Skip to contentSkip to conversation

perf tools: Annotate fixes, stdio progress indication, debuginfo-client in more places

PATCH Superseded Arnaldo Carvalho de Melo 6 participants 42 messages Sep 16 – Sep 22 perf events

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Arnaldo Carvalho de Melo

author maintainer
#0 lore raw

v6 cover perf tools: Annotate fixes, stdio progress indication, debuginfo-client in more places

Hi,

This came out of work on data type profiling, but none of them depends on that work and nothing in this series needs anything from it, so sending them separately.

The fixes:

- 'perf report -s type' spins forever, burning all of a CPU with no output, on the dwz compressed debug info of zlib-ng (libz.so.1):
die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, the dwz common file for the types shared by more than one CU, and resolving one of those offsets in the main debug file does not fail, it parses whatever is at that offset there, in this case a typedef whose DW_AT_type refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin. The offset is now resolved in the file it was recorded as coming from, which the CU of the collected DIE tells exactly rather than having to be inferred from what is at the offset, and the type chases and the struct/union member nesting are bounded, so a debug info file broken in some other way makes perf give up on a type with a pr_debug instead of looking like it hung (patch 10);

- the data type browser prints, in its samples view (-n, annotate.show_nr_samples), a local variable initialized to zero and never updated, so every member shows up as having no samples while the period and percent columns for the same entry are filled in (patch 7);

- the 'perf data type profiling' shell test turns "this PMU cannot record these events" into a test failure: the script runs under 'set -e', so the bare 'perf mem record' aborts it through the EXIT trap, which reports a signal that never happened, instead of reaching the code right below meant to report the failure (patch 1).

The features:

- 'perf report --progress' prints the phase it is in and how far along it is to stderr: the TUI shows that with a progress bar, but in the stdio case the ui_progress updates that are already there are dropped on the floor. It is what makes a slow run tell itself apart from a stuck one (patch 8);

- the debuginfo for a DSO, and the kernel symbols, can now be fetched keyed by the build ID recorded in the perf.data file, using the debuginfod client, for the cases where they are not available locally under the name the DSO was opened with: a vmlinux for a kernel that since got upgraded, or a profile recorded on another machine. It is off with --no-debuginfod, with core.debuginfod=false, and when the build-id cache is turned off (patch 2), opt-outs that also clear DEBUGINFOD_URLS, that libdwfl's own debuginfod client reads, so that it doesn't fetch behind perf's back;

- a fetch that ends up taking a while, e.g. that vmlinux, or one for the vmlinux of a kernel profiled on another machine, doesn't leave the user staring at a stuck looking terminal: in the stdio interface, the progress is a line on stderr, in the TUI a window over the browser, and 's' aborts the current fetch while remembering the build ID, so that the rest of the session doesn't ask for it again, the user may have skipped it for being too big; 'd' additionally disables debuginfod for the rest of the session, clearing DEBUGINFOD_URLS so that libdwfl's own client stops too, and writes core.debuginfod=false into the configuration file perf is using, ~/.perfconfig or the one named by PERF_CONFIG, making it permanent. This is spread over three patches, with the move of perf_config__set_variable() to util/config.c as a prep patch, the stdio keys and the TUI window (patches 3 to 5);

- 'perf mem record' asks for PERF_SAMPLE_CPU by default, as without it the cpu field is the (u32)-1 "no CPU info" sentinel and per-sample analysis cannot tell accesses from different cores apart from same-CPU traffic (patch 11), and uses the IBS swfilt software privilege filter, when the kernel exposes it, so that per-thread recording works where only system wide recording did (patch 12).

Which file a saved type DIE offset belongs to is settled with dwarf_cu_getdwarf(), new in elfutils 0.160, so the libdw feature test now probes for it and the message in Makefile.config says 0.160 where it said 0.157: 0.157 to 0.159 is from 2014 and has no such symbol, and those versions now lose dwarf support with that message instead of failing to link util/dwarf-aux.c.

Best regards,

- Arnaldo

What changed from v5 (ef0ac8e44d43d2aa86842d2f83ad41d1c058d949):

PATCH 2/9, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod", was split [Namhyung Kim review of PATCH v3 2/8, <[email protected]>]: the fetch stays in patch 2, the way to get out of a slow fetch, that was folded at the end of patch 2 in v5, is now patches 3 to 5, each separately cherry-pickable, and carrying on their own the answer to his "I'm not sure what would be the good default. But with this, it can slow down the process especially when the binary is not in the debuginfod": on by default, with the way to skip the one in progress and to disable the feature right where the wait is felt. Patch 3, "perf config: Move perf_config__set_variable() to util/config.c", is the prep patch, patch 4, "perf debuginfo: Let the user skip and disable debuginfod fetches", has the stdio 's'/'d' keys, and patch 5, "perf debuginfo:
Show the debuginfod fetch progress and keys in the TUI", the TUI window over the browser.

PATCH 4/12, "perf debuginfo: Let the user skip and disable debuginfod fetches" [that same review of the 'd' key semantics]:

- 'd' now writes core.debuginfod=false into the configuration file directly, the user's ~/.perfconfig or the one named by PERF_CONFIG, via perf_config__set_variable(), moved from builtin-config.c to util/config.c by the prep patch so that util/debuginfo.c can use it, and clears DEBUGINFOD_URLS so that libdwfl's own debuginfod client stops for the rest of the session too: in v5 it only disabled debuginfod for the rest of the session and pointed at 'perf config' to make that permanent.

PATCH 5/12, "perf debuginfo: Show the debuginfod fetch progress and keys in the TUI":

- The progress and the keys are no longer stdio-only: printing to stderr would garble the TUI browser display, so in v5 a fetch in progress in the TUI showed nothing and there was no way to skip or disable it, the thing that makes a slow fetch indistinguishable from a stuck perf. The TUI now draws a window over the browser with the fetch progress and drains the 's'/'d' keys from the TUI input queue: the TUI owns the terminal and its input queue, and the browser thread is the one doing the fetch, so nobody else is reading from it while the fetch blocks it. No terminal settings or signal dispositions are touched in that case, the terminal is the TUI's own and so are the SIGINT/SIGTERM handlers, that restore it before exiting.

- The slang calls stay in ui/tui/, behind the ui__key_pending()/ ui__key_read()/ui__progress_window()/ui__progress_window_end() primitives declared in ui/util.h and implemented in ui/tui/util.c, with inline stubs for builds without slang: the plan is to reduce the number of libraries perf needs to build, or swap slang for something else, such as ncurses, at some point, so nothing outside ui/ calls slang directly. The text in the window is word wrapped to the available columns, as constrained terminals, such as a smartphone running termux, crop the tail of a long line and would hide the 'd' option hint.

Patches 1, 3, 4, 5, 6, 7, 8 and 9 are unchanged from v5, renumbered as 1, 6, 7, 8, 9, 10, 11 and 12 with the split.

What changed from v4 (f801de4e80289566):

PATCH 1/8, "perf test: Skip data_type_profiling when the PMU cannot record memory events" [Namhyung Kim review of PATCH v3 1/8]:

- Namhyung suggested adding a fallback to the 'swfilt' term for ibs_op instead of only skipping the test on a PMU that refuses per-thread memory events. The new PATCH 9 does that for the ibs_op memory events, using the swfilt software privilege filter the kernel exposes as a format term since v6.14, which makes the per-thread 'perf mem record' the test does work on AMD; the skip stays for kernels without it. The term is in the event name even for records that end up without exclude bits, e.g. plain system wide ones: 'perf record' adds those bits itself when the first open fails with EACCES, after the name was built, and the retry only succeeds with the term already in the name:

$ perf record -v -e ibs_op/ldlat=0/ -- true
kernel.perf_event_paranoid=2, trying to fall back to excluding kernel and hypervisor samples Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed. Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

with no exclude bits the kernel discards nothing. The 'Test data symbol' shell test matches the event string, so its regex now accepts terms added after ldlat=150.

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v4 2/8]:

- debuginfod__setup_urls_env() moved from util/debuginfo.c to util/util.c, next to perf_debuginfod_setup(), and its declaration from util/debuginfo.h to util/util.h: debuginfo.o is only built with CONFIG_LIBDW, so with libdebuginfod but without libdw the declaration was an empty stub and the setup was skipped exactly for build-id.c and probe-event.c, the debuginfod users that are built without libdw and that the commit message lists as reading DEBUGINFOD_URLS in every debuginfod_begin().

PATCH 8/8, "perf mem record: Request PERF_SAMPLE_CPU by default" [sashiko-bot review of PATCH v4 8/8]:

- The rec_argv array now reserves room for the fixed arguments __cmd_record() adds, up to eight with all the optional flags; a pre-existing under-allocation, the array was nine arguments per memory PMU plus the user arguments, and on a PMU with separate load and store events the four event arguments plus the fixed ones already filled it to the last slot, so this new --sample-cpu argument would write past it, e.g. with --phys-data --data-page-size --all-user.

Patches 1, 3, 4, 5, 6 and 7 are unchanged from v4: sashiko-bot found no issues in them.

What changed from v3 (2433e9f0bf86cfc3):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v3 2/8]:

- DEBUGINFOD_URLS is no longer set from the fetch path: setenv() is not thread safe and libdebuginfod reads it with getenv() in every debuginfod_begin(), which perf also does from build-id.c, probe-event.c and probe-finder.c, paths that don't go through the fetch lock, so a thread taking one of them could race with the setenv() a fetch was doing. This is now done from symbol__init(), before the threads are started, from the .urls files in /etc/debuginfod when it isn't set, and the pthread_once() and the call in debuginfod__fetch() are gone with it.

- The in-flight fetch wait was dead code and is gone: with debuginfod__fetch_lock held across the whole lookup and fetch, no second thread could ever see an entry being fetched, so the condition variable, the waiters and the publication dance could not run. debuginfo_lookup__new() became debuginfo_lookup__add(), called after a fetch that brought a file back: lookups remember only the successful fetches, one entry each, answered with a strdup() and re-fetched if that file goes away, and a request for a build ID being fetched waits for the lock and is answered from the misses list or from the entry, so a download still can't happen twice. debuginfod__missed_lock, always taken under the fetch lock, is gone with it.

PATCH 3/8, "perf symbol: Fall back to fetching the vmlinux by build ID" [sashiko-bot review of PATCH v3 3/8]:

- The check for a concurrent load, made after taking dso->lock back from the fetch, is now dso__has_symbols() and not dso__loaded():
dso__load() sets the latter even for an attempt that failed, so a thread whose fetch failed could make a thread whose fetch succeeded discard the vmlinux it just downloaded and carry on with an empty kernel dso, as loaded is set and nothing would try again.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH v3 6/8]:

- The review asked whether the %s of a dwarf_diename() that returns NULL in perf-die-chain would abort the macro. Checked, not changed:
with GDB 17.2, printf prints "(null)" for a NULL char *, including for an inferior call returning one, and the macro carries on. The other %s in the file, dloc->ms->sym->name in perf-dso, can't be NULL
either: the only caller of find_data_type() dereferences ms->sym and ms->map to build the data_loc_info before that frame can exist, so no change there as well.

- Patches 1, 4, 5, 7 and 8 are unchanged from v3: sashiko-bot found no issues in them.

What changed from v2 (00cffc7df5abfa68):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v2 2/8]:

- The conditions to fetch, debuginfod being enabled, the build-id cache being on and the build ID not having been a miss already, are now looked at with the fetch lock held, as they are the state the fetch itself changes: deciding them outside the lock and fetching inside it let a second thread repeat the fetch the first one had just made, or repeat one that had just been turned off, and put the same build ID on the misses list twice.

- A build ID that is already being fetched is waited for, instead of fetched again: there was nothing that said "this one is in flight", so a second thread wanting it blocked on the fetch lock and then did its own complete fetch, which, while the first one is still running, is a second download of the same file, with a second client, a second progress line and a second turn at the terminal, for the same answer. It now sleeps on a condition variable and is woken with whatever the fetch settled: the file, when there is one, or the not available, when it found nothing or the user cancelled it, with no retry.

- And a fetch that brought a file back is remembered, so that the ones that need the same build ID later, e.g. annotating another symbol of the same DSO, are answered with a strdup() instead of another client:
the file stays in the debuginfod client cache, and its path is checked before being handed out, in case that cache gets cleaned from under us.

- The fetches are still one at a time, so a fetch for one build ID waits for a fetch for another one: the terminal mode, the signal dispositions, the progress line and the 's'/'d' keys are process global, and sharing them between concurrent fetches, refcounted, is left for later, noted in the code. It matters less than it sounds, now that the build IDs of a workload are answered from memory after the first pass over them.

PATCH 3/8, "perf symbol: Fall back to fetching the vmlinux by build ID" [sashiko-bot review of PATCH v2 3/8]:

- The fetch is no longer made with dso->lock held: dso__load() holds it across dso__load_kernel_sym(), and a fetch can block for a long time on the network, or on the terminal, waiting for the user, which would stall every other thread that needs the kernel dso. It is dropped just around the fetch, nothing of the dso is touched while it is, and the symbols are loaded with the lock held again, the same way dso__debuginfo() already does it for the debuginfo of a DSO. If another thread got the symbols for the dso in the meantime, the file that came back is not loaded a second time.

PATCH 5/8, "perf report: Add --progress option" [sashiko-bot review of PATCH v2 5/8]:

- A phase that starts when the progress stack is full is now just not shown, and its finish() is swallowed, instead of force completing the phase that encloses it to make room: that one still gets its own finish() later, which would then complete the phase above it, and so on, one premature completion per nesting level. Reproduced by building with STDIO_PROGRESS__MAX_DEPTH set to 1: v2 prints "Processing events... [100.0%]" while it is at 99.7%, before the nested phase even runs, v3 warns and keeps the outer phase to complete when it really does. With the 8 slots in the tree this cannot be hit, the phases perf has nest at most 2 deep, but the bookkeeping was wrong.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH v2 6/8]:

- The perf-dso GDB macro read dloc->ms.map->dso->name and

    dloc->ms.sym->name, but struct data_loc_info.ms is a
    'struct map_symbol *', so GDB's C evaluator needs -> there: as it
    was, 'perf-stuck.sh -g' would have failed to evaluate it.

- The one pre-existing issue in PATCH 7/8, the unbounded member type chase in die_get_member_type() and the unbounded recursion in die_find_member(), is not addressed here, it went to the perf TODO list as item 180, as it is not on the path this series fixes.

Other changes since v2:

- PATCH 2/8: a refetch made because the cached file was cleaned from under us now updates the entry the same way the first fetch does, and an entry with no path and no error is fetched again instead of answered; as it was, the entry kept the first fetch's success with no path, so the later lookups it existed for returned success with a NULL path, which debuginfo__new_build_id() cannot open and the vmlinux fallback in PATCH 3 would take as a file that came back and dereference.

- PATCH 3/8: the commit message said the ignore_vmlinux_buildid that 'perf record' and 'perf probe' set internally is what keeps them away from the fallback, but 'perf probe' only sets it for the commands other than --list, --del and --add when given an offline vmlinux; the message now says that and that a 'perf probe' that does not set it, e.g. --funcs or --add without --vmlinux on a system with a restricted /proc/kallsyms, can have the kernel debuginfo fetched, like the other tools.

- PATCH 7/8: the kerneldoc of die_get_type_die() said the type would be looked up in the main file and then in the alt file, using the one that has a DIE with the saved tag, but the function deliberately resolves in only the file the offset was recorded as belonging to, with no fallback, and @from_alt was missing from the parameter list; it now describes that, and the die_collect_vars() and die_collect_global_vars() kerneldocs, which still said the type could be retrieved with dwarf_offdie() and the offset alone, now point at die_get_type_die().

- PATCH 7/8: the comment on the member->truncated flag said the browser reads it, but the browser picks between ';' and '{' from the children list, so a member cut by the nesting limit is drawn like a complete leaf; the comment now says the flag is for the JSON exporter added in a later series.

- PATCH 2/8: a search aborted by the user, with 's' or a signal, is now remembered for the rest of the session: the threads already waiting for that fetch were told there was nothing to share, but a request that arrived later started a new download of the same file, which the user may have skipped for being too big. debuginfod__misses now records such a cancellation as a cancellation, the 's' message says the build ID will not be fetched again in this session and the debug message tells a cancellation and a server miss apart. The lookup entry is also now allocated and published by a debuginfo_lookup__new() helper called before the fetch, instead of an allocation whose failure was only checked after it.

- Patches 1, 4 and 8 are unchanged from v2: sashiko-bot found no issues in them.

What changed from v1 (060dccedab1617dc):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH 2/8]:

- Included <limits.h> for PATH_MAX, that was coming in through a glibc transitive include and fails to build with musl libc.

- debuginfod__setup_urls_env() now goes through pthread_once, so that the setenv() of DEBUGINFOD_URLS happens exactly once: setenv() is not thread safe and this is on the fetch path, which dso__debuginfo() deliberately takes outside dso__lock.

- The fetch now runs with debuginfod__fetch_lock held, so that the process global state it uses — the terminal settings, the SIGINT/ SIGTERM dispositions and the progress and cancellation state — is only ever touched by one fetch at a time. A second concurrent fetch would otherwise have taken the first one's raw mode as the state to restore, leaving the terminal broken when it was done, and would reset the cancellation state from under the fetch already in progress. Serializing also keeps two fetches from racing for the same keypresses and the same progress line, and a second fetch has nothing to gain from running in parallel with a first one reading the same kind of file off the same servers.

- An interrupt that lands after the fetch already succeeded is no longer swallowed: at that point the terminal and the signal dispositions are the original ones again, so the signal is re-raised, the same way the failure path does, instead of perf carrying on as if the user had not asked it to stop.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH 6/8]:

- /proc/<pid>/stat is now parsed with the parenthesized command name removed first, so that a process with spaces in its name no longer shifts every field after it: watching a process named 'my sleep' used to report "sleep)" as its state and a garbage RSS.

- The CPU time is printed with awk's printf "%d" instead of relying on its default output format, that switches to scientific notation, which the bash arithmetic below cannot parse, once the sum of utime and stime goes past six digits, i.e. some 16 minutes of CPU at 100 Hz.

- The process can go away between the '-d /proc/<pid>' check and the read of /proc/<pid>/stat, so the read is now checked: a failed or empty one reports "process gone" instead of 'set -u' aborting the script on an unbound field, which is what would have hidden the very exit the script is meant to report.

- The command line printed when the script starts has its control characters stripped, so that a process started with escape sequences in its arguments, e.g. one replaying a log line, does not get them replayed on the terminal of whoever runs this.

- The two pre-existing issues sashiko-bot raised that are outside the scope of this series, the child entries and their hists arrays leaked by the data type browser teardown and the missing <stdio.h> and <stdlib.h> includes in ui/browsers/annotate-data.c, are recorded as items 178 and 179 of the perf TODO list for follow-up work.

- Patches 1, 3, 4, 5, 7 and 8 are unchanged from v1: sashiko-bot found no issues in them.

Arnaldo Carvalho de Melo (12):
perf test: Skip data_type_profiling when the PMU cannot record memory
events
perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod perf config: Move perf_config__set_variable() to util/config.c perf debuginfo: Let the user skip and disable debuginfod fetches perf debuginfo: Show the debuginfod fetch progress and keys in the TUI perf symbol: Fall back to fetching the vmlinux by build ID perf annotate-data: Show the sample count in the data-type browser perf report: Add --progress option
perf scripts: Add perf-stuck, to tell where a running perf is stuck perf annotate-data: Resolve type DIEs in the debug file they came from perf mem record: Request PERF_SAMPLE_CPU by default
perf mem record: Use the IBS swfilt filter when available

 tools/build/feature/test-libdw.c              |  15 +-
 tools/perf/Documentation/perf-annotate.txt    |  10 +
 tools/perf/Documentation/perf-config.txt      |  19 +
 tools/perf/Documentation/perf-mem.txt         |   4 +
 tools/perf/Documentation/perf-report.txt      |  31 +
 tools/perf/Documentation/perf-top.txt         |  16 +
 tools/perf/Makefile.config                    |   2 +-
 tools/perf/arch/x86/util/mem-events.c         |  17 +-
 tools/perf/builtin-annotate.c                 |   3 +
 tools/perf/builtin-config.c                   |  70 +-
 tools/perf/builtin-mem.c                      |  18 +-
 tools/perf/builtin-report.c                   |  18 +
 tools/perf/builtin-top.c                      |   7 +
 tools/perf/scripts/perf-stuck.gdb             | 104 +++
 tools/perf/scripts/perf-stuck.sh              | 194 ++++++
 tools/perf/tests/shell/data_type_profiling.sh |  46 +-
 tools/perf/tests/shell/test_data_symbol.sh    |   6 +-
 tools/perf/ui/Build                           |   1 +
 tools/perf/ui/browsers/annotate-data.c        |   2 +-
 tools/perf/ui/progress.h                      |   2 +
 tools/perf/ui/stdio/progress.c                | 185 ++++++
 tools/perf/ui/tui/util.c                      | 177 +++++
 tools/perf/ui/util.h                          |  42 ++
 tools/perf/util/annotate-data.c               |  59 +-
 tools/perf/util/annotate-data.h               |   3 +
 tools/perf/util/config.c                      |  73 +++
 tools/perf/util/config.h                      |   2 +
 tools/perf/util/debuginfo.c                   | 608 ++++++++++++++++++
 tools/perf/util/debuginfo.h                   |  36 ++
 tools/perf/util/dso.c                         |  19 +
 tools/perf/util/dwarf-aux.c                   | 171 ++++-
 tools/perf/util/dwarf-aux.h                   |  33 +
 tools/perf/util/mem-events.c                  |  21 +-
 tools/perf/util/mem-events.h                  |   2 +
 tools/perf/util/ordered-events.c              |  17 +-
 tools/perf/util/session.c                     |  12 +-
 tools/perf/util/symbol.c                      |  83 ++-
 tools/perf/util/symbol_conf.h                 |   1 +
 tools/perf/util/util.c                        | 103 +++
 tools/perf/util/util.h                        |   8 +
 40 files changed, 2105 insertions(+), 135 deletions(-)

base-commit: 91b0782fc9e9d2f0a40b5256146e014802fdbb36
v5-head: ef0ac8e44d43d2aa86842d2f83ad41d1c058d949
v4-head: f801de4e80289566670e6b735198606b672e74e8
v3-head: 2433e9f0bf86cfc3d9966b467042563596c9ebdb
v2-head: 00cffc7df5abfa6850beb360deb830d8495163ce
v1-head: 060dccedab1617dcbf6e2be64ed97afe3f3fac63 --

Assisted-by LLM
author maintainer
#1 lore raw

PATCH v6 1/12 perf test: Skip data_type_profiling when the PMU cannot record memory events

From: Arnaldo Carvalho de Melo <[email protected]>

The test records with 'perf mem record' in per-thread mode and its only guard matches one specific message:

  perf mem record -o /dev/null -- true 2>&1 | \
        grep -q "failed: no PMU supports the memory events" && exit 2

A PMU that has memory events but refuses them per-thread falls through it, and AMD IBS does:

  $ perf mem record -o /dev/null -- true
  Error:
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.
  Error:
  Failure to open any events for recording.

What follows is not a skip either. The script runs under 'set -e', so the bare 'perf mem record' in test_basic_annotate() aborts it through the EXIT trap, which reports a signal that never happened and exits 1:

Basic Rust perf annotate test
Unexpected signal in test_basic_annotate

so 'perf test' turns "this PMU cannot record these events" into a test failure:

  $ perf test "data type profiling"
   87: perf data type profiling tests             : FAILED!

The 'if [ "x$?" != "x0" ]' right below the record was meant to report "[Failed: perf record]" and carry on with the other cases, but under 'set -e' it is dead code: the shell is gone before it is reached. The same goes for the 'perf annotate' call below it.

Skip when a trivial per-thread 'perf mem record' fails, and put both commands in the condition of an 'if', where 'set -e' leaves them alone, so that the
reporting works for the cases where only one workload cannot be recorded or annotated:

  $ perf test "data type profiling"
   87: perf data type profiling tests             : Skip

and, with just the new skip guard disabled on the same box, every case reports its failure instead of the first one aborting the run:

Basic Rust perf annotate test
Basic annotate [Failed: perf record]
Pipe Rust perf annotate test
Pipe annotate [Failed: perf record]
Basic C perf annotate test
Basic annotate [Failed: perf record]
Pipe C perf annotate test
Pipe annotate [Failed: perf record]

Fixes f60a5c22967b ("perf tests: Test annotate with data type profiling and rust")
Assisted-by LLM
---
 tools/perf/tests/shell/data_type_profiling.sh | 46 +++++++++++++++----
 1 file changed, 36 insertions(+), 10 deletions(-)
tools/perf/tests/shell/data_type_profiling.sh +36 -10
index eca694600a0478d2..a916c410274aa888 100755
@@ -19,6 +19,15 @@ perfout=$(mktemp /tmp/__perf_test.perf.out.XXXXX)
 perf mem record -o /dev/null -- true  2>&1 | \
   		grep -q "failed: no PMU supports the memory events" && exit 2
 
+# Skip if per-thread mem record is not supported on this PMU (e.g. AMD IBS
+# needs system-wide '-a'): it is what the test records with below, and a
+# failing record must not be reported as a test failure.
+if ! perf mem record -o /dev/null -- true 2>/dev/null
+then
+  echo "Skip: cannot record memory events on this PMU"
+  exit 2
+fi
+
 cleanup() {
   rm -rf "${perfdata}" "${perfout}"
   rm -rf "${perfdata}".old
@@ -52,25 +61,42 @@ test_basic_annotate() {
     index=1 ;;
   esac
 
+  # Under 'set -e' a bare failing command aborts the script through the EXIT
+  # trap, so the commands that report a failure have to be the condition of
+  # an 'if' for that reporting to ever happen.
   if [ "x${mode}" == "xBasic" ]
   then
-    perf mem record -o "${perfdata}" ${testprogs[$index]} 2> /dev/null
+    if ! perf mem record -o "${perfdata}" ${testprogs[$index]} 2> /dev/null
+    then
+      echo "${mode} annotate [Failed: perf record]"
+      err=1
+      return
+    fi
   else
-    perf mem record -o - ${testprogs[$index]} 2> /dev/null > "${perfdata}"
-  fi
-  if [ "x$?" != "x0" ]
-  then
-    echo "${mode} annotate [Failed: perf record]"
-    err=1
-    return
+    if ! perf mem record -o - ${testprogs[$index]} 2> /dev/null > "${perfdata}"
+    then
+      echo "${mode} annotate [Failed: perf record]"
+      err=1
+      return
+    fi
   fi
 
   # Generate the annotated output file
   if [ "x${mode}" == "xBasic" ]
   then
-    perf annotate --code-with-type -i "${perfdata}" --stdio --percent-limit 1 2> /dev/null > "${perfout}"
+    if ! perf annotate --code-with-type -i "${perfdata}" --stdio --percent-limit 1 2> /dev/null > "${perfout}"
+    then
+      echo "${mode} annotate [Failed: perf annotate]"
+      err=1
+      return
+    fi
   else
-    perf annotate --code-with-type -i - --stdio 2> /dev/null --percent-limit 1 < "${perfdata}" > "${perfout}"
+    if ! perf annotate --code-with-type -i - --stdio 2> /dev/null --percent-limit 1 < "${perfdata}" > "${perfout}"
+    then
+      echo "${mode} annotate [Failed: perf annotate]"
+      err=1
+      return
+    fi
   fi
 
   # check if it has the target data type
Signature · 1 line
2.55.0
author maintainer
#3 lore raw

PATCH v6 2/12 perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.

The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set. That has to happen before any thread that can call getenv() is started, as setenv() is not thread safe, and there are getenv()s outside the fetch
lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(), which perf also does in build-id.c, probe-event.c and probe-finder.c, so do it from symbol__init(), on the single-threaded setup, and not lazily from the fetch path.

Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.

The opt-outs also cover libdwfl's own debuginfod client, that reads DEBUGINFOD_URLS in every query: when debuginfod is off, be it with --no-debuginfod, core.debuginfod=false or by disabling the build-id cache, the variable is set to the empty string, that the client treats as an opt-out and fails the query without even looking at its cache, instead of being exported from /etc/debuginfod. Tools that manage DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep doing so.

The fetches are serialized, as the lookup state they use is process global, so two concurrent fetches would race for it, and serializing also means that a request for a build ID that is being fetched doesn't start a second download of the same file: it waits for the fetch to finish and is answered from the entry the fetch published, keyed by the build ID, with the path of the file that came back, checked before being handed out in case the debuginfod client cache is cleaned from under us.

Make dso__debuginfo() use debuginfo__new_build_id() as a fallback, keyed by the build ID recorded in the perf.data file, so that consumers such as the data type profiler can resolve the types of DSOs whose debuginfo can be fetched this way. Do the fetch outside dso__lock, a fetch from a remote debuginfod server can take a while and would otherwise block anything else using this dso, and remember the build IDs that were a miss, so that consumers revisiting a set of DSOs repeatedly, such as the data type profiler on every hist entry DSO switch, don't pay server round trips per attempt.

debuginfo__new_build_id() needs libdw to open the DWARF, so it lives in debuginfo.o, built only with CONFIG_LIBDW, while the libdebuginfod feature check is independent of NO_LIBDW; keep the new build ID prototypes under HAVE_LIBDW_SUPPORT too, with stubs otherwise, so that make NO_LIBDW=1 on a system that has the debuginfod client keeps linking.

---
 tools/perf/Documentation/perf-annotate.txt |   8 +
 tools/perf/Documentation/perf-config.txt   |  19 ++
 tools/perf/Documentation/perf-report.txt   |  12 +
 tools/perf/Documentation/perf-top.txt      |  11 +
 tools/perf/builtin-annotate.c              |   3 +
 tools/perf/builtin-report.c                |   7 +
 tools/perf/builtin-top.c                   |   7 +
 tools/perf/util/config.c                   |   3 +
 tools/perf/util/debuginfo.c                | 327 +++++++++++++++++++++
 tools/perf/util/debuginfo.h                |  36 +++
 tools/perf/util/dso.c                      |  19 ++
 tools/perf/util/symbol.c                   |  12 +
 tools/perf/util/symbol_conf.h              |   1 +
 tools/perf/util/util.c                     | 103 +++++++
 tools/perf/util/util.h                     |   8 +
 15 files changed, 576 insertions(+)
tools/perf/Documentation/perf-annotate.txt +8 -0
index 1a90b09a12d5abb1..12edc3337b86f099 100644
@@ -58,6 +58,14 @@ OPTIONS
 --ignore-vmlinux::
 	Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+	Fetch debuginfo keyed by build ID from the debuginfod servers
+	configured in DEBUGINFOD_URLS, checking the local debuginfod
+	client cache first, when it is not available locally, on for
+	these commands by default.  See the --debuginfod option of
+	'perf report' for how to turn it off.
+
 --itrace::
 	Options for decoding instruction tracing data. The options are:
 
tools/perf/Documentation/perf-config.txt +19 -0
index 9b223f8928299945..688306abe847a0df 100644
@@ -216,6 +216,14 @@ core.*::
 	addr2line-timeout::
 		Sets a timeout (in milliseconds) for parsing 'addr2line'
 		output.  The default timeout is 5s.
+	debuginfod::
+		When set to 'false', disable fetching debuginfo keyed by
+		build ID from the debuginfod servers configured in
+		DEBUGINFOD_URLS.  It is on by default, can be overridden per
+		tool with the 'report.debuginfod' and 'top.debuginfod'
+		options and per invocation with --no-debuginfod; it is off
+		too when the local build-id cache is disabled, e.g.
+		'buildid.dir' set to /dev/null.
 
 tui.*, gtk.*::
 	Subcommands that can be configured here are 'top', 'report' and 'annotate'.
@@ -562,6 +570,14 @@ report.*::
 		This option can change default stat behavior with empty results.
 		If it's set true, 'perf report --stat' will not show 0 stats.
 
+	report.debuginfod::
+		Fetch debuginfo keyed by build ID from the debuginfod
+		servers configured in DEBUGINFOD_URLS, checking the local
+		debuginfod client cache first, when it is not available
+		locally.  On by default, set to 'false' to disable it for
+		'perf report', globally with 'core.debuginfod=false' or per
+		invocation with --no-debuginfod.
+
 top.*::
 	top.children::
 		Same as 'report.children'. So if it is enabled, the output of 'top'
@@ -569,6 +585,9 @@ top.*::
 		column by default.
 		The default is 'true'.
 
+	top.debuginfod::
+		Same as 'report.debuginfod', for 'perf top'.
+
 	top.call-graph::
 		This is identical to 'call-graph.record-mode', except it is
 		applicable only for 'top' subcommand. This option ONLY setup
tools/perf/Documentation/perf-report.txt +12 -0
index 1a4706329c6cef1d..2e11c0a97a016853 100644
@@ -366,6 +366,18 @@ OPTIONS
 --ignore-vmlinux::
 	Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+	Fetch debuginfo keyed by build ID from the debuginfod servers
+	configured in DEBUGINFOD_URLS, checking the local debuginfod
+	client cache first, when it is not available locally, on for
+	these commands by default.  It can be turned off per invocation
+	with --no-debuginfod, per tool with the "report.debuginfod"
+	config option or globally with "core.debuginfod" set to false.
+	It is disabled as well when the local build-id cache is turned
+	off, e.g. "buildid.dir" set to /dev/null, as that asks for
+	fetched files not to be kept on the box.
+
 --kallsyms=<file>::
         kallsyms pathname
 
tools/perf/Documentation/perf-top.txt +11 -0
index 2da2a16bbf260685..345dd9ee9b6bab88 100644
@@ -83,6 +83,17 @@ Default is to monitor all CPUS.
 --ignore-vmlinux::
 	Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+	Fetch debuginfo keyed by build ID from the debuginfod servers
+	configured in DEBUGINFOD_URLS, checking the local debuginfod
+	client cache first, when it is not available locally, on for
+	these commands by default.  Turn it off per invocation with
+	--no-debuginfod, with the "top.debuginfod" config option or
+	globally with "core.debuginfod" set to false.  Disabled as well
+	when the build-id cache is off, e.g. "buildid.dir" set to
+	/dev/null.
+
 --kallsyms=<file>::
 	kallsyms pathname
 
tools/perf/builtin-annotate.c +3 -0
index 850fc72fa75fe7df..d6a201c1866d363f 100644
@@ -733,6 +733,9 @@ int cmd_annotate(int argc, const char **argv)
 	OPT_BOOLEAN(0, "stdio2", &annotate.use_stdio2, "Use the stdio interface"),
 	OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
                     "don't load vmlinux even if found"),
+	OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+		    "fetch debuginfo keyed by build ID from the debuginfod "
+		    "servers, on by default, use --no-debuginfod to turn off"),
 	OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name,
 		   "file", "vmlinux pathname"),
 	OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
tools/perf/builtin-report.c +7 -0
index 279e61c2366cb294..442c0822e614197f 100644
@@ -134,6 +134,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		symbol_conf.event_group = perf_config_bool(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.debuginfod")) {
+		symbol_conf.debuginfod = perf_config_bool(var, value);
+		return 0;
+	}
 	if (!strcmp(var, "report.percent-limit")) {
 		double pcnt = strtof(value, NULL);
 
@@ -1346,6 +1350,9 @@ int cmd_report(int argc, const char **argv)
 		   "file", "vmlinux pathname"),
 	OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
                     "don't load vmlinux even if found"),
+	OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+		    "fetch debuginfo keyed by build ID from the debuginfod "
+		    "servers, on by default, use --no-debuginfod to turn off"),
 	OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
 		   "file", "kallsyms pathname"),
 	OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
tools/perf/builtin-top.c +7 -0
index c2562d49be46a9a1..2be6859fd4aee141 100644
@@ -1436,6 +1436,10 @@ static int perf_top_config(const char *var, const char *value, void *cb __maybe_
 		symbol_conf.cumulate_callchain = perf_config_bool(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "top.debuginfod")) {
+		symbol_conf.debuginfod = perf_config_bool(var, value);
+		return 0;
+	}
 
 	return 0;
 }
@@ -1508,6 +1512,9 @@ int cmd_top(int argc, const char **argv)
 		   "file", "vmlinux pathname"),
 	OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
 		    "don't load vmlinux even if found"),
+	OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+		    "fetch debuginfo keyed by build ID from the debuginfod "
+		    "servers, on by default, use --no-debuginfod to turn off"),
 	OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
 		   "file", "kallsyms pathname"),
 	OPT_BOOLEAN('K', "hide_kernel_symbols", &top.hide_kernel_symbols,
tools/perf/util/config.c +3 -0
index b2972c35c1eca68c..31c6618d3b3daf22 100644
@@ -470,6 +470,9 @@ static int perf_default_core_config(const char *var, const char *value)
 	if (!strcmp(var, "core.addr2line-disable-warn"))
 		symbol_conf.addr2line_disable_warn = perf_config_bool(var, value);
 
+	if (!strcmp(var, "core.debuginfod"))
+		symbol_conf.debuginfod = perf_config_bool(var, value);
+
 	/* Add other config variables here. */
 	return 0;
 }
tools/perf/util/debuginfo.c +327 -0

349 lines not on this page · raw message

tools/perf/util/debuginfo.h +36 -0
index a52d69932815cd72..43b211a0dec174f5 100644
@@ -5,6 +5,8 @@
 #include <errno.h>
 #include <linux/compiler.h>
 
+struct build_id;
+
 #ifdef HAVE_LIBDW_SUPPORT
 
 #include "dwarf-aux.h"
@@ -54,6 +56,28 @@ static inline int debuginfo__get_text_offset(struct debuginfo *dbg __maybe_unuse
 #ifdef HAVE_DEBUGINFOD_SUPPORT
 int get_source_from_debuginfod(const char *raw_path, const char *sbuild_id,
 			       char **new_path);
+
+/*
+ * Finding a debuginfo file keyed by build ID uses the debuginfod client,
+ * but opening the DWARF in it needs libdw, i.e. these live in
+ * debuginfo.o, which is only built with CONFIG_LIBDW.
+ */
+#ifdef HAVE_LIBDW_SUPPORT
+int debuginfo__find_build_id(const struct build_id *bid, char **path);
+struct debuginfo *debuginfo__new_build_id(const struct build_id *bid);
+#else
+static inline int debuginfo__find_build_id(const struct build_id *bid __maybe_unused,
+					   char **path __maybe_unused)
+{
+	return -ENOTSUP;
+}
+
+static inline struct debuginfo *
+debuginfo__new_build_id(const struct build_id *bid __maybe_unused)
+{
+	return NULL;
+}
+#endif /* HAVE_LIBDW_SUPPORT */
 #else /* HAVE_DEBUGINFOD_SUPPORT */
 static inline int get_source_from_debuginfod(const char *raw_path __maybe_unused,
 					     const char *sbuild_id __maybe_unused,
@@ -61,6 +85,18 @@ static inline int get_source_from_debuginfod(const char *raw_path __maybe_unused
 {
 	return -ENOTSUP;
 }
+
+static inline int debuginfo__find_build_id(const struct build_id *bid __maybe_unused,
+					   char **path __maybe_unused)
+{
+	return -ENOTSUP;
+}
+
+static inline struct debuginfo *
+debuginfo__new_build_id(const struct build_id *bid __maybe_unused)
+{
+	return NULL;
+}
 #endif /* HAVE_DEBUGINFOD_SUPPORT */
 
 #endif /* _PERF_DEBUGINFO_H */
tools/perf/util/dso.c +19 -0
index 42bfe30a3b518e80..fe7c3b0265ba633a 100644
@@ -32,6 +32,7 @@
 #include "string2.h"
 #include "vdso.h"
 #include "annotate-data.h"
+#include "debuginfo.h"
 #include "libdw.h"
 
 static const char * const debuglink_paths[] = {
@@ -2073,5 +2074,23 @@ struct debuginfo *dso__debuginfo(struct dso *dso)
 
 	mutex_unlock(dso__lock(dso));
 	free(name);
+
+	/*
+	 * The debuginfo for a DSO in the profile may not be installed
+	 * locally, for instance the vmlinux for the kernel the profile was
+	 * recorded on when processing it on another machine, or after the
+	 * kernel and its debuginfo got upgraded in between.  Fall back to
+	 * fetching it keyed by the build ID recorded in the perf.data file,
+	 * using the debuginfod client, which checks its local cache first.
+	 *
+	 * Do it outside dso__lock, a fetch from a remote debuginfod server
+	 * can take a while and would otherwise block anything else using
+	 * this dso, and honour the opt-out, the user may have asked for
+	 * no debuginfod via --no-debuginfod, core.debuginfod=false or by
+	 * disabling the build-id cache.
+	 */
+	if (dinfo == NULL)
+		dinfo = debuginfo__new_build_id(dso__bid(dso));
+
 	return dinfo;
 }
tools/perf/util/symbol.c +12 -0
index 3587ad243159074f..1d70c43bb24ee279 100644
@@ -76,6 +76,8 @@ struct symbol_conf symbol_conf = {
 	.inline_name		= true,
 	.res_sample		= 0,
 	.addr2line_timeout_ms	= 5 * 1000,
+	/* Fetching debuginfo by build ID, off via --no-debuginfod, etc */
+	.debuginfod		= true,
 };
 
 struct map_list_node {
@@ -2534,6 +2536,16 @@ int symbol__init(struct perf_env *env)
 	if (symbol_conf.initialized)
 		return 0;
 
+	/*
+	 * Have DEBUGINFOD_URLS set before the threads that can take the
+	 * debuginfod fetch path, and that getenv() it, e.g. in the every
+	 * debuginfod_begin() of the other debuginfod client users, are
+	 * started: set it up here, from the .urls files in /etc/debuginfod
+	 * when it isn't set, instead of doing it lazily from the fetch
+	 * path itself, setenv() not being thread safe.
+	 */
+	debuginfod__setup_urls_env();
+
 	symbol_conf.priv_size = PERF_ALIGN(symbol_conf.priv_size, sizeof(u64));
 
 	symbol__elf_init();
tools/perf/util/symbol_conf.h +1 -0
index 71f60081a85bb18d..a56b1d2d843b9ff1 100644
@@ -45,6 +45,7 @@ struct symbol_conf {
 			force,
 			ignore_vmlinux,
 			ignore_vmlinux_buildid,
+			debuginfod,
 			show_kernel_path,
 			use_modules,
 			allow_aliases,
tools/perf/util/util.c +103 -0

130 lines not on this page · raw message

tools/perf/util/util.h +8 -0
index f8f1ff603c728086..1f763f9960bd0756 100644
@@ -82,6 +82,14 @@ struct perf_debuginfod {
 };
 void perf_debuginfod_setup(struct perf_debuginfod *di);
 
+#ifdef HAVE_DEBUGINFOD_SUPPORT
+void debuginfod__setup_urls_env(void);
+#else
+static inline void debuginfod__setup_urls_env(void)
+{
+}
+#endif
+
 const char *perf_basename(const char *path);
 
 char *filename_with_chroot(int pid, const char *filename);
Signature · 1 line
2.55.0
author maintainer
#9 lore raw

PATCH v6 3/12 perf config: Move perf_config__set_variable() to util/config.c

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

It is code from the 'perf config' builtin to set a variable in the configuration file perf is using, the user's ~/.perfconfig, or the file named by the PERF_CONFIG environment variable when it is set, rewriting that file from the config set, the same rewrite 'perf config' does, i.e. the comments in it are not preserved, as the config set carries just the key-value pairs. It has to honour PERF_CONFIG, that makes perf read only that file: the config set is loaded from it, so writing ~/.perfconfig would replace it with the exclusive file's entries.

The next patch wants to use it from util/debuginfo.c, so that when the user asks for a debuginfod fetch to be skipped, pressing 'd', the disable can also be made permanent, moving perf_config__set_variable() to util/config.c, together with the set_config() body it uses, parameterized as perf_config_set__write() with the system_config choice as an argument, since the builtin's static use_system_config and use_user_config option variables are not available outside it. 'perf config' keeps its behavior via the set_config() wrapper.

---
 tools/perf/builtin-config.c | 70 +------------------------------------
 tools/perf/util/config.c    | 70 +++++++++++++++++++++++++++++++++++++
 tools/perf/util/config.h    |  2 ++
 3 files changed, 73 insertions(+), 69 deletions(-)
tools/perf/builtin-config.c +1 -69
index cefd042e4f853466..3b074aca8d344539 100644
@@ -41,37 +41,7 @@ static struct option config_options[] = {
 
 static int set_config(struct perf_config_set *set, const char *file_name)
 {
-	struct perf_config_section *section = NULL;
-	struct perf_config_item *item = NULL;
-	const char *first_line = "# this file is auto-generated.";
-	FILE *fp;
-
-	if (set == NULL)
-		return -1;
-
-	fp = fopen(file_name, "w");
-	if (!fp)
-		return -1;
-
-	fprintf(fp, "%s\n", first_line);
-
-	/* overwrite configvariables */
-	perf_config_items__for_each_entry(&set->sections, section) {
-		if (!use_system_config && section->from_system_config)
-			continue;
-		fprintf(fp, "[%s]\n", section->name);
-
-		perf_config_items__for_each_entry(&section->items, item) {
-			if (!use_system_config && item->from_system_config)
-				continue;
-			if (item->value)
-				fprintf(fp, "\t%s = %s\n",
-					item->name, item->value);
-		}
-	}
-	fclose(fp);
-
-	return 0;
+	return perf_config_set__write(set, file_name, use_system_config);
 }
 
 static int show_spec_config(struct perf_config_set *set, const char *var)
@@ -158,44 +128,6 @@ static int parse_config_arg(char *arg, char **var, char **value)
 	return 0;
 }
 
-int perf_config__set_variable(const char *var, const char *value)
-{
-	char path[PATH_MAX];
-	char *user_config = mkpath(path, sizeof(path), "%s/.perfconfig", getenv("HOME"));
-	const char *config_filename;
-	struct perf_config_set *set;
-	int ret = -1;
-
-	if (use_system_config)
-		config_exclusive_filename = perf_etc_perfconfig();
-	else if (use_user_config)
-		config_exclusive_filename = user_config;
-
-	if (!config_exclusive_filename)
-		config_filename = user_config;
-	else
-		config_filename = config_exclusive_filename;
-
-	set = perf_config_set__new();
-	if (!set)
-		goto out_err;
-
-	if (perf_config_set__collect(set, config_filename, var, value) < 0) {
-		pr_err("Failed to add '%s=%s'\n", var, value);
-		goto out_err;
-	}
-
-	if (set_config(set, config_filename) < 0) {
-		pr_err("Failed to set the configs on %s\n", config_filename);
-		goto out_err;
-	}
-
-	ret = 0;
-out_err:
-	perf_config_set__delete(set);
-	return ret;
-}
-
 int cmd_config(int argc, const char **argv)
 {
 	int i, ret = -1;
tools/perf/util/config.c +70 -0

76 lines not on this page · raw message

tools/perf/util/config.h +2 -0
index 987b47cf54c350ba..9098f8a045850c97 100644
@@ -33,6 +33,8 @@ int perf_config_scan(const char *name, const char *fmt, ...) __scanf(2, 3);
 const char *perf_config_get(const char *name);
 int perf_config_set(struct perf_config_set *set,
 		    config_fn_t fn, void *data);
+int perf_config_set__write(struct perf_config_set *set,
+			   const char *file_name, bool system_config);
 int perf_config_int(int *dest, const char *, const char *);
 int perf_config_u8(u8 *dest, const char *name, const char *value);
 int perf_config_u64(u64 *dest, const char *, const char *);
Signature · 1 line
2.55.0
author maintainer
#11 lore raw

PATCH v6 4/12 perf debuginfo: Let the user skip and disable debuginfod fetches

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

When a fetch is in progress, the way it prints progress is how one gets out of it: 's' aborts the current fetch via the debuginfod client's progress callback protocol and remembers the build ID, so that the rest of the session doesn't ask for it again, the user may have skipped it for being too big; 'd' additionally disables debuginfod for the rest of the session, clearing DEBUGINFOD_URLS so that libdwfl's own client, that reads it in every query, stops fetching too, and, using perf_config__set_variable(), moved to util/config.c in the previous patch, writes core.debuginfod=false to the configuration file directly, the user's ~/.perfconfig or the one named by PERF_CONFIG, the same rewrite 'perf config' does -- the comments are not preserved, as the config set carries just the key-value pairs -- and SIGINT, SIGQUIT and SIGTERM are intercepted while the terminal is in raw mode, so that it is restored and the signal is re-raised when the user interrupts a fetch.

In the TUI nothing is shown yet, printing to stderr would garble the browser display and draining stdin would eat the keys the browser wants, that comes in the next patch.

The interaction state -- the terminal settings, the signal dispositions and the progress and cancellation state -- is process global, so the fetches stay serialized: a second fetch resetting the cancellation state would drop the 's'/'d' keypress that was meant for the fetch already in progress, and the serialization the lookup state needs already provides the lock the interaction state uses.

Since the debuginfo fetching this builds on is opt-out, a fetch that the user asked to skip is not restarted by the next request for the same build ID in the same session, it is remembered as a cancellation on the misses list, where the debug message can tell it apart from a plain miss.

---
 tools/perf/Documentation/perf-annotate.txt |   4 +-
 tools/perf/Documentation/perf-report.txt   |  11 +-
 tools/perf/Documentation/perf-top.txt      |  10 +-
 tools/perf/util/debuginfo.c                | 311 ++++++++++++++++++---
 4 files changed, 296 insertions(+), 40 deletions(-)
tools/perf/Documentation/perf-annotate.txt +3 -1
index 12edc3337b86f099..25af1d166dc4c877 100644
@@ -64,7 +64,9 @@ OPTIONS
 	configured in DEBUGINFOD_URLS, checking the local debuginfod
 	client cache first, when it is not available locally, on for
 	these commands by default.  See the --debuginfod option of
-	'perf report' for how to turn it off.
+	'perf report' for how to turn it off, including the 's' and
+	'd' keys that skip a fetch in progress while 'perf' is
+	waiting for it.
 
 --itrace::
 	Options for decoding instruction tracing data. The options are:
tools/perf/Documentation/perf-report.txt +8 -3
index 2e11c0a97a016853..0a821490d6a0f895 100644
@@ -374,9 +374,14 @@ OPTIONS
 	these commands by default.  It can be turned off per invocation
 	with --no-debuginfod, per tool with the "report.debuginfod"
 	config option or globally with "core.debuginfod" set to false.
-	It is disabled as well when the local build-id cache is turned
-	off, e.g. "buildid.dir" set to /dev/null, as that asks for
-	fetched files not to be kept on the box.
+	While a fetch is in progress, 's' skips the current fetch and
+	'd' skips it and disables debuginfod for the rest of the
+	session, writing core.debuginfod=false to the configuration
+	file so that it stays disabled in the next runs too; in the
+	stdio interface the progress is a line on stderr.  It is
+	disabled as well when the local build-id cache is turned off,
+	e.g. "buildid.dir" set to /dev/null, as that asks for fetched
+	files not to be kept on the box.
 
 --kallsyms=<file>::
         kallsyms pathname
tools/perf/Documentation/perf-top.txt +7 -3
index 345dd9ee9b6bab88..f38b54267125cb08 100644
@@ -90,9 +90,13 @@ Default is to monitor all CPUS.
 	client cache first, when it is not available locally, on for
 	these commands by default.  Turn it off per invocation with
 	--no-debuginfod, with the "top.debuginfod" config option or
-	globally with "core.debuginfod" set to false.  Disabled as well
-	when the build-id cache is off, e.g. "buildid.dir" set to
-	/dev/null.
+	globally with "core.debuginfod" set to false.  While a fetch is
+	in progress, 's' skips the current fetch and 'd' skips it and
+	disables debuginfod for the rest of the session, writing
+	core.debuginfod=false to the configuration file so that it
+	stays disabled in the next runs too; in the stdio interface the
+	progress is a line on stderr.  Disabled as well when the
+	build-id cache is off, e.g. "buildid.dir" set to /dev/null.
 
 --kallsyms=<file>::
 	kallsyms pathname
tools/perf/util/debuginfo.c +278 -33

443 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#15 lore raw

PATCH v6 5/12 perf debuginfo: Show the debuginfod fetch progress and keys in the TUI

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The stdio only fetch UI left a fetch in progress in the TUI with nothing on screen, indistinguishable from perf being stuck, with no way out of it: the browser thread is the one doing the fetch, so it draws a window over itself with the fetch progress and drains the 's'/'d' keys from the input queue it owns, via the new ui__key_pending()/ ui__key_read()/ui__progress_window()/ui__progress_window_end() primitives declared in ui/util.h and implemented in ui/tui/util.c, with inline no-ops for builds without slang, keeping callers free of #ifdefs:
slang stays behind the ui/ layer, as the plan is to reduce the number of libraries perf needs to build, or swap slang for something else, such as ncurses, at some point. No terminal settings or signal dispositions are touched in that case: the terminal is the TUI's own and so are the SIGINT/SIGTERM handlers, that restore it before exiting.

The text in the window is word wrapped to the available columns, so that constrained terminals, such as a smartphone running termux, get all of it, including the 'd' option hint, instead of having the tail of the long progress line cropped; the title line is written without padding, so that the box border characters fill the rest of it.

---
 tools/perf/Documentation/perf-report.txt |   9 +-
 tools/perf/Documentation/perf-top.txt    |   5 +-
 tools/perf/ui/tui/util.c                 | 177 +++++++++++++++++++++++
 tools/perf/ui/util.h                     |  42 ++++++
 tools/perf/util/debuginfo.c              |  64 ++++++--
 5 files changed, 277 insertions(+), 20 deletions(-)
tools/perf/Documentation/perf-report.txt +5 -4
index 0a821490d6a0f895..0a4f61283b9542f7 100644
@@ -378,10 +378,11 @@ OPTIONS
 	'd' skips it and disables debuginfod for the rest of the
 	session, writing core.debuginfod=false to the configuration
 	file so that it stays disabled in the next runs too; in the
-	stdio interface the progress is a line on stderr.  It is
-	disabled as well when the local build-id cache is turned off,
-	e.g. "buildid.dir" set to /dev/null, as that asks for fetched
-	files not to be kept on the box.
+	stdio interface the progress is a line on stderr, in the TUI a
+	window over the browser.  It is disabled as well when the local
+	build-id cache is turned off, e.g. "buildid.dir" set to
+	/dev/null, as that asks for fetched files not to be kept on the
+	box.
 
 --kallsyms=<file>::
         kallsyms pathname
tools/perf/Documentation/perf-top.txt +3 -2
index f38b54267125cb08..3ab8121480175b51 100644
@@ -95,8 +95,9 @@ Default is to monitor all CPUS.
 	disables debuginfod for the rest of the session, writing
 	core.debuginfod=false to the configuration file so that it
 	stays disabled in the next runs too; in the stdio interface the
-	progress is a line on stderr.  Disabled as well when the
-	build-id cache is off, e.g. "buildid.dir" set to /dev/null.
+	progress is a line on stderr, in the TUI a window over the
+	browser.  Disabled as well when the build-id cache is off, e.g.
+	"buildid.dir" set to /dev/null.
 
 --kallsyms=<file>::
 	kallsyms pathname
tools/perf/ui/tui/util.c +177 -0

192 lines not on this page · raw message

tools/perf/ui/util.h +42 -0

51 lines not on this page · raw message

tools/perf/util/debuginfo.c +50 -14

117 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#17 lore raw

PATCH v6 6/12 perf symbol: Fall back to fetching the vmlinux by build ID

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

A profile recorded on a kernel that is no longer installed, be it because the machine was rebooted into a new kernel or because the profile is being processed on another machine, can't have its kernel symbols resolved: /proc/kallsyms matches the running kernel, not the one in the profile, and is refused when restricted, e.g. with kernel.perf_event_paranoid > 1, while the build-id cache may carry just a kallsyms copy with zeroed addresses. With no kernel symbols the kernel samples can't be annotated, so they all end up in the '(unknown)' data type and the kernel is missing from the data type profile JSON "dsos" entry.

As a last resort, when the kernel symbols can't be found locally and the user didn't specify a kallsyms file, fetch the vmlinux keyed by the kernel build ID recorded in the perf.data file using the debuginfod client and use it for symbols, which also provides the DWARF debuginfo needed for data type profiling.

The fetch is made with dso->lock dropped, the same way dso__debuginfo() does it for the debuginfo of a DSO, and the fetched file is loaded with the lock held again, only if another thread didn't get the symbols in the meantime, checked with dso__has_symbols(): dso__loaded() is set by dso__load() even for an attempt that failed, and would make a successful fetch throw the file away.

Like the other vmlinux sources this honors --ignore-vmlinux and --ignore-vmlinux_buildid: a fetched vmlinux is still a vmlinux, and the fetch is keyed by the build ID, so both flags skip it. 'perf record' sets ignore_vmlinux_buildid internally, and so does 'perf probe' for the commands other than --list, --del and --add when given an offline vmlinux, keeping those away from the fetch as well. A 'perf probe' that doesn't set it, e.g. --funcs or --add without --vmlinux on a system with a restricted /proc/kallsyms, can have the kernel debuginfo fetched, like the other tools. The fetch itself follows the opt-out controls added in the previous patch, --no-debuginfod, core.debuginfod=false and the build-id-cache-is-off case, and can be skipped with the 's'/'d' keys while it runs. Build IDs that were a miss are remembered, not queried again on every dso__load() retry.

With a profile recorded on a system running kernel 7.1.10, later processed after it was upgraded to 7.1.13, 'perf report -s type ' went from having all 24.91% of the kernel samples as '(unknown)' data types to resolving struct task_struct, struct rq, struct tty_struct, struct qspinlock, etc, with the kernel DSO, keyed by its build ID, appearing in the data type profile JSON.

---
 tools/perf/util/symbol.c | 71 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +70 -1
index 1d70c43bb24ee279..ccb6217119416dec 100644
@@ -20,6 +20,7 @@
 #include "cap.h"
 #include "cpumap.h"
 #include "debug.h"
+#include "debuginfo.h"
 #include "demangle-cxx.h"
 #include "demangle-java.h"
 #include "demangle-ocaml.h"
@@ -2191,12 +2192,33 @@ static char *dso__find_kallsyms(struct dso *dso, struct map *map)
 	return strdup(path);
 }
 
+/*
+ * Last resort when the symbols for the kernel the profile was recorded
+ * on can't be found locally: fetch the vmlinux keyed by the build ID
+ * recorded in the perf.data file using the debuginfod client, which
+ * checks its local cache first, e.g. when processing the profile on
+ * another machine or after the kernel and its debuginfo package got
+ * upgraded in between.
+ */
+/*
+ * The fetch itself, that dso__load_kernel_sym() calls with dso->lock
+ * dropped, see the comment there.
+ */
+static int dso__fetch_vmlinux_build_id(struct dso *dso, char **path)
+{
+	if (!dso__has_build_id(dso))
+		return -1;
+
+	return debuginfo__find_build_id(dso__bid(dso), path);
+}
+
 static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 {
 	int err;
 	const char *kallsyms_filename = NULL;
 	char *kallsyms_allocated_filename = NULL;
 	char *filename = NULL;
+	bool user_kallsyms = false;
 
 	/*
 	 * Step 1: if the user specified a kallsyms or vmlinux filename, use
@@ -2215,6 +2237,7 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 	 */
 	if (symbol_conf.kallsyms_name != NULL) {
 		kallsyms_filename = symbol_conf.kallsyms_name;
+		user_kallsyms = true;
 		goto do_kallsyms;
 	}
 
@@ -2257,7 +2280,53 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 		pr_debug("Using %s for symbols\n", kallsyms_filename);
 	free(kallsyms_allocated_filename);
 
-	if (err > 0 && !dso__is_kcore(dso)) {
+	/*
+	 * The kallsyms may be unavailable or restricted, e.g.
+	 * /proc/kallsyms with kernel.perf_event_paranoid > 1, try to fetch
+	 * the vmlinux keyed by the build ID using debuginfod as a last
+	 * resort, honoring --ignore-vmlinux and --ignore-vmlinux_buildid
+	 * like the other vmlinux sources above.
+	 */
+	if (err <= 0 && !user_kallsyms &&
+	    !symbol_conf.ignore_vmlinux &&
+	    !symbol_conf.ignore_vmlinux_buildid) {
+		char *fetched_path = NULL;
+
+		/*
+		 * dso__load() holds dso->lock while it calls us, and the
+		 * fetch below can take a long time, blocked on the network
+		 * or on the terminal, waiting for the user: do it with the
+		 * lock dropped, as dso__debuginfo() does for the debuginfo
+		 * of a DSO, so that the threads that need this dso don't get
+		 * stuck behind a server round trip.  Nothing of the dso is
+		 * touched by the fetch, the symbols are loaded with the lock
+		 * held again, and only if the fetch brought a file back.
+		 */
+		mutex_unlock(dso__lock(dso));
+		err = dso__fetch_vmlinux_build_id(dso, &fetched_path);
+		mutex_lock(dso__lock(dso));
+
+		if (err) {
+			zfree(&fetched_path);
+		} else if (dso__has_symbols(dso)) {
+			/*
+			 * Somebody else got the symbols for this dso while
+			 * the lock was dropped for the fetch, use those
+			 * instead of loading the file that came back a
+			 * second time.  dso__has_symbols() and not
+			 * dso__loaded(): dso__load() sets the latter even
+			 * when it fails, so an attempt that failed would
+			 * have us throw a file that came back away.
+			 */
+			pr_debug("%s got its symbols while its vmlinux was being fetched, using them\n",
+				 dso__name(dso));
+			zfree(&fetched_path);
+			err = 1;
+		} else {
+			/* Takes ownership of 'fetched_path' even when it fails */
+			err = dso__load_vmlinux(dso, map, fetched_path, true);
+		}
+	} else if (err > 0 && !dso__is_kcore(dso)) {
 		struct maps *kmaps = map__kmaps(map);
 
 		dso__set_binary_type(dso, DSO_BINARY_TYPE__KALLSYMS);
Signature · 1 line
2.55.0
author maintainer
#19 lore raw

PATCH v6 7/12 perf annotate-data: Show the sample count in the data-type browser

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.

Print the histogram entry's own count instead.

This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.

Fixes d001c7a7f4736743 ("perf annotate-data: Add hist_entry__annotate_data_tui()")
Assisted-by LLM
---
 tools/perf/ui/browsers/annotate-data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/ui/browsers/annotate-data.c +1 -1
index c6e07a9b64089ab5..a15608f6ee53335f 100644
@@ -374,7 +374,7 @@ static void browser__write_overhead(struct ui_browser *uib,
 	u64 period = hist->period;
 	double percent = total->period ? (100.0 * period / total->period) : 0;
 	bool current = ui_browser__is_current_entry(uib, row);
-	int nr_samples = 0;
+	int nr_samples = hist->nr_samples;
 
 	ui_browser__set_percent_color(uib, percent, current);
 
Signature · 1 line
2.55.0
author maintainer
#23 lore raw

PATCH v6 8/12 perf report: Add --progress option

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

Processing a large data type profiling session, e.g. an AMD IBS one, can take a long time and, when using the stdio output, 'perf report' gives no feedback about which phase it is in nor about how far along it is: the TUI has a progress bar for that, but in the stdio case the ui_progress updates, that are already there, are dropped on the floor.

Add a --progress option that installs a stdio backend for ui_progress, ui/stdio/progress.c, printing the phase title, the percentage done and the current/total counts to stderr:

Processing events... [ 42.3%] 317M / 746M Merging related events... [ 61.0%] 309026 / 506686 Sorting events for output... [ 98.2%] 14132 / 14387

The first one is the perf.data file size based progress already kept while reading events, sized with the same unit_number__scnprintf() used by the TUI progress bar title; the other two are the hist entry based ones for the hist entry merging (collapse) and output sorting phases, that print raw counts.

Steps are 1% of the phase total, the default (total / 16) is tuned for the character cell based TUI bar, and the last update normally stops short of the total, so finishing a phase prints it as complete.

When stderr is a tty the line is updated in place, with each phase getting a line of its own, otherwise one line is printed per update, so that redirecting stderr to a file leaves a readable log of the phases.

Phases can be nested, e.g. the ordered events flushes that take place while the "Processing events..." phase is still in progress, so the backend keeps track of the ones started so far to be able to complete the right one when a phase finishes, as ui_progress__finish() gets no arguments.

That bookkeeping requires ui_progress__init() and ui_progress__finish() to be paired, and a few callers had paths returning early without the
finish: do_flush() in ordered-events.c on session_done() and on deliver() errors, and the ENOMEM paths right after the init in __perf_session__process_pipe_events() and __perf_session__process_dir_events().

Fix those, with a backend that kept the stale entries it would print through dangling pointers to returned stack frames, and have the update() method ignore anything that doesn't match the innermost running phase, printing nothing is better than printing another phase's numbers or reading the stack array with index -1.

Committer testing:

⬢ [acme@toolbx perf-tools-next]$ perf record -o perf.data.small -- sleep 0.2 [ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.002 MB perf.data.small (7 samples) ] ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -i perf.data.small > /dev/null Processing events... [ 0.0%] 0B / 2K
Processing events... [ 99.6%] 2040B / 2K
Processing time ordered events... [100.0%] 13 / 13
Processing events... [100.0%] 2K / 2K
Sorting events for output... [100.0%] 5 / 5
⬢ [acme@toolbx perf-tools-next]$

Nothing goes to stderr without --progress:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --stdio -i perf.data.small > /dev/null 2> stderr.txt
  ⬢ [acme@toolbx perf-tools-next]$ wc -c stderr.txt
  0 stderr.txt
  ⬢ [acme@toolbx perf-tools-next]$

With a 321 MB perf.data (perf mem record, i7-14700K):

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -i perf.data.i7 > /dev/null
  Processing events... [  0.0%] 0B / 306M
  [...]
  Sorting events for output... [100.0%] 29996 / 29996
  ⬢ [acme@toolbx perf-tools-next]$

Which is what prompted this: an AMD IBS data type profile session hangs in the DWARF type resolution done when merging hist entries, with --progress it now is visible that it is not the event loading that is stuck, but the merging, at a specific hist entry:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [  0.0%] 0B / 746M
  [...]
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 58.0%] 293828 / 506686
  Merging related events... [ 59.0%] 298894 / 506686
  Merging related events... [ 60.0%] 303960 / 506686
  Merging related events... [ 61.0%] 309026 / 506686

  ⬢ [acme@toolbx perf-tools-next]$ gdb -p $(pidof perf) -batch -ex 'bt 6' -ex detach
  #0  0x0000000000773f31 in die_get_pointer_type ()
  #1  0x000000000077b817 in find_data_type ()
  #2  0x0000000000636f20 in __hist_entry__get_data_type ()
  #3  0x0000000000639c65 in hist_entry__get_data_type ()
  #4  0x00000000006e57a8 in sort__type_collapse ()
  #5  0x00000000006ef0ee in hists__collapse_resort ()
  ⬢ [acme@toolbx perf-tools-next]$

The report related entries in 'perf test' pass:

⬢ [acme@toolbx perf-tools-next]$ perf test 17 27 30 31 85 88

  17: Match and link multiple hists                              : Ok
  27: Filter hist entries                                        : Ok
  30: Sort output of hist entries                                : Ok
  31: Cumulate child hist entries                                : Ok

85: Test that perf report includes file offsets and event type names in diagnostic messages. : Skip 88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Ok ⬢ [acme@toolbx perf-tools-next]$

---
 tools/perf/Documentation/perf-report.txt |  13 ++
 tools/perf/builtin-report.c              |  11 ++
 tools/perf/ui/Build                      |   1 +
 tools/perf/ui/progress.h                 |   2 +
 tools/perf/ui/stdio/progress.c           | 185 +++++++++++++++++++++++
 tools/perf/util/ordered-events.c         |  17 ++-
 tools/perf/util/session.c                |  12 +-
 7 files changed, 234 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/ui/stdio/progress.c
tools/perf/Documentation/perf-report.txt +13 -0
index 0a4f61283b9542f7..8fc1d5d56d54f698 100644
@@ -29,6 +29,19 @@ OPTIONS
 --quiet::
 	Do not show any warnings or messages.  (Suppress -v)
 
+--progress::
+	Show progress for each of the processing phases, printing the
+	percentage done and the current/total counts: the first phase
+	counts the bytes of the perf.data file processed so far, the
+	merge and sort phases count hist entries, e.g.:
+
+	Processing events... [ 42.3%] 4G / 10G
+	Merging related events... [  7.1%] 1024 / 14387
+	Sorting events for output... [ 98.2%] 14132 / 14387
+
+	It is a no-op when using the TUI or GTK browsers, that already
+	present progress information.
+
 -n::
 --show-nr-samples::
 	Show the number of samples for each symbol
tools/perf/builtin-report.c +11 -0
index 442c0822e614197f..32d5b35772cf374a 100644
@@ -87,6 +87,7 @@ struct report {
 	bool			use_gtk;
 #endif
 	bool			use_stdio;
+	bool			progress;
 	bool			show_full_info;
 	bool			show_threads;
 	bool			inverted_callchain;
@@ -1374,6 +1375,8 @@ int cmd_report(int argc, const char **argv)
 		    "Use the stdio interface"),
 	OPT_BOOLEAN(0, "weights", &symbol_conf.annotate_weight,
 			"Show or hide weight columns in annotation. Default show if non-zero."),
+	OPT_BOOLEAN(0, "progress", &report.progress,
+		    "Show progress while processing the perf.data file"),
 	OPT_BOOLEAN(0, "header", &report.header, "Show data header."),
 	OPT_BOOLEAN(0, "header-only", &report.header_only,
 		    "Show only data header."),
@@ -1766,6 +1769,14 @@ int cmd_report(int argc, const char **argv)
 	else
 		use_browser = 0;
 
+	/*
+	 * The TUI/GTK browsers already show progress, this is for the stdio
+	 * case, where we print the percentage of the perf.data file that was
+	 * processed so far, for each of the processing phases.
+	 */
+	if (report.progress && use_browser == 0)
+		stdio_progress__init();
+
 	if (report.data_type && use_browser == 1) {
 		symbol_conf.annotate_data_member = true;
 		symbol_conf.annotate_data_sample = true;
tools/perf/ui/Build +1 -0
index 6005f813c9e3990c..a7b1740d51c80f23 100644
@@ -4,6 +4,7 @@ perf-ui-y += progress.o
 perf-ui-y += util.o
 perf-ui-y += hist.o
 perf-ui-y += stdio/hist.o
+perf-ui-y += stdio/progress.o
 
 CFLAGS_setup.o += -DLIBDIR="BUILD_STR($(LIBDIR))"
 
tools/perf/ui/progress.h +2 -0
index 4f52c37b2f099a82..03f1a8bb260ba076 100644
@@ -23,6 +23,8 @@ void __ui_progress__init(struct ui_progress *p, u64 total,
 
 void ui_progress__update(struct ui_progress *p, u64 adv);
 
+void stdio_progress__init(void);
+
 struct ui_progress_ops {
 	void (*init)(struct ui_progress *p);
 	void (*update)(struct ui_progress *p);
tools/perf/ui/stdio/progress.c new file +185 -0

185 lines not on this page · raw message

tools/perf/util/ordered-events.c +13 -4
index a5857f9f5af2d3de..4063403d4978b45e 100644
@@ -237,14 +237,16 @@ static int do_flush(struct ordered_events *oe, bool show_progress)
 		ui_progress__init(&prog, oe->nr_events, "Processing time ordered events...");
 
 	list_for_each_entry_safe(iter, tmp, head, list) {
-		if (session_done())
-			return 0;
+		if (session_done()) {
+			ret = 0;
+			goto out_progress;
+		}
 
 		if (iter->timestamp > limit)
 			break;
 		ret = oe->deliver(oe, iter);
 		if (ret < 0)
-			return ret;
+			goto out_progress;
 
 		ordered_events__delete(oe, iter);
 		oe->last_flush = iter->timestamp;
@@ -258,10 +260,17 @@ static int do_flush(struct ordered_events *oe, bool show_progress)
 	else if (last_ts <= limit)
 		oe->last = list_entry(head->prev, struct ordered_event, list);
 
+	ret = 0;
+out_progress:
+	/*
+	 * Always pair ui_progress__init() with ui_progress__finish(), the
+	 * stdio progress backend tracks phases on a stack and an early
+	 * return that skipped the finish would leave the dead 'prog' on it.
+	 */
 	if (show_progress)
 		ui_progress__finish();
 
-	return 0;
+	return ret;
 }
 
 static int __ordered_events__flush(struct ordered_events *oe, enum oe_flush how,
tools/perf/util/session.c +9 -3
index a5b596cd14be6186..50f3e1d6c17d3147 100644
@@ -3248,8 +3248,12 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 	cur_size = sizeof(union perf_event);
 
 	buf = malloc(cur_size);
-	if (!buf)
-		return -errno;
+	if (!buf) {
+		err = -errno;
+		if (update_prog)
+			ui_progress__finish();
+		return err;
+	}
 	ordered_events__set_copy_on_queue(oe, true);
 more:
 	event = buf;
@@ -3748,8 +3752,10 @@ static int __perf_session__process_dir_events(struct perf_session *session)
 	}
 
 	rd = calloc(nr_readers, sizeof(struct reader));
-	if (!rd)
+	if (!rd) {
+		ui_progress__finish();
 		return -ENOMEM;
+	}
 
 	rd[0] = (struct reader) {
 		.fd		 = perf_data__fd(session->data),
Signature · 1 line
2.55.0
author maintainer
#25 lore raw

PATCH v6 9/12 perf scripts: Add perf-stuck, to tell where a running perf is stuck

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

A perf that takes forever is hard to tell apart from one that is stuck in a loop, and when it is stuck there is no way to know where without attaching gdb to it and looking around, which is what this does, from the outside, sampling /proc/<pid> at a fixed interval:

  ⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf)
  watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s
  09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB  Merging related events... [ 39.0%] 197574 / 506686
  09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686
  09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686
  09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686

The CPU time used grows by a whole interval on every sample while the [stack] mapping, that would be moving down if this was recursion, stays put, so that one is spinning, and the last line of the progress log of 'perf report --progress' tells in which phase.

With -g it runs gdb when no progress is made for two consecutive samples, using the perf-stuck.gdb that sits next to it, which adds the perf-die-chain command used by the next patch, to print the DIE chain a DWARF type chasing loop is walking when the perf being watched is stuck in one of those.

It is a prototype: this wants to become a first class 'perf stuck' command, sampling a running process from inside perf, instead of this shell script poking at /proc and shelling out to gdb.

Example:

⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 1 -n 3 -g $(pgrep -x sleep) watching 2342826 (sleep 45 ) every 1s
11:58:36 state=S cpu=+0 (0.00s) rss=492kB stack=7ffe21f89000-7ffe21faa000 size=132kB stuck=2 (no progress log) ... gdb output of 2342826 in /tmp/perf-stuck-gdb.Uz9i5p
⬢ [acme@toolbx perf-tools-next]$ tail -4 /tmp/perf-stuck-gdb.Uz9i5p
#3 0x0000555bc6ddc28f in main ()
not in a DWARF type chaser, try: bt
not in find_data_type()
[Inferior 1 (process 2342826) detached]
⬢ [acme@toolbx perf-tools-next]$

---
 tools/perf/scripts/perf-stuck.gdb | 104 ++++++++++++++++
 tools/perf/scripts/perf-stuck.sh  | 194 ++++++++++++++++++++++++++++++
 2 files changed, 298 insertions(+)
 create mode 100644 tools/perf/scripts/perf-stuck.gdb
 create mode 100755 tools/perf/scripts/perf-stuck.sh
tools/perf/scripts/perf-stuck.gdb new file +104 -0
new file mode 100644
index 0000000000000000..53e9019828b6ef62
@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# gdb commands for a perf that is stuck, used by perf-stuck.sh -g and usable
+# directly:
+#
+#   gdb -p $(pgrep -x perf) -batch -x perf-stuck.gdb -ex bt
+#
+# PROTOTYPE: part of the perf-stuck.sh stopgap, see the note at the start of
+# that script: this wants to move into a first class 'perf stuck' command,
+# which would print these DIE chains by itself, without gdb.
+#
+# The settings are the ones that keep a batch attach from stopping to ask
+# questions (debuginfod, pagination) and that make the output readable.
+#
+# The commands are for the DWARF type chasers in util/dwarf-aux.c, the
+# functions a data type profiling 'perf report -s type' spins in when a
+# debug info file has a type chain that got into a cycle:
+#
+#   perf-die-chain <function> <die variable> [iterations]
+#   perf-die-chain-all [iterations]
+#   perf-dso
+#
+# For each iteration of the chasing loop they print the DIE address, the
+# CU it came from, its offset in the debug file, its tag and its name: a
+# cycle shows up as the same handful of (addr, cu) pairs repeating, and a
+# CU that changes from one iteration to the next means the chase is
+# hopping between a debug file and its dwz common file.
+
+set pagination off
+set confirm off
+set debuginfod enabled off
+set print pretty on
+set height 0
+set width 0
+
+define perf-die-chain
+  if $argc < 2
+    printf "usage: perf-die-chain <function> <die variable> [iterations]\n"
+  else
+    frame function $arg0
+    if $argc == 3
+      set $perf_die_chain_n = $arg2
+    else
+      set $perf_die_chain_n = 10
+    end
+    set $perf_die_chain_head = $pc
+    set $perf_die_chain_i = 0
+    while $perf_die_chain_i < $perf_die_chain_n
+      printf "chain[%d] die=%p addr=%p cu=%p off=0x%lx tag=%d name=%s\n", $perf_die_chain_i, $arg1, $arg1->addr, $arg1->cu, ((Dwarf_Off) dwarf_dieoffset($arg1)), ((int) dwarf_tag($arg1)), ((char *) dwarf_diename($arg1))
+      until *$perf_die_chain_head
+      set $perf_die_chain_i = $perf_die_chain_i + 1
+    end
+  end
+end
+
+document perf-die-chain
+Print the DIE chain being walked by a DWARF type chasing loop.
+usage: perf-die-chain <function> <die variable> [iterations]
+  perf-die-chain die_get_pointer_type type_die
+  perf-die-chain __die_get_real_type vr_die
+  perf-die-chain die_get_real_type vr_die
+end
+
+define perf-die-chain-all
+  if $argc == 0
+    set $perf_die_chain_n = 10
+  else
+    set $perf_die_chain_n = $arg0
+  end
+  if $_any_caller_is("die_get_pointer_type", 20)
+    printf "stuck in die_get_pointer_type():\n"
+    perf-die-chain die_get_pointer_type type_die $perf_die_chain_n
+  else
+    if $_any_caller_is("__die_get_real_type", 20)
+      printf "stuck in __die_get_real_type():\n"
+      perf-die-chain __die_get_real_type vr_die $perf_die_chain_n
+    else
+      if $_any_caller_is("die_get_real_type", 20)
+        printf "stuck in die_get_real_type():\n"
+        perf-die-chain die_get_real_type vr_die $perf_die_chain_n
+      else
+        printf "not in a DWARF type chaser, try: bt\n"
+      end
+    end
+  end
+end
+
+document perf-die-chain-all
+Find which DWARF type chaser the process is in and print the DIE chain.
+usage: perf-die-chain-all [iterations]
+end
+
+define perf-dso
+  if $_any_caller_is("find_data_type", 20)
+    frame function find_data_type
+    printf "dso=%s ip=0x%lx sym=%s\n", dloc->ms->map->dso->name, dloc->ip, dloc->ms->sym->name
+  else
+    printf "not in find_data_type()\n"
+  end
+end
+
+document perf-dso
+Print the dso, ip and symbol of the data location being resolved.
+end
tools/perf/scripts/perf-stuck.sh new file +194 -0

194 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#27 lore raw

PATCH v6 10/12 perf annotate-data: Resolve type DIEs in the debug file they came from

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

A 'perf report -s type' on a 783 MB AMD IBS data type profiling session hangs, burning all of a CPU and producing no output:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 58.0%] 293828 / 506686
  Merging related events... [ 59.0%] 298894 / 506686
  Merging related events... [ 60.0%] 303960 / 506686
  Merging related events... [ 61.0%] 309026 / 506686
  [ ... nothing else, ever ... ]

It is a spin and not a slow path: with perf-stuck the CPU time used grows by a whole interval on every sample, while the [stack] mapping, which would be moving down if this was recursion, stays put:

  ⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf)
  watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s
  09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB  Merging related events... [ 39.0%] 197574 / 506686
  09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686
  09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686
  09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686

Attaching gdb and stepping the loop, printing dwarf_dieoffset() and dwarf_tag() for the DIE being chased on each trip round it, shows the chase never moving, on a typedef that refers to itself:

  ⬢ [acme@toolbx perf-tools-next]$ gdb -p $(pgrep -x perf) -batch -x tools/perf/scripts/perf-stuck.gdb -ex 'perf-die-chain die_get_pointer_type type_die 8' -ex perf-dso -ex detach
  stuck in die_get_pointer_type():
  #3  0x0000000000772421 in die_get_pointer_type (type_die=0x7ffc3ae4b710, type_die@entry=0x7ffc3ae4b6f0, die_mem=die_mem@entry=0x7ffc3ae4b710) at util/dwarf-aux.c:327
  327			type_die = die_get_type(type_die, die_mem);
  chain[0] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null)
  chain[1] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null)
  [ ... the very same DIE, forever ... ]
  dso=/usr/lib64/libz.so.1.3.1.zlib-ng ip=0xe2e sym=build_tree

The DIE is at offset 0x1f of the debug info of libz.so.1, which is zlib-ng, and is one of the dwz compressed ones: the type DIEs shared by more than one CU live in the common file, where 0x1f is a perfectly good DW_TAG_base_type:

⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/.dwz/zlib-ng-2.3.3-3.fc44.x86_64 | sed -n '/Compilation Unit @ offset 0:/,/Compilation Unit @ offset 0x5f:/p'
Compilation Unit @ offset 0:
Length: 0x5b (32-bit)
Version: 5
Unit Type: DW_UT_partial (3)
<0><c>: Abbrev Number: 1 (DW_TAG_partial_unit)
[...]
<1><1f>: Abbrev Number: 62 (DW_TAG_base_type)

      <20>   DW_AT_byte_size   : 4
      <21>   DW_AT_encoding    : 5	(signed)
      <22>   DW_AT_name        : int

while in the main debug file that same offset is not a DIE at all, it is the start of another unit's header:

⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/usr/lib64/libz.so.1.3.1.zlib-ng-2.3.3-3.fc44.x86_64.debug | grep 'Compilation Unit @' | head -2
Compilation Unit @ offset 0:
Compilation Unit @ offset 0x1f:

die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, and update_var_state() then hands that offset to dwarf_offdie() together with the main debug file, where dwarf_offdie() parses whatever is there as a DIE, in this case the one typedef that refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin.

So record, next to the offset, whether the type DIE was in the file the variable DIE came from or in the dwz common one, and resolve the offset in the file it came from, which is what the new die_get_type_die() does.

Which of the two that is does not have to be guessed from what is at the
offset: dwz encodes the references into its common file as DW_FORM_GNU_ref_alt, so elfutils resolves them into the alt Dwarf and the CU of the resulting type DIE belongs to that other file, so comparing the Dwarf each of the two CUs belongs to, die_same_file(), settles it exactly, for any number of hops from the variable DIE.

Then bound the chases themselves: no sane chain of typedefs and qualifiers is 32 DIEs long, no sane nesting for the struct and union members that __add_member_cb() follows recursively is 8 deep, and the same recursion bound covers the type names die_get_typename_from_type() builds by following pointers and arrays, so that a debug info file broken in some other way makes perf give up on a type, telling about it with pr_debug, visible with -v, instead of looking like it hung. The member nesting bound is reported with pr_debug_dtp (visible with -vvv or -D type-profile).

Cycles can't occur in member trees from valid DWARF, embedded members can't be recursive in C, so the nesting bound only ever bites legitimate depth and the member where the recursion was cut is marked 'truncated', which the JSON exporter added in a subsequent series reports to its consumers, so that they can tell a truncated tree from one that really ends there.

die_get_type_die() then has no fallback to the other file: resolving an alt file offset in the main file is the misparse above, so it resolves in the file the offset was recorded as belonging to, and gives up on the type when the offset does not resolve there. The tag the type DIE had is kept as a sanity check, a mismatch means the debug info changed under perf or is broken in yet another way, and the bounds on the type chasers remain the backstop: a debug info file broken in some other way still makes perf give up on a type with a pr_debug instead of hanging.

Testing:

Before, on the 783 MB AMD IBS session, killed after some 8 minutes stuck at 61%:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 61.0%] 309026 / 506686
  ⬢ [acme@toolbx perf-tools-next]$

After, the whole session is processed, and the zlib-ng types, from the build_tree() hist entry that used to hang it, show up:

⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > ibs.out Processing events... [100.0%] 746M / 746M
Merging related events... [100.0%] 506686 / 506686
Sorting events for output... [100.0%] 1428 / 1428
⬢ [acme@toolbx perf-tools-next]$ grep -E "deflate_state|inflate_state|internal_state" ibs.out

      0.00%  deflate_state
      0.00%  deflate_state*
      0.00%  struct inflate_state
      0.00%  struct inflate_state*
      0.00%  struct internal_state
  ⬢ [acme@toolbx perf-tools-next]$ perf test 17 27 30 31 85 88
  17: Match and link multiple hists                              : Ok
  27: Filter hist entries                                        : Ok
  30: Sort output of hist entries                                : Ok
  31: Cumulate child hist entries                                : Ok

85: Test that perf report includes file offsets and event type names in diagnostic messages. : Ok 88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Skip ⬢ [acme@toolbx perf-tools-next]$

Requiring elfutils 0.160 for this: dwarf_cu_getdwarf(), the function that tells which Dwarf a CU belongs to, first appeared in 0.160 ("libdw:
New functions dwarf_cu_getdwarf, dwarf_cu_die", elfutils NEWS), so the libdw feature test now probes for it, in tools/build/feature/test-libdw.c, and Makefile.config says 0.160 where it said 0.157. The probe takes the address instead of calling it, as that is all that is needed to make a 0.157-0.159 elfutils, from 2014 and without the symbol, disable dwarf support with the existing message rather than fail to link dwarf-aux.c.

Fixes 06b2ce75386df04b ("perf annotate-data: Maintain variable type info")
Fixes 55ee3d005d62279d ("perf annotate-data: Add a cache for global variable types")
Assisted-by LLM
---
 tools/build/feature/test-libdw.c |  15 ++-
 tools/perf/Makefile.config       |   2 +-
 tools/perf/util/annotate-data.c  |  59 ++++++++---
 tools/perf/util/annotate-data.h  |   3 +
 tools/perf/util/dwarf-aux.c      | 171 +++++++++++++++++++++++++++----
 tools/perf/util/dwarf-aux.h      |  33 ++++++
 6 files changed, 247 insertions(+), 36 deletions(-)
tools/build/feature/test-libdw.c +14 -1
index aabd63ca76b4d7e6..23e1ba6ff3466b9f 100644
@@ -49,8 +49,21 @@ int test_elfutils(void)
 	return 0;
 }
 
+/*
+ * elfutils 0.160 and later: used to tell which debug file a DIE lives in,
+ * the dwz alt file or the main one, see die_same_file() in
+ * tools/perf/util/dwarf-aux.c.  Only the symbol is needed, so take its
+ * address instead of calling it.
+ */
+int test_libdw_cu_getdwarf(void)
+{
+	void *sym = (void *)dwarf_cu_getdwarf;
+
+	return sym == NULL;
+}
+
 int main(void)
 {
 	return test_libdw() + test_libdw_unwind() + test_libdw_getlocations() +
-	       test_libdw_getcfi() + test_elfutils();
+	       test_libdw_getcfi() + test_libdw_cu_getdwarf() + test_elfutils();
 }
tools/perf/Makefile.config +1 -1
index 4ee7393a39f91d85..0585d71e3182c6dd 100644
@@ -470,7 +470,7 @@ else
   else
     ifneq ($(feature-libdw), 1)
       ifndef NO_LIBDW
-        $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.157, disables dwarf support. Please install new elfutils-devel/libdw-dev)
+        $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.160, disables dwarf support. Please install new elfutils-devel/libdw-dev)
         NO_LIBDW := 1
       endif
     endif # Dwarf support
tools/perf/util/annotate-data.c +47 -12

133 lines not on this page · raw message

tools/perf/util/annotate-data.h +3 -0
index ca2096a9ee62cbfe..cc576232f55b5fb0 100644
@@ -57,6 +57,9 @@ struct annotated_member {
 	char *var_name;
 	int offset;
 	int size;
+	unsigned int depth;
+	/* Children not expanded because the nesting limit was reached */
+	bool truncated;
 };
 
 /**
tools/perf/util/dwarf-aux.c +149 -22

266 lines not on this page · raw message

tools/perf/util/dwarf-aux.h +33 -0
index 161f0bf980b6ee6a..149e0cbf63ccc82a 100644
@@ -152,6 +152,8 @@ int die_get_scopes(Dwarf_Die *cu_die, Dwarf_Addr pc, Dwarf_Die **scopes);
 struct die_var_type {
 	struct die_var_type *next;
 	u64 die_off;
+	int die_tag;
+	bool from_alt;	/* die_off is relative to the alt (dwz) file */
 	u64 addr;
 	u64 end;        /* end address of location range */
 	int reg;
@@ -183,6 +185,37 @@ Dwarf_Die *die_find_variable_by_addr(Dwarf_Die *sc_die, Dwarf_Addr addr,
 /* Save all variables and parameters in this scope */
 void die_collect_vars(Dwarf_Die *sc_die, struct die_var_type **var_types);
 
+/*
+ * Get the type DIE saved by die_collect_vars()/die_collect_global_vars().
+ *
+ * The offsets those save are the dwarf_dieoffset() of the type DIE, which is
+ * relative to the debug file that DIE lives in: the dwz common file, the alt
+ * file in libdw terms, for the types shared by more than one CU, the main
+ * file for the rest.  Resolving an alt file offset in the main file does not
+ * fail: dwarf_offdie() parses whatever is at that offset there, and an offset
+ * that is a CU header in the main file reads back as a typedef whose
+ * DW_AT_type refers to itself, which is what hung 'perf report -s type' on
+ * the dwz compressed debug info of zlib-ng (libz.so.1).
+ *
+ * So @from_alt, recorded when the offset was saved, says which of the two
+ * files to resolve it in.  It is not inferred from the DIE contents: dwz
+ * encodes references into the common file as DW_FORM_GNU_ref_alt, so
+ * elfutils resolves them into the alt Dwarf and the CU of the type DIE then
+ * belongs to that other file, which die_same_file() compares exactly.
+ *
+ * There is deliberately no fallback to the other file when the offset does
+ * not resolve: that fallback is the misparse above.
+ *
+ * @die_tag is then only a sanity check: the offset is of a DIE that had this
+ * tag when it was saved, so a mismatch means the debug info changed under us,
+ * or is broken, and giving up on the type is the right answer.
+ */
+Dwarf_Die *die_get_type_die(Dwarf *dbg, u64 die_off, int die_tag, bool from_alt,
+			    Dwarf_Die *die_mem);
+
+/* Whether two DIEs live in the same debug file */
+bool die_same_file(Dwarf_Die *die_a, Dwarf_Die *die_b);
+
 /* Save all global variables in this CU */
 void die_collect_global_vars(Dwarf_Die *cu_die, struct die_var_type **var_types);
 
Signature · 1 line
2.55.0
author maintainer
#31 lore raw

PATCH v6 11/12 perf mem record: Request PERF_SAMPLE_CPU by default

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type profiling per-sample stream keys cross-CPU contention on sample->cpu: without PERF_SAMPLE_CPU the cpu field is the (u32)-1 "no CPU info" sentinel, documented as such in perf_session__deliver_event(), and same-instance reads and writes from different cores are indistinguishable from same-CPU traffic, so pahole's false-sharing detector cannot tell them apart.

builtin-record.c already defines --sample-cpu and 'perf mem record' forwards unknown options to the record parser, so passing it explicitly works today; make it the default, next to the -d (addr) and -W (weight) the command already requests, documenting it in perf-mem(1).

The rec_argv array is sized as nine arguments per memory PMU plus the user arguments, not counting the arguments __cmd_record() adds itself, up to eight with all the optional flags. On PMUs with separate load and store events the four event arguments plus the fixed ones already filled the array to its last slot, so this new argument would write past it; reserve space for the fixed arguments explicitly.

---
 tools/perf/Documentation/perf-mem.txt |  4 ++++
 tools/perf/builtin-mem.c              | 18 ++++++++++++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-mem.txt +4 -0
index 4d164836d0943119..fe51c5e3333dc4a0 100644
@@ -14,6 +14,10 @@ DESCRIPTION
 -----------
 "perf mem record" runs a command and gathers memory operation data
 from it, into perf.data. Perf record options are accepted and are passed through.
+It also requests the address (-d), the weight (-W, where supported) and the
+CPU id (--sample-cpu) of every sampled access by default; the CPU id is what
+lets per-sample analysis tell reads and writes to the same data from
+different cores apart from same-CPU traffic.
 
 "perf mem report" displays the result. It invokes perf report with the
 right set of options to display a memory access profile. By default, loads
tools/perf/builtin-mem.c +16 -2
index 6101a26b3a781e69..6f38cda1a45ada16 100644
@@ -99,8 +99,13 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem,
 	argc = parse_options(argc, argv, options, record_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
 
-	/* Max number of arguments multiplied by number of PMUs that can support them. */
-	rec_argc = argc + 9 * (perf_pmu__mem_events_num_mem_pmus(pmu) + 1);
+	/*
+	 * Max number of arguments multiplied by number of PMUs that can
+	 * support them, plus the arguments added directly below, at most:
+	 * "record", "-W", "-d", "--sample-cpu", "--phys-data",
+	 * "--data-page-size", "--all-user" and "--all-kernel".
+	 */
+	rec_argc = argc + 8 + 9 * (perf_pmu__mem_events_num_mem_pmus(pmu) + 1);
 
 	if (mem->cpu_list)
 		rec_argc += 2;
@@ -135,6 +140,15 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem,
 
 	rec_argv[i++] = "-d";
 
+	/*
+	 * The data-type profiling per-sample stream keys cross-CPU
+	 * contention on sample->cpu (PERF_SAMPLE_CPU); without it the cpu
+	 * field is the (u32)-1 'no CPU info' sentinel and same-instance
+	 * reads and writes from different cores are indistinguishable
+	 * from same-CPU traffic.
+	 */
+	rec_argv[i++] = "--sample-cpu";
+
 	if (mem->phys_addr)
 		rec_argv[i++] = "--phys-data";
 
Signature · 1 line
2.55.0
author maintainer
#34 lore raw

PATCH v6 12/12 perf mem record: Use the IBS swfilt filter when available

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

IBS events with exclude_{user,kernel} bits, as used in per-thread mode when kernel samples are not allowed, are rejected by the kernel on hardware without the privilege filter, so a per-thread 'perf mem record' fails:

  $ perf mem record -o /dev/null -- true
  Error:
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

Kernel v6.14 added swfilt, a software privilege filter that makes those events usable per-thread, exposed as the 'swfilt' format term (d29e744c71673a71 "perf/x86: Relax privilege filter restriction on AMD IBS"):

  $ perf record -e ibs_op/ldlat=0,swfilt=1/ -- true

Give the ibs_op memory events a name variant with the swfilt term, used when the PMU exposes it as a format term, keeping the names that need system wide mode otherwise.

The variant is used even for records that end up without exclude bits, e.g. a plain system wide one: perf record adds those bits itself when the first open fails with EACCES on an unprivileged setup, after the event name has been built, so the term has to be in it for that retry:

  $ perf record -v -e ibs_op/ldlat=0/ -- true
  kernel.perf_event_paranoid=2, trying to fall back to excluding kernel and hypervisor  samples
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

With no exclude bits the kernel discards nothing.

This makes the per-thread 'perf mem record' that the data type profiling shell test does work: the skip added by patch 1 is driven by that record failing, so it now runs on AMD kernels with swfilt, and keeps skipping on kernels without it. The 'Test data symbol' shell test matches the exact ibs_op event string, so its regex now accepts terms added after ldlat=150, and it only checks ldlat when the captured term has it: on uarch without ldlat the swfilt variant makes the capture non-empty ("swfilt=1"), that the old 'ldlat=0' check would reject.

---
 tools/perf/arch/x86/util/mem-events.c      | 17 ++++++++++++++---
 tools/perf/tests/shell/test_data_symbol.sh |  6 ++++--
 tools/perf/util/mem-events.c               | 21 +++++++++++++++++----
 tools/perf/util/mem-events.h               |  2 ++
 4 files changed, 37 insertions(+), 9 deletions(-)
tools/perf/arch/x86/util/mem-events.c +14 -3
index b38f519020ff8c6f..034053dc762101dc 100644
@@ -7,7 +7,10 @@
 
 #define MEM_LOADS_AUX		0x8203
 
-#define E(t, n, s, l, a) { .tag = t, .name = n, .event_name = s, .ldlat = l, .aux_event = a }
+#define E_INIT(t, n, s, l, a, sf) {					\
+	.tag = t, .name = n, .event_name = s, .swfilt_name = sf,	\
+	.ldlat = l, .aux_event = a }
+#define E(t, n, s, l, a) E_INIT(t, n, s, l, a, NULL)
 
 struct perf_mem_event perf_mem_events_intel[PERF_MEM_EVENTS__MAX] = {
 	E("ldlat-loads",	"%s/mem-loads,ldlat=%u/P",	"mem-loads",	true,	0),
@@ -21,14 +24,22 @@ struct perf_mem_event perf_mem_events_intel_aux[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,			NULL,				NULL,		false,	0),
 };
 
+/*
+ * IBS events with exclude_{user,kernel} bits set, as used by perf to
+ * record per-thread when kernel samples are not allowed, are rejected
+ * by the kernel on hardware without the privilege filter unless the
+ * swfilt software filter is used, so these events carry a variant of
+ * their names with the swfilt term, used by perf_pmu__mem_events_name()
+ * when the kernel exposes the term.
+ */
 struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
-	E("mem-ldst",	"%s//",		NULL,	false,	0),
+	E_INIT("mem-ldst",	"%s//",		NULL,	false,	0, "%s/swfilt=1/"),
 };
 
 struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
-	E("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0),
+	E_INIT("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0, "%s/ldlat=%u,swfilt=1/"),
 };
tools/perf/tests/shell/test_data_symbol.sh +4 -2
index d61b5659a46d9a77..52c837fddb639595 100755
@@ -65,15 +65,17 @@ if (($is_amd >= 1)); then
 	# --ldlat on AMD:
 	# o Zen4 and earlier uarch does not support ldlat
 	# o Even on supported platforms, it's disabled (--ldlat=0) by default.
+	# o Kernels with the swfilt term add it even when ldlat is not
+	#   supported, so only check ldlat when the term is present.
 	ldlat=${BASH_REMATCH[1]}
-	if [[ -n $ldlat ]]; then
+	if [[ $ldlat == *ldlat=* ]]; then
 		if ! [[ "$ldlat" =~ ldlat=0 ]]; then
 			echo "ERROR: ldlat not initialized to 0?"
 			exit 1
 		fi
 
 		mem_events="$(perf mem record -v --ldlat=150 -e list 2>&1)"
-		if ! [[ "$mem_events" =~ ^mem-ldst.*ibs_op/ldlat=150/.*available ]]; then
+		if ! [[ "$mem_events" =~ ^mem-ldst.*ibs_op/ldlat=150[,/].*available ]]; then
 			echo "ERROR: --ldlat not honored?"
 			exit 1
 		fi
tools/perf/util/mem-events.c +17 -4
index 0b49fce251fcc184..8f74cd085e500231 100644
@@ -82,6 +82,7 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 					     char *buf, size_t buf_size)
 {
 	struct perf_mem_event *e;
+	const char *name;
 
 	if (i >= PERF_MEM_EVENTS__MAX || !pmu)
 		return NULL;
@@ -90,24 +91,36 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 	if (!e || !e->name)
 		return NULL;
 
+	/*
+	 * Use the swfilt variant of the name when the PMU exposes the term.
+	 * It is not conditional on the event already having exclude bits:
+	 * perf record adds those bits itself when the first open fails with
+	 * EACCES on an unprivileged setup, after this name has been built,
+	 * and that retry only succeeds with the term in the name.  With no
+	 * exclude bits the kernel doesn't discard anything.
+	 */
+	name = e->name;
+	if (e->swfilt_name && perf_pmu__has_format(pmu, "swfilt"))
+		name = e->swfilt_name;
+
 	if (i == PERF_MEM_EVENTS__LOAD || i == PERF_MEM_EVENTS__LOAD_STORE) {
 		if (e->ldlat) {
 			if (!e->aux_event) {
 				/* ARM and Most of Intel */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name,
+					  name, pmu->name,
 					  perf_mem_events__loads_ldlat);
 			} else {
 				/* Intel with mem-loads-aux event */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name, pmu->name,
+					  name, pmu->name, pmu->name,
 					  perf_mem_events__loads_ldlat);
 			}
 		} else {
 			if (!e->aux_event) {
 				/* AMD and POWER */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name);
+					  name, pmu->name);
 			} else {
 				return NULL;
 			}
@@ -117,7 +130,7 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 
 	if (i == PERF_MEM_EVENTS__STORE) {
 		scnprintf(buf, buf_size,
-			  e->name, pmu->name);
+			  name, pmu->name);
 		return buf;
 	}
 
tools/perf/util/mem-events.h +2 -0
index 5b98076904b0b689..41f628fad10709b9 100644
@@ -11,6 +11,8 @@ struct perf_mem_event {
 	u32		aux_event;
 	const char	*tag;
 	const char	*name;
+	/* Name with the swfilt software privilege filter, when supported. */
+	const char	*swfilt_name;
 	const char	*event_name;
 };
 
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH 01/12] perf test: Skip data_type_profiling when the PMU cannot record memory events

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

From: Arnaldo Carvalho de Melo <[email protected]>

The test records with 'perf mem record' in per-thread mode and its only guard matches one specific message:

  perf mem record -o /dev/null -- true 2>&1 | \
        grep -q "failed: no PMU supports the memory events" && exit 2

A PMU that has memory events but refuses them per-thread falls through it, and AMD IBS does:

  $ perf mem record -o /dev/null -- true
  Error:
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.
[ ... ]
Quoted 4 lines
Fixes f60a5c22967b ("perf tests: Test annotate with data type profiling and rust")
Assisted-by LLM

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#20 lore raw

Re: [PATCH 07/12] perf annotate-data: Show the sample count in the data-type browser

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.

Print the histogram entry's own count instead.

Quoted 7 lines

This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.

Fixes d001c7a7f4736743 ("perf annotate-data: Add hist_entry__annotate_data_tui()")
Assisted-by LLM

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#4 lore raw

Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

From: Arnaldo Carvalho de Melo <[email protected]>

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Quoted 8 lines

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local [ ... ]
that make NO_LIBDW=1 on a system that has the debuginfod client keeps linking.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#24 lore raw

Re: [PATCH 08/12] perf report: Add --progress option

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines

From: Arnaldo Carvalho de Melo <[email protected]>

Processing a large data type profiling session, e.g. an AMD IBS one, can take a long time and, when using the stdio output, 'perf report' gives no feedback about which phase it is in nor about how far along it is: the TUI has a progress bar for that, but in the stdio case the ui_progress updates, that are already there, are dropped on the floor.

Add a --progress option that installs a stdio backend for ui_progress, ui/stdio/progress.c, printing the phase title, the percentage done and the current/total counts to stderr:

Processing events... [ 42.3%] 317M / 746M
Merging related events... [ 61.0%] 309026 / 506686
Sorting events for output... [ 98.2%] 14132 / 14387
[ ... ]
88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Ok ⬢ [acme@toolbx perf-tools-next]$

Quoted 2 lines

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#12 lore raw

Re: [PATCH 04/12] perf debuginfo: Let the user skip and disable debuginfod fetches

In reply to Arnaldo Carvalho de Melo
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf: Thread-unsafe setenv() during concurrent execution risks crashes --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf: Thread-unsafe setenv() during concurrent execution risks crashes --

commit f05833c6d0f3f46ab299b21fc5d5d3a85ee8249e
Author: Arnaldo Carvalho de Melo <[email protected]>

perf debuginfo: Let the user skip and disable debuginfod fetches

This commit introduces interactivity during debuginfod fetches, allowing the user to press 's' to skip the current fetch or 'd' to cancel it and disable debuginfod for the rest of the session. The terminal is placed in raw mode to intercept these keys.

Quoted 4 lines
tools/perf/util/debuginfo.c +0 -0
index be5882d9c2204..e2066fce2450c 100644

[ ... ]

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 210

Quoted 13 lines
tools/perf/util/debuginfo.c +13 -0
+static void debuginfod__cancel_key(int key)
+{
+	if (key == 's' || key == 'S') {
+		debuginfod_fetch_cancelled = true;
+		debuginfod__skipped("Skipping this debuginfod fetch, this build ID will not be fetched again in this session, press 'd' to also disable it for the other ones");
+	} else if (key == 'd' || key == 'D') {
+		debuginfod_fetch_cancelled = true;
+		symbol_conf.debuginfod = false;
+		/*
+		 * libdwfl's own debuginfod client reads DEBUGINFOD_URLS
+		 * in every query, so clear it to stop it as well.
+		 */
+		setenv("DEBUGINFOD_URLS", "", 1);

[Severity: Medium]
Is it safe to call setenv() here? Modifying the global environment can reallocate the environ array or free existing environment strings.

If another thread in a multithreaded tool like perf top concurrently calls getenv() (either explicitly, or implicitly via libc functions like localtime or gettext), could it read from a freed pointer, resulting in a use-after-free or crash?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#26 lore raw

Re: [PATCH 09/12] perf scripts: Add perf-stuck, to tell where a running perf is stuck

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 14 lines

From: Arnaldo Carvalho de Melo <[email protected]>

A perf that takes forever is hard to tell apart from one that is stuck in a loop, and when it is stuck there is no way to know where without attaching gdb to it and looking around, which is what this does, from the outside, sampling /proc/<pid> at a fixed interval:

  ⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf)
  watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s
  09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB  Merging related events... [ 39.0%] 197574 / 506686
  09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686
  09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686
  09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686
Quoted 7 lines

The CPU time used grows by a whole interval on every sample while the [ ... ]
[Inferior 1 (process 2342826) detached]
⬢ [acme@toolbx perf-tools-next]$

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#35 lore raw

Re: [PATCH 12/12] perf mem record: Use the IBS swfilt filter when available

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines

From: Arnaldo Carvalho de Melo <[email protected]>

IBS events with exclude_{user,kernel} bits, as used in per-thread mode when kernel samples are not allowed, are rejected by the kernel on hardware without the privilege filter, so a per-thread 'perf mem record' fails:

  $ perf mem record -o /dev/null -- true
  Error:
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

Kernel v6.14 added swfilt, a software privilege filter that makes those events usable per-thread, exposed as the 'swfilt' format term (d29e744c71673a71 "perf/x86: Relax privilege filter restriction on AMD [ ... ]
("swfilt=1"), that the old 'ldlat=0' check would reject.

Quoted 3 lines

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
#16 lore raw

Re: [PATCH 05/12] perf debuginfo: Show the debuginfod fetch progress and keys in the TUI

In reply to Arnaldo Carvalho de Melo
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf tui: infinite padding loop on narrow terminal width --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf tui: infinite padding loop on narrow terminal width
--

commit bf7df34cfc153e952fa2cf73c078a1a435dfeb1c
Author: Arnaldo Carvalho de Melo <[email protected]>

perf debuginfo: Show the debuginfod fetch progress and keys in the TUI

The stdio only fetch UI left a fetch in progress in the TUI with nothing on screen, indistinguishable from perf being stuck, with no way out of it. The browser thread is the one doing the fetch, so it draws a window over itself with the fetch progress and drains the 's'/'d' keys from the input queue it owns.

Quoted 4 lines
tools/perf/ui/tui/util.c +0 -0
index e4d322ce0b54c..c3274b92b0e22 100644

[ ... ]

Quoted 24 lines
tools/perf/ui/tui/util.c +24 -0
+void ui__progress_window(const char *title, const char *text,
+			 u64 fetched, u64 total)
+{
+	static char lines[PROGRESS_WINDOW_MAX_LINES][256];
+	char buf_cur[20], buf_tot[20], bytes[64];
+	size_t len;
+	int y, height, nr_lines, inner, i;
+
+	if (use_browser != 1)
+		return;
+
+	unit_number__scnprintf(buf_cur, sizeof(buf_cur), fetched);
+	if (total) {
+		unit_number__scnprintf(buf_tot, sizeof(buf_tot), total);
+		scnprintf(bytes, sizeof(bytes), "  %s / %s fetched",
+			  buf_cur, buf_tot);
+	} else {
+		scnprintf(bytes, sizeof(bytes),
+			  "  %s fetched, size unknown", buf_cur);
+	}
+
+	if (progress_window__shown && !strcmp(bytes, progress_window__bytes))
+		return;
+

tools/perf/ui/tui/util.c struct perf_error_ops perf_tui_eops = { · line 390

Quoted 10 lines
tools/perf/ui/tui/util.c +10 -0
+	scnprintf(progress_window__bytes, sizeof(progress_window__bytes),
+		  "%s", bytes);
+	progress_window__shown = true;
+
+	ui__refresh_dimensions(false);
+	mutex_lock(&ui__lock);
+	inner = SLtt_Screen_Cols - 2;
+	if (inner > 255)
+		inner = 255;
+	nr_lines = progress_window__wrap(text, inner, lines);

[Severity: Medium]
If the terminal is resized to 0 or 1 columns during a fetch, could SLtt_Screen_Cols - 2 underflow to a negative value, allowing inner to become negative without bounds checking here?

Quoted 8 lines
tools/perf/ui/tui/util.c +8 -0
+	height = nr_lines + 3;
+
+	SLsmg_set_color(0);
+	if (progress_window__rows)
+		SLsmg_fill_region(progress_window__y, 0, progress_window__rows,
+				  SLtt_Screen_Cols, ' ');
+	y = (SLtt_Screen_Rows - height) / 2;
+	if (y < 0)

tools/perf/ui/tui/util.c struct perf_error_ops perf_tui_eops = { · line 408

Quoted 10 lines
tools/perf/ui/tui/util.c +10 -0
+		y = 0;
+	progress_window__y = y;
+	progress_window__rows = height;
+
+	SLsmg_draw_box(y, 0, height, SLtt_Screen_Cols);
+	SLsmg_gotorc(y++, 1);
+	len = strlen(title);
+	if (len > (size_t)inner)
+		len = inner;
+	SLsmg_write_nchars(title, len);

[Severity: Medium]
If inner is negative, won't casting it to size_t result in a huge unsigned value? This seems like it would bypass the length bounds check and could read out of bounds.

tools/perf/ui/tui/util.c struct perf_error_ops perf_tui_eops = { · line 412

Quoted 4 lines
tools/perf/ui/tui/util.c +4 -0
+	for (i = 0; i < nr_lines; i++, y++) {
+		SLsmg_gotorc(y, 1);
+		SLsmg_write_nstring(lines[i], SLtt_Screen_Cols - 2);
+	}

[Severity: Medium]
Could passing SLtt_Screen_Cols - 2 as a negative value here convert to a massive unsigned parameter in the TUI renderer SLsmg_write_nstring()? This seems like it would cause an endless padding loop that spikes the CPU and locks up the TUI.

tools/perf/ui/tui/util.c

Quoted 2 lines
tools/perf/ui/tui/util.c +2 -0
+	SLsmg_gotorc(y, 1);
+	SLsmg_write_nstring(bytes, SLtt_Screen_Cols - 2);

[Severity: Medium]
Does this also pass the same potentially underflowed width value to the padding function?

Quoted 3 lines
tools/perf/ui/tui/util.c +3 -0
+	SLsmg_refresh();
+	mutex_unlock(&ui__lock);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#28 lore raw

Re: [PATCH 10/12] perf annotate-data: Resolve type DIEs in the debug file they came from

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

From: Arnaldo Carvalho de Melo <[email protected]>

A 'perf report -s type' on a 783 MB AMD IBS data type profiling session hangs, burning all of a CPU and producing no output:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 58.0%] 293828 / 506686
  Merging related events... [ 59.0%] 298894 / 506686
  Merging related events... [ 60.0%] 303960 / 506686
  Merging related events... [ 61.0%] 309026 / 506686
  [ ... nothing else, ever ... ]
Quoted 8 lines

It is a spin and not a slow path: with perf-stuck the CPU time used grows by a whole interval on every sample, while the [stack] mapping, [ ... ]

Fixes 06b2ce75386df04b ("perf annotate-data: Maintain variable type info")
Fixes 55ee3d005d62279d ("perf annotate-data: Add a cache for global variable types")
Assisted-by LLM

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#10 lore raw

Re: [PATCH 03/12] perf config: Move perf_config__set_variable() to util/config.c

In reply to Arnaldo Carvalho de Melo
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] perf: hardcoded system_config=false wipes out system config file - [Low] perf: missing limits.h…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] perf: hardcoded system_config=false wipes out system config file
- [Low] perf: missing limits.h inclusion for PATH_MAX causes musl build failure --

commit dd0d85968ed396c9cbcce556b124511aed73efb9
Author: Arnaldo Carvalho de Melo <[email protected]>

perf config: Move perf_config__set_variable() to util/config.c

It is code from the 'perf config' builtin to set a variable in the configuration file perf is using, the user's ~/.perfconfig, or the file named by the PERF_CONFIG environment variable when it is set, rewriting that file from the config set, the same rewrite 'perf config' does, i.e. the comments in it are not preserved, as the config set carries just the key-value pairs. It has to honour PERF_CONFIG, that makes perf read only that file: the config set is loaded from it, so writing ~/.perfconfig would replace it with the exclusive file's entries.

Quoted 4 lines
tools/perf/util/config.c +0 -0
index 31c6618d3b3da..3f9753b2a7bdf 100644

[ ... ]

Quoted 39 lines
tools/perf/util/config.c +35 -0
@@ -876,6 +876,76 @@ void perf_config__exit(void)
 	config_set = NULL;
 }
 
+int perf_config_set__write(struct perf_config_set *set,
+			   const char *file_name, bool system_config)
+{
+	struct perf_config_section *section = NULL;
+	struct perf_config_item *item = NULL;
+	FILE *fp;
+
+	fp = fopen(file_name, "w");
+	if (!fp)
+		return -1;
+
+	fprintf(fp, "# this file is auto-generated.\n");
+
+	/* overwrite configvariables */
+	perf_config_sections__for_each_entry(&set->sections, section) {
+		if (!system_config && section->from_system_config)
+			continue;
+		fprintf(fp, "[%s]\n", section->name);
+
+		perf_config_items__for_each_entry(&section->items, item) {
+			if (!system_config && item->from_system_config)
+				continue;
+			if (item->value)
+				fprintf(fp, "\t%s = %s\n",
+					item->name, item->value);
+		}
+	}
+	fclose(fp);
+
+	return 0;
+}
+
+/*
+ * Set @var=@value in the configuration file perf is using: the user's
+ * ~/.perfconfig, or the file named by the PERF_CONFIG environment

tools/perf/util/config.c void perf_config__exit(void) · line 923

Quoted 10 lines
tools/perf/util/config.c +10 -0
+ * variable.  The latter has to be honoured as it makes perf read only
+ * that file, so the config set came from it and writing ~/.perfconfig
+ * would replace it with the exclusive file's entries.  Rewriting the
+ * file from the config set is the same rewrite 'perf config' does: the
+ * comments are not preserved, as the config set carries just the
+ * key-value pairs.
+ */
+int perf_config__set_variable(const char *var, const char *value)
+{
+	char path[PATH_MAX];

[Severity: Low]
Does this file need an explicit #include <limits.h>? The function uses the PATH_MAX macro, and while glibc may implicitly include it through other headers, strict musl compilation might fail without the required POSIX-specified header.

Quoted 5 lines
tools/perf/util/config.c +5 -0
+	char *user_config = mkpath(path, sizeof(path), "%s/.perfconfig", getenv("HOME"));
+	const char *config_filename = config_exclusive_filename ?: user_config;
+	struct perf_config_set *set;
+	int ret = -1;
+

tools/perf/util/config.c void perf_config__exit(void) · line 938

Quoted 10 lines
tools/perf/util/config.c +10 -0
+	set = perf_config_set__new();
+	if (!set)
+		goto out_err;
+
+	if (perf_config_set__collect(set, config_filename, var, value) < 0) {
+		pr_err("Failed to add '%s=%s'\n", var, value);
+		goto out_err;
+	}
+
+	if (perf_config_set__write(set, config_filename, /*system_config=*/false) < 0) {

[Severity: High]
Could hardcoding system_config to false here risk wiping out the system configuration file?

If a user sets PERF_CONFIG=/etc/perfconfig (setting config_exclusive_filename), this call to perf_config__set_variable() will read that file. Because the file path matches perf_etc_perfconfig(), all loaded items are flagged with from_system_config = true.

When perf_config_set__write() is then called with system_config=false, it checks:

tools/perf/util/config.c:perf_config_set__write() {
...

		if (!system_config && section->from_system_config)
			continue;

...
}

This actively filters out items that originated from the system config, which could effectively truncate the file and cause data loss for the system-wide configuration.

Quoted 9 lines
tools/perf/util/config.c +9 -0
+		pr_err("Failed to set the configs on %s\n", config_filename);
+		goto out_err;
+	}
+
+	ret = 0;
+out_err:
+	perf_config_set__delete(set);
+	return ret;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#18 lore raw

Re: [PATCH 06/12] perf symbol: Fall back to fetching the vmlinux by build ID

In reply to Arnaldo Carvalho de Melo
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf symbol: NULL pointer dereference in trampoline map adjustment due to lock dropping --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf symbol: NULL pointer dereference in trampoline map adjustment due to lock dropping --

commit 54cc3e4d87729d86b17ea862346d56f11117f916
Author: Arnaldo Carvalho de Melo <[email protected]>

perf symbol: Fall back to fetching the vmlinux by build ID

A profile recorded on a kernel that is no longer installed, be it because the machine was rebooted into a new kernel or because the profile is being processed on another machine, can't have its kernel symbols resolved. As a last resort, when the kernel symbols can't be found locally and the user didn't specify a kallsyms file, fetch the vmlinux keyed by the kernel build ID recorded in the perf.data file using the debuginfod client and use it for symbols.

Quoted 4 lines
tools/perf/util/symbol.c +0 -0
index 1d70c43bb24ee..ccb6217119416 100644

[ ... ]

Quoted 20 lines
tools/perf/util/symbol.c +15 -1
@@ -2257,7 +2280,53 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 		pr_debug("Using %s for symbols\n", kallsyms_filename);
 	free(kallsyms_allocated_filename);
 
-	if (err > 0 && !dso__is_kcore(dso)) {
+	/*
+	 * The kallsyms may be unavailable or restricted, e.g.
+	 * /proc/kallsyms with kernel.perf_event_paranoid > 1, try to fetch
+	 * the vmlinux keyed by the build ID using debuginfod as a last
+	 * resort, honoring --ignore-vmlinux and --ignore-vmlinux_buildid
+	 * like the other vmlinux sources above.
+	 */
+	if (err <= 0 && !user_kallsyms &&
+	    !symbol_conf.ignore_vmlinux &&
+	    !symbol_conf.ignore_vmlinux_buildid) {
+		char *fetched_path = NULL;
+
+		/*
+		 * dso__load() holds dso->lock while it calls us, and the
+		 * fetch below can take a long time, blocked on the network

tools/perf/util/symbol.c static int dso__load_kernel_sym(struct dso *dso, struct map *map) · line 2307

Quoted 10 lines
tools/perf/util/symbol.c +10 -0
+		 * or on the terminal, waiting for the user: do it with the
+		 * lock dropped, as dso__debuginfo() does for the debuginfo
+		 * of a DSO, so that the threads that need this dso don't get
+		 * stuck behind a server round trip.  Nothing of the dso is
+		 * touched by the fetch, the symbols are loaded with the lock
+		 * held again, and only if the fetch brought a file back.
+		 */
+		mutex_unlock(dso__lock(dso));
+		err = dso__fetch_vmlinux_build_id(dso, &fetched_path);
+		mutex_lock(dso__lock(dso));

[Severity: Medium]
Could dropping the lock here lead to a race condition that causes a NULL pointer dereference?

If a second thread enters dso__load() while the lock is dropped here, it would see dso__loaded(dso) is still false and also enter dso__load_kernel_sym(). Both threads would eventually return and consecutively call machine__map_x86_64_entry_trampolines().

Since machine__map_x86_64_entry_trampolines_cb() translates map->pgoff in place without idempotency guarantees:

tools/perf/util/machine.c:machine__map_x86_64_entry_trampolines_cb()

    dest_map = maps__find(args->kmaps, map__pgoff(map));
    if (RC_CHK_ACCESS(dest_map) != RC_CHK_ACCESS(map))
        map__set_pgoff(map, map__map_ip(dest_map, map__pgoff(map)));

Would the second execution fail to find dest_map due to the modified pgoff, and subsequently pass NULL to map__map_ip(), resulting in a fatal NULL pointer dereference?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#32 lore raw

Re: [PATCH 11/12] perf mem record: Request PERF_SAMPLE_CPU by default

In reply to Arnaldo Carvalho de Melo
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 21 lines

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type profiling per-sample stream keys cross-CPU contention on sample->cpu: without PERF_SAMPLE_CPU the cpu field is the (u32)-1 "no CPU info" sentinel, documented as such in perf_session__deliver_event(), and same-instance reads and writes from different cores are indistinguishable from same-CPU traffic, so pahole's false-sharing detector cannot tell them apart.

builtin-record.c already defines --sample-cpu and 'perf mem record' forwards unknown options to the record parser, so passing it explicitly works today; make it the default, next to the -d (addr) and -W (weight) the command already requests, documenting it in perf-mem(1).

The rec_argv array is sized as nine arguments per memory PMU plus the user arguments, not counting the arguments __cmd_record() adds itself, up to eight with all the optional flags. On PMUs with separate load and store events the four event arguments plus the fixed ones already filled the array to its last slot, so this new argument would write past it; reserve space for the fixed arguments explicitly.

Quoted 2 lines

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
reviewer
#5 lore raw

Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Arnaldo Carvalho de Melo
Quoted 14 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Quoted 4 lines

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.

When do we have a build ID but not a DSO? The current intent is that dso__debuginfo hide these complexities. We currently don't purge DSOs and the first call to dso__debuginfo should trigger loading with the DSO owning the debuginfo. With this change we now have a duplication of DSO's data, keyed by build ID and mapping directly to the debuginfo. I can't see a performance or efficiency gain and we could potentially implement some kind of LRU mechanism for DSOs, which would benefit memory usage for things like perf top.

Quoted 5 lines

The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set.

I found the explanation above hard to follow. Are we setting an environment variable because debuginfod isn't respecting its /etc setup?

Quoted 16 lines

That has
to happen before any thread that can call getenv() is started, as setenv() is not thread safe, and there are getenv()s outside the fetch
lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(), which perf also does in build-id.c, probe-event.c and probe-finder.c, so do it from symbol__init(), on the single-threaded setup, and not lazily from the fetch path.

Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.

Quoted 8 lines

The opt-outs also cover libdwfl's own debuginfod client, that reads DEBUGINFOD_URLS in every query: when debuginfod is off, be it with --no-debuginfod, core.debuginfod=false or by disabling the build-id cache, the variable is set to the empty string, that the client treats as an opt-out and fails the query without even looking at its cache, instead of being exported from /etc/debuginfod. Tools that manage DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep doing so.

Libdwfl is part of elfutils and so is debuginfod. Is it possible to share clients?

Quoted 100 lines

The fetches are serialized, as the lookup state they use is process global, so two concurrent fetches would race for it, and serializing also means that a request for a build ID that is being fetched doesn't start a second download of the same file: it waits for the fetch to finish and is answered from the entry the fetch published, keyed by the build ID, with the path of the file that came back, checked before being handed out in case the debuginfod client cache is cleaned from under us.

Make dso__debuginfo() use debuginfo__new_build_id() as a fallback, keyed by the build ID recorded in the perf.data file, so that consumers such as the data type profiler can resolve the types of DSOs whose debuginfo can be fetched this way. Do the fetch outside dso__lock, a fetch from a remote debuginfod server can take a while and would otherwise block anything else using this dso, and remember the build IDs that were a miss, so that consumers revisiting a set of DSOs repeatedly, such as the data type profiler on every hist entry DSO switch, don't pay server round trips per attempt.

debuginfo__new_build_id() needs libdw to open the DWARF, so it lives in debuginfo.o, built only with CONFIG_LIBDW, while the libdebuginfod feature check is independent of NO_LIBDW; keep the new build ID prototypes under HAVE_LIBDW_SUPPORT too, with stubs otherwise, so that make NO_LIBDW=1 on a system that has the debuginfod client keeps linking.

---
 tools/perf/Documentation/perf-annotate.txt |   8 +
 tools/perf/Documentation/perf-config.txt   |  19 ++
 tools/perf/Documentation/perf-report.txt   |  12 +
 tools/perf/Documentation/perf-top.txt      |  11 +
 tools/perf/builtin-annotate.c              |   3 +
 tools/perf/builtin-report.c                |   7 +
 tools/perf/builtin-top.c                   |   7 +
 tools/perf/util/config.c                   |   3 +
 tools/perf/util/debuginfo.c                | 327 +++++++++++++++++++++
 tools/perf/util/debuginfo.h                |  36 +++
 tools/perf/util/dso.c                      |  19 ++
 tools/perf/util/symbol.c                   |  12 +
 tools/perf/util/symbol_conf.h              |   1 +
 tools/perf/util/util.c                     | 103 +++++++
 tools/perf/util/util.h                     |   8 +
 15 files changed, 576 insertions(+)
tools/perf/Documentation/perf-annotate.txt +8 -0
index 1a90b09a12d5abb1..12edc3337b86f099 100644
@@ -58,6 +58,14 @@ OPTIONS
 --ignore-vmlinux::
        Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+       Fetch debuginfo keyed by build ID from the debuginfod servers
+       configured in DEBUGINFOD_URLS, checking the local debuginfod
+       client cache first, when it is not available locally, on for
+       these commands by default.  See the --debuginfod option of
+       'perf report' for how to turn it off.
+
 --itrace::
        Options for decoding instruction tracing data. The options are:
 
tools/perf/Documentation/perf-config.txt +16 -0
index 9b223f8928299945..688306abe847a0df 100644
@@ -216,6 +216,14 @@ core.*::
        addr2line-timeout::
                Sets a timeout (in milliseconds) for parsing 'addr2line'
                output.  The default timeout is 5s.
+       debuginfod::
+               When set to 'false', disable fetching debuginfo keyed by
+               build ID from the debuginfod servers configured in
+               DEBUGINFOD_URLS.  It is on by default, can be overridden per
+               tool with the 'report.debuginfod' and 'top.debuginfod'
+               options and per invocation with --no-debuginfod; it is off
+               too when the local build-id cache is disabled, e.g.
+               'buildid.dir' set to /dev/null.
 
 tui.*, gtk.*::
        Subcommands that can be configured here are 'top', 'report' and 'annotate'.
@@ -562,6 +570,14 @@ report.*::
                This option can change default stat behavior with empty results.
                If it's set true, 'perf report --stat' will not show 0 stats.
 
+       report.debuginfod::
+               Fetch debuginfo keyed by build ID from the debuginfod
+               servers configured in DEBUGINFOD_URLS, checking the local
+               debuginfod client cache first, when it is not available
+               locally.  On by default, set to 'false' to disable it for
+               'perf report', globally with 'core.debuginfod=false' or per
+               invocation with --no-debuginfod.
+
 top.*::
        top.children::
                Same as 'report.children'. So if it is enabled, the output of 'top'

tools/perf/Documentation/perf-config.txt top.*:: · line 590

Quoted 6 lines
tools/perf/Documentation/perf-config.txt +3 -0
@@ -569,6 +585,9 @@ top.*::
                column by default.
                The default is 'true'.
 
+       top.debuginfod::
+               Same as 'report.debuginfod', for 'perf top'.
+

It seems unfortunate to duplicate this option. I wonder if it should be a "core" option. There's likely similar clean up necessary for addr2line.

Quoted 275 lines
tools/perf/Documentation/perf-config.txt +0 -0
        top.call-graph::
                This is identical to 'call-graph.record-mode', except it is
                applicable only for 'top' subcommand. This option ONLY setup
tools/perf/Documentation/perf-report.txt +12 -0
index 1a4706329c6cef1d..2e11c0a97a016853 100644
@@ -366,6 +366,18 @@ OPTIONS
 --ignore-vmlinux::
        Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+       Fetch debuginfo keyed by build ID from the debuginfod servers
+       configured in DEBUGINFOD_URLS, checking the local debuginfod
+       client cache first, when it is not available locally, on for
+       these commands by default.  It can be turned off per invocation
+       with --no-debuginfod, per tool with the "report.debuginfod"
+       config option or globally with "core.debuginfod" set to false.
+       It is disabled as well when the local build-id cache is turned
+       off, e.g. "buildid.dir" set to /dev/null, as that asks for
+       fetched files not to be kept on the box.
+
 --kallsyms=<file>::
         kallsyms pathname
 
tools/perf/Documentation/perf-top.txt +11 -0
index 2da2a16bbf260685..345dd9ee9b6bab88 100644
@@ -83,6 +83,17 @@ Default is to monitor all CPUS.
 --ignore-vmlinux::
        Ignore vmlinux files.
 
+--debuginfod::
+--no-debuginfod::
+       Fetch debuginfo keyed by build ID from the debuginfod servers
+       configured in DEBUGINFOD_URLS, checking the local debuginfod
+       client cache first, when it is not available locally, on for
+       these commands by default.  Turn it off per invocation with
+       --no-debuginfod, with the "top.debuginfod" config option or
+       globally with "core.debuginfod" set to false.  Disabled as well
+       when the build-id cache is off, e.g. "buildid.dir" set to
+       /dev/null.
+
 --kallsyms=<file>::
        kallsyms pathname
 
tools/perf/builtin-annotate.c +3 -0
index 850fc72fa75fe7df..d6a201c1866d363f 100644
@@ -733,6 +733,9 @@ int cmd_annotate(int argc, const char **argv)
        OPT_BOOLEAN(0, "stdio2", &annotate.use_stdio2, "Use the stdio interface"),
        OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
                     "don't load vmlinux even if found"),
+       OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+                   "fetch debuginfo keyed by build ID from the debuginfod "
+                   "servers, on by default, use --no-debuginfod to turn off"),
        OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name,
                   "file", "vmlinux pathname"),
        OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
tools/perf/builtin-report.c +7 -0
index 279e61c2366cb294..442c0822e614197f 100644
@@ -134,6 +134,10 @@ static int report__config(const char *var, const char *value, void *cb)
                symbol_conf.event_group = perf_config_bool(var, value);
                return 0;
        }
+       if (!strcmp(var, "report.debuginfod")) {
+               symbol_conf.debuginfod = perf_config_bool(var, value);
+               return 0;
+       }
        if (!strcmp(var, "report.percent-limit")) {
                double pcnt = strtof(value, NULL);
 
@@ -1346,6 +1350,9 @@ int cmd_report(int argc, const char **argv)
                   "file", "vmlinux pathname"),
        OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
                     "don't load vmlinux even if found"),
+       OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+                   "fetch debuginfo keyed by build ID from the debuginfod "
+                   "servers, on by default, use --no-debuginfod to turn off"),
        OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
                   "file", "kallsyms pathname"),
        OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
tools/perf/builtin-top.c +7 -0
index c2562d49be46a9a1..2be6859fd4aee141 100644
@@ -1436,6 +1436,10 @@ static int perf_top_config(const char *var, const char *value, void *cb __maybe_
                symbol_conf.cumulate_callchain = perf_config_bool(var, value);
                return 0;
        }
+       if (!strcmp(var, "top.debuginfod")) {
+               symbol_conf.debuginfod = perf_config_bool(var, value);
+               return 0;
+       }
 
        return 0;
 }
@@ -1508,6 +1512,9 @@ int cmd_top(int argc, const char **argv)
                   "file", "vmlinux pathname"),
        OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
                    "don't load vmlinux even if found"),
+       OPT_BOOLEAN(0, "debuginfod", &symbol_conf.debuginfod,
+                   "fetch debuginfo keyed by build ID from the debuginfod "
+                   "servers, on by default, use --no-debuginfod to turn off"),
        OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
                   "file", "kallsyms pathname"),
        OPT_BOOLEAN('K', "hide_kernel_symbols", &top.hide_kernel_symbols,
tools/perf/util/config.c +3 -0
index b2972c35c1eca68c..31c6618d3b3daf22 100644
@@ -470,6 +470,9 @@ static int perf_default_core_config(const char *var, const char *value)
        if (!strcmp(var, "core.addr2line-disable-warn"))
                symbol_conf.addr2line_disable_warn = perf_config_bool(var, value);
 
+       if (!strcmp(var, "core.debuginfod"))
+               symbol_conf.debuginfod = perf_config_bool(var, value);
+
        /* Add other config variables here. */
        return 0;
 }
tools/perf/util/debuginfo.c +124 -0
index 84a78b30ceac1066..be5882d9c2204797 100644
@@ -7,16 +7,22 @@
 
 #include <errno.h>
 #include <fcntl.h>
+#include <limits.h>
+#include <pthread.h>
+#include <stdbool.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <linux/list.h>
 #include <linux/zalloc.h>
+#include <api/fs/fs.h>
 
 #include "build-id.h"
 #include "dso.h"
 #include "debug.h"
 #include "debuginfo.h"
+#include "mutex.h"
 #include "symbol.h"
 
 #ifdef HAVE_DEBUGINFOD_SUPPORT
@@ -139,6 +145,327 @@ struct debuginfo *debuginfo__new(const char *path)
        return __debuginfo__new(buf);
 }
 
+#ifdef HAVE_DEBUGINFOD_SUPPORT
+/*
+ * Users can disable the local build-id/.debug cache by setting
+ * buildid.dir to /dev/null, meaning they don't want fetched
+ * binaries/debuginfo stored on the box; the debuginfod client keeps
+ * its own cache in ~/.cache/debuginfod_client, so honour that intent
+ * and don't fetch at all in that case.
+ */
+static bool debuginfod__cache_disabled(void)
+{
+       return !strcmp(buildid_dir, "/dev/null");
+}
+
+/*
+ * Build IDs that shouldn't be searched for again in this session: the
+ * ones already searched for on the debuginfod servers without success,
+ * so that callers that see the same DSO over and over, such as the data
+ * type profiler switching between DSOs on every hist entry, don't pay a
+ * server round trip again for each miss.  The cache of successes is the
+ * debuginfod client's own, in the local filesystem.
+ *
+ * Guarded by debuginfod__fetch_lock: it is only read by the lookups
+ * below, that run with that lock held, and written by fetches, that run
+ * with it held too.
+ */
+struct debuginfod_miss {
+       struct list_head        node;
+       struct build_id         bid;
+};
+
+static LIST_HEAD(debuginfod__misses);
+
+/*
+ * Was the search for this build ID already made and settled, i.e. the
+ * servers had nothing for it?
+ */
+static bool debuginfod__missed(const struct build_id *bid)
+{
+       struct debuginfod_miss *miss;
+
+       list_for_each_entry(miss, &debuginfod__misses, node) {
+               if (miss->bid.size == bid->size &&
+                   !memcmp(miss->bid.data, bid->data, bid->size))
+                       return true;
+       }
+
+       return false;
+}
+
+static void debuginfod__miss_add(const struct build_id *bid)
+{
+       struct debuginfod_miss *miss = zalloc(sizeof(*miss));
+
+       if (miss == NULL)
+               return;
+
+       miss->bid = *bid;
+
+       list_add(&miss->node, &debuginfod__misses);
+}
+
+/*
+ * One fetch at a time.
+ *
+ * The lookup state below is process global, so two concurrent fetches,
+ * which dso__debuginfo() makes possible by taking the fetch out of
+ * dso__lock, would race for it: the second one could answer from a list
+ * the first one is concurrently updating, and put the same build ID on
+ * the misses list twice.
+ *
+ * Serializing also means a second request for a build ID that is being
+ * fetched waits here for the fetch to finish, instead of starting a second
+ * download of the same file, and is then answered from the entry the fetch
+ * published, or from the misses list, with no client at all.  Should the
+ * fetches ever run in parallel, that wait has to come back explicitly, with
+ * this lock split in two: one only for the lookup state, that the waiters
+ * sleep on, and one held around each fetch, with the thread that finds a
+ * fetch in progress waiting on the former for the entry to be published.
+ *
+ * What that costs is that a fetch for one build ID blocks a fetch for
+ * another one, and it is what parallel downloads would fix.  Worth doing
+ * only if the wait turns out to be long, because it mostly is not: the
+ * lookups below answer the second and later requests for a build ID from
+ * memory, so after the first pass over the build IDs of a workload, which
+ * is the only time anything is fetched at all, the serialization has
+ * nothing left to serialize.  Start there if a profile with many DSOs to
+ * fetch shows up in a profile of perf itself.
+ */
+static struct mutex debuginfod__fetch_lock;
+
+static void debuginfod__fetch_lock_setup(void)
+{
+       mutex_init(&debuginfod__fetch_lock);
+}
+
+static void debuginfod__fetch_lock_init(void)
+{
+       static pthread_once_t once = PTHREAD_ONCE_INIT;
+
+       pthread_once(&once, debuginfod__fetch_lock_setup);
+}
+
+/*
+ * The build IDs already fetched in this session, and the path of the file
+ * that came back for each, so that the repeated requests for the same build
+ * ID, dso__debuginfo() is called per symbol annotated, are answered with a
+ * strdup() instead of another client: the file is in the debuginfod client
+ * cache already and its path checked before being handed out, in case that
+ * cache is cleaned from under us.
+ *
+ * Guarded by debuginfod__fetch_lock.  Only a fetch that brought a file back
+ * gets an entry: one that didn't is recorded in debuginfod__misses as a miss,
+ * and that is what keeps the rest of the session from asking for it again.
+ * Like debuginfod__misses this grows with the number of build IDs in the
+ * workload, one small entry each, and is not trimmed.
+ */
+struct debuginfo_lookup {
+       struct list_head         node;

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 275

Quoted 10 lines
tools/perf/util/debuginfo.c +10 -0
+       struct build_id          bid;
+       char                    *path;
+};
+
+static LIST_HEAD(debuginfo_lookups);
+
+static bool build_id__equal(const struct build_id *a, const struct build_id *b)
+{
+       return a->size == b->size && memcmp(a->data, b->data, a->size) == 0;
+}

This is probably worth moving to the build-id.[ch] file. I see similar logic in places like __dso_id__cmp, dso__missing_buildid_cache in builitin-buildid-cache.c and sort__dcacheline_cmp. dso__build_id_equal has some special backward compatibility checks.

[snip]

Thanks,
Ian

maintainer
#8 lore raw

Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Arnaldo Carvalho de Melo
Quoted 18 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.

Quoted 11 lines

The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set. That has to happen before any thread that can call getenv() is started, as setenv() is not thread safe, and there are getenv()s outside the fetch
lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(), which perf also does in build-id.c, probe-event.c and probe-finder.c, so do it from symbol__init(), on the single-threaded setup, and not lazily from the fetch path.

I think it's better to split the URL handling into a separate patch. Also it seems we have "buildid-cache.debuginfod" config option.

Thanks,
Namhyung

Quoted 46 lines

Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.

The opt-outs also cover libdwfl's own debuginfod client, that reads DEBUGINFOD_URLS in every query: when debuginfod is off, be it with --no-debuginfod, core.debuginfod=false or by disabling the build-id cache, the variable is set to the empty string, that the client treats as an opt-out and fails the query without even looking at its cache, instead of being exported from /etc/debuginfod. Tools that manage DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep doing so.

The fetches are serialized, as the lookup state they use is process global, so two concurrent fetches would race for it, and serializing also means that a request for a build ID that is being fetched doesn't start a second download of the same file: it waits for the fetch to finish and is answered from the entry the fetch published, keyed by the build ID, with the path of the file that came back, checked before being handed out in case the debuginfod client cache is cleaned from under us.

Make dso__debuginfo() use debuginfo__new_build_id() as a fallback, keyed by the build ID recorded in the perf.data file, so that consumers such as the data type profiler can resolve the types of DSOs whose debuginfo can be fetched this way. Do the fetch outside dso__lock, a fetch from a remote debuginfod server can take a while and would otherwise block anything else using this dso, and remember the build IDs that were a miss, so that consumers revisiting a set of DSOs repeatedly, such as the data type profiler on every hist entry DSO switch, don't pay server round trips per attempt.

debuginfo__new_build_id() needs libdw to open the DWARF, so it lives in debuginfo.o, built only with CONFIG_LIBDW, while the libdebuginfod feature check is independent of NO_LIBDW; keep the new build ID prototypes under HAVE_LIBDW_SUPPORT too, with stubs otherwise, so that make NO_LIBDW=1 on a system that has the debuginfod client keeps linking.

maintainer
#13 lore raw

Re: [PATCH 04/12] perf debuginfo: Let the user skip and disable debuginfod fetches

In reply to Arnaldo Carvalho de Melo
Quoted 37 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

When a fetch is in progress, the way it prints progress is how one gets out of it: 's' aborts the current fetch via the debuginfod client's progress callback protocol and remembers the build ID, so that the rest of the session doesn't ask for it again, the user may have skipped it for being too big; 'd' additionally disables debuginfod for the rest of the session, clearing DEBUGINFOD_URLS so that libdwfl's own client, that reads it in every query, stops fetching too, and, using perf_config__set_variable(), moved to util/config.c in the previous patch, writes core.debuginfod=false to the configuration file directly, the user's ~/.perfconfig or the one named by PERF_CONFIG, the same rewrite 'perf config' does -- the comments are not preserved, as the config set carries just the key-value pairs -- and SIGINT, SIGQUIT and SIGTERM are intercepted while the terminal is in raw mode, so that it is restored and the signal is re-raised when the user interrupts a fetch.

In the TUI nothing is shown yet, printing to stderr would garble the browser display and draining stdin would eat the keys the browser wants, that comes in the next patch.

The interaction state -- the terminal settings, the signal dispositions and the progress and cancellation state -- is process global, so the fetches stay serialized: a second fetch resetting the cancellation state would drop the 's'/'d' keypress that was meant for the fetch already in progress, and the serialization the lookup state needs already provides the lock the interaction state uses.

Since the debuginfo fetching this builds on is opt-out, a fetch that the user asked to skip is not restarted by the next request for the same build ID in the same session, it is remembered as a cancellation on the misses list, where the debug message can tell it apart from a plain miss.

---

[SNIP]

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 229

Quoted 10 lines
tools/perf/util/debuginfo.c +10 -0
+/*
+ * The stdio fetch UI: the terminal is in raw mode, see
+ * debuginfod__fetch(), so the keypresses are on stdin, drain them.
+ */
+static void debuginfod__poll_cancel_keys(void)
+{
+	char ch;
+
+	while (read(STDIN_FILENO, &ch, 1) == 1)
+		debuginfod__cancel_key(ch);

Isn't it block indefinitely when users don't press any key? I'm curious how it can escape from the loop when it finishes to fetch.

Thanks,
Namhyung

Quoted 50 lines
tools/perf/util/debuginfo.c +50 -0
+}
+
+/*
+ * Print a warning and a progress indicator when the debuginfod client
+ * ends up fetching a file, which can be big, such as the vmlinux for a
+ * kernel profiled on another machine or before it got upgraded, so that
+ * users know perf is not stuck, and let them bail out: 's' skips this
+ * fetch and remembers the build ID, so that the rest of the session
+ * doesn't ask for it again, 'd' also disables debuginfod for the rest
+ * of the session.
+ *
+ * The client invokes this both while fetching, where 'a' is the number
+ * of bytes transferred so far and 'b' the total size, zero when it
+ * doesn't know it yet, and, before committing to a server, from the
+ * cache cleanup, that scans the debuginfod client cache, with 'a' being
+ * the number of cache files scanned so far and 'b' zero.
+ *
+ * In the stdio case the progress goes to stderr, a \r terminated line,
+ * the keys are drained from stdin, that debuginfod__fetch() put in raw
+ * mode.
+ */
+static int debuginfod_progress_fn(debuginfod_client *c __maybe_unused,
+				  long a, long b)
+{
+	if (debuginfod_signal)
+		return 1;
+
+	if (!isatty(STDERR_FILENO) || use_browser)
+		return 0;
+
+	if (isatty(STDIN_FILENO)) {
+		debuginfod__poll_cancel_keys();
+		if (debuginfod_fetch_cancelled)
+			return 1;
+	}
+
+	if (!debuginfod_progress_started) {
+		fprintf(stderr, "Fetching debuginfo by build ID from the debuginfod servers, this may take a while for large files such as the vmlinux, press 's' to skip, 'd' to skip and disable\n");
+		debuginfod_progress_started = true;
+	}
+
+	if (a >= 0) {
+		if (b > 0)
+			fprintf(stderr, "  %ld/%ld MiB fetched\r", a >> 20, b >> 20);
+		else
+			fprintf(stderr, "  %ld MiB fetched\r", a >> 20);
+	}
+
+	return 0;
+}
author maintainer
#6 lore raw

Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Ian Rogers
Quoted 13 lines · Ian Rogers
Quoted 12 lines · Arnaldo Carvalho de Melo

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Quoted 6 lines
Quoted 4 lines

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.

When do we have a build ID but not a DSO? The current intent is that

Trying to parse this: Before we had a PERF_RECORD_MMAP with a dso name that we, at the end, when enabled, would look for a build-id to add to the perf.data headers, now we get both in the PERF_RECORD_MMAP3, right?

Quoted 1 line

dso__debuginfo hide these complexities. We currently don't purge DSOs

From the cache, right, and that is a problem, we need to do that LRU you mention, to not have a ever growing cache.

There is a recent patch from someone at Uber about another aspect of this, the symtabs loaded in memory for resolving symbols are not in any way constrained, we go on loading, not purging, hope that patch gets resubmitted addressing the sashiko reviews that were acknowledged.

Quoted 13 lines

and the first call to dso__debuginfo should trigger loading with the DSO owning the debuginfo. With this change we now have a duplication of DSO's data, keyed by build ID and mapping directly to the debuginfo. I can't see a performance or efficiency gain and we could potentially implement some kind of LRU mechanism for DSOs, which would benefit memory usage for things like perf top.

Quoted 5 lines

The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set.

Quoted 3 lines

I found the explanation above hard to follow. Are we setting an environment variable because debuginfod isn't respecting its /etc setup?

IIRC what I saw was the debuginfod client not finding things in the cache when that DEBUGINFO_URLS variable wasn't set, so setting it doesn't mean to ask for downloads necessarily, but to use what is already cached locally.

Quoted 25 lines
Quoted 16 lines

That has
to happen before any thread that can call getenv() is started, as setenv() is not thread safe, and there are getenv()s outside the fetch
lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(), which perf also does in build-id.c, probe-event.c and probe-finder.c, so do it from symbol__init(), on the single-threaded setup, and not lazily from the fetch path.

Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.

Quoted 8 lines

The opt-outs also cover libdwfl's own debuginfod client, that reads DEBUGINFOD_URLS in every query: when debuginfod is off, be it with --no-debuginfod, core.debuginfod=false or by disabling the build-id cache, the variable is set to the empty string, that the client treats as an opt-out and fails the query without even looking at its cache, instead of being exported from /etc/debuginfod. Tools that manage DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep doing so.

Quoted 2 lines

Libdwfl is part of elfutils and so is debuginfod. Is it possible to share clients?

We need to stop using ~/.debug/ and move to have the cache where elfutils libraries have it.

Transitioning should just use ~/.debug if available but saving copies of local DSOs like perf does in the elfutils cache directory.

<SNIP>

Quoted 9 lines

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 275

Quoted 4 lines
tools/perf/util/debuginfo.c +4 -0
+static bool build_id__equal(const struct build_id *a, const struct build_id *b)
+{
+       return a->size == b->size && memcmp(a->data, b->data, a->size) == 0;
+}

This is probably worth moving to the build-id.[ch] file. I see similar logic in places like __dso_id__cmp, dso__missing_buildid_cache in builitin-buildid-cache.c and sort__dcacheline_cmp. dso__build_id_equal has some special backward compatibility checks.

Agreed, will do.

- Arnaldo

author maintainer
#14 lore raw

Re: [PATCH 04/12] perf debuginfo: Let the user skip and disable debuginfod fetches

In reply to Namhyung Kim
Quoted 39 lines · Namhyung Kim
Quoted 37 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

When a fetch is in progress, the way it prints progress is how one gets out of it: 's' aborts the current fetch via the debuginfod client's progress callback protocol and remembers the build ID, so that the rest of the session doesn't ask for it again, the user may have skipped it for being too big; 'd' additionally disables debuginfod for the rest of the session, clearing DEBUGINFOD_URLS so that libdwfl's own client, that reads it in every query, stops fetching too, and, using perf_config__set_variable(), moved to util/config.c in the previous patch, writes core.debuginfod=false to the configuration file directly, the user's ~/.perfconfig or the one named by PERF_CONFIG, the same rewrite 'perf config' does -- the comments are not preserved, as the config set carries just the key-value pairs -- and SIGINT, SIGQUIT and SIGTERM are intercepted while the terminal is in raw mode, so that it is restored and the signal is re-raised when the user interrupts a fetch.

In the TUI nothing is shown yet, printing to stderr would garble the browser display and draining stdin would eat the keys the browser wants, that comes in the next patch.

The interaction state -- the terminal settings, the signal dispositions and the progress and cancellation state -- is process global, so the fetches stay serialized: a second fetch resetting the cancellation state would drop the 's'/'d' keypress that was meant for the fetch already in progress, and the serialization the lookup state needs already provides the lock the interaction state uses.

Since the debuginfo fetching this builds on is opt-out, a fetch that the user asked to skip is not restarted by the next request for the same build ID in the same session, it is remembered as a cancellation on the misses list, where the debug message can tell it apart from a plain miss.

---

[SNIP]

Quoted 13 lines

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 229

Quoted 10 lines
tools/perf/util/debuginfo.c +10 -0
+/*
+ * The stdio fetch UI: the terminal is in raw mode, see
+ * debuginfod__fetch(), so the keypresses are on stdin, drain them.
+ */
+static void debuginfod__poll_cancel_keys(void)
+{
+	char ch;
+
+	while (read(STDIN_FILENO, &ch, 1) == 1)
+		debuginfod__cancel_key(ch);

Isn't it block indefinitely when users don't press any key? I'm curious how it can escape from the loop when it finishes to fetch.

With the help of AI:

it can't block — it's a drain, not a wait. set_term_quiet_input() (util/term.c) puts the terminal in raw mode with VMIN = 0; VTIME = 0, so read() returns 0 when there is nothing typed; the while (read(...) == 1) loop drains what was typed and returns to the progress callback, which returns to libdebuginfod, which carries on fetching and calls it again at the next update. When the fetch finishes there are no more callbacks.

But then there are other issues in that series, from Ian, and from myself, locally, I'll submit a new version soon.

- Arnaldo

maintainer Acked-by on v6 7/12
#21 lore raw

Re: [PATCH 07/12] perf annotate-data: Show the sample count in the data-type browser

In reply to Arnaldo Carvalho de Melo
Quoted 10 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.

Print the histogram entry's own count instead.

Quoted 7 lines

This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.

Fixes d001c7a7f4736743 ("perf annotate-data: Add hist_entry__annotate_data_tui()")
Assisted-by LLM

This can be applied independently.

Thanks,
Namhyung

Quoted 20 lines
---
 tools/perf/ui/browsers/annotate-data.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/ui/browsers/annotate-data.c +1 -1
index c6e07a9b64089ab5..a15608f6ee53335f 100644
@@ -374,7 +374,7 @@ static void browser__write_overhead(struct ui_browser *uib,
 	u64 period = hist->period;
 	double percent = total->period ? (100.0 * period / total->period) : 0;
 	bool current = ui_browser__is_current_entry(uib, row);
-	int nr_samples = 0;
+	int nr_samples = hist->nr_samples;
 
 	ui_browser__set_percent_color(uib, percent, current);
 
Signature · 1 line
2.55.0
reviewer
#7 lore raw

Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

In reply to Arnaldo Carvalho de Melo
Quoted 20 lines · Arnaldo Carvalho de Melo
Quoted 17 lines · Ian Rogers
Quoted 16 lines · Arnaldo Carvalho de Melo

perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.

Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.

Quoted 5 lines
Quoted 1 line

When do we have a build ID but not a DSO? The current intent is that

Trying to parse this: Before we had a PERF_RECORD_MMAP with a dso name that we, at the end, when enabled, would look for a build-id to add to the perf.data headers, now we get both in the PERF_RECORD_MMAP3, right?

I believe the dso_id allows look up of a DSO before having its name.

Quoted 4 lines
Quoted 1 line

dso__debuginfo hide these complexities. We currently don't purge DSOs

From the cache, right, and that is a problem, we need to do that LRU you mention, to not have a ever growing cache.

Yeah, I think we can periodically scan the machine's DSOs, and if the reference count is one and some LRU metric is met, we can drop the reference count to 0 and remove it from the DSOs array. Since the debuginfo is owned by the DSO, it gets cleaned up as well.

Quoted 4 lines

There is a recent patch from someone at Uber about another aspect of this, the symtabs loaded in memory for resolving symbols are not in any way constrained, we go on loading, not purging, hope that patch gets resubmitted addressing the sashiko reviews that were acknowledged.

I've missed that one, but it sounds good.

Quoted 13 lines
Quoted 12 lines

and the first call to dso__debuginfo should trigger loading with the DSO owning the debuginfo. With this change we now have a duplication of DSO's data, keyed by build ID and mapping directly to the debuginfo. I can't see a performance or efficiency gain and we could potentially implement some kind of LRU mechanism for DSOs, which would benefit memory usage for things like perf top.

Quoted 5 lines

The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set.

Quoted 8 lines
Quoted 3 lines

I found the explanation above hard to follow. Are we setting an environment variable because debuginfod isn't respecting its /etc setup?

IIRC what I saw was the debuginfod client not finding things in the cache when that DEBUGINFO_URLS variable wasn't set, so setting it doesn't mean to ask for downloads necessarily, but to use what is already cached locally.

Perhaps we need to get this fixed in libdebuginfod. Or should the local connection to debuginfod automatically scan the /etc directory. It sounds like an elfutils bug that we may need to work around, but we should probably also raise the issue with elfutils.

Quoted 25 lines
Quoted 24 lines
Quoted 24 lines

That has
to happen before any thread that can call getenv() is started, as setenv() is not thread safe, and there are getenv()s outside the fetch
lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(), which perf also does in build-id.c, probe-event.c and probe-finder.c, so do it from symbol__init(), on the single-threaded setup, and not lazily from the fetch path.

Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.

The opt-outs also cover libdwfl's own debuginfod client, that reads DEBUGINFOD_URLS in every query: when debuginfod is off, be it with --no-debuginfod, core.debuginfod=false or by disabling the build-id cache, the variable is set to the empty string, that the client treats as an opt-out and fails the query without even looking at its cache, instead of being exported from /etc/debuginfod. Tools that manage DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep doing so.

Quoted 5 lines
Quoted 2 lines

Libdwfl is part of elfutils and so is debuginfod. Is it possible to share clients?

We need to stop using ~/.debug/ and move to have the cache where elfutils libraries have it.

+1

Quoted 18 lines

Transitioning should just use ~/.debug if available but saving copies of local DSOs like perf does in the elfutils cache directory.

<SNIP>

Quoted 9 lines

tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 275

Quoted 4 lines
tools/perf/util/debuginfo.c +4 -0
+static bool build_id__equal(const struct build_id *a, const struct build_id *b)
+{
+       return a->size == b->size && memcmp(a->data, b->data, a->size) == 0;
+}

This is probably worth moving to the build-id.[ch] file. I see similar logic in places like __dso_id__cmp, dso__missing_buildid_cache in builitin-buildid-cache.c and sort__dcacheline_cmp. dso__build_id_equal has some special backward compatibility checks.

Agreed, will do.

- Arnaldo

maintainer
#29 lore raw

Re: [PATCH 10/12] perf annotate-data: Resolve type DIEs in the debug file they came from

In reply to Arnaldo Carvalho de Melo
Quoted 239 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

A 'perf report -s type' on a 783 MB AMD IBS data type profiling session hangs, burning all of a CPU and producing no output:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 58.0%] 293828 / 506686
  Merging related events... [ 59.0%] 298894 / 506686
  Merging related events... [ 60.0%] 303960 / 506686
  Merging related events... [ 61.0%] 309026 / 506686
  [ ... nothing else, ever ... ]

It is a spin and not a slow path: with perf-stuck the CPU time used grows by a whole interval on every sample, while the [stack] mapping, which would be moving down if this was recursion, stays put:

  ⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf)
  watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s
  09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB  Merging related events... [ 39.0%] 197574 / 506686
  09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686
  09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686
  09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686

Attaching gdb and stepping the loop, printing dwarf_dieoffset() and dwarf_tag() for the DIE being chased on each trip round it, shows the chase never moving, on a typedef that refers to itself:

  ⬢ [acme@toolbx perf-tools-next]$ gdb -p $(pgrep -x perf) -batch -x tools/perf/scripts/perf-stuck.gdb -ex 'perf-die-chain die_get_pointer_type type_die 8' -ex perf-dso -ex detach
  stuck in die_get_pointer_type():
  #3  0x0000000000772421 in die_get_pointer_type (type_die=0x7ffc3ae4b710, type_die@entry=0x7ffc3ae4b6f0, die_mem=die_mem@entry=0x7ffc3ae4b710) at util/dwarf-aux.c:327
  327			type_die = die_get_type(type_die, die_mem);
  chain[0] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null)
  chain[1] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null)
  [ ... the very same DIE, forever ... ]
  dso=/usr/lib64/libz.so.1.3.1.zlib-ng ip=0xe2e sym=build_tree

The DIE is at offset 0x1f of the debug info of libz.so.1, which is zlib-ng, and is one of the dwz compressed ones: the type DIEs shared by more than one CU live in the common file, where 0x1f is a perfectly good DW_TAG_base_type:

⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/.dwz/zlib-ng-2.3.3-3.fc44.x86_64 | sed -n '/Compilation Unit @ offset 0:/,/Compilation Unit @ offset 0x5f:/p'
Compilation Unit @ offset 0:
Length: 0x5b (32-bit)
Version: 5
Unit Type: DW_UT_partial (3)
<0><c>: Abbrev Number: 1 (DW_TAG_partial_unit)
[...]
<1><1f>: Abbrev Number: 62 (DW_TAG_base_type)

      <20>   DW_AT_byte_size   : 4
      <21>   DW_AT_encoding    : 5	(signed)
      <22>   DW_AT_name        : int

while in the main debug file that same offset is not a DIE at all, it is the start of another unit's header:

⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/usr/lib64/libz.so.1.3.1.zlib-ng-2.3.3-3.fc44.x86_64.debug | grep 'Compilation Unit @' | head -2
Compilation Unit @ offset 0:
Compilation Unit @ offset 0x1f:

die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, and update_var_state() then hands that offset to dwarf_offdie() together with the main debug file, where dwarf_offdie() parses whatever is there as a DIE, in this case the one typedef that refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin.

So record, next to the offset, whether the type DIE was in the file the variable DIE came from or in the dwz common one, and resolve the offset in the file it came from, which is what the new die_get_type_die() does.

Which of the two that is does not have to be guessed from what is at the
offset: dwz encodes the references into its common file as DW_FORM_GNU_ref_alt, so elfutils resolves them into the alt Dwarf and the CU of the resulting type DIE belongs to that other file, so comparing the Dwarf each of the two CUs belongs to, die_same_file(), settles it exactly, for any number of hops from the variable DIE.

Then bound the chases themselves: no sane chain of typedefs and qualifiers is 32 DIEs long, no sane nesting for the struct and union members that __add_member_cb() follows recursively is 8 deep, and the same recursion bound covers the type names die_get_typename_from_type() builds by following pointers and arrays, so that a debug info file broken in some other way makes perf give up on a type, telling about it with pr_debug, visible with -v, instead of looking like it hung. The member nesting bound is reported with pr_debug_dtp (visible with -vvv or -D type-profile).

Cycles can't occur in member trees from valid DWARF, embedded members can't be recursive in C, so the nesting bound only ever bites legitimate depth and the member where the recursion was cut is marked 'truncated', which the JSON exporter added in a subsequent series reports to its consumers, so that they can tell a truncated tree from one that really ends there.

die_get_type_die() then has no fallback to the other file: resolving an alt file offset in the main file is the misparse above, so it resolves in the file the offset was recorded as belonging to, and gives up on the type when the offset does not resolve there. The tag the type DIE had is kept as a sanity check, a mismatch means the debug info changed under perf or is broken in yet another way, and the bounds on the type chasers remain the backstop: a debug info file broken in some other way still makes perf give up on a type with a pr_debug instead of hanging.

Testing:

Before, on the 783 MB AMD IBS session, killed after some 8 minutes stuck at 61%:

  ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null
  Processing events... [100.0%] 746M / 746M
  Merging related events... [ 61.0%] 309026 / 506686
  ⬢ [acme@toolbx perf-tools-next]$

After, the whole session is processed, and the zlib-ng types, from the build_tree() hist entry that used to hang it, show up:

⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > ibs.out Processing events... [100.0%] 746M / 746M
Merging related events... [100.0%] 506686 / 506686
Sorting events for output... [100.0%] 1428 / 1428
⬢ [acme@toolbx perf-tools-next]$ grep -E "deflate_state|inflate_state|internal_state" ibs.out

      0.00%  deflate_state
      0.00%  deflate_state*
      0.00%  struct inflate_state
      0.00%  struct inflate_state*
      0.00%  struct internal_state
  ⬢ [acme@toolbx perf-tools-next]$ perf test 17 27 30 31 85 88
  17: Match and link multiple hists                              : Ok
  27: Filter hist entries                                        : Ok
  30: Sort output of hist entries                                : Ok
  31: Cumulate child hist entries                                : Ok

85: Test that perf report includes file offsets and event type names in diagnostic messages. : Ok 88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Skip ⬢ [acme@toolbx perf-tools-next]$

Requiring elfutils 0.160 for this: dwarf_cu_getdwarf(), the function that tells which Dwarf a CU belongs to, first appeared in 0.160 ("libdw:
New functions dwarf_cu_getdwarf, dwarf_cu_die", elfutils NEWS), so the libdw feature test now probes for it, in tools/build/feature/test-libdw.c, and Makefile.config says 0.160 where it said 0.157. The probe takes the address instead of calling it, as that is all that is needed to make a 0.157-0.159 elfutils, from 2014 and without the symbol, disable dwarf support with the existing message rather than fail to link dwarf-aux.c.

Fixes 06b2ce75386df04b ("perf annotate-data: Maintain variable type info")
Fixes 55ee3d005d62279d ("perf annotate-data: Add a cache for global variable types")
Assisted-by LLM
---
 tools/build/feature/test-libdw.c |  15 ++-
 tools/perf/Makefile.config       |   2 +-
 tools/perf/util/annotate-data.c  |  59 ++++++++---
 tools/perf/util/annotate-data.h  |   3 +
 tools/perf/util/dwarf-aux.c      | 171 +++++++++++++++++++++++++++----
 tools/perf/util/dwarf-aux.h      |  33 ++++++
 6 files changed, 247 insertions(+), 36 deletions(-)
tools/build/feature/test-libdw.c +14 -1
index aabd63ca76b4d7e6..23e1ba6ff3466b9f 100644
@@ -49,8 +49,21 @@ int test_elfutils(void)
 	return 0;
 }
 
+/*
+ * elfutils 0.160 and later: used to tell which debug file a DIE lives in,
+ * the dwz alt file or the main one, see die_same_file() in
+ * tools/perf/util/dwarf-aux.c.  Only the symbol is needed, so take its
+ * address instead of calling it.
+ */
+int test_libdw_cu_getdwarf(void)
+{
+	void *sym = (void *)dwarf_cu_getdwarf;
+
+	return sym == NULL;
+}
+
 int main(void)
 {
 	return test_libdw() + test_libdw_unwind() + test_libdw_getlocations() +
-	       test_libdw_getcfi() + test_elfutils();
+	       test_libdw_getcfi() + test_libdw_cu_getdwarf() + test_elfutils();
 }
tools/perf/Makefile.config +1 -1
index 4ee7393a39f91d85..0585d71e3182c6dd 100644
@@ -470,7 +470,7 @@ else
   else
     ifneq ($(feature-libdw), 1)
       ifndef NO_LIBDW
-        $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.157, disables dwarf support. Please install new elfutils-devel/libdw-dev)
+        $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.160, disables dwarf support. Please install new elfutils-devel/libdw-dev)
         NO_LIBDW := 1
       endif
     endif # Dwarf support
tools/perf/util/annotate-data.c +19 -0
index aff60a630fd05b01..e67c9a21a7a27af2 100644
@@ -221,6 +221,15 @@ static bool data_type_less(struct rb_node *node_a, const struct rb_node *node_b)
 	return strcmp(a->self.type_name, b->self.type_name) < 0;
 }
 
+/*
+ * Members of struct/union members are added recursively, and the same DIE
+ * that is not what it looks like, the one that makes the type chasers in
+ * util/dwarf-aux.c spin, can make a member's type point back at one of its
+ * own ancestors, recursing until the stack is gone.  Nothing usable comes
+ * out of nesting members this deep anyway.
+ */
+#define MAX_MEMBER_DEPTH 8
+
 /* Recursively add new members for struct/union */
 static int __add_member_cb(Dwarf_Die *die, void *arg)
 {
@@ -235,6 +244,16 @@ static int __add_member_cb(Dwarf_Die *die, void *arg)
 	if (dwarf_tag(die) != DW_TAG_member)
 		return DIE_FIND_CB_SIBLING;
 
+	if (__die_get_real_type(die, &member_type) == NULL)
+		return DIE_FIND_CB_SIBLING;
+
+	if (dwarf_tag(&member_type) == DW_TAG_typedef) {
+		if (die_get_real_type(&member_type, &die_mem) == NULL)
+			return DIE_FIND_CB_SIBLING;
+	} else {
+		die_mem = member_type;
+	}
+
 	member = zalloc(sizeof(*member));
 	if (member == NULL)
 		return DIE_FIND_CB_END;
@@ -242,12 +261,6 @@ static int __add_member_cb(Dwarf_Die *die, void *arg)
 	strbuf_init(&sb, 32);

tools/perf/util/annotate-data.c static int __add_member_cb(Dwarf_Die *die, void *arg) · line 265

Quoted 10 lines
tools/perf/util/annotate-data.c +0 -6
 	die_get_typename(die, &sb);
 
-	__die_get_real_type(die, &member_type);
-	if (dwarf_tag(&member_type) == DW_TAG_typedef)
-		die_get_real_type(&member_type, &die_mem);
-	else
-		die_mem = member_type;
-
 	if (dwarf_aggregate_size(&die_mem, &size) < 0)
 		size = 0;

This is gonna conflict with my change. :)

Quoted 12 lines
tools/perf/util/annotate-data.c +3 -0
@@ -289,10 +302,23 @@ static int __add_member_cb(Dwarf_Die *die, void *arg)
 	}
 	member->size = size;
 	member->offset = loc + parent->offset;
+	member->depth = parent->depth + 1;
 	INIT_LIST_HEAD(&member->children);
 	list_add_tail(&member->node, &parent->children);
 
 	tag = dwarf_tag(&die_mem);
+	if (member->depth >= MAX_MEMBER_DEPTH) {
+		/*

tools/perf/util/annotate-data.c static int __add_member_cb(Dwarf_Die *die, void *arg) · line 321

Quoted 10 lines
tools/perf/util/annotate-data.c +10 -0
+		 * The JSON exporter added in a later series reports
+		 * this to its consumers, so that they can tell a
+		 * truncated tree from one that really ends here.
+		 */
+		member->truncated = true;
+		pr_debug_dtp("member nesting limit reached at %s\n",
+			     member->type_name ?: "(unknown type)");
+		return DIE_FIND_CB_SIBLING;
+	}
+

I think it's worth to be a separate change.

tools/perf/util/dwarf-aux.c

Quoted 95 lines
tools/perf/util/annotate-data.c +15 -6
 	switch (tag) {
 	case DW_TAG_structure_type:
 	case DW_TAG_union_type:
@@ -645,6 +671,8 @@ struct global_var_entry {
 	u64 start;
 	u64 end;
 	u64 die_offset;
+	int die_tag;
+	bool from_alt;	/* die_offset is relative to the alt (dwz) file */
 };
 
 static int global_var_cmp(const void *_key, const struct rb_node *node)
@@ -682,7 +710,7 @@ static struct global_var_entry *global_var__find(struct data_loc_info *dloc, u64
 }
 
 static bool global_var__add(struct data_loc_info *dloc, u64 addr,
-			    const char *name, Dwarf_Die *type_die)
+			    const char *name, Dwarf_Die *type_die, bool from_alt)
 {
 	struct dso *dso = map__dso(dloc->ms->map);
 	struct global_var_entry *gvar;
@@ -704,6 +732,8 @@ static bool global_var__add(struct data_loc_info *dloc, u64 addr,
 	gvar->start = addr;
 	gvar->end = addr + size;
 	gvar->die_offset = dwarf_dieoffset(type_die);
+	gvar->die_tag = dwarf_tag(type_die);
+	gvar->from_alt = from_alt;
 
 	rb_add(&gvar->node, dso__global_vars(dso), global_var_less);
 	return true;
@@ -778,12 +808,14 @@ static void global_var__collect(struct data_loc_info *dloc)
 			if (pos->reg != -1)
 				continue;
 
-			if (!dwarf_offdie(dwarf, pos->die_off, &type_die))
+			if (!die_get_type_die(dwarf, pos->die_off, pos->die_tag,
+					      pos->from_alt, &type_die))
 				continue;
 
 			get_global_var_info(dloc, pos->addr, &var_name, &var_offset);
 
-			global_var__add(dloc, pos->addr, var_name, &type_die);
+			global_var__add(dloc, pos->addr, var_name, &type_die,
+					pos->from_alt);
 		}
 
 		delete_var_types(var_types);
@@ -808,7 +840,8 @@ bool get_global_var_type(Dwarf_Die *cu_die, struct data_loc_info *dloc,
 
 	gvar = global_var__find(dloc, var_addr);
 	if (gvar) {
-		if (!dwarf_offdie(dloc->di->dbg, gvar->die_offset, type_die))
+		if (!die_get_type_die(dloc->di->dbg, gvar->die_offset,
+				      gvar->die_tag, gvar->from_alt, type_die))
 			return false;
 
 		*var_offset = var_addr - gvar->start;
@@ -838,7 +871,8 @@ bool get_global_var_type(Dwarf_Die *cu_die, struct data_loc_info *dloc,
 
 ok:
 	/* The address should point to the start of the variable */
-	global_var__add(dloc, var_addr - *var_offset, var_name, type_die);
+	global_var__add(dloc, var_addr - *var_offset, var_name, type_die,
+			!die_same_file(cu_die, type_die));
 	return true;
 }
 
@@ -893,7 +927,8 @@ static void update_var_state(struct type_state *state, struct data_loc_info *dlo
 				continue;
 		}
 		/* Get the type DIE using the offset */
-		if (!dwarf_offdie(dloc->di->dbg, var->die_off, &mem_die))
+		if (!die_get_type_die(dloc->di->dbg, var->die_off,
+				      var->die_tag, var->from_alt, &mem_die))
 			continue;
 
 		if (var->reg == DWARF_REG_FB || var->reg == fbreg || var->reg == state->stack_reg) {
tools/perf/util/annotate-data.h +3 -0
index ca2096a9ee62cbfe..cc576232f55b5fb0 100644
@@ -57,6 +57,9 @@ struct annotated_member {
 	char *var_name;
 	int offset;
 	int size;
+	unsigned int depth;
+	/* Children not expanded because the nesting limit was reached */
+	bool truncated;
 };
 
 /**
tools/perf/util/dwarf-aux.c +0 -0
index d7160f87ac7d7ab3..7acb431fd34a8ecb 100644

I tend to split changes in the dwarf-aux.[ch] as they are primitive helpers and need to work in generic way.

Thanks,
Namhyung

Quoted 334 lines
tools/perf/util/dwarf-aux.c +149 -22
@@ -266,16 +266,35 @@ Dwarf_Die *die_get_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem)
 		return NULL;
 }
 
+/*
+ * The chases below cross typedefs and qualifiers to get to the type that
+ * is actually meant, and a DIE that is not what it looks like, e.g. one
+ * parsed at an offset that is not the start of a DIE in the file it was
+ * resolved in, can have a DW_AT_type that refers back to itself, which
+ * makes them spin forever: 'perf report -s type' did exactly that on the
+ * dwz compressed debug info of zlib-ng (libz.so.1), burning all of a CPU
+ * with no output while resolving a hist entry in build_tree().
+ *
+ * No sane chain is this long, so give up instead of hanging, telling about
+ * it so that the broken debug info can be looked at.
+ */
+#define MAX_TYPE_CHASE 32
+
 /* Get a type die, but skip qualifiers */
 Dwarf_Die *__die_get_real_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem)
 {
-	int tag;
+	int tag, chase = 0;
 
 	do {
 		vr_die = die_get_type(vr_die, die_mem);
 		if (!vr_die)
-			break;
+			return NULL;
 		tag = dwarf_tag(vr_die);
+		if (++chase > MAX_TYPE_CHASE) {
+			pr_debug("DWARF: qualifier chase limit reached at DIE 0x%lx\n",
+				 (unsigned long)dwarf_dieoffset(vr_die));
+			return NULL;
+		}
 	} while (tag == DW_TAG_const_type ||
 		 tag == DW_TAG_restrict_type ||
 		 tag == DW_TAG_volatile_type ||
@@ -296,8 +315,15 @@ Dwarf_Die *__die_get_real_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem)
  */
 Dwarf_Die *die_get_real_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem)
 {
+	int chase = 0;
+
 	do {
 		vr_die = __die_get_real_type(vr_die, die_mem);
+		if (++chase > MAX_TYPE_CHASE) {
+			pr_debug("DWARF: typedef chase limit reached at DIE 0x%lx\n",
+				 vr_die ? (unsigned long)dwarf_dieoffset(vr_die) : 0);
+			return NULL;
+		}
 	} while (vr_die && dwarf_tag(vr_die) == DW_TAG_typedef);
 
 	return vr_die;
@@ -314,7 +340,7 @@ Dwarf_Die *die_get_real_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem)
  */
 Dwarf_Die *die_get_pointer_type(Dwarf_Die *type_die, Dwarf_Die *die_mem)
 {
-	int tag;
+	int tag, chase = 0;
 
 	do {
 		tag = dwarf_tag(type_die);
@@ -324,6 +350,11 @@ Dwarf_Die *die_get_pointer_type(Dwarf_Die *type_die, Dwarf_Die *die_mem)
 		    tag != DW_TAG_restrict_type && tag != DW_TAG_volatile_type &&
 		    tag != DW_TAG_shared_type)
 			return NULL;
+		if (++chase > MAX_TYPE_CHASE) {
+			pr_debug("DWARF: pointer type chase limit reached at DIE 0x%lx\n",
+				 (unsigned long)dwarf_dieoffset(type_die));
+			return NULL;
+		}
 		type_die = die_get_type(type_die, die_mem);
 	} while (type_die);
 
@@ -1118,17 +1149,27 @@ Dwarf_Die *die_find_member(Dwarf_Die *st_die, const char *name,
 			      die_mem);
 }
 
-/**
- * die_get_typename_from_type - Get the name of given type DIE
- * @type_die: a type DIE
- * @buf: a strbuf for result type name
- *
- * Get the name of @type_die and stores it to @buf. Return 0 if succeeded.
- * and Return -ENOENT if failed to find type name.
- * Note that the result will stores typedef name if possible, and stores
- * "*(function_type)" if the type is a function pointer.
+/*
+ * The name of a pointer or array type is built from the name of the type
+ * it points to or holds, so the recursion below follows DW_AT_type; a
+ * garbage DIE whose DW_AT_type refers back to itself makes it recurse
+ * forever, just like the chases above, so it gets the same bound.
  */
-int die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf)
+static int __die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf,
+					int depth);
+
+static int __die_get_typename(Dwarf_Die *vr_die, struct strbuf *buf, int depth)
+{
+	Dwarf_Die type;
+
+	if (__die_get_real_type(vr_die, &type) == NULL)
+		return -ENOENT;
+
+	return __die_get_typename_from_type(&type, buf, depth);
+}
+
+static int __die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf,
+					int depth)
 {
 	int tag, ret;
 	const char *tmp = "";
@@ -1155,7 +1196,12 @@ int die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf)
 		/* Write a base name */
 		return strbuf_addf(buf, "%s%s", tmp, name ?: "");
 	}
-	ret = die_get_typename(type_die, buf);
+	if (depth >= MAX_TYPE_CHASE) {
+		pr_debug("DWARF: type name recursion limit reached at DIE 0x%lx\n",
+			 (unsigned long)dwarf_dieoffset(type_die));
+		return -ENOENT;
+	}
+	ret = __die_get_typename(type_die, buf, depth + 1);
 	if (ret < 0) {
 		/* void pointer has no type attribute */
 		if (tag == DW_TAG_pointer_type && ret == -ENOENT)
@@ -1166,6 +1212,21 @@ int die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf)
 	return strbuf_addstr(buf, tmp);
 }
 
+/**
+ * die_get_typename_from_type - Get the name of given type DIE
+ * @type_die: a type DIE
+ * @buf: a strbuf for result type name
+ *
+ * Get the name of @type_die and stores it to @buf. Return 0 if succeeded.
+ * and Return -ENOENT if failed to find type name.
+ * Note that the result will stores typedef name if possible, and stores
+ * "*(function_type)" if the type is a function pointer.
+ */
+int die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf)
+{
+	return __die_get_typename_from_type(type_die, buf, 0);
+}
+
 /**
  * die_get_typename - Get the name of given variable DIE
  * @vr_die: a variable DIE
@@ -1178,12 +1239,7 @@ int die_get_typename_from_type(Dwarf_Die *type_die, struct strbuf *buf)
  */
 int die_get_typename(Dwarf_Die *vr_die, struct strbuf *buf)
 {
-	Dwarf_Die type;
-
-	if (__die_get_real_type(vr_die, &type) == NULL)
-		return -ENOENT;
-
-	return die_get_typename_from_type(&type, buf);
+	return __die_get_typename(vr_die, buf, 0);
 }
 
 /**
@@ -1632,6 +1688,22 @@ Dwarf_Die *die_find_variable_by_addr(Dwarf_Die *sc_die, Dwarf_Addr addr,
 	return result;
 }
 
+/*
+ * Whether two DIEs live in the same debug file.
+ *
+ * dwarf_dieoffset() is relative to the file the DIE is in, so this is what
+ * tells an offset that has to be resolved in the dwz alt file, where dwz
+ * moved the type, from one that belongs to the main file: dwz encodes the
+ * references into its common file as DW_FORM_GNU_ref_alt, elfutils resolves
+ * them into the alt Dwarf and the CU of the resulting DIE belongs to that
+ * other file, so comparing the Dwarf each CU belongs to settles it exactly,
+ * rather than inferring it from what happens to be at the offset.
+ */
+bool die_same_file(Dwarf_Die *die_a, Dwarf_Die *die_b)
+{
+	return dwarf_cu_getdwarf(die_a->cu) == dwarf_cu_getdwarf(die_b->cu);
+}
+
 static int __die_collect_vars_cb(Dwarf_Die *die_mem, void *arg)
 {
 	struct die_var_type **var_types = arg;
@@ -1676,6 +1748,8 @@ static int __die_collect_vars_cb(Dwarf_Die *die_mem, void *arg)
 			vt->is_reg_var_addr = true;
 
 		vt->die_off = dwarf_dieoffset(&type_die);
+		vt->die_tag = dwarf_tag(&type_die);
+		vt->from_alt = !die_same_file(die_mem, &type_die);
 		vt->addr = start;
 		vt->end = end;
 		vt->has_range = (end != 0 || start != 0);
@@ -1695,7 +1769,8 @@ static int __die_collect_vars_cb(Dwarf_Die *die_mem, void *arg)
  *
  * Save all variables and parameters in the @sc_die and save them to @var_types.
  * The @var_types is a singly-linked list containing type and location info.
- * Actual type can be retrieved using dwarf_offdie() with 'die_off' later.
+ * Actual type can be retrieved using die_get_type_die() with 'die_off',
+ * 'die_tag' and 'from_alt' later.
  *
  * Callers should free @var_types.
  */
@@ -1741,6 +1816,8 @@ static int __die_collect_global_vars_cb(Dwarf_Die *die_mem, void *arg)
 		return DIE_FIND_CB_END;
 
 	vt->die_off = dwarf_dieoffset(&type_die);
+	vt->die_tag = dwarf_tag(&type_die);
+	vt->from_alt = !die_same_file(die_mem, &type_die);
 	vt->addr = ops->number;
 	vt->end = 0;
 	vt->has_range = false;
@@ -1752,6 +1829,55 @@ static int __die_collect_global_vars_cb(Dwarf_Die *die_mem, void *arg)
 	return DIE_FIND_CB_SIBLING;
 }
 
+/**
+ * die_get_type_die - Get a type DIE saved by die_collect_vars()
+ * @dbg: the main debug info
+ * @die_off: offset of the type DIE, from dwarf_dieoffset()
+ * @die_tag: tag that DIE had when the offset was saved
+ * @from_alt: whether the type DIE is in the dwz alt file
+ * @die_mem: where to store the resulting DIE
+ *
+ * See the comment in util/dwarf-aux.h: the offset is only meaningful in the
+ * file the DIE was in, which can be the dwz common file, so resolve it in
+ * the file @from_alt says it was in, the main file or its alt file, and use
+ * the DIE only when it has the @die_tag it had when the offset was saved.
+ * There is deliberately no fallback to the other file: resolving an alt
+ * file offset in the main file does not fail, it parses whatever is there
+ * as a DIE, and that is what hung 'perf report -s type'.
+ */
+Dwarf_Die *die_get_type_die(Dwarf *dbg, u64 die_off, int die_tag, bool from_alt,
+			    Dwarf_Die *die_mem)
+{
+	Dwarf *target = dbg;
+	Dwarf_Die die;
+
+	if (from_alt) {
+		/*
+		 * Deliberately no fallback to the main file when there is
+		 * no alt file, or when the offset does not resolve in it:
+		 * resolving an alt file offset in the main file does not
+		 * fail, it parses whatever is there as a DIE, and that is
+		 * what hung 'perf report -s type', see the comment in
+		 * util/dwarf-aux.h.
+		 */
+		target = dwarf_getalt(dbg);
+		if (target == NULL) {
+			pr_debug("DWARF: no alt (dwz) debug file to resolve the type DIE at offset 0x%lx in\n",
+				 (unsigned long)die_off);
+			return NULL;
+		}
+	}
+
+	if (dwarf_offdie(target, die_off, &die) && dwarf_tag(&die) == die_tag) {
+		*die_mem = die;
+		return die_mem;
+	}
+
+	pr_debug("DWARF: no DIE with tag %d at offset 0x%lx in the %s debug file\n",
+		 die_tag, (unsigned long)die_off, from_alt ? "alt" : "main");
+	return NULL;
+}
+
 /**
  * die_collect_global_vars - Save all global variables
  * @cu_die: a CU DIE
@@ -1759,7 +1885,8 @@ static int __die_collect_global_vars_cb(Dwarf_Die *die_mem, void *arg)
  *
  * Save all global variables in the @cu_die and save them to @var_types.
  * The @var_types is a singly-linked list containing type and location info.
- * Actual type can be retrieved using dwarf_offdie() with 'die_off' later.
+ * Actual type can be retrieved using die_get_type_die() with 'die_off',
+ * 'die_tag' and 'from_alt' later.
  *
  * Callers should free @var_types.
  */
tools/perf/util/dwarf-aux.h +33 -0
index 161f0bf980b6ee6a..149e0cbf63ccc82a 100644
@@ -152,6 +152,8 @@ int die_get_scopes(Dwarf_Die *cu_die, Dwarf_Addr pc, Dwarf_Die **scopes);
 struct die_var_type {
 	struct die_var_type *next;
 	u64 die_off;
+	int die_tag;
+	bool from_alt;	/* die_off is relative to the alt (dwz) file */
 	u64 addr;
 	u64 end;        /* end address of location range */
 	int reg;
@@ -183,6 +185,37 @@ Dwarf_Die *die_find_variable_by_addr(Dwarf_Die *sc_die, Dwarf_Addr addr,
 /* Save all variables and parameters in this scope */
 void die_collect_vars(Dwarf_Die *sc_die, struct die_var_type **var_types);
 
+/*
+ * Get the type DIE saved by die_collect_vars()/die_collect_global_vars().
+ *
+ * The offsets those save are the dwarf_dieoffset() of the type DIE, which is
+ * relative to the debug file that DIE lives in: the dwz common file, the alt
+ * file in libdw terms, for the types shared by more than one CU, the main
+ * file for the rest.  Resolving an alt file offset in the main file does not
+ * fail: dwarf_offdie() parses whatever is at that offset there, and an offset
+ * that is a CU header in the main file reads back as a typedef whose
+ * DW_AT_type refers to itself, which is what hung 'perf report -s type' on
+ * the dwz compressed debug info of zlib-ng (libz.so.1).
+ *
+ * So @from_alt, recorded when the offset was saved, says which of the two
+ * files to resolve it in.  It is not inferred from the DIE contents: dwz
+ * encodes references into the common file as DW_FORM_GNU_ref_alt, so
+ * elfutils resolves them into the alt Dwarf and the CU of the type DIE then
+ * belongs to that other file, which die_same_file() compares exactly.
+ *
+ * There is deliberately no fallback to the other file when the offset does
+ * not resolve: that fallback is the misparse above.
+ *
+ * @die_tag is then only a sanity check: the offset is of a DIE that had this
+ * tag when it was saved, so a mismatch means the debug info changed under us,
+ * or is broken, and giving up on the type is the right answer.
+ */
+Dwarf_Die *die_get_type_die(Dwarf *dbg, u64 die_off, int die_tag, bool from_alt,
+			    Dwarf_Die *die_mem);
+
+/* Whether two DIEs live in the same debug file */
+bool die_same_file(Dwarf_Die *die_a, Dwarf_Die *die_b);
+
 /* Save all global variables in this CU */
 void die_collect_global_vars(Dwarf_Die *cu_die, struct die_var_type **var_types);
 
Signature · 1 line
2.55.0
maintainer
#33 lore raw

Re: [PATCH 11/12] perf mem record: Request PERF_SAMPLE_CPU by default

In reply to Arnaldo Carvalho de Melo
Quoted 33 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type profiling per-sample stream keys cross-CPU contention on sample->cpu: without PERF_SAMPLE_CPU the cpu field is the (u32)-1 "no CPU info" sentinel, documented as such in perf_session__deliver_event(), and same-instance reads and writes from different cores are indistinguishable from same-CPU traffic, so pahole's false-sharing detector cannot tell them apart.

builtin-record.c already defines --sample-cpu and 'perf mem record' forwards unknown options to the record parser, so passing it explicitly works today; make it the default, next to the -d (addr) and -W (weight) the command already requests, documenting it in perf-mem(1).

The rec_argv array is sized as nine arguments per memory PMU plus the user arguments, not counting the arguments __cmd_record() adds itself, up to eight with all the optional flags. On PMUs with separate load and store events the four event arguments plus the fixed ones already filled the array to its last slot, so this new argument would write past it; reserve space for the fixed arguments explicitly.

---
 tools/perf/Documentation/perf-mem.txt |  4 ++++
 tools/perf/builtin-mem.c              | 18 ++++++++++++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-mem.txt +0 -0
index 4d164836d0943119..fe51c5e3333dc4a0 100644

tools/perf/Documentation/perf-mem.txt DESCRIPTION · line 20

Quoted 7 lines
tools/perf/Documentation/perf-mem.txt +4 -0
@@ -14,6 +14,10 @@ DESCRIPTION
 -----------
 "perf mem record" runs a command and gathers memory operation data
 from it, into perf.data. Perf record options are accepted and are passed through.
+It also requests the address (-d), the weight (-W, where supported) and the
+CPU id (--sample-cpu) of every sampled access by default; the CPU id is what
+lets per-sample analysis tell reads and writes to the same data from
+different cores apart from same-CPU traffic.

I guess the data source of samples can tell that on supported platforms. Probably we can check TID for concurrent access, wdyt?

Thanks,
Namhyung

Quoted 42 lines
tools/perf/Documentation/perf-mem.txt +0 -0
 "perf mem report" displays the result. It invokes perf report with the
 right set of options to display a memory access profile. By default, loads
tools/perf/builtin-mem.c +16 -2
index 6101a26b3a781e69..6f38cda1a45ada16 100644
@@ -99,8 +99,13 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem,
 	argc = parse_options(argc, argv, options, record_usage,
 			     PARSE_OPT_KEEP_UNKNOWN);
 
-	/* Max number of arguments multiplied by number of PMUs that can support them. */
-	rec_argc = argc + 9 * (perf_pmu__mem_events_num_mem_pmus(pmu) + 1);
+	/*
+	 * Max number of arguments multiplied by number of PMUs that can
+	 * support them, plus the arguments added directly below, at most:
+	 * "record", "-W", "-d", "--sample-cpu", "--phys-data",
+	 * "--data-page-size", "--all-user" and "--all-kernel".
+	 */
+	rec_argc = argc + 8 + 9 * (perf_pmu__mem_events_num_mem_pmus(pmu) + 1);
 
 	if (mem->cpu_list)
 		rec_argc += 2;
@@ -135,6 +140,15 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem,
 
 	rec_argv[i++] = "-d";
 
+	/*
+	 * The data-type profiling per-sample stream keys cross-CPU
+	 * contention on sample->cpu (PERF_SAMPLE_CPU); without it the cpu
+	 * field is the (u32)-1 'no CPU info' sentinel and same-instance
+	 * reads and writes from different cores are indistinguishable
+	 * from same-CPU traffic.
+	 */
+	rec_argv[i++] = "--sample-cpu";
+
 	if (mem->phys_addr)
 		rec_argv[i++] = "--phys-data";
 
Signature · 1 line
2.55.0
maintainer
#36 lore raw

Re: [PATCH 12/12] perf mem record: Use the IBS swfilt filter when available

In reply to Arnaldo Carvalho de Melo

+Ravi

Quoted 136 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

IBS events with exclude_{user,kernel} bits, as used in per-thread mode when kernel samples are not allowed, are rejected by the kernel on hardware without the privilege filter, so a per-thread 'perf mem record' fails:

  $ perf mem record -o /dev/null -- true
  Error:
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

Kernel v6.14 added swfilt, a software privilege filter that makes those events usable per-thread, exposed as the 'swfilt' format term (d29e744c71673a71 "perf/x86: Relax privilege filter restriction on AMD IBS"):

  $ perf record -e ibs_op/ldlat=0,swfilt=1/ -- true

Give the ibs_op memory events a name variant with the swfilt term, used when the PMU exposes it as a format term, keeping the names that need system wide mode otherwise.

The variant is used even for records that end up without exclude bits, e.g. a plain system wide one: perf record adds those bits itself when the first open fails with EACCES on an unprivileged setup, after the event name has been built, so the term has to be in it for that retry:

  $ perf record -v -e ibs_op/ldlat=0/ -- true
  kernel.perf_event_paranoid=2, trying to fall back to excluding kernel and hypervisor  samples
  Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed.
  Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

With no exclude bits the kernel discards nothing.

This makes the per-thread 'perf mem record' that the data type profiling shell test does work: the skip added by patch 1 is driven by that record failing, so it now runs on AMD kernels with swfilt, and keeps skipping on kernels without it. The 'Test data symbol' shell test matches the exact ibs_op event string, so its regex now accepts terms added after ldlat=150, and it only checks ldlat when the captured term has it: on uarch without ldlat the swfilt variant makes the capture non-empty ("swfilt=1"), that the old 'ldlat=0' check would reject.

---
 tools/perf/arch/x86/util/mem-events.c      | 17 ++++++++++++++---
 tools/perf/tests/shell/test_data_symbol.sh |  6 ++++--
 tools/perf/util/mem-events.c               | 21 +++++++++++++++++----
 tools/perf/util/mem-events.h               |  2 ++
 4 files changed, 37 insertions(+), 9 deletions(-)
tools/perf/arch/x86/util/mem-events.c +14 -3
index b38f519020ff8c6f..034053dc762101dc 100644
@@ -7,7 +7,10 @@
 
 #define MEM_LOADS_AUX		0x8203
 
-#define E(t, n, s, l, a) { .tag = t, .name = n, .event_name = s, .ldlat = l, .aux_event = a }
+#define E_INIT(t, n, s, l, a, sf) {					\
+	.tag = t, .name = n, .event_name = s, .swfilt_name = sf,	\
+	.ldlat = l, .aux_event = a }
+#define E(t, n, s, l, a) E_INIT(t, n, s, l, a, NULL)
 
 struct perf_mem_event perf_mem_events_intel[PERF_MEM_EVENTS__MAX] = {
 	E("ldlat-loads",	"%s/mem-loads,ldlat=%u/P",	"mem-loads",	true,	0),
@@ -21,14 +24,22 @@ struct perf_mem_event perf_mem_events_intel_aux[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,			NULL,				NULL,		false,	0),
 };
 
+/*
+ * IBS events with exclude_{user,kernel} bits set, as used by perf to
+ * record per-thread when kernel samples are not allowed, are rejected
+ * by the kernel on hardware without the privilege filter unless the
+ * swfilt software filter is used, so these events carry a variant of
+ * their names with the swfilt term, used by perf_pmu__mem_events_name()
+ * when the kernel exposes the term.
+ */
 struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
-	E("mem-ldst",	"%s//",		NULL,	false,	0),
+	E_INIT("mem-ldst",	"%s//",		NULL,	false,	0, "%s/swfilt=1/"),
 };
 
 struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
-	E("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0),
+	E_INIT("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0, "%s/ldlat=%u,swfilt=1/"),
 };
tools/perf/tests/shell/test_data_symbol.sh +4 -2
index d61b5659a46d9a77..52c837fddb639595 100755
@@ -65,15 +65,17 @@ if (($is_amd >= 1)); then
 	# --ldlat on AMD:
 	# o Zen4 and earlier uarch does not support ldlat
 	# o Even on supported platforms, it's disabled (--ldlat=0) by default.
+	# o Kernels with the swfilt term add it even when ldlat is not
+	#   supported, so only check ldlat when the term is present.
 	ldlat=${BASH_REMATCH[1]}
-	if [[ -n $ldlat ]]; then
+	if [[ $ldlat == *ldlat=* ]]; then
 		if ! [[ "$ldlat" =~ ldlat=0 ]]; then
 			echo "ERROR: ldlat not initialized to 0?"
 			exit 1
 		fi
 
 		mem_events="$(perf mem record -v --ldlat=150 -e list 2>&1)"
-		if ! [[ "$mem_events" =~ ^mem-ldst.*ibs_op/ldlat=150/.*available ]]; then
+		if ! [[ "$mem_events" =~ ^mem-ldst.*ibs_op/ldlat=150[,/].*available ]]; then
 			echo "ERROR: --ldlat not honored?"
 			exit 1
 		fi
tools/perf/util/mem-events.c +2 -0
index 0b49fce251fcc184..8f74cd085e500231 100644
@@ -82,6 +82,7 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 					     char *buf, size_t buf_size)
 {
 	struct perf_mem_event *e;
+	const char *name;
 
 	if (i >= PERF_MEM_EVENTS__MAX || !pmu)
 		return NULL;
@@ -90,24 +91,36 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 	if (!e || !e->name)
 		return NULL;
 
+	/*

tools/perf/util/mem-events.c static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i, · line 104

Quoted 10 lines
tools/perf/util/mem-events.c +10 -0
+	 * Use the swfilt variant of the name when the PMU exposes the term.
+	 * It is not conditional on the event already having exclude bits:
+	 * perf record adds those bits itself when the first open fails with
+	 * EACCES on an unprivileged setup, after this name has been built,
+	 * and that retry only succeeds with the term in the name.  With no
+	 * exclude bits the kernel doesn't discard anything.
+	 */
+	name = e->name;
+	if (e->swfilt_name && perf_pmu__has_format(pmu, "swfilt"))
+		name = e->swfilt_name;

It's a bit unfortunate we hardcode "swfilt" here. Probably better to have it in the mem-events array and add it to the name format dynamically. But as it's only needed for IBS, I think it's ok for now.

Thanks,
Namhyung

Quoted 50 lines
tools/perf/util/mem-events.c +5 -4
+
 	if (i == PERF_MEM_EVENTS__LOAD || i == PERF_MEM_EVENTS__LOAD_STORE) {
 		if (e->ldlat) {
 			if (!e->aux_event) {
 				/* ARM and Most of Intel */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name,
+					  name, pmu->name,
 					  perf_mem_events__loads_ldlat);
 			} else {
 				/* Intel with mem-loads-aux event */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name, pmu->name,
+					  name, pmu->name, pmu->name,
 					  perf_mem_events__loads_ldlat);
 			}
 		} else {
 			if (!e->aux_event) {
 				/* AMD and POWER */
 				scnprintf(buf, buf_size,
-					  e->name, pmu->name);
+					  name, pmu->name);
 			} else {
 				return NULL;
 			}
@@ -117,7 +130,7 @@ static const char *perf_pmu__mem_events_name(struct perf_pmu *pmu, int i,
 
 	if (i == PERF_MEM_EVENTS__STORE) {
 		scnprintf(buf, buf_size,
-			  e->name, pmu->name);
+			  name, pmu->name);
 		return buf;
 	}
 
tools/perf/util/mem-events.h +2 -0
index 5b98076904b0b689..41f628fad10709b9 100644
@@ -11,6 +11,8 @@ struct perf_mem_event {
 	u32		aux_event;
 	const char	*tag;
 	const char	*name;
+	/* Name with the swfilt software privilege filter, when supported. */
+	const char	*swfilt_name;
 	const char	*event_name;
 };
 
Signature · 1 line
2.55.0
maintainer
#40 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 450 lines · Arnaldo Carvalho de Melo

Hi,

This came out of work on data type profiling, but none of them depends on that work and nothing in this series needs anything from it, so sending them separately.

The fixes:

- 'perf report -s type' spins forever, burning all of a CPU with no output, on the dwz compressed debug info of zlib-ng (libz.so.1):
die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, the dwz common file for the types shared by more than one CU, and resolving one of those offsets in the main debug file does not fail, it parses whatever is at that offset there, in this case a typedef whose DW_AT_type refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin. The offset is now resolved in the file it was recorded as coming from, which the CU of the collected DIE tells exactly rather than having to be inferred from what is at the offset, and the type chases and the struct/union member nesting are bounded, so a debug info file broken in some other way makes perf give up on a type with a pr_debug instead of looking like it hung (patch 10);

- the data type browser prints, in its samples view (-n, annotate.show_nr_samples), a local variable initialized to zero and never updated, so every member shows up as having no samples while the period and percent columns for the same entry are filled in (patch 7);

- the 'perf data type profiling' shell test turns "this PMU cannot record these events" into a test failure: the script runs under 'set -e', so the bare 'perf mem record' aborts it through the EXIT trap, which reports a signal that never happened, instead of reaching the code right below meant to report the failure (patch 1).

The features:

- 'perf report --progress' prints the phase it is in and how far along it is to stderr: the TUI shows that with a progress bar, but in the stdio case the ui_progress updates that are already there are dropped on the floor. It is what makes a slow run tell itself apart from a stuck one (patch 8);

- the debuginfo for a DSO, and the kernel symbols, can now be fetched keyed by the build ID recorded in the perf.data file, using the debuginfod client, for the cases where they are not available locally under the name the DSO was opened with: a vmlinux for a kernel that since got upgraded, or a profile recorded on another machine. It is off with --no-debuginfod, with core.debuginfod=false, and when the build-id cache is turned off (patch 2), opt-outs that also clear DEBUGINFOD_URLS, that libdwfl's own debuginfod client reads, so that it doesn't fetch behind perf's back;

- a fetch that ends up taking a while, e.g. that vmlinux, or one for the vmlinux of a kernel profiled on another machine, doesn't leave the user staring at a stuck looking terminal: in the stdio interface, the progress is a line on stderr, in the TUI a window over the browser, and 's' aborts the current fetch while remembering the build ID, so that the rest of the session doesn't ask for it again, the user may have skipped it for being too big; 'd' additionally disables debuginfod for the rest of the session, clearing DEBUGINFOD_URLS so that libdwfl's own client stops too, and writes core.debuginfod=false into the configuration file perf is using, ~/.perfconfig or the one named by PERF_CONFIG, making it permanent. This is spread over three patches, with the move of perf_config__set_variable() to util/config.c as a prep patch, the stdio keys and the TUI window (patches 3 to 5);

- 'perf mem record' asks for PERF_SAMPLE_CPU by default, as without it the cpu field is the (u32)-1 "no CPU info" sentinel and per-sample analysis cannot tell accesses from different cores apart from same-CPU traffic (patch 11), and uses the IBS swfilt software privilege filter, when the kernel exposes it, so that per-thread recording works where only system wide recording did (patch 12).

Which file a saved type DIE offset belongs to is settled with dwarf_cu_getdwarf(), new in elfutils 0.160, so the libdw feature test now probes for it and the message in Makefile.config says 0.160 where it said 0.157: 0.157 to 0.159 is from 2014 and has no such symbol, and those versions now lose dwarf support with that message instead of failing to link util/dwarf-aux.c.

Best regards,

- Arnaldo

What changed from v5 (ef0ac8e44d43d2aa86842d2f83ad41d1c058d949):

PATCH 2/9, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod", was split [Namhyung Kim review of PATCH v3 2/8, <[email protected]>]: the fetch stays in patch 2, the way to get out of a slow fetch, that was folded at the end of patch 2 in v5, is now patches 3 to 5, each separately cherry-pickable, and carrying on their own the answer to his "I'm not sure what would be the good default. But with this, it can slow down the process especially when the binary is not in the debuginfod": on by default, with the way to skip the one in progress and to disable the feature right where the wait is felt. Patch 3, "perf config: Move perf_config__set_variable() to util/config.c", is the prep patch, patch 4, "perf debuginfo: Let the user skip and disable debuginfod fetches", has the stdio 's'/'d' keys, and patch 5, "perf debuginfo:
Show the debuginfod fetch progress and keys in the TUI", the TUI window over the browser.

PATCH 4/12, "perf debuginfo: Let the user skip and disable debuginfod fetches" [that same review of the 'd' key semantics]:

- 'd' now writes core.debuginfod=false into the configuration file directly, the user's ~/.perfconfig or the one named by PERF_CONFIG, via perf_config__set_variable(), moved from builtin-config.c to util/config.c by the prep patch so that util/debuginfo.c can use it, and clears DEBUGINFOD_URLS so that libdwfl's own debuginfod client stops for the rest of the session too: in v5 it only disabled debuginfod for the rest of the session and pointed at 'perf config' to make that permanent.

PATCH 5/12, "perf debuginfo: Show the debuginfod fetch progress and keys in the TUI":

- The progress and the keys are no longer stdio-only: printing to stderr would garble the TUI browser display, so in v5 a fetch in progress in the TUI showed nothing and there was no way to skip or disable it, the thing that makes a slow fetch indistinguishable from a stuck perf. The TUI now draws a window over the browser with the fetch progress and drains the 's'/'d' keys from the TUI input queue: the TUI owns the terminal and its input queue, and the browser thread is the one doing the fetch, so nobody else is reading from it while the fetch blocks it. No terminal settings or signal dispositions are touched in that case, the terminal is the TUI's own and so are the SIGINT/SIGTERM handlers, that restore it before exiting.

- The slang calls stay in ui/tui/, behind the ui__key_pending()/ ui__key_read()/ui__progress_window()/ui__progress_window_end() primitives declared in ui/util.h and implemented in ui/tui/util.c, with inline stubs for builds without slang: the plan is to reduce the number of libraries perf needs to build, or swap slang for something else, such as ncurses, at some point, so nothing outside ui/ calls slang directly. The text in the window is word wrapped to the available columns, as constrained terminals, such as a smartphone running termux, crop the tail of a long line and would hide the 'd' option hint.

Patches 1, 3, 4, 5, 6, 7, 8 and 9 are unchanged from v5, renumbered as 1, 6, 7, 8, 9, 10, 11 and 12 with the split.

What changed from v4 (f801de4e80289566):

PATCH 1/8, "perf test: Skip data_type_profiling when the PMU cannot record memory events" [Namhyung Kim review of PATCH v3 1/8]:

- Namhyung suggested adding a fallback to the 'swfilt' term for ibs_op instead of only skipping the test on a PMU that refuses per-thread memory events. The new PATCH 9 does that for the ibs_op memory events, using the swfilt software privilege filter the kernel exposes as a format term since v6.14, which makes the per-thread 'perf mem record' the test does work on AMD; the skip stays for kernels without it. The term is in the event name even for records that end up without exclude bits, e.g. plain system wide ones: 'perf record' adds those bits itself when the first open fails with EACCES, after the name was built, and the retry only succeeds with the term already in the name:

$ perf record -v -e ibs_op/ldlat=0/ -- true
kernel.perf_event_paranoid=2, trying to fall back to excluding kernel and hypervisor samples Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed. Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'.

with no exclude bits the kernel discards nothing. The 'Test data symbol' shell test matches the event string, so its regex now accepts terms added after ldlat=150.

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v4 2/8]:

- debuginfod__setup_urls_env() moved from util/debuginfo.c to util/util.c, next to perf_debuginfod_setup(), and its declaration from util/debuginfo.h to util/util.h: debuginfo.o is only built with CONFIG_LIBDW, so with libdebuginfod but without libdw the declaration was an empty stub and the setup was skipped exactly for build-id.c and probe-event.c, the debuginfod users that are built without libdw and that the commit message lists as reading DEBUGINFOD_URLS in every debuginfod_begin().

PATCH 8/8, "perf mem record: Request PERF_SAMPLE_CPU by default" [sashiko-bot review of PATCH v4 8/8]:

- The rec_argv array now reserves room for the fixed arguments __cmd_record() adds, up to eight with all the optional flags; a pre-existing under-allocation, the array was nine arguments per memory PMU plus the user arguments, and on a PMU with separate load and store events the four event arguments plus the fixed ones already filled it to the last slot, so this new --sample-cpu argument would write past it, e.g. with --phys-data --data-page-size --all-user.

Patches 1, 3, 4, 5, 6 and 7 are unchanged from v4: sashiko-bot found no issues in them.

What changed from v3 (2433e9f0bf86cfc3):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v3 2/8]:

- DEBUGINFOD_URLS is no longer set from the fetch path: setenv() is not thread safe and libdebuginfod reads it with getenv() in every debuginfod_begin(), which perf also does from build-id.c, probe-event.c and probe-finder.c, paths that don't go through the fetch lock, so a thread taking one of them could race with the setenv() a fetch was doing. This is now done from symbol__init(), before the threads are started, from the .urls files in /etc/debuginfod when it isn't set, and the pthread_once() and the call in debuginfod__fetch() are gone with it.

- The in-flight fetch wait was dead code and is gone: with debuginfod__fetch_lock held across the whole lookup and fetch, no second thread could ever see an entry being fetched, so the condition variable, the waiters and the publication dance could not run. debuginfo_lookup__new() became debuginfo_lookup__add(), called after a fetch that brought a file back: lookups remember only the successful fetches, one entry each, answered with a strdup() and re-fetched if that file goes away, and a request for a build ID being fetched waits for the lock and is answered from the misses list or from the entry, so a download still can't happen twice. debuginfod__missed_lock, always taken under the fetch lock, is gone with it.

PATCH 3/8, "perf symbol: Fall back to fetching the vmlinux by build ID" [sashiko-bot review of PATCH v3 3/8]:

- The check for a concurrent load, made after taking dso->lock back from the fetch, is now dso__has_symbols() and not dso__loaded():
dso__load() sets the latter even for an attempt that failed, so a thread whose fetch failed could make a thread whose fetch succeeded discard the vmlinux it just downloaded and carry on with an empty kernel dso, as loaded is set and nothing would try again.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH v3 6/8]:

- The review asked whether the %s of a dwarf_diename() that returns NULL in perf-die-chain would abort the macro. Checked, not changed:
with GDB 17.2, printf prints "(null)" for a NULL char *, including for an inferior call returning one, and the macro carries on. The other %s in the file, dloc->ms->sym->name in perf-dso, can't be NULL
either: the only caller of find_data_type() dereferences ms->sym and ms->map to build the data_loc_info before that frame can exist, so no change there as well.

- Patches 1, 4, 5, 7 and 8 are unchanged from v3: sashiko-bot found no issues in them.

What changed from v2 (00cffc7df5abfa68):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH v2 2/8]:

- The conditions to fetch, debuginfod being enabled, the build-id cache being on and the build ID not having been a miss already, are now looked at with the fetch lock held, as they are the state the fetch itself changes: deciding them outside the lock and fetching inside it let a second thread repeat the fetch the first one had just made, or repeat one that had just been turned off, and put the same build ID on the misses list twice.

- A build ID that is already being fetched is waited for, instead of fetched again: there was nothing that said "this one is in flight", so a second thread wanting it blocked on the fetch lock and then did its own complete fetch, which, while the first one is still running, is a second download of the same file, with a second client, a second progress line and a second turn at the terminal, for the same answer. It now sleeps on a condition variable and is woken with whatever the fetch settled: the file, when there is one, or the not available, when it found nothing or the user cancelled it, with no retry.

- And a fetch that brought a file back is remembered, so that the ones that need the same build ID later, e.g. annotating another symbol of the same DSO, are answered with a strdup() instead of another client:
the file stays in the debuginfod client cache, and its path is checked before being handed out, in case that cache gets cleaned from under us.

- The fetches are still one at a time, so a fetch for one build ID waits for a fetch for another one: the terminal mode, the signal dispositions, the progress line and the 's'/'d' keys are process global, and sharing them between concurrent fetches, refcounted, is left for later, noted in the code. It matters less than it sounds, now that the build IDs of a workload are answered from memory after the first pass over them.

PATCH 3/8, "perf symbol: Fall back to fetching the vmlinux by build ID" [sashiko-bot review of PATCH v2 3/8]:

- The fetch is no longer made with dso->lock held: dso__load() holds it across dso__load_kernel_sym(), and a fetch can block for a long time on the network, or on the terminal, waiting for the user, which would stall every other thread that needs the kernel dso. It is dropped just around the fetch, nothing of the dso is touched while it is, and the symbols are loaded with the lock held again, the same way dso__debuginfo() already does it for the debuginfo of a DSO. If another thread got the symbols for the dso in the meantime, the file that came back is not loaded a second time.

PATCH 5/8, "perf report: Add --progress option" [sashiko-bot review of PATCH v2 5/8]:

- A phase that starts when the progress stack is full is now just not shown, and its finish() is swallowed, instead of force completing the phase that encloses it to make room: that one still gets its own finish() later, which would then complete the phase above it, and so on, one premature completion per nesting level. Reproduced by building with STDIO_PROGRESS__MAX_DEPTH set to 1: v2 prints "Processing events... [100.0%]" while it is at 99.7%, before the nested phase even runs, v3 warns and keeps the outer phase to complete when it really does. With the 8 slots in the tree this cannot be hit, the phases perf has nest at most 2 deep, but the bookkeeping was wrong.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH v2 6/8]:

- The perf-dso GDB macro read dloc->ms.map->dso->name and

    dloc->ms.sym->name, but struct data_loc_info.ms is a
    'struct map_symbol *', so GDB's C evaluator needs -> there: as it
    was, 'perf-stuck.sh -g' would have failed to evaluate it.

- The one pre-existing issue in PATCH 7/8, the unbounded member type chase in die_get_member_type() and the unbounded recursion in die_find_member(), is not addressed here, it went to the perf TODO list as item 180, as it is not on the path this series fixes.

Other changes since v2:

- PATCH 2/8: a refetch made because the cached file was cleaned from under us now updates the entry the same way the first fetch does, and an entry with no path and no error is fetched again instead of answered; as it was, the entry kept the first fetch's success with no path, so the later lookups it existed for returned success with a NULL path, which debuginfo__new_build_id() cannot open and the vmlinux fallback in PATCH 3 would take as a file that came back and dereference.

- PATCH 3/8: the commit message said the ignore_vmlinux_buildid that 'perf record' and 'perf probe' set internally is what keeps them away from the fallback, but 'perf probe' only sets it for the commands other than --list, --del and --add when given an offline vmlinux; the message now says that and that a 'perf probe' that does not set it, e.g. --funcs or --add without --vmlinux on a system with a restricted /proc/kallsyms, can have the kernel debuginfo fetched, like the other tools.

- PATCH 7/8: the kerneldoc of die_get_type_die() said the type would be looked up in the main file and then in the alt file, using the one that has a DIE with the saved tag, but the function deliberately resolves in only the file the offset was recorded as belonging to, with no fallback, and @from_alt was missing from the parameter list; it now describes that, and the die_collect_vars() and die_collect_global_vars() kerneldocs, which still said the type could be retrieved with dwarf_offdie() and the offset alone, now point at die_get_type_die().

- PATCH 7/8: the comment on the member->truncated flag said the browser reads it, but the browser picks between ';' and '{' from the children list, so a member cut by the nesting limit is drawn like a complete leaf; the comment now says the flag is for the JSON exporter added in a later series.

- PATCH 2/8: a search aborted by the user, with 's' or a signal, is now remembered for the rest of the session: the threads already waiting for that fetch were told there was nothing to share, but a request that arrived later started a new download of the same file, which the user may have skipped for being too big. debuginfod__misses now records such a cancellation as a cancellation, the 's' message says the build ID will not be fetched again in this session and the debug message tells a cancellation and a server miss apart. The lookup entry is also now allocated and published by a debuginfo_lookup__new() helper called before the fetch, instead of an allocation whose failure was only checked after it.

- Patches 1, 4 and 8 are unchanged from v2: sashiko-bot found no issues in them.

What changed from v1 (060dccedab1617dc):

PATCH 2/8, "perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod" [sashiko-bot review of PATCH 2/8]:

- Included <limits.h> for PATH_MAX, that was coming in through a glibc transitive include and fails to build with musl libc.

- debuginfod__setup_urls_env() now goes through pthread_once, so that the setenv() of DEBUGINFOD_URLS happens exactly once: setenv() is not thread safe and this is on the fetch path, which dso__debuginfo() deliberately takes outside dso__lock.

- The fetch now runs with debuginfod__fetch_lock held, so that the process global state it uses — the terminal settings, the SIGINT/ SIGTERM dispositions and the progress and cancellation state — is only ever touched by one fetch at a time. A second concurrent fetch would otherwise have taken the first one's raw mode as the state to restore, leaving the terminal broken when it was done, and would reset the cancellation state from under the fetch already in progress. Serializing also keeps two fetches from racing for the same keypresses and the same progress line, and a second fetch has nothing to gain from running in parallel with a first one reading the same kind of file off the same servers.

- An interrupt that lands after the fetch already succeeded is no longer swallowed: at that point the terminal and the signal dispositions are the original ones again, so the signal is re-raised, the same way the failure path does, instead of perf carrying on as if the user had not asked it to stop.

PATCH 6/8, "perf scripts: Add perf-stuck, to tell where a running perf is stuck" [sashiko-bot review of PATCH 6/8]:

- /proc/<pid>/stat is now parsed with the parenthesized command name removed first, so that a process with spaces in its name no longer shifts every field after it: watching a process named 'my sleep' used to report "sleep)" as its state and a garbage RSS.

- The CPU time is printed with awk's printf "%d" instead of relying on its default output format, that switches to scientific notation, which the bash arithmetic below cannot parse, once the sum of utime and stime goes past six digits, i.e. some 16 minutes of CPU at 100 Hz.

- The process can go away between the '-d /proc/<pid>' check and the read of /proc/<pid>/stat, so the read is now checked: a failed or empty one reports "process gone" instead of 'set -u' aborting the script on an unbound field, which is what would have hidden the very exit the script is meant to report.

- The command line printed when the script starts has its control characters stripped, so that a process started with escape sequences in its arguments, e.g. one replaying a log line, does not get them replayed on the terminal of whoever runs this.

- The two pre-existing issues sashiko-bot raised that are outside the scope of this series, the child entries and their hists arrays leaked by the data type browser teardown and the missing <stdio.h> and <stdlib.h> includes in ui/browsers/annotate-data.c, are recorded as items 178 and 179 of the perf TODO list for follow-up work.

- Patches 1, 3, 4, 5, 7 and 8 are unchanged from v1: sashiko-bot found no issues in them.

Quoted 14 lines

Arnaldo Carvalho de Melo (12):
perf test: Skip data_type_profiling when the PMU cannot record memory
events
perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod perf config: Move perf_config__set_variable() to util/config.c perf debuginfo: Let the user skip and disable debuginfod fetches perf debuginfo: Show the debuginfod fetch progress and keys in the TUI perf symbol: Fall back to fetching the vmlinux by build ID perf annotate-data: Show the sample count in the data-type browser perf report: Add --progress option
perf scripts: Add perf-stuck, to tell where a running perf is stuck perf annotate-data: Resolve type DIEs in the debug file they came from perf mem record: Request PERF_SAMPLE_CPU by default
perf mem record: Use the IBS swfilt filter when available

In general, I feel like AI generated patches are too verbose - not only for the cover letter, it's about commit log and the code comments. It adds more burdens on the reviewer side. Is it possible to guide your agent to produce less messages with important informations only?

Thanks,
Namhyung

Quoted 51 lines
 tools/build/feature/test-libdw.c              |  15 +-
 tools/perf/Documentation/perf-annotate.txt    |  10 +
 tools/perf/Documentation/perf-config.txt      |  19 +
 tools/perf/Documentation/perf-mem.txt         |   4 +
 tools/perf/Documentation/perf-report.txt      |  31 +
 tools/perf/Documentation/perf-top.txt         |  16 +
 tools/perf/Makefile.config                    |   2 +-
 tools/perf/arch/x86/util/mem-events.c         |  17 +-
 tools/perf/builtin-annotate.c                 |   3 +
 tools/perf/builtin-config.c                   |  70 +-
 tools/perf/builtin-mem.c                      |  18 +-
 tools/perf/builtin-report.c                   |  18 +
 tools/perf/builtin-top.c                      |   7 +
 tools/perf/scripts/perf-stuck.gdb             | 104 +++
 tools/perf/scripts/perf-stuck.sh              | 194 ++++++
 tools/perf/tests/shell/data_type_profiling.sh |  46 +-
 tools/perf/tests/shell/test_data_symbol.sh    |   6 +-
 tools/perf/ui/Build                           |   1 +
 tools/perf/ui/browsers/annotate-data.c        |   2 +-
 tools/perf/ui/progress.h                      |   2 +
 tools/perf/ui/stdio/progress.c                | 185 ++++++
 tools/perf/ui/tui/util.c                      | 177 +++++
 tools/perf/ui/util.h                          |  42 ++
 tools/perf/util/annotate-data.c               |  59 +-
 tools/perf/util/annotate-data.h               |   3 +
 tools/perf/util/config.c                      |  73 +++
 tools/perf/util/config.h                      |   2 +
 tools/perf/util/debuginfo.c                   | 608 ++++++++++++++++++
 tools/perf/util/debuginfo.h                   |  36 ++
 tools/perf/util/dso.c                         |  19 +
 tools/perf/util/dwarf-aux.c                   | 171 ++++-
 tools/perf/util/dwarf-aux.h                   |  33 +
 tools/perf/util/mem-events.c                  |  21 +-
 tools/perf/util/mem-events.h                  |   2 +
 tools/perf/util/ordered-events.c              |  17 +-
 tools/perf/util/session.c                     |  12 +-
 tools/perf/util/symbol.c                      |  83 ++-
 tools/perf/util/symbol_conf.h                 |   1 +
 tools/perf/util/util.c                        | 103 +++
 tools/perf/util/util.h                        |   8 +
 40 files changed, 2105 insertions(+), 135 deletions(-)

base-commit: 91b0782fc9e9d2f0a40b5256146e014802fdbb36
v5-head: ef0ac8e44d43d2aa86842d2f83ad41d1c058d949
v4-head: f801de4e80289566670e6b735198606b672e74e8
v3-head: 2433e9f0bf86cfc3d9966b467042563596c9ebdb
v2-head: 00cffc7df5abfa6850beb360deb830d8495163ce
v1-head: 060dccedab1617dcbf6e2be64ed97afe3f3fac63 --

Assisted-by LLM
#41 lore raw
Quoted 4 lines · Namhyung Kim

In general, I feel like AI generated patches are too verbose - not only for the cover letter, it's about commit log and the code comments. It adds more burdens on the reviewer side. Is it possible to guide your agent to produce less messages with important informations only?

Yeah, on one side it provides more context and explanations that can be useful in understanding new code and changes in the patch series versions, but gets tiresome at times, I'll change it to be more succinct.

Thanks for the feedback :-)
- Arnaldo

author maintainer
#30 lore raw

Re: [PATCH 10/12] perf annotate-data: Resolve type DIEs in the debug file they came from

In reply to Namhyung Kim
Quoted 12 lines · Namhyung Kim
Quoted 10 lines · Arnaldo Carvalho de Melo

--- a/tools/perf/util/annotate-data.c

tools/perf/util/annotate-data.c +0 -5
@@ -242,12 +261,6 @@ static int __add_member_cb(Dwarf_Die *die, void *arg)
 	strbuf_init(&sb, 32);
 	die_get_typename(die, &sb);
 
-	__die_get_real_type(die, &member_type);
-	if (dwarf_tag(&member_type) == DW_TAG_typedef)
-		die_get_real_type(&member_type, &die_mem);
-	else
-		die_mem = member_type;
Quoted 4 lines

tools/perf/util/annotate-data.c static int __add_member_cb(Dwarf_Die *die, void *arg) · line 265

Quoted 2 lines
tools/perf/util/annotate-data.c +0 -0
 	if (dwarf_aggregate_size(&die_mem, &size) < 0)
 		size = 0;

This is gonna conflict with my change. :)

We can get yours first and I'll adapt :-)

- Arnaldo

#37 lore raw

Re: [PATCH 12/12] perf mem record: Use the IBS swfilt filter when available

In reply to Namhyung Kim
Quoted 3 lines

It's a bit unfortunate we hardcode "swfilt" here. Probably better to have it in the mem-events array and add it to the name format dynamically. But as it's only needed for IBS, I think it's ok for now.

Something like this? (Untested):

tools/perf/arch/x86/util/mem-events.c +12 -0
index b38f519020ff..816c73d75bc4 100644
@@ -27,8 +27,20 @@ struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX] = {
 	E("mem-ldst",	"%s//",		NULL,	false,	0),
 };
 
+struct perf_mem_event perf_mem_events_amd_swfilt[PERF_MEM_EVENTS__MAX] = {
+	E(NULL,		NULL,		NULL,	false,	0),
+	E(NULL,		NULL,		NULL,	false,	0),
+	E("mem-ldst",	"%s/swfilt=1/",	NULL,	false,	0),
+};
+
 struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
 	E("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0),
 };
+
+struct perf_mem_event perf_mem_events_amd_ldlat_swfilt[PERF_MEM_EVENTS__MAX] = {
+	E(NULL,		NULL,				NULL,	false,	0),
+	E(NULL,		NULL,				NULL,	false,	0),
+	E("mem-ldst",	"%s/ldlat=%u,swfilt=1/",	NULL,	true,	0),
+};
tools/perf/arch/x86/util/mem-events.h +2 -0
index 11e09a256f5b..f707de380370 100644
@@ -6,6 +6,8 @@ extern struct perf_mem_event perf_mem_events_intel[PERF_MEM_EVENTS__MAX];
 extern struct perf_mem_event perf_mem_events_intel_aux[PERF_MEM_EVENTS__MAX];
 
 extern struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX];
+extern struct perf_mem_event perf_mem_events_amd_swfilt[PERF_MEM_EVENTS__MAX];
 extern struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX];
+extern struct perf_mem_event perf_mem_events_amd_ldlat_swfilt[PERF_MEM_EVENTS__MAX];
 
 #endif /* _X86_MEM_EVENTS_H */
tools/perf/arch/x86/util/pmu.c +8 -2
index 2c24ef3140da..fd4491fda117 100644
@@ -333,6 +333,7 @@ static void uncore_cha_imc_adjust_cpumask_for_snc(struct perf_pmu *pmu, bool cha
 void perf_pmu__arch_init(struct perf_pmu *pmu)
 {
 	struct perf_pmu_caps *ldlat_cap;
+	bool swfilt_format;
 
 	if (!strcmp(pmu->name, INTEL_PT_PMU_NAME)) {
 		pmu->auxtrace = true;
@@ -348,7 +349,10 @@ void perf_pmu__arch_init(struct perf_pmu *pmu)
 		if (strcmp(pmu->name, "ibs_op"))
 			return;
 
-		pmu->mem_events = perf_mem_events_amd;
+		swfilt_format = perf_pmu__has_format(pmu, "swfilt");
+		pmu->mem_events = swfilt_format ?
+				  perf_mem_events_amd_swfilt :
+				  perf_mem_events_amd;
 
 		if (!perf_pmu__caps_parse(pmu))
 			return;
@@ -358,7 +362,9 @@ void perf_pmu__arch_init(struct perf_pmu *pmu)
 			return;
 
 		perf_mem_events__loads_ldlat = 0;
-		pmu->mem_events = perf_mem_events_amd_ldlat;
+		pmu->mem_events = swfilt_format ?
+				  perf_mem_events_amd_ldlat_swfilt :
+				  perf_mem_events_amd_ldlat;
 	} else {
 		if (pmu->is_core) {
 			if (perf_pmu__have_event(pmu, "mem-loads-aux"))

---

Thanks,
Ravi

author maintainer
#38 lore raw

Re: [PATCH 12/12] perf mem record: Use the IBS swfilt filter when available

Quoted 5 lines · Ravi Bangoria
Quoted 3 lines

It's a bit unfortunate we hardcode "swfilt" here. Probably better to have it in the mem-events array and add it to the name format dynamically. But as it's only needed for IBS, I think it's ok for now.

Something like this? (Untested):

Seems ok, I'm trying to use it now, ok?

- Arnaldo

Quoted 78 lines
tools/perf/arch/x86/util/mem-events.c +12 -0
index b38f519020ff..816c73d75bc4 100644
@@ -27,8 +27,20 @@ struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX] = {
 	E("mem-ldst",	"%s//",		NULL,	false,	0),
 };
 
+struct perf_mem_event perf_mem_events_amd_swfilt[PERF_MEM_EVENTS__MAX] = {
+	E(NULL,		NULL,		NULL,	false,	0),
+	E(NULL,		NULL,		NULL,	false,	0),
+	E("mem-ldst",	"%s/swfilt=1/",	NULL,	false,	0),
+};
+
 struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX] = {
 	E(NULL,		NULL,		NULL,	false,	0),
 	E(NULL,		NULL,		NULL,	false,	0),
 	E("mem-ldst",	"%s/ldlat=%u/",	NULL,	true,	0),
 };
+
+struct perf_mem_event perf_mem_events_amd_ldlat_swfilt[PERF_MEM_EVENTS__MAX] = {
+	E(NULL,		NULL,				NULL,	false,	0),
+	E(NULL,		NULL,				NULL,	false,	0),
+	E("mem-ldst",	"%s/ldlat=%u,swfilt=1/",	NULL,	true,	0),
+};
tools/perf/arch/x86/util/mem-events.h +2 -0
index 11e09a256f5b..f707de380370 100644
@@ -6,6 +6,8 @@ extern struct perf_mem_event perf_mem_events_intel[PERF_MEM_EVENTS__MAX];
 extern struct perf_mem_event perf_mem_events_intel_aux[PERF_MEM_EVENTS__MAX];
 
 extern struct perf_mem_event perf_mem_events_amd[PERF_MEM_EVENTS__MAX];
+extern struct perf_mem_event perf_mem_events_amd_swfilt[PERF_MEM_EVENTS__MAX];
 extern struct perf_mem_event perf_mem_events_amd_ldlat[PERF_MEM_EVENTS__MAX];
+extern struct perf_mem_event perf_mem_events_amd_ldlat_swfilt[PERF_MEM_EVENTS__MAX];
 
 #endif /* _X86_MEM_EVENTS_H */
tools/perf/arch/x86/util/pmu.c +8 -3
index 2c24ef3140da..fd4491fda117 100644
@@ -333,6 +333,7 @@ static void uncore_cha_imc_adjust_cpumask_for_snc(struct perf_pmu *pmu, bool cha
 void perf_pmu__arch_init(struct perf_pmu *pmu)
 {
 	struct perf_pmu_caps *ldlat_cap;
+	bool swfilt_format;
 
 	if (!strcmp(pmu->name, INTEL_PT_PMU_NAME)) {
 		pmu->auxtrace = true;
@@ -348,7 +349,10 @@ void perf_pmu__arch_init(struct perf_pmu *pmu)
 		if (strcmp(pmu->name, "ibs_op"))
 			return;
 
-		pmu->mem_events = perf_mem_events_amd;
+		swfilt_format = perf_pmu__has_format(pmu, "swfilt");
+		pmu->mem_events = swfilt_format ?
+				  perf_mem_events_amd_swfilt :
+				  perf_mem_events_amd;
 
 		if (!perf_pmu__caps_parse(pmu))
 			return;
@@ -358,7 +362,9 @@ void perf_pmu__arch_init(struct perf_pmu *pmu)
 			return;
 
 		perf_mem_events__loads_ldlat = 0;
-		pmu->mem_events = perf_mem_events_amd_ldlat;
+		pmu->mem_events = swfilt_format ?
+				  perf_mem_events_amd_ldlat_swfilt :
+				  perf_mem_events_amd_ldlat;
 	} else {
 		if (pmu->is_core) {
 			if (perf_pmu__have_event(pmu, "mem-loads-aux"))
 
---

Thanks,
Ravi

#39 lore raw

Re: [PATCH 12/12] perf mem record: Use the IBS swfilt filter when available

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

It's a bit unfortunate we hardcode "swfilt" here. Probably better to have it in the mem-events array and add it to the name format dynamically. But as it's only needed for IBS, I think it's ok for now.

Something like this? (Untested):

Seems ok, I'm trying to use it now, ok?

Sure, Thanks.

Ravi

author maintainer
#22 lore raw

Re: [PATCH 07/12] perf annotate-data: Show the sample count in the data-type browser

In reply to Namhyung Kim
Quoted 15 lines · Namhyung Kim
Quoted 14 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.

Print the histogram entry's own count instead.

This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.

Quoted 7 lines
Quoted 3 lines
Fixes d001c7a7f4736743 ("perf annotate-data: Add hist_entry__annotate_data_tui()")
Assisted-by LLM

This can be applied independently.

Thanks, cherry picked into perf-tools-next, for v7.4.

- Arnaldo