Skip to contentSkip to conversation

perf machine: Don't unwind the arm64 leaf frame caller without symbols

PATCH In review Matthew Wood 3 participants 4 messages Oct 14 – Oct 15 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf machine: Don't unwind the arm64 leaf frame caller without symbols

On arm64, 'perf record --call-graph fp' also samples LR so that thread__resolve_callchain_sample() can recover the leaf function's caller, which is missing from the frame pointer chain. It does so via get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The unwinder looks up the symbol of every frame it visits, so the first lookup in a DSO loads its full symbol table and demangles every symbol in it.

Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain") added the symbols argument so that callers which only need maps can avoid loading symbols, and passed it down to every add_callchain_ip() call. The leaf frame unwind was missed and still runs for symbols=false, so 'perf inject' on arm64 has been loading symbols it doesn't need ever since.

Commit 29be60c93d2d ("perf build-id: Mark DSO in sample callchains") compounded the issue: process_buildids() now walks the callchain of every sample at the end of 'perf record' to mark DSOs hit. With -a, that loads and demangles the symbol tables of every user binary sampled on the system. This can send things into exponential time and memory (over 1GB of heap growth in 5 seconds), so

  perf record -a -e cycles:u --call-graph fp -- sleep 2

never finishes writing perf.data after the workload exits.

Callers that pass symbols=false only want the maps that callchain IPs fall in, and add_callchain_ip() already honours that by using thread__find_map() instead of thread__find_symbol(). Skip the leaf frame unwind for them as well. The leaf frame caller is almost always in the same DSO as the sampled IP or the next frame in the chain, so build-id marking loses nothing in practice.

Fixes 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")
---
 tools/perf/util/machine.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/util/machine.c +7 -1
index f86b3b7df742..51597cddd239 100644
@@ -2989,7 +2989,13 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 
 		usr_idx = callchain_param.order == ORDER_CALLEE ? j-2 : j-1;
 
-		if (usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {
+		/*
+		 * Finding the leaf frame caller needs a DWARF unwind, and the
+		 * unwinder looks up the symbol of every frame it visits. Skip
+		 * it when the caller asked for maps only, as for build-id
+		 * marking, so that no symbol tables are loaded.
+		 */
+		if (symbols && usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {
 
 			leaf_frame_caller = get_leaf_frame_caller(sample, thread, usr_idx);
 

--
2.53.0-Meta

#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

On arm64, 'perf record --call-graph fp' also samples LR so that thread__resolve_callchain_sample() can recover the leaf function's caller, which is missing from the frame pointer chain. It does so via get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The unwinder looks up the symbol of every frame it visits, so the first lookup in a DSO loads its full symbol table and demangles every symbol in it.

Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain") added the symbols argument so that callers which only need maps can avoid loading symbols, and passed it down to every add_callchain_ip() call. The leaf frame unwind was missed and still runs for symbols=false, so 'perf inject' on arm64 has been loading symbols it doesn't need ever since.

Quoted 6 lines

[ ... ]
in the same DSO as the sampled IP or the next frame in the chain, so build-id marking loses nothing in practice.

Fixes 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#2 lore raw
In reply to Matthew Wood
Quoted 46 lines · Matthew Wood

On arm64, 'perf record --call-graph fp' also samples LR so that thread__resolve_callchain_sample() can recover the leaf function's caller, which is missing from the frame pointer chain. It does so via get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The unwinder looks up the symbol of every frame it visits, so the first lookup in a DSO loads its full symbol table and demangles every symbol in it.

Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain") added the symbols argument so that callers which only need maps can avoid loading symbols, and passed it down to every add_callchain_ip() call. The leaf frame unwind was missed and still runs for symbols=false, so 'perf inject' on arm64 has been loading symbols it doesn't need ever since.

Commit 29be60c93d2d ("perf build-id: Mark DSO in sample callchains") compounded the issue: process_buildids() now walks the callchain of every sample at the end of 'perf record' to mark DSOs hit. With -a, that loads and demangles the symbol tables of every user binary sampled on the system. This can send things into exponential time and memory (over 1GB of heap growth in 5 seconds), so

  perf record -a -e cycles:u --call-graph fp -- sleep 2

never finishes writing perf.data after the workload exits.

Callers that pass symbols=false only want the maps that callchain IPs fall in, and add_callchain_ip() already honours that by using thread__find_map() instead of thread__find_symbol(). Skip the leaf frame unwind for them as well. The leaf frame caller is almost always in the same DSO as the sampled IP or the next frame in the chain, so build-id marking loses nothing in practice.

Fixes 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")
---
 tools/perf/util/machine.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/util/machine.c +0 -0
index f86b3b7df742..51597cddd239 100644
@@ -2989,7 +2989,13 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 

tools/perf/util/machine.c static int thread__resolve_callchain_sample(struct thread *thread, · line 2998

Quoted 10 lines
tools/perf/util/machine.c +7 -1
                usr_idx = callchain_param.order == ORDER_CALLEE ? j-2 : j-1;
 
-               if (usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {
+               /*
+                * Finding the leaf frame caller needs a DWARF unwind, and the
+                * unwinder looks up the symbol of every frame it visits. Skip
+                * it when the caller asked for maps only, as for build-id
+                * marking, so that no symbol tables are loaded.
+                */
+               if (symbols && usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {

Hi Matthew, I'm not sure if this is a serious patch or a bug report about the slowness of DSO marking. Broadly DSO marking is known to be slow and many perf tests run with -B and -N for this reason. I believe the intent behind DSO marking as a default is to ensure that perf provides the best profiling experience. This comes at a cost, but this fix looks like it will break things because IPs will be lost.

Trying to look a little more at the ARM64 performance problem, I see that we're trying to unwind 2 frames:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/arm64-frame-pointer-unwind-support.c?h=perf-tools-next#n70
but often just 1 is read. For libunwind the 1 frame case has a special case:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libunwind.c?h=perf-tools-next#n671
I wonder if we could replicate the 1 frame case with libdw to avoid processing the DSO, and update the caller to be:

```
ret = unwind__get_entries(add_entry, &entries, thread, sample,
callchain_param.order == ORDER_CALLER ? 1 : 2, true);
```

Thanks,
Ian

Quoted 5 lines
tools/perf/util/machine.c +0 -1
                        leaf_frame_caller = get_leaf_frame_caller(sample, thread, usr_idx);
 
--

2.53.0-Meta

#3 lore raw
Quoted 8 lines · Ian Rogers

Hi Matthew, I'm not sure if this is a serious patch or a bug report about the slowness of DSO marking. Broadly DSO marking is known to be slow and many perf tests run with -B and -N for this reason. I believe the intent behind DSO marking as a default is to ensure that perf provides the best profiling experience. This comes at a cost, but this fix looks like it will break things because IPs will be lost.

Hi Ian,

Thanks for looking. It is definitely meant as an attempted fix. On a 128-CPU arm64 host, 'perf record -a -e cycles:u --call-graph fp -- sleep 2' never finishes writing perf.data, and I'm trying to save some unnecessary work to solve that. perf sits in __cxa_demangle() on one C++ symbol, with heap growing by over 1GB every 5 seconds.

You're right about the lost IPs, thanks for mentioning. v1 drops the leaf frame caller for symbols=false so a DSO that is only reached through that frame wouldn't be marked.

Quoted 11 lines

Trying to look a little more at the ARM64 performance problem, I see that we're trying to unwind 2 frames:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/arm64-frame-pointer-unwind-support.c?h=perf-tools-next#n70
but often just 1 is read. For libunwind the 1 frame case has a special case:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libunwind.c?h=perf-tools-next#n671
I wonder if we could replicate the 1 frame case with libdw to avoid processing the DSO, and update the caller to be:

```
ret = unwind__get_entries(add_entry, &entries, thread, sample,
callchain_param.order == ORDER_CALLER ? 1 : 2, true);
```

I looked at this but I don't think it would avoid the hang, unless I'm misreading the code:

- The cost is in loading symbols, not in the DWARF unwind itself. libunwind's 1 frame case still goes through entry(), which calls thread__find_symbol(). libdw's __report_module() does the same for every frame. Either way, the DSO's symbol table gets loaded and every name in it demangled.

- callchain_param.order defaults to ORDER_CALLEE so process_buildids() would still ask for 2 frames.

- The 1 frame case returns only the sample PC. That's the IP we already have, not its caller, and get_leaf_frame_caller_aarch64() needs entries.length == 2. So in caller order I think it would stop recovering the leaf frame caller altogether.

I'm working on and testing a v2 that instead keeps the unwind and frames as they are, but passes a symbols flag down to unwind__get_entries(). When it's false, libdw and libunwind look up only the map of each frame. The leaf frame code only uses the addresses, so it never asks for symbols, and thread__resolve_callchain_unwind() passes its symbols argument through. That also removes the "Not resolving symbols with an unwinder isn't currently supported" limitation for 'perf inject' with DWARF callchains.

On the arm64 host, the 'perf record -a' command above now finishes. To check that no IPs are lost, I used a small program where one DSO can only be reached through the leaf frame caller:

app -> libmid.so:mid() -> libleaf.so:leaf_spin()

leaf_spin() has no frame record, so the kernel's frame pointer chain goes straight from leaf_spin() to main(). No samples land in mid() itself, so libmid.so is only marked hit if the leaf frame caller is kept. Running 'perf record --call-graph fp' without -N and checking which DSOs end up in the build-id cache:

unpatched: libmid.so cached
v1: libmid.so missing
v2: libmid.so cached

'perf script' output is identical between the unpatched perf and v2, with mid() recovered in the same 2053 samples.

On x86 with libdw, 'perf inject -b' on a --call-graph dwarf profile goes from 0.24s and 58.5MB max RSS to 0.04s and 15.5MB, with the same build-ids injected. 'perf report' output and the dwarf unwind test are unchanged.

I'll send v2 shortly.

Thanks,
Matthew