perf: Add a libdw addr2line implementation
Ian Rogers
reviewerv3 cover perf: Add a libdw addr2line implementation
addr2line is a performance bottleneck in perf, add a libdw based implementation that avoids forking addr2line and caches the decoded debug information.
Allow the addr2line implementation to be picked via the configuration file or --addr2line-style with perf report.
Test/fix that inline callchains are properly displayed by perf script.
An example:
```
$ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
[ perf record: Woken up 132 times to write data ]
[ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
$ perf script --fields +srcline
...
perf-inlineloop 1814670 293100.228871: 640004 cpu_core/cycles/u:
55a11d6e61ee leaf+0x2e
inlineloop.c:21 (inlined)
55a11d6e61ee middle+0x2e
inlineloop.c:27 (inlined)
55a11d6e61ee parent+0x2e (perf)
inlineloop.c:32
55a11d6e629b inlineloop+0x8b (perf)
inlineloop.c:47
55a11d69a3bc run_workload+0x5a (perf)
builtin-test.c:715
55a11d69aa9f cmd_test+0x417 (perf)
builtin-test.c:825
55a11d6155f5 run_builtin+0xd4 (perf)
perf.c:349
55a11d61588d handle_internal_command+0xdd (perf)
perf.c:401
55a11d6159e6 run_argv+0x35 (perf)
perf.c:445
55a11d615d2f main+0x2cb (perf)
perf.c:553
7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
libc_start_call_main.h:58
7fae3d233d64 __libc_start_main_impl+0x84
libc-start.c:360 (inlined)
55a11d565f80 _start+0x20 (perf)
??:0
...
```v3: Make the caller inline file and line number accurate in the libdw
addr2line, rather than using the function's declared location. Fix reference counts in unwind-libdw. Add fixes tag for srcline inline printing.
v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
from perf's dwarf-aux to fully walk inline functions. Add testing that inlined functions are shown in the perf script srcline callchain information. Add configurability as to which addr2line style to use.
https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/
Ian Rogers (7):
perf unwind-libdw: Fix invalid reference counts
perf addr2line: Add a libdw implementation
perf addr2line.c: Rename a2l_style to cmd_a2l_style perf srcline: Add configuration support for the addr2line style perf callchain: Fix srcline printing with inlines perf test workload: Add inlineloop test workload
perf test: Test addr2line unwinding works with inline functions
tools/perf/builtin-report.c | 10 ++ tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/addr2line_inlines.sh | 47 ++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++ tools/perf/util/Build | 1 + tools/perf/util/addr2line.c | 20 +-- tools/perf/util/config.c | 4 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 ++ tools/perf/util/evsel_fprintf.c | 8 +- tools/perf/util/libdw.c | 153 ++++++++++++++++++++ tools/perf/util/libdw.h | 60 ++++++++ tools/perf/util/srcline.c | 116 ++++++++++++++- tools/perf/util/srcline.h | 3 + tools/perf/util/symbol_conf.h | 10 ++ tools/perf/util/unwind-libdw.c | 7 +- 18 files changed, 486 insertions(+), 22 deletions(-) create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh create mode 100644 tools/perf/tests/workloads/inlineloop.c create mode 100644 tools/perf/util/libdw.c create mode 100644 tools/perf/util/libdw.h
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 1/7 perf unwind-libdw: Fix invalid reference counts
The addition of addr_location__exit causes use-after put on the maps and map references in the unwind info. Add the gets and then add the map_symbol__exits.
--- tools/perf/util/unwind-libdw.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/perf/util/unwind-libdw.c +5 -2
@@ -136,8 +136,8 @@ static int entry(u64 ip, struct unwind_info *ui) } e->ip = ip; - e->ms.maps = al.maps; - e->ms.map = al.map; + e->ms.maps =maps__get (al.maps); + e->ms.map =map__get (al.map); e->ms.sym = al.sym; pr_debug("unwind: %s:ip = 0x%" PRIx64 " (0x%" PRIx64 ")\n",
@@ -325,6 +325,9 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,if (err)pr_debug ("unwind: failed with '%s'\n" ,dwfl_errmsg (-1 )); +for (i =0 ; i < ui->idx; i++) +map_symbol__exit (&ui->entries[i].ms); +dwfl_end (ui->dwfl);free (ui);return 0 ;
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 2/7 perf addr2line: Add a libdw implementation
In reply to Ian RogersAdd an implementation of addr2line that uses libdw. Other addr2line implementations are slow, particularly in the case of forking addr2line. Add an implementation that caches the libdw information in the dso and uses it to find the file and line number information. Inline information is supported but because cu_walk_functions_at visits the leaf function last add a inline_list__append_tail to reverse the lists order.
--- tools/perf/util/Build | 1 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 +++ tools/perf/util/libdw.c | 153 ++++++++++++++++++++++++++++++++++++++ tools/perf/util/libdw.h | 60 +++++++++++++++ tools/perf/util/srcline.c | 24 ++++++ tools/perf/util/srcline.h | 1 + 7 files changed, 252 insertions(+) create mode 100644 tools/perf/util/libdw.c create mode 100644 tools/perf/util/libdw.h
tools/perf/util/Build +1 -0
@@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o perf-util-$(CONFIG_LIBDW) += debuginfo.o perf-util-$(CONFIG_LIBDW) += annotate-data.o +perf-util-$(CONFIG_LIBDW) += libdw.o perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o perf-util-$(CONFIG_LOCAL_LIBUNWIND) += unwind-libunwind-local.o
tools/perf/util/dso.c +2 -0
@@ -32,6 +32,7 @@#include "string2.h" #include "vdso.h" #include "annotate-data.h" +#include "libdw.h" static const char *const debuglink_paths[] = {"%.0s%s" ,
@@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)auxtrace_cache__free (RC_CHK_ACCESS (dso)->auxtrace_cache);dso_cache__free (dso);dso__free_a2l (dso); +dso__free_a2l_libdw (dso);dso__free_symsrc_filename (dso);nsinfo__zput (RC_CHK_ACCESS (dso)->nsinfo);mutex_destroy (dso__lock (dso));
tools/perf/util/dso.h +11 -0
@@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {const char *short_name;const char *long_name;void *a2l; +void *a2l_libdw;char *symsrc_filename;#if defined(__powerpc__)void *dwfl;/* DWARF debug info */
@@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)RC_CHK_ACCESS (dso)->a2l = val; } +static inline void *dso__a2l_libdw (const struct dso *dso) +{ +return RC_CHK_ACCESS (dso)->a2l_libdw; +} + +static inline void dso__set_a2l_libdw (struct dso *dso,void *val) +{ +RC_CHK_ACCESS (dso)->a2l_libdw = val; +} +static inline unsigned int dso__a2l_fails (const struct dso *dso) {return RC_CHK_ACCESS (dso)->a2l_fails;
tools/perf/util/libdw.c new file +153 -0
153 lines not on this page · raw message
tools/perf/util/libdw.h new file +60 -0
60 lines not on this page · raw message
tools/perf/util/srcline.c +24 -0
@@ -6,6 +6,7 @@#include "libbfd.h" #include "llvm.h" #include "symbol.h" +#include "libdw.h" #include <inttypes.h> #include <string.h>
@@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_nodereturn 0 ; } +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node) +{ +struct inline_list *ilist; + + ilist =zalloc (sizeof (*ilist)); +if (ilist ==NULL ) +return -1 ; + + ilist->symbol = symbol; + ilist->srcline = srcline; + +if (callchain_param.order ==ORDER_CALLEE ) +list_add (&ilist->list, &node->val); +else +list_add_tail (&ilist->list, &node->val); + +return 0 ; +} +/* basename version that takes a const input string */ static const char *gnu_basename (const char *path) {
@@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int * {int ret; + ret =libdw__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); +if (ret >0 ) +return ret; + ret =llvm__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);if (ret >0 )return ret;
tools/perf/util/srcline.h +1 -0
@@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);void inlines__tree_delete (struct rb_root_cached *tree);int inline_list__append (struct symbol *symbol,char *srcline,struct inline_node *node); +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node);char *srcline_from_fileline (const char *file,unsigned int line); struct symbol *new_inline_sym(struct dso *dso, struct symbol *base_sym,
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Arnaldo Carvalho de Melo
maintainerRe: [PATCH v3 2/7] perf addr2line: Add a libdw implementation
Quoted 7 lines · Ian Rogers
Add an implementation of addr2line that uses libdw. Other addr2line implementations are slow, particularly in the case of forking addr2line. Add an implementation that caches the libdw information in the dso and uses it to find the file and line number information. Inline information is supported but because
cu_walk_functions_atvisits the leaf function last add ainline_list__append_tailto reverse the lists order.
So while testing I moved the inlineloop 'perf test' workload to the front to test without this new libdw implementation and then with it, and tested using perf probe on it:
root@number:/home/acme# perf probe -x ~/bin/perf libdw__addr2line%return 'ret=$retval' Added new event: probe_perf:libdw_addr2line__return (on libdw__addr2line%return in /home/acme/bin/perf with ret=$retval)
You can now use it in all perf tools, such as:
perf record -e probe_perf:libdw_addr2line__return -aR sleep 1
root@number:/home/acme# perf trace -e probe_perf:libdw_addr2line__return perf report -f --dso perf --stdio -s srcfile,srcline
0.000 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
0.007 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.680 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.685 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.689 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.692 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.696 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.699 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.702 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.705 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.709 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.712 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.715 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.718 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.721 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.724 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.727 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.729 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.732 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.734 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.737 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.740 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
0.771 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
13.563 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
13.766 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
14.099 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
14.103 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.105 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.134 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.143 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.242 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.244 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.245 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.246 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.248 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.250 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.251 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.252 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.258 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.260 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.265 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.267 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.268 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
14.270 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 4K of event 'cpu/cycles/Pu'
# Event count (approx.): 5535180842
#
# Overhead Source File Source:Line
# ........ ............ ...............
#
99.04% inlineloop.c inlineloop.c:21
0.46% inlineloop.c inlineloop.c:20#
# (Tip: Save output of perf stat using: perf stat record <target workload>) #
root@number:/home/acme#
Putting more probes to see the fallbacks...
root@number:/home/acme# perf probe -l probe_perf:cmd_addr2line (on cmd__addr2line@util/addr2line.c in /home/acme/bin/perf) probe_perf:libbfd_addr2line (on libbfd__addr2line@util/libbfd.h in /home/acme/bin/perf) probe_perf:libdw_addr2line (on libdw__addr2line@util/libdw.c in /home/acme/bin/perf) probe_perf:llvm_addr2line (on llvm__addr2line@util/llvm.c in /home/acme/bin/perf) probe_perf:libdw_addr2line__return (on libdw__addr2line%return@util/libdw.c in /home/acme/bin/perf with ret) root@number:/home/acme#
As we have:
static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *line_nr,
struct dso *dso, bool unwind_inlines, struct inline_node *node, struct symbol *sym)
{
int ret;
ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
if (ret > 0)
return ret;
ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
if (ret > 0)
return ret;
ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
if (ret > 0)
return ret;
return cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
}root@number:/home/acme# perf stat -e probe_perf:*_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 4K of event 'cpu/cycles/Pu'
# Event count (approx.): 5535180842
#
# Overhead Source File Source:Line
# ........ ............ ...............
#
99.04% inlineloop.c inlineloop.c:21
0.46% inlineloop.c inlineloop.c:20#
# (Tip: To set sampling period of individual events use perf record -e cpu/cpu-cycles,period=100001/,cpu/branches,period=10001/ ...) #
Performance counter stats for 'perf report -f --dso perf --stdio -s srcfile,srcline':
44 probe_perf:libdw_addr2line
23 probe_perf:llvm_addr2line
0 probe_perf:libbfd_addr2line
0 probe_perf:cmd_addr2line 0.036554827 seconds time elapsed
0.030570000 seconds user
0.006049000 seconds sysroot@number:/home/acme#
Maybe asking for the dso_name may help understand why it is falling back 23 times to the llvm based one...
root@number:/home/acme# perf probe -x ~/bin/perf llvm__addr2line dso_name:string Target program is compiled without optimization. Skipping prologue. Probe on address 0x5f73e4 to force probing at the function entry.
Added new event:
probe_perf:llvm_addr2line (on llvm__addr2line in /home/acme/bin/perf with dso_name:string)
You can now use it in all perf tools, such as:
perf record -e probe_perf:llvm_addr2line -aR sleep 1
root@number:/home/acme# perf trace --libtrace -e probe_perf:llvm_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
0.000 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
0.697 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
0.704 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
0.708 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
0.712 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
<SNIP>root@number:/home/acme# readelf -wi /root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf | head -30 Contents of the .debug_info section:
Compilation Unit @ offset 0:
Length: 0x20 (32-bit)
Version: 5
Unit Type: DW_UT_partial (3)
Abbrev Offset: 0xad87
Pointer Size: 8
<0><c>: Abbrev Number: 38 (DW_TAG_partial_unit)
<d> DW_AT_stmt_list : 0
<11> DW_AT_comp_dir : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
<1><15>: Abbrev Number: 57 (DW_TAG_base_type)
<16> DW_AT_byte_size : 8
<17> DW_AT_encoding : 7 (unsigned)
<18> DW_AT_name : (indirect string, offset: 0x9bf1): long unsigned int
<1><1c>: Abbrev Number: 57 (DW_TAG_base_type)
<1d> DW_AT_byte_size : 1
<1e> DW_AT_encoding : 6 (signed char)
<1f> DW_AT_name : (indirect string, offset: 0x7bab): char
<1><23>: Abbrev Number: 0
Compilation Unit @ offset 0x24:
Length: 0x20 (32-bit)
Version: 5
Unit Type: DW_UT_partial (3)
Abbrev Offset: 0xad87
Pointer Size: 8
<0><30>: Abbrev Number: 38 (DW_TAG_partial_unit)
<31> DW_AT_stmt_list : 0
<35> DW_AT_comp_dir : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf <1><39>: Abbrev Number: 57 (DW_TAG_base_type)
root@number:/home/acme#
Some partial unit glibc DWARF 5. I think we need some more libdw setup to handle those?
But then we can merge what James already reviewed and go on from here, right?
- Arnaldo
Quoted 363 lines
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/util/Build | 1 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 +++ tools/perf/util/libdw.c | 153 ++++++++++++++++++++++++++++++++++++++ tools/perf/util/libdw.h | 60 +++++++++++++++ tools/perf/util/srcline.c | 24 ++++++ tools/perf/util/srcline.h | 1 + 7 files changed, 252 insertions(+) create mode 100644 tools/perf/util/libdw.c create mode 100644 tools/perf/util/libdw.htools/perf/util/Build +1
-0@@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o perf-util-$(CONFIG_LIBDW) += debuginfo.o perf-util-$(CONFIG_LIBDW) += annotate-data.o +perf-util-$(CONFIG_LIBDW) += libdw.o perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o perf-util-$(CONFIG_LOCAL_LIBUNWIND) += unwind-libunwind-local.otools/perf/util/dso.c +2
-0@@ -32,6 +32,7 @@#include "string2.h" #include "vdso.h" #include "annotate-data.h" +#include "libdw.h" static const char *const debuglink_paths[] = {"%.0s%s" ,@@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)auxtrace_cache__free (RC_CHK_ACCESS (dso)->auxtrace_cache);dso_cache__free (dso);dso__free_a2l (dso); +dso__free_a2l_libdw (dso);dso__free_symsrc_filename (dso);nsinfo__zput (RC_CHK_ACCESS (dso)->nsinfo);mutex_destroy (dso__lock (dso));tools/perf/util/dso.h +11
-0@@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {const char *short_name;const char *long_name;void *a2l; +void *a2l_libdw;char *symsrc_filename;#if defined(__powerpc__)void *dwfl;/* DWARF debug info */ @@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)RC_CHK_ACCESS (dso)->a2l = val; } +static inline void *dso__a2l_libdw (const struct dso *dso) +{ +return RC_CHK_ACCESS (dso)->a2l_libdw; +} + +static inline void dso__set_a2l_libdw (struct dso *dso,void *val) +{ +RC_CHK_ACCESS (dso)->a2l_libdw = val; +} +static inline unsigned int dso__a2l_fails (const struct dso *dso) {return RC_CHK_ACCESS (dso)->a2l_fails;tools/perf/util/libdw.c new file +153
-0@@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "dso.h" +#include "libdw.h" +#include "srcline.h" +#include "symbol.h" +#include "dwarf-aux.h" +#include <fcntl.h> +#include <unistd.h> +#include <elfutils/libdwfl.h> + +void dso__free_a2l_libdw (struct dso *dso) +{ +Dwfl *dwfl =dso__a2l_libdw (dso); + +if (dwfl) { +dwfl_end (dwfl); +dso__set_a2l_libdw (dso,NULL ); + } +} + +struct libdw_a2l_cb_args { +struct dso *dso; +struct symbol *sym; +struct inline_node *node; +char *leaf_srcline; +bool leaf_srcline_used; +}; + +static int libdw_a2l_cb (Dwarf_Die *die,void *_args) +{ +struct libdw_a2l_cb_args *args = _args; +struct symbol *inline_sym =new_inline_sym (args->dso, args->sym,dwarf_diename (die)); +const char *call_fname =die_get_call_file (die); +char *call_srcline = srcline__unknown; +struct inline_list *ilist; + +if (!inline_sym) +return -ENOMEM ; + +/* Assign caller information to the parent. */ +if (call_fname) + call_srcline =srcline_from_fileline (call_fname,die_get_call_lineno (die)); + +list_for_each_entry (ilist, &args->node->val, list) { + ilist->srcline = call_srcline; + call_srcline =NULL ; +break ; + } +if (call_srcline && call_fname) +free (call_srcline); + +/* Add this symbol to the chain as the leaf. */ +inline_list__append_tail (inline_sym, args->leaf_srcline, args->node); + args->leaf_srcline_used = true; +return 0 ; +} + +int libdw__addr2line (const char *dso_name,u64 addr, +char **file,unsigned int *line_nr, +struct dso *dso,bool unwind_inlines, +struct inline_node *node,struct symbol *sym) +{ +static const Dwfl_Callbacks offline_callbacks = { + .find_debuginfo = dwfl_standard_find_debuginfo, + .section_address = dwfl_offline_section_address, + .find_elf = dwfl_build_id_find_elf, + }; +Dwfl *dwfl =dso__a2l_libdw (dso); +Dwfl_Module *mod; +Dwfl_Line *dwline; +Dwarf_Addr bias; +const char *src; +int lineno =0 ; + +if (!dwfl) { +/* +* Initialize Dwfl session. +* We need to open the DSO file to report it to libdw. +*/ +int fd; + + fd =open (dso_name,O_RDONLY ); +if (fd <0 ) +return 0 ; + + dwfl =dwfl_begin (&offline_callbacks); +if (!dwfl) { +close (fd); +return 0 ; + } + +/* +* If the report is successful, the file descriptor fd is consumed +* and closed by the Dwfl. If not, it is not closed. +*/ + mod =dwfl_report_offline (dwfl, dso_name, dso_name, fd); +if (!mod) { +dwfl_end (dwfl); +close (fd); +return 0 ; + } + +dwfl_report_end (dwfl,/*removed=*/ NULL ,/*arg=*/ NULL ); +dso__set_a2l_libdw (dso, dwfl); + }else { +/* Dwfl session already initialized, get module for address. */ + mod =dwfl_addrmodule (dwfl, addr); + } + +if (!mod) +return 0 ; + +/* +* Get/ignore the dwarf information. Determine the bias, difference +* between the regular ELF addr2line addresses and those to use with +* libdw. +*/ +if (!dwfl_module_getdwarf (mod, &bias)) +return 0 ; + +/* Find source line information for the address. */ + dwline =dwfl_module_getsrc (mod, addr + bias); +if (!dwline) +return 0 ; + +/* Get line information. */ + src =dwfl_lineinfo (dwline,/*addr=*/ NULL , &lineno,/*col=*/ NULL ,/*mtime=*/ NULL , +/*length=*/ NULL ); + +if (file) + *file = src ?strdup (src) :NULL ; +if (line_nr) + *line_nr = lineno; + +/* Optionally unwind inline function call chain. */ +if (unwind_inlines && node) { +Dwarf_Addr unused_bias; +Dwarf_Die *cudie =dwfl_module_addrdie (mod, addr + bias, &unused_bias); +struct libdw_a2l_cb_args args = { + .dso = dso, + .sym = sym, + .node = node, + .leaf_srcline =srcline_from_fileline (src ?:"<unknown>" , lineno), + }; + +/* Walk from the parent down to the leaf. */ +cu_walk_functions_at (cudie, addr, libdw_a2l_cb, &args); + +if (!args.leaf_srcline_used) +free (args.leaf_srcline); + } +return 1 ; +}tools/perf/util/libdw.h new file +60
-0@@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef PERF_LIBDW_H +#define PERF_LIBDW_H + +#include <linux/types.h> + +struct dso ; +struct inline_node ; +struct symbol ; + +#ifdef HAVE_LIBDW_SUPPORT +/* +* libdw__addr2line - Convert address to source location using libdw +* @dso_name: Name of the DSO +* @addr: Address to resolve +* @file: Pointer to return filename (caller must free) +* @line_nr: Pointer to return line number +* @dso: The dso struct +* @unwind_inlines: Whether to unwind inline function calls +* @node: Inline node list to append to +* @sym: The symbol associated with the address +* +* This function initializes a Dwfl context for the DSO if not already present, +* finds the source line information for the given address, and optionally +* resolves inline function call chains. +* +* Returns 1 on success (found), 0 on failure (not found). +*/ +int libdw__addr2line (const char *dso_name,u64 addr,char **file, +unsigned int *line_nr,struct dso *dso, +bool unwind_inlines,struct inline_node *node, +struct symbol *sym); + +/* +* dso__free_a2l_libdw - Free libdw resources associated with the DSO +* @dso: The dso to free resources for +* +* This function cleans up the Dwfl context used for addr2line lookups. +*/ +void dso__free_a2l_libdw (struct dso *dso); + +#else /* HAVE_LIBDW_SUPPORT */ + +static inline int libdw__addr2line (const char *dso_name__maybe_unused , +u64 addr__maybe_unused ,char **file__maybe_unused , +unsigned int *line_nr__maybe_unused , +struct dso *dso__maybe_unused , +bool unwind_inlines__maybe_unused , +struct inline_node *node__maybe_unused , +struct symbol *sym__maybe_unused ) +{ +return 0 ; +} + +static inline void dso__free_a2l_libdw (struct dso *dso__maybe_unused ) +{ +} +#endif /* HAVE_LIBDW_SUPPORT */ + +#endif /* PERF_LIBDW_H */ tools/perf/util/srcline.c +24
-0@@ -6,6 +6,7 @@#include "libbfd.h" #include "llvm.h" #include "symbol.h" +#include "libdw.h" #include <inttypes.h> #include <string.h> @@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_nodereturn 0 ; } +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node) +{ +struct inline_list *ilist; + + ilist =zalloc (sizeof (*ilist)); +if (ilist ==NULL ) +return -1 ; + + ilist->symbol = symbol; + ilist->srcline = srcline; + +if (callchain_param.order ==ORDER_CALLEE ) +list_add (&ilist->list, &node->val); +else +list_add_tail (&ilist->list, &node->val); + +return 0 ; +} +/* basename version that takes a const input string */ static const char *gnu_basename (const char *path) {@@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int * {int ret; + ret =libdw__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); +if (ret >0 ) +return ret; + ret =llvm__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);if (ret >0 )return ret;tools/perf/util/srcline.h +1
-0@@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);void inlines__tree_delete (struct rb_root_cached *tree);int inline_list__append (struct symbol *symbol,char *srcline,struct inline_node *node); +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node);char *srcline_from_fileline (const char *file,unsigned int line); struct symbol *new_inline_sym(struct dso *dso, struct symbol *base_sym,Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerRe: [PATCH v3 2/7] perf addr2line: Add a libdw implementation
Quoted 207 lines · Arnaldo Carvalho de Melo
Quoted 7 lines · Ian Rogers
Add an implementation of addr2line that uses libdw. Other addr2line implementations are slow, particularly in the case of forking addr2line. Add an implementation that caches the libdw information in the dso and uses it to find the file and line number information. Inline information is supported but because
cu_walk_functions_atvisits the leaf function last add ainline_list__append_tailto reverse the lists order.So while testing I moved the inlineloop 'perf test' workload to the front to test without this new libdw implementation and then with it, and tested using perf probe on it:
root@number:/home/acme# perf probe -x ~/bin/perf libdw__addr2line%return 'ret=$retval' Added new event: probe_perf:libdw_addr2line__return (on libdw__addr2line%return in /home/acme/bin/perf with ret=$retval)You can now use it in all perf tools, such as:
perf record -e probe_perf:libdw_addr2line__return -aR sleep 1
root@number:/home/acme# perf trace -e probe_perf:libdw_addr2line__return perf report -f --dso perf --stdio -s srcfile,srcline 0.000 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 0.007 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.680 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.685 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.689 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.692 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.696 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.699 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.702 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.705 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.709 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.712 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.715 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.718 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.721 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.724 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.727 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.729 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.732 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.734 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.737 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.740 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 0.771 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 13.563 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 13.766 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 14.099 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548) 14.103 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.105 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.134 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.143 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.242 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.244 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.245 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.246 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.248 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.250 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.251 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.252 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.258 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.260 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.265 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.267 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.268 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) 14.270 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1) # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 4K of event 'cpu/cycles/Pu' # Event count (approx.): 5535180842 # # Overhead Source File Source:Line # ........ ............ ............... # 99.04% inlineloop.c inlineloop.c:21 0.46% inlineloop.c inlineloop.c:20#
# (Tip: Save output of perf stat using: perf stat record <target workload>) #
root@number:/home/acme#Putting more probes to see the fallbacks...
root@number:/home/acme# perf probe -l probe_perf:cmd_addr2line (on cmd__addr2line@util/addr2line.c in /home/acme/bin/perf) probe_perf:libbfd_addr2line (on libbfd__addr2line@util/libbfd.h in /home/acme/bin/perf) probe_perf:libdw_addr2line (on libdw__addr2line@util/libdw.c in /home/acme/bin/perf) probe_perf:llvm_addr2line (on llvm__addr2line@util/llvm.c in /home/acme/bin/perf) probe_perf:libdw_addr2line__return (on libdw__addr2line%return@util/libdw.c in /home/acme/bin/perf with ret) root@number:/home/acme#As we have:
static int addr2line(const char
*dso_name, u64 addr, char **file, unsigned int*line_nr,
struct dso *dso, boolunwind_inlines, structinline_node*node, struct symbol *sym){ int ret; ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); if (ret > 0) return ret; ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); if (ret > 0) return ret; ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); if (ret > 0) return ret; return cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); }root@number:/home/acme# perf stat -e probe_perf:*_addr2line perf report -f --dso perf --stdio -s srcfile,srcline # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 4K of event 'cpu/cycles/Pu' # Event count (approx.): 5535180842 # # Overhead Source File Source:Line # ........ ............ ............... # 99.04% inlineloop.c inlineloop.c:21 0.46% inlineloop.c inlineloop.c:20#
# (Tip: To set sampling period of individual events use perf record -e cpu/cpu-cycles,period=100001/,cpu/branches,period=10001/ ...) #Performance counter stats for 'perf report -f --dso perf --stdio -s srcfile,srcline':
44 probe_perf:libdw_addr2line 23 probe_perf:llvm_addr2line 0 probe_perf:libbfd_addr2line 0 probe_perf:cmd_addr2line0.036554827 seconds time elapsed 0.030570000 seconds user 0.006049000 seconds sysroot@number:/home/acme#
Maybe asking for the
dso_namemay help understand why it is falling back 23 times to the llvm based one...root@number:/home/acme# perf probe -x ~/bin/perf llvm__addr2line dso_name:string Target program is compiled without optimization. Skipping prologue. Probe on address 0x5f73e4 to force probing at the function entry.Added new event:
probe_perf:llvm_addr2line (onllvm__addr2linein/home/acme/bin/perfwith dso_name:string)You can now use it in all perf tools, such as:
perf record -e probe_perf:llvm_addr2line -aR sleep 1
root@number:/home/acme# perf trace --libtrace -e probe_perf:llvm_addr2line perf report -f --dso perf --stdio -s srcfile,srcline 0.000 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf") 0.697 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf") 0.704 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf") 0.708 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf") 0.712 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf") <SNIP>root@number:/home/acme# readelf -wi /root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf | head -30 Contents of the .debug_info section:Compilation Unit @ offset 0:
Length: 0x20 (32-bit) Version: 5 Unit Type: DW_UT_partial (3) Abbrev Offset: 0xad87 Pointer Size: 8 <0><c>: Abbrev Number: 38 (DW_TAG_partial_unit) <d> DW_AT_stmt_list : 0 <11> DW_AT_comp_dir : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf <1><15>: Abbrev Number: 57 (DW_TAG_base_type) <16> DW_AT_byte_size : 8 <17> DW_AT_encoding : 7 (unsigned) <18> DW_AT_name : (indirect string, offset: 0x9bf1): long unsigned int <1><1c>: Abbrev Number: 57 (DW_TAG_base_type) <1d> DW_AT_byte_size : 1 <1e> DW_AT_encoding : 6 (signed char) <1f> DW_AT_name : (indirect string, offset: 0x7bab): char <1><23>: Abbrev Number: 0 Compilation Unit @ offset 0x24: Length: 0x20 (32-bit) Version: 5 Unit Type: DW_UT_partial (3) Abbrev Offset: 0xad87 Pointer Size: 8 <0><30>: Abbrev Number: 38 (DW_TAG_partial_unit) <31> DW_AT_stmt_list : 0 <35> DW_AT_comp_dir : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
Quoted 5 lines
<1><39>: Abbrev Number: 57 (
DW_TAG_base_type)
root@number:/home/acme#Some partial unit glibc DWARF 5. I think we need some more libdw setup to handle those?
Could be. Thanks for the analysis!
Quoted 2 lines
But then we can merge what James already reviewed and go on from here, right?
+Masami Hiramatsu
I believe so. I'm not aware of anything outstanding on the patches - v2 was using the function's declaration file:line rather than its calling file:line, but that's all resolved in v3 and I'm not aware of anything else left outstanding. Perhaps James can comment more on the (non-blocking) issues he faced with LLVM and perhaps Masami can comment on the libdw usage (I just checked that he did a lot of the dwarf-aux.c work).
Thanks,
Ian
Quoted 365 lines
- Arnaldo
Quoted 363 lines
Signed-off-by Ian Rogers <[email protected]>--- tools/perf/util/Build | 1 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 +++ tools/perf/util/libdw.c | 153 ++++++++++++++++++++++++++++++++++++++ tools/perf/util/libdw.h | 60 +++++++++++++++ tools/perf/util/srcline.c | 24 ++++++ tools/perf/util/srcline.h | 1 + 7 files changed, 252 insertions(+) create mode 100644 tools/perf/util/libdw.c create mode 100644 tools/perf/util/libdw.htools/perf/util/Build +1
-0@@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o perf-util-$(CONFIG_LIBDW) += debuginfo.o perf-util-$(CONFIG_LIBDW) += annotate-data.o +perf-util-$(CONFIG_LIBDW) += libdw.o perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o perf-util-$(CONFIG_LOCAL_LIBUNWIND) += unwind-libunwind-local.otools/perf/util/dso.c +2
-0@@ -32,6 +32,7 @@#include "string2.h" #include "vdso.h" #include "annotate-data.h" +#include "libdw.h" static const char *const debuglink_paths[] = {"%.0s%s" ,@@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)auxtrace_cache__free (RC_CHK_ACCESS (dso)->auxtrace_cache);dso_cache__free (dso);dso__free_a2l (dso); +dso__free_a2l_libdw (dso);dso__free_symsrc_filename (dso);nsinfo__zput (RC_CHK_ACCESS (dso)->nsinfo);mutex_destroy (dso__lock (dso));tools/perf/util/dso.h +11
-0@@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {const char *short_name;const char *long_name;void *a2l; +void *a2l_libdw;char *symsrc_filename;#if defined(__powerpc__)void *dwfl;/* DWARF debug info */ @@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)RC_CHK_ACCESS (dso)->a2l = val; } +static inline void *dso__a2l_libdw (const struct dso *dso) +{ +return RC_CHK_ACCESS (dso)->a2l_libdw; +} + +static inline void dso__set_a2l_libdw (struct dso *dso,void *val) +{ +RC_CHK_ACCESS (dso)->a2l_libdw = val; +} +static inline unsigned int dso__a2l_fails (const struct dso *dso) {return RC_CHK_ACCESS (dso)->a2l_fails;tools/perf/util/libdw.c new file +153
-0@@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "dso.h" +#include "libdw.h" +#include "srcline.h" +#include "symbol.h" +#include "dwarf-aux.h" +#include <fcntl.h> +#include <unistd.h> +#include <elfutils/libdwfl.h> + +void dso__free_a2l_libdw (struct dso *dso) +{ +Dwfl *dwfl =dso__a2l_libdw (dso); + +if (dwfl) { +dwfl_end (dwfl); +dso__set_a2l_libdw (dso,NULL ); + } +} + +struct libdw_a2l_cb_args { +struct dso *dso; +struct symbol *sym; +struct inline_node *node; +char *leaf_srcline; +bool leaf_srcline_used; +}; + +static int libdw_a2l_cb (Dwarf_Die *die,void *_args) +{ +struct libdw_a2l_cb_args *args = _args; +struct symbol *inline_sym =new_inline_sym (args->dso, args->sym,dwarf_diename (die)); +const char *call_fname =die_get_call_file (die); +char *call_srcline = srcline__unknown; +struct inline_list *ilist; + +if (!inline_sym) +return -ENOMEM ; + +/* Assign caller information to the parent. */ +if (call_fname) + call_srcline =srcline_from_fileline (call_fname,die_get_call_lineno (die)); + +list_for_each_entry (ilist, &args->node->val, list) { + ilist->srcline = call_srcline; + call_srcline =NULL ; +break ; + } +if (call_srcline && call_fname) +free (call_srcline); + +/* Add this symbol to the chain as the leaf. */ +inline_list__append_tail (inline_sym, args->leaf_srcline, args->node); + args->leaf_srcline_used = true; +return 0 ; +} + +int libdw__addr2line (const char *dso_name,u64 addr, +char **file,unsigned int *line_nr, +struct dso *dso,bool unwind_inlines, +struct inline_node *node,struct symbol *sym) +{ +static const Dwfl_Callbacks offline_callbacks = { + .find_debuginfo = dwfl_standard_find_debuginfo, + .section_address = dwfl_offline_section_address, + .find_elf = dwfl_build_id_find_elf, + }; +Dwfl *dwfl =dso__a2l_libdw (dso); +Dwfl_Module *mod; +Dwfl_Line *dwline; +Dwarf_Addr bias; +const char *src; +int lineno =0 ; + +if (!dwfl) { +/* +* Initialize Dwfl session. +* We need to open the DSO file to report it to libdw. +*/ +int fd; + + fd =open (dso_name,O_RDONLY ); +if (fd <0 ) +return 0 ; + + dwfl =dwfl_begin (&offline_callbacks); +if (!dwfl) { +close (fd); +return 0 ; + } + +/* +* If the report is successful, the file descriptor fd is consumed +* and closed by the Dwfl. If not, it is not closed. +*/ + mod =dwfl_report_offline (dwfl, dso_name, dso_name, fd); +if (!mod) { +dwfl_end (dwfl); +close (fd); +return 0 ; + } + +dwfl_report_end (dwfl,/*removed=*/ NULL ,/*arg=*/ NULL ); +dso__set_a2l_libdw (dso, dwfl); + }else { +/* Dwfl session already initialized, get module for address. */ + mod =dwfl_addrmodule (dwfl, addr); + } + +if (!mod) +return 0 ; + +/* +* Get/ignore the dwarf information. Determine the bias, difference +* between the regular ELF addr2line addresses and those to use with +* libdw. +*/ +if (!dwfl_module_getdwarf (mod, &bias)) +return 0 ; + +/* Find source line information for the address. */ + dwline =dwfl_module_getsrc (mod, addr + bias); +if (!dwline) +return 0 ; + +/* Get line information. */ + src =dwfl_lineinfo (dwline,/*addr=*/ NULL , &lineno,/*col=*/ NULL ,/*mtime=*/ NULL , +/*length=*/ NULL ); + +if (file) + *file = src ?strdup (src) :NULL ; +if (line_nr) + *line_nr = lineno; + +/* Optionally unwind inline function call chain. */ +if (unwind_inlines && node) { +Dwarf_Addr unused_bias; +Dwarf_Die *cudie =dwfl_module_addrdie (mod, addr + bias, &unused_bias); +struct libdw_a2l_cb_args args = { + .dso = dso, + .sym = sym, + .node = node, + .leaf_srcline =srcline_from_fileline (src ?:"<unknown>" , lineno), + }; + +/* Walk from the parent down to the leaf. */ +cu_walk_functions_at (cudie, addr, libdw_a2l_cb, &args); + +if (!args.leaf_srcline_used) +free (args.leaf_srcline); + } +return 1 ; +}tools/perf/util/libdw.h new file +60
-0@@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef PERF_LIBDW_H +#define PERF_LIBDW_H + +#include <linux/types.h> + +struct dso ; +struct inline_node ; +struct symbol ; + +#ifdef HAVE_LIBDW_SUPPORT +/* +* libdw__addr2line - Convert address to source location using libdw +* @dso_name: Name of the DSO +* @addr: Address to resolve +* @file: Pointer to return filename (caller must free) +* @line_nr: Pointer to return line number +* @dso: The dso struct +* @unwind_inlines: Whether to unwind inline function calls +* @node: Inline node list to append to +* @sym: The symbol associated with the address +* +* This function initializes a Dwfl context for the DSO if not already present, +* finds the source line information for the given address, and optionally +* resolves inline function call chains. +* +* Returns 1 on success (found), 0 on failure (not found). +*/ +int libdw__addr2line (const char *dso_name,u64 addr,char **file, +unsigned int *line_nr,struct dso *dso, +bool unwind_inlines,struct inline_node *node, +struct symbol *sym); + +/* +* dso__free_a2l_libdw - Free libdw resources associated with the DSO +* @dso: The dso to free resources for +* +* This function cleans up the Dwfl context used for addr2line lookups. +*/ +void dso__free_a2l_libdw (struct dso *dso); + +#else /* HAVE_LIBDW_SUPPORT */ + +static inline int libdw__addr2line (const char *dso_name__maybe_unused , +u64 addr__maybe_unused ,char **file__maybe_unused , +unsigned int *line_nr__maybe_unused , +struct dso *dso__maybe_unused , +bool unwind_inlines__maybe_unused , +struct inline_node *node__maybe_unused , +struct symbol *sym__maybe_unused ) +{ +return 0 ; +} + +static inline void dso__free_a2l_libdw (struct dso *dso__maybe_unused ) +{ +} +#endif /* HAVE_LIBDW_SUPPORT */ + +#endif /* PERF_LIBDW_H */ tools/perf/util/srcline.c +24
-0@@ -6,6 +6,7 @@#include "libbfd.h" #include "llvm.h" #include "symbol.h" +#include "libdw.h" #include <inttypes.h> #include <string.h> @@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_nodereturn 0 ; } +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node) +{ +struct inline_list *ilist; + + ilist =zalloc (sizeof (*ilist)); +if (ilist ==NULL ) +return -1 ; + + ilist->symbol = symbol; + ilist->srcline = srcline; + +if (callchain_param.order ==ORDER_CALLEE ) +list_add (&ilist->list, &node->val); +else +list_add_tail (&ilist->list, &node->val); + +return 0 ; +} +/* basename version that takes a const input string */ static const char *gnu_basename (const char *path) {@@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int * {int ret; + ret =libdw__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym); +if (ret >0 ) +return ret; + ret =llvm__addr2line (dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);if (ret >0 )return ret;tools/perf/util/srcline.h +1
-1@@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);void inlines__tree_delete (struct rb_root_cached *tree);int inline_list__append (struct symbol *symbol,char *srcline,struct inline_node *node); +int inline_list__append_tail (struct symbol *symbol,char *srcline,struct inline_node *node);char *srcline_from_fileline (const char *file,unsigned int line); struct symbol *new_inline_sym(struct dso *dso, struct symbol *base_sym, --2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 3/7 perf addr2line.c: Rename a2l_style to cmd_a2l_style
In reply to Ian RogersThe a2l_style is only relevant to the command line version, so rename to make this clearer.
--- tools/perf/util/addr2line.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
tools/perf/util/addr2line.c +10 -10
@@ -90,16 +90,16 @@ static struct child_process *addr2line_subprocess_init(const char *addr2line_patreturn a2l; } -enum a2l_style { +enum cmd_a2l_style {BROKEN ,GNU_BINUTILS ,LLVM , }; -static enum a2l_style addr2line_configure (struct child_process *a2l,const char *dso_name) +static enum cmd_a2l_style cmd_addr2line_configure (struct child_process *a2l,const char *dso_name) {static bool cached; -static enum a2l_style style; +static enum cmd_a2l_style style;if (!cached) {char buf[128 ];
@@ -149,7 +149,7 @@ static enum a2l_style addr2line_configure(struct child_process *a2l, const char } static int read_addr2line_record(struct io *io, - enum a2l_style style, + enum cmd_a2l_style style, const char *dso_name, u64 addr, bool first,
@@ -298,7 +298,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,char buf[128 ];ssize_t written;struct io io = { .eof = false }; -enum a2l_style a2l_style; +enum cmd_a2l_style cmd_a2l_style;if (!a2l) {if (!filename__has_section (dso_name,".debug_line" ))
@@ -314,8 +314,8 @@ int cmd__addr2line(const char *dso_name, u64 addr,pr_warning ("%s %s: addr2line_subprocess_init failed\n" , __func__, dso_name); goto out; } - a2l_style =addr2line_configure (a2l, dso_name); -if (a2l_style ==BROKEN ) + cmd_a2l_style =cmd_addr2line_configure (a2l, dso_name); +if (cmd_a2l_style ==BROKEN ) goto out;/*
@@ -336,7 +336,7 @@ int cmd__addr2line(const char *dso_name, u64 addr, }io__init (&io, a2l->out, buf,sizeof (buf)); io.timeout_ms = addr2line_timeout_ms; -switch (read_addr2line_record (&io, a2l_style, dso_name, addr,/*first=*/ true, +switch (read_addr2line_record (&io, cmd_a2l_style, dso_name, addr,/*first=*/ true, &record_function, &record_filename, &record_line_nr)) {case -1 :if (!symbol_conf.disable_add2line_warn)
@@ -351,7 +351,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,* binutils, also force a non-zero address as we're no longer * reading that record. */ -switch (read_addr2line_record (&io, a2l_style, dso_name, +switch (read_addr2line_record (&io, cmd_a2l_style, dso_name,/*addr=*/ 1 ,/*first=*/ true,NULL ,NULL ,NULL )) {case -1 :
@@ -397,7 +397,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,* as we're reading records beyond the first. */ while ((record_status = read_addr2line_record(&io, - a2l_style, + cmd_a2l_style, dso_name, /*addr=*/1, /*first=*/false,
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 4/7 perf srcline: Add configuration support for the addr2line style
In reply to Ian RogersAllow the addr2line style to be specified on the perf report command line or in the .perfconfig file.
--- tools/perf/builtin-report.c | 10 ++++ tools/perf/util/config.c | 4 ++ tools/perf/util/srcline.c | 98 +++++++++++++++++++++++++++++++---- tools/perf/util/srcline.h | 2 + tools/perf/util/symbol_conf.h | 10 ++++ 5 files changed, 113 insertions(+), 11 deletions(-)
tools/perf/builtin-report.c +10 -0
@@ -1271,6 +1271,13 @@ parse_percent_limit(const struct option *opt, const char *str,return 0 ; } +static int +report_parse_addr2line_config (const struct option *opt__maybe_unused , +const char *arg,int unset__maybe_unused ) +{ +return addr2line_configure ("addr2line.style" , arg,NULL ); +} +static int process_attr (const struct perf_tool *tool__maybe_unused ,union perf_event *event,struct evlist **pevlist)
@@ -1447,6 +1454,9 @@ int cmd_report(int argc, const char **argv) "objdump binary to use for disassembly and annotations"),OPT_STRING (0 ,"addr2line" , &addr2line_path,"path" ,"addr2line binary to use for line numbers" ), +OPT_CALLBACK (0 ,"addr2line-style" ,NULL ,"addr2line style" , +"addr2line styles (libdw,llvm,libbfd,addr2line)" , + report_parse_addr2line_config),OPT_BOOLEAN (0 ,"demangle" , &symbol_conf.demangle,"Symbol demangling. Enabled by default, use --no-demangle to disable." ), OPT_BOOLEAN(0, "demangle-kernel", &symbol_conf.demangle_kernel,
tools/perf/util/config.c +4 -0
@@ -20,6 +20,7 @@#include "util/stat.h" /* perf_stat__set_big_num */ #include "util/evsel.h" /* evsel__hw_names, evsel__use_bpf_counters */ #include "util/addr2line.h" /* addr2line_timeout_ms */ +#include "srcline.h" #include "build-id.h" #include "debug.h" #include "config.h"
@@ -519,6 +520,9 @@ int perf_default_config(const char *var, const char *value,if (strstarts (var,"stat." ))return perf_stat_config (var, value); +if (strstarts (var,"addr2line." )) +return addr2line_configure (var, value, dummy); +/* Add other config variables here. */ return 0 ; }
tools/perf/util/srcline.c +87 -11
117 lines not on this page · raw message
tools/perf/util/srcline.h +2 -0
@@ -63,4 +63,6 @@ struct symbol *new_inline_sym(struct dso *dso,struct symbol *base_sym, const char *funcname); +int addr2line_configure (const char *var,const char *value,void *cb); +#endif /* PERF_SRCLINE_H */
tools/perf/util/symbol_conf.h +10 -0
@@ -9,6 +9,15 @@struct strlist ;struct intlist ; +enum a2l_style { +A2L_STYLE_UNKNOWN =0 , +A2L_STYLE_LIBDW , +A2L_STYLE_LLVM , +A2L_STYLE_LIBBFD , +A2L_STYLE_CMD , +}; +#define MAX_A2L_STYLE (A2L_STYLE_CMD + 1) +struct symbol_conf {bool nanosecs;unsigned short priv_size;
@@ -70,6 +79,7 @@ struct symbol_conf { *col_width_list_str, *bt_stop_list_str;const char *addr2line_path; +enum a2l_style addr2line_style[MAX_A2L_STYLE ];unsigned long time_quantum;struct strlist *dso_list, *comm_list,
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 5/7 perf callchain: Fix srcline printing with inlines
In reply to Ian Rogerssample__fprintf_callchain was using map__fprintf_srcline which won't report inline line numbers. Fix by using the srcline from the callchain and falling back to the map variant.
--- tools/perf/util/evsel_fprintf.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tools/perf/util/evsel_fprintf.c +6 -2
@@ -185,8 +185,12 @@ int sample__fprintf_callchain(struct perf_sample *sample, int left_alignment,if (print_dso && (!sym || !sym->inlined)) printed +=map__fprintf_dsoname_dsoff (map, print_dsoff, addr, fp); -if (print_srcline) - printed +=map__fprintf_srcline (map, addr,"\n " , fp); +if (print_srcline) { +if (node->srcline) + printed +=fprintf (fp,"\n %s" , node->srcline); +else + printed +=map__fprintf_srcline (map, addr,"\n " , fp); + }if (sym && sym->inlined) printed +=fprintf (fp," (inlined)" );
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 6/7 perf test workload: Add inlineloop test workload
In reply to Ian RogersThe purpose of this workload is to gather samples in an inlined function. This can be used to test whether inlined addr2line works correctly.
--- tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++++++++++++++++++++ 4 files changed, 56 insertions(+) create mode 100644 tools/perf/tests/workloads/inlineloop.c
tools/perf/tests/builtin-test.c +1 -0
@@ -153,6 +153,7 @@ static struct test_workload *workloads[] = { &workload__datasym, &workload__landlock, &workload__traploop, + &workload__inlineloop, };#define workloads__for_each (workload) \
tools/perf/tests/tests.h +1 -0
@@ -240,6 +240,7 @@ DECLARE_WORKLOAD(brstack);DECLARE_WORKLOAD (datasym);DECLARE_WORKLOAD (landlock);DECLARE_WORKLOAD (traploop); +DECLARE_WORKLOAD (inlineloop);extern const char *dso_to_test;extern const char *test_objdump_path;
tools/perf/tests/workloads/Build +2 -0
@@ -8,9 +8,11 @@ perf-test-y += brstack.o perf-test-y += datasym.o perf-test-y += landlock.o perf-test-y += traploop.o +perf-test-y += inlineloop.o CFLAGS_sqrtloop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_leafloop.o = -g -O0 -fno-inline -fno-omit-frame-pointer -U_FORTIFY_SOURCE CFLAGS_brstack.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_datasym.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_traploop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE +CFLAGS_inlineloop.o = -g -O2
tools/perf/tests/workloads/inlineloop.c new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Ian Rogers
reviewerPATCH v3 7/7 perf test: Test addr2line unwinding works with inline functions
In reply to Ian RogersAdd a test that seeks to see inline functions correctly displayed in perf script from the inlineloop workload.
--- tools/perf/tests/shell/addr2line_inlines.sh | 47 +++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
tools/perf/tests/shell/addr2line_inlines.sh new file +47 -0
@@ -0,0 +1,47 @@ +#!/bin/bash +# test addr2line inline unwinding +# SPDX-License-Identifier: GPL-2.0 + +set -e + +err=0 +test_dir=$(mktemp -d /tmp/perf-test-inline-addr2line.XXXXXXXXXX) +perf_data="${test_dir}/perf.data" +perf_script_txt="${test_dir}/perf_script.txt" + +cleanup() { + rm -rf "${test_dir}" + trap - EXIT TERM INT +} + +trap_cleanup() { + echo "Unexpected signal in ${FUNCNAME[1]}" + cleanup + exit 1 +} +trap trap_cleanup EXIT TERM INT + +test_inlinedloop() { + echo "Inline unwinding verification test" + # Record data. Currently only dwarf callchains support inlined functions. + perf record --call-graph dwarf -e task-clock:u -o "${perf_data}" -- perf test -w inlineloop 1 + + # Check output with inline (default) and srcline + perf script -i "${perf_data}" --fields +srcline > "${perf_script_txt}" + + # Expect the leaf and middle functions to occur on lines in the 20s, with + # the non-inlined parent function on a line in the 30s. + if grep -q "inlineloop.c:2. (inlined)" "${perf_script_txt}" && + grep -q "inlineloop.c:3.$" "${perf_script_txt}" + then + echo "Inline unwinding verification test [Success]" + else + echo "Inline unwinding verification test [Failed missing inlined functions]" + err=1 + fi +} + +test_inlinedloop + +cleanup +exit $err
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Quoted 88 lines · Ian Rogers
addr2line is a performance bottleneck in perf, add a libdw based implementation that avoids forking addr2line and caches the decoded debug information.
Allow the addr2line implementation to be picked via the configuration file or --addr2line-style with
perf report.Test/fix that inline callchains are properly displayed by perf script.
An example:
``` $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1 [ perf record: Woken up 132 times to write data ] [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ] $ perf script --fields +srcline ... perf-inlineloop 1814670 293100.228871: 640004 cpu_core/cycles/u: 55a11d6e61ee leaf+0x2e inlineloop.c:21 (inlined) 55a11d6e61ee middle+0x2e inlineloop.c:27 (inlined) 55a11d6e61ee parent+0x2e (perf) inlineloop.c:32 55a11d6e629b inlineloop+0x8b (perf) inlineloop.c:47 55a11d69a3bc run_workload+0x5a (perf) builtin-test.c:715 55a11d69aa9f cmd_test+0x417 (perf) builtin-test.c:825 55a11d6155f5 run_builtin+0xd4 (perf) perf.c:349 55a11d61588d handle_internal_command+0xdd (perf) perf.c:401 55a11d6159e6 run_argv+0x35 (perf) perf.c:445 55a11d615d2f main+0x2cb (perf) perf.c:553 7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6) libc_start_call_main.h:58 7fae3d233d64 __libc_start_main_impl+0x84 libc-start.c:360 (inlined) 55a11d565f80 _start+0x20 (perf) ??:0 ... ```v3: Make the caller inline file and line number accurate in the libdw
addr2line, rather than using the function's declared location. Fix reference counts in unwind-libdw. Add fixes tag for srcline inline printing.v2: Fix bias issue with libdwfl functions. Use
cu_walk_functions_at
from perf's dwarf-aux to fully walk inline functions. Add testing that inlined functions are shown in the perf script srcline callchain information. Add configurability as to which addr2line style to use.
https://lore.kernel.org/lkml/[email protected]/v1: https://lore.kernel.org/lkml/[email protected]/
Ian Rogers (7):
perf unwind-libdw: Fix invalid reference counts
perf addr2line: Add a libdw implementation
perfaddr2line.c: Renamea2l_styletocmd_a2l_styleperf srcline: Add configuration support for the addr2line style perf callchain: Fix srcline printing with inlines perf test workload: Add inlineloop test workload perf test: Test addr2line unwinding works with inline functionstools/perf/builtin-report.c | 10 ++ tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/addr2line_inlines.sh | 47 ++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++ tools/perf/util/Build | 1 + tools/perf/util/addr2line.c | 20 +-- tools/perf/util/config.c | 4 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 ++ tools/perf/util/evsel_fprintf.c | 8 +- tools/perf/util/libdw.c | 153 ++++++++++++++++++++ tools/perf/util/libdw.h | 60 ++++++++ tools/perf/util/srcline.c | 116 ++++++++++++++- tools/perf/util/srcline.h | 3 + tools/perf/util/symbol_conf.h | 10 ++ tools/perf/util/unwind-libdw.c | 7 +- 18 files changed, 486 insertions(+), 22 deletions(-)
Quoted 5 lines
create mode 100755
tools/perf/tests/shell/addr2line_inlines.shcreate mode 100644tools/perf/tests/workloads/inlineloop.ccreate mode 100644tools/perf/util/libdw.c
create mode 100644tools/perf/util/libdw.h
I don't see the differences to the other addr2line implementations anymore, but only because it falls through to the old ones when libdw fails now.
For example when building Perf with LLVM it can't get the line in the inlineloop workload, and there's still a few things in libc and other system libraries it fails on.
But I think it's fine because it doesn't give the wrong line anymore, it just falls through to another working addr2line implementation.
Ian Rogers
reviewerQuoted 96 lines · James Clark
Quoted 88 lines · Ian Rogers
addr2line is a performance bottleneck in perf, add a libdw based implementation that avoids forking addr2line and caches the decoded debug information.
Allow the addr2line implementation to be picked via the configuration file or --addr2line-style with
perf report.Test/fix that inline callchains are properly displayed by perf script.
An example:
``` $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1 [ perf record: Woken up 132 times to write data ] [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ] $ perf script --fields +srcline ... perf-inlineloop 1814670 293100.228871: 640004 cpu_core/cycles/u: 55a11d6e61ee leaf+0x2e inlineloop.c:21 (inlined) 55a11d6e61ee middle+0x2e inlineloop.c:27 (inlined) 55a11d6e61ee parent+0x2e (perf) inlineloop.c:32 55a11d6e629b inlineloop+0x8b (perf) inlineloop.c:47 55a11d69a3bc run_workload+0x5a (perf) builtin-test.c:715 55a11d69aa9f cmd_test+0x417 (perf) builtin-test.c:825 55a11d6155f5 run_builtin+0xd4 (perf) perf.c:349 55a11d61588d handle_internal_command+0xdd (perf) perf.c:401 55a11d6159e6 run_argv+0x35 (perf) perf.c:445 55a11d615d2f main+0x2cb (perf) perf.c:553 7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6) libc_start_call_main.h:58 7fae3d233d64 __libc_start_main_impl+0x84 libc-start.c:360 (inlined) 55a11d565f80 _start+0x20 (perf) ??:0 ... ```v3: Make the caller inline file and line number accurate in the libdw
addr2line, rather than using the function's declared location. Fix reference counts in unwind-libdw. Add fixes tag for srcline inline printing.v2: Fix bias issue with libdwfl functions. Use
cu_walk_functions_at
from perf's dwarf-aux to fully walk inline functions. Add testing that inlined functions are shown in the perf script srcline callchain information. Add configurability as to which addr2line style to use.
https://lore.kernel.org/lkml/[email protected]/v1: https://lore.kernel.org/lkml/[email protected]/
Ian Rogers (7):
perf unwind-libdw: Fix invalid reference counts
perf addr2line: Add a libdw implementation
perfaddr2line.c: Renamea2l_styletocmd_a2l_styleperf srcline: Add configuration support for the addr2line style perf callchain: Fix srcline printing with inlines perf test workload: Add inlineloop test workload perf test: Test addr2line unwinding works with inline functionstools/perf/builtin-report.c | 10 ++ tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/addr2line_inlines.sh | 47 ++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++ tools/perf/util/Build | 1 + tools/perf/util/addr2line.c | 20 +-- tools/perf/util/config.c | 4 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 ++ tools/perf/util/evsel_fprintf.c | 8 +- tools/perf/util/libdw.c | 153 ++++++++++++++++++++ tools/perf/util/libdw.h | 60 ++++++++ tools/perf/util/srcline.c | 116 ++++++++++++++- tools/perf/util/srcline.h | 3 + tools/perf/util/symbol_conf.h | 10 ++ tools/perf/util/unwind-libdw.c | 7 +- 18 files changed, 486 insertions(+), 22 deletions(-)Quoted 5 lines
create mode 100755
tools/perf/tests/shell/addr2line_inlines.shcreate mode 100644tools/perf/tests/workloads/inlineloop.ccreate mode 100644tools/perf/util/libdw.c
create mode 100644tools/perf/util/libdw.h
Quoted 7 lines
I don't see the differences to the other addr2line implementations anymore, but only because it falls through to the old ones when libdw fails now.
For example when building Perf with LLVM it can't get the line in the inlineloop workload, and there's still a few things in libc and other system libraries it fails on.
Hmm.. I wonder what the issue is. I was looking at the dwarf output from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are doing something to confuse libdw? I'll try to investigate. There are quite a few levels of libdw: there's the raw libdw, libdwfl (frontend to libdw) that does the parsing and tries to give things like nested debug scopes (libdwfl is the one needing addresses with a module bias rather than raw file offsets), and then there is the dwarf-aux.c that is in perf and is used by things like probe finding (I believe this doesn't need biases addresses). Anyway, with the biases there are things I can screw up (like in the v1 patch) but maybe the LLVM issue is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific :-/
Quoted 2 lines
But I think it's fine because it doesn't give the wrong line anymore, it just falls through to another working addr2line implementation.
Just to confirm that with gcc builds it isn't failing now? ie it isn't just an addr2line implementation that falls through all the time? I was seeing things working/testing on x86 with gcc.
Quoted 1 line
Reviewed-by James Clark <[email protected]>
Thanks,
Ian
Ian Rogers
reviewerQuoted 106 lines · Ian Rogers
Quoted 96 lines · James Clark
Quoted 88 lines · Ian Rogers
addr2line is a performance bottleneck in perf, add a libdw based implementation that avoids forking addr2line and caches the decoded debug information.
Allow the addr2line implementation to be picked via the configuration file or --addr2line-style with
perf report.Test/fix that inline callchains are properly displayed by perf script.
An example:
``` $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1 [ perf record: Woken up 132 times to write data ] [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ] $ perf script --fields +srcline ... perf-inlineloop 1814670 293100.228871: 640004 cpu_core/cycles/u: 55a11d6e61ee leaf+0x2e inlineloop.c:21 (inlined) 55a11d6e61ee middle+0x2e inlineloop.c:27 (inlined) 55a11d6e61ee parent+0x2e (perf) inlineloop.c:32 55a11d6e629b inlineloop+0x8b (perf) inlineloop.c:47 55a11d69a3bc run_workload+0x5a (perf) builtin-test.c:715 55a11d69aa9f cmd_test+0x417 (perf) builtin-test.c:825 55a11d6155f5 run_builtin+0xd4 (perf) perf.c:349 55a11d61588d handle_internal_command+0xdd (perf) perf.c:401 55a11d6159e6 run_argv+0x35 (perf) perf.c:445 55a11d615d2f main+0x2cb (perf) perf.c:553 7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6) libc_start_call_main.h:58 7fae3d233d64 __libc_start_main_impl+0x84 libc-start.c:360 (inlined) 55a11d565f80 _start+0x20 (perf) ??:0 ... ```v3: Make the caller inline file and line number accurate in the libdw
addr2line, rather than using the function's declared location. Fix reference counts in unwind-libdw. Add fixes tag for srcline inline printing.v2: Fix bias issue with libdwfl functions. Use
cu_walk_functions_at
from perf's dwarf-aux to fully walk inline functions. Add testing that inlined functions are shown in the perf script srcline callchain information. Add configurability as to which addr2line style to use.
https://lore.kernel.org/lkml/[email protected]/v1: https://lore.kernel.org/lkml/[email protected]/
Ian Rogers (7):
perf unwind-libdw: Fix invalid reference counts
perf addr2line: Add a libdw implementation
perfaddr2line.c: Renamea2l_styletocmd_a2l_styleperf srcline: Add configuration support for the addr2line style perf callchain: Fix srcline printing with inlines perf test workload: Add inlineloop test workload perf test: Test addr2line unwinding works with inline functionstools/perf/builtin-report.c | 10 ++ tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/addr2line_inlines.sh | 47 ++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++ tools/perf/util/Build | 1 + tools/perf/util/addr2line.c | 20 +-- tools/perf/util/config.c | 4 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 ++ tools/perf/util/evsel_fprintf.c | 8 +- tools/perf/util/libdw.c | 153 ++++++++++++++++++++ tools/perf/util/libdw.h | 60 ++++++++ tools/perf/util/srcline.c | 116 ++++++++++++++- tools/perf/util/srcline.h | 3 + tools/perf/util/symbol_conf.h | 10 ++ tools/perf/util/unwind-libdw.c | 7 +- 18 files changed, 486 insertions(+), 22 deletions(-)Quoted 5 lines
create mode 100755
tools/perf/tests/shell/addr2line_inlines.shcreate mode 100644tools/perf/tests/workloads/inlineloop.ccreate mode 100644tools/perf/util/libdw.c
create mode 100644tools/perf/util/libdw.hQuoted 7 lines
I don't see the differences to the other addr2line implementations anymore, but only because it falls through to the old ones when libdw fails now.
For example when building Perf with LLVM it can't get the line in the inlineloop workload, and there's still a few things in libc and other system libraries it fails on.
Quoted 12 lines
Hmm.. I wonder what the issue is. I was looking at the dwarf output from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are doing something to confuse libdw? I'll try to investigate. There are quite a few levels of libdw: there's the raw libdw, libdwfl (frontend to libdw) that does the parsing and tries to give things like nested debug scopes (libdwfl is the one needing addresses with a module bias rather than raw file offsets), and then there is the
dwarf-aux.cthat is in perf and is used by things like probe finding (I believe this doesn't need biases addresses). Anyway, with the biases there are things I can screw up (like in the v1 patch) but maybe the LLVM issue is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific :-/
Testing with clang/llvm on x86-64 (dwarf5):
```
$ make -C tools/perf O=/tmp/perf DEBUG=1 CC=clang CXX=clang++
HOSTCC=clang clean all
...
$ llvm-dwarfdump /tmp/perf/perf
...
0x0014f852: Compile Unit: length = 0x00000294, format = DWARF32,
version = 0x0005, unit_type = DW_UT_compile,
abbr_offset = 0x1879a, addr_size = 0x08 (next unit at 0x0014faea)
0x0014f85e: DW_TAG_compile_unit
DW_AT_producer ("Debian clang version 19.1.7 (3+build5)")
DW_AT_language (DW_LANG_C11)
DW_AT_name ("tests/workloads/inlineloop.c")
DW_AT_str_offsets_base (0x0004a550)
DW_AT_stmt_list (0x0008c3f2)
DW_AT_comp_dir ("linux/tools/perf")
DW_AT_low_pc (0x00000000001e61c0)
DW_AT_high_pc (0x00000000001e62e9)
DW_AT_addr_base (0x00022248)
DW_AT_loclists_base (0x0000018a)
...
$ sudo /tmp/perf/perf record --call-graph dwarf -e cycles:u --
/tmp/perf/perf test -w inlineloop 1
...
$ sudo /tmp/perf/perf script --fields +srcline
...
perf-inlineloop 2284167 423038.015394: 569917 cpu_core/cycles/u:
56390020d2c6 leaf+0x26
inlineloop.c:21 (inlined)
56390020d2c6 middle+0x26
inlineloop.c:27 (inlined)
56390020d2c6 parent+0x26 (/tmp/perf/perf)
...
```I ran inside of gdb and confirmed that the libdw code is creating the inlined information (breakpoint on libdw_a2l_cb, etc.). So I'm not able to reproduce the LLVM issue for now on x86-64.
Thanks,
Ian
Quoted 11 lines
Quoted 2 lines
But I think it's fine because it doesn't give the wrong line anymore, it just falls through to another working addr2line implementation.
Just to confirm that with gcc builds it isn't failing now? ie it isn't just an addr2line implementation that falls through all the time? I was seeing things working/testing on x86 with gcc.
Quoted 1 line
Reviewed-by James Clark <[email protected]>Thanks,
Ian
James Clark
reviewerQuoted 103 lines · Ian Rogers
Quoted 102 lines · Ian Rogers
Quoted 100 lines · James Clark
Quoted 88 lines · Ian Rogers
addr2line is a performance bottleneck in perf, add a libdw based implementation that avoids forking addr2line and caches the decoded debug information.
Allow the addr2line implementation to be picked via the configuration file or --addr2line-style with
perf report.Test/fix that inline callchains are properly displayed by perf script.
An example:
``` $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1 [ perf record: Woken up 132 times to write data ] [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ] $ perf script --fields +srcline ... perf-inlineloop 1814670 293100.228871: 640004 cpu_core/cycles/u: 55a11d6e61ee leaf+0x2e inlineloop.c:21 (inlined) 55a11d6e61ee middle+0x2e inlineloop.c:27 (inlined) 55a11d6e61ee parent+0x2e (perf) inlineloop.c:32 55a11d6e629b inlineloop+0x8b (perf) inlineloop.c:47 55a11d69a3bc run_workload+0x5a (perf) builtin-test.c:715 55a11d69aa9f cmd_test+0x417 (perf) builtin-test.c:825 55a11d6155f5 run_builtin+0xd4 (perf) perf.c:349 55a11d61588d handle_internal_command+0xdd (perf) perf.c:401 55a11d6159e6 run_argv+0x35 (perf) perf.c:445 55a11d615d2f main+0x2cb (perf) perf.c:553 7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6) libc_start_call_main.h:58 7fae3d233d64 __libc_start_main_impl+0x84 libc-start.c:360 (inlined) 55a11d565f80 _start+0x20 (perf) ??:0 ... ```v3: Make the caller inline file and line number accurate in the libdw
addr2line, rather than using the function's declared location. Fix reference counts in unwind-libdw. Add fixes tag for srcline inline printing.v2: Fix bias issue with libdwfl functions. Use
cu_walk_functions_at
from perf's dwarf-aux to fully walk inline functions. Add testing that inlined functions are shown in the perf script srcline callchain information. Add configurability as to which addr2line style to use.
https://lore.kernel.org/lkml/[email protected]/v1: https://lore.kernel.org/lkml/[email protected]/
Ian Rogers (7):
perf unwind-libdw: Fix invalid reference counts
perf addr2line: Add a libdw implementation
perfaddr2line.c: Renamea2l_styletocmd_a2l_styleperf srcline: Add configuration support for the addr2line style perf callchain: Fix srcline printing with inlines perf test workload: Add inlineloop test workload perf test: Test addr2line unwinding works with inline functionstools/perf/builtin-report.c | 10 ++ tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/addr2line_inlines.sh | 47 ++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/inlineloop.c | 52 +++++++ tools/perf/util/Build | 1 + tools/perf/util/addr2line.c | 20 +-- tools/perf/util/config.c | 4 + tools/perf/util/dso.c | 2 + tools/perf/util/dso.h | 11 ++ tools/perf/util/evsel_fprintf.c | 8 +- tools/perf/util/libdw.c | 153 ++++++++++++++++++++ tools/perf/util/libdw.h | 60 ++++++++ tools/perf/util/srcline.c | 116 ++++++++++++++- tools/perf/util/srcline.h | 3 + tools/perf/util/symbol_conf.h | 10 ++ tools/perf/util/unwind-libdw.c | 7 +- 18 files changed, 486 insertions(+), 22 deletions(-)Quoted 5 lines
create mode 100755
tools/perf/tests/shell/addr2line_inlines.shcreate mode 100644tools/perf/tests/workloads/inlineloop.ccreate mode 100644tools/perf/util/libdw.c
create mode 100644tools/perf/util/libdw.hI don't see the differences to the other addr2line implementations anymore, but only because it falls through to the old ones when libdw fails now.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
For example when building Perf with LLVM it can't get the line in the inlineloop workload, and there's still a few things in libc and other system libraries it fails on.
Hmm.. I wonder what the issue is. I was looking at the dwarf output from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are
I see some issues in libc on Ubuntu though, which I assume is compiled with GCC, although there's no .comment section in it so I can't be sure. So it's not exclusively LLVM but it does seem like LLVM builds cause a lot more failures.
Quoted 10 lines
Quoted 10 lines
doing something to confuse libdw? I'll try to investigate. There are quite a few levels of libdw: there's the raw libdw, libdwfl (frontend to libdw) that does the parsing and tries to give things like nested debug scopes (libdwfl is the one needing addresses with a module bias rather than raw file offsets), and then there is the
dwarf-aux.cthat is in perf and is used by things like probe finding (I believe this doesn't need biases addresses). Anyway, with the biases there are things I can screw up (like in the v1 patch) but maybe the LLVM issue is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific :-/
Actually I get the same behavior on Arm and x86.
Quoted 37 lines
Testing with clang/llvm on x86-64 (dwarf5):
``` $ make -C tools/perf O=/tmp/perf DEBUG=1 CC=clang CXX=clang++ HOSTCC=clang clean all ... $ llvm-dwarfdump /tmp/perf/perf ... 0x0014f852: Compile Unit: length = 0x00000294, format = DWARF32, version = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x1879a, addr_size = 0x08 (next unit at 0x0014faea) 0x0014f85e: DW_TAG_compile_unit DW_AT_producer ("Debian clang version 19.1.7 (3+build5)") DW_AT_language (DW_LANG_C11) DW_AT_name ("tests/workloads/inlineloop.c") DW_AT_str_offsets_base (0x0004a550) DW_AT_stmt_list (0x0008c3f2) DW_AT_comp_dir ("linux/tools/perf") DW_AT_low_pc (0x00000000001e61c0) DW_AT_high_pc (0x00000000001e62e9) DW_AT_addr_base (0x00022248) DW_AT_loclists_base (0x0000018a) ... $ sudo /tmp/perf/perf record --call-graph dwarf -e cycles:u -- /tmp/perf/perf test -w inlineloop 1 ... $ sudo /tmp/perf/perf script --fields +srcline ... perf-inlineloop 2284167 423038.015394: 569917 cpu_core/cycles/u: 56390020d2c6 leaf+0x26 inlineloop.c:21 (inlined) 56390020d2c6 middle+0x26 inlineloop.c:27 (inlined) 56390020d2c6 parent+0x26 (/tmp/perf/perf) ... ```
Quoted 7 lines
I ran inside of gdb and confirmed that the libdw code is creating the inlined information (breakpoint on
libdw_a2l_cb, etc.). So I'm not able to reproduce the LLVM issue for now on x86-64.Thanks,
Ian
If I set this in ~/.perfconfig so the fallback is disabled:
[addr2line]
style = libdw
Then:
$ make LLVM=1 -C tools/perf DEBUG=1 clean all
$ perf record --delay 1000 -- perf test -w inlineloop 2
$ perf script --fields ip,srcline
6012b5957b70
perf[1f7b70]
6012b5957b70
perf[1f7b70]
...x86:
$ clang -v Ubuntu clang version 15.0.7
Arm:
$ clang -v Ubuntu clang version 18.1.8 (11~20.04.2)
Disabling the ~/.perfconfig to re-enable the LLVM fallback works:
(x86)
$ perf script --fields ip,srcline
6012b5957b70
inlineloop.c:20
6012b5957b70
inlineloop.c:20Interestingly, on Arm this results in zeros for line numbers. This is a completely different issue though which I didn't notice before because I built with GCC. It falls all the way back to A2L_STYLE_CMD:
(Arm)
$ perf script --fields ip,srcline
aaaad0a7828c
inlineloop.c:0
aaaad0a7828c
inlineloop.c:0$ addr2line -e `which perf` -a -i -f aaaad0a7828c 0x0000aaaad0a7828c ?? ??:0
Probably shouldn't get sidetracked by that here though. It's at least working when compiled with GCC, and neither LLVM or libdw work, so it's no worse.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
But I think it's fine because it doesn't give the wrong line anymore, it just falls through to another working addr2line implementation.
Just to confirm that with gcc builds it isn't failing now? ie it isn't just an addr2line implementation that falls through all the time? I was seeing things working/testing on x86 with gcc.
No, the GCC Perf build always works with libdw as far as I can see. Just the occasional fall through to LLVM with some libc addresses.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Reviewed-by James Clark <[email protected]>Thanks,
Ian
Ian Rogers
reviewerThanks Arnaldo for landing the series in perf-tools-next!
James mentioned frame pointer unwinding lacking inlines. I had a look at and I think this patch may suffice (although on some quick testing I wasn't able to get inlines other than at the leaf):
```
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -2090,6 +2090,8 @@ struct iterations {
u64 cycles;
};
+static int append_inlines(struct callchain_cursor *cursor, struct
map_symbol *ms, u64 ip);
+
static int add_callchain_ip(struct thread *thread,
struct callchain_cursor *cursor,
struct symbol **parent,
@@ -2170,6 +2172,10 @@ static int add_callchain_ip(struct thread *thread,
ms.maps = maps__get(al.maps);
ms.map = map__get(al.map);
ms.sym = al.sym;
+
+ if (append_inlines(cursor, &ms, ip) == 0)
+ goto out;
+
srcline = callchain_srcline(&ms, al.addr);
err = callchain_cursor_append(cursor, ip, &ms,
branch, flags, nr_loop_iter,
```Having inline information seems like a good thing regardless of the stack trace format, so it'd be nice to move a patch like this forward.
Thanks,
Ian