Skip to contentSkip to conversation

perf report: Support LLVM for addr2line()

PATCH Superseded Steinar H. Gunderson 2 participants 5 messages May 23, 2024 – May 24, 2024 perf events

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/3 perf report: Support LLVM for addr2line()

In addition to the existing support for libbfd and calling out to an external addr2line command, add support for using libllvm directly. This is both faster than libbfd, and can be enabled in distro builds (the LLVM license has an explicit provision for GPLv2 compatibility). Thus, it is set as the primary choice if available.

As an example, running perf report on a medium-size profile with DWARF-based backtraces took 58 seconds with LLVM, 78 seconds with libbfd, 153 seconds with external llvm-addr2line, and I got tired and aborted the test after waiting for 55 minutes with external bfd addr2line (which is the default for perf as compiled by distributions today). Evidently, for this case, the bfd addr2line process needs 18 seconds (on a 5.2 GHz Zen 3) to load the .debug ELF in question, hits the 1-second timeout and gets killed during initialization, getting restarted anew every time. Having an in-process addr2line makes this much more robust.

As future extensions, libllvm can be used in many other places where we currently use libbfd or other libraries:

- Symbol enumeration (in particular, for PE binaries).
- Demangling (including non-Itanium demangling, e.g. Microsoft or Rust).
- Disassembling (perf annotate).

However, these are much less pressing; most people don't profile PE binaries, and perf has non-bfd paths for ELF. The same with demangling; the default _cxa_demangle path works fine for most users. Disassembling is coming in a later patch in the series; however do note that while bfd objdump can be slow on large binaries, it is possible to use --objdump=llvm-objdump to get the speed benefits. (It appears LLVM-based demangling is very simple, should we want that.)

Tested with LLVM 14, 15, 16, 18 and 19. For some reason, LLVM 12 was not correctly detected using feature_check, and thus was not tested.

---
 tools/build/Makefile.feature       |   1 +
 tools/perf/Makefile.config         |  15 ++++
 tools/perf/builtin-version.c       |   1 +
 tools/perf/util/Build              |   1 +
 tools/perf/util/llvm-c-helpers.cpp | 130 +++++++++++++++++++++++++++++
 tools/perf/util/llvm-c-helpers.h   |  47 +++++++++++
 tools/perf/util/srcline.c          |  57 ++++++++++++-
 7 files changed, 251 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/llvm-c-helpers.cpp
 create mode 100644 tools/perf/util/llvm-c-helpers.h
tools/build/Makefile.feature +1 -0
index 1e2ab148d5db..278b26216254 100644
@@ -136,6 +136,7 @@ FEATURE_DISPLAY ?=              \
          libunwind              \
          libdw-dwarf-unwind     \
          libcapstone            \
+         llvm                   \
          zlib                   \
          lzma                   \
          get_cpuid              \
tools/perf/Makefile.config +15 -0
index 7f1e016a9253..414a37f712bd 100644
@@ -969,6 +969,21 @@ ifdef BUILD_NONDISTRO
   endif
 endif
 
+ifndef NO_LLVM
+  $(call feature_check,llvm)
+  ifeq ($(feature-llvm), 1)
+    CFLAGS += -DHAVE_LLVM_SUPPORT
+    CXXFLAGS += -DHAVE_LLVM_SUPPORT
+    CXXFLAGS += $(shell $(LLVM_CONFIG) --cxxflags)
+    LIBLLVM = $(shell $(LLVM_CONFIG) --libs all) $(shell $(LLVM_CONFIG) --system-libs)
+    EXTLIBS += -L$(shell $(LLVM_CONFIG) --libdir) $(LIBLLVM)
+    $(call detected,CONFIG_LLVM)
+  else
+    $(warning No libllvm found, slower source file resolution, please install llvm-devel/llvm-dev)
+    NO_LLVM := 1
+  endif
+endif
+
 ifndef NO_DEMANGLE
   $(call feature_check,cxa-demangle)
   ifeq ($(feature-cxa-demangle), 1)
tools/perf/builtin-version.c +1 -0
index 398aa53e9e2e..94d8e13a688f 100644
@@ -65,6 +65,7 @@ static void library_status(void)
 	STATUS(HAVE_LIBBFD_SUPPORT, libbfd);
 	STATUS(HAVE_DEBUGINFOD_SUPPORT, debuginfod);
 	STATUS(HAVE_LIBELF_SUPPORT, libelf);
+	STATUS(HAVE_LLVM_SUPPORT, libllvm);
 	STATUS(HAVE_LIBNUMA_SUPPORT, libnuma);
 	STATUS(HAVE_LIBNUMA_SUPPORT, numa_num_possible_cpus);
 	STATUS(HAVE_LIBPERL_SUPPORT, libperl);
tools/perf/util/Build +1 -0
index da64efd8718f..32c4e5e634ed 100644
@@ -226,6 +226,7 @@ perf-$(CONFIG_CXX_DEMANGLE) += demangle-cxx.o
 perf-y += demangle-ocaml.o
 perf-y += demangle-java.o
 perf-y += demangle-rust.o
+perf-$(CONFIG_LLVM) += llvm-c-helpers.o
 
 ifdef CONFIG_JITDUMP
 perf-$(CONFIG_LIBELF) += jitdump.o
tools/perf/util/llvm-c-helpers.cpp new file +130 -0

130 lines not on this page · raw message

tools/perf/util/llvm-c-helpers.h new file +47 -0

47 lines not on this page · raw message

tools/perf/util/srcline.c +56 -1

69 lines not on this page · raw message

Signature · 1 line
2.45.1
#1 lore raw

PATCH v5 2/3 perf annotate: split out read_symbol()

The Capstone disassembler code has a useful code snippet to read the bytes for a given code symbol into memory. Split it out into its own function, so that the LLVM disassembler can use it in the next patch.

---
 tools/perf/util/disasm.c | 90 +++++++++++++++++++++++++---------------
 1 file changed, 56 insertions(+), 34 deletions(-)
tools/perf/util/disasm.c +56 -34
index 72aec8f61b94..c0dbb955e61a 100644
@@ -1396,6 +1396,53 @@ static int find_file_offset(u64 start, u64 len, u64 pgoff, void *arg)
 	return 0;
 }
 
+static u8 *
+read_symbol(const char *filename, struct map *map, struct symbol *sym,
+	    u64 *len, bool *is_64bit)
+{
+	struct dso *dso = map__dso(map);
+	struct nscookie nsc;
+	u64 start = map__rip_2objdump(map, sym->start);
+	u64 end = map__rip_2objdump(map, sym->end);
+	int fd, count;
+	u8 *buf = NULL;
+	struct find_file_offset_data data = {
+		.ip = start,
+	};
+
+	*is_64bit = false;
+
+	nsinfo__mountns_enter(dso__nsinfo(dso), &nsc);
+	fd = open(filename, O_RDONLY);
+	nsinfo__mountns_exit(&nsc);
+	if (fd < 0)
+		return NULL;
+
+	if (file__read_maps(fd, /*exe=*/true, find_file_offset, &data,
+			    is_64bit) == 0)
+		goto err;
+
+	*len = end - start;
+	buf = malloc(*len);
+	if (buf == NULL)
+		goto err;
+
+	count = pread(fd, buf, *len, data.offset);
+	close(fd);
+	fd = -1;
+
+	if ((u64)count != *len)
+		goto err;
+
+	return buf;
+
+err:
+	if (fd >= 0)
+		close(fd);
+	free(buf);
+	return NULL;
+}
+
 static void print_capstone_detail(cs_insn *insn, char *buf, size_t len,
 				  struct annotate_args *args, u64 addr)
 {
@@ -1458,19 +1505,13 @@ static int symbol__disassemble_capstone(char *filename, struct symbol *sym,
 {
 	struct annotation *notes = symbol__annotation(sym);
 	struct map *map = args->ms.map;
-	struct dso *dso = map__dso(map);
-	struct nscookie nsc;
 	u64 start = map__rip_2objdump(map, sym->start);
-	u64 end = map__rip_2objdump(map, sym->end);
-	u64 len = end - start;
+	u64 len;
 	u64 offset;
-	int i, fd, count;
+	int i, count;
 	bool is_64bit = false;
 	bool needs_cs_close = false;
 	u8 *buf = NULL;
-	struct find_file_offset_data data = {
-		.ip = start,
-	};
 	csh handle;
 	cs_insn *insn;
 	char disasm_buf[512];
@@ -1479,31 +1520,9 @@ static int symbol__disassemble_capstone(char *filename, struct symbol *sym,
 	if (args->options->objdump_path)
 		return -1;
 
-	nsinfo__mountns_enter(dso__nsinfo(dso), &nsc);
-	fd = open(filename, O_RDONLY);
-	nsinfo__mountns_exit(&nsc);
-	if (fd < 0)
-		return -1;
-
-	if (file__read_maps(fd, /*exe=*/true, find_file_offset, &data,
-			    &is_64bit) == 0)
-		goto err;
-
-	if (open_capstone_handle(args, is_64bit, &handle) < 0)
-		goto err;
-
-	needs_cs_close = true;
-
-	buf = malloc(len);
+	buf = read_symbol(filename, map, sym, &len, &is_64bit);
 	if (buf == NULL)
-		goto err;
-
-	count = pread(fd, buf, len, data.offset);
-	close(fd);
-	fd = -1;
-
-	if ((u64)count != len)
-		goto err;
+		return -1;
 
 	/* add the function address and name */
 	scnprintf(disasm_buf, sizeof(disasm_buf), "%#"PRIx64" <%s>:",
@@ -1521,6 +1540,11 @@ static int symbol__disassemble_capstone(char *filename, struct symbol *sym,
 
 	annotation_line__add(&dl->al, &notes->src->source);
 
+	if (open_capstone_handle(args, is_64bit, &handle) < 0)
+		goto err;
+
+	needs_cs_close = true;
+
 	count = cs_disasm(handle, buf, len, start, len, &insn);
 	for (i = 0, offset = 0; i < count; i++) {
 		int printed;
@@ -1565,8 +1589,6 @@ static int symbol__disassemble_capstone(char *filename, struct symbol *sym,
 	return count < 0 ? count : 0;
 
 err:
-	if (fd >= 0)
-		close(fd);
 	if (needs_cs_close) {
 		struct disasm_line *tmp;
 
Signature · 1 line
2.45.1
#2 lore raw

PATCH v5 3/3 perf annotate: LLVM-based disassembler

In reply to Steinar H. Gunderson

Support using LLVM as a disassembler method, allowing helperless annotation in non-distro builds. (It is also much faster than using libbfd or bfd objdump on binaries with a lot of debug information.)

This is nearly identical to the output of llvm-objdump; there are some very rare whitespace differences, some minor changes to demangling (since we use perf's regular demangling and not LLVM's own) and the occasional case where llvm-objdump makes a different choice when multiple symbols share the same address. It should work across all of LLVM's supported architectures, although I've only tested 64-bit x86, and finding the right triple from perf's idea of machine architecture can sometimes be a bit tricky. Ideally, we should have some way of finding the triplet just from the file itself.

---
 tools/perf/util/disasm.c           | 195 +++++++++++++++++++++++++++++
 tools/perf/util/llvm-c-helpers.cpp |  62 +++++++++
 tools/perf/util/llvm-c-helpers.h   |  11 ++
 3 files changed, 268 insertions(+)
tools/perf/util/disasm.c +195 -0
index c0dbb955e61a..9c07d2a8c8a8 100644
@@ -43,6 +43,7 @@ static int call__scnprintf(struct ins *ins, char *bf, size_t size,
 
 static void ins__sort(struct arch *arch);
 static int disasm_line__parse(char *line, const char **namep, char **rawp);
+static char *expand_tabs(char *line, char **storage, size_t *storage_len);
 
 static __attribute__((constructor)) void symbol__init_regexpr(void)
 {
@@ -1378,7 +1379,9 @@ static int open_capstone_handle(struct annotate_args *args, bool is_64bit,
 
 	return 0;
 }
+#endif
 
+#if defined(HAVE_LIBCAPSTONE_SUPPORT) || defined(HAVE_LLVM_SUPPORT)
 struct find_file_offset_data {
 	u64 ip;
 	u64 offset;
@@ -1442,7 +1445,9 @@ read_symbol(const char *filename, struct map *map, struct symbol *sym,
 	free(buf);
 	return NULL;
 }
+#endif
 
+#ifdef HAVE_LIBCAPSTONE_SUPPORT
 static void print_capstone_detail(cs_insn *insn, char *buf, size_t len,
 				  struct annotate_args *args, u64 addr)
 {
@@ -1606,6 +1611,191 @@ static int symbol__disassemble_capstone(char *filename, struct symbol *sym,
 }
 #endif
 
+#ifdef HAVE_LLVM_SUPPORT
+#include <llvm-c/Disassembler.h>
+#include <llvm-c/Target.h>
+#include "util/llvm-c-helpers.h"
+
+struct symbol_lookup_storage {
+	u64 branch_addr;
+	u64 pcrel_load_addr;
+};
+
+/*
+ * Whenever LLVM wants to resolve an address into a symbol, it calls this
+ * callback. We don't ever actually _return_ anything (in particular, because
+ * it puts quotation marks around what we return), but we use this as a hint
+ * that there is a branch or PC-relative address in the expression that we
+ * should add some textual annotation for after the instruction. The caller
+ * will use this information to add the actual annotation.
+ */
+static const char *
+symbol_lookup_callback(void *disinfo, uint64_t value,
+		       uint64_t *ref_type,
+		       uint64_t address __maybe_unused,
+		       const char **ref __maybe_unused)
+{
+	struct symbol_lookup_storage *storage =
+		(struct symbol_lookup_storage *)disinfo;
+	if (*ref_type == LLVMDisassembler_ReferenceType_In_Branch)
+		storage->branch_addr = value;
+	else if (*ref_type == LLVMDisassembler_ReferenceType_In_PCrel_Load)
+		storage->pcrel_load_addr = value;
+	*ref_type = LLVMDisassembler_ReferenceType_InOut_None;
+	return NULL;
+}
+
+static int symbol__disassemble_llvm(char *filename, struct symbol *sym,
+				    struct annotate_args *args)
+{
+	struct annotation *notes = symbol__annotation(sym);
+	struct map *map = args->ms.map;
+	struct dso *dso = map__dso(map);
+	u64 start = map__rip_2objdump(map, sym->start);
+	u8 *buf;
+	u64 len;
+	u64 pc;
+	bool is_64bit;
+	char triplet[64];
+	char disasm_buf[2048];
+	size_t disasm_len;
+	struct disasm_line *dl;
+	LLVMDisasmContextRef disasm = NULL;
+	struct symbol_lookup_storage storage;
+	char *line_storage = NULL;
+	size_t line_storage_len = 0;
+
+	if (args->options->objdump_path)
+		return -1;
+
+	LLVMInitializeAllTargetInfos();
+	LLVMInitializeAllTargetMCs();
+	LLVMInitializeAllDisassemblers();
+
+	buf = read_symbol(filename, map, sym, &len, &is_64bit);
+	if (buf == NULL)
+		return -1;
+
+	if (arch__is(args->arch, "x86")) {
+		if (is_64bit)
+			scnprintf(triplet, sizeof(triplet), "x86_64-pc-linux");
+		else
+			scnprintf(triplet, sizeof(triplet), "i686-pc-linux");
+	} else {
+		scnprintf(triplet, sizeof(triplet), "%s-linux-gnu",
+			  args->arch->name);
+	}
+
+	disasm = LLVMCreateDisasm(
+		triplet, &storage, 0, NULL, symbol_lookup_callback);
+	if (disasm == NULL)
+		goto err;
+
+	if (args->options->disassembler_style &&
+	    !strcmp(args->options->disassembler_style, "intel"))
+		LLVMSetDisasmOptions(
+			disasm, LLVMDisassembler_Option_AsmPrinterVariant);
+
+	/*
+	 * This needs to be set after AsmPrinterVariant, due to a bug in LLVM;
+	 * setting AsmPrinterVariant makes a new instruction printer, making it
+	 * forget about the PrintImmHex flag (which is applied before if both
+	 * are given to the same call).
+	 */
+	LLVMSetDisasmOptions(disasm, LLVMDisassembler_Option_PrintImmHex);
+
+	/* add the function address and name */
+	scnprintf(disasm_buf, sizeof(disasm_buf), "%#"PRIx64" <%s>:",
+		  start, sym->name);
+
+	args->offset = -1;
+	args->line = disasm_buf;
+	args->line_nr = 0;
+	args->fileloc = NULL;
+	args->ms.sym = sym;
+
+	dl = disasm_line__new(args);
+	if (dl == NULL)
+		goto err;
+
+	annotation_line__add(&dl->al, &notes->src->source);
+
+	pc = start;
+	for (u64 offset = 0; offset < len; ) {
+		unsigned int ins_len;
+
+		storage.branch_addr = 0;
+		storage.pcrel_load_addr = 0;
+
+		ins_len = LLVMDisasmInstruction(
+			disasm, buf + offset, len - offset, pc,
+			disasm_buf, sizeof(disasm_buf));
+		if (ins_len == 0)
+			goto err;
+		disasm_len = strlen(disasm_buf);
+
+		if (storage.branch_addr != 0) {
+			char *name = llvm_name_for_code(
+				dso, filename, storage.branch_addr);
+			if (name != NULL) {
+				disasm_len += scnprintf(
+					disasm_buf + disasm_len,
+					sizeof(disasm_buf) - disasm_len,
+					" <%s>", name);
+				free(name);
+			}
+		}
+		if (storage.pcrel_load_addr != 0) {
+			char *name = llvm_name_for_data(
+				dso, filename, storage.pcrel_load_addr);
+			disasm_len += scnprintf(disasm_buf + disasm_len,
+						sizeof(disasm_buf) - disasm_len,
+						"  # %#"PRIx64,
+						storage.pcrel_load_addr);
+			if (name) {
+				disasm_len += scnprintf(
+					disasm_buf + disasm_len,
+					sizeof(disasm_buf) - disasm_len,
+					" <%s>", name);
+				free(name);
+			}
+		}
+
+		args->offset = offset;
+		args->line = expand_tabs(
+			disasm_buf, &line_storage, &line_storage_len);
+		args->line_nr = 0;
+		args->fileloc = NULL;
+		args->ms.sym = sym;
+
+		llvm_addr2line(filename, pc, &args->fileloc,
+			       (unsigned int *)&args->line_nr, false, NULL);
+
+		dl = disasm_line__new(args);
+		if (dl == NULL)
+			goto err;
+
+		annotation_line__add(&dl->al, &notes->src->source);
+
+		free(args->fileloc);
+		pc += ins_len;
+		offset += ins_len;
+	}
+
+	LLVMDisasmDispose(disasm);
+	free(buf);
+	free(line_storage);
+	return 0;
+
+err:
+	LLVMDisasmDispose(disasm);
+	free(buf);
+	free(line_storage);
+	return -1;
+}
+#endif
+
+
 /*
  * Possibly create a new version of line with tabs expanded. Returns the
  * existing or new line, storage is updated if a new line is allocated. If
@@ -1730,6 +1920,11 @@ int symbol__disassemble(struct symbol *sym, struct annotate_args *args)
 		strcpy(symfs_filename, tmp);
 	}
 
+#ifdef HAVE_LLVM_SUPPORT
+	err = symbol__disassemble_llvm(symfs_filename, sym, args);
+	if (err == 0)
+		goto out_remove_tmp;
+#endif
 #ifdef HAVE_LIBCAPSTONE_SUPPORT
 	err = symbol__disassemble_capstone(symfs_filename, sym, args);
 	if (err == 0)
tools/perf/util/llvm-c-helpers.cpp +62 -0

77 lines not on this page · raw message

tools/perf/util/llvm-c-helpers.h +11 -0
index 1b28cdc9f9b7..e00e3f73a906 100644
@@ -11,6 +11,8 @@
 extern "C" {
 #endif
 
+struct dso;
+
 struct llvm_a2l_frame {
 	char *filename;
 	char *funcname;
@@ -40,6 +42,15 @@ int llvm_addr2line(const char *dso_name,
 		   bool unwind_inlines,
 		   struct llvm_a2l_frame **inline_frames);
 
+/*
+ * Simple symbolizers for addresses; will convert something like
+ * 0x12345 to "func+0x123". Will return NULL if no symbol was found.
+ *
+ * The returned value must be freed by the caller, with free().
+ */
+char *llvm_name_for_code(struct dso *dso, const char *dso_name, u64 addr);
+char *llvm_name_for_data(struct dso *dso, const char *dso_name, u64 addr);
+
 #ifdef __cplusplus
 }
 #endif
Signature · 1 line
2.45.1
maintainer Tested-by on v5 1/3
#3 lore raw
In reply to Steinar H. Gunderson
Quoted 11 lines · Steinar H. Gunderson

In addition to the existing support for libbfd and calling out to an external addr2line command, add support for using libllvm directly. This is both faster than libbfd, and can be enabled in distro builds (the LLVM license has an explicit provision for GPLv2 compatibility). Thus, it is set as the primary choice if available.

As an example, running perf report on a medium-size profile with DWARF-based backtraces took 58 seconds with LLVM, 78 seconds with libbfd, 153 seconds with external llvm-addr2line, and I got tired and aborted the test after waiting for 55 minutes with external bfd addr2line (which is the default for perf as compiled by distributions

Before this patch, using BFD's addr2line:

root@x1:~# echo 1 > /proc/sys/vm/drop_caches 
root@x1:~# perf record --call-graph dwarf find / > /dev/null
[ perf record: Woken up 428 times to write data ]
[ perf record: Captured and wrote 107.598 MB perf.data (13355 samples) ]
root@x1:~#
root@x1:~# dnf debuginfo-install findutils
root@x1:~# perf report --no-children --stdio -s srcfile,srcline
# To display the perf.data header info, please use --header/--header-only options.
#
# Total Lost Samples: 0
#
# Samples: 75  of event 'cpu_atom/cycles/P'
# Event count (approx.): 22073647
#
# Overhead  Source File        Source:Line                     
# ........  .................  ................................
#
     4.59%  disk-io.c          disk-io.c:108
            |
            ---btrfs_buffer_uptodate
               read_block_for_search
               |          
                --4.13%--btrfs_search_slot
                          btrfs_real_readdir
                          iterate_dir
                          __x64_sys_getdents64
                          do_syscall_64
                          entry_SYSCALL_64
<SNIP>
root@x1:~# echo 1 > /proc/sys/vm/drop_caches 
root@x1:~# taskset -c 0 perf stat -e cpu_core/cycles/P perf report --no-children --stdio -s srcfile,srcline > /dev/null

Performance counter stats for 'perf report --no-children --stdio -s srcfile,srcline':

225,837,039,942 cpu_core/cycles/P

     113.192464730 seconds time elapsed

     102.789965000 seconds user
       6.478551000 seconds sys

root@x1:~#

Now with your patches:

Brownie points++:

Makefile.config:982: No libllvm found, slower source file resolution, please install llvm-devel/llvm-dev

... llvm: [ OFF ]

But maybe use "libllvm"?

⬢[acme@toolbox perf-tools]$ sudo dnf install llvm-devel

... libcapstone: [ on ]
... llvm: [ on ]

root@x1:~# echo 1 > /proc/sys/vm/drop_caches 
root@x1:~# taskset -c 0 perf stat -e cpu_core/cycles/P perf report --no-children --stdio -s srcfile,srcline > /dev/null

Performance counter stats for 'perf report --no-children --stdio -s srcfile,srcline':

     2,462,040,637      cpu_core/cycles/P                                                     

       1.696059762 seconds time elapsed

       1.013055000 seconds user
       0.423232000 seconds sys

root@x1:~#

root@x1:~# head -25 report.srcfile.srcline.libllvm 
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 75  of event 'cpu_atom/cycles/P'
# Event count (approx.): 22073647
#
# Overhead  Source File        Source:Line                     
# ........  .................  ................................
#
     4.59%  disk-io.c          disk-io.c:108
            |
            ---btrfs_buffer_uptodate
               read_block_for_search
               |          
                --4.13%--btrfs_search_slot
                          btrfs_real_readdir
                          iterate_dir
                          __x64_sys_getdents64
                          do_syscall_64
                          entry_SYSCALL_64
                          __GI___getdents64
                          __GI___readdir64
                          fts_build.constprop.0
root@x1:~#



Matches, now everything:

root@x1:~# diff -u report.srcfile.srcline.bfd report.srcfile.srcline.libllvm
report.srcfile.srcline.libllvm renamed from report.srcfile.srcline.bfd +1 -0
@@ -23,6 +23,7 @@
                           __GI___getdents64
                           __GI___readdir64
                           fts_build.constprop.0
+                          rpl_fts_read (inlined)
                           find
                           process_all_startpoints
                           main

So mostly the above, and:

root@x1:~# perf probe -x /usr/bin/find -L find | grep fts_read
     44        while ( (errno=0, ent=fts_read (p)) != NULL )
               /* fts_read returned NULL; distinguish between "finished" and "error". */
root@x1:~#
root@x1:~# readelf -wi /usr/lib/debug/usr/bin/find-4.9.0-5.fc39.x86_64.debug | grep "DW_AT_name.*rpl_fts_read" -B2
 <1><2c69c>: Abbrev Number: 11 (DW_TAG_subprogram)
    <2c69d>   DW_AT_external    : 1
    <2c69d>   DW_AT_name        : (indirect string, offset: 0xda3): rpl_fts_read
--
 <1><364b8>: Abbrev Number: 25 (DW_TAG_subprogram)
    ^^^^^^^
    ^^^^^^^
    ^^^^^^^
    <364b9>   DW_AT_external    : 1
    <364b9>   DW_AT_name        : (indirect string, offset: 0xda3): rpl_fts_read

root@x1:~#

<1><2c952>: Abbrev Number: 81 (DW_TAG_subprogram)

    ^^^^^^
    ^^^^^^
    ^^^^^^
    <2c953>   DW_AT_name        : (indirect string, offset: 0x118f): find
    <2c957>   DW_AT_decl_file   : 22
    <2c957>   DW_AT_decl_line   : 482
    <2c959>   DW_AT_decl_column : 1
    <2c959>   DW_AT_prototyped  : 1
    <2c959>   DW_AT_type        : <alt 0x573b>
    <2c95d>   DW_AT_sibling     : <0x2c996>

Then the function referring to the above that has inlined functions:

<1><def8>: Abbrev Number: 2 (DW_TAG_subprogram)

    <def9>   DW_AT_abstract_origin: <0x2c952> <=============================
    <defd>   DW_AT_ranges      : 0xff0
    <df01>   DW_AT_frame_base  : 1 byte block: 9c 	(DW_OP_call_frame_cfa)
    <df03>   DW_AT_call_all_calls: 1
    <df03>   DW_AT_sibling     : <0xe84a>
 <2><df06>: Abbrev Number: 62 (DW_TAG_formal_parameter)
    <df07>   DW_AT_abstract_origin: <0x2c95f>
    <df0b>   DW_AT_location    : 0x8345 (location list)
    <df0f>   DW_AT_GNU_locviews: 0x833f
 <2><df13>: Abbrev Number: 52 (DW_TAG_variable)
    <df14>   DW_AT_abstract_origin: <0x2c96b>
    <df18>   DW_AT_location    : 3 byte block: 91 a0 7f 	(DW_OP_fbreg: -96)
<SNIP>
 <3><df77>: Abbrev Number: 51 (DW_TAG_inlined_subroutine)
    <df78>   DW_AT_abstract_origin: <0x364b8> <=========================
    <df7c>   DW_AT_entry_pc    : 0xeddf
    <df84>   DW_AT_GNU_entry_view: 0
    <df86>   DW_AT_ranges      : 0x1042
    <df8a>   DW_AT_call_file   : 6
    <df8b>   DW_AT_call_line   : 526
    <df8d>   DW_AT_call_column : 29
    <df8e>   DW_AT_sibling     : <0xe60d>
 <4><df91>: Abbrev Number: 62 (DW_TAG_formal_parameter)
    <df92>   DW_AT_abstract_origin: <0x364c3>
    <df96>   DW_AT_location    : 0x848f (location list)
    <df9a>   DW_AT_GNU_locviews: 0x8473
 <4><df9e>: Abbrev Number: 14 (DW_TAG_lexical_block)
    <df9f>   DW_AT_ranges      : 0x1042

<SNIP>

So the libllvm is even producing a better result, showing those inlines not seen in the BFD based output.

So going from (BFD):

225,837,039,942 cpu_core/cycles/P

     113.192464730 seconds time elapsed

     102.789965000 seconds user
       6.478551000 seconds sys

To (using libllvm):

     2,462,040,637      cpu_core/cycles/P                                                     

       1.696059762 seconds time elapsed

       1.013055000 seconds user
       0.423232000 seconds sys

100x speedup, looks like a win! 8-)

Thanks a lot, the comments I made on the patch are mostly coding style, please consider them, but I wouldn't get in the way for this patch to get merged because of that albeit would be nice to try to fit in more nicely with the existing source code base.

Assuming the code remains the same after an eventual coding style polish:

- Arnaldo

The rest, more comments in the patch, after the diff output.

@@ -35,6 +36,7 @@

             ---__GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -56,6 +58,7 @@

             |          __GI___getdents64
             |          __GI___readdir64
             |          fts_build.constprop.0
+            |          rpl_fts_read (inlined)
             |          find
             |          process_all_startpoints
             |          main
@@ -75,6 +78,8 @@
                        0x55f742a0e9a3
                        0x55f742a0e9a3
                        0x55f742a02735
+                       0x55f742a02735
+                       0x55f742a02735
                        0x55f742a02f6d
                        0x55f742a037c4
                        0x55f7429fc0f7
@@ -90,6 +95,7 @@
                |          __fdopendir
                |          opendirat
                |          fts_build.constprop.0
+               |          rpl_fts_read (inlined)
                |          find
                |          process_all_startpoints
                |          main
@@ -98,6 +104,8 @@
                |          _start
                |          
                 --2.09%--__close

+ left_dir (inlined)
+ left_dir (inlined)
consider_visiting find process_all_startpoints
@@ -117,6 +125,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -152,6 +161,7 @@
openat_safer
opendirat
fts_build.constprop.0
+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -170,11 +180,13 @@

                do_syscall_64
                entry_SYSCALL_64
                __mmap64 (inlined)
-               __mmap64 (inlined)
+               mmap64
+               _dl_map_segment (inlined)
+               _dl_map_segments (inlined)

_dl_map_object_from_fd
_dl_map_object
openaux
- __GI__dl_catch_exception (inlined)
+ _dl_catch_exception
_dl_map_object_deps
dl_main
_dl_sysdep_start
@@ -202,7 +214,8 @@

      2.21%  do-rel.h           do-rel.h:133
             |
-            ---_dl_relocate_object
+            ---elf_dynamic_do_Rela (inlined)

+ _dl_relocate_object
dl_main
_dl_sysdep_start
_dl_start_final (inlined)
@@ -216,6 +229,7 @@

                entry_SYSCALL_64
                __GI___fstatat64
                fts_stat.isra.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -231,6 +245,8 @@

                do_syscall_64
                entry_SYSCALL_64
                __close

+ left_dir (inlined)
+ left_dir (inlined)
consider_visiting
find
process_all_startpoints
@@ -250,6 +266,7 @@

                entry_SYSCALL_64
                __GI___fstatat64
                fts_stat.isra.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -272,6 +289,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -303,6 +321,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -318,6 +337,7 @@
pred_print
apply_predicate
apply_predicate
+ visit (inlined)
consider_visiting
find
process_all_startpoints
@@ -331,6 +351,7 @@

             ---_int_free_create_chunk
                _int_free_merge_chunk
                cfree@GLIBC_2.2.5

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -340,7 +361,7 @@

      2.09%  fts-cycle.c        fts-cycle.c:126
             |
-            ---leave_dir (inlined)
+            ---leave_dir.lto_priv.0
                0x7ffd0a6562af
 
      2.09%                     __printf_buffer_to_file_switch+0

@@ -351,6 +372,7 @@
pred_print
apply_predicate
apply_predicate
+ visit (inlined)
consider_visiting
find
process_all_startpoints
@@ -371,6 +393,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -387,6 +410,8 @@

                do_syscall_64
                entry_SYSCALL_64
                __close

+ left_dir (inlined)
+ left_dir (inlined)
consider_visiting
find
process_all_startpoints
@@ -403,6 +428,7 @@

                __GI___fstatat64
                fts_stat.isra.0
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -421,6 +447,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -432,7 +459,8 @@

             |
             ---_int_free
                cfree@GLIBC_2.2.5
-               leave_dir (inlined)
+               leave_dir.lto_priv.0
+               rpl_fts_read (inlined)

find
process_all_startpoints
main
@@ -456,6 +484,7 @@
openat_safer
opendirat
fts_build.constprop.0
+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -474,6 +503,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -492,6 +522,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -505,6 +536,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -516,6 +548,7 @@

             |
             ---pred_and
                apply_predicate

+ visit (inlined)
consider_visiting
find
process_all_startpoints
@@ -536,6 +569,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -553,6 +587,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -572,6 +607,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -591,6 +627,7 @@
openat_safer
opendirat
fts_build.constprop.0
+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -610,6 +647,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -620,6 +658,7 @@

      1.98%                     cfree@GLIBC_2.2.5+139
             |
             ---cfree@GLIBC_2.2.5

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -635,6 +674,8 @@

                do_syscall_64
                entry_SYSCALL_64
                __close

+ left_dir (inlined)
+ left_dir (inlined)
consider_visiting
find
process_all_startpoints
@@ -655,6 +696,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -677,6 +719,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -695,6 +738,7 @@

                __GI___fstatat64
                fts_stat.isra.0
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -706,8 +750,9 @@

             |
             ---hash_find_entry
                hash_remove (inlined)

- leave_dir (inlined)
+ leave_dir.lto_priv.0
fts_build.constprop.0
+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -740,6 +785,7 @@

                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -753,8 +799,9 @@
safe_hasher (inlined)
hash_find_entry
hash_remove (inlined)
- leave_dir (inlined)
+ leave_dir.lto_priv.0
fts_build.constprop.0
+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -799,6 +846,7 @@

                        |          0x7fc0e8e9ce16
                        |          0x55f742a1769f
                        |          0x55f742a03319
+                       |          0x55f742a03319
                        |          0x55f742a037c4
                        |          0x55f7429fc0f7
                        |          0x7fc0e8deb149
@@ -815,6 +863,7 @@
                                   0x7fc0e8e9ce16
                                   0x55f742a1769f
                                   0x55f742a03319
+                                  0x55f742a03319
                                   0x55f742a037c4
                                   0x55f7429fc0f7
                                   0x7fc0e8deb149
@@ -837,6 +886,7 @@
             |                     |          0x7fc0e8e9ce16
             |                     |          0x55f742a1769f
             |                     |          0x55f742a03319
+            |                     |          0x55f742a03319
             |                     |          0x55f742a037c4
             |                     |          0x55f7429fc0f7
             |                     |          0x7fc0e8deb149
@@ -852,6 +902,7 @@
             |                                0x7fc0e8e9ce16
             |                                0x55f742a1769f
             |                                0x55f742a03319
+            |                                0x55f742a03319
             |                                0x55f742a037c4
             |                                0x55f7429fc0f7
             |                                0x7fc0e8deb149
@@ -870,6 +921,7 @@
                        entry_SYSCALL_64
                        __GI___fstatat64
                        fts_stat.isra.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -898,6 +950,7 @@

                        |          __GI___getdents64
                        |          __GI___readdir64
                        |          fts_build.constprop.0
+                       |          rpl_fts_read (inlined)
                        |          find
                        |          process_all_startpoints
                        |          main
@@ -913,6 +966,7 @@
                                   __GI___getdents64
                                   __GI___readdir64
                                   fts_build.constprop.0

+ rpl_fts_read (inlined)
find process_all_startpoints main
@@ -935,6 +989,7 @@

                                              __GI___getdents64
                                              __GI___readdir64
                                              fts_build.constprop.0

+ rpl_fts_read (inlined)
find process_all_startpoints main
@@ -947,6 +1002,7 @@

             |
             ---__GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -971,6 +1027,7 @@

                                   0x55f742a17439
                                   |          
                                    --0.90%--0x55f742a03545
+                                             0x55f742a03545
                                              0x55f742a037c4
                                              0x55f7429fc0f7
                                              0x7fc0e8deb149
@@ -988,6 +1045,7 @@
                __GI___getdents64
                __GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -1019,6 +1077,7 @@

                        0x7fc0e8e9ce16
                        0x55f742a1769f
                        0x55f742a03319
+                       0x55f742a03319
                        0x55f742a037c4
                        0x55f7429fc0f7
                        0x7fc0e8deb149
@@ -1042,6 +1101,7 @@
                                   0x7fc0e8e9ce16
                                   0x55f742a1769f
                                   0x55f742a03319
+                                  0x55f742a03319
                                   0x55f742a037c4
                                   0x55f7429fc0f7
                                   0x7fc0e8deb149
@@ -1052,6 +1112,7 @@
             |
             ---__GI___readdir64
                fts_build.constprop.0

+ rpl_fts_read (inlined)
find
process_all_startpoints
main
@@ -3292,5 +3353,5 @@

#
-# (Tip: For hierarchical output, try: perf report --hierarchy) +# (Tip: For memory address profiling, try: perf mem record / perf mem report)
#

Quoted 44 lines

today). Evidently, for this case, the bfd addr2line process needs 18 seconds (on a 5.2 GHz Zen 3) to load the .debug ELF in question, hits the 1-second timeout and gets killed during initialization, getting restarted anew every time. Having an in-process addr2line makes this much more robust.

As future extensions, libllvm can be used in many other places where we currently use libbfd or other libraries:

- Symbol enumeration (in particular, for PE binaries).
- Demangling (including non-Itanium demangling, e.g. Microsoft or Rust).
- Disassembling (perf annotate).

However, these are much less pressing; most people don't profile PE binaries, and perf has non-bfd paths for ELF. The same with demangling; the default _cxa_demangle path works fine for most users. Disassembling is coming in a later patch in the series; however do note that while bfd objdump can be slow on large binaries, it is possible to use --objdump=llvm-objdump to get the speed benefits. (It appears LLVM-based demangling is very simple, should we want that.)

Tested with LLVM 14, 15, 16, 18 and 19. For some reason, LLVM 12 was not correctly detected using feature_check, and thus was not tested.

---
 tools/build/Makefile.feature       |   1 +
 tools/perf/Makefile.config         |  15 ++++
 tools/perf/builtin-version.c       |   1 +
 tools/perf/util/Build              |   1 +
 tools/perf/util/llvm-c-helpers.cpp | 130 +++++++++++++++++++++++++++++
 tools/perf/util/llvm-c-helpers.h   |  47 +++++++++++
 tools/perf/util/srcline.c          |  57 ++++++++++++-
 7 files changed, 251 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/llvm-c-helpers.cpp
 create mode 100644 tools/perf/util/llvm-c-helpers.h
tools/build/Makefile.feature +0 -0
index 1e2ab148d5db..278b26216254 100644

tools/build/Makefile.feature FEATURE_DISPLAY ?= \ · line 139

Quoted 4 lines
tools/build/Makefile.feature +1 -0
@@ -136,6 +136,7 @@ FEATURE_DISPLAY ?=              \
          libunwind              \
          libdw-dwarf-unwind     \
          libcapstone            \
+         llvm                   \

libllvm please

tools/perf/Makefile.config ifdef BUILD_NONDISTRO · line 972

Quoted 12 lines
tools/build/Makefile.feature +0 -0
          zlib                   \
          lzma                   \
          get_cpuid              \
tools/perf/Makefile.config +1 -0
index 7f1e016a9253..414a37f712bd 100644
@@ -969,6 +969,21 @@ ifdef BUILD_NONDISTRO
   endif
 endif
 
+ifndef NO_LLVM

NO_LIBLLVM

Quoted 22 lines
tools/perf/Makefile.config +14 -0
+  $(call feature_check,llvm)
+  ifeq ($(feature-llvm), 1)
+    CFLAGS += -DHAVE_LLVM_SUPPORT
+    CXXFLAGS += -DHAVE_LLVM_SUPPORT
+    CXXFLAGS += $(shell $(LLVM_CONFIG) --cxxflags)
+    LIBLLVM = $(shell $(LLVM_CONFIG) --libs all) $(shell $(LLVM_CONFIG) --system-libs)
+    EXTLIBS += -L$(shell $(LLVM_CONFIG) --libdir) $(LIBLLVM)
+    $(call detected,CONFIG_LLVM)
+  else
+    $(warning No libllvm found, slower source file resolution, please install llvm-devel/llvm-dev)
+    NO_LLVM := 1
+  endif
+endif
+
 ifndef NO_DEMANGLE
   $(call feature_check,cxa-demangle)
   ifeq ($(feature-cxa-demangle), 1)
tools/perf/builtin-version.c +0 -0
index 398aa53e9e2e..94d8e13a688f 100644

tools/perf/builtin-version.c static void library_status(void) · line 68

Quoted 4 lines
tools/perf/builtin-version.c +1 -0
@@ -65,6 +65,7 @@ static void library_status(void)
 	STATUS(HAVE_LIBBFD_SUPPORT, libbfd);
 	STATUS(HAVE_DEBUGINFOD_SUPPORT, debuginfod);
 	STATUS(HAVE_LIBELF_SUPPORT, libelf);
+	STATUS(HAVE_LLVM_SUPPORT, libllvm);

HAVE_LIBLLVM_SUPPORT to be consistent

Quoted 100 lines
tools/perf/builtin-version.c +0 -0
 	STATUS(HAVE_LIBNUMA_SUPPORT, libnuma);
 	STATUS(HAVE_LIBNUMA_SUPPORT, numa_num_possible_cpus);
 	STATUS(HAVE_LIBPERL_SUPPORT, libperl);
tools/perf/util/Build +1 -0
index da64efd8718f..32c4e5e634ed 100644
@@ -226,6 +226,7 @@ perf-$(CONFIG_CXX_DEMANGLE) += demangle-cxx.o
 perf-y += demangle-ocaml.o
 perf-y += demangle-java.o
 perf-y += demangle-rust.o
+perf-$(CONFIG_LLVM) += llvm-c-helpers.o
 
 ifdef CONFIG_JITDUMP
 perf-$(CONFIG_LIBELF) += jitdump.o
tools/perf/util/llvm-c-helpers.cpp new file +79 -0
new file mode 100644
index 000000000000..cde3f1f03af0
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Must come before the linux/compiler.h include, which defines several
+ * macros (e.g. noinline) that conflict with compiler builtins used
+ * by LLVM.
+ */
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wunused-parameter"  /* Needed for LLVM <= 15 */
+#include <llvm/DebugInfo/Symbolize/Symbolize.h>
+#pragma GCC diagnostic pop
+
+#include <stdio.h>
+#include <sys/types.h>
+#include <linux/compiler.h>
+#include "symbol_conf.h"
+#include "llvm-c-helpers.h"
+
+using namespace llvm;
+using llvm::symbolize::LLVMSymbolizer;
+
+/*
+ * Allocate a static LLVMSymbolizer, which will live to the end of the program.
+ * Unlike the bfd paths, LLVMSymbolizer has its own cache, so we do not need
+ * to store anything in the dso struct.
+ */
+static LLVMSymbolizer *get_symbolizer()
+{
+	static LLVMSymbolizer *instance = nullptr;
+	if (instance == nullptr) {
+		LLVMSymbolizer::Options opts;
+		/*
+		 * LLVM sometimes demangles slightly different from the rest
+		 * of the code, and this mismatch can cause new_inline_sym()
+		 * to get confused and mark non-inline symbol as inlined
+		 * (since the name does not properly match up with base_sym).
+		 * Thus, disable the demangling and let the rest of the code
+		 * handle it.
+		 */
+		opts.Demangle = false;
+		instance = new LLVMSymbolizer(opts);
+	}
+	return instance;
+}
+
+/* Returns 0 on error, 1 on success. */
+static int extract_file_and_line(const DILineInfo& line_info, char **file,
+				 unsigned int *line)
+{
+	if (file) {
+		if (line_info.FileName == "<invalid>") {
+			/* Match the convention of libbfd. */
+			*file = nullptr;
+		} else {
+			/* The caller expects to get something it can free(). */
+			*file = strdup(line_info.FileName.c_str());
+			if (*file == nullptr)
+				return 0;
+		}
+	}
+	if (line)
+		*line = line_info.Line;
+	return 1;
+}
+
+extern "C"
+int llvm_addr2line(const char *dso_name, u64 addr,
+		   char **file, unsigned int *line,
+		   bool unwind_inlines,
+		   llvm_a2l_frame** inline_frames)
+{
+	LLVMSymbolizer *symbolizer = get_symbolizer();
+	object::SectionedAddress sectioned_addr = {
+		addr,
+		object::SectionedAddress::UndefSection
+	};
+
+	if (unwind_inlines) {
+		Expected<DIInliningInfo> res_or_err =

tools/perf/util/llvm-c-helpers.cpp line 89

Quoted 10 lines
tools/perf/util/llvm-c-helpers.cpp new file +10 -0
+			symbolizer->symbolizeInlinedCode(dso_name,
+							 sectioned_addr);
+		if (!res_or_err)
+			return 0;
+		unsigned num_frames = res_or_err->getNumberOfFrames();
+		if (num_frames == 0)
+			return 0;
+
+		if (extract_file_and_line(
+			res_or_err->getFrame(0), file, line) == 0)

Please don't break this line or at least align 'file' with res_or_err.

tools/perf/util/llvm-c-helpers.cpp line 93

Quoted 4 lines
tools/perf/util/llvm-c-helpers.cpp new file +4 -0
+			return 0;
+
+		*inline_frames = (llvm_a2l_frame*)malloc(
+			sizeof(**inline_frames) * num_frames);

Do we really need to cast here (my C++ is super rusty), and why not calloc()?

tools/perf/util/llvm-c-helpers.cpp line 98

Quoted 5 lines
tools/perf/util/llvm-c-helpers.cpp new file +5 -0
+		if (*inline_frames == nullptr)
+			return 0;
+
+		for (unsigned i = 0; i < num_frames; ++i) {
+			const DILineInfo& src = res_or_err->getFrame(i);

Add blank line here

tools/perf/util/llvm-c-helpers.cpp line 105

Quoted 7 lines
tools/perf/util/llvm-c-helpers.cpp new file +7 -0
+			llvm_a2l_frame& dst = (*inline_frames)[i];
+			if (src.FileName == "<invalid>")
+				/* Match the convention of libbfd. */
+				dst.filename = nullptr;
+			else
+				dst.filename = strdup(src.FileName.c_str());
+			dst.funcname = strdup(src.FunctionName.c_str());

If strdup fails, users will cope?

tools/perf/util/llvm-c-helpers.cpp line 112

Quoted 7 lines
tools/perf/util/llvm-c-helpers.cpp new file +7 -0
+			dst.line = src.Line;
+
+			if (dst.filename == nullptr ||
+			    dst.funcname == nullptr) {
+				for (unsigned j = 0; j <= i; ++j) {
+					free((*inline_frames)[j].filename);
+					free((*inline_frames)[j].funcname);

Normally we use zfree()

tools/perf/util/llvm-c-helpers.cpp line 125

Quoted 13 lines
tools/perf/util/llvm-c-helpers.cpp new file +13 -0
+				}
+				free(*inline_frames);
+				return 0;
+			}
+		}
+
+		return num_frames;
+	} else {
+		if (inline_frames)
+			*inline_frames = nullptr;
+
+		Expected<DILineInfo> res_or_err =
+			symbolizer->symbolizeCode(dso_name, sectioned_addr);

same line?

Quoted 76 lines
tools/perf/util/llvm-c-helpers.cpp new file +5 -0
+		if (!res_or_err)
+			return 0;
+		return extract_file_and_line(*res_or_err, file, line);
+	}
+}
tools/perf/util/llvm-c-helpers.h new file +47 -0
new file mode 100644
index 000000000000..1b28cdc9f9b7
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __PERF_LLVM_C_HELPERS
+#define __PERF_LLVM_C_HELPERS 1
+
+/*
+ * Helpers to call into LLVM C++ code from C, for the parts that do not have
+ * C APIs.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct llvm_a2l_frame {
+	char *filename;
+	char *funcname;
+	unsigned int line;
+};
+
+/*
+ * Implement addr2line() using libLLVM. LLVM is a C++ API, and
+ * many of the linux/ headers cannot be included in a C++ compile unit,
+ * so we need to make a little bridge code here. llvm_addr2line() will
+ * convert the inline frame information from LLVM's internal structures
+ * and put them into a flat array given in inline_frames. The caller
+ * is then responsible for taking that array and convert it into perf's
+ * regular inline frame structures (which depend on e.g. struct list_head).
+ *
+ * If the address could not be resolved, or an error occurred (e.g. OOM),
+ * returns 0. Otherwise, returns the number of inline frames (which means 1
+ * if the address was not part of an inlined function). If unwind_inlines
+ * is set and the return code is nonzero, inline_frames will be set to
+ * a newly allocated array with that length. The caller is then responsible
+ * for freeing both the strings and the array itself.
+ */
+int llvm_addr2line(const char *dso_name,
+		   u64 addr,
+		   char **file,
+		   unsigned int *line,
+		   bool unwind_inlines,
+		   struct llvm_a2l_frame **inline_frames);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __PERF_LLVM_C_HELPERS */
tools/perf/util/srcline.c +3 -0
index 9d670d8c1c08..0505b4c16608 100644
@@ -16,6 +16,9 @@
 #include "util/debug.h"
 #include "util/callchain.h"
 #include "util/symbol_conf.h"
+#ifdef HAVE_LLVM_SUPPORT
+#include "util/llvm-c-helpers.h"
+#endif
 #include "srcline.h"
 #include "string2.h"
 #include "symbol.h"
@@ -130,7 +133,59 @@ static struct symbol *new_inline_sym(struct dso *dso,
 
 #define MAX_INLINE_NEST 1024
 

tools/perf/util/srcline.c static struct symbol *new_inline_sym(struct dso *dso, · line 144

Quoted 10 lines
tools/perf/util/srcline.c +9 -1
-#ifdef HAVE_LIBBFD_SUPPORT
+#ifdef HAVE_LLVM_SUPPORT
+
+static void free_llvm_inline_frames(struct llvm_a2l_frame *inline_frames,
+				    int num_frames)
+{
+	if (inline_frames != NULL) {
+		for (int i = 0; i < num_frames; ++i) {
+			free(inline_frames[i].filename);
+			free(inline_frames[i].funcname);

zfree()?

Quoted 16 lines
tools/perf/util/srcline.c +16 -0
+		}
+		free(inline_frames);
+	}
+}
+
+static int addr2line(const char *dso_name, u64 addr,
+		     char **file, unsigned int *line, struct dso *dso,
+		     bool unwind_inlines, struct inline_node *node,
+		     struct symbol *sym)
+{
+	struct llvm_a2l_frame *inline_frames = NULL;
+	int num_frames = llvm_addr2line(dso_name, addr, file, line,
+					node && unwind_inlines, &inline_frames);
+
+	if (num_frames == 0 || !inline_frames) {
+		/* Error, or we didn't want inlines. */

tools/perf/util/srcline.c static struct symbol *new_inline_sym(struct dso *dso, · line 170

Quoted 10 lines
tools/perf/util/srcline.c +10 -0
+		return num_frames;
+	}
+
+	for (int i = 0; i < num_frames; ++i) {
+		struct symbol *inline_sym =
+			new_inline_sym(dso, sym, inline_frames[i].funcname);
+		char *srcline = NULL;
+
+		if (inline_frames[i].filename)
+			srcline = srcline_from_fileline(

move first arg to same line of function call

Quoted 23 lines
tools/perf/util/srcline.c +18 -0
+				inline_frames[i].filename,
+				inline_frames[i].line);
+		if (inline_list__append(inline_sym, srcline, node) != 0) {
+			free_llvm_inline_frames(inline_frames, num_frames);
+			return 0;
+		}
+	}
+	free_llvm_inline_frames(inline_frames, num_frames);
+
+	return num_frames;
+}
+
+void dso__free_a2l(struct dso *)
+{
+	/* Nothing to free. */
+}
+
+#elif defined(HAVE_LIBBFD_SUPPORT)
 
 /*
  * Implement addr2line using libbfd.
Signature · 1 line
2.45.1
#4 lore raw
Quoted 5 lines · Arnaldo Carvalho de Melo

Makefile.config:982: No libllvm found, slower source file resolution, please install llvm-devel/llvm-dev

... llvm: [ OFF ]

But maybe use "libllvm"?

I think I was trying to be consistent with the previous patch using LLVM (something with Clang and eBPF, I think?), which used llvm and not libllvm for the name here :-) And I don't think upstream actually uses the libllvm name much (e.g. as you can see, it's llvm-devel, not libllvm-devel). But I can change it if you think it's a better name; I don't mind much either way.

Quoted 6 lines

So mostly the above, and:

root@x1:~# perf probe -x /usr/bin/find -L find | grep fts_read
     44        while ( (errno=0, ent=fts_read (p)) != NULL )
               /* fts_read returned NULL; distinguish between "finished" and "error". */
root@x1:~#

This part went a bit over my head, I'll just assume it's good. :-)

Quoted 2 lines

So the libllvm is even producing a better result, showing those inlines not seen in the BFD based output.

I think maybe you need to give --inlines to BFD objdump to get it to show inlines. (It can be useful in perf annotate sometimes.) But annoyingly enough, LLVM objdump does not understand the flag.

Quoted 1 line

100x speedup, looks like a win! 8-)

Great, thanks for testing. :-)

Quoted 4 lines

Thanks a lot, the comments I made on the patch are mostly coding style, please consider them, but I wouldn't get in the way for this patch to get merged because of that albeit would be nice to try to fit in more nicely with the existing source code base.

Sure, I'll make a v6 with the requested changes. Just some questions/comments below.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		*inline_frames = (llvm_a2l_frame*)malloc(
+			sizeof(**inline_frames) * num_frames);

Do we really need to cast here (my C++ is super rusty), and why not calloc()?

Yes, C++ does not have implicit cast-from-void*. The C++ way would be new[] or use std::vector, but, that would make it impossible for the caller to free(). I can use calloc, it won't change much either way.

Quoted 2 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+			dst.funcname = strdup(src.FunctionName.c_str());

If strdup fails, users will cope?

Yes, same as dst.filename. new_inline_sym() (which is where we give this data to) has an explicit check for if (!funcname).

Quoted 3 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		Expected<DILineInfo> res_or_err =
+			symbolizer->symbolizeCode(dso_name, sectioned_addr);

same line?

That takes it way over 80 characters, is that OK?

/* Steinar */