Fix 'faddr2line' for LLVM arm64 builds

GNU utilities cannot necessarily parse objects built by LLVM, which can result in confusing errors when using 'faddr2line':

$ CROSS_COMPILE=aarch64-linux-gnu- ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
do_one_initcall+0xf4/0x260:
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
$x.73 at main.c:?

Although this can be worked around by setting CROSS_COMPILE to "llvm=-", it's cleaner to follow the same syntax as the top-level Makefile and accept LLVM=1 as an indication to use the llvm- tools.

Cc Josh Poimboeuf <[email protected]>

Cc John Stultz <[email protected]>

Signed-off-by Will Deacon <[email protected]>

---
 scripts/faddr2line | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

scripts/faddr2line +8 -3

index 0e73aca4f908..62a3fa6f6f59 100755
@@ -58,8 +58,14 @@ die() {
        exit 1
 }
 
-READELF="${CROSS_COMPILE:-}readelf"
-ADDR2LINE="${CROSS_COMPILE:-}addr2line"
+if [ "${LLVM:-}" == "1" ]; then
+       UTIL_PREFIX=llvm-
+else
+       UTIL_PREFIX=${CROSS_COMPILE:-}
+fi
+
+READELF="${UTIL_PREFIX}readelf"
+ADDR2LINE="${UTIL_PREFIX}addr2line"
 AWK="awk"
 GREP="grep"
 
--

2.41.0.487.g6d72f3e995-goog

Quoted 6 lines · Will Deacon

GNU utilities cannot necessarily parse objects built by LLVM, which can result in confusing errors when using 'faddr2line':

$ CROSS_COMPILE=aarch64-linux-gnu- ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'

Quoted 1 line

aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25

Quoted 8 lines

do_one_initcall+0xf4/0x260:
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn' aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25 $x.73 at main.c:?

Although this can be worked around by setting CROSS_COMPILE to "llvm=-", it's cleaner to follow the same syntax as the top-level Makefile and accept LLVM=1 as an indication to use the llvm- tools.

Just a note.
The top Makefile accepts not only LLVM=1 but also LLVM=/usr/lib/llvm-16/bin/.
The latter is useful when you want to use a particular version or a custom one.

Another idea might be to use a generic '${prefix}' as this is not hooked to the Makefile,
but I do not have a strong opinion.

Quoted 30 lines

Cc Josh Poimboeuf <[email protected]>

Cc John Stultz <[email protected]>

Signed-off-by Will Deacon <[email protected]>

---
 scripts/faddr2line | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

scripts/faddr2line +8 -3

index 0e73aca4f908..62a3fa6f6f59 100755
@@ -58,8 +58,14 @@ die() {
        exit 1
 }
 
-READELF="${CROSS_COMPILE:-}readelf"
-ADDR2LINE="${CROSS_COMPILE:-}addr2line"
+if [ "${LLVM:-}" == "1" ]; then
+       UTIL_PREFIX=llvm-
+else
+       UTIL_PREFIX=${CROSS_COMPILE:-}
+fi
+
+READELF="${UTIL_PREFIX}readelf"
+ADDR2LINE="${UTIL_PREFIX}addr2line"
 AWK="awk"
 GREP="grep"
 
--

Quoted 2 lines

2.41.0.487.g6d72f3e995-goog

--
Best Regards
Masahiro Yamada

To prepare for 'faddr2line' reusing the same ignored symbols list as 'mksysmap', factor out the relevant sed expression into its own script, removing the double-escapes for '$' symbols as they are no longer required.

Cc Masahiro Yamada <[email protected]>

Cc Nathan Chancellor <[email protected]>

Cc Nick Desaulniers <[email protected]>

Cc Nicolas Schier <[email protected]>

Cc Josh Poimboeuf <[email protected]>

Cc John Stultz <[email protected]>

Cc [email protected]

Signed-off-by Will Deacon <[email protected]>

---

 scripts/mksysmap                | 77 +--------------------------------
 scripts/sysmap-ignored-syms.sed | 74 +++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+), 76 deletions(-)
 create mode 100644 scripts/sysmap-ignored-syms.sed

scripts/mksysmap +1 -76

index 9ba1c9da0a40..a98b34363258 100755
@@ -16,7 +16,7 @@
 # 'W' or 'w'.
 #
 
-${NM} -n ${1} | sed >${2} -e "
+${NM} -n ${1} | sed >${2} -f $(dirname $0)/sysmap-ignored-syms.sed -e "
 # ---------------------------------------------------------------------------
 # Ignored symbol types
 #

@@ -27,81 +27,6 @@ ${NM} -n ${1} | sed >${2} -e "
 # w: local weak symbols
 / [aNUw] /d
 
-# ---------------------------------------------------------------------------
-# Ignored prefixes
-#  (do not forget a space before each pattern)
-
-# local symbols for ARM, MIPS, etc.
-/ \\$/d
-
-# local labels, .LBB, .Ltmpxxx, .L__unnamed_xx, .LASANPC, etc.
-/ \.L/d
-
-# arm64 EFI stub namespace
-/ __efistub_/d
-
-# arm64 local symbols in PIE namespace
-/ __pi_\\$/d
-/ __pi_\.L/d
-
-# arm64 local symbols in non-VHE KVM namespace
-/ __kvm_nvhe_\\$/d
-/ __kvm_nvhe_\.L/d
-
-# arm64 lld
-/ __AArch64ADRPThunk_/d
-
-# arm lld
-/ __ARMV5PILongThunk_/d
-/ __ARMV7PILongThunk_/d
-/ __ThumbV7PILongThunk_/d
-
-# mips lld
-/ __LA25Thunk_/d
-/ __microLA25Thunk_/d
-
-# CFI type identifiers
-/ __kcfi_typeid_/d
-/ __kvm_nvhe___kcfi_typeid_/d
-/ __pi___kcfi_typeid_/d
-
-# CRC from modversions
-/ __crc_/d
-
-# EXPORT_SYMBOL (symbol name)
-/ __kstrtab_/d
-
-# EXPORT_SYMBOL (namespace)
-/ __kstrtabns_/d
-
-# ---------------------------------------------------------------------------
-# Ignored suffixes
-#  (do not forget '$' after each pattern)
-
-# arm
-/_from_arm$/d
-/_from_thumb$/d
-/_veneer$/d
-
-# ---------------------------------------------------------------------------
-# Ignored symbols (exact match)
-#  (do not forget a space before and '$' after each pattern)
-
-# for LoongArch?
-/ L0$/d
-
-# ppc
-/ _SDA_BASE_$/d
-/ _SDA2_BASE_$/d
-
-# ---------------------------------------------------------------------------
-# Ignored patterns
-#  (symbols that contain the pattern are ignored)
-
-# ppc stub
-/\.long_branch\./d
-/\.plt_branch\./d
-
 # ---------------------------------------------------------------------------
 # Ignored kallsyms symbols
 #

scripts/sysmap-ignored-syms.sed new file +74 -0

new file mode 100644
index 000000000000..14b9eb2c9ed9
@@ -0,0 +1,74 @@
+# ---------------------------------------------------------------------------
+# Ignored prefixes
+#  (do not forget a space before each pattern)
+
+# local symbols for ARM, MIPS, etc.
+/ \$/d
+
+# local labels, .LBB, .Ltmpxxx, .L__unnamed_xx, .LASANPC, etc.
+/ \.L/d
+
+# arm64 EFI stub namespace
+/ __efistub_/d
+
+# arm64 local symbols in PIE namespace
+/ __pi_\$/d
+/ __pi_\.L/d
+
+# arm64 local symbols in non-VHE KVM namespace
+/ __kvm_nvhe_\$/d
+/ __kvm_nvhe_\.L/d
+
+# arm64 lld
+/ __AArch64ADRPThunk_/d
+
+# arm lld
+/ __ARMV5PILongThunk_/d
+/ __ARMV7PILongThunk_/d
+/ __ThumbV7PILongThunk_/d
+
+# mips lld
+/ __LA25Thunk_/d
+/ __microLA25Thunk_/d
+
+# CFI type identifiers
+/ __kcfi_typeid_/d
+/ __kvm_nvhe___kcfi_typeid_/d
+/ __pi___kcfi_typeid_/d
+
+# CRC from modversions
+/ __crc_/d
+
+# EXPORT_SYMBOL (symbol name)
+/ __kstrtab_/d
+
+# EXPORT_SYMBOL (namespace)
+/ __kstrtabns_/d
+
+# ---------------------------------------------------------------------------
+# Ignored suffixes
+#  (do not forget '$' after each pattern)
+
+# arm
+/_from_arm$/d
+/_from_thumb$/d
+/_veneer$/d
+
+# ---------------------------------------------------------------------------
+# Ignored symbols (exact match)
+#  (do not forget a space before and '$' after each pattern)
+
+# for LoongArch?
+/ L0$/d
+
+# ppc
+/ _SDA_BASE_$/d
+/ _SDA2_BASE_$/d
+
+# ---------------------------------------------------------------------------
+# Ignored patterns
+#  (symbols that contain the pattern are ignored)
+
+# ppc stub
+/\.long_branch\./d
+/\.plt_branch\./d

Signature · 1 line

2.41.0.487.g6d72f3e995-goog

Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example:

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Adding a warning to do_one_initcall() results in:

| WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260

Which 'faddr2line' refuses to accept:

$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260

Filter out entries from readelf using the 'sysmap-ignored-syms.sed' script used to construct System.map, so that the size of a symbol is calculated as a delta to the next symbol present in ksymtab.

Cc Josh Poimboeuf <[email protected]>

Cc John Stultz <[email protected]>

Signed-off-by Will Deacon <[email protected]>

---
 scripts/faddr2line | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

scripts/faddr2line +2 -2

index 62a3fa6f6f59..da734af90036 100755
@@ -64,6 +64,7 @@ else
        UTIL_PREFIX=${CROSS_COMPILE:-}
 fi
 
+IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed
 READELF="${UTIL_PREFIX}readelf"
 ADDR2LINE="${UTIL_PREFIX}addr2line"
 AWK="awk"

@@ -185,7 +186,7 @@ __faddr2line() {
                                found=2
                                break
                        fi
-               done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
+               done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
 
                if [[ $found = 0 ]]; then
                        warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
--

2.41.0.487.g6d72f3e995-goog

Quoted 7 lines · Will Deacon

Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example:

Quoted 6 lines

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Sami,
Should we change the llvm-ir linkage type for these symbols from internal to private?
https://llvm.org/docs/LangRef.html#linkage-types

Then they would not appear in the symbol table.

At first, I thought other modules might need to directly reference this data, but with the local binding, I don't think they can.

Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example:

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Adding a warning to do_one_initcall() results in:

| WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260

Which 'faddr2line' refuses to accept:

$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260

Filter out entries from readelf using the 'sysmap-ignored-syms.sed' script used to construct System.map, so that the size of a symbol is calculated as a delta to the next symbol present in ksymtab.

Cc Josh Poimboeuf <[email protected]>

Cc John Stultz <[email protected]>

Signed-off-by Will Deacon <[email protected]>

---
 scripts/faddr2line | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

scripts/faddr2line +2 -2

index 62a3fa6f6f59..da734af90036 100755
@@ -64,6 +64,7 @@ else
        UTIL_PREFIX=${CROSS_COMPILE:-}
 fi
 
+IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed
 READELF="${UTIL_PREFIX}readelf"
 ADDR2LINE="${UTIL_PREFIX}addr2line"
 AWK="awk"

@@ -185,7 +186,7 @@ __faddr2line() {
                                found=2
                                break
                        fi
-               done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
+               done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
 
                if [[ $found = 0 ]]; then
                        warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
--

2.41.0.487.g6d72f3e995-goog

Quoted 20 lines · Will Deacon

Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example:

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Adding a warning to do_one_initcall() results in:

| WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260

Which 'faddr2line' refuses to accept:

Quoted 7 lines

$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260

Filter out entries from readelf using the 'sysmap-ignored-syms.sed' script used to construct System.map, so that the size of a symbol is calculated as a delta to the next symbol present in ksymtab.

I do not think this patch set is the right approach.

I assume faddr2line is meant to work with both vmlinux and modules.

A problem is that we have different filtering policies wrt kallsyms.

scripts/mksysmap filters symbols in vmlinux, while kernel/module/kallsyms.c filters ones in modules.

This patch tries to get aligned with the stacktrace of vmlinux, but that does not seem optimal to the stacktrace of modules.

I have not checked the details, but I guess
the module kallsyms filters less symbols.

https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288

I prefer filtering symbols in the intersection of vmlinux and modules.

is_mapping_symbol() filters symbols you are addressing.

Quoted 30 lines · Masahiro Yamada

Quoted 20 lines · Will Deacon

Some symbols emitted in the readelf output but filtered from System.map can confuse the 'faddr2line' symbol size calculation, resulting in the erroneous rejection of valid offsets. This is especially prevalent when building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are prefixed with a 32-bit data value in a '$d.n' section. For example:

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Adding a warning to do_one_initcall() results in:

| WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260

Which 'faddr2line' refuses to accept:

Quoted 7 lines

$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260

Filter out entries from readelf using the 'sysmap-ignored-syms.sed' script used to construct System.map, so that the size of a symbol is calculated as a delta to the next symbol present in ksymtab.

Quoted 4 lines

I do not think this patch set is the right approach.

I assume faddr2line is meant to work with both vmlinux and modules.

Huh, it seems to be busted for modules :/ I get:

| error: unknown argument '--section=.text'

with llvm and:

| addr2line: DWARF error: invalid or unhandled FORM value: 0x25

with binutils.

I'll look into this, as I don't think it's related to symbol filtering.

Quoted 2 lines

A problem is that we have different filtering policies wrt kallsyms.

Quoted 2 lines

scripts/mksysmap filters symbols in vmlinux, while kernel/module/kallsyms.c filters ones in modules.

I don't understand why we need two different ways of filtering out symbols, but it appears that the module case only filters out local labels and mapping symbols, both of which are filtered out of vmlinux as well. Is that right?

Quoted 7 lines

This patch tries to get aligned with the stacktrace of vmlinux, but that does not seem optimal to the stacktrace of modules.

I have not checked the details, but I guess
the module kallsyms filters less symbols.

Quoted 3 lines

https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288

I prefer filtering symbols in the intersection of vmlinux and modules.

I think mksysmap filters out a superset of the symbols which are filtered for modules, so why is the intersection the right thing to do? That will mean that faddr2line considers a whole bunch of symbols that aren't in the ksymtab of vmlinux.

Quoted 2 lines

is_mapping_symbol() filters symbols you are addressing.

That's a C function and faddr2line is a shell script. What exactly do you want me to do? My first hack just matched on symbols starting with '$' but I ended up with this after other review feedback.

Josh -- how do you want to proceed here?

Will

Problem is, I think the kernel's symbol printing code prints the nearest kallsyms symbol, and there are some valid non-FUNC code symbols. For example, syscall_return_via_sysret.