perf symbol: Do not use debug file as the binary type
Adrian Hunter
reviewerPATCH 1/1 perf symbol: Do not use debug file as the binary type
dso__load() sets the binary type of a DSO to the type of the first symbol source found. For a DSO with a separate debug file linked via .gnu-debuglink, that is DSO_BINARY_TYPE__DEBUGLINK, which makes dso__get_filename() return the name of the debug file instead of the file that was actually executed.
Consumers that need to read instruction bytes, such as Intel PT decoding in 'perf script', then read from the debug file and produce wrong instructions.
Prefer DSO_BINARY_TYPE__BUILD_ID_CACHE, and otherwise DSO_BINARY_TYPE__SYSTEM_PATH_DSO, over debug-only types, which restores the behaviour of using a file that contains the executed instructions.
This is a workaround. Properly separating the binary file used for instructions from the file used for debug symbols is left for later.
Example:
Create a shared object with a separate .gnu_debuglink debug file. Note that 'objcopy --only-keep-debug' leaves .text as NOBITS, so instructions read from the debug file are zeros:
# cat > foo.c << EOF
unsigned long foo_work(unsigned long n)
{
unsigned long s = 0; for (unsigned long i = 0; i < n; i++)
s = s * 31 + i;
return s;
}
EOF
# cat > main.c << EOF
#include <stdio.h>
unsigned long foo_work(unsigned long n);
int main(void)
{
printf("%lu\n", foo_work(1000));
return 0;
}
EOF
# gcc -g -O2 -shared -fPIC -o libfoo.so foo.c
# gcc -g -O2 -o main main.c -L. -lfoo -Wl,-rpath,'$ORIGIN'
# objcopy --only-keep-debug libfoo.so libfoo.so.debug
# objcopy --strip-debug libfoo.so
# objcopy --add-gnu-debuglink=libfoo.so.debug libfoo.so
# perf record -e intel_pt//u ./main Note that branch samples must be requested, because it is the resolving of the branch target symbol that causes dso__load() to be called, and hence the binary type to be set, before the decoder walks the code. With '--itrace=e' alone, nothing loads symbols for libfoo.so, the binary type is left as DSO_BINARY_TYPE__NOT_FOUND, the correct file is read anyway, and no errors are reported either way.
Before:
# perf.before script --itrace=be 2>&1 | grep "instruction trace error" instruction trace error type 1 time 2350.467489498 cpu 9 pid 75634 tid 75634 ip 0x77d48480718f code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467489832 cpu 9 pid 75634 tid 75634 ip 0x77d484807341 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467496412 cpu 9 pid 75634 tid 75634 ip 0x5b4de37a8074 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593393 cpu 9 pid 75634 tid 75634 ip 0x77d4848070d0 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593954 cpu 9 pid 75634 tid 75634 ip 0x77d4848075a8 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467595728 cpu 9 pid 75634 tid 75634 ip 0x77d4848324de code 6: Trace doesn't match instruction 6 instruction trace errors
After:
# perf script --itrace=be 2>&1 | grep "instruction trace error" #
--- tools/perf/util/symbol.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +10 -1
@@ -1947,7 +1947,16 @@ int dso__load(struct dso *dso, struct map *map)if (next_slot) { ss_pos++; -if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND ) +/* +* The binary type is used to find the file containing +* the executed instructions, so prefer the types that +* refer to the actual object over debug-only files such +* as DSO_BINARY_TYPE__DEBUGLINK. +*/ +if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND || + symtab_type ==DSO_BINARY_TYPE__BUILD_ID_CACHE || + (symtab_type ==DSO_BINARY_TYPE__SYSTEM_PATH_DSO && +dso__binary_type (dso) !=DSO_BINARY_TYPE__BUILD_ID_CACHE ))dso__set_binary_type (dso, symtab_type);if (syms_ss && runtime_ss)
Signature · 1 line
2.53.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The patch fails to apply the instruction trace fix to kernel modules, leaving them vulnerable to…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch fails to apply the instruction trace fix to kernel modules, leaving them vulnerable to the exact same decoding error. --
--- Patch [1]: [PATCH] perf symbol: Do not use debug file as the binary type --- commit af2af0a33f456210f477b124b3165ecd7e50b2ad
Author: Adrian Hunter <[email protected]>
perf symbol: Do not use debug file as the binary type
This commit updates dso__load() to prefer files that contain executed instructions over debug-only files (like .gnu-debuglink) for the binary type. It achieves this by explicitly favoring build ID cache files and system path DSOs over previously discovered debug files.
Quoted 9 lines
tools/perf/util/symbol.c +0
-1@@ -1947,7 +1947,16 @@ int dso__load(struct dso *dso, struct map *map)if (next_slot) { ss_pos++; -if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND )
tools/perf/util/symbol.c int dso__load(struct dso *dso, struct map *map) · line 1959
Quoted 10 lines
tools/perf/util/symbol.c +10
-0+/* +* The binary type is used to find the file containing +* the executed instructions, so prefer the types that +* refer to the actual object over debug-only files such +* as DSO_BINARY_TYPE__DEBUGLINK. +*/ +if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND || + symtab_type ==DSO_BINARY_TYPE__BUILD_ID_CACHE || + (symtab_type ==DSO_BINARY_TYPE__SYSTEM_PATH_DSO && +dso__binary_type (dso) !=DSO_BINARY_TYPE__BUILD_ID_CACHE ))
[Severity: Medium]
Does this new condition also need to check for kernel module object types like DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE?
If a trace contains kernel module execution and the module's debuginfo is cached in the perf build-id cache, dso__load() might encounter the debuginfo file first. When the actual kernel module binary is found later, the check above will fail because it strictly looks for DSO_BINARY_TYPE__SYSTEM_PATH_DSO.
Could this leave kernel modules vulnerable to the same decoding errors, where the debuginfo file is retained and produces wrong instructions during Intel PT decoding?
Quoted 3 lines
tools/perf/util/symbol.c +0
-0dso__set_binary_type (dso, symtab_type);if (syms_ss && runtime_ss)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerQuoted 70 lines · Adrian Hunter
dso__load()sets the binary type of a DSO to the type of the first symbol source found. For a DSO with a separate debug file linked via .gnu-debuglink, that isDSO_BINARY_TYPE__DEBUGLINK, which makesdso__get_filename()return the name of the debug file instead of the file that was actually executed.Consumers that need to read instruction bytes, such as Intel PT decoding in 'perf script', then read from the debug file and produce wrong instructions.
Prefer
DSO_BINARY_TYPE__BUILD_ID_CACHE, and otherwiseDSO_BINARY_TYPE__SYSTEM_PATH_DSO, over debug-only types, which restores the behaviour of using a file that contains the executed instructions.This is a workaround. Properly separating the binary file used for instructions from the file used for debug symbols is left for later.
Example:
Create a shared object with a separate .gnu_debuglink debug file. Note that 'objcopy --only-keep-debug' leaves .text as NOBITS, so instructions read from the debug file are zeros:
# cat > foo.c << EOF unsigned long foo_work(unsigned long n) { unsigned long s = 0;for (unsigned long i = 0; i < n; i++) s = s * 31 + i; return s; } EOF # cat > main.c << EOF #include <stdio.h> unsigned long foo_work(unsigned long n); int main(void) { printf("%lu\n", foo_work(1000)); return 0; } EOF # gcc -g -O2 -shared -fPIC -o libfoo.so foo.c # gcc -g -O2 -o main main.c -L. -lfoo -Wl,-rpath,'$ORIGIN' # objcopy --only-keep-debug libfoo.so libfoo.so.debug # objcopy --strip-debug libfoo.so # objcopy --add-gnu-debuglink=libfoo.so.debug libfoo.so # perf record -e intel_pt//u ./mainNote that branch samples must be requested, because it is the resolving of the branch target symbol that causes
dso__load()to be called, and hence the binary type to be set, before the decoder walks the code. With '--itrace=e' alone, nothing loads symbols for libfoo.so, the binary type is left asDSO_BINARY_TYPE__NOT_FOUND, the correct file is read anyway, and no errors are reported either way.Before:
# perf.before script --itrace=be 2>&1 | grep "instruction trace error" instruction trace error type 1 time 2350.467489498 cpu 9 pid 75634 tid 75634 ip 0x77d48480718f code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467489832 cpu 9 pid 75634 tid 75634 ip 0x77d484807341 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467496412 cpu 9 pid 75634 tid 75634 ip 0x5b4de37a8074 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593393 cpu 9 pid 75634 tid 75634 ip 0x77d4848070d0 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593954 cpu 9 pid 75634 tid 75634 ip 0x77d4848075a8 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467595728 cpu 9 pid 75634 tid 75634 ip 0x77d4848324de code 6: Trace doesn't match instruction 6 instruction trace errorsAfter:
Quoted 7 lines
# perf script --itrace=be 2>&1 | grep "instruction trace error" #Fixes 5363c306787c8 ("perf symbol: Set binary_type of dso when loading")Reported-by Todd Lipcon <[email protected]>Closes https://lore.kernel.org/all/CAGH6UiG=RJLqBU3kLu9XJciPyPO1HZkbAPERguVUMRuWQgqf=A@mail.gmail.com/ open hereSigned-off-by Adrian Hunter <[email protected]>
Just a heads up of a different using the ELF rather than the debug problem in addr2line caught in:
https://lore.kernel.org/linux-perf-users/[email protected]/
As you mention, there's probably a wider clean up needed. I dislike that we currently have dso__name and dso__long_name where the long name is generally the path to the binary. There should probably be a helper to get the DSO debuginfo for split cases.
Thanks,
Ian
Quoted 29 lines
--- tools/perf/util/symbol.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)tools/perf/util/symbol.c +10
-2@@ -1947,7 +1947,16 @@ int dso__load(struct dso *dso, struct map *map)if (next_slot) { ss_pos++; -if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND ) +/* +* The binary type is used to find the file containing +* the executed instructions, so prefer the types that +* refer to the actual object over debug-only files such +* as DSO_BINARY_TYPE__DEBUGLINK. +*/ +if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND || + symtab_type ==DSO_BINARY_TYPE__BUILD_ID_CACHE || + (symtab_type ==DSO_BINARY_TYPE__SYSTEM_PATH_DSO && +dso__binary_type (dso) !=DSO_BINARY_TYPE__BUILD_ID_CACHE ))dso__set_binary_type (dso, symtab_type);if (syms_ss && runtime_ss) --2.53.0
Namhyung Kim
maintainerHello,
Quoted 69 lines · Adrian Hunter
dso__load()sets the binary type of a DSO to the type of the first symbol source found. For a DSO with a separate debug file linked via .gnu-debuglink, that isDSO_BINARY_TYPE__DEBUGLINK, which makesdso__get_filename()return the name of the debug file instead of the file that was actually executed.Consumers that need to read instruction bytes, such as Intel PT decoding in 'perf script', then read from the debug file and produce wrong instructions.
Prefer
DSO_BINARY_TYPE__BUILD_ID_CACHE, and otherwiseDSO_BINARY_TYPE__SYSTEM_PATH_DSO, over debug-only types, which restores the behaviour of using a file that contains the executed instructions.This is a workaround. Properly separating the binary file used for instructions from the file used for debug symbols is left for later.
Example:
Create a shared object with a separate .gnu_debuglink debug file. Note that 'objcopy --only-keep-debug' leaves .text as NOBITS, so instructions read from the debug file are zeros:
# cat > foo.c << EOF unsigned long foo_work(unsigned long n) { unsigned long s = 0;for (unsigned long i = 0; i < n; i++) s = s * 31 + i; return s; } EOF # cat > main.c << EOF #include <stdio.h> unsigned long foo_work(unsigned long n); int main(void) { printf("%lu\n", foo_work(1000)); return 0; } EOF # gcc -g -O2 -shared -fPIC -o libfoo.so foo.c # gcc -g -O2 -o main main.c -L. -lfoo -Wl,-rpath,'$ORIGIN' # objcopy --only-keep-debug libfoo.so libfoo.so.debug # objcopy --strip-debug libfoo.so # objcopy --add-gnu-debuglink=libfoo.so.debug libfoo.so # perf record -e intel_pt//u ./mainNote that branch samples must be requested, because it is the resolving of the branch target symbol that causes
dso__load()to be called, and hence the binary type to be set, before the decoder walks the code. With '--itrace=e' alone, nothing loads symbols for libfoo.so, the binary type is left asDSO_BINARY_TYPE__NOT_FOUND, the correct file is read anyway, and no errors are reported either way.Before:
# perf.before script --itrace=be 2>&1 | grep "instruction trace error" instruction trace error type 1 time 2350.467489498 cpu 9 pid 75634 tid 75634 ip 0x77d48480718f code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467489832 cpu 9 pid 75634 tid 75634 ip 0x77d484807341 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467496412 cpu 9 pid 75634 tid 75634 ip 0x5b4de37a8074 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593393 cpu 9 pid 75634 tid 75634 ip 0x77d4848070d0 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593954 cpu 9 pid 75634 tid 75634 ip 0x77d4848075a8 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467595728 cpu 9 pid 75634 tid 75634 ip 0x77d4848324de code 6: Trace doesn't match instruction 6 instruction trace errorsAfter:
Quoted 7 lines
# perf script --itrace=be 2>&1 | grep "instruction trace error" #Fixes 5363c306787c8 ("perf symbol: Set binary_type of dso when loading")Reported-by Todd Lipcon <[email protected]>Closes https://lore.kernel.org/all/CAGH6UiG=RJLqBU3kLu9XJciPyPO1HZkbAPERguVUMRuWQgqf=A@mail.gmail.com/ open hereSigned-off-by Adrian Hunter <[email protected]>
Thanks for the patch and a test case.
Arnaldo, I can take this to the perf-tools tree for v7.3.
Thanks,
Namhyung
Quoted 29 lines
--- tools/perf/util/symbol.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)tools/perf/util/symbol.c +10
-1@@ -1947,7 +1947,16 @@ int dso__load(struct dso *dso, struct map *map)if (next_slot) { ss_pos++; -if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND ) +/* +* The binary type is used to find the file containing +* the executed instructions, so prefer the types that +* refer to the actual object over debug-only files such +* as DSO_BINARY_TYPE__DEBUGLINK. +*/ +if (dso__binary_type (dso) ==DSO_BINARY_TYPE__NOT_FOUND || + symtab_type ==DSO_BINARY_TYPE__BUILD_ID_CACHE || + (symtab_type ==DSO_BINARY_TYPE__SYSTEM_PATH_DSO && +dso__binary_type (dso) !=DSO_BINARY_TYPE__BUILD_ID_CACHE ))dso__set_binary_type (dso, symtab_type);if (syms_ss && runtime_ss)Signature · 1 line
2.53.0
Arnaldo Carvalho de Melo
maintainerQuoted 75 lines · Namhyung Kim
Hello,
Quoted 72 lines · Adrian Hunter
dso__load()sets the binary type of a DSO to the type of the first symbol source found. For a DSO with a separate debug file linked via .gnu-debuglink, that isDSO_BINARY_TYPE__DEBUGLINK, which makesdso__get_filename()return the name of the debug file instead of the file that was actually executed.Consumers that need to read instruction bytes, such as Intel PT decoding in 'perf script', then read from the debug file and produce wrong instructions.
Prefer
DSO_BINARY_TYPE__BUILD_ID_CACHE, and otherwiseDSO_BINARY_TYPE__SYSTEM_PATH_DSO, over debug-only types, which restores the behaviour of using a file that contains the executed instructions.This is a workaround. Properly separating the binary file used for instructions from the file used for debug symbols is left for later.
Example:
Create a shared object with a separate .gnu_debuglink debug file. Note that 'objcopy --only-keep-debug' leaves .text as NOBITS, so instructions read from the debug file are zeros:
# cat > foo.c << EOF unsigned long foo_work(unsigned long n) { unsigned long s = 0;for (unsigned long i = 0; i < n; i++) s = s * 31 + i; return s; } EOF # cat > main.c << EOF #include <stdio.h> unsigned long foo_work(unsigned long n); int main(void) { printf("%lu\n", foo_work(1000)); return 0; } EOF # gcc -g -O2 -shared -fPIC -o libfoo.so foo.c # gcc -g -O2 -o main main.c -L. -lfoo -Wl,-rpath,'$ORIGIN' # objcopy --only-keep-debug libfoo.so libfoo.so.debug # objcopy --strip-debug libfoo.so # objcopy --add-gnu-debuglink=libfoo.so.debug libfoo.so # perf record -e intel_pt//u ./mainNote that branch samples must be requested, because it is the resolving of the branch target symbol that causes
dso__load()to be called, and hence the binary type to be set, before the decoder walks the code. With '--itrace=e' alone, nothing loads symbols for libfoo.so, the binary type is left asDSO_BINARY_TYPE__NOT_FOUND, the correct file is read anyway, and no errors are reported either way.Before:
# perf.before script --itrace=be 2>&1 | grep "instruction trace error" instruction trace error type 1 time 2350.467489498 cpu 9 pid 75634 tid 75634 ip 0x77d48480718f code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467489832 cpu 9 pid 75634 tid 75634 ip 0x77d484807341 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467496412 cpu 9 pid 75634 tid 75634 ip 0x5b4de37a8074 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593393 cpu 9 pid 75634 tid 75634 ip 0x77d4848070d0 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467593954 cpu 9 pid 75634 tid 75634 ip 0x77d4848075a8 code 6: Trace doesn't match instruction instruction trace error type 1 time 2350.467595728 cpu 9 pid 75634 tid 75634 ip 0x77d4848324de code 6: Trace doesn't match instruction 6 instruction trace errorsAfter:
# perf script --itrace=be 2>&1 | grep "instruction trace error" #
Quoted 8 lines
Quoted 4 lines
Fixes 5363c306787c8 ("perf symbol: Set binary_type of dso when loading")Reported-by Todd Lipcon <[email protected]>Closes https://lore.kernel.org/all/CAGH6UiG=RJLqBU3kLu9XJciPyPO1HZkbAPERguVUMRuWQgqf=A@mail.gmail.com/ open hereSigned-off-by Adrian Hunter <[email protected]>Thanks for the patch and a test case.
Arnaldo, I can take this to the perf-tools tree for v7.3.
Ok!
- Arnaldo
Quoted 11 lines · Adrian Hunter
dso__load()sets the binary type of a DSO to the type of the first symbol source found. For a DSO with a separate debug file linked via .gnu-debuglink, that isDSO_BINARY_TYPE__DEBUGLINK, which makesdso__get_filename()return the name of the debug file instead of the file that was actually executed.Consumers that need to read instruction bytes, such as Intel PT decoding in 'perf script', then read from the debug file and produce wrong instructions.
[...]
Applied to perf-tools, thanks!
Best regards,
Namhyung