Fix duplicate caller frame in arm64 stack traces
Subsystems (6)
Andrii Nakryiko
maintainerv2 cover Fix duplicate caller frame in arm64 stack traces
On arm64 the caller is reported twice when the kernel builds a callchain for the current task from regs filled by perf_fetch_caller_regs(), instead of regs saved by a real exception. There was a similar problem on x86-64, but it was since fixed. Now it's time for arm64.
Patch 1 has the fix, patches 2-3 add BPF selftests for this.
I'm targeting bpf-next because of BPF selftests and to have BPF CI test everything. Patch 1 doesn't have to go through bpf-next, but it's simple enough, so if arm64 folks are ok with that, please ack and then we'll land everything in one go into bpf-next.
v1->v2:
- keep __builtin_frame_address(0) in perf_arch_fetch_caller_regs(), mark
these regs with PSR_MODE_EL1t instead, and have perf_callchain_kernel()
skip the duplicate leaf regs->pc for them, like x86 does with
perf_hw_regs() (Will);v1: https://lore.kernel.org/bpf/[email protected]/
Andrii Nakryiko (3):
arm64: perf: Don't report the caller twice in caller-regs callchains selftests/bpf: Add stacktrace_ips test for tracepoint selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64
arch/arm64/include/asm/perf_event.h | 2 +- arch/arm64/kernel/perf_callchain.c | 33 ++++++++- .../selftests/bpf/prog_tests/stacktrace_ips.c | 67 ++++++++++++++++++- .../selftests/bpf/progs/stacktrace_ips.c | 18 ++++- 4 files changed, 113 insertions(+), 7 deletions(-)
Signature · 1 line
2.53.1-Meta
Andrii Nakryiko
maintainerPATCH v2 1/3 arm64: perf: Don't report the caller twice in caller-regs callchains
perf_arch_fetch_caller_regs() sets regs->pc to the return address of the function it is used in (CALLER_ADDR0) and points regs->regs[29] at that function's own frame record, whose saved LR is that same return address. perf_callchain_kernel() reports regs->pc and then unwinds from the frame record at x29, so the caller shows up twice at the top of every kernel callchain built from perf_fetch_caller_regs(): tracepoint events, software events emitted by __perf_sw_event_sched() (context-switches, cgroup-switches, cpu-migrations), and bpf_get_stack() and bpf_get_stackid() in tracepoint, raw tracepoint, tp_btf and fentry/fexit programs. E.g., for a tracepoint program attached to sched_process_fork:
__traceiter_sched_process_fork+0x44
__traceiter_sched_process_fork+0x44
kernel_clone+0x250
...
x86 tells such regs apart in perf_callchain_kernel() with perf_hw_regs(): perf_arch_fetch_caller_regs() sets regs->flags to 0, while hardware-saved RFLAGS always has X86_EFLAGS_FIXED set, and regs->ip is not reported for regs that fail the check.
Do the same on arm64. Set pstate to PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leaf regs->pc in perf_callchain_kernel() for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.
regs->pstate of software events emitted by __perf_sw_event_sched(), which BPF perf_event programs see in bpf_perf_event_data::regs, changes from PSR_MODE_EL1h to PSR_MODE_EL1t.
x86 had the same duplicate entry, fixed by commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"").
--- arch/arm64/include/asm/perf_event.h | 2 +- arch/arm64/kernel/perf_callchain.c | 33 ++++++++++++++++++++++++++++- 2 files changed, 33 insertions(+), 2 deletions(-)
arch/arm64/include/asm/perf_event.h +1 -1
@@ -17,7 +17,7 @@ (regs)->pc = (__ip); \ (regs)->regs[29 ] = (unsigned long )__builtin_frame_address (0 ); \ (regs)->sp = current_stack_pointer; \ - (regs)->pstate = PSR_MODE_EL1h; \ + (regs)->pstate = PSR_MODE_EL1t; \ }#endif
arch/arm64/kernel/perf_callchain.c +32 -1
@@ -28,6 +28,31 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry,arch_stack_walk_user (callchain_trace, entry, regs); } +/* +* Determine whether the regs were taken from an exception handler rather than +* from perf_arch_fetch_caller_regs(). +*/ +static bool perf_hw_regs (struct pt_regs *regs) +{ +return (regs->pstate &PSR_MODE_MASK ) != PSR_MODE_EL1t; +} + +struct caller_regs_callchain { +struct perf_callchain_entry_ctx *entry; +bool skip_leaf; +}; + +static bool caller_regs_callchain_trace (void *data,unsigned long pc) +{ +struct caller_regs_callchain *cc = data; + +if (cc->skip_leaf) { + cc->skip_leaf = false; +return true; + } +return callchain_trace (cc->entry, pc); +} +void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs) {
@@ -36,5 +61,11 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,return ; } -arch_stack_walk (callchain_trace, entry, current, regs); +if (perf_hw_regs (regs)) { +arch_stack_walk (callchain_trace, entry, current, regs); + }else { +struct caller_regs_callchain cc = { .entry = entry, .skip_leaf = true }; + +arch_stack_walk (caller_regs_callchain_trace, &cc, current, regs); + } }
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
perf_arch_fetch_caller_regs()setsregs->pcto the return address of the function it is used in (CALLER_ADDR0) and points regs->regs[29] at that function's own frame record, whose saved LR is that same return address.perf_callchain_kernel()reportsregs->pcand then unwinds from the frame record at x29, so the caller shows up twice at the top of every kernel callchain built fromperf_fetch_caller_regs(): tracepoint events, software events emitted by__perf_sw_event_sched()(context-switches, cgroup-switches, cpu-migrations), andbpf_get_stack()andbpf_get_stackid()in tracepoint, raw tracepoint,tp_btfand fentry/fexit programs. E.g., for a tracepoint program attached tosched_process_fork:
__traceiter_sched_process_fork+0x44__traceiter_sched_process_fork+0x44
kernel_clone+0x250
[ ... ]
x86 had the same duplicate entry, fixed by commit6d08340d1e35("Revert "perf/x86: Always storeregs->ipinperf_callchain_kernel()"").
Quoted 2 lines
Fixes 5b09a094f2fb ("arm64: perf: Fix callchain parse error with kernel tracepoint events")Signed-off-by Andrii Nakryiko <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains
In reply to Andrii NakryikoQuoted 5 lines · Andrii Nakryiko
Do the same on arm64. Set pstate to
PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leafregs->pcinperf_callchain_kernel()for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.
my bot thinks the following will be a more accurate fix...
arch/arm64/include/asm/perf_event.h +2 -1
@@ -13,11 +13,12 @@#define perf_arch_bpf_user_pt_regs (regs) ®s->user_regs#endif +/* PSR_MODE_EL1t marks these regs for kunwind_stack_walk() */ #define perf_arch_fetch_caller_regs (regs, __ip) { \ (regs)->pc = (__ip); \ (regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \ (regs)->sp = current_stack_pointer; \ - (regs)->pstate = PSR_MODE_EL1h; \ + (regs)->pstate = PSR_MODE_EL1t; \ }#endif
arch/arm64/kernel/stacktrace.c +7 -0
@@ -357,6 +357,13 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,if (task != current)return -EINVAL ;kunwind_init_from_regs (&state, regs); +/* +* With perf_arch_fetch_caller_regs() the PC is a copy of the +* LR in the frame record at FP. Start from that record. +*/ +if (processor_mode (regs) == PSR_MODE_EL1t && +kunwind_next (&state)) +return -EINVAL ; }else if (task == current) {kunwind_init_from_caller (&state); }else {
Re: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains
Quoted 33 lines · Alexei Starovoitov
Quoted 5 lines · Andrii Nakryiko
Do the same on arm64. Set pstate to
PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leafregs->pcinperf_callchain_kernel()for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.my bot thinks the following will be a more accurate fix...
arch/arm64/include/asm/perf_event.h +2
-1@@ -13,11 +13,12 @@#define perf_arch_bpf_user_pt_regs (regs) ®s->user_regs#endif +/* PSR_MODE_EL1t marks these regs for kunwind_stack_walk() */ #define perf_arch_fetch_caller_regs (regs, __ip) { \ (regs)->pc = (__ip); \ (regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \ (regs)->sp = current_stack_pointer; \ - (regs)->pstate = PSR_MODE_EL1h; \ + (regs)->pstate = PSR_MODE_EL1t; \ }#endif arch/arm64/kernel/stacktrace.c +0
-0
arch/arm64/kernel/stacktrace.c kunwind_stack_walk(kunwind_consume_fn consume_state, · line 366
Quoted 10 lines
arch/arm64/kernel/stacktrace.c +7
-0@@ -357,6 +357,13 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,if (task != current)return -EINVAL ;kunwind_init_from_regs (&state, regs); +/* +* With perf_arch_fetch_caller_regs() the PC is a copy of the +* LR in the frame record at FP. Start from that record. +*/ +if (processor_mode (regs) == PSR_MODE_EL1t && +kunwind_next (&state)) +return -EINVAL ;
if arm folks approve, I'm all for this fix deeper in the generic unwinder code, smaller and cleaner change, thanks!
Quoted 4 lines
arch/arm64/kernel/stacktrace.c +0
-0}else if (task == current) {kunwind_init_from_caller (&state); }else {
Andrii Nakryiko
maintainerPATCH v2 2/3 selftests/bpf: Add stacktrace_ips test for tracepoint
In reply to Andrii NakryikoAdd a tp subtest that checks the stack captured by bpf_get_stack() from a tracepoint program, recorded only for the test's own process. Depending on static calls and inlining, the stack may or may not have __traceiter_bpf_testmod_test_read in front of bpf_testmod_test_read, so accept both. Also check that the entry after it differs: x86 kernels without commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"") report the tracepoint's caller twice.
--- .../selftests/bpf/prog_tests/stacktrace_ips.c | 54 +++++++++++++++++++ .../selftests/bpf/progs/stacktrace_ips.c | 13 +++++ 2 files changed, 67 insertions(+)
tools/testing/selftests/bpf/prog_tests/stacktrace_ips.c +54 -0
@@ -137,6 +137,58 @@ static void test_stacktrace_ips_raw_tp(void)stacktrace_ips__destroy (skel); } +static void test_stacktrace_ips_tp (void ) +{ +__u64 traceiter_addr, read_addr, *ips; +struct stacktrace_ips *skel; +struct ksyms *ksyms =NULL ; +struct ksym *ksym; +int i; + + skel =stacktrace_ips__open_and_load (); +if (!ASSERT_OK_PTR (skel,"stacktrace_ips__open_and_load" )) +return ; + +if (!skel->kconfig->CONFIG_UNWINDER_ORC) { +test__skip (); + goto cleanup; + } + + skel->bss->my_pid =getpid (); + + skel->links.tp_test =bpf_program__attach (skel->progs.tp_test); +if (!ASSERT_OK_PTR (skel->links.tp_test,"bpf_program__attach" )) + goto cleanup; + +trigger_module_test_read (1 ); + + ksyms =load_kallsyms_local (); +if (!ASSERT_OK_PTR (ksyms,"load_kallsyms_local" )) + goto cleanup; + traceiter_addr =ksym_get_addr_local (ksyms,"__traceiter_bpf_testmod_test_read" ); + read_addr =ksym_get_addr_local (ksyms,"bpf_testmod_test_read" ); + +/* +* Depending on static calls and inlining, __traceiter_bpf_testmod_test_read +* may or may not be in front of bpf_testmod_test_read. +*/ + ips = skel->bss->tp_ips; + ksym =ksym_search_local (ksyms, ips[0 ]); +if (!ASSERT_OK_PTR (ksym,"ksym_search_local" )) + goto cleanup; + i = ksym->addr == traceiter_addr ?1 :0 ; + + ksym =ksym_search_local (ksyms, ips[i]); +if (!ASSERT_OK_PTR (ksym,"ksym_search_local" )) + goto cleanup; +ASSERT_EQ (ksym->addr, read_addr,"stack_cmp" ); +ASSERT_NEQ (ips[i], ips[i +1 ],"stack_dup" ); + +cleanup: +free_kallsyms_local (ksyms); +stacktrace_ips__destroy (skel); +} +static void test_stacktrace_ips_kprobe (bool retprobe) { LIBBPF_OPTS(bpf_kprobe_opts, opts,
@@ -238,6 +290,8 @@ static void __test_stacktrace_ips(void)test_stacktrace_ips_kprobe_multi (true);if (test__start_subtest ("raw_tp" ))test_stacktrace_ips_raw_tp (); +if (test__start_subtest ("tp" )) +test_stacktrace_ips_tp ();if (test__start_subtest ("kprobe" ))test_stacktrace_ips_kprobe (false);if (test__start_subtest ("kretprobe" ))
tools/testing/selftests/bpf/progs/stacktrace_ips.c +13 -0
@@ -53,6 +53,19 @@ int rawtp_test(void *ctx)return 0 ; } +__u64 tp_ips[128 ]; +int my_pid; + +SEC("tp/bpf_testmod/bpf_testmod_test_read") +int tp_test (void *ctx) +{ +if (bpf_get_current_pid_tgid () >>32 != my_pid) +return 0 ; + +bpf_get_stack (ctx, tp_ips,sizeof (tp_ips),0 ); +return 0 ; +} + SEC("fentry/bpf_testmod_stacktrace_test")int fentry_test (struct pt_regs *ctx) {
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 2/3] selftests/bpf: Add stacktrace_ips test for tracepoint
Sashiko has reviewed this patch and found no issues.
Quoted 10 lines
Add a tp subtest that checks the stack captured by
bpf_get_stack()from a tracepoint program, recorded only for the test's own process. Depending on static calls and inlining, the stack may or may not have__traceiter_bpf_testmod_test_readin front ofbpf_testmod_test_read, so accept both. Also check that the entry after it differs: x86 kernels without commit6d08340d1e35("Revert "perf/x86: Always storeregs->ipinperf_callchain_kernel()"") report the tracepoint's caller twice.Signed-off-by Andrii Nakryiko <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Andrii Nakryiko
maintainerPATCH v2 3/3 selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64
In reply to Andrii NakryikoThe raw_tp, tp and fentry subtests check stacks unwound from perf_fetch_caller_regs() regs. Now that arm64 perf_callchain_kernel() skips regs->pc of such regs and no longer reports the caller twice, run these subtests on arm64 as well, where kernel stacks are always unwound through frame records.
Leave the kprobe, kretprobe, kprobe_multi, kretprobe_multi and fexit subtests x86-only: on arm64, kprobes at function entry miss the probed function's caller, and fexit stacks have an extra entry for the traced function. The kretprobe subtests pass on arm64, but don't use perf_fetch_caller_regs() regs.
--- .../selftests/bpf/prog_tests/stacktrace_ips.c | 15 +++++++++++---- .../testing/selftests/bpf/progs/stacktrace_ips.c | 5 +++-- 2 files changed, 14 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/prog_tests/stacktrace_ips.c +11 -4
@@ -2,7 +2,7 @@#include <test_progs.h> #include "stacktrace_ips.skel.h" -#ifdef __x86_64__ +#if defined(__x86_64__) || defined(__aarch64__)static int check_stacktrace_ips (int fd,__u32 key,int cnt, ...) {__u64 ips[PERF_MAX_STACK_DEPTH ];
@@ -47,6 +47,12 @@ static int check_stacktrace_ips(int fd, __u32 key, int cnt, ...)return err; } +/* arm64 always unwinds kernel stacks through frame records */ +static bool unwinder_ok (struct stacktrace_ips *skel) +{ +return skel->kconfig->CONFIG_UNWINDER_ORC || skel->kconfig->CONFIG_ARM64; +} +static void test_stacktrace_ips_kprobe_multi (bool retprobe) { LIBBPF_OPTS(bpf_kprobe_multi_opts, opts,
@@ -106,7 +112,7 @@ static void test_stacktrace_ips_raw_tp(void)if (!ASSERT_OK_PTR (skel,"stacktrace_ips__open_and_load" ))return ; -if (!skel->kconfig->CONFIG_UNWINDER_ORC) { +if (!unwinder_ok (skel)) {test__skip (); goto cleanup; }
@@ -149,7 +155,7 @@ static void test_stacktrace_ips_tp(void)if (!ASSERT_OK_PTR (skel,"stacktrace_ips__open_and_load" ))return ; -if (!skel->kconfig->CONFIG_UNWINDER_ORC) { +if (!unwinder_ok (skel)) {test__skip (); goto cleanup; }
@@ -244,7 +250,8 @@ static void test_stacktrace_ips_trampoline(bool retprobe)if (!ASSERT_OK_PTR (skel,"stacktrace_ips__open_and_load" ))return ; -if (!skel->kconfig->CONFIG_UNWINDER_ORC) { +/* on arm64, fexit stacks also have a frame for the traced function */ +if ((retprobe && !skel->kconfig->CONFIG_UNWINDER_ORC) || !unwinder_ok (skel)) {test__skip (); goto cleanup; }
tools/testing/selftests/bpf/progs/stacktrace_ips.c +3 -2
@@ -19,14 +19,15 @@ struct { } stackmap SEC(".maps"); extern bool CONFIG_UNWINDER_ORC __kconfig __weak; +extern bool CONFIG_ARM64 __kconfig __weak;/* -* This function is here to have CONFIG_UNWINDER_ORC +* This function is here to have CONFIG_UNWINDER_ORC and CONFIG_ARM64 * used and added to object BTF. */ int unused (void ) { -return CONFIG_UNWINDER_ORC ?0 :1 ; +return CONFIG_UNWINDER_ORC ||CONFIG_ARM64 ?0 :1 ; }__u32 stack_key;
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 3/3] selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64
Sashiko has reviewed this patch and found no issues.
Quoted 6 lines
The
raw_tp, tp and fentry subtests check stacks unwound fromperf_fetch_caller_regs()regs. Now that arm64perf_callchain_kernel()skipsregs->pcof such regs and no longer reports the caller twice, run these subtests on arm64 as well, where kernel stacks are always unwound through frame records.
Quoted 7 lines
Leave the kprobe, kretprobe,
kprobe_multi,kretprobe_multiand fexit subtests x86-only: on arm64, kprobes at function entry miss the probed function's caller, and fexit stacks have an extra entry for the traced function. The kretprobe subtests pass on arm64, but don't useperf_fetch_caller_regs()regs.Signed-off-by Andrii Nakryiko <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3