Skip to contentSkip to conversation

Fix duplicate caller frame in arm64 stack traces

PATCH In review Andrii Nakryiko 4 participants 9 messages Nov 19

Subsystems (6)
Kselftest BPF perf events arm64 +2 more subsystems
lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

v2 cover Fix duplicate caller frame in arm64 stack traces

On arm64 the caller is reported twice when the kernel builds a callchain for the current task from regs filled by perf_fetch_caller_regs(), instead of regs saved by a real exception. There was a similar problem on x86-64, but it was since fixed. Now it's time for arm64.

Patch 1 has the fix, patches 2-3 add BPF selftests for this.

I'm targeting bpf-next because of BPF selftests and to have BPF CI test everything. Patch 1 doesn't have to go through bpf-next, but it's simple enough, so if arm64 folks are ok with that, please ack and then we'll land everything in one go into bpf-next.

v1->v2:
- keep __builtin_frame_address(0) in perf_arch_fetch_caller_regs(), mark

    these regs with PSR_MODE_EL1t instead, and have perf_callchain_kernel()
    skip the duplicate leaf regs->pc for them, like x86 does with
    perf_hw_regs() (Will);

v1: https://lore.kernel.org/bpf/[email protected]/

Andrii Nakryiko (3):
arm64: perf: Don't report the caller twice in caller-regs callchains selftests/bpf: Add stacktrace_ips test for tracepoint selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64

 arch/arm64/include/asm/perf_event.h           |  2 +-
 arch/arm64/kernel/perf_callchain.c            | 33 ++++++++-
 .../selftests/bpf/prog_tests/stacktrace_ips.c | 67 ++++++++++++++++++-
 .../selftests/bpf/progs/stacktrace_ips.c      | 18 ++++-
 4 files changed, 113 insertions(+), 7 deletions(-)
Signature · 1 line
2.53.1-Meta
author maintainer
#1 lore raw

PATCH v2 1/3 arm64: perf: Don't report the caller twice in caller-regs callchains

perf_arch_fetch_caller_regs() sets regs->pc to the return address of the function it is used in (CALLER_ADDR0) and points regs->regs[29] at that function's own frame record, whose saved LR is that same return address. perf_callchain_kernel() reports regs->pc and then unwinds from the frame record at x29, so the caller shows up twice at the top of every kernel callchain built from perf_fetch_caller_regs(): tracepoint events, software events emitted by __perf_sw_event_sched() (context-switches, cgroup-switches, cpu-migrations), and bpf_get_stack() and bpf_get_stackid() in tracepoint, raw tracepoint, tp_btf and fentry/fexit programs. E.g., for a tracepoint program attached to sched_process_fork:

__traceiter_sched_process_fork+0x44
__traceiter_sched_process_fork+0x44
kernel_clone+0x250
...

x86 tells such regs apart in perf_callchain_kernel() with perf_hw_regs(): perf_arch_fetch_caller_regs() sets regs->flags to 0, while hardware-saved RFLAGS always has X86_EFLAGS_FIXED set, and regs->ip is not reported for regs that fail the check.

Do the same on arm64. Set pstate to PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leaf regs->pc in perf_callchain_kernel() for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.

regs->pstate of software events emitted by __perf_sw_event_sched(), which BPF perf_event programs see in bpf_perf_event_data::regs, changes from PSR_MODE_EL1h to PSR_MODE_EL1t.

x86 had the same duplicate entry, fixed by commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"").

Fixes 5b09a094f2fb ("arm64: perf: Fix callchain parse error with kernel tracepoint events")
---
 arch/arm64/include/asm/perf_event.h |  2 +-
 arch/arm64/kernel/perf_callchain.c  | 33 ++++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 2 deletions(-)
arch/arm64/include/asm/perf_event.h +1 -1
index ee45b4e77347..b3e7c24a5aab 100644
@@ -17,7 +17,7 @@
 	(regs)->pc = (__ip);    \
 	(regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \
 	(regs)->sp = current_stack_pointer; \
-	(regs)->pstate = PSR_MODE_EL1h;	\
+	(regs)->pstate = PSR_MODE_EL1t;	\
 }
 
 #endif
arch/arm64/kernel/perf_callchain.c +32 -1
index 9b7f26b128b5..3e956c4b0798 100644
@@ -28,6 +28,31 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
 	arch_stack_walk_user(callchain_trace, entry, regs);
 }
 
+/*
+ * Determine whether the regs were taken from an exception handler rather than
+ * from perf_arch_fetch_caller_regs().
+ */
+static bool perf_hw_regs(struct pt_regs *regs)
+{
+	return (regs->pstate & PSR_MODE_MASK) != PSR_MODE_EL1t;
+}
+
+struct caller_regs_callchain {
+	struct perf_callchain_entry_ctx *entry;
+	bool skip_leaf;
+};
+
+static bool caller_regs_callchain_trace(void *data, unsigned long pc)
+{
+	struct caller_regs_callchain *cc = data;
+
+	if (cc->skip_leaf) {
+		cc->skip_leaf = false;
+		return true;
+	}
+	return callchain_trace(cc->entry, pc);
+}
+
 void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 			   struct pt_regs *regs)
 {
@@ -36,5 +61,11 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 		return;
 	}
 
-	arch_stack_walk(callchain_trace, entry, current, regs);
+	if (perf_hw_regs(regs)) {
+		arch_stack_walk(callchain_trace, entry, current, regs);
+	} else {
+		struct caller_regs_callchain cc = { .entry = entry, .skip_leaf = true };
+
+		arch_stack_walk(caller_regs_callchain_trace, &cc, current, regs);
+	}
 }
Signature · 1 line
2.53.1-Meta
#2 lore raw

Re: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains

Sashiko has reviewed this patch and found no issues.
Quoted 19 lines

perf_arch_fetch_caller_regs() sets regs->pc to the return address of the function it is used in (CALLER_ADDR0) and points regs->regs[29] at that function's own frame record, whose saved LR is that same return address. perf_callchain_kernel() reports regs->pc and then unwinds from the frame record at x29, so the caller shows up twice at the top of every kernel callchain built from perf_fetch_caller_regs(): tracepoint events, software events emitted by __perf_sw_event_sched() (context-switches, cgroup-switches, cpu-migrations), and bpf_get_stack() and bpf_get_stackid() in tracepoint, raw tracepoint, tp_btf and fentry/fexit programs. E.g., for a tracepoint program attached to sched_process_fork:

__traceiter_sched_process_fork+0x44 __traceiter_sched_process_fork+0x44
kernel_clone+0x250
[ ... ]
x86 had the same duplicate entry, fixed by commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"").

Quoted 2 lines
Fixes 5b09a094f2fb ("arm64: perf: Fix callchain parse error with kernel tracepoint events")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains

In reply to Andrii Nakryiko
Quoted 5 lines · Andrii Nakryiko

Do the same on arm64. Set pstate to PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leaf regs->pc in perf_callchain_kernel() for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.

my bot thinks the following will be a more accurate fix...

arch/arm64/include/asm/perf_event.h +2 -1
index ee45b4e..7c5ab65 100644
@@ -13,11 +13,12 @@
 #define perf_arch_bpf_user_pt_regs(regs) &regs->user_regs
 #endif
 
+/* PSR_MODE_EL1t marks these regs for kunwind_stack_walk() */
 #define perf_arch_fetch_caller_regs(regs, __ip) { \
 	(regs)->pc = (__ip);    \
 	(regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \
 	(regs)->sp = current_stack_pointer; \
-	(regs)->pstate = PSR_MODE_EL1h;	\
+	(regs)->pstate = PSR_MODE_EL1t;	\
 }
 
 #endif
arch/arm64/kernel/stacktrace.c +7 -0
index 3ebcf8c..de394ff 100644
@@ -357,6 +357,13 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
 		if (task != current)
 			return -EINVAL;
 		kunwind_init_from_regs(&state, regs);
+		/*
+		 * With perf_arch_fetch_caller_regs() the PC is a copy of the
+		 * LR in the frame record at FP. Start from that record.
+		 */
+		if (processor_mode(regs) == PSR_MODE_EL1t &&
+		    kunwind_next(&state))
+			return -EINVAL;
 	} else if (task == current) {
 		kunwind_init_from_caller(&state);
 	} else {
#4 lore raw

Re: [PATCH bpf-next v2 1/3] arm64: perf: Don't report the caller twice in caller-regs callchains

Quoted 33 lines · Alexei Starovoitov
Quoted 5 lines · Andrii Nakryiko

Do the same on arm64. Set pstate to PSR_MODE_EL1t, which exceptions taken from the kernel never have (the kernel runs in EL1h, or EL2h with VHE), and skip the leaf regs->pc in perf_callchain_kernel() for regs with that mode. The unwind still starts from regs, with the frame record at x29 walked by the unwinder.

my bot thinks the following will be a more accurate fix...

arch/arm64/include/asm/perf_event.h +2 -1
index ee45b4e..7c5ab65 100644
@@ -13,11 +13,12 @@
 #define perf_arch_bpf_user_pt_regs(regs) &regs->user_regs
 #endif
 
+/* PSR_MODE_EL1t marks these regs for kunwind_stack_walk() */
 #define perf_arch_fetch_caller_regs(regs, __ip) { \
        (regs)->pc = (__ip);    \
        (regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \
        (regs)->sp = current_stack_pointer; \
-       (regs)->pstate = PSR_MODE_EL1h; \
+       (regs)->pstate = PSR_MODE_EL1t; \
 }
 
 #endif
arch/arm64/kernel/stacktrace.c +0 -0
index 3ebcf8c..de394ff 100644

arch/arm64/kernel/stacktrace.c kunwind_stack_walk(kunwind_consume_fn consume_state, · line 366

Quoted 10 lines
arch/arm64/kernel/stacktrace.c +7 -0
@@ -357,6 +357,13 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
                if (task != current)
                        return -EINVAL;
                kunwind_init_from_regs(&state, regs);
+               /*
+                * With perf_arch_fetch_caller_regs() the PC is a copy of the
+                * LR in the frame record at FP. Start from that record.
+                */
+               if (processor_mode(regs) == PSR_MODE_EL1t &&
+                   kunwind_next(&state))
+                       return -EINVAL;

if arm folks approve, I'm all for this fix deeper in the generic unwinder code, smaller and cleaner change, thanks!

Quoted 4 lines
arch/arm64/kernel/stacktrace.c +0 -0
        } else if (task == current) {
                kunwind_init_from_caller(&state);
        } else {
author maintainer
#5 lore raw

PATCH v2 2/3 selftests/bpf: Add stacktrace_ips test for tracepoint

In reply to Andrii Nakryiko

Add a tp subtest that checks the stack captured by bpf_get_stack() from a tracepoint program, recorded only for the test's own process. Depending on static calls and inlining, the stack may or may not have __traceiter_bpf_testmod_test_read in front of bpf_testmod_test_read, so accept both. Also check that the entry after it differs: x86 kernels without commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"") report the tracepoint's caller twice.

---
 .../selftests/bpf/prog_tests/stacktrace_ips.c | 54 +++++++++++++++++++
 .../selftests/bpf/progs/stacktrace_ips.c      | 13 +++++
 2 files changed, 67 insertions(+)
tools/testing/selftests/bpf/prog_tests/stacktrace_ips.c +54 -0
index da42b00e3d1f..a03d4c10c469 100644
@@ -137,6 +137,58 @@ static void test_stacktrace_ips_raw_tp(void)
 	stacktrace_ips__destroy(skel);
 }
 
+static void test_stacktrace_ips_tp(void)
+{
+	__u64 traceiter_addr, read_addr, *ips;
+	struct stacktrace_ips *skel;
+	struct ksyms *ksyms = NULL;
+	struct ksym *ksym;
+	int i;
+
+	skel = stacktrace_ips__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "stacktrace_ips__open_and_load"))
+		return;
+
+	if (!skel->kconfig->CONFIG_UNWINDER_ORC) {
+		test__skip();
+		goto cleanup;
+	}
+
+	skel->bss->my_pid = getpid();
+
+	skel->links.tp_test = bpf_program__attach(skel->progs.tp_test);
+	if (!ASSERT_OK_PTR(skel->links.tp_test, "bpf_program__attach"))
+		goto cleanup;
+
+	trigger_module_test_read(1);
+
+	ksyms = load_kallsyms_local();
+	if (!ASSERT_OK_PTR(ksyms, "load_kallsyms_local"))
+		goto cleanup;
+	traceiter_addr = ksym_get_addr_local(ksyms, "__traceiter_bpf_testmod_test_read");
+	read_addr = ksym_get_addr_local(ksyms, "bpf_testmod_test_read");
+
+	/*
+	 * Depending on static calls and inlining, __traceiter_bpf_testmod_test_read
+	 * may or may not be in front of bpf_testmod_test_read.
+	 */
+	ips = skel->bss->tp_ips;
+	ksym = ksym_search_local(ksyms, ips[0]);
+	if (!ASSERT_OK_PTR(ksym, "ksym_search_local"))
+		goto cleanup;
+	i = ksym->addr == traceiter_addr ? 1 : 0;
+
+	ksym = ksym_search_local(ksyms, ips[i]);
+	if (!ASSERT_OK_PTR(ksym, "ksym_search_local"))
+		goto cleanup;
+	ASSERT_EQ(ksym->addr, read_addr, "stack_cmp");
+	ASSERT_NEQ(ips[i], ips[i + 1], "stack_dup");
+
+cleanup:
+	free_kallsyms_local(ksyms);
+	stacktrace_ips__destroy(skel);
+}
+
 static void test_stacktrace_ips_kprobe(bool retprobe)
 {
 	LIBBPF_OPTS(bpf_kprobe_opts, opts,
@@ -238,6 +290,8 @@ static void __test_stacktrace_ips(void)
 		test_stacktrace_ips_kprobe_multi(true);
 	if (test__start_subtest("raw_tp"))
 		test_stacktrace_ips_raw_tp();
+	if (test__start_subtest("tp"))
+		test_stacktrace_ips_tp();
 	if (test__start_subtest("kprobe"))
 		test_stacktrace_ips_kprobe(false);
 	if (test__start_subtest("kretprobe"))
tools/testing/selftests/bpf/progs/stacktrace_ips.c +13 -0
index 6830f2978613..df10ef3cb14d 100644
@@ -53,6 +53,19 @@ int rawtp_test(void *ctx)
 	return 0;
 }
 
+__u64 tp_ips[128];
+int my_pid;
+
+SEC("tp/bpf_testmod/bpf_testmod_test_read")
+int tp_test(void *ctx)
+{
+	if (bpf_get_current_pid_tgid() >> 32 != my_pid)
+		return 0;
+
+	bpf_get_stack(ctx, tp_ips, sizeof(tp_ips), 0);
+	return 0;
+}
+
 SEC("fentry/bpf_testmod_stacktrace_test")
 int fentry_test(struct pt_regs *ctx)
 {
Signature · 1 line
2.53.1-Meta
#6 lore raw

Re: [PATCH bpf-next v2 2/3] selftests/bpf: Add stacktrace_ips test for tracepoint

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

Add a tp subtest that checks the stack captured by bpf_get_stack() from a tracepoint program, recorded only for the test's own process. Depending on static calls and inlining, the stack may or may not have __traceiter_bpf_testmod_test_read in front of bpf_testmod_test_read, so accept both. Also check that the entry after it differs: x86 kernels without commit 6d08340d1e35 ("Revert "perf/x86: Always store regs->ip in perf_callchain_kernel()"") report the tracepoint's caller twice.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author maintainer
#7 lore raw

PATCH v2 3/3 selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64

In reply to Andrii Nakryiko

The raw_tp, tp and fentry subtests check stacks unwound from perf_fetch_caller_regs() regs. Now that arm64 perf_callchain_kernel() skips regs->pc of such regs and no longer reports the caller twice, run these subtests on arm64 as well, where kernel stacks are always unwound through frame records.

Leave the kprobe, kretprobe, kprobe_multi, kretprobe_multi and fexit subtests x86-only: on arm64, kprobes at function entry miss the probed function's caller, and fexit stacks have an extra entry for the traced function. The kretprobe subtests pass on arm64, but don't use perf_fetch_caller_regs() regs.

---
 .../selftests/bpf/prog_tests/stacktrace_ips.c     | 15 +++++++++++----
 .../testing/selftests/bpf/progs/stacktrace_ips.c  |  5 +++--
 2 files changed, 14 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/prog_tests/stacktrace_ips.c +11 -4
index a03d4c10c469..aee46543f390 100644
@@ -2,7 +2,7 @@
 #include <test_progs.h>
 #include "stacktrace_ips.skel.h"
 
-#ifdef __x86_64__
+#if defined(__x86_64__) || defined(__aarch64__)
 static int check_stacktrace_ips(int fd, __u32 key, int cnt, ...)
 {
 	__u64 ips[PERF_MAX_STACK_DEPTH];
@@ -47,6 +47,12 @@ static int check_stacktrace_ips(int fd, __u32 key, int cnt, ...)
 	return err;
 }
 
+/* arm64 always unwinds kernel stacks through frame records */
+static bool unwinder_ok(struct stacktrace_ips *skel)
+{
+	return skel->kconfig->CONFIG_UNWINDER_ORC || skel->kconfig->CONFIG_ARM64;
+}
+
 static void test_stacktrace_ips_kprobe_multi(bool retprobe)
 {
 	LIBBPF_OPTS(bpf_kprobe_multi_opts, opts,
@@ -106,7 +112,7 @@ static void test_stacktrace_ips_raw_tp(void)
 	if (!ASSERT_OK_PTR(skel, "stacktrace_ips__open_and_load"))
 		return;
 
-	if (!skel->kconfig->CONFIG_UNWINDER_ORC) {
+	if (!unwinder_ok(skel)) {
 		test__skip();
 		goto cleanup;
 	}
@@ -149,7 +155,7 @@ static void test_stacktrace_ips_tp(void)
 	if (!ASSERT_OK_PTR(skel, "stacktrace_ips__open_and_load"))
 		return;
 
-	if (!skel->kconfig->CONFIG_UNWINDER_ORC) {
+	if (!unwinder_ok(skel)) {
 		test__skip();
 		goto cleanup;
 	}
@@ -244,7 +250,8 @@ static void test_stacktrace_ips_trampoline(bool retprobe)
 	if (!ASSERT_OK_PTR(skel, "stacktrace_ips__open_and_load"))
 		return;
 
-	if (!skel->kconfig->CONFIG_UNWINDER_ORC) {
+	/* on arm64, fexit stacks also have a frame for the traced function */
+	if ((retprobe && !skel->kconfig->CONFIG_UNWINDER_ORC) || !unwinder_ok(skel)) {
 		test__skip();
 		goto cleanup;
 	}
tools/testing/selftests/bpf/progs/stacktrace_ips.c +3 -2
index df10ef3cb14d..11dfebd4ddb2 100644
@@ -19,14 +19,15 @@ struct {
 } stackmap SEC(".maps");
 
 extern bool CONFIG_UNWINDER_ORC __kconfig __weak;
+extern bool CONFIG_ARM64 __kconfig __weak;
 
 /*
- * This function is here to have CONFIG_UNWINDER_ORC
+ * This function is here to have CONFIG_UNWINDER_ORC and CONFIG_ARM64
  * used and added to object BTF.
  */
 int unused(void)
 {
-	return CONFIG_UNWINDER_ORC ? 0 : 1;
+	return CONFIG_UNWINDER_ORC || CONFIG_ARM64 ? 0 : 1;
 }
 
 __u32 stack_key;
Signature · 1 line
2.53.1-Meta
#8 lore raw

Re: [PATCH bpf-next v2 3/3] selftests/bpf: Run stacktrace_ips raw_tp, tp and fentry tests on arm64

Sashiko has reviewed this patch and found no issues.
Quoted 6 lines

The raw_tp, tp and fentry subtests check stacks unwound from perf_fetch_caller_regs() regs. Now that arm64 perf_callchain_kernel() skips regs->pc of such regs and no longer reports the caller twice, run these subtests on arm64 as well, where kernel stacks are always unwound through frame records.

Quoted 7 lines

Leave the kprobe, kretprobe, kprobe_multi, kretprobe_multi and fexit subtests x86-only: on arm64, kprobes at function entry miss the probed function's caller, and fexit stacks have an extra entry for the traced function. The kretprobe subtests pass on arm64, but don't use perf_fetch_caller_regs() regs.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3