Skip to contentSkip to conversation

bpf, arm64: Support Exceptions

PATCH Superseded Puranjay Mohan 5 participants 12 messages Sep 17, 2023 – Nov 16, 2023

Subsystems (5)
Kselftest BPF arm64 BPF selftests +1 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover bpf, arm64: Support Exceptions

Changes in V1->V2:
V1: https://lore.kernel.org/all/[email protected]/
- Remove exceptions from DENYLIST.aarch64 as they are supported now.

The base support for exceptions was merged with [1] and it was enabled for x86-64.

This patch enables the support on ARM64, all sefltests are passing:

# ./test_progs -a exceptions
#74/1    exceptions/exception_throw_always_1:OK
#74/2    exceptions/exception_throw_always_2:OK
#74/3    exceptions/exception_throw_unwind_1:OK
#74/4    exceptions/exception_throw_unwind_2:OK
#74/5    exceptions/exception_throw_default:OK
#74/6    exceptions/exception_throw_default_value:OK
#74/7    exceptions/exception_tail_call:OK
#74/8    exceptions/exception_ext:OK
#74/9    exceptions/exception_ext_mod_cb_runtime:OK
#74/10   exceptions/exception_throw_subprog:OK
#74/11   exceptions/exception_assert_nz_gfunc:OK
#74/12   exceptions/exception_assert_zero_gfunc:OK
#74/13   exceptions/exception_assert_neg_gfunc:OK
#74/14   exceptions/exception_assert_pos_gfunc:OK
#74/15   exceptions/exception_assert_negeq_gfunc:OK
#74/16   exceptions/exception_assert_poseq_gfunc:OK
#74/17   exceptions/exception_assert_nz_gfunc_with:OK
#74/18   exceptions/exception_assert_zero_gfunc_with:OK
#74/19   exceptions/exception_assert_neg_gfunc_with:OK
#74/20   exceptions/exception_assert_pos_gfunc_with:OK
#74/21   exceptions/exception_assert_negeq_gfunc_with:OK
#74/22   exceptions/exception_assert_poseq_gfunc_with:OK
#74/23   exceptions/exception_bad_assert_nz_gfunc:OK
#74/24   exceptions/exception_bad_assert_zero_gfunc:OK
#74/25   exceptions/exception_bad_assert_neg_gfunc:OK
#74/26   exceptions/exception_bad_assert_pos_gfunc:OK
#74/27   exceptions/exception_bad_assert_negeq_gfunc:OK
#74/28   exceptions/exception_bad_assert_poseq_gfunc:OK
#74/29   exceptions/exception_bad_assert_nz_gfunc_with:OK
#74/30   exceptions/exception_bad_assert_zero_gfunc_with:OK
#74/31   exceptions/exception_bad_assert_neg_gfunc_with:OK
#74/32   exceptions/exception_bad_assert_pos_gfunc_with:OK
#74/33   exceptions/exception_bad_assert_negeq_gfunc_with:OK
#74/34   exceptions/exception_bad_assert_poseq_gfunc_with:OK
#74/35   exceptions/exception_assert_range:OK
#74/36   exceptions/exception_assert_range_with:OK
#74/37   exceptions/exception_bad_assert_range:OK
#74/38   exceptions/exception_bad_assert_range_with:OK
#74/39   exceptions/non-throwing fentry -> exception_cb:OK
#74/40   exceptions/throwing fentry -> exception_cb:OK
#74/41   exceptions/non-throwing fexit -> exception_cb:OK
#74/42   exceptions/throwing fexit -> exception_cb:OK
#74/43   exceptions/throwing extension (with custom cb) -> exception_cb:OK
#74/44   exceptions/throwing extension -> global func in exception_cb:OK
#74/45   exceptions/exception_ext_mod_cb_runtime:OK
#74/46   exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
#74/47   exceptions/exception_ext:OK
#74/48   exceptions/non-throwing fentry -> non-throwing subprog:OK
#74/49   exceptions/throwing fentry -> non-throwing subprog:OK
#74/50   exceptions/non-throwing fentry -> throwing subprog:OK
#74/51   exceptions/throwing fentry -> throwing subprog:OK
#74/52   exceptions/non-throwing fexit -> non-throwing subprog:OK
#74/53   exceptions/throwing fexit -> non-throwing subprog:OK
#74/54   exceptions/non-throwing fexit -> throwing subprog:OK
#74/55   exceptions/throwing fexit -> throwing subprog:OK
#74/56   exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
#74/57   exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
#74/58   exceptions/non-throwing extension -> non-throwing subprog:OK
#74/59   exceptions/non-throwing extension -> throwing subprog:OK
#74/60   exceptions/non-throwing extension -> non-throwing subprog:OK
#74/61   exceptions/non-throwing extension -> throwing global subprog:OK
#74/62   exceptions/throwing extension -> throwing global subprog:OK
#74/63   exceptions/throwing extension -> non-throwing global subprog:OK
#74/64   exceptions/non-throwing extension -> main subprog:OK
#74/65   exceptions/throwing extension -> main subprog:OK
#74/66   exceptions/reject_exception_cb_type_1:OK
#74/67   exceptions/reject_exception_cb_type_2:OK
#74/68   exceptions/reject_exception_cb_type_3:OK
#74/69   exceptions/reject_exception_cb_type_4:OK
#74/70   exceptions/reject_async_callback_throw:OK
#74/71   exceptions/reject_with_lock:OK
#74/72   exceptions/reject_subprog_with_lock:OK
#74/73   exceptions/reject_with_rcu_read_lock:OK
#74/74   exceptions/reject_subprog_with_rcu_read_lock:OK
#74/75   exceptions/reject_with_rbtree_add_throw:OK
#74/76   exceptions/reject_with_reference:OK
#74/77   exceptions/reject_with_cb_reference:OK
#74/78   exceptions/reject_with_cb:OK
#74/79   exceptions/reject_with_subprog_reference:OK
#74/80   exceptions/reject_throwing_exception_cb:OK
#74/81   exceptions/reject_exception_cb_call_global_func:OK
#74/82   exceptions/reject_exception_cb_call_static_func:OK
#74/83   exceptions/reject_multiple_exception_cb:OK
#74/84   exceptions/reject_exception_throw_cb:OK
#74/85   exceptions/reject_exception_throw_cb_diff:OK
#74/86   exceptions/reject_set_exception_cb_bad_ret1:OK
#74/87   exceptions/reject_set_exception_cb_bad_ret2:OK
#74/88   exceptions/check_assert_eq_int_min:OK
#74/89   exceptions/check_assert_eq_int_max:OK
#74/90   exceptions/check_assert_eq_zero:OK
#74/91   exceptions/check_assert_eq_llong_min:OK
#74/92   exceptions/check_assert_eq_llong_max:OK
#74/93   exceptions/check_assert_lt_pos:OK
#74/94   exceptions/check_assert_lt_zero:OK
#74/95   exceptions/check_assert_lt_neg:OK
#74/96   exceptions/check_assert_le_pos:OK
#74/97   exceptions/check_assert_le_zero:OK
#74/98   exceptions/check_assert_le_neg:OK
#74/99   exceptions/check_assert_gt_pos:OK
#74/100  exceptions/check_assert_gt_zero:OK
#74/101  exceptions/check_assert_gt_neg:OK
#74/102  exceptions/check_assert_ge_pos:OK
#74/103  exceptions/check_assert_ge_zero:OK
#74/104  exceptions/check_assert_ge_neg:OK
#74/105  exceptions/check_assert_range_s64:OK
#74/106  exceptions/check_assert_range_u64:OK
#74/107  exceptions/check_assert_single_range_s64:OK
#74/108  exceptions/check_assert_single_range_u64:OK
#74/109  exceptions/check_assert_generic:OK
#74/110  exceptions/check_assert_with_return:OK
#74      exceptions:OK
Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED

[1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?h=for-next&id=ec6f1b4db95b7eedb3fe85f4f14e08fa0e9281c3

Puranjay Mohan (1):
bpf, arm64: support exceptions

 arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
 tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
 2 files changed, 79 insertions(+), 20 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH v2 1/1 bpf, arm64: support exceptions

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---
 arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
 tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
 2 files changed, 79 insertions(+), 20 deletions(-)
arch/arm64/net/bpf_jit_comp.c +79 -19
index 7d4af64e3982..fcc55e558863 100644
@@ -21,6 +21,7 @@
 #include <asm/insn.h>
 #include <asm/patching.h>
 #include <asm/set_memory.h>
+#include <asm/stacktrace.h>
 
 #include "bpf_jit.h"
 
@@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale)
 /* Tail call offset to jump into */
 #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
 
-static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
+static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb)
 {
 	const struct bpf_prog *prog = ctx->prog;
 	const bool is_main_prog = !bpf_is_subprog(prog);
@@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
 	emit(A64_NOP, ctx);
 
-	/* Sign lr */
-	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
-		emit(A64_PACIASP, ctx);
-
-	/* Save FP and LR registers to stay align with ARM64 AAPCS */
-	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
-	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
-
-	/* Save callee-saved registers */
-	emit(A64_PUSH(r6, r7, A64_SP), ctx);
-	emit(A64_PUSH(r8, r9, A64_SP), ctx);
-	emit(A64_PUSH(fp, tcc, A64_SP), ctx);
-	emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	if (!is_exception_cb) {
+		/* Sign lr */
+		if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
+			emit(A64_PACIASP, ctx);
+		/* Save FP and LR registers to stay align with ARM64 AAPCS */
+		emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
+		/* Save callee-saved registers */
+		emit(A64_PUSH(r6, r7, A64_SP), ctx);
+		emit(A64_PUSH(r8, r9, A64_SP), ctx);
+		emit(A64_PUSH(fp, tcc, A64_SP), ctx);
+		emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	} else {
+		/* Exception callback receives FP of Main Program as third parameter */
+		emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
+		/*
+		 * Main Program already pushed the frame record and the callee-saved registers. The
+		 * exception callback will not push anything and re-use the main program's stack.
+		 */
+		emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */
+	}
 
 	/* Set up BPF prog stack base register */
 	emit(A64_MOV(1, fp, A64_SP), ctx);
@@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 		emit_bti(A64_BTI_J, ctx);
 	}
 
+	/*
+	 * Program acting as exception boundary should save all ARM64 Callee-saved registers as the
+	 * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue.
+	 */
+	if (prog->aux->exception_boundary)
+		emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);
+
 	emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
 
 	/* Stack must be multiples of 16B */
@@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx)
 		plt->target = (u64)&dummy_tramp;
 }
 
-static void build_epilogue(struct jit_ctx *ctx)
+static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
 {
 	const u8 r0 = bpf2a64[BPF_REG_0];
 	const u8 r6 = bpf2a64[BPF_REG_6];
@@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx)
 	/* We're done with BPF stack */
 	emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
+	/*
+	 * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved
+	 * registers. Exception callback uses the boundary program's stack frame, so recover these
+	 * extra registers in the above two cases.
+	 */
+	if (ctx->prog->aux->exception_boundary || is_exception_cb)
+		emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
+
 	/* Restore x27 and x28 */
 	emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
 	/* Restore fs (x25) and x26 */
@@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * BPF line info needs ctx->offset[i] to be the offset of
 	 * instruction[i] in jited image, so build prologue first.
 	 */
-	if (build_prologue(&ctx, was_classic)) {
+	if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
 		prog = orig_prog;
 		goto out_off;
 	}
@@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	}
 
 	ctx.epilogue_offset = ctx.idx;
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
 	build_plt(&ctx);
 
 	extable_align = __alignof__(struct exception_table_entry);
@@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.idx = 0;
 	ctx.exentry_idx = 0;
 
-	build_prologue(&ctx, was_classic);
+	build_prologue(&ctx, was_classic, prog->aux->exception_cb);
 
 	if (build_body(&ctx, extra_pass)) {
 		bpf_jit_binary_free(header);
@@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		goto out_off;
 	}
 
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
 	build_plt(&ctx);
 
 	/* 3. Extra pass to validate JITed code. */
@@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 
 	return ret;
 }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* We unwind through both kernel frames (starting from within bpf_throw call) and
+	 * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and
+	 * reach BPF frames in the stack trace.
+	 * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y
+	 */
+	return true;
+}
+
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};
+
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))
+			break;
+	}
+}
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
index f5065576cae9..7f768d335698 100644
@@ -1,6 +1,5 @@
 bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
 bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
 fexit_sleep                                      # The test never returns. The remaining tests cannot start.
 kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
 kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95
Signature · 1 line
2.40.1
maintainer Acked-by on v2 1/1
#2 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 20 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

Quoted 7 lines

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

We need reviews from arm64 JIT experts, but otherwise, given we've discussed this offline as well:

reviewer
#3 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

In reply to Puranjay Mohan
Quoted 79 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---
  arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
  tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
  2 files changed, 79 insertions(+), 20 deletions(-)
arch/arm64/net/bpf_jit_comp.c +13 -14
index 7d4af64e3982..fcc55e558863 100644
@@ -21,6 +21,7 @@
  #include <asm/insn.h>
  #include <asm/patching.h>
  #include <asm/set_memory.h>
+#include <asm/stacktrace.h>
 
  #include "bpf_jit.h"
 
@@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale)
  /* Tail call offset to jump into */
  #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
 
-static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
+static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb)
  {
  	const struct bpf_prog *prog = ctx->prog;
  	const bool is_main_prog = !bpf_is_subprog(prog);
@@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
  	emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
  	emit(A64_NOP, ctx);
 
-	/* Sign lr */
-	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
-		emit(A64_PACIASP, ctx);
-
-	/* Save FP and LR registers to stay align with ARM64 AAPCS */
-	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
-	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
-
-	/* Save callee-saved registers */
-	emit(A64_PUSH(r6, r7, A64_SP), ctx);
-	emit(A64_PUSH(r8, r9, A64_SP), ctx);
-	emit(A64_PUSH(fp, tcc, A64_SP), ctx);
-	emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	if (!is_exception_cb) {
+		/* Sign lr */
+		if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
+			emit(A64_PACIASP, ctx);
+		/* Save FP and LR registers to stay align with ARM64 AAPCS */
+		emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
+		/* Save callee-saved registers */
+		emit(A64_PUSH(r6, r7, A64_SP), ctx);
+		emit(A64_PUSH(r8, r9, A64_SP), ctx);

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 357

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+		emit(A64_PUSH(fp, tcc, A64_SP), ctx);
+		emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	} else {
+		/* Exception callback receives FP of Main Program as third parameter */
+		emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
+		/*
+		 * Main Program already pushed the frame record and the callee-saved registers. The
+		 * exception callback will not push anything and re-use the main program's stack.
+		 */
+		emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */

To ensure th calculated A6_SP is always correct, add an assertion to ensure the distance between A64_FP and A64_SP is 80 after all callee-registers are pushed to the stack?

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+	}
 
  	/* Set up BPF prog stack base register */
  	emit(A64_MOV(1, fp, A64_SP), ctx);

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 383

Quoted 9 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
@@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
  		emit_bti(A64_BTI_J, ctx);
  	}
 
+	/*
+	 * Program acting as exception boundary should save all ARM64 Callee-saved registers as the
+	 * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue.
+	 */
+	if (prog->aux->exception_boundary)
+		emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);

Blindly storing x23/x24 to BPF_FP -8/16 is incorrect, as the stack space below BPF_FP might be written with other values by the bpf prog.

Quoted 14 lines
arch/arm64/net/bpf_jit_comp.c +2 -1
+
  	emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
 
  	/* Stack must be multiples of 16B */
@@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx)
  		plt->target = (u64)&dummy_tramp;
  }
 
-static void build_epilogue(struct jit_ctx *ctx)
+static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
  {
  	const u8 r0 = bpf2a64[BPF_REG_0];
  	const u8 r6 = bpf2a64[BPF_REG_6];

arch/arm64/net/bpf_jit_comp.c static void build_epilogue(struct jit_ctx *ctx) · line 688

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +3 -0
@@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx)
  	/* We're done with BPF stack */
  	emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
+	/*
+	 * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved
+	 * registers. Exception callback uses the boundary program's stack frame, so recover these

Keep the line width within 80 characters?

Quoted 55 lines
arch/arm64/net/bpf_jit_comp.c +16 -4
+	 * extra registers in the above two cases.
+	 */
+	if (ctx->prog->aux->exception_boundary || is_exception_cb)
+		emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
+
  	/* Restore x27 and x28 */
  	emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
  	/* Restore fs (x25) and x26 */
@@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	 * BPF line info needs ctx->offset[i] to be the offset of
  	 * instruction[i] in jited image, so build prologue first.
  	 */
-	if (build_prologue(&ctx, was_classic)) {
+	if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
  		prog = orig_prog;
  		goto out_off;
  	}
@@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	}
 
  	ctx.epilogue_offset = ctx.idx;
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
  	build_plt(&ctx);
 
  	extable_align = __alignof__(struct exception_table_entry);
@@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	ctx.idx = 0;
  	ctx.exentry_idx = 0;
 
-	build_prologue(&ctx, was_classic);
+	build_prologue(&ctx, was_classic, prog->aux->exception_cb);
 
  	if (build_body(&ctx, extra_pass)) {
  		bpf_jit_binary_free(header);
@@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  		goto out_off;
  	}
 
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
  	build_plt(&ctx);
 
  	/* 3. Extra pass to validate JITed code. */
@@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 
  	return ret;
  }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* We unwind through both kernel frames (starting from within bpf_throw call) and
+	 * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and
+	 * reach BPF frames in the stack trace.
+	 * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2330

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+	 */
+	return true;
+}
+
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};
+

Seems there is no need to define "stacks" as an array

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2345

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))

When PTR_AUTH is implemented, lr is encoded before being pushed to the stack, but unwind_next_frame_record() does not decode state.pc when fetching it from the stack.

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2347

Quoted 2 lines
arch/arm64/net/bpf_jit_comp.c +2 -0
+			break;
+	}

And it's better to simplify the if-while(1)-if to:

while (!unwind_next_frame_record(&state)) {
    ...
}

tools/testing/selftests/bpf/DENYLIST.aarch64 line 5

Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+}
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
index f5065576cae9..7f768d335698 100644
@@ -1,6 +1,5 @@
  bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
  bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
  fexit_sleep                                      # The test never returns. The remaining tests cannot start.
  kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
  kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#4 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 86 lines · Xu Kuohai
Quoted 85 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---
  arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
  tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
  2 files changed, 79 insertions(+), 20 deletions(-)
arch/arm64/net/bpf_jit_comp.c +19 -14
index 7d4af64e3982..fcc55e558863 100644
@@ -21,6 +21,7 @@
  #include <asm/insn.h>
  #include <asm/patching.h>
  #include <asm/set_memory.h>
+#include <asm/stacktrace.h>
 
  #include "bpf_jit.h"
 
@@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale)
  /* Tail call offset to jump into */
  #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
 
-static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
+static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb)
  {
  	const struct bpf_prog *prog = ctx->prog;
  	const bool is_main_prog = !bpf_is_subprog(prog);
@@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
  	emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
  	emit(A64_NOP, ctx);
 
-	/* Sign lr */
-	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
-		emit(A64_PACIASP, ctx);
-
-	/* Save FP and LR registers to stay align with ARM64 AAPCS */
-	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
-	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
-
-	/* Save callee-saved registers */
-	emit(A64_PUSH(r6, r7, A64_SP), ctx);
-	emit(A64_PUSH(r8, r9, A64_SP), ctx);
-	emit(A64_PUSH(fp, tcc, A64_SP), ctx);
-	emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	if (!is_exception_cb) {
+		/* Sign lr */
+		if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
+			emit(A64_PACIASP, ctx);
+		/* Save FP and LR registers to stay align with ARM64 AAPCS */
+		emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
+		/* Save callee-saved registers */
+		emit(A64_PUSH(r6, r7, A64_SP), ctx);
+		emit(A64_PUSH(r8, r9, A64_SP), ctx);
+		emit(A64_PUSH(fp, tcc, A64_SP), ctx);
+		emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	} else {
+		/* Exception callback receives FP of Main Program as third parameter */
+		emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
+		/*
Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 357

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+		 * Main Program already pushed the frame record and the callee-saved registers. The
+		 * exception callback will not push anything and re-use the main program's stack.
+		 */
+		emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */

To ensure th calculated A6_SP is always correct, add an assertion to ensure the distance between A64_FP and A64_SP is 80 after all callee-registers are pushed to the stack?

I agree that this should be done. Can you give an example how this should be implemented?

Quoted 8 lines
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+	}
 
  	/* Set up BPF prog stack base register */
  	emit(A64_MOV(1, fp, A64_SP), ctx);
@@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
  		emit_bti(A64_BTI_J, ctx);
  	}
 
Quoted 11 lines

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 383

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+	/*
+	 * Program acting as exception boundary should save all ARM64 Callee-saved registers as the
+	 * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue.
+	 */
+	if (prog->aux->exception_boundary)
+		emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);

Blindly storing x23/x24 to BPF_FP -8/16 is incorrect, as the stack space below BPF_FP might be written with other values by the bpf prog.

Thanks for pointing this out. I will set fp = A64_SP - 16 so to allocate space for saving x23/x24. And I will take care while poping back in the epilogue.

Quoted 17 lines
Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +2 -1
+
  	emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
 
  	/* Stack must be multiples of 16B */
@@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx)
  		plt->target = (u64)&dummy_tramp;
  }
 
-static void build_epilogue(struct jit_ctx *ctx)
+static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
  {
  	const u8 r0 = bpf2a64[BPF_REG_0];
  	const u8 r6 = bpf2a64[BPF_REG_6];
@@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx)
  	/* We're done with BPF stack */
  	emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
Quoted 5 lines

arch/arm64/net/bpf_jit_comp.c static void build_epilogue(struct jit_ctx *ctx) · line 688

Quoted 3 lines
arch/arm64/net/bpf_jit_comp.c +3 -0
+	/*
+	 * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved
+	 * registers. Exception callback uses the boundary program's stack frame, so recover these

Keep the line width within 80 characters?

bdc48fa11e46 ("checkpatch/coding-style: deprecate 80-column warning") removed the warning so I started using 100 character lines.

Quoted 60 lines
Quoted 60 lines
arch/arm64/net/bpf_jit_comp.c +20 -4
+	 * extra registers in the above two cases.
+	 */
+	if (ctx->prog->aux->exception_boundary || is_exception_cb)
+		emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
+
  	/* Restore x27 and x28 */
  	emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
  	/* Restore fs (x25) and x26 */
@@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	 * BPF line info needs ctx->offset[i] to be the offset of
  	 * instruction[i] in jited image, so build prologue first.
  	 */
-	if (build_prologue(&ctx, was_classic)) {
+	if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
  		prog = orig_prog;
  		goto out_off;
  	}
@@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	}
 
  	ctx.epilogue_offset = ctx.idx;
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
  	build_plt(&ctx);
 
  	extable_align = __alignof__(struct exception_table_entry);
@@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	ctx.idx = 0;
  	ctx.exentry_idx = 0;
 
-	build_prologue(&ctx, was_classic);
+	build_prologue(&ctx, was_classic, prog->aux->exception_cb);
 
  	if (build_body(&ctx, extra_pass)) {
  		bpf_jit_binary_free(header);
@@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  		goto out_off;
  	}
 
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
  	build_plt(&ctx);
 
  	/* 3. Extra pass to validate JITed code. */
@@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 
  	return ret;
  }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* We unwind through both kernel frames (starting from within bpf_throw call) and
+	 * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and
+	 * reach BPF frames in the stack trace.
+	 * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y
+	 */
+	return true;
+}
+
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2330

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};
+

Seems there is no need to define "stacks" as an array

Sure, will change in next version.

Quoted 12 lines
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +11 -0
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2345

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))

When PTR_AUTH is implemented, lr is encoded before being pushed to the stack, but unwind_next_frame_record() does not decode state.pc when fetching it from the stack.

Thanks for pointing this out. I will fix this in the next version.

Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2347

Quoted 2 lines
arch/arm64/net/bpf_jit_comp.c +2 -0
+			break;
+	}

And it's better to simplify the if-while(1)-if to:

while (!unwind_next_frame_record(&state)) {
     ...
}

Sure,
Will use this method in the next version.

Quoted 13 lines

tools/testing/selftests/bpf/DENYLIST.aarch64 line 5

Quoted 13 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+}
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
index f5065576cae9..7f768d335698 100644
@@ -1,6 +1,5 @@
  bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
  bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
  fexit_sleep                                      # The test never returns. The remaining tests cannot start.
  kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
  kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95

Thanks,
Puranjay

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#5 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 98 lines · Puranjay Mohan
Quoted 86 lines · Xu Kuohai
Quoted 85 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---
   arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
   tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
   2 files changed, 79 insertions(+), 20 deletions(-)
arch/arm64/net/bpf_jit_comp.c +19 -14
index 7d4af64e3982..fcc55e558863 100644
@@ -21,6 +21,7 @@
   #include <asm/insn.h>
   #include <asm/patching.h>
   #include <asm/set_memory.h>
+#include <asm/stacktrace.h>
 
   #include "bpf_jit.h"
 
@@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale)
   /* Tail call offset to jump into */
   #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
 
-static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
+static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb)
   {
   	const struct bpf_prog *prog = ctx->prog;
   	const bool is_main_prog = !bpf_is_subprog(prog);
@@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
   	emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
   	emit(A64_NOP, ctx);
 
-	/* Sign lr */
-	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
-		emit(A64_PACIASP, ctx);
-
-	/* Save FP and LR registers to stay align with ARM64 AAPCS */
-	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
-	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
-
-	/* Save callee-saved registers */
-	emit(A64_PUSH(r6, r7, A64_SP), ctx);
-	emit(A64_PUSH(r8, r9, A64_SP), ctx);
-	emit(A64_PUSH(fp, tcc, A64_SP), ctx);
-	emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	if (!is_exception_cb) {
+		/* Sign lr */
+		if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
+			emit(A64_PACIASP, ctx);
+		/* Save FP and LR registers to stay align with ARM64 AAPCS */
+		emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
+		/* Save callee-saved registers */
+		emit(A64_PUSH(r6, r7, A64_SP), ctx);
+		emit(A64_PUSH(r8, r9, A64_SP), ctx);
+		emit(A64_PUSH(fp, tcc, A64_SP), ctx);
+		emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	} else {
+		/* Exception callback receives FP of Main Program as third parameter */
+		emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
+		/*
Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 357

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+		 * Main Program already pushed the frame record and the callee-saved registers. The
+		 * exception callback will not push anything and re-use the main program's stack.
+		 */
+		emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */

To ensure th calculated A6_SP is always correct, add an assertion to ensure the distance between A64_FP and A64_SP is 80 after all callee-registers are pushed to the stack?

Quoted 3 lines

I agree that this should be done. Can you give an example how this should be implemented?

IIUC, bpf_throw is essentially a tail call to the exception boundary prog, so can we reset the SP to the PROLOGUE_OFFSET position? If so, we can rely on the assertion of PROLOGUE_OFFSET in build_prologue, or we can add a simliar assertion.

Quoted 40 lines
Quoted 8 lines
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+	}
 
   	/* Set up BPF prog stack base register */
   	emit(A64_MOV(1, fp, A64_SP), ctx);
@@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
   		emit_bti(A64_BTI_J, ctx);
   	}
 
Quoted 11 lines

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 383

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+	/*
+	 * Program acting as exception boundary should save all ARM64 Callee-saved registers as the
+	 * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue.
+	 */
+	if (prog->aux->exception_boundary)
+		emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);

Blindly storing x23/x24 to BPF_FP -8/16 is incorrect, as the stack space below BPF_FP might be written with other values by the bpf prog.

Thanks for pointing this out. I will set fp = A64_SP - 16 so to allocate space for saving x23/x24. And I will take care while poping back in the epilogue.

Quoted 17 lines
Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +2 -1
+
   	emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
 
   	/* Stack must be multiples of 16B */
@@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx)
   		plt->target = (u64)&dummy_tramp;
   }
 
-static void build_epilogue(struct jit_ctx *ctx)
+static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
   {
   	const u8 r0 = bpf2a64[BPF_REG_0];
   	const u8 r6 = bpf2a64[BPF_REG_6];
@@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx)
   	/* We're done with BPF stack */
   	emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
Quoted 9 lines
Quoted 5 lines

arch/arm64/net/bpf_jit_comp.c static void build_epilogue(struct jit_ctx *ctx) · line 688

Quoted 3 lines
arch/arm64/net/bpf_jit_comp.c +3 -0
+	/*
+	 * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved
+	 * registers. Exception callback uses the boundary program's stack frame, so recover these

Keep the line width within 80 characters?

bdc48fa11e46 ("checkpatch/coding-style: deprecate 80-column warning") removed the warning so I started using 100 character lines.

80-column is not a hard limit, but it's still preferred, right? And I don't think it's a good idea to include two different line width styles in a single file.

Quoted 107 lines
Quoted 60 lines
Quoted 60 lines
arch/arm64/net/bpf_jit_comp.c +20 -4
+	 * extra registers in the above two cases.
+	 */
+	if (ctx->prog->aux->exception_boundary || is_exception_cb)
+		emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
+
   	/* Restore x27 and x28 */
   	emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
   	/* Restore fs (x25) and x26 */
@@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	 * BPF line info needs ctx->offset[i] to be the offset of
   	 * instruction[i] in jited image, so build prologue first.
   	 */
-	if (build_prologue(&ctx, was_classic)) {
+	if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
   		prog = orig_prog;
   		goto out_off;
   	}
@@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	}
 
   	ctx.epilogue_offset = ctx.idx;
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
   	build_plt(&ctx);
 
   	extable_align = __alignof__(struct exception_table_entry);
@@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	ctx.idx = 0;
   	ctx.exentry_idx = 0;
 
-	build_prologue(&ctx, was_classic);
+	build_prologue(&ctx, was_classic, prog->aux->exception_cb);
 
   	if (build_body(&ctx, extra_pass)) {
   		bpf_jit_binary_free(header);
@@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   		goto out_off;
   	}
 
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
   	build_plt(&ctx);
 
   	/* 3. Extra pass to validate JITed code. */
@@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 
   	return ret;
   }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* We unwind through both kernel frames (starting from within bpf_throw call) and
+	 * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and
+	 * reach BPF frames in the stack trace.
+	 * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y
+	 */
+	return true;
+}
+
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2330

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};
+

Seems there is no need to define "stacks" as an array

Sure, will change in next version.

Quoted 12 lines
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +11 -0
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2345

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))

When PTR_AUTH is implemented, lr is encoded before being pushed to the stack, but unwind_next_frame_record() does not decode state.pc when fetching it from the stack.

Thanks for pointing this out. I will fix this in the next version.

Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2347

Quoted 2 lines
arch/arm64/net/bpf_jit_comp.c +2 -0
+			break;
+	}

And it's better to simplify the if-while(1)-if to:

while (!unwind_next_frame_record(&state)) {
      ...
}

Sure,
Will use this method in the next version.

Quoted 1 line
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+}
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
index f5065576cae9..7f768d335698 100644
--- a/tools/testing/selftests/bpf/DENYLIST.aarch64
Quoted 8 lines

tools/testing/selftests/bpf/DENYLIST.aarch64 line 5

Quoted 8 lines

+++ b/tools/testing/selftests/bpf/DENYLIST.aarch64

tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
@@ -1,6 +1,5 @@
   bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
   bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
   fexit_sleep                                      # The test never returns. The remaining tests cannot start.
   kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
   kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95

Thanks,
Puranjay

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#6 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 99 lines · Xu Kuohai
Quoted 98 lines · Puranjay Mohan
Quoted 95 lines · Xu Kuohai
Quoted 79 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---
   arch/arm64/net/bpf_jit_comp.c                | 98 ++++++++++++++++----
   tools/testing/selftests/bpf/DENYLIST.aarch64 |  1 -
   2 files changed, 79 insertions(+), 20 deletions(-)
arch/arm64/net/bpf_jit_comp.c +13 -14
index 7d4af64e3982..fcc55e558863 100644
@@ -21,6 +21,7 @@
   #include <asm/insn.h>
   #include <asm/patching.h>
   #include <asm/set_memory.h>
+#include <asm/stacktrace.h>
 
   #include "bpf_jit.h"
 
@@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale)
   /* Tail call offset to jump into */
   #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
 
-static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
+static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb)
   {
   	const struct bpf_prog *prog = ctx->prog;
   	const bool is_main_prog = !bpf_is_subprog(prog);
@@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
   	emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
   	emit(A64_NOP, ctx);
 
-	/* Sign lr */
-	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
-		emit(A64_PACIASP, ctx);
-
-	/* Save FP and LR registers to stay align with ARM64 AAPCS */
-	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
-	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
-
-	/* Save callee-saved registers */
-	emit(A64_PUSH(r6, r7, A64_SP), ctx);
-	emit(A64_PUSH(r8, r9, A64_SP), ctx);
-	emit(A64_PUSH(fp, tcc, A64_SP), ctx);
-	emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	if (!is_exception_cb) {
+		/* Sign lr */
+		if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
+			emit(A64_PACIASP, ctx);
+		/* Save FP and LR registers to stay align with ARM64 AAPCS */
+		emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+		emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
+		/* Save callee-saved registers */
+		emit(A64_PUSH(r6, r7, A64_SP), ctx);
+		emit(A64_PUSH(r8, r9, A64_SP), ctx);

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 357

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+		emit(A64_PUSH(fp, tcc, A64_SP), ctx);
+		emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
+	} else {
+		/* Exception callback receives FP of Main Program as third parameter */
+		emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
+		/*
+		 * Main Program already pushed the frame record and the callee-saved registers. The
+		 * exception callback will not push anything and re-use the main program's stack.
+		 */
+		emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */

To ensure th calculated A6_SP is always correct, add an assertion to ensure the distance between A64_FP and A64_SP is 80 after all callee-registers are pushed to the stack?

Quoted 8 lines
Quoted 3 lines

I agree that this should be done. Can you give an example how this should be implemented?

IIUC, bpf_throw is essentially a tail call to the exception boundary prog, so can we reset the SP to the PROLOGUE_OFFSET position? If so, we can rely on the assertion of PROLOGUE_OFFSET in build_prologue, or we can add a simliar assertion.

I will think more about this but the PROLOGUE_OFFSET assertion is about the number of emitted instructions, here we want to see how many registers are pushed on the stack. This can only be asseted at runtime.

IMHO we don't need to add an assetion here because we are pushing 10 registers for every main program.

Quoted 46 lines
Quoted 46 lines
Quoted 8 lines
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+	}
 
   	/* Set up BPF prog stack base register */
   	emit(A64_MOV(1, fp, A64_SP), ctx);
@@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
   		emit_bti(A64_BTI_J, ctx);
   	}
 
Quoted 11 lines

arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 383

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+	/*
+	 * Program acting as exception boundary should save all ARM64 Callee-saved registers as the
+	 * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue.
+	 */
+	if (prog->aux->exception_boundary)
+		emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);

Blindly storing x23/x24 to BPF_FP -8/16 is incorrect, as the stack space below BPF_FP might be written with other values by the bpf prog.

Thanks for pointing this out. I will set fp = A64_SP - 16 so to allocate space for saving x23/x24. And I will take care while poping back in the epilogue.

Quoted 22 lines
Quoted 14 lines
arch/arm64/net/bpf_jit_comp.c +2 -1
+
   	emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
 
   	/* Stack must be multiples of 16B */
@@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx)
   		plt->target = (u64)&dummy_tramp;
   }
 
-static void build_epilogue(struct jit_ctx *ctx)
+static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
   {
   	const u8 r0 = bpf2a64[BPF_REG_0];
   	const u8 r6 = bpf2a64[BPF_REG_6];

arch/arm64/net/bpf_jit_comp.c static void build_epilogue(struct jit_ctx *ctx) · line 688

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +3 -0
@@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx)
   	/* We're done with BPF stack */
   	emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
+	/*
+	 * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved
+	 * registers. Exception callback uses the boundary program's stack frame, so recover these

Keep the line width within 80 characters?

Quoted 7 lines
Quoted 3 lines

bdc48fa11e46 ("checkpatch/coding-style: deprecate 80-column warning") removed the warning so I started using 100 character lines.

80-column is not a hard limit, but it's still preferred, right? And I don't think it's a good idea to include two different line width styles in a single file.

Okay,
I will change to 80 everywhere in the patch.

Quoted 108 lines
Quoted 108 lines
Quoted 61 lines
Quoted 61 lines
arch/arm64/net/bpf_jit_comp.c +20 -4
+	 * extra registers in the above two cases.
+	 */
+	if (ctx->prog->aux->exception_boundary || is_exception_cb)
+		emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
+
   	/* Restore x27 and x28 */
   	emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
   	/* Restore fs (x25) and x26 */
@@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	 * BPF line info needs ctx->offset[i] to be the offset of
   	 * instruction[i] in jited image, so build prologue first.
   	 */
-	if (build_prologue(&ctx, was_classic)) {
+	if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
   		prog = orig_prog;
   		goto out_off;
   	}
@@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	}
 
   	ctx.epilogue_offset = ctx.idx;
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
   	build_plt(&ctx);
 
   	extable_align = __alignof__(struct exception_table_entry);
@@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   	ctx.idx = 0;
   	ctx.exentry_idx = 0;
 
-	build_prologue(&ctx, was_classic);
+	build_prologue(&ctx, was_classic, prog->aux->exception_cb);
 
   	if (build_body(&ctx, extra_pass)) {
   		bpf_jit_binary_free(header);
@@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
   		goto out_off;
   	}
 
-	build_epilogue(&ctx);
+	build_epilogue(&ctx, prog->aux->exception_cb);
   	build_plt(&ctx);
 
   	/* 3. Extra pass to validate JITed code. */
@@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 
   	return ret;
   }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* We unwind through both kernel frames (starting from within bpf_throw call) and
+	 * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and
+	 * reach BPF frames in the stack trace.
+	 * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y
+	 */
+	return true;
+}
+
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2330

Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};
+

Seems there is no need to define "stacks" as an array

Sure, will change in next version.

Quoted 12 lines
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +11 -0
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2345

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))

When PTR_AUTH is implemented, lr is encoded before being pushed to the stack, but unwind_next_frame_record() does not decode state.pc when fetching it from the stack.

Thanks for pointing this out. I will fix this in the next version.

Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2347

Quoted 2 lines
arch/arm64/net/bpf_jit_comp.c +2 -0
+			break;
+	}

And it's better to simplify the if-while(1)-if to:

while (!unwind_next_frame_record(&state)) {
      ...
}

Sure,
Will use this method in the next version.

Quoted 1 line
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
+}
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -0
index f5065576cae9..7f768d335698 100644
@@ -1,6 +1,5 @@
   bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
   bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
Quoted 8 lines
Quoted 4 lines

tools/testing/selftests/bpf/DENYLIST.aarch64 line 5

Quoted 4 lines
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
   fexit_sleep                                      # The test never returns. The remaining tests cannot start.
   kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
   kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95

Thanks,
Puranjay

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

reviewer
#7 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

In reply to Puranjay Mohan
Quoted 26 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

[...]

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2329

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+	struct stack_info stacks[] = {
+		stackinfo_get_task(current),
+	};

Can bpf_throw() only be used by BPF programs that run in task context, or is it possible e.g. for those to run within an IRQ handler (or otherwise on the IRQ stack)?

Quoted 9 lines
arch/arm64/net/bpf_jit_comp.c +9 -0
+
+	struct unwind_state state = {
+		.stacks = stacks,
+		.nr_stacks = ARRAY_SIZE(stacks),
+	};
+	unwind_init_common(&state, current);
+	state.fp = (unsigned long)__builtin_frame_address(1);
+	state.pc = (unsigned long)__builtin_return_address(0);
+

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2348

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+	if (unwind_next_frame_record(&state))
+		return;
+	while (1) {
+		/* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+		if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
+			break;
+		if (unwind_next_frame_record(&state))
+			break;
+	}
+}

IIUC you're not using arch_stack_walk() because you need the FP in addition to the PC.

Is there any other reason you need to open-code this?

If not, I'd rather rework the common unwinder so that it's possible to get at the FP. I had patches for that a while back:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/metadata

... and I'm happy to rebase that and pull out the minimum necessary to make that possible.

Mark.

Quoted 15 lines
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
index f5065576cae9..7f768d335698 100644
@@ -1,6 +1,5 @@
 bpf_cookie/multi_kprobe_attach_api               # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
 bpf_cookie/multi_kprobe_link_api                 # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
-exceptions					 # JIT does not support calling kfunc bpf_throw: -524
 fexit_sleep                                      # The test never returns. The remaining tests cannot start.
 kprobe_multi_bench_attach                        # bpf_program__attach_kprobe_multi_opts unexpected error: -95
 kprobe_multi_test/attach_api_addrs               # bpf_program__attach_kprobe_multi_opts unexpected error: -95
Signature · 1 line
2.40.1
#8 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Hi Mark,

Quoted 31 lines · Mark Rutland
Quoted 26 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

[...]

Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2329

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+     struct stack_info stacks[] = {
+             stackinfo_get_task(current),
+     };

Can bpf_throw() only be used by BPF programs that run in task context, or is it possible e.g. for those to run within an IRQ handler (or otherwise on the IRQ stack)?

I will get back on this with more information.

Quoted 15 lines
Quoted 15 lines
arch/arm64/net/bpf_jit_comp.c +14 -0
+
+     struct unwind_state state = {
+             .stacks = stacks,
+             .nr_stacks = ARRAY_SIZE(stacks),
+     };
+     unwind_init_common(&state, current);
+     state.fp = (unsigned long)__builtin_frame_address(1);
+     state.pc = (unsigned long)__builtin_return_address(0);
+
+     if (unwind_next_frame_record(&state))
+             return;
+     while (1) {
+             /* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+             if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2348

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+                     break;
+             if (unwind_next_frame_record(&state))
+                     break;
+     }
+}

IIUC you're not using arch_stack_walk() because you need the FP in addition to the PC.

Yes,

Quoted 2 lines

Is there any other reason you need to open-code this?

No,

Quoted 8 lines

If not, I'd rather rework the common unwinder so that it's possible to get at the FP. I had patches for that a while back:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/metadata

... and I'm happy to rebase that and pull out the minimum necessary to make that possible.

It would be great if you can rebase and push the code, I can rebase this on your work and
not open code this implementation.

Quoted 3 lines

Mark.

Thanks,
Puranjay

reviewer
#9 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 80 lines · Puranjay Mohan

Hi Mark,

Quoted 31 lines · Mark Rutland
Quoted 26 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

[...]

Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2329

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+     struct stack_info stacks[] = {
+             stackinfo_get_task(current),
+     };

Can bpf_throw() only be used by BPF programs that run in task context, or is it possible e.g. for those to run within an IRQ handler (or otherwise on the IRQ stack)?

I will get back on this with more information.

Quoted 15 lines
Quoted 15 lines
arch/arm64/net/bpf_jit_comp.c +14 -0
+
+     struct unwind_state state = {
+             .stacks = stacks,
+             .nr_stacks = ARRAY_SIZE(stacks),
+     };
+     unwind_init_common(&state, current);
+     state.fp = (unsigned long)__builtin_frame_address(1);
+     state.pc = (unsigned long)__builtin_return_address(0);
+
+     if (unwind_next_frame_record(&state))
+             return;
+     while (1) {
+             /* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+             if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2348

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+                     break;
+             if (unwind_next_frame_record(&state))
+                     break;
+     }
+}

IIUC you're not using arch_stack_walk() because you need the FP in addition to the PC.

Yes,

Quoted 1 line

Is there any other reason you need to open-code this?

No,

Quoted 4 lines

If not, I'd rather rework the common unwinder so that it's possible to get at the FP. I had patches for that a while back:

Quoted 7 lines
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/metadata

... and I'm happy to rebase that and pull out the minimum necessary to make that possible.

It would be great if you can rebase and push the code, I can rebase this on your work and not open code this implementation.

I've rebased the core of that atop v6.6, and pushed that out to my arm64/stacktrace/kunwind branch:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/kunwind

Once v6.7-rc1 is out, I'll rebase that and post it out (possibly with some of the other patches atop).

With that I think you can implement arch_bpf_stack_walk() in stacktrace.c using kunwind_stack_walk() in a similar way to how arch_stack_walk() is implemented in that branch.

If BPF only needs a single consume_fn, that can probably be even simpler as you won't need a struct to hold the consume_fn and cookie value.

Mark.

#10 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Hi Mark,

Quoted 98 lines · Mark Rutland
Quoted 80 lines · Puranjay Mohan

Hi Mark,

Quoted 31 lines · Mark Rutland
Quoted 26 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

[...]

Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2329

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+     struct stack_info stacks[] = {
+             stackinfo_get_task(current),
+     };

Can bpf_throw() only be used by BPF programs that run in task context, or is it possible e.g. for those to run within an IRQ handler (or otherwise on the IRQ stack)?

I will get back on this with more information.

Quoted 15 lines
Quoted 15 lines
arch/arm64/net/bpf_jit_comp.c +14 -0
+
+     struct unwind_state state = {
+             .stacks = stacks,
+             .nr_stacks = ARRAY_SIZE(stacks),
+     };
+     unwind_init_common(&state, current);
+     state.fp = (unsigned long)__builtin_frame_address(1);
+     state.pc = (unsigned long)__builtin_return_address(0);
+
+     if (unwind_next_frame_record(&state))
+             return;
+     while (1) {
+             /* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+             if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2348

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+                     break;
+             if (unwind_next_frame_record(&state))
+                     break;
+     }
+}

IIUC you're not using arch_stack_walk() because you need the FP in addition to the PC.

Yes,

Quoted 1 line

Is there any other reason you need to open-code this?

No,

Quoted 4 lines

If not, I'd rather rework the common unwinder so that it's possible to get at the FP. I had patches for that a while back:

Quoted 7 lines
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/metadata

... and I'm happy to rebase that and pull out the minimum necessary to make that possible.

It would be great if you can rebase and push the code, I can rebase this on your work and not open code this implementation.

I've rebased the core of that atop v6.6, and pushed that out to my arm64/stacktrace/kunwind branch:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/kunwind

Once v6.7-rc1 is out, I'll rebase that and post it out (possibly with some of the other patches atop).

Quoted 6 lines

With that I think you can implement arch_bpf_stack_walk() in stacktrace.c using kunwind_stack_walk() in a similar way to how arch_stack_walk() is implemented in that branch.

If BPF only needs a single consume_fn, that can probably be even simpler as you won't need a struct to hold the consume_fn and cookie value.

Thanks for the help.
I am planning to do something like the following:
let me know if this can be done in a better way:

+struct bpf_unwind_consume_entry_data {
+ bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
+ void *cookie;
+};
+
+static bool
+arch_bpf_unwind_consume_entry (const struct kunwind_state *state, void *cookie) +{
+ struct bpf_unwind_consume_entry_data *data = cookie;
+ return data->consume_entry(data->cookie, state->common.pc, 0,
state->common.fp);
+}
+
+noinline noinstr void arch_bpf_stack_walk(bool (*consume_entry)(void *cookie, u64 ip, u64 sp,
+ u64 fp), void *cookie)
+{
+ struct bpf_unwind_consume_entry_data data = {
+ .consume_entry = consume_entry,
+ .cookie = cookie,
+ };

I need to get the task and regs here so it can work from all contexts. How can I do it?

+
+ kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, task, regs);
+}

Quoted 2 lines

Mark.

Thanks,
Puranjay

reviewer
#11 lore raw

Re: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions

Quoted 127 lines · Puranjay Mohan

Hi Mark,

Quoted 98 lines · Mark Rutland
Quoted 80 lines · Puranjay Mohan

Hi Mark,

Quoted 31 lines · Mark Rutland
Quoted 26 lines · Puranjay Mohan

Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program.

The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed.

As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue.

The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them.

These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged.

[1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst

---

[...]

Quoted 9 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2329

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
+{
+     struct stack_info stacks[] = {
+             stackinfo_get_task(current),
+     };

Can bpf_throw() only be used by BPF programs that run in task context, or is it possible e.g. for those to run within an IRQ handler (or otherwise on the IRQ stack)?

I will get back on this with more information.

Quoted 15 lines
Quoted 15 lines
arch/arm64/net/bpf_jit_comp.c +14 -0
+
+     struct unwind_state state = {
+             .stacks = stacks,
+             .nr_stacks = ARRAY_SIZE(stacks),
+     };
+     unwind_init_common(&state, current);
+     state.fp = (unsigned long)__builtin_frame_address(1);
+     state.pc = (unsigned long)__builtin_return_address(0);
+
+     if (unwind_next_frame_record(&state))
+             return;
+     while (1) {
+             /* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/
+             if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp))
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, · line 2348

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+                     break;
+             if (unwind_next_frame_record(&state))
+                     break;
+     }
+}

IIUC you're not using arch_stack_walk() because you need the FP in addition to the PC.

Yes,

Quoted 1 line

Is there any other reason you need to open-code this?

No,

Quoted 4 lines

If not, I'd rather rework the common unwinder so that it's possible to get at the FP. I had patches for that a while back:

Quoted 7 lines
Quoted 4 lines

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/metadata

... and I'm happy to rebase that and pull out the minimum necessary to make that possible.

It would be great if you can rebase and push the code, I can rebase this on your work and not open code this implementation.

I've rebased the core of that atop v6.6, and pushed that out to my arm64/stacktrace/kunwind branch:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/stacktrace/kunwind

Once v6.7-rc1 is out, I'll rebase that and post it out (possibly with some of the other patches atop).

Quoted 6 lines

With that I think you can implement arch_bpf_stack_walk() in stacktrace.c using kunwind_stack_walk() in a similar way to how arch_stack_walk() is implemented in that branch.

If BPF only needs a single consume_fn, that can probably be even simpler as you won't need a struct to hold the consume_fn and cookie value.

Thanks for the help.
I am planning to do something like the following:
let me know if this can be done in a better way:

(quoted diff) +10 -0
+struct bpf_unwind_consume_entry_data {
+       bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
+       void *cookie;
+};
+
+static bool
+arch_bpf_unwind_consume_entry (const struct kunwind_state *state, void *cookie)
+{
+       struct bpf_unwind_consume_entry_data *data = cookie;
+       return data->consume_entry(data->cookie, state->common.pc, 0,

state->common.fp);

arch/arm64/net/bpf_jit_comp.c +3 -0
+}
+
+noinline noinstr void arch_bpf_stack_walk(bool (*consume_entry)(void

*cookie, u64 ip, u64 sp,

Review

Quoted 9 lines
(quoted diff) +9 -0
+                                         u64 fp), void *cookie)
+{
+       struct bpf_unwind_consume_entry_data data = {
+               .consume_entry = consume_entry,
+               .cookie = cookie,
+       };
+
+       kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, task, regs);
+}

That's roughly what I had expected, so that looks good to me.

Quoted 2 lines

I need to get the task and regs here so it can work from all contexts. How can I do it?

Are you asking because that's what the kunwind_stack_walk() prototype takes?

If so, I believe you just need:

	kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, current, NULL);

Note that we currently *cannot* reliably unwind across an exception boundary, so if you have non-NULL regs the unwind will be unsafe. IIUC the BPF exceptions you're adding support for are handled via a branch rather than via an architectural exception, so there are no regs to pass (and so NULL is correct).

Thanks,
Mark.