Skip to contentSkip to conversation

bpf: Follow-up fixes for stack argument support

PATCH In review Yonghong Song 3 participants 15 messages May 15 – May 17

Subsystems (6)
x86 Kselftest BPF BPF selftests +2 more subsystems
lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v3 cover bpf: Follow-up fixes for stack argument support

Commit cd59fa185a03 ("bpf: Support stack arguments for BPF functions and kfuncs") added stack argument support for bpf functions and kfuncs. This patch set is to fix various issues related to stack arguments, mainly include:
- Validate outgoing stack args when btf_prepare_func_args fails
- Fix arg_track_join log to use sa prefix for stack arg slots
- Clean up redundant stack arg checks for non-JITed programs
- Fix exception unwinding with outgoing stack arguments

Changelog:
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Add additional fix (fix arg_track_join log, fix exception with outgoing stack arguments, some cleanup, etc.).
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Remove 'Reported-by: Sashiko <[email protected]>'.
- Add proper fix tag.

Yonghong Song (7):
bpf: Validate outgoing stack args when btf_prepare_func_args fails selftests/bpf: Add test for stack arg read without caller write selftests/bpf: Log arg_track_join for stack arg slots in liveness
analysis
bpf: Fix arg_track_join log to use sa prefix for stack arg slots
bpf: Clean up redundant stack arg checks for non-JITed programs bpf,x86: Fix exception unwinding with outgoing stack arguments selftests/bpf: Add exception tests with stack arguments

 arch/x86/net/bpf_jit_comp.c                   |   9 +-
 include/linux/bpf.h                           |   3 +-
 kernel/bpf/core.c                             |   2 +-
 kernel/bpf/fixups.c                           |   3 +-
 kernel/bpf/helpers.c                          |   2 +-
 kernel/bpf/liveness.c                         |   4 +-
 kernel/bpf/verifier.c                         |  10 +-
 .../selftests/bpf/prog_tests/exceptions.c     |   7 ++
 .../bpf/progs/btf__verifier_stack_arg_order.c |   8 ++
 .../testing/selftests/bpf/progs/exceptions.c  | 113 ++++++++++++++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  |   8 +-
 .../bpf/progs/verifier_stack_arg_order.c      |  58 +++++++++
 12 files changed, 216 insertions(+), 11 deletions(-)
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v3 1/7 bpf: Validate outgoing stack args when btf_prepare_func_args fails

btf_prepare_func_args() sets sub->arg_cnt before validating arg types. If validation fails (e.g. unsupported pointer type in a static subprog), check_outgoing_stack_args() is skipped because btf_check_func_arg_match() returns early. For static subprogs, check_func_call() ignores non-EFAULT errors and proceeds with the call.

This causes the callee to read stack arg slots that the caller never stored or not initialized, potentially dereferencing NULL caller->stack_arg_regs or getting no-initialized value.

To fix the issue, when btf_prepare_func_args() fails and the subprog expects stack args, call check_outgoing_stack_args() to verify the caller initialized the slots. Return -EFAULT on failure so the error is not ignored.

Fixes 3ab5bd317ee2 ("bpf: Set sub->arg_cnt earlier in btf_prepare_func_args()")
---
 kernel/bpf/verifier.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +8 -2
index 76a07f09ab64..8dd79b735a69 100644
@@ -9118,11 +9118,17 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	struct bpf_func_state *caller = cur_func(env);
 	struct bpf_verifier_log *log = &env->log;
 	u32 i;
-	int ret;
+	int ret, err;
 
 	ret = btf_prepare_func_args(env, subprog);
-	if (ret)
+	if (ret) {
+		if (bpf_in_stack_arg_cnt(sub) > 0) {
+			err = check_outgoing_stack_args(env, caller, sub->arg_cnt);
+			if (err)
+				return err;
+		}
 		return ret;
+	}
 
 	ret = check_outgoing_stack_args(env, caller, sub->arg_cnt);
 	if (ret)
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH v3 2/7 selftests/bpf: Add test for stack arg read without caller write

In reply to Yonghong Song

Add negative tests for the outgoing stack arg validation. A static subprog with a 'long *' arg causes btf_prepare_func_args() to fail after setting arg_cnt. The validation ensures check_outgoing_stack_args() still runs.

Also update two existing tests (release_ref, stale_pkt_ptr) whose expected error messages changed: invalidated stack arg slots are now caught by check_outgoing_stack_args() at the call site instead of at the callee's dereference.

---
 .../bpf/progs/btf__verifier_stack_arg_order.c |  8 +++
 .../selftests/bpf/progs/verifier_stack_arg.c  |  4 +-
 .../bpf/progs/verifier_stack_arg_order.c      | 58 +++++++++++++++++++
 3 files changed, 68 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c +8 -0
index da34e8456b6c..99bc115f8380 100644
@@ -21,6 +21,10 @@ int subprog_pruning_call_before_load_6args(int a, int b, int c, int d, int e, in
 	return a + b + c + d + e + f;
 }
 
+void subprog_bad_ptr_7args(long *a, int b, int c, int d, int e, int f, int g)
+{
+}
+
 #else
 
 int subprog_bad_order_6args(void)
@@ -38,4 +42,8 @@ int subprog_pruning_call_before_load_6args(void)
 	return 0;
 }
 
+void subprog_bad_ptr_7args(void)
+{
+}
+
 #endif
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +2 -2
index d43a9b42034c..d45339b83795 100644
@@ -152,7 +152,7 @@ __naked void stack_arg_pruning_type_mismatch(void)
 SEC("tc")
 __description("stack_arg: release_reference invalidates stack arg slot")
 __failure
-__msg("R{{[0-9]}} !read_ok")
+__msg("callee expects 6 args, stack arg1 is not initialized")
 __naked void stack_arg_release_ref(void)
 {
 	asm volatile (
@@ -201,7 +201,7 @@ __naked void stack_arg_release_ref(void)
 SEC("tc")
 __description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data")
 __failure
-__msg("R{{[0-9]}} !read_ok")
+__msg("callee expects 6 args, stack arg1 is not initialized")
 __naked void stack_arg_stale_pkt_ptr(void)
 {
 	asm volatile (
tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +58 -0

64 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH v3 3/7 selftests/bpf: Log arg_track_join for stack arg slots in liveness analysis

In reply to Yonghong Song

Commit 2af4e792773f ("bpf: Extend liveness analysis to track stack argument slots") added stack arg supports. For selftest
verifier_stack_arg/stack_arg: pruning with different stack arg types
the following are two arg JOIN messages:
arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0
arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0

Here the "r11:" label for stack arg slot 0 is misleading since r11 is a special register (BPF_REG_PARAMS). The next patch corrects this to "sa0:", properly representing the 'stack arg slot 0'.

---
 tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +3 -1
index d45339b83795..df0c3438529e 100644
@@ -114,8 +114,10 @@ __naked void stack_arg_gap_at_minus8(void)
 
 SEC("tc")
 __description("stack_arg: pruning with different stack arg types")
-__failure
+__failure __log_level(2)
 __flag(BPF_F_TEST_STATE_FREQ)
+__msg("arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0")
+__msg("arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0")
 __msg("R{{[0-9]}} invalid mem access 'scalar'")
 __naked void stack_arg_pruning_type_mismatch(void)
 {
Signature · 1 line
2.53.0-Meta
author reviewer
#4 lore raw

PATCH v3 4/7 bpf: Fix arg_track_join log to use sa prefix for stack arg slots

In reply to Yonghong Song

arg_track_join() logs state transitions at CFG merge points. For stack arg slots (r >= MAX_BPF_REG), it printed "r11:", "r12:", etc., which is misleading since r11 is a special register (BPF_REG_PARAMS) not meaningful to the user.

Fix it to print "sa0:", "sa1:", etc., matching the per-instruction transition log in arg_track_log() which already uses the "sa" prefix.

Update the existing stack_arg_pruning_type_mismatch selftest to expect the corrected format.

Fixes 2af4e792773f ("bpf: Extend liveness analysis to track stack argument slots")
---
 kernel/bpf/liveness.c                                  | 4 +++-
 tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
kernel/bpf/liveness.c +3 -1
index 7f4a0e4c2c49..0aadfbae0acc 100644
@@ -806,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in
 		return true;
 
 	verbose(env, "arg JOIN insn %d -> %d ", idx, target);
-	if (r >= 0)
+	if (r >= MAX_BPF_REG)
+		verbose(env, "sa%d: ", r - MAX_BPF_REG);
+	else if (r >= 0)
 		verbose(env, "r%d: ", r);
 	else
 		verbose(env, "fp%+d: ", r * 8);
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +1 -1
index df0c3438529e..7e0ce5db28a0 100644
@@ -117,7 +117,7 @@ __description("stack_arg: pruning with different stack arg types")
 __failure __log_level(2)
 __flag(BPF_F_TEST_STATE_FREQ)
 __msg("arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0")
-__msg("arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0")
+__msg("arg JOIN insn 9 -> 10 sa0: fp0-8 + _ => fp0-8|fp0+0")
 __msg("R{{[0-9]}} invalid mem access 'scalar'")
 __naked void stack_arg_pruning_type_mismatch(void)
 {
Signature · 1 line
2.53.0-Meta
author reviewer
#5 lore raw

PATCH v3 5/7 bpf: Clean up redundant stack arg checks for non-JITed programs

In reply to Yonghong Song

Remove a redundant stack_arg_cnt check in __bpf_prog_select_runtime() and start the stack arg loop from index 0 in bpf_fixup_call_args(). Both changes are no-ops that simplify the code:

In __bpf_prog_select_runtime(), the subprog_info[0].stack_arg_cnt check is unreachable:
- when there is only a main program (no bpf-to-bpf calls), subprog_info[0].stack_arg_cnt is always 0 because the main program's arg_cnt is forced to 1
- when bpf-to-bpf calls use stack args and JIT succeeds,

    fp->bpf_func is set and this code is skipped

- when JIT fails, bpf_fixup_call_args() rejects the program before we get to __bpf_prog_select_runtime().

In bpf_fixup_call_args(), starting the loop at i=1 skipped subprog 0, which is safe since the main program always has arg_cnt=1 and thus bpf_in_stack_arg_cnt() returns 0. Starting at i=0 removes the need to reason about this invariant.

---
 kernel/bpf/core.c   | 2 +-
 kernel/bpf/fixups.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/core.c +1 -1
index 427a6d828e01..cdbe9fdf474f 100644
@@ -2609,7 +2609,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp) || (env && env->subprog_info[0].stack_arg_cnt))
+	    bpf_prog_has_kfunc_call(fp))
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/fixups.c +1 -1
index 19056016eed8..2cec4e8cd4a0 100644
@@ -1407,7 +1407,7 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}
-	for (i = 1; i < env->subprog_cnt; i++) {
+	for (i = 0; i < env->subprog_cnt; i++) {
 		if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
 			verbose(env, "stack args are not supported in non-JITed programs\n");
 			return -EINVAL;
Signature · 1 line
2.53.0-Meta
author reviewer
#6 lore raw

PATCH v3 6/7 bpf,x86: Fix exception unwinding with outgoing stack arguments

In reply to Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
 arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
 include/linux/bpf.h         | 3 ++-
 kernel/bpf/fixups.c         | 1 +
 kernel/bpf/helpers.c        | 2 +-
 4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +8 -1
index ceefefb4da21..f4fdceedaad7 100644
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 			/* Keep the same instruction layout. */
 			emit_nops(&prog, 3);     /* nop3 */
 	}
-	/* Exception callback receives FP as third parameter */
+	/*
+	 * Exception callback receives:
+	 *   rsi = main program's SP, rdx = main program's FP,
+	 *   rcx = main program's outgoing stack arg area size
+	 */
 	if (is_exception_cb) {
 		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
 		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */
 		/* The main frame must have exception_boundary as true, so we
 		 * first restore those callee-saved regs from stack, before
 		 * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	 * Arg 6 goes into r9 register, not on stack.
 	 */
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
 	emit_sub_rsp(&prog, outgoing_rsp);
 
 	if (arena_vm_start)
include/linux/bpf.h +2 -1
index 242f9597d9ab..2a1616c769a9 100644
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
 	int cgroup_atype; /* enum cgroup_bpf_attach_type */
 	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
 	char name[BPF_OBJ_NAME_LEN];
-	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);
+	u16 stack_arg_adjust;
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
kernel/bpf/fixups.c +1 -0
index 2cec4e8cd4a0..52aaf2863648 100644
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	prog->aux->real_func_cnt = env->subprog_cnt;
 	prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
 	prog->aux->exception_boundary = func[0]->aux->exception_boundary;
+	prog->aux->stack_arg_adjust = func[0]->aux->stack_arg_adjust;
 	bpf_prog_jit_attempt_done(prog);
 	return 0;
 out_free:
kernel/bpf/helpers.c +1 -1
index baa12b24bb64..53a0430f60b3 100644
@@ -3301,7 +3301,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	 * which skips compiler generated instrumentation to do the same.
 	 */
 	kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
-	ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
+	ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, ctx.aux->stack_arg_adjust, 0);
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v3 6/7
#7 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

Quoted 19 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Quoted 3 lines
Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")

---

Do we need any adjustment for arm64?

For this patch:

Quoted 1 line

[...]

author reviewer
#8 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

Quoted 20 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Quoted 4 lines
Quoted 3 lines
Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")

---

Do we need any adjustment for arm64?

Yes, after this patch lands, Puranjay can help with arm64 side.

Quoted 5 lines

For this patch:

Quoted 1 line

[...]

#9 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

In reply to Yonghong Song
Quoted 36 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
 arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
 include/linux/bpf.h         | 3 ++-
 kernel/bpf/fixups.c         | 1 +
 kernel/bpf/helpers.c        | 2 +-
 4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +0 -0
index ceefefb4da21..f4fdceedaad7 100644
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 			/* Keep the same instruction layout. */
 			emit_nops(&prog, 3);     /* nop3 */
 	}

arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +6 -1
-	/* Exception callback receives FP as third parameter */
+	/*
+	 * Exception callback receives:
+	 *   rsi = main program's SP, rdx = main program's FP,
+	 *   rcx = main program's outgoing stack arg area size
+	 */
 	if (is_exception_cb) {
 		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
 		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */

Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0); Avoids the need to use 'rcx'.

Quoted 17 lines
arch/x86/net/bpf_jit_comp.c +2 -0
 		/* The main frame must have exception_boundary as true, so we
 		 * first restore those callee-saved regs from stack, before
 		 * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	 * Arg 6 goes into r9 register, not on stack.
 	 */
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
 	emit_sub_rsp(&prog, outgoing_rsp);
 
 	if (arena_vm_start)
include/linux/bpf.h +0 -0
index 242f9597d9ab..2a1616c769a9 100644

include/linux/bpf.h struct bpf_prog_aux { · line 1738

Quoted 5 lines
include/linux/bpf.h +1 -1
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
 	int cgroup_atype; /* enum cgroup_bpf_attach_type */
 	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
 	char name[BPF_OBJ_NAME_LEN];
-	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);

no need to change this.

include/linux/bpf.h struct bpf_prog_aux { · line 1739

Quoted 1 line
include/linux/bpf.h +1 -0
+	u16 stack_arg_adjust;

this one is still needed, but maybe let's call it stack_arg_sp_adjust?

Looking at arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.

and arm64 seems to work already?

and the field x86 specific? not sure about other archs.

author reviewer
#10 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
  arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
  include/linux/bpf.h         | 3 ++-
  kernel/bpf/fixups.c         | 1 +
  kernel/bpf/helpers.c        | 2 +-
  4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +3 -1
index ceefefb4da21..f4fdceedaad7 100644
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
  			/* Keep the same instruction layout. */
  			emit_nops(&prog, 3);     /* nop3 */
  	}
-	/* Exception callback receives FP as third parameter */
+	/*
+	 * Exception callback receives:
+	 *   rsi = main program's SP, rdx = main program's FP,
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3 -0
+	 *   rcx = main program's outgoing stack arg area size
+	 */
  	if (is_exception_cb) {
  		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
  		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */

Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);

This sounds better!

Quoted 13 lines

Avoids the need to use 'rcx'.

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2 -0
  		/* The main frame must have exception_boundary as true, so we
  		 * first restore those callee-saved regs from stack, before
  		 * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  	 * Arg 6 goes into r9 register, not on stack.
  	 */
  	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
  	emit_sub_rsp(&prog, outgoing_rsp);
 
Quoted 12 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1738

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0 -0
  	if (arena_vm_start)
include/linux/bpf.h +1 -1
index 242f9597d9ab..2a1616c769a9 100644
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
  	int cgroup_atype; /* enum cgroup_bpf_attach_type */
  	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
  	char name[BPF_OBJ_NAME_LEN];
-	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);

no need to change this.

Indeed, with the above 'ctx.sp + ctx.aux->stack_arg_adjust', the 4th argument does not need any change.

Quoted 2 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1739

Quoted 1 line
include/linux/bpf.h +1 -0
+	u16 stack_arg_adjust;

this one is still needed, but maybe let's call it stack_arg_sp_adjust?

Okay. Will use the stack_arg_sp_adjust.

Quoted 5 lines

Looking at arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.

and arm64 seems to work already?

I will take a look at arm64 as well.

Quoted 4 lines

and the field x86 specific? not sure about other archs.

#11 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

Quoted 90 lines · Yonghong Song
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
  arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
  include/linux/bpf.h         | 3 ++-
  kernel/bpf/fixups.c         | 1 +
  kernel/bpf/helpers.c        | 2 +-
  4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +3 -1
index ceefefb4da21..f4fdceedaad7 100644
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
  			/* Keep the same instruction layout. */
  			emit_nops(&prog, 3);     /* nop3 */
  	}
-	/* Exception callback receives FP as third parameter */
+	/*
+	 * Exception callback receives:
+	 *   rsi = main program's SP, rdx = main program's FP,
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3 -0
+	 *   rcx = main program's outgoing stack arg area size
+	 */
  	if (is_exception_cb) {
  		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
  		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */

Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);

This sounds better!

Quoted 13 lines

Avoids the need to use 'rcx'.

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2 -0
  		/* The main frame must have exception_boundary as true, so we
  		 * first restore those callee-saved regs from stack, before
  		 * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  	 * Arg 6 goes into r9 register, not on stack.
  	 */
  	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
  	emit_sub_rsp(&prog, outgoing_rsp);
 
Quoted 12 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1738

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0 -0
  	if (arena_vm_start)
include/linux/bpf.h +1 -1
index 242f9597d9ab..2a1616c769a9 100644
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
  	int cgroup_atype; /* enum cgroup_bpf_attach_type */
  	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
  	char name[BPF_OBJ_NAME_LEN];
-	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);

no need to change this.

Indeed, with the above 'ctx.sp + ctx.aux->stack_arg_adjust', the 4th argument does not need any change.

Quoted 2 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1739

Quoted 1 line
include/linux/bpf.h +1 -0
+	u16 stack_arg_adjust;

this one is still needed, but maybe let's call it stack_arg_sp_adjust?

Okay. Will use the stack_arg_sp_adjust.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Looking at arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.

and arm64 seems to work already?

I will take a look at arm64 as well.

Please also remove guards on tests to allow them to run on arm64, once it is handled properly.

author reviewer
#12 lore raw

Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

Quoted 86 lines · Kumar Kartikeya Dwivedi
Quoted 85 lines · Yonghong Song
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
   arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
   include/linux/bpf.h         | 3 ++-
   kernel/bpf/fixups.c         | 1 +
   kernel/bpf/helpers.c        | 2 +-
   4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +3 -1
index ceefefb4da21..f4fdceedaad7 100644
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
   			/* Keep the same instruction layout. */
   			emit_nops(&prog, 3);     /* nop3 */
   	}
-	/* Exception callback receives FP as third parameter */
+	/*
+	 * Exception callback receives:
+	 *   rsi = main program's SP, rdx = main program's FP,
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3 -0
+	 *   rcx = main program's outgoing stack arg area size
+	 */
   	if (is_exception_cb) {
   		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
   		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
+		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */

Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);

This sounds better!

Quoted 13 lines

Avoids the need to use 'rcx'.

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2 -0
   		/* The main frame must have exception_boundary as true, so we
   		 * first restore those callee-saved regs from stack, before
   		 * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
   	 * Arg 6 goes into r9 register, not on stack.
   	 */
   	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
   	emit_sub_rsp(&prog, outgoing_rsp);
 
Quoted 12 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1738

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0 -0
   	if (arena_vm_start)
include/linux/bpf.h +1 -1
index 242f9597d9ab..2a1616c769a9 100644
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
   	int cgroup_atype; /* enum cgroup_bpf_attach_type */
   	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
   	char name[BPF_OBJ_NAME_LEN];
-	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);

no need to change this.

Indeed, with the above 'ctx.sp + ctx.aux->stack_arg_adjust', the 4th argument does not need any change.

Quoted 2 lines

include/linux/bpf.h struct bpf_prog_aux { · line 1739

Quoted 1 line
include/linux/bpf.h +1 -0
+	u16 stack_arg_adjust;

this one is still needed, but maybe let's call it stack_arg_sp_adjust?

Okay. Will use the stack_arg_sp_adjust.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Looking at arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.

and arm64 seems to work already?

I will take a look at arm64 as well.

Please also remove guards on tests to allow them to run on arm64, once it is handled properly.

Done. See

    https://lore.kernel.org/bpf/[email protected]/

No change is needed for arm64. It already can handle exception and stack arg's together.

author reviewer
#13 lore raw

PATCH v3 7/7 selftests/bpf: Add exception tests with stack arguments

In reply to Yonghong Song

Add tests to verify that bpf_throw() correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding fix, these tests crash the kernel due to corrupted callee-saved register restore.

---
 .../selftests/bpf/prog_tests/exceptions.c     |   7 ++
 .../testing/selftests/bpf/progs/exceptions.c  | 113 ++++++++++++++++++
 2 files changed, 120 insertions(+)
tools/testing/selftests/bpf/prog_tests/exceptions.c +7 -0
index e8cbaf2a3e82..3588d6f97fd4 100644
@@ -85,6 +85,13 @@ static void test_exceptions_success(void)
 	RUN_SUCCESS(exception_bad_assert_range_with, 10);
 	RUN_SUCCESS(exception_throw_from_void_global, 11);
 
+	if (skel->rodata->has_stack_arg) {
+		RUN_SUCCESS(exception_throw_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_after_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_subprog_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_subprog_after_stack_arg, 56);
+	}
+
 #define RUN_EXT(load_ret, attach_err, expr, msg, after_link)			  \
 	{									  \
 		LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf,		 \
tools/testing/selftests/bpf/progs/exceptions.c +113 -0

117 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v3 7/7
#14 lore raw

Re: [PATCH bpf-next v3 7/7] selftests/bpf: Add exception tests with stack arguments

Quoted 7 lines · Yonghong Song

Add tests to verify that bpf_throw() correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding fix, these tests crash the kernel due to corrupted callee-saved register restore.

---

Quoted 1 line

[...]