Skip to contentSkip to conversation

bpf,x86: Fix exception unwinding with outgoing stack arguments

PATCH Applied Yonghong Song 2 participants 3 messages May 17

Subsystems (6)
x86 Kselftest BPF BPF selftests +2 more subsystems
lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

PATCH 1/2 bpf,x86: Fix exception unwinding with outgoing stack arguments

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Fix this by adjusting the sp (adding stack_arg_sp_adjust) passed to the exception callback, so it points to the bottom of the callee-saved registers instead of the outgoing arg area. When stack_arg_sp_adjust is 0 (the common case), this is a no-op.

Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
---
 arch/x86/net/bpf_jit_comp.c | 2 ++
 include/linux/bpf.h         | 1 +
 kernel/bpf/fixups.c         | 1 +
 kernel/bpf/helpers.c        | 2 +-
 4 files changed, 5 insertions(+), 1 deletion(-)

NOTE:
Existing arm64 jit already supports exception unwinding with stack arguments. In arch/arm64/net/bpf_jit_comp.c, we have

        if (!prog->aux->exception_cb) {
                ...
        } else {
                /* FP = x2 */
                emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
                /* SP = FP - 96 */
                emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
        }

There is no SP adjustment on arm64 as SP points directly at the bottom of the main program's callee-saved registers.

arch/x86/net/bpf_jit_comp.c +2 -0
index ceefefb4da21..a0c541a441cf 100644
@@ -1789,6 +1789,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	 * Arg 6 goes into r9 register, not on stack.
 	 */
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	if (bpf_prog->aux->exception_boundary)
+		bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
 	emit_sub_rsp(&prog, outgoing_rsp);
 
 	if (arena_vm_start)
include/linux/bpf.h +1 -0
index 242f9597d9ab..1b28cacc3075 100644
@@ -1736,6 +1736,7 @@ struct bpf_prog_aux {
 	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
 	char name[BPF_OBJ_NAME_LEN];
 	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
+	u16 stack_arg_sp_adjust;
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
kernel/bpf/fixups.c +1 -0
index 2cec4e8cd4a0..52535671cb9a 100644
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	prog->aux->real_func_cnt = env->subprog_cnt;
 	prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
 	prog->aux->exception_boundary = func[0]->aux->exception_boundary;
+	prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
 	bpf_prog_jit_attempt_done(prog);
 	return 0;
 out_free:
kernel/bpf/helpers.c +1 -1
index baa12b24bb64..07de26e7314c 100644
@@ -3301,7 +3301,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	 * which skips compiler generated instrumentation to do the same.
 	 */
 	kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
-	ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
+	ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH 2/2 selftests/bpf: Add exception tests with stack arguments

Add tests to verify that bpf_throw() correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding x86 fix, these tests crash the kernel on x86 due to corrupted callee-saved register restore. There is no change for arm64 to support exception with stack arguments.

---
 .../selftests/bpf/prog_tests/exceptions.c     |   7 ++
 .../testing/selftests/bpf/progs/exceptions.c  | 114 ++++++++++++++++++
 2 files changed, 121 insertions(+)
tools/testing/selftests/bpf/prog_tests/exceptions.c +7 -0
index e8cbaf2a3e82..3588d6f97fd4 100644
@@ -85,6 +85,13 @@ static void test_exceptions_success(void)
 	RUN_SUCCESS(exception_bad_assert_range_with, 10);
 	RUN_SUCCESS(exception_throw_from_void_global, 11);
 
+	if (skel->rodata->has_stack_arg) {
+		RUN_SUCCESS(exception_throw_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_after_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_subprog_stack_arg, 56);
+		RUN_SUCCESS(exception_throw_subprog_after_stack_arg, 56);
+	}
+
 #define RUN_EXT(load_ret, attach_err, expr, msg, after_link)			  \
 	{									  \
 		LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf,		 \
tools/testing/selftests/bpf/progs/exceptions.c +114 -0

118 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#2 lore raw
In reply to Yonghong Song
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 13 lines · On Sun, 17 May 2026 08:07:02 -0700 you wrote:

When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.

The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.

Quoted 1 line

[...]

Here is the summary with links:
- [bpf-next,1/2] bpf,x86: Fix exception unwinding with outgoing stack arguments

    https://git.kernel.org/bpf/bpf-next/c/18a37465b0ab
  - [bpf-next,2/2] selftests/bpf: Add exception tests with stack arguments
    https://git.kernel.org/bpf/bpf-next/c/576482b55c19

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html