bpf,x86: Fix exception unwinding with outgoing stack arguments
Subsystems (6)
Yonghong Song
reviewerPATCH 1/2 bpf,x86: Fix exception unwinding with outgoing stack arguments
When a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.
The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.
Fix this by adjusting the sp (adding stack_arg_sp_adjust) passed to the exception callback, so it points to the bottom of the callee-saved registers instead of the outgoing arg area. When stack_arg_sp_adjust is 0 (the common case), this is a no-op.
--- arch/x86/net/bpf_jit_comp.c | 2 ++ include/linux/bpf.h | 1 + kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 5 insertions(+), 1 deletion(-)
NOTE:
Existing arm64 jit already supports exception unwinding with stack arguments. In arch/arm64/net/bpf_jit_comp.c, we have
if (!prog->aux->exception_cb) {
...
} else {
/* FP = x2 */
emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
/* SP = FP - 96 */
emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
}There is no SP adjustment on arm64 as SP points directly at the bottom of the main program's callee-saved registers.
arch/x86/net/bpf_jit_comp.c +2 -0
@@ -1789,6 +1789,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);if (arena_vm_start)
include/linux/bpf.h +1 -0
@@ -1736,6 +1736,7 @@ struct bpf_prog_aux {struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ];u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u16 stack_arg_sp_adjust;#ifdef CONFIG_SECURITY void *security;#endif
kernel/bpf/fixups.c +1 -0
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->real_func_cnt = env->subprog_cnt; prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; prog->aux->exception_boundary = func[0 ]->aux->exception_boundary; + prog->aux->stack_arg_sp_adjust = func[0 ]->aux->stack_arg_sp_adjust;bpf_prog_jit_attempt_done (prog);return 0 ; out_free:
kernel/bpf/helpers.c +1 -1
@@ -3301,7 +3301,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)* which skips compiler generated instrumentation to do the same. */ kasan_unpoison_task_stack_below ((void *)(long )ctx.sp); - ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp,0 ,0 ); + ctx.aux->bpf_exception_cb (cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp,0 ,0 );WARN (1 ,"A call to BPF exception callback should never return\n" ); }
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 2/2 selftests/bpf: Add exception tests with stack arguments
Add tests to verify that bpf_throw() correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding x86 fix, these tests crash the kernel on x86 due to corrupted callee-saved register restore. There is no change for arm64 to support exception with stack arguments.
--- .../selftests/bpf/prog_tests/exceptions.c | 7 ++ .../testing/selftests/bpf/progs/exceptions.c | 114 ++++++++++++++++++ 2 files changed, 121 insertions(+)
tools/testing/selftests/bpf/prog_tests/exceptions.c +7 -0
@@ -85,6 +85,13 @@ static void test_exceptions_success(void)RUN_SUCCESS (exception_bad_assert_range_with,10 );RUN_SUCCESS (exception_throw_from_void_global,11 ); +if (skel->rodata->has_stack_arg) { +RUN_SUCCESS (exception_throw_stack_arg,56 ); +RUN_SUCCESS (exception_throw_after_stack_arg,56 ); +RUN_SUCCESS (exception_throw_subprog_stack_arg,56 ); +RUN_SUCCESS (exception_throw_subprog_after_stack_arg,56 ); + } +#define RUN_EXT (load_ret, attach_err, expr, msg, after_link) \ { \ LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf, \
tools/testing/selftests/bpf/progs/exceptions.c +114 -0
118 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 13 lines · On Sun, 17 May 2026 08:07:02 -0700 you wrote:
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next,1/2] bpf,x86: Fix exception unwinding with outgoing stack arguments
https://git.kernel.org/bpf/bpf-next/c/18a37465b0ab - [bpf-next,2/2] selftests/bpf: Add exception tests with stack arguments https://git.kernel.org/bpf/bpf-next/c/576482b55c19
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html