bpf: Follow-up fixes for stack argument support
Subsystems (6)
Yonghong Song
reviewerv3 cover bpf: Follow-up fixes for stack argument support
Commit cd59fa185a03 ("bpf: Support stack arguments for BPF functions and kfuncs") added stack argument support for bpf functions and kfuncs. This patch set is to fix various issues related to stack arguments, mainly include:
- Validate outgoing stack args when btf_prepare_func_args fails
- Fix arg_track_join log to use sa prefix for stack arg slots
- Clean up redundant stack arg checks for non-JITed programs
- Fix exception unwinding with outgoing stack arguments
Changelog:
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Add additional fix (fix arg_track_join log, fix exception with outgoing stack arguments, some cleanup, etc.).
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Remove 'Reported-by: Sashiko <[email protected]>'.
- Add proper fix tag.
Yonghong Song (7):
bpf: Validate outgoing stack args when btf_prepare_func_args fails selftests/bpf: Add test for stack arg read without caller write selftests/bpf: Log arg_track_join for stack arg slots in liveness
analysis
bpf: Fix arg_track_join log to use sa prefix for stack arg slots
bpf: Clean up redundant stack arg checks for non-JITed programs bpf,x86: Fix exception unwinding with outgoing stack arguments selftests/bpf: Add exception tests with stack arguments
arch/x86/net/bpf_jit_comp.c | 9 +- include/linux/bpf.h | 3 +- kernel/bpf/core.c | 2 +- kernel/bpf/fixups.c | 3 +- kernel/bpf/helpers.c | 2 +- kernel/bpf/liveness.c | 4 +- kernel/bpf/verifier.c | 10 +- .../selftests/bpf/prog_tests/exceptions.c | 7 ++ .../bpf/progs/btf__verifier_stack_arg_order.c | 8 ++ .../testing/selftests/bpf/progs/exceptions.c | 113 ++++++++++++++++++ .../selftests/bpf/progs/verifier_stack_arg.c | 8 +- .../bpf/progs/verifier_stack_arg_order.c | 58 +++++++++ 12 files changed, 216 insertions(+), 11 deletions(-)
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 1/7 bpf: Validate outgoing stack args when btf_prepare_func_args fails
btf_prepare_func_args() sets sub->arg_cnt before validating arg types. If validation fails (e.g. unsupported pointer type in a static subprog), check_outgoing_stack_args() is skipped because btf_check_func_arg_match() returns early. For static subprogs, check_func_call() ignores non-EFAULT errors and proceeds with the call.
This causes the callee to read stack arg slots that the caller never stored or not initialized, potentially dereferencing NULL caller->stack_arg_regs or getting no-initialized value.
To fix the issue, when btf_prepare_func_args() fails and the subprog expects stack args, call check_outgoing_stack_args() to verify the caller initialized the slots. Return -EFAULT on failure so the error is not ignored.
--- kernel/bpf/verifier.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +8 -2
@@ -9118,11 +9118,17 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,struct bpf_func_state *caller =cur_func (env);struct bpf_verifier_log *log = &env->log;u32 i; -int ret; +int ret, err; ret =btf_prepare_func_args (env, subprog); -if (ret) +if (ret) { +if (bpf_in_stack_arg_cnt (sub) >0 ) { + err =check_outgoing_stack_args (env, caller, sub->arg_cnt); +if (err) +return err; + }return ret; + } ret =check_outgoing_stack_args (env, caller, sub->arg_cnt);if (ret)
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 2/7 selftests/bpf: Add test for stack arg read without caller write
In reply to Yonghong SongAdd negative tests for the outgoing stack arg validation. A static subprog with a 'long *' arg causes btf_prepare_func_args() to fail after setting arg_cnt. The validation ensures check_outgoing_stack_args() still runs.
Also update two existing tests (release_ref, stale_pkt_ptr) whose expected error messages changed: invalidated stack arg slots are now caught by check_outgoing_stack_args() at the call site instead of at the callee's dereference.
--- .../bpf/progs/btf__verifier_stack_arg_order.c | 8 +++ .../selftests/bpf/progs/verifier_stack_arg.c | 4 +- .../bpf/progs/verifier_stack_arg_order.c | 58 +++++++++++++++++++ 3 files changed, 68 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c +8 -0
@@ -21,6 +21,10 @@ int subprog_pruning_call_before_load_6args(int a, int b, int c, int d, int e, in return a + b + c + d + e + f; } +void subprog_bad_ptr_7args (long *a,int b,int c,int d,int e,int f,int g) +{ +} +#else int subprog_bad_order_6args (void )
@@ -38,4 +42,8 @@ int subprog_pruning_call_before_load_6args(void) return 0; } +void subprog_bad_ptr_7args (void ) +{ +} +#endif
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +2 -2
@@ -152,7 +152,7 @@ __naked void stack_arg_pruning_type_mismatch(void) SEC("tc") __description("stack_arg: release_reference invalidates stack arg slot") __failure -__msg("R{{[0-9]}} !read_ok") +__msg("callee expects 6 args, stack arg1 is not initialized")__naked void stack_arg_release_ref (void ) { asm volatile (
@@ -201,7 +201,7 @@ __naked void stack_arg_release_ref(void) SEC("tc") __description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data") __failure -__msg("R{{[0-9]}} !read_ok") +__msg("callee expects 6 args, stack arg1 is not initialized")__naked void stack_arg_stale_pkt_ptr (void ) { asm volatile (
tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +58 -0
64 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 3/7 selftests/bpf: Log arg_track_join for stack arg slots in liveness analysis
In reply to Yonghong SongCommit 2af4e792773f ("bpf: Extend liveness analysis to track stack argument slots") added stack arg supports. For selftest
verifier_stack_arg/stack_arg: pruning with different stack arg types
the following are two arg JOIN messages:
arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0
arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0
Here the "r11:" label for stack arg slot 0 is misleading since r11 is a special register (BPF_REG_PARAMS). The next patch corrects this to "sa0:", properly representing the 'stack arg slot 0'.
--- tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +3 -1
@@ -114,8 +114,10 @@ __naked void stack_arg_gap_at_minus8(void) SEC("tc") __description("stack_arg: pruning with different stack arg types") -__failure +__failure __log_level(2) __flag(BPF_F_TEST_STATE_FREQ) +__msg("arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0") +__msg("arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0") __msg("R{{[0-9]}} invalid mem access 'scalar'")__naked void stack_arg_pruning_type_mismatch (void ) {
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 4/7 bpf: Fix arg_track_join log to use sa prefix for stack arg slots
In reply to Yonghong Songarg_track_join() logs state transitions at CFG merge points. For stack arg slots (r >= MAX_BPF_REG), it printed "r11:", "r12:", etc., which is misleading since r11 is a special register (BPF_REG_PARAMS) not meaningful to the user.
Fix it to print "sa0:", "sa1:", etc., matching the per-instruction transition log in arg_track_log() which already uses the "sa" prefix.
Update the existing stack_arg_pruning_type_mismatch selftest to expect the corrected format.
--- kernel/bpf/liveness.c | 4 +++- tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)
kernel/bpf/liveness.c +3 -1
@@ -806,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, inreturn true;verbose (env,"arg JOIN insn %d -> %d " , idx, target); -if (r >=0 ) +if (r >=MAX_BPF_REG ) +verbose (env,"sa%d: " , r -MAX_BPF_REG ); +else if (r >=0 )verbose (env,"r%d: " , r);else verbose (env,"fp%+d: " , r *8 );
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +1 -1
@@ -117,7 +117,7 @@ __description("stack_arg: pruning with different stack arg types") __failure __log_level(2) __flag(BPF_F_TEST_STATE_FREQ) __msg("arg JOIN insn 9 -> 10 r1: fp0-8 + _ => fp0-8|fp0+0") -__msg("arg JOIN insn 9 -> 10 r11: fp0-8 + _ => fp0-8|fp0+0") +__msg("arg JOIN insn 9 -> 10 sa0: fp0-8 + _ => fp0-8|fp0+0") __msg("R{{[0-9]}} invalid mem access 'scalar'")__naked void stack_arg_pruning_type_mismatch (void ) {
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 5/7 bpf: Clean up redundant stack arg checks for non-JITed programs
In reply to Yonghong SongRemove a redundant stack_arg_cnt check in __bpf_prog_select_runtime() and start the stack arg loop from index 0 in bpf_fixup_call_args(). Both changes are no-ops that simplify the code:
In __bpf_prog_select_runtime(), the subprog_info[0].stack_arg_cnt check is unreachable:
- when there is only a main program (no bpf-to-bpf calls), subprog_info[0].stack_arg_cnt is always 0 because the main program's arg_cnt is forced to 1
- when bpf-to-bpf calls use stack args and JIT succeeds,
fp->bpf_func is set and this code is skipped
- when JIT fails, bpf_fixup_call_args() rejects the program before we get to __bpf_prog_select_runtime().
In bpf_fixup_call_args(), starting the loop at i=1 skipped subprog 0, which is safe since the main program always has arg_cnt=1 and thus bpf_in_stack_arg_cnt() returns 0. Starting at i=0 removes the need to reason about this invariant.
--- kernel/bpf/core.c | 2 +- kernel/bpf/fixups.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/core.c +1 -1
@@ -2609,7 +2609,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp) || (env && env->subprog_info[0 ].stack_arg_cnt)) +bpf_prog_has_kfunc_call (fp)) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/fixups.c +1 -1
@@ -1407,7 +1407,7 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)verbose (env,"calling kernel functions are not allowed in non-JITed programs\n" );return -EINVAL ; } -for (i =1 ; i < env->subprog_cnt; i++) { +for (i =0 ; i < env->subprog_cnt; i++) {if (bpf_in_stack_arg_cnt (&env->subprog_info[i])) {verbose (env,"stack args are not supported in non-JITed programs\n" );return -EINVAL ;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 6/7 bpf,x86: Fix exception unwinding with outgoing stack arguments
In reply to Yonghong SongWhen a main program with exception_boundary has outgoing stack arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails to correctly restore callee-saved registers, causing a kernel crash.
The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue. When bpf_throw() unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed by pop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.
Fix this by passing the main program's outgoing_rsp as the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. When outgoing_rsp is 0 (the common case), this is a no-op.
--- arch/x86/net/bpf_jit_comp.c | 9 ++++++++- include/linux/bpf.h | 3 ++- kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +8 -1
@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb/* Keep the same instruction layout. */ emit_nops (&prog,3 );/* nop3 */ } -/* Exception callback receives FP as third parameter */ +/* +* Exception callback receives: +* rsi = main program's SP, rdx = main program's FP, +* rcx = main program's outgoing stack arg area size +*/ if (is_exception_cb) {EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +EMIT3 (0x48 ,0x01 ,0xCC );/* add rsp, rcx */ /* The main frame must have exception_boundary as true, so we * first restore those callee-saved regs from stack, before * reusing the stack frame.
@@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);if (arena_vm_start)
include/linux/bpf.h +2 -1
@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 stack_arg_adjust,u64 ); +u16 stack_arg_adjust;#ifdef CONFIG_SECURITY void *security;#endif
kernel/bpf/fixups.c +1 -0
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->real_func_cnt = env->subprog_cnt; prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; prog->aux->exception_boundary = func[0 ]->aux->exception_boundary; + prog->aux->stack_arg_adjust = func[0 ]->aux->stack_arg_adjust;bpf_prog_jit_attempt_done (prog);return 0 ; out_free:
kernel/bpf/helpers.c +1 -1
@@ -3301,7 +3301,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)* which skips compiler generated instrumentation to do the same. */ kasan_unpoison_task_stack_below ((void *)(long )ctx.sp); - ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp,0 ,0 ); + ctx.aux->bpf_exception_cb (cookie, ctx.sp, ctx.bp, ctx.aux->stack_arg_adjust,0 );WARN (1 ,"A call to BPF exception callback should never return\n" ); }
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
Quoted 19 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.
Quoted 3 lines
Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>---
Do we need any adjustment for arm64?
For this patch:
Quoted 1 line
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
Quoted 20 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.
Quoted 4 lines
Quoted 3 lines
Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>---
Do we need any adjustment for arm64?
Yes, after this patch lands, Puranjay can help with arm64 side.
Quoted 5 lines
For this patch:
Quoted 1 line
[...]
Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
In reply to Yonghong SongQuoted 36 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 9 ++++++++- include/linux/bpf.h | 3 ++- kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 12 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +0
-0@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb/* Keep the same instruction layout. */ emit_nops (&prog,3 );/* nop3 */ }
arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +6
-1-/* Exception callback receives FP as third parameter */ +/* +* Exception callback receives: +* rsi = main program's SP, rdx = main program's FP, +* rcx = main program's outgoing stack arg area size +*/ if (is_exception_cb) {EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +EMIT3 (0x48 ,0x01 ,0xCC );/* add rsp, rcx */
Maybe let's do it on C side like:bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0); Avoids the need to use 'rcx'.
Quoted 17 lines
arch/x86/net/bpf_jit_comp.c +2
-0/* The main frame must have exception_boundary as true, so we * first restore those callee-saved regs from stack, before * reusing the stack frame. @@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);if (arena_vm_start)include/linux/bpf.h +0
-0
include/linux/bpf.h struct bpf_prog_aux { · line 1738
Quoted 5 lines
include/linux/bpf.h +1
-1@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 stack_arg_adjust,u64 );
no need to change this.
include/linux/bpf.h struct bpf_prog_aux { · line 1739
Quoted 1 line
include/linux/bpf.h +1
-0+u16 stack_arg_adjust;
this one is still needed, but maybe let's call it stack_arg_sp_adjust?
Looking at arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.
and arm64 seems to work already?
and the field x86 specific? not sure about other archs.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 9 ++++++++- include/linux/bpf.h | 3 ++- kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 12 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb/* Keep the same instruction layout. */ emit_nops (&prog,3 );/* nop3 */ } -/* Exception callback receives FP as third parameter */ +/* +* Exception callback receives: +* rsi = main program's SP, rdx = main program's FP,
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0+* rcx = main program's outgoing stack arg area size +*/ if (is_exception_cb) {EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +EMIT3 (0x48 ,0x01 ,0xCC );/* add rsp, rcx */ Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);
This sounds better!
Quoted 13 lines
Avoids the need to use 'rcx'.
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2
-0/* The main frame must have exception_boundary as true, so we * first restore those callee-saved regs from stack, before * reusing the stack frame. @@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);
Quoted 12 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1738
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0
-0if (arena_vm_start)include/linux/bpf.h +1
-1@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 stack_arg_adjust,u64 );no need to change this.
Indeed, with the above 'ctx.sp + ctx.aux->stack_arg_adjust', the 4th argument does not need any change.
Quoted 2 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1739
Quoted 1 line
include/linux/bpf.h +1
-0+u16 stack_arg_adjust;this one is still needed, but maybe let's call it
stack_arg_sp_adjust?
Okay. Will use the stack_arg_sp_adjust.
Quoted 5 lines
Looking at
arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.and arm64 seems to work already?
I will take a look at arm64 as well.
Quoted 4 lines
and the field x86 specific? not sure about other archs.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
Quoted 90 lines · Yonghong Song
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 9 ++++++++- include/linux/bpf.h | 3 ++- kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 12 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb/* Keep the same instruction layout. */ emit_nops (&prog,3 );/* nop3 */ } -/* Exception callback receives FP as third parameter */ +/* +* Exception callback receives: +* rsi = main program's SP, rdx = main program's FP, Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0+* rcx = main program's outgoing stack arg area size +*/ if (is_exception_cb) {EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +EMIT3 (0x48 ,0x01 ,0xCC );/* add rsp, rcx */ Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);This sounds better!
Quoted 13 lines
Avoids the need to use 'rcx'.
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2
-0/* The main frame must have exception_boundary as true, so we * first restore those callee-saved regs from stack, before * reusing the stack frame. @@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);Quoted 12 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1738
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0
-0if (arena_vm_start)include/linux/bpf.h +1
-1@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 stack_arg_adjust,u64 );no need to change this.
Indeed, with the above 'ctx.sp +
ctx.aux->stack_arg_adjust', the 4th argument does not need any change.Quoted 2 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1739
Quoted 1 line
include/linux/bpf.h +1
-0+u16 stack_arg_adjust;this one is still needed, but maybe let's call it
stack_arg_sp_adjust?Okay. Will use the
stack_arg_sp_adjust.Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Looking at
arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.and arm64 seems to work already?
I will take a look at arm64 as well.
Please also remove guards on tests to allow them to run on arm64, once it is handled properly.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments
Quoted 86 lines · Kumar Kartikeya Dwivedi
Quoted 85 lines · Yonghong Song
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Yonghong Song
When a main program with
exception_boundaryhas outgoing stack arguments (e.g. from calling subprogs with >5 args),bpf_throw()fails to correctly restore callee-saved registers, causing a kernel crash.The x86 JIT allocates the outgoing stack arg area below the callee-saved registers via 'sub rsp,
outgoing_rsp' in the prologue. Whenbpf_throw()unwinds, it captures the main program's sp (which includes this outgoing area) and passes it to the exception callback. The callback gets rsp and rbp, followed bypop_callee_regs, but rsp points into the outgoing arg area rather than the callee-saved registers, so the pops restore garbage values. Returning to the kernel with corrupted callee-saved registers causes a crash.Fix this by passing the main program's
outgoing_rspas the 4th argument to the exception callback. The callback adjusts rsp with 'add rsp, rcx' before popping callee-saved registers, correctly skipping the outgoing arg area. Whenoutgoing_rspis 0 (the common case), this is a no-op.Fixes 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 9 ++++++++- include/linux/bpf.h | 3 ++- kernel/bpf/fixups.c | 1 + kernel/bpf/helpers.c | 2 +- 4 files changed, 12 insertions(+), 3 deletions(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb/* Keep the same instruction layout. */ emit_nops (&prog,3 );/* nop3 */ } -/* Exception callback receives FP as third parameter */ +/* +* Exception callback receives: +* rsi = main program's SP, rdx = main program's FP, Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb · line 568
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0+* rcx = main program's outgoing stack arg area size +*/ if (is_exception_cb) {EMIT3 (0x48 ,0x89 ,0xF4 );/* mov rsp, rsi */ EMIT3 (0x48 ,0x89 ,0xD5 );/* mov rbp, rdx */ +EMIT3 (0x48 ,0x01 ,0xCC );/* add rsp, rcx */ Maybe let's do it on C side like:
bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);This sounds better!
Quoted 13 lines
Avoids the need to use 'rcx'.
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +2
-0/* The main frame must have exception_boundary as true, so we * first restore those callee-saved regs from stack, before * reusing the stack frame. @@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int ** Arg 6 goes into r9 register, not on stack. */ outgoing_rsp = out_stack_arg_cnt >1 ? (out_stack_arg_cnt -1 ) *8 :0 ; +if (bpf_prog->aux->exception_boundary) + bpf_prog->aux->stack_arg_adjust = outgoing_rsp;emit_sub_rsp (&prog, outgoing_rsp);Quoted 12 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1738
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +0
-0if (arena_vm_start)include/linux/bpf.h +1
-1@@ -1735,7 +1735,8 @@ struct bpf_prog_aux {int cgroup_atype;/* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE ];char name[BPF_OBJ_NAME_LEN ]; -u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 ); +u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 stack_arg_adjust,u64 );no need to change this.
Indeed, with the above 'ctx.sp +
ctx.aux->stack_arg_adjust', the 4th argument does not need any change.Quoted 2 lines
include/linux/bpf.h struct bpf_prog_aux { · line 1739
Quoted 1 line
include/linux/bpf.h +1
-0+u16 stack_arg_adjust;this one is still needed, but maybe let's call it
stack_arg_sp_adjust?Okay. Will use the
stack_arg_sp_adjust.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Looking at
arch/arm64/net/bpf_jit_comp.c:590
it doesn't use SP, so should it fine.and arm64 seems to work already?
I will take a look at arm64 as well.
Please also remove guards on tests to allow them to run on arm64, once it is handled properly.
Done. See
https://lore.kernel.org/bpf/[email protected]/
No change is needed for arm64. It already can handle exception and stack arg's together.
Yonghong Song
reviewerPATCH v3 7/7 selftests/bpf: Add exception tests with stack arguments
In reply to Yonghong SongAdd tests to verify that bpf_throw() correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding fix, these tests crash the kernel due to corrupted callee-saved register restore.
--- .../selftests/bpf/prog_tests/exceptions.c | 7 ++ .../testing/selftests/bpf/progs/exceptions.c | 113 ++++++++++++++++++ 2 files changed, 120 insertions(+)
tools/testing/selftests/bpf/prog_tests/exceptions.c +7 -0
@@ -85,6 +85,13 @@ static void test_exceptions_success(void)RUN_SUCCESS (exception_bad_assert_range_with,10 );RUN_SUCCESS (exception_throw_from_void_global,11 ); +if (skel->rodata->has_stack_arg) { +RUN_SUCCESS (exception_throw_stack_arg,56 ); +RUN_SUCCESS (exception_throw_after_stack_arg,56 ); +RUN_SUCCESS (exception_throw_subprog_stack_arg,56 ); +RUN_SUCCESS (exception_throw_subprog_after_stack_arg,56 ); + } +#define RUN_EXT (load_ret, attach_err, expr, msg, after_link) \ { \ LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf, \
tools/testing/selftests/bpf/progs/exceptions.c +113 -0
117 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next v3 7/7] selftests/bpf: Add exception tests with stack arguments
Quoted 7 lines · Yonghong Song
Add tests to verify that
bpf_throw()correctly unwinds the stack when the program uses outgoing stack arguments (functions with >5 args). Without the preceding fix, these tests crash the kernel due to corrupted callee-saved register restore.Signed-off-by Yonghong Song <[email protected]>---
Quoted 1 line
[...]