bpf: Support stack arguments for BPF functions and kfuncs
Yonghong Song
reviewercover bpf: Support stack arguments for BPF functions and kfuncs
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits stores/loads through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.
The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by (incoming + max_outgoing) bytes to hold the stack arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL and popped after return. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).
Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.
For the rest of patches, patches 1-5 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 6 enables x86_64 for stack arguments. Patch 7 implemented JIT for x86_64. Patches 8-10 are some selftests.
[1] https://github.com/llvm/llvm-project/pull/189060
Yonghong Song (10):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for oversized kfunc stack argument selftests/bpf: Add verifier tests for stack argument validation
arch/x86/net/bpf_jit_comp.c | 150 +++++++- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 15 +- include/linux/filter.h | 4 +- kernel/bpf/btf.c | 21 +- kernel/bpf/core.c | 12 +- kernel/bpf/verifier.c | 351 ++++++++++++++++-- .../selftests/bpf/prog_tests/stack_arg.c | 143 +++++++ .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++ .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/stack_arg.c | 111 ++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 ++ .../selftests/bpf/progs/stack_arg_kfunc.c | 59 +++ .../selftests/bpf/progs/verifier_stack_arg.c | 122 ++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 29 ++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 14 + 16 files changed, 1045 insertions(+), 50 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 1/10 bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf register R12 added by [1]. R12 is used as the base for stack arguments so it won't mess out R10 based stacks.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/filter.h | 3 ++- kernel/bpf/core.c | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/filter.h +2 -1
@@ -59,7 +59,8 @@ struct ctl_table_header;/* Kernel hidden auxiliary/helper register. */ #define BPF_REG_AX MAX_BPF_REG -#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) +#define BPF_REG_STACK_ARG_BASE (MAX_BPF_REG + 1) +#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG/* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,u32 imm_rnd =get_random_u32 ();s16 off; -BUILD_BUG_ON (BPF_REG_AX +1 !=MAX_BPF_JIT_REG ); -BUILD_BUG_ON (MAX_BPF_REG +1 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_AX +2 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_STACK_ARG_BASE +1 !=MAX_BPF_JIT_REG );/* Constraints on AX register: *
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 2/10 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongCurrently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.
--- include/linux/bpf.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 3/10 bpf: Support stack arguments for bpf functions
In reply to Yonghong SongCurrently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.
The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar(a1, a2, a3, a4, a5, a6, a7, a8);
...
}The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.
If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar1(a1, a2, a3, a4, a5, a6, a7, a8);
...
bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
...
Back from bar1
... Callee (bar2)
=== =============
+---> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
|+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
|||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+||| ...
r12-32: [outgoing arg 7 to callee]-+||
r12-40: [outgoing arg 8 to callee]--+|r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 15 ++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 248 ++++++++++++++++++++++++++++++++--- 4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +13 -2
@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; }; +struct bpf_stack_arg_state { +struct bpf_reg_state spilled_ptr;/* for spilled scalar/pointer semantics */ +u8 slot_type[BPF_REG_SIZE ]; +}; +/* state of the program: * type of all registers and stack info */
@@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };#define MAX_CALL_FRAMES 8
@@ -674,10 +683,12 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; };struct bpf_verifier_env ;
kernel/bpf/btf.c +10 -4
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))
kernel/bpf/verifier.c +232 -16
340 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
The new stack_arg_slots state is part of bpf_func_state and is properly copied in copy_stack_state() and freed in free_func_state(), but stacksafe() does not compare it during state pruning.
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0
[ ... ]
include/linux/bpf_verifier.h struct bpf_func_state { · line 331
Quoted 9 lines
include/linux/bpf_verifier.h +4
-0@@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };
The new stack_arg_slots state is part of bpf_func_state and is properly copied in copy_stack_state() and freed in free_func_state(), but stacksafe() does not compare it during state pruning.
stacksafe() only iterates the regular stack:
kernel/bpf/verifier.c:stacksafe() {
for (i = 0; i < old->allocated_stack; i++) {
...
old->stack[spi].slot_type[...]
...
}
return true;
}It never examines old->stack_arg_slots or cur->stack_arg_slots. This means func_states_equal() -> stacksafe() treats two states as equivalent even when they differ in their stack arg slot contents.
Could this allow the verifier to incorrectly prune a path where outgoing stack arg slots are uninitialized? For example, if a BPF program has two paths to a call instruction -- one that writes the outgoing stack args and one that does not -- the verifier could explore the initialized path first, save the state at a checkpoint, then prune the uninitialized path as equivalent. The initialization check in set_callee_state():
kernel/bpf/verifier.c:set_callee_state() {
for (i = 0; i < callee_incoming_slots; i++) {
int caller_spi = i + caller_incoming_slots;
if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
verbose(env, "stack arg#%d not properly initialized\n",
i + 1 + MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
}
}would never run on the pruned path, potentially allowing the callee to receive uninitialized data.
It seems like stacksafe() needs a companion loop for stack_arg_slots, similar to how it walks the regular stack, to maintain verifier soundness for this new state.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 55 lines · On 4/1/26 8:18 PM, [email protected] wrote:
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0[ ... ]
include/linux/bpf_verifier.h struct bpf_func_state { · line 331
Quoted 9 lines
include/linux/bpf_verifier.h +4
-0@@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };The new
stack_arg_slotsstate is part ofbpf_func_stateand is properly copied incopy_stack_state()and freed infree_func_state(), butstacksafe()does not compare it during state pruning.
stacksafe()only iterates the regular stack:kernel/bpf/verifier.c:stacksafe() { for (i = 0; i < old->allocated_stack; i++) { ... old->stack[spi].slot_type[...] ... } return true; }It never examines
old->stack_arg_slotsorcur->stack_arg_slots. This meansfunc_states_equal()->stacksafe()treats two states as equivalent even when they differ in their stack arg slot contents.Could this allow the verifier to incorrectly prune a path where outgoing stack arg slots are uninitialized? For example, if a BPF program has two paths to a call instruction -- one that writes the outgoing stack args and one that does not -- the verifier could explore the initialized path first, save the state at a checkpoint, then prune the uninitialized path as equivalent. The initialization check in
set_callee_state():kernel/bpf/verifier.c:set_callee_state() { for (i = 0; i < callee_incoming_slots; i++) { int caller_spi = i + caller_incoming_slots; if (!is_stack_arg_slot_initialized(caller, caller_spi)) { verbose(env, "stack arg#%d not properly initialized\n", i + 1 + MAX_BPF_FUNC_REG_ARGS); return -EINVAL; } callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi]; } }
Quoted 6 lines
would never run on the pruned path, potentially allowing the callee to receive uninitialized data.
It seems like
stacksafe()needs a companion loop forstack_arg_slots, similar to how it walks the regular stack, to maintain verifier soundness for this new state.
You are absolutely correct. Indeed, we should do some checking for stack_arg_slots along side stacksafe(). I will fix it in the next revision.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongQuoted 277 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundaryr12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new '
bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr+ slot_type[]) but lives in a dedicated 'stack_arg_slots' array inbpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar1(a1, a2, a3, a4, a5, a6, a7, a8); ... bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9); ... }The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+ ... Back from bar1 ... Callee (bar2) === ============= +---> r12-8: [incoming arg 6] (from caller's outgoing r12-24) |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40) |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48) ---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+||| ... r12-32: [outgoing arg 7 to callee]-+|| r12-40: [outgoing arg 8 to callee]--+|r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 15 ++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 248 ++++++++++++++++++++++++++++++++--- 4 files changed, 257 insertions(+), 22 deletions(-)include/linux/bpf.h +2
-0@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */ include/linux/bpf_verifier.h +13
-2@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; }; +struct bpf_stack_arg_state { +struct bpf_reg_state spilled_ptr;/* for spilled scalar/pointer semantics */ +u8 slot_type[BPF_REG_SIZE ]; +}; +/* state of the program: * type of all registers and stack info */ @@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };#define MAX_CALL_FRAMES 8@@ -674,10 +683,12 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; };struct bpf_verifier_env ;kernel/bpf/btf.c +10
-4@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))kernel/bpf/verifier.c +76
-0@@ -1488,6 +1488,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_streturn -ENOMEM ; dst->allocated_stack = src->allocated_stack; + +/* copy stack_arg_slots state */ + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_slots =copy_array (dst->stack_arg_slots, src->stack_arg_slots, n, +sizeof (struct bpf_stack_arg_state ), +GFP_KERNEL_ACCOUNT ); +if (!dst->stack_arg_slots) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + }return 0 ; }@@ -1529,6 +1542,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_statereturn 0 ; } +static int grow_stack_arg_slots (struct bpf_verifier_env *env, +struct bpf_func_state *state,int size) +{ +size_t old_n = state->stack_arg_depth /BPF_REG_SIZE , n; + + size =round_up (size,BPF_REG_SIZE ); + n = size /BPF_REG_SIZE ; +if (old_n >= n) +return 0 ; + + state->stack_arg_slots =realloc_array (state->stack_arg_slots, old_n, n, +sizeof (struct bpf_stack_arg_state )); +if (!state->stack_arg_slots) +return -ENOMEM ; + + state->stack_arg_depth = size; +return 0 ; +} +/* Acquire a pointer id from the env and update the state->refs to include * this new pointer reference. * On success, returns a valid pointer id to associate with the register @@ -1699,6 +1731,7 @@ static void free_func_state(struct bpf_func_state *state) {if (!state)return ; +kfree (state->stack_arg_slots);kfree (state->stack);kfree (state); }@@ -5848,6 +5881,101 @@ static int check_stack_write(struct bpf_verifier_env *env,return err; } +/* Validate that a stack arg access is 8-byte sized and aligned. */ +static int check_stack_arg_access (struct bpf_verifier_env *env, +struct bpf_insn *insn,const char *op) +{ +int size =bpf_size_to_bytes (BPF_SIZE (insn->code)); + +if (size !=BPF_REG_SIZE ) { +verbose (env,"stack arg %s must be %d bytes, got %d\n" , + op,BPF_REG_SIZE , size); +return -EINVAL ; + } +if (insn->off %BPF_REG_SIZE ) { +verbose (env,"stack arg %s offset %d not aligned to %d\n" , + op, insn->off,BPF_REG_SIZE ); +return -EINVAL ; + } +return 0 ; +} + +/* Check that a stack arg slot has been properly initialized. */ +static bool is_stack_arg_slot_initialized (struct bpf_func_state *state,int spi) +{ +u8 type; + +if (spi >= (int )(state->stack_arg_depth /BPF_REG_SIZE )) +return false; + type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE -1 ]; +return type ==STACK_SPILL || type ==STACK_MISC ; +} + +/* +* Write a value to the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ; +struct bpf_func_state *cur; +struct bpf_reg_state *reg; +int i, err; +u8 type;
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 5936
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ + err =grow_stack_arg_slots (env, state, -off); +if (err) +return err; + + cur = env->cur_state->frame[env->cur_state->curframe]; +if (value_regno >=0 ) { + reg = &cur->regs[value_regno]; + state->stack_arg_slots[spi].spilled_ptr = *reg; + type =is_spillable_regtype (reg->type) ?STACK_SPILL :STACK_MISC ;
It seems any spillable register can be passed to the callee, so reg containing ref_obj_id can be spilled to stack_arg_slots. However, release_reference() does not invalidate ref_obj_id in stack_arg_slots. Can this cause UAF like below?
a6 = bpf_task_acquire(t);
if (!a6)
goto err;
// a6 now has a valid ref_obj_id
// foo1 calls bpf_task_release(a6);foo1(a1, a2, a3, a4, a5, a6);
// a6 still has a valid ref_obj_id
// foo2 dereference a6 -> UAFfoo2(a1, a2, a3, a4, a5, a6);
Since stack_arg_slots is separated from the normal stack slots, other types of stale registers may exist in the outgoing stack slots. For example:
- stale pkt pointer after calling clear_all_pkt_pointers() in callee
- register with inprecise nullness after calling mark_ptr_or_null_regs() in callee
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] = type; + }else { +/* BPF_ST: store immediate, treat as scalar */
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 109 lines · Amery Hung
Quoted 108 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new '
bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr+ slot_type[]) but lives in a dedicated 'stack_arg_slots' array inbpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar1(a1, a2, a3, a4, a5, a6, a7, a8); ... bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9); ... }The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+ ... Back from bar1 ... Callee (bar2) === ============= +---> r12-8: [incoming arg 6] (from caller's outgoing r12-24) |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40) |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48) ---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+||| ... r12-32: [outgoing arg 7 to callee]-+|| r12-40: [outgoing arg 8 to callee]--+| r12-48: [outgoing arg 9 to callee]---+Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 15 ++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 248 ++++++++++++++++++++++++++++++++--- 4 files changed, 257 insertions(+), 22 deletions(-)include/linux/bpf.h +2
-0@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */ include/linux/bpf_verifier.h +0
-0@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; };
...
Quoted 41 lines
Quoted 14 lines
include/linux/bpf_verifier.h +14
-0+ +/* +* Write a value to the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ; +struct bpf_func_state *cur; +struct bpf_reg_state *reg; +int i, err; +u8 type;include/linux/bpf_verifier.h
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+ + err =grow_stack_arg_slots (env, state, -off); +if (err) +return err; + + cur = env->cur_state->frame[env->cur_state->curframe]; +if (value_regno >=0 ) { + reg = &cur->regs[value_regno]; + state->stack_arg_slots[spi].spilled_ptr = *reg; + type =is_spillable_regtype (reg->type) ?STACK_SPILL :STACK_MISC ;It seems any spillable register can be passed to the callee, so reg containing
ref_obj_idcan be spilled tostack_arg_slots. However,release_reference()does not invalidateref_obj_idinstack_arg_slots. Can this cause UAF like below?a6 =
bpf_task_acquire(t);
if (!a6)goto err;// a6 now has a valid
ref_obj_id
// foo1 callsbpf_task_release(a6);foo1(a1, a2, a3, a4, a5, a6);// a6 still has a valid
ref_obj_id
// foo2 dereference a6 -> UAFfoo2(a1, a2, a3, a4, a5, a6);
Quoted 6 lines
Since
stack_arg_slotsis separated from the normal stack slots, other types of stale registers may exist in the outgoing stack slots. For example:
- stale pkt pointer after callingclear_all_pkt_pointers()in callee
- register with inprecise nullness after callingmark_ptr_or_null_regs()in callee
Thanks for pointing this out. Indeed, a lot of checking are needed for stack arguments. Looks like I need to add something like
bpf_for_each_spilled_stack_arg
in bpf_for_each_reg_in_vstate_mask for full coverage.
Will fix in the next resivion.
Quoted 6 lines
Quoted 6 lines
include/linux/bpf_verifier.h +4
-0+for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] = type; + }else { +/* BPF_ST: store immediate, treat as scalar */
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongQuoted 408 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundaryr12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new '
bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr+ slot_type[]) but lives in a dedicated 'stack_arg_slots' array inbpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar1(a1, a2, a3, a4, a5, a6, a7, a8); ... bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9); ... }The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+ ... Back from bar1 ... Callee (bar2) === ============= +---> r12-8: [incoming arg 6] (from caller's outgoing r12-24) |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40) |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48) ---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+||| ... r12-32: [outgoing arg 7 to callee]-+|| r12-40: [outgoing arg 8 to callee]--+|r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 15 ++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 248 ++++++++++++++++++++++++++++++++--- 4 files changed, 257 insertions(+), 22 deletions(-)include/linux/bpf.h +2
-0@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */ include/linux/bpf_verifier.h +13
-2@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; }; +struct bpf_stack_arg_state { +struct bpf_reg_state spilled_ptr;/* for spilled scalar/pointer semantics */ +u8 slot_type[BPF_REG_SIZE ]; +}; +/* state of the program: * type of all registers and stack info */ @@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };#define MAX_CALL_FRAMES 8@@ -674,10 +683,12 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; };struct bpf_verifier_env ;kernel/bpf/btf.c +10
-4@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))kernel/bpf/verifier.c +165
-0@@ -1488,6 +1488,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_streturn -ENOMEM ; dst->allocated_stack = src->allocated_stack; + +/* copy stack_arg_slots state */ + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_slots =copy_array (dst->stack_arg_slots, src->stack_arg_slots, n, +sizeof (struct bpf_stack_arg_state ), +GFP_KERNEL_ACCOUNT ); +if (!dst->stack_arg_slots) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + }return 0 ; }@@ -1529,6 +1542,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_statereturn 0 ; } +static int grow_stack_arg_slots (struct bpf_verifier_env *env, +struct bpf_func_state *state,int size) +{ +size_t old_n = state->stack_arg_depth /BPF_REG_SIZE , n; + + size =round_up (size,BPF_REG_SIZE ); + n = size /BPF_REG_SIZE ; +if (old_n >= n) +return 0 ; + + state->stack_arg_slots =realloc_array (state->stack_arg_slots, old_n, n, +sizeof (struct bpf_stack_arg_state )); +if (!state->stack_arg_slots) +return -ENOMEM ; + + state->stack_arg_depth = size; +return 0 ; +} +/* Acquire a pointer id from the env and update the state->refs to include * this new pointer reference. * On success, returns a valid pointer id to associate with the register @@ -1699,6 +1731,7 @@ static void free_func_state(struct bpf_func_state *state) {if (!state)return ; +kfree (state->stack_arg_slots);kfree (state->stack);kfree (state); }@@ -5848,6 +5881,101 @@ static int check_stack_write(struct bpf_verifier_env *env,return err; } +/* Validate that a stack arg access is 8-byte sized and aligned. */ +static int check_stack_arg_access (struct bpf_verifier_env *env, +struct bpf_insn *insn,const char *op) +{ +int size =bpf_size_to_bytes (BPF_SIZE (insn->code)); + +if (size !=BPF_REG_SIZE ) { +verbose (env,"stack arg %s must be %d bytes, got %d\n" , + op,BPF_REG_SIZE , size); +return -EINVAL ; + } +if (insn->off %BPF_REG_SIZE ) { +verbose (env,"stack arg %s offset %d not aligned to %d\n" , + op, insn->off,BPF_REG_SIZE ); +return -EINVAL ; + } +return 0 ; +} + +/* Check that a stack arg slot has been properly initialized. */ +static bool is_stack_arg_slot_initialized (struct bpf_func_state *state,int spi) +{ +u8 type; + +if (spi >= (int )(state->stack_arg_depth /BPF_REG_SIZE )) +return false; + type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE -1 ]; +return type ==STACK_SPILL || type ==STACK_MISC ; +} + +/* +* Write a value to the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ; +struct bpf_func_state *cur; +struct bpf_reg_state *reg; +int i, err; +u8 type; + + err =grow_stack_arg_slots (env, state, -off); +if (err) +return err; + + cur = env->cur_state->frame[env->cur_state->curframe]; +if (value_regno >=0 ) { + reg = &cur->regs[value_regno]; + state->stack_arg_slots[spi].spilled_ptr = *reg; + type =is_spillable_regtype (reg->type) ?STACK_SPILL :STACK_MISC ; +for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] = type; + }else { +/* BPF_ST: store immediate, treat as scalar */ + reg = &state->stack_arg_slots[spi].spilled_ptr; + reg->type =SCALAR_VALUE ; +__mark_reg_known (reg, (u32 )env->prog->insnsi[env->insn_idx].imm); +for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] =STACK_MISC ; + } +return 0 ; +} + +/* +* Read a value from the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_read (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int dst_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ; +struct bpf_func_state *cur; +u8 *stype; + +if (-off > state->stack_arg_depth) { +verbose (env,"invalid read from stack arg off %d depth %d\n" , + off, state->stack_arg_depth); +return -EACCES ; + } + + stype = state->stack_arg_slots[spi].slot_type; + cur = env->cur_state->frame[env->cur_state->curframe]; + +if (stype[BPF_REG_SIZE -1 ] ==STACK_SPILL ) +copy_register_state (&cur->regs[dst_regno], + &state->stack_arg_slots[spi].spilled_ptr); +else +mark_reg_unknown (env, cur->regs, dst_regno); +return 0 ; +} +static int check_map_access_type (struct bpf_verifier_env *env,u32 regno,int off,int size,enum bpf_access_type type) {@@ -8022,10 +8150,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, bool strict_alignment_once, bool is_ldsx, bool allow_trust_mismatch, const char *ctx) { + struct bpf_verifier_state *vstate = env->cur_state; +struct bpf_func_state *state = vstate->frame[vstate->curframe];struct bpf_reg_state *regs =cur_regs (env);enum bpf_reg_type src_reg_type;int err; +/* Handle stack arg access */ +if (insn->src_reg ==BPF_REG_STACK_ARG_BASE ) { + err =check_reg_arg (env, insn->dst_reg,DST_OP_NO_MARK ); +if (err) +return err; + err =check_stack_arg_access (env, insn,"read" ); +if (err) +return err; +return check_stack_arg_read (env, state, insn->off, insn->dst_reg); + } +/* check src operand */ err =check_reg_arg (env, insn->src_reg,SRC_OP );if (err)@@ -8054,10 +8195,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,static int check_store_reg (struct bpf_verifier_env *env,struct bpf_insn *insn,bool strict_alignment_once) { +struct bpf_verifier_state *vstate = env->cur_state; +struct bpf_func_state *state = vstate->frame[vstate->curframe];struct bpf_reg_state *regs =cur_regs (env);enum bpf_reg_type dst_reg_type;int err; +/* Handle stack arg write */ +if (insn->dst_reg ==BPF_REG_STACK_ARG_BASE ) { + err =check_reg_arg (env, insn->src_reg,SRC_OP ); +if (err) +return err; + err =check_stack_arg_access (env, insn,"write" ); +if (err) +return err; +return check_stack_arg_write (env, state, insn->off, insn->src_reg); + } +/* check src1 operand */ err =check_reg_arg (env, insn->src_reg,SRC_OP );if (err)@@ -10940,8 +11094,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx) {struct bpf_verifier_state *state = env->cur_state; +struct bpf_subprog_info *caller_info;struct bpf_func_state *caller;int err, subprog, target_insn; +u16 callee_incoming; target_insn = *insn_idx + insn->imm +1 ; subprog =find_subprog (env, target_insn);@@ -10993,6 +11149,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return 0 ; } +/* +* Track caller's outgoing stack arg depth (max across all callees). +* This is needed so the JIT knows how much stack arg space to allocate. +*/ + caller_info = &env->subprog_info[caller->subprogno]; + callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth; +if (callee_incoming > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = callee_incoming; +/* for regular function entry setup new frame and continue * from that frame. */
kernel/bpf/verifier.c static int set_callee_state(struct bpf_verifier_env *env, · line 11215
Quoted 3 lines
kernel/bpf/verifier.c +0
-0@@ -11048,13 +11213,41 @@ static int set_callee_state(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee, int insn_idx) {
Taking note when reading the change to set_callee_state():
The function is not called when handling callback function, which uses push_callback_call() -> setup_func_entry() -> callback specific set_callee_state_cb. So caller stack argument will not be transferred.
This should be fine as callee's stack_arg_depth will remain zero and then when callee tries to do r12 based load, check_stack_arg_read() should reject the program. Not sure if this needs a selftest since callbacks' set_callee_state_cb will also transfer register state very intentionally.
Quoted 121 lines
kernel/bpf/verifier.c +67
-17-int i; +struct bpf_subprog_info *callee_info; +int i, err;/* copy r1 - r5 args that callee can access. The copy includes parent * pointers, which connects us up to the liveness chain */ for (i =BPF_REG_1 ; i <=BPF_REG_5 ; i++) callee->regs[i] = caller->regs[i]; + +/* +* Transfer stack args from caller's outgoing area to callee's incoming area. +* Caller wrote outgoing args at offsets '-(incoming + 8)', '-(incoming + 16)', ... +* These outgoing args will go to callee's incoming area. +*/ + callee_info = &env->subprog_info[callee->subprogno]; +if (callee_info->incoming_stack_arg_depth) { +int caller_incoming_slots = caller->incoming_stack_arg_depth /BPF_REG_SIZE ; +int callee_incoming_slots = callee_info->incoming_stack_arg_depth /BPF_REG_SIZE ; + + callee->incoming_stack_arg_depth = callee_info->incoming_stack_arg_depth; + err =grow_stack_arg_slots (env, callee, callee_info->incoming_stack_arg_depth); +if (err) +return err; + +for (i =0 ; i < callee_incoming_slots; i++) { +int caller_spi = i + caller_incoming_slots; + +if (!is_stack_arg_slot_initialized (caller, caller_spi)) { +verbose (env,"stack arg#%d not properly initialized\n" , + i +1 +MAX_BPF_FUNC_REG_ARGS ); +return -EINVAL ; + } + callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi]; + } + }return 0 ; }@@ -21262,23 +21455,37 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)verbose (env,"BPF_ST uses reserved fields\n" );return -EINVAL ; } -/* check src operand */ - err =check_reg_arg (env, insn->dst_reg,SRC_OP ); -if (err) -return err; - dst_reg_type =cur_regs (env)[insn->dst_reg].type; +/* Handle stack arg write (store immediate) */ +if (insn->dst_reg ==BPF_REG_STACK_ARG_BASE ) { +struct bpf_verifier_state *vstate = env->cur_state; +struct bpf_func_state *state = vstate->frame[vstate->curframe]; -/* check that memory (dst_reg + off) is writeable */ - err =check_mem_access (env, env->insn_idx, insn->dst_reg, - insn->off,BPF_SIZE (insn->code), -BPF_WRITE ,-1 , false, false); -if (err) -return err; + err =check_stack_arg_access (env, insn,"write" ); +if (err) +return err; + err =check_stack_arg_write (env, state, insn->off,-1 ); +if (err) +return err; + }else { +/* check src operand */ + err =check_reg_arg (env, insn->dst_reg,SRC_OP ); +if (err) +return err; - err =save_aux_ptr_type (env, dst_reg_type, false); -if (err) -return err; + dst_reg_type =cur_regs (env)[insn->dst_reg].type; + +/* check that memory (dst_reg + off) is writeable */ + err =check_mem_access (env, env->insn_idx, insn->dst_reg, + insn->off,BPF_SIZE (insn->code), +BPF_WRITE ,-1 , false, false); +if (err) +return err; + + err =save_aux_ptr_type (env, dst_reg_type, false); +if (err) +return err; + } }else if (class ==BPF_JMP || class ==BPF_JMP32 ) {u8 opcode =BPF_OP (insn->code);@@ -22974,8 +23181,14 @@ static int jit_subprogs(struct bpf_verifier_env *env)int err, num_exentries;int old_len, subprog_start_adjustment =0 ; -if (env->subprog_cnt <=1 ) +if (env->subprog_cnt <=1 ) { +/* +* Even without subprogs, kfunc calls with >5 args need stack arg space +* allocated by the root program. +*/ + prog->aux->stack_arg_depth = env->subprog_info[0 ].outgoing_stack_arg_depth;return 0 ; + }for (i =0 , insn = prog->insnsi; i < prog->len; i++, insn++) {if (!bpf_pseudo_func (insn) && !bpf_pseudo_call (insn))@@ -23065,6 +23278,9 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->name[0 ] ='F' ; func[i]->aux->stack_depth = env->subprog_info[i].stack_depth; + func[i]->aux->incoming_stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth; + func[i]->aux->stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth + + env->subprog_info[i].outgoing_stack_arg_depth;if (env->subprog_info[i].priv_stack_mode ==PRIV_STACK_ADAPTIVE ) func[i]->aux->jits_use_priv_stack = true; --2.52.0
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 102 lines · Amery Hung
Quoted 101 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new '
bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr+ slot_type[]) but lives in a dedicated 'stack_arg_slots' array inbpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar1(a1, a2, a3, a4, a5, a6, a7, a8); ... bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9); ... }The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+ ... Back from bar1 ... Callee (bar2) === ============= +---> r12-8: [incoming arg 6] (from caller's outgoing r12-24) |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40) |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48) ---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+||| ... r12-32: [outgoing arg 7 to callee]-+|| r12-40: [outgoing arg 8 to callee]--+| r12-48: [outgoing arg 9 to callee]---+Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 15 ++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 248 ++++++++++++++++++++++++++++++++--- 4 files changed, 257 insertions(+), 22 deletions(-)include/linux/bpf.h +2
-0@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */
[...]
Quoted 61 lines
Quoted 52 lines
include/linux/bpf.h +24
-0@@ -8054,10 +8195,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,static int check_store_reg (struct bpf_verifier_env *env,struct bpf_insn *insn,bool strict_alignment_once) { +struct bpf_verifier_state *vstate = env->cur_state; +struct bpf_func_state *state = vstate->frame[vstate->curframe];struct bpf_reg_state *regs =cur_regs (env);enum bpf_reg_type dst_reg_type;int err; +/* Handle stack arg write */ +if (insn->dst_reg ==BPF_REG_STACK_ARG_BASE ) { + err =check_reg_arg (env, insn->src_reg,SRC_OP ); +if (err) +return err; + err =check_stack_arg_access (env, insn,"write" ); +if (err) +return err; +return check_stack_arg_write (env, state, insn->off, insn->src_reg); + } +/* check src1 operand */ err =check_reg_arg (env, insn->src_reg,SRC_OP );if (err)@@ -10940,8 +11094,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx) {struct bpf_verifier_state *state = env->cur_state; +struct bpf_subprog_info *caller_info;struct bpf_func_state *caller;int err, subprog, target_insn; +u16 callee_incoming; target_insn = *insn_idx + insn->imm +1 ; subprog =find_subprog (env, target_insn);@@ -10993,6 +11149,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return 0 ; } +/* +* Track caller's outgoing stack arg depth (max across all callees). +* This is needed so the JIT knows how much stack arg space to allocate. +*/ + caller_info = &env->subprog_info[caller->subprogno]; + callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth; +if (callee_incoming > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = callee_incoming; +/* for regular function entry setup new frame and continue * from that frame. */ include/linux/bpf.h static int set_callee_state(struct bpf_verifier_env *env, · line 11215
Quoted 3 lines
include/linux/bpf.h +0
-0@@ -11048,13 +11213,41 @@ static int set_callee_state(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee, int insn_idx) {Taking note when reading the change to
set_callee_state():The function is not called when handling callback function, which uses
push_callback_call()->setup_func_entry()-> callback specificset_callee_state_cb. So caller stack argument will not be transferred.
Quoted 5 lines
This should be fine as callee's
stack_arg_depthwill remain zero and then when callee tries to do r12 based load,check_stack_arg_read()should reject the program. Not sure if this needs a selftest since callbacks'set_callee_state_cbwill also transfer register state very intentionally.
All callback functions are carefully designed in kernel. So far all callback functions are within 5 register parameters. I ignore them for now. If in the future, there is a need for callback functions with more than 5 arguments, we can deal with them at that time.
Quoted 10 lines
Quoted 10 lines
include/linux/bpf.h +2
-1-int i; +struct bpf_subprog_info *callee_info; +int i, err;/* copy r1 - r5 args that callee can access. The copy includes parent * pointers, which connects us up to the liveness chain */ for (i =BPF_REG_1 ; i <=BPF_REG_5 ; i++) callee->regs[i] = caller->regs[i];
[...]
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongQuoted 21 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:
Quoted 7 lines
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundaryr12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 28 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:Quoted 7 lines
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+
Quoted 6 lines
I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?
This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 38 lines · Yonghong Song
Quoted 28 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:Quoted 7 lines
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+Quoted 6 lines
I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?
Quoted 3 lines
This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.
There shouldn't be any extra copy.
For 7th and higher argument:
the caller does 'stx [r12 - N]' and the callee does 'ldx [r12 + M]' and JIT emits them as-is only adjusting N and M constants.
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 38 lines · Alexei Starovoitov
Quoted 37 lines · Yonghong Song
Quoted 34 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song
Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE(r12), introduced in the previous patch.The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... bar(a1, a2, a3, a4, a5, a6, a7, a8); ... }The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:Quoted 7 lines
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24) r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32) ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40) ---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ... r12-32: [outgoing arg 7 to callee]-+| r12-40: [outgoing arg 8 to callee]--+I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?
Quoted 7 lines
Quoted 3 lines
This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.
There shouldn't be any extra copy.
For 7th and higher argument:
the caller does 'stx [r12 - N]' and the callee does 'ldx [r12 + M]' and JIT emits them as-is only adjusting N and M constants.
Indeed, for 7th and higher arguments, ldx and stx and directly get or store values in expected calling-convention places.
Quoted 2 lines
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
For ldx case, this is a bpf callee to access a bpf caller. In such cases, r9 is not involved.
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 7 lines · Yonghong Song
Quoted 4 lines
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 8 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
Quoted 2 lines
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.
The following stack layout can avoid the above push/pop issue:
incoming stack arg N -> 1
return adderss
saved rbp
BPF program stack
tail call cnt <== if tail call reachable
callee-saved regs
r9 <== if priv_frame_ptr is not null
outgoing stack arg 1
outgoing stack arg M -> 2
After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 13 lines · Yonghong Song
Quoted 8 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
Quoted 2 lines
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.
Quoted 7 lines
The following stack layout can avoid the above push/pop issue:
incoming stack arg N -> 1 return adderss saved rbp BPF program stack tail call cnt <== if tail call reachable
let's disallow mixing 6+ args and tailcalls.
Quoted 2 lines
callee-saved regs
r9 <== if priv_frame_ptr is not nullI guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N], src_reg // store of outgoing 6th arg call
For priv stack JIT will emit push_r9 while JITing call insn. But it should push_r9 before stx converts to 'mov %r9 <- %src_reg'. Maybe we can reload r9 after the call instead of push/pop.
Quoted 6 lines
outgoing stack arg 1
outgoing stack arg M -> 2After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.
Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',
so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.
It feels to me that it's easier to disallow priv stack for now.
Yonghong Song
reviewerRe: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 11 lines · Alexei Starovoitov
Quoted 10 lines · Yonghong Song
Quoted 7 lines · Alexei Starovoitov
Quoted 3 lines · Yonghong Song
Quoted 1 line
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.
Quoted 8 lines
Quoted 7 lines
The following stack layout can avoid the above push/pop issue:
incoming stack arg N -> 1 return adderss saved rbp BPF program stack tail call cnt <== if tail call reachablelet's disallow mixing 6+ args and tailcalls.
Okay.
Quoted 11 lines
Quoted 3 lines
callee-saved regs r9 <== if priv_frame_ptr is not nullI guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N],src_reg// store of outgoing 6th arg callFor priv stack JIT will emit
push_r9while JITing call insn. But it shouldpush_r9before stx converts to 'mov %r9 <- %src_reg'.
I guess this is not enough. we can push_r9 in prologue. But for priv stack, r9 is still used in the normal bpf load/store codes. This will interfere with jitting 'mov %r9 <- %src_reg'.
As you mentioned later, disable priv_stack would work.
Quoted 13 lines
Maybe we can reload r9 after the call instead of push/pop.
Quoted 6 lines
outgoing stack arg 1
outgoing stack arg M -> 2After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.
Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',
Quoted 7 lines
so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.It feels to me that it's easier to disallow priv stack for now.
So if we disallow priv_stack, we can do
stx[r12 - first_arg_off] = val/reg => mov %r9, val/reg
other arg_off will be pushed to stack in reverse orders.
So for kfunc, 'mov %r9, val' already there, so we should be okay. For bpf-to-bpf, do 'push %r9'.
Is this correct?
Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions
Quoted 63 lines · Yonghong Song
Quoted 11 lines · Alexei Starovoitov
Quoted 10 lines · Yonghong Song
Quoted 7 lines · Alexei Starovoitov
Quoted 3 lines · Yonghong Song
Quoted 1 line
For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.
For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).
That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.Quoted 8 lines
Quoted 7 lines
The following stack layout can avoid the above push/pop issue:
incoming stack arg N -> 1 return adderss saved rbp BPF program stack tail call cnt <== if tail call reachablelet's disallow mixing 6+ args and tailcalls.
Okay.
Quoted 11 lines
Quoted 3 lines
callee-saved regs r9 <== if priv_frame_ptr is not nullI guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N],src_reg// store of outgoing 6th arg callFor priv stack JIT will emit
push_r9while JITing call insn. But it shouldpush_r9before stx converts to 'mov %r9 <- %src_reg'.I guess this is not enough. we can
push_r9in prologue. But for priv stack, r9 is still used in the normal bpf load/store codes. This will interfere with jitting 'mov %r9 <- %src_reg'.As you mentioned later, disable
priv_stackwould work.Quoted 19 lines
Maybe we can reload r9 after the call instead of push/pop.
Quoted 6 lines
outgoing stack arg 1
outgoing stack arg M -> 2After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.
Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.
Quoted 4 lines
Quoted 1 line
It feels to me that it's easier to disallow priv stack for now.
So if we disallow
priv_stack, we can dostx[r12 - first_arg_off] = val/reg => mov %r9, val/reg
yes.
Quoted 1 line
other
arg_offwill be pushed to stack in reverse orders.
no. Not pushed. they will be JITed in place stx[r12 -off] -> mov[%rbp], val Probably needs add %rsp somewhere to grow the stack.
Quoted 2 lines
So for kfunc, 'mov %r9, val' already there, so we should be okay. For bpf-to-bpf, do 'push %r9'.
Don't follow. why should they be different?
Yonghong Song
reviewerPATCH 4/10 bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongExtend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.
The following is an example to show how stack arguments are saved:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo)
============
r12-relative stack arg area:
r12-8: [incoming arg 6]
r12-16: [incoming arg 7]
---- incoming/outgoing boundary (kfunc1)
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
...
Back from kfunc1
...
---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
r12-48: [outgoing arg 9 to callee]Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().
In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. Also in fixup_kfunc_call(), repurpose insn->off (no longer needed after kfunc address resolution) to store the number of stack args, so the JIT knows how many args to marshal.
--- kernel/bpf/verifier.c | 97 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 17 deletions(-)
kernel/bpf/verifier.c +80 -17
@@ -3502,7 +3502,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)struct bpf_kfunc_meta kfunc;struct bpf_kfunc_desc *desc;unsigned long addr; -int err; +int i, err; prog_aux = env->prog->aux; tab = prog_aux->kfunc_tab;
@@ -3578,6 +3578,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)if (err)return err; +for (i =MAX_BPF_FUNC_REG_ARGS ; i < func_model.nr_args; i++) { +if (func_model.arg_size[i] >sizeof (u64)) { +verbose (env,"kfunc %s arg#%d size %d > %zu not supported for stack args\n" , + kfunc.name, i, func_model.arg_size[i],sizeof (u64)); +return -EINVAL ; + } + } + desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; desc->offset = offset;
@@ -12995,9 +13003,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,struct bpf_kfunc_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, - int argno, int nargs) + int argno, int nargs, u32 regno) { - u32 regno = argno + 1; struct bpf_reg_state *regs = cur_regs(env);struct bpf_reg_state *reg = ®s[regno];bool arg_mem_size = false;
@@ -13677,9 +13684,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (nargs >MAX_BPF_FUNC_ARGS ) {verbose (env,"Function %s has %d > %d args\n" , func_name, nargs, -MAX_BPF_FUNC_REG_ARGS ); +MAX_BPF_FUNC_ARGS );return -EINVAL ; }
@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 regno = i +1 , ref_id, type_size;bool is_ret_buf_sz = false; +bool is_stack_arg = false;int kf_arg_type; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/ +struct bpf_func_state *caller =cur_func (env); +int spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + + (i -MAX_BPF_FUNC_REG_ARGS ); + +if (!is_stack_arg_slot_initialized (caller, spi)) { +verbose (env,"stack arg#%d not properly initialized\n" , i); +return -EINVAL ; + } + + is_stack_arg = true; + regno =BPF_REG_1 ; + saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {
@@ -13702,7 +13737,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->arg_prog = true;cur_aux (env)->arg_prog = regno; -continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))
@@ -13725,9 +13760,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"R%d must be a known constant\n" , regno);return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + } meta->arg_constant.found = true; meta->arg_constant.value = reg->var_off.value; }else if (is_kfunc_arg_scalar_with_name (btf, &args[i],"rdonly_buf_size" )) {
@@ -13749,11 +13786,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; - ret =mark_chain_precision (env, regno); -if (ret) -return ret; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + ret =mark_chain_precision (env, regno); +if (ret) +return ret; + } } -continue ; + goto next_arg; }if (!btf_type_is_ptr (t)) {
@@ -13782,13 +13821,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id); ref_tname =btf_name_by_offset (btf, ref_t->name_off); - kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs); + kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs, + regno);if (kf_arg_type <0 )return kf_arg_type;switch (kf_arg_type) {case KF_ARG_PTR_TO_NULL : -continue ; + goto next_arg;case KF_ARG_PTR_TO_MAP :if (!reg->map_ptr) {verbose (env,"pointer in R%d isn't map pointer\n" , regno);
@@ -14201,6 +14241,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ; } } +next_arg: +if (is_stack_arg) + regs[BPF_REG_1 ] = saved_reg; }if (is_kfunc_release (meta) && !meta->release_regno) {
@@ -14778,7 +14821,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, nargs =btf_type_vlen (meta.func_proto); args = (const struct btf_param *)(meta.func_proto +1 ); -for (i =0 ; i < nargs; i++) { +for (i =0 ; i < nargs && i <MAX_BPF_FUNC_REG_ARGS ; i++) {u32 regno = i +1 ; t =btf_type_skip_modifiers (desc_btf, args[i].type,NULL );
@@ -14789,6 +14832,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,mark_btf_func_reg_size (env, regno, t->size); } +/* Track outgoing stack arg depth for kfuncs with >5 args */ +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +struct bpf_func_state *caller =cur_func (env); +struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; +u16 kfunc_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + +if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth; + } +if (is_iter_next_kfunc (&meta)) { err =process_iter_next_call (env, insn_idx, &meta);if (err)
@@ -23615,6 +23668,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) };
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls
The is_kfunc_arg_prog_aux path above was converted from continue to goto next_arg so that regs[BPF_REG_1] gets restored via the next_arg label. The scalar path and the KF_ARG_PTR_TO_NULL path were…
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 7 lines
kernel/bpf/verifier.c +2
-1@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;
[ ... ]
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/
[ ... ]
Quoted 8 lines
kernel/bpf/verifier.c +7
-0+ is_stack_arg = true; + regno =BPF_REG_1 ; + saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {
[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
Quoted 6 lines
kernel/bpf/verifier.c +1
-1-continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))continue ;
The is_kfunc_arg_prog_aux path above was converted from continue to goto next_arg so that regs[BPF_REG_1] gets restored via the next_arg label. The scalar path and the KF_ARG_PTR_TO_NULL path were also converted the same way.
Should this continue for is_kfunc_arg_ignore/is_kfunc_arg_implicit also be goto next_arg? When a stack arg (i >= MAX_BPF_FUNC_REG_ARGS) hits this continue, is_stack_arg is true but the next_arg restore is skipped, so regs[BPF_REG_1] keeps the spilled pointer from the current iteration. On the next iteration, saved_reg then captures that stale value instead of the original register state.
The comment above says "restore it after checking", which this continue path does not do. The impact is currently masked because each iteration independently overwrites regs[BPF_REG_1] and the caller_saved clearing in check_kfunc_call() resets R1 after the loop, but it breaks the save/restore invariant the rest of this patch establishes.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767
Yonghong Song
reviewerRe: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls
Quoted 48 lines · On 4/1/26 8:18 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 7 lines
kernel/bpf/verifier.c +2
-1@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;[ ... ]
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/ [ ... ]
Quoted 8 lines
kernel/bpf/verifier.c +7
-0+ is_stack_arg = true; + regno =BPF_REG_1 ; + saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
Quoted 6 lines
kernel/bpf/verifier.c +1
-1-continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))continue ;The
is_kfunc_arg_prog_auxpath above was converted from continue to gotonext_argso that regs[BPF_REG_1] gets restored via thenext_arglabel. The scalar path and theKF_ARG_PTR_TO_NULLpath were also converted the same way.
Quoted 2 lines
Should this continue for is_kfunc_arg_ignore/is_kfunc_arg_implicit also be goto
next_arg? When a stack arg (i >=MAX_BPF_FUNC_REG_ARGS)
You are right. My initial version with a early bpf-next is okay. But after rebase on top of master, I missed the above as it is newly introduced. Will fix in the next revision.
Quoted 17 lines
hits this continue,
is_stack_argis true but thenext_argrestore is skipped, so regs[BPF_REG_1] keeps the spilled pointer from the current iteration. On the next iteration,saved_regthen captures that stale value instead of the original register state.The comment above says "restore it after checking", which this continue path does not do. The impact is currently masked because each iteration independently overwrites regs[BPF_REG_1] and the
caller_savedclearing incheck_kfunc_call()resets R1 after the loop, but it breaks the save/restore invariant the rest of this patch establishes.---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767
Re: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongQuoted 96 lines · Yonghong Song
Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.
The following is an example to show how stack arguments are saved:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) { ... kfunc1(a1, a2, a3, a4, a5, a6, a7, a8); ... kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9); ... }The following is an illustration:
Caller (foo) ============ r12-relative stack arg area: r12-8: [incoming arg 6] r12-16: [incoming arg 7] ---- incoming/outgoing boundary (kfunc1) r12-24: [outgoing arg 6 to callee] r12-32: [outgoing arg 7 to callee] r12-40: [outgoing arg 8 to callee] ... Back from kfunc1 ... ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee] r12-32: [outgoing arg 7 to callee] r12-40: [outgoing arg 8 to callee] r12-48: [outgoing arg 9 to callee]Later JIT will marshal outgoing arguments to the native calling convention for
kfunc1()andkfunc2().In
check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. Also infixup_kfunc_call(), repurposeinsn->off(no longer needed after kfunc address resolution) to store the number of stack args, so the JIT knows how many args to marshal.Signed-off-by Yonghong Song <[email protected]>--- kernel/bpf/verifier.c | 97 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 17 deletions(-)kernel/bpf/verifier.c +10
-3@@ -3502,7 +3502,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)struct bpf_kfunc_meta kfunc;struct bpf_kfunc_desc *desc;unsigned long addr; -int err; +int i, err; prog_aux = env->prog->aux; tab = prog_aux->kfunc_tab;@@ -3578,6 +3578,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)if (err)return err; +for (i =MAX_BPF_FUNC_REG_ARGS ; i < func_model.nr_args; i++) { +if (func_model.arg_size[i] >sizeof (u64)) { +verbose (env,"kfunc %s arg#%d size %d > %zu not supported for stack args\n" , + kfunc.name, i, func_model.arg_size[i],sizeof (u64)); +return -EINVAL ; + } + } + desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; desc->offset = offset;@@ -12995,9 +13003,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,struct bpf_kfunc_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, - int argno, int nargs) + int argno, int nargs, u32 regno) { - u32 regno = argno + 1; struct bpf_reg_state *regs = cur_regs(env);struct bpf_reg_state *reg = ®s[regno];bool arg_mem_size = false;
[...]
Quoted 37 lines
kernel/bpf/verifier.c +16
-3@@ -13677,9 +13684,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (nargs >MAX_BPF_FUNC_ARGS ) {verbose (env,"Function %s has %d > %d args\n" , func_name, nargs, -MAX_BPF_FUNC_REG_ARGS ); +MAX_BPF_FUNC_ARGS );return -EINVAL ; }@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 regno = i +1 , ref_id, type_size;bool is_ret_buf_sz = false; +bool is_stack_arg = false;int kf_arg_type; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/ +struct bpf_func_state *caller =cur_func (env);
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13726
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+int spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + + (i -MAX_BPF_FUNC_REG_ARGS ); + +if (!is_stack_arg_slot_initialized (caller, spi)) { +verbose (env,"stack arg#%d not properly initialized\n" , i); +return -EINVAL ; + } + + is_stack_arg = true; + regno =BPF_REG_1 ;
is_kfunc_arg_mem_size() and is_kfunc_arg_const_mem_size() will not work properly in get_kfunc_ptr_arg_type(). Since they assume regno + 1 to be the size, R2 would be accidentally treated as size.
Maybe consider creating a helper to retrieve bpf_reg_state of a specific arg and use it in get_kfunc_ptr_arg_type() instead of copying and restoring?
Quoted 124 lines
kernel/bpf/verifier.c +44
-12+ saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {@@ -13702,7 +13737,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->arg_prog = true;cur_aux (env)->arg_prog = regno; -continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))@@ -13725,9 +13760,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"R%d must be a known constant\n" , regno);return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + } meta->arg_constant.found = true; meta->arg_constant.value = reg->var_off.value; }else if (is_kfunc_arg_scalar_with_name (btf, &args[i],"rdonly_buf_size" )) {@@ -13749,11 +13786,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; - ret =mark_chain_precision (env, regno); -if (ret) -return ret; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + ret =mark_chain_precision (env, regno); +if (ret) +return ret; + } } -continue ; + goto next_arg; }if (!btf_type_is_ptr (t)) {@@ -13782,13 +13821,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id); ref_tname =btf_name_by_offset (btf, ref_t->name_off); - kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs); + kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs, + regno);if (kf_arg_type <0 )return kf_arg_type;switch (kf_arg_type) {case KF_ARG_PTR_TO_NULL : -continue ; + goto next_arg;case KF_ARG_PTR_TO_MAP :if (!reg->map_ptr) {verbose (env,"pointer in R%d isn't map pointer\n" , regno);@@ -14201,6 +14241,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ; } } +next_arg: +if (is_stack_arg) + regs[BPF_REG_1 ] = saved_reg; }if (is_kfunc_release (meta) && !meta->release_regno) {@@ -14778,7 +14821,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, nargs =btf_type_vlen (meta.func_proto); args = (const struct btf_param *)(meta.func_proto +1 ); -for (i =0 ; i < nargs; i++) { +for (i =0 ; i < nargs && i <MAX_BPF_FUNC_REG_ARGS ; i++) {u32 regno = i +1 ; t =btf_type_skip_modifiers (desc_btf, args[i].type,NULL );@@ -14789,6 +14832,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,mark_btf_func_reg_size (env, regno, t->size); } +/* Track outgoing stack arg depth for kfuncs with >5 args */ +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +struct bpf_func_state *caller =cur_func (env); +struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; +u16 kfunc_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + +if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth; + } +if (is_iter_next_kfunc (&meta)) { err =process_iter_next_call (env, insn_idx, &meta);if (err)@@ -23615,6 +23668,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) }; --2.52.0
Yonghong Song
reviewerPATCH 5/10 bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong SongThe interpreter does not understand the bpf register r12 (BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.
--- kernel/bpf/core.c | 3 ++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
kernel/bpf/core.c +2 -1
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || + fp->aux->stack_arg_depth) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/verifier.c +6 -0
@@ -23542,6 +23542,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)verbose (env,"calling kernel functions are not allowed in non-JITed programs\n" );return -EINVAL ; } +for (i =0 ; i < env->subprog_cnt; i++) { +if (env->subprog_info[i].incoming_stack_arg_depth) { +verbose (env,"stack args are not supported in non-JITed programs\n" ); +return -EINVAL ; + } + }if (env->subprog_cnt >1 && env->prog->aux->tail_call_reachable) {/* When JIT fails the progs with bpf2bpf calls and tail_calls * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 6/10 bpf: Enable stack argument support for x86_64
In reply to Yonghong SongAdd stack argument support for x86_64.
--- arch/x86/net/bpf_jit_comp.c | 5 +++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 9 ++++++++- kernel/bpf/core.c | 5 +++++ 4 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
include/linux/filter.h +1 -0
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void ); +bool bpf_jit_supports_stack_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );
kernel/bpf/btf.c +8 -1
@@ -20,6 +20,7 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/bpf.h> +#include <linux/filter.h> #include <linux/bpf_lsm.h> #include <linux/skmsg.h> #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tname, nargs, MAX_BPF_FUNC_REG_ARGS);return -EINVAL ; } -if (nargs >MAX_BPF_FUNC_REG_ARGS ) +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d args\n" , + tname, nargs); +return -ENOTSUPP ; + } sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + }/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);
kernel/bpf/core.c +5 -0
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)return false; } +bool __weak bpf_jit_supports_stack_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 7/10 bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongAdd x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.
There are two possible approaches to allocate the stack arg area:
Option 1: Allocate a single combined region (incoming + max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp - prog_stack_depth - offset] where the 'offset' is the offset value in (incoming + max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.
Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.
This patch uses Option 1 for simplicity.
The native x86_64 stack layout for a function with incoming and outgoing stack args:
high address
┌─────────────────────────┐
│ incoming stack arg N │ [rbp + 16 + (N - 1) * 8] (pushed by caller)
│ ... │
│ incoming stack arg 1 │ [rbp + 16]
├─────────────────────────┤
│ return address │ [rbp + 8]
│ saved rbp │ [rbp]
├─────────────────────────┤
│ callee-saved regs │
│ BPF program stack │ (stack_depth bytes)
├─────────────────────────┤
│ incoming stack arg 1 │ [rbp - prog_stack_depth - 8]
│ ... (copied from │ (copied in prologue)
│ caller's push) │
│ incoming stack arg N │ [rbp - prog_stack_depth - N * 8]
├─────────────────────────┤
│ outgoing stack arg 1 │ (written via r12-relative STX/ST,
│ ... │ JIT translates to RBP-relative)
│ outgoing stack arg M │
└─────────────────────────┘
... Other stack usage
┌─────────────────────────┐
│ incoming stack arg M │ (copy from outgoing stack arg to
│ ... │ incoming stack arg)
│ incoming stack arg 1 │
├─────────────────────────┤
│ return address │
│ saved rbp │
├─────────────────────────┤
│ ... │
└─────────────────────────┘low address
In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.
For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.
For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the x86_64 calling convention, then clean up RSP after return.
--- arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++-- 1 file changed, 138 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +138 -7
@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static int push_stack_args (u8 **pprog,s32 base_off,int from,int to) +{ +u8 *prog = *pprog; +int j, off, cnt =0 ; + +for (j = from; j >= to; j--) { + off = base_off - j *8 ; + +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + cnt +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + cnt +=6 ; + } + } + *pprog = prog; +return cnt; +} +static void pop_r12 (u8 **pprog) {u8 *prog = *pprog;
@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; -u32 stack_depth; +u16 stack_arg_depth, incoming_stack_arg_depth; +u32 prog_stack_depth, stack_depth; +bool has_stack_args;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Save program stack depth before adding stack arg space. +* Each function allocates its own stack arg space +* (incoming + outgoing) below its BPF stack. +* Stack args are accessed via RBP-based addressing. +*/ + prog_stack_depth =round_up (stack_depth,8 ); +if (stack_arg_depth) + stack_depth += stack_arg_depth; + has_stack_args = stack_arg_depth >0 ; + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena); +detect_reg_usage (insn, insn_cnt, callee_regs_used); emit_prologue(&prog, image, stack_depth,
@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageemit_mov_imm64 (&prog,X86_REG_R12 , arena_vm_start >>32 , (u32 ) arena_vm_start); +if (incoming_stack_arg_depth &&bpf_is_subprog (bpf_prog)) { +int n = incoming_stack_arg_depth /8 ; + +/* +* Caller pushed stack args before CALL, so after prologue +* (CALL saves ret addr, then PUSH saves old RBP) they sit +* above RBP: +* +* [rbp + 16 + (n - 1) * 8] stack_arg n +* ... +* [rbp + 24] stack_arg 2 +* [rbp + 16] stack_arg 1 +* [rbp + 8] return address +* [rbp + 0] saved rbp +* +* Copy each into callee's own region below the program stack: +* [rbp - prog_stack_depth - i * 8] +*/ +for (i =0 ; i < n; i++) { +s32 src =16 + i *8 ; +s32 dst = -prog_stack_depth - (i +1 ) *8 ; + +/* mov rax, [rbp + src] */ +EMIT4 (0x48 ,0x8B ,0x45 , src); +/* mov [rbp + dst], rax */ +if (is_imm8 (dst)) +EMIT4 (0x48 ,0x89 ,0x45 , dst); +else +EMIT3_off32 (0x48 ,0x89 ,0x85 , dst); + } + } +if (priv_frame_ptr)emit_priv_frame_ptr (&prog, priv_frame_ptr);
@@ -1715,13 +1784,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +bool adjust_stack_arg_off = false;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;u8 b2 =0 , b3 =0 ;u8 *start_of_ldx;s64 jmp_offset; -s16 insn_off; +s32 insn_off;u8 jmp_cond;u8 *func;int nops;
@@ -1734,6 +1804,21 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image dst_reg =X86_REG_R9 ; } +if (has_stack_args) { +u8 class =BPF_CLASS (insn->code); + +if (class ==BPF_LDX && + src_reg ==BPF_REG_STACK_ARG_BASE ) { + src_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } +if ((class ==BPF_STX || class ==BPF_ST ) && + dst_reg ==BPF_REG_STACK_ARG_BASE ) { + dst_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } + } +switch (insn->code) {/* ALU */ case BPF_ALU |BPF_ADD |BPF_X :
@@ -2131,10 +2216,16 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ST |BPF_MEM |BPF_DW :EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: { +int off = insn->off; + +if (adjust_stack_arg_off) + off -= prog_stack_depth; +if (is_imm8 (off)) +EMIT2 (add_1reg (0x40 , dst_reg), off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), off); + }EMIT (imm32,bpf_size_to_x86_bytes (BPF_SIZE (insn->code)));break ;
@@ -2143,9 +2234,14 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_B :case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W : -case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off); +case BPF_STX |BPF_MEM |BPF_DW : { +int off = insn->off; + +if (adjust_stack_arg_off) + off -= prog_stack_depth; +emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, off);break ; + }case BPF_ST |BPF_PROBE_MEM32 |BPF_B :case BPF_ST |BPF_PROBE_MEM32 |BPF_H :
@@ -2243,6 +2339,8 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off -= prog_stack_depth;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ||BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {
@@ -2440,6 +2538,8 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +int off, base_off, n_stack_args, kfunc_stack_args =0 , stack_args =0 ; +u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32;
@@ -2449,6 +2549,29 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { + n_stack_args = outgoing_stack_args /8 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, n_stack_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { + kfunc_stack_args = insn->off; + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, kfunc_stack_args,2 ); + +/* mov r9, [rbp + base_off - 8] */ + off = base_off -8 ; +if (is_imm8 (off)) { +EMIT4 (0x4C ,0x8B ,0x4D , off); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , off); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;
@@ -2458,6 +2581,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { +/* add rsp, outgoing_stack_args */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_args); + }break ; }
Signature · 1 line
2.52.0
Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments
Quoted 1 line · Yonghong Song
[...]
Quoted 23 lines
case BPF_JMP | BPF_CALL: {arch/x86/net/bpf_jit_comp.c +15
-0+int off, base_off, n_stack_args, kfunc_stack_args =0 , stack_args =0 ; +u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32;@@ -2449,6 +2549,29 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { + n_stack_args = outgoing_stack_args /8 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, n_stack_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { + kfunc_stack_args = insn->off; + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, kfunc_stack_args,2 ); +
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2574
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+/* mov r9, [rbp + base_off - 8] */ + off = base_off -8 ; +if (is_imm8 (off)) { +EMIT4 (0x4C ,0x8B ,0x4D , off); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , off); + ip +=7 ; + } + }
Do we need to make sure RSP is 16-byte aligned before passing arg through stack per x86-64 ABI?
Quoted 22 lines
arch/x86/net/bpf_jit_comp.c +8
-1if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;@@ -2458,6 +2581,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { +/* add rsp, outgoing_stack_args */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_args); + }break ; } --2.52.0
Yonghong Song
reviewerRe: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments
Quoted 30 lines · Amery Hung
On Wed, Apr 1, 2026 at 6:28 PM Yonghong Song <[email protected]> wrote:
[...]Quoted 27 lines
case BPF_JMP | BPF_CALL: {arch/x86/net/bpf_jit_comp.c +19
-0+int off, base_off, n_stack_args, kfunc_stack_args =0 , stack_args =0 ; +u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32;@@ -2449,6 +2549,29 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { + n_stack_args = outgoing_stack_args /8 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, n_stack_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { + kfunc_stack_args = insn->off; + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + base_off = -(prog_stack_depth + incoming_stack_arg_depth); + ip +=push_stack_args (&prog, base_off, kfunc_stack_args,2 ); + +/* mov r9, [rbp + base_off - 8] */ + off = base_off -8 ; +if (is_imm8 (off)) { +EMIT4 (0x4C ,0x8B ,0x4D , off);
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2574
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+ ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , off); + ip +=7 ; + } + }Do we need to make sure RSP is 16-byte aligned before passing arg through stack per x86-64 ABI?
Good question. Without this patch set, looks like bpf jit here does not enforce 16 byte aligned. IIUC, 16 byte aligned requirement will be necessary for some special SSE/AVX/128-bit XMM etc. The bpf jit does not really emit them, so I guess it is okay?
Quoted 23 lines
Quoted 23 lines
arch/x86/net/bpf_jit_comp.c +8
-1if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;@@ -2458,6 +2581,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_args >0 ) { +/* add rsp, outgoing_stack_args */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_args); + }break ; } --2.52.0
Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongQuoted 172 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.There are two possible approaches to allocate the stack arg area:
Option 1: Allocate a single combined region (incoming +
max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp -prog_stack_depth- offset] where the 'offset' is the offset value in (incoming +max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.This patch uses Option 1 for simplicity.
The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N - 1) * 8] (pushed by caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ callee-saved regs │ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ incoming stack arg 1 │ [rbp - prog_stack_depth - 8] │ ... (copied from │ (copied in prologue) │ caller's push) │ │ incoming stack arg N │ [rbp - prog_stack_depth - N * 8] ├─────────────────────────┤ │ outgoing stack arg 1 │ (written via r12-relative STX/ST, │ ... │ JIT translates to RBP-relative) │ outgoing stack arg M │ └─────────────────────────┘ ... Other stack usage ┌─────────────────────────┐ │ incoming stack arg M │ (copy from outgoing stack arg to │ ... │ incoming stack arg) │ incoming stack arg 1 │ ├─────────────────────────┤ │ return address │ │ saved rbp │ ├─────────────────────────┤ │ ... │ └─────────────────────────┘low address
In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.
For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.
For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the
x86_64calling convention, then clean up RSP after return.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++-- 1 file changed, 138 insertions(+), 7 deletions(-)arch/x86/net/bpf_jit_comp.c +58
-1@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static int push_stack_args (u8 **pprog,s32 base_off,int from,int to) +{ +u8 *prog = *pprog; +int j, off, cnt =0 ; + +for (j = from; j >= to; j--) { + off = base_off - j *8 ; + +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + cnt +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + cnt +=6 ; + } + } + *pprog = prog; +return cnt; +} +static void pop_r12 (u8 **pprog) {u8 *prog = *pprog;@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; -u32 stack_depth; +u16 stack_arg_depth, incoming_stack_arg_depth; +u32 prog_stack_depth, stack_depth; +bool has_stack_args;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Save program stack depth before adding stack arg space. +* Each function allocates its own stack arg space +* (incoming + outgoing) below its BPF stack. +* Stack args are accessed via RBP-based addressing. +*/ + prog_stack_depth =round_up (stack_depth,8 ); +if (stack_arg_depth) + stack_depth += stack_arg_depth; + has_stack_args = stack_arg_depth >0 ; + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena); +detect_reg_usage (insn, insn_cnt, callee_regs_used); emit_prologue(&prog, image, stack_depth,@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageemit_mov_imm64 (&prog,X86_REG_R12 , arena_vm_start >>32 , (u32 ) arena_vm_start); +if (incoming_stack_arg_depth &&bpf_is_subprog (bpf_prog)) { +int n = incoming_stack_arg_depth /8 ; + +/* +* Caller pushed stack args before CALL, so after prologue +* (CALL saves ret addr, then PUSH saves old RBP) they sit +* above RBP: +* +* [rbp + 16 + (n - 1) * 8] stack_arg n +* ... +* [rbp + 24] stack_arg 2 +* [rbp + 16] stack_arg 1 +* [rbp + 8] return address +* [rbp + 0] saved rbp +* +* Copy each into callee's own region below the program stack: +* [rbp - prog_stack_depth - i * 8] +*/ +for (i =0 ; i < n; i++) { +s32 src =16 + i *8 ;
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1773
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+s32 dst = -prog_stack_depth - (i +1 ) *8 ; + +/* mov rax, [rbp + src] */ +EMIT4 (0x48 ,0x8B ,0x45 , src); +/* mov [rbp + dst], rax */ +if (is_imm8 (dst)) +EMIT4 (0x48 ,0x89 ,0x45 , dst); +else +EMIT3_off32 (0x48 ,0x89 ,0x85 , dst); + }
This is really suboptimal.
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
pw-bot: cr
Yonghong Song
reviewerRe: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments
Quoted 188 lines · Alexei Starovoitov
Quoted 171 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.There are two possible approaches to allocate the stack arg area:
Option 1: Allocate a single combined region (incoming +
max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp -prog_stack_depth- offset] where the 'offset' is the offset value in (incoming +max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.This patch uses Option 1 for simplicity.
The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N - 1) * 8] (pushed by caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ callee-saved regs │ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ incoming stack arg 1 │ [rbp - prog_stack_depth - 8] │ ... (copied from │ (copied in prologue) │ caller's push) │ │ incoming stack arg N │ [rbp - prog_stack_depth - N * 8] ├─────────────────────────┤ │ outgoing stack arg 1 │ (written via r12-relative STX/ST, │ ... │ JIT translates to RBP-relative) │ outgoing stack arg M │ └─────────────────────────┘ ... Other stack usage ┌─────────────────────────┐ │ incoming stack arg M │ (copy from outgoing stack arg to │ ... │ incoming stack arg) │ incoming stack arg 1 │ ├─────────────────────────┤ │ return address │ │ saved rbp │ ├─────────────────────────┤ │ ... │ └─────────────────────────┘low address
In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.
For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.
For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the
x86_64calling convention, then clean up RSP after return.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++-- 1 file changed, 138 insertions(+), 7 deletions(-)arch/x86/net/bpf_jit_comp.c +58
-1@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static int push_stack_args (u8 **pprog,s32 base_off,int from,int to) +{ +u8 *prog = *pprog; +int j, off, cnt =0 ; + +for (j = from; j >= to; j--) { + off = base_off - j *8 ; + +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + cnt +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + cnt +=6 ; + } + } + *pprog = prog; +return cnt; +} +static void pop_r12 (u8 **pprog) {u8 *prog = *pprog;@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; -u32 stack_depth; +u16 stack_arg_depth, incoming_stack_arg_depth; +u32 prog_stack_depth, stack_depth; +bool has_stack_args;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Save program stack depth before adding stack arg space. +* Each function allocates its own stack arg space +* (incoming + outgoing) below its BPF stack. +* Stack args are accessed via RBP-based addressing. +*/ + prog_stack_depth =round_up (stack_depth,8 ); +if (stack_arg_depth) + stack_depth += stack_arg_depth; + has_stack_args = stack_arg_depth >0 ; + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena); +detect_reg_usage (insn, insn_cnt, callee_regs_used); emit_prologue(&prog, image, stack_depth,@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageemit_mov_imm64 (&prog,X86_REG_R12 , arena_vm_start >>32 , (u32 ) arena_vm_start); +if (incoming_stack_arg_depth &&bpf_is_subprog (bpf_prog)) { +int n = incoming_stack_arg_depth /8 ; + +/* +* Caller pushed stack args before CALL, so after prologue +* (CALL saves ret addr, then PUSH saves old RBP) they sit +* above RBP: +* +* [rbp + 16 + (n - 1) * 8] stack_arg n +* ... +* [rbp + 24] stack_arg 2 +* [rbp + 16] stack_arg 1 +* [rbp + 8] return address +* [rbp + 0] saved rbp +* +* Copy each into callee's own region below the program stack: +* [rbp - prog_stack_depth - i * 8] +*/ +for (i =0 ; i < n; i++) { +s32 src =16 + i *8 ;arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1773
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+s32 dst = -prog_stack_depth - (i +1 ) *8 ; + +/* mov rax, [rbp + src] */ +EMIT4 (0x48 ,0x8B ,0x45 , src); +/* mov [rbp + dst], rax */ +if (is_imm8 (dst)) +EMIT4 (0x48 ,0x89 ,0x45 , dst); +else +EMIT3_off32 (0x48 ,0x89 ,0x85 , dst); + }This is really suboptimal.
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N],src_reg
Quoted 8 lines
into 'mov r9,
src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'movdst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
Good point. I will try to simplify the JIT by following x86_64 calling convention.
Quoted 2 lines
pw-bot: cr
Yonghong Song
reviewerPATCH 8/10 selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongAdd selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.
--- .../selftests/bpf/prog_tests/stack_arg.c | 143 ++++++++++++++++++ tools/testing/selftests/bpf/progs/stack_arg.c | 111 ++++++++++++++ .../selftests/bpf/progs/stack_arg_kfunc.c | 59 ++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 22 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 7 + 5 files changed, 342 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +143 -0
@@ -0,0 +1,143 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void test_nesting (void ) +{ +struct stack_arg *skel; +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + + err =stack_arg__load (skel); +if (!ASSERT_OK (err,"load" )) + goto out; + + skel->bss->a =0 ; + skel->bss->b =0 ; + skel->bss->c =0 ; + skel->bss->d =0 ; + skel->bss->e =0 ; + skel->bss->f =6 ; + skel->bss->g =7 ; + skel->bss->i =8 ; + + prog_fd =bpf_program__fd (skel->progs.test); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,29 ,"retval" ); + +out: +stack_arg__destroy (skel); +} + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +} + +static void test_global_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_global_many_args,36 ); + +out: +stack_arg__destroy (skel); +} + +static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +} + +static void test_kfunc (void ) +{ +struct stack_arg_kfunc *skel; + + skel =stack_arg_kfunc__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg_kfunc__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_stack_arg_scalar,36 ); +run_subtest (skel->progs.test_stack_arg_ptr,45 ); +run_subtest (skel->progs.test_stack_arg_mix,51 ); + +out: +stack_arg_kfunc__destroy (skel); +} + +void test_stack_arg (void ) +{ +if (test__start_subtest ("nesting" )) +test_nesting (); +if (test__start_subtest ("global_many_args" )) +test_global_many (); +if (test__start_subtest ("async_cb_many_args" )) +test_async_cb_many (); +if (test__start_subtest ("kfunc" )) +test_kfunc (); +}
tools/testing/selftests/bpf/progs/stack_arg.c new file +111 -0
111 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +59 -0
59 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +22 -0
@@ -824,6 +824,25 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)return 0 ; } +__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d, +u64 e,u64 f,u64 g,u64 h) +{ +return a + b + c + d + e + f + g + h; +} + +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_ptr(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_pass1 *p) +{ +return a + b + c + d + e + p->x0 + p->x1; +} + +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_pass1 *p,u64 f, +struct prog_test_pass1 *q) +{ +return a + b + c + d + e + p->x0 + f + q->x1; +} +static struct prog_test_ref_kfunc prog_test_struct = { .a =42 , .b =108 ,
@@ -1287,6 +1306,9 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)BTF_ID_FLAGS (func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +7 -0
@@ -111,6 +111,13 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, + __u64 e, __u64 f, __u64 g, __u64 h) __ksym; +__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p) __ksym; +__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p, __u64 f, + struct prog_test_pass1 *q) __ksym; void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 9/10 selftests/bpf: Add negative test for oversized kfunc stack argument
In reply to Yonghong SongAdd a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).
--- .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 7 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 7 ++++ 4 files changed, 70 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "stack_arg_fail.skel.h" + +void test_stack_arg_fail (void ) +{ +struct stack_arg_fail *skel; + + skel =stack_arg_fail__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" ); + +out: +stack_arg_fail__destroy (skel); +}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" + +#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +const volatile bool has_stack_arg = true; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +struct prog_test_big_arg s = { .a =1 , .b =2 }; + +return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); +} + +#else + +const volatile bool has_stack_arg = false; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +return 0 ; +} + +#endif + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
@@ -843,6 +843,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e,return a + b + c + d + e + p->x0 + f + q->x1; } +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_big_arg s) +{ +return a + b + c + d + e + s.a + s.b; +} +static struct prog_test_ref_kfunc prog_test_struct = { .a =42 , .b =108 ,
@@ -1309,6 +1315,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test5) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)BTF_ID_FLAGS (func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +7 -0
@@ -48,6 +48,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +struct prog_test_fail1 {void *p;int x;
@@ -118,6 +123,8 @@ __u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, struct prog_test_pass1 *p, __u64 f, struct prog_test_pass1 *q) __ksym; +__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_big_arg s) __ksym; void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH 10/10 selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongAdd inline-asm-based verifier tests that exercise the stack argument validation logic directly.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_stack_arg.c | 122 ++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -90,6 +90,7 @@#include "verifier_sockmap_mutate.skel.h" #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" +#include "verifier_stack_arg.skel.h" #include "verifier_stack_ptr.skel.h" #include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h"
@@ -235,6 +236,7 @@ void test_verifier_sock_addr(void) { RUN(verifier_sock_addr); }void test_verifier_sockmap_mutate (void ) {RUN (verifier_sockmap_mutate); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }void test_verifier_spin_lock (void ) {RUN (verifier_spin_lock); } +void test_verifier_stack_arg (void ) {RUN (verifier_stack_arg); }void test_verifier_stack_ptr (void ) {RUN (verifier_stack_ptr); }void test_verifier_store_release (void ) {RUN (verifier_store_release); }void test_verifier_subprog_precision (void ) {RUN (verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +122 -0
122 lines not on this page · raw message
Signature · 1 line
2.52.0