bpf: Support stack arguments for BPF functions and kfuncs
Yonghong Song
reviewerv3 cover bpf: Support stack arguments for BPF functions and kfuncs
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.
The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL. The incoming parameters can directly get the value from pushed native stack from caller. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).
Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.
For the rest of patches, patches 1-6 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 7 enables x86_64 for stack arguments. Patch 8 implemented JIT for x86_64. Patches 9-11 are some selftests.
[1] https://github.com/llvm/llvm-project/pull/189060
Changelogs:
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Fix selftest stack_arg_gap_at_minus8().
- Fix a few 'UTF-8' issues.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.
Yonghong Song (11):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Refactor process_iter_arg() to have proper argument index
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validation
arch/x86/net/bpf_jit_comp.c | 140 +++++- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 31 +- include/linux/filter.h | 4 +- kernel/bpf/btf.c | 21 +- kernel/bpf/core.c | 12 +- kernel/bpf/verifier.c | 474 ++++++++++++++++-- .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++ .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 + .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 ++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++ .../selftests/bpf/progs/verifier_stack_arg.c | 302 +++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 72 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 26 + 16 files changed, 1594 insertions(+), 60 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 1/11 bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf register R12 added by [1]. R12 is used as the base for stack arguments so it won't mess out R10 based stacks.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/filter.h | 3 ++- kernel/bpf/core.c | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/filter.h +2 -1
@@ -59,7 +59,8 @@ struct ctl_table_header;/* Kernel hidden auxiliary/helper register. */ #define BPF_REG_AX MAX_BPF_REG -#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) +#define BPF_REG_STACK_ARG_BASE (MAX_BPF_REG + 1) +#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG/* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,u32 imm_rnd =get_random_u32 ();s16 off; -BUILD_BUG_ON (BPF_REG_AX +1 !=MAX_BPF_JIT_REG ); -BUILD_BUG_ON (MAX_BPF_REG +1 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_AX +2 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_STACK_ARG_BASE +1 !=MAX_BPF_JIT_REG );/* Constraints on AX register: *
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 2/11 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongCurrently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.
--- include/linux/bpf.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 3/11 bpf: Support stack arguments for bpf functions
In reply to Yonghong SongCurrently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.
The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar(a1, a2, a3, a4, a5, a6, a7, a8);
...
}The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe() to do pruning check.
If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar1(a1, a2, a3, a4, a5, a6, a7, a8);
...
bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
...
Back from bar1
... Callee (bar2)
=== =============
+---> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
|+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
|||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+||| ...
r12-32: [outgoing arg 7 to callee]-+||
r12-40: [outgoing arg 8 to callee]--+|r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 31 +++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 311 +++++++++++++++++++++++++++++++++-- 4 files changed, 334 insertions(+), 24 deletions(-)
include/linux/bpf.h +2 -0
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +29 -2
@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; }; +struct bpf_stack_arg_state { +struct bpf_reg_state spilled_ptr;/* for spilled scalar/pointer semantics */ +u8 slot_type[BPF_REG_SIZE ]; +}; +/* state of the program: * type of all registers and stack info */
@@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };#define MAX_CALL_FRAMES 8
@@ -456,6 +465,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_slots[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \ + ? &frame->stack_arg_slots[slot].spilled_ptr : NULL) + +/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */ +#define bpf_for_each_spilled_stack_arg (iter, frame, reg, mask) \ + for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \ + iter < frame->stack_arg_depth / BPF_REG_SIZE; \ + iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask)) +#define bpf_for_each_reg_in_vstate_mask (__vst, __state, __reg, __mask, __expr) \ ({ \ struct bpf_verifier_state *___vstate = __vst; \
@@ -473,6 +493,11 @@ struct bpf_verifier_state {continue ; \ (void )(__expr); \ } \ + bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \ +if (!__reg) \ +continue ; \ + (void )(__expr); \ + } \ } \ })
@@ -686,10 +711,12 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; };struct bpf_verifier_env ;
kernel/bpf/btf.c +10 -4
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))
kernel/bpf/verifier.c +293 -18
424 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 03/11] bpf: Support stack arguments for bpf functions
Should this also reject non-negative offsets? The commit message and check_stack_arg_write() comment both state that off is a negative offset from the stack arg frame pointer, and the diagram shows…
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+/* Validate that a stack arg access is 8-byte sized and aligned. */ +static int check_stack_arg_access (struct bpf_verifier_env *env, +struct bpf_insn *insn,const char *op) +{ +int size =bpf_size_to_bytes (BPF_SIZE (insn->code)); + +if (size !=BPF_REG_SIZE ) { + verbose(env, "stack arg %s must be %d bytes, got %d\n",
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 5987
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ op, BPF_REG_SIZE, size); +return -EINVAL ; + } +if (insn->off %BPF_REG_SIZE ) { +verbose (env,"stack arg %s offset %d not aligned to %d\n" , + op, insn->off,BPF_REG_SIZE ); +return -EINVAL ; + } +return 0 ; +}
Should this also reject non-negative offsets? The commit message and check_stack_arg_write() comment both state that off is a negative offset from the stack arg frame pointer, and the diagram shows only negative offsets (r12-8, r12-16, etc.).
If a crafted BPF program uses a positive offset (e.g., off=+8), this function accepts it because +8 is 8-byte aligned. Then in check_stack_arg_write():
int spi = (-off - 1) / BPF_REG_SIZE;
With off=+8, spi becomes (-8 - 1) / 8 = -1. The grow_stack_arg_slots() call receives a negative size, which round_up() turns to 0, so no allocation happens. The subsequent access to stack_arg_slots[-1] is an out-of-bounds write (heap underflow if previously allocated, or a crash if stack_arg_slots is NULL).
The same issue applies to the read path in check_stack_arg_read() where -off becomes negative, passing the depth check, and spi again computes to -1.
resolve_pseudo_ldimm64() explicitly allows BPF_REG_STACK_ARG_BASE in both src_reg and dst_reg positions, so a BPF program can encode these instructions.
Something like the following may be sufficient:
if (insn->off >= 0) {
verbose(env, "stack arg %s offset must be negative\n", op);
return -EINVAL;
}Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ +/* +* Write a value to the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ;
[ ... ]
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 6040
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ }else { +/* BPF_ST: store immediate, treat as scalar */ + reg = &state->stack_arg_slots[spi].spilled_ptr; + reg->type =SCALAR_VALUE ; +__mark_reg_known (reg, (u32 )env->prog->insnsi[env->insn_idx].imm); +for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] =STACK_MISC ; + } +return 0 ; +}
Is the (u32) cast on the imm field intentional here? The regular stack path in check_stack_write_fixed_off() passes insn->imm directly to __mark_reg_known() without a cast:
__mark_reg_known(tmp_reg, insn->imm);
Since insn->imm is __s32 and __mark_reg_known() takes u64, the regular path sign-extends (e.g., imm=-1 becomes 0xffffffffffffffff). The (u32) cast here zero-extends instead (imm=-1 becomes 0x00000000ffffffff).
BPF_ST with BPF_DW stores the 32-bit immediate sign-extended to 64 bits at runtime, so the verifier's tracked value would not match the actual runtime value for negative immediates.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Yonghong Song
reviewerRe: [PATCH bpf-next v3 03/11] bpf: Support stack arguments for bpf functions
Quoted 20 lines · On 4/5/26 11:20 AM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 14 lines
kernel/bpf/verifier.c +14
-0+/* Validate that a stack arg access is 8-byte sized and aligned. */ +static int check_stack_arg_access (struct bpf_verifier_env *env, +struct bpf_insn *insn,const char *op) +{ +int size =bpf_size_to_bytes (BPF_SIZE (insn->code)); + +if (size !=BPF_REG_SIZE ) { +verbose (env,"stack arg %s must be %d bytes, got %d\n" , + op,BPF_REG_SIZE , size); +return -EINVAL ; + } +if (insn->off %BPF_REG_SIZE ) { +verbose (env,"stack arg %s offset %d not aligned to %d\n" , + op, insn->off,BPF_REG_SIZE );
Quoted 8 lines
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 5987
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+return -EINVAL ; + } +return 0 ; +}Should this also reject non-negative offsets? The commit message and
check_stack_arg_write()comment both state that off is a negative offset from the stack arg frame pointer, and the diagram shows only negative offsets (r12-8, r12-16, etc.).
We should. I will add more checking about positive or negative offset (as applied) in the next revision.
Quoted 40 lines
If a crafted BPF program uses a positive offset (e.g., off=+8), this function accepts it because +8 is 8-byte aligned. Then in
check_stack_arg_write():int spi = (-off - 1) / BPF_REG_SIZE;With off=+8, spi becomes (-8 - 1) / 8 = -1. The
grow_stack_arg_slots()call receives a negative size, whichround_up()turns to 0, so no allocation happens. The subsequent access to stack_arg_slots[-1] is an out-of-bounds write (heap underflow if previously allocated, or a crash ifstack_arg_slotsis NULL).The same issue applies to the read path in
check_stack_arg_read()where -off becomes negative, passing the depth check, and spi again computes to -1.
resolve_pseudo_ldimm64()explicitly allowsBPF_REG_STACK_ARG_BASEin bothsrc_reganddst_regpositions, so a BPF program can encode these instructions.Something like the following may be sufficient:
if (insn->off >= 0) { verbose(env, "stack arg %s offset must be negative\n", op); return -EINVAL; }Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ +/* +* Write a value to the stack arg area. +* off is the negative offset from the stack arg frame pointer. +* Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{ +int spi = (-off -1 ) /BPF_REG_SIZE ;[ ... ]
Quoted 12 lines
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 6040
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ }else { +/* BPF_ST: store immediate, treat as scalar */ + reg = &state->stack_arg_slots[spi].spilled_ptr; + reg->type =SCALAR_VALUE ; +__mark_reg_known (reg, (u32 )env->prog->insnsi[env->insn_idx].imm); +for (i =0 ; i <BPF_REG_SIZE ; i++) + state->stack_arg_slots[spi].slot_type[i] =STACK_MISC ; + } +return 0 ; +}Is the (u32) cast on the imm field intentional here? The regular stack path in
check_stack_write_fixed_off()passesinsn->imm
Yes, good point. Will remove (u32).
Quoted 19 lines
directly to
__mark_reg_known()without a cast:__mark_reg_known(tmp_reg, insn->imm);Since
insn->immis__s32and__mark_reg_known()takes u64, the regular path sign-extends (e.g., imm=-1 becomes0xffffffffffffffff). The (u32) cast here zero-extends instead (imm=-1 becomes0x00000000ffffffff).
BPF_STwithBPF_DWstores the 32-bit immediate sign-extended to 64 bits at runtime, so the verifier's tracked value would not match the actual runtime value for negative immediates.---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Yonghong Song
reviewerPATCH v3 4/11 bpf: Refactor process_iter_arg() to have proper argument index
In reply to Yonghong SongIn the next patch for kfunc stack arguments, a faked register is used to do proper verification checking. For process_iter_arg(), the regno is passed in and the iterator assumes 'regno - 1' as the argument index. This is wrong as regno is fake. So refactor process_iter_arg() by adding actual argument index which is used inside the function.
--- kernel/bpf/verifier.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +7 -7
@@ -9259,14 +9259,14 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, }static int process_iter_arg (struct bpf_verifier_env *env,int regno,int insn_idx, -struct bpf_kfunc_call_arg_meta *meta) +int argno,struct bpf_kfunc_call_arg_meta *meta) {struct bpf_reg_state *reg =reg_state (env, regno);const struct btf_type *t;int spi, err, i, nr_slots, btf_id;if (reg->type !=PTR_TO_STACK ) { -verbose (env,"arg#%d expected pointer to an iterator on stack\n" , regno -1 ); +verbose (env,"arg#%d expected pointer to an iterator on stack\n" , argno);return -EINVAL ; }
@@ -9276,9 +9276,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id* to any kfunc, if arg has "__iter" suffix, we need to be a bit more * conservative here. */ - btf_id =btf_check_iter_arg (meta->btf, meta->func_proto, regno -1 ); + btf_id =btf_check_iter_arg (meta->btf, meta->func_proto, argno);if (btf_id <0 ) { -verbose (env,"expected valid iter pointer as arg #%d\n" , regno -1 ); +verbose (env,"expected valid iter pointer as arg #%d\n" , argno);return -EINVAL ; } t =btf_type_by_id (meta->btf, btf_id);
@@ -9288,7 +9288,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id/* bpf_iter_<type>_new() expects pointer to uninit iter state */ if (!is_iter_reg_valid_uninit (env, reg, nr_slots)) {verbose (env,"expected uninitialized iter_%s as arg #%d\n" , -iter_type_str (meta->btf, btf_id), regno -1 ); +iter_type_str (meta->btf, btf_id), argno);return -EINVAL ; }
@@ -9312,7 +9312,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idbreak ;case -EINVAL :verbose (env,"expected an initialized iter_%s as arg #%d\n" , -iter_type_str (meta->btf, btf_id), regno -1 ); +iter_type_str (meta->btf, btf_id), argno);return err;case -EPROTO :verbose (env,"expected an RCU CS when using %s\n" , meta->func_name);
@@ -14063,7 +14063,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EINVAL ; } } - ret =process_iter_arg (env, regno, insn_idx, meta); + ret =process_iter_arg (env, regno, insn_idx, i, meta);if (ret <0 )return ret;break ;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 5/11 bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongExtend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.
The following is an example to show how stack arguments are saved:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo)
============
r12-relative stack arg area:
r12-8: [incoming arg 6]
r12-16: [incoming arg 7]
---- incoming/outgoing boundary (kfunc1)
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
...
Back from kfunc1
...
---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
r12-48: [outgoing arg 9 to callee]Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().
In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. The following is one of examples based on a later selftest:
13: (85) call bpf_kfunc_call_stack_arg_mem#152105 Use reg 1 for stack arg#5
Use reg 2 to represent mem_size
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2 End of using reg 1 for stack arg#6
14: R0=scalar()
14: (95) exit
The above example is for KF_ARG_PTR_TO_MEM_SIZE case. Registers 1 and 2 are used as an temporary register for argument checking. The verifier log will identify when the temporary register is used and when the temporary register is not used any more.
There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno' is also keep regno for later fixup. Since regno is a faked one such three cases are rejected for now if they are in stack arguments. If possible, new kfuncs could keep them in first 5 registers so there are no issues at all.
If faked register approach is not favored, the following is an alternative approach with the following data structure:
struct reg_or_arg_t {
struct bpf_reg_state *state;
union {
int regno;
int argno;
};
bool is_reg; /* distinguish between reg and arg */
};Such a struct can replace existing regno/argno/reg_state parameters to make it easy to distinguish reg vs. arg.
--- kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 19 deletions(-)
kernel/bpf/verifier.c +124 -19
@@ -3491,7 +3491,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)struct bpf_kfunc_meta kfunc;struct bpf_kfunc_desc *desc;unsigned long addr; -int err; +int i, err; prog_aux = env->prog->aux; tab = prog_aux->kfunc_tab;
@@ -3567,6 +3567,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)if (err)return err; +for (i =MAX_BPF_FUNC_REG_ARGS ; i < func_model.nr_args; i++) { +if (func_model.arg_size[i] >sizeof (u64)) { +verbose (env,"kfunc %s arg#%d size %d > %zu not supported for stack args\n" , + kfunc.name, i, func_model.arg_size[i],sizeof (u64)); +return -EINVAL ; + } + } + desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; desc->offset = offset;
@@ -13083,6 +13091,19 @@ static bool is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + (argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_slots[spi].spilled_ptr; +} + static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,
@@ -13101,8 +13122,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,return KF_ARG_PTR_TO_CTX ;if (argno +1 < nargs && - (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]) || -is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]))) + (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )) || +is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )))) arg_mem_size = true;/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -13770,9 +13791,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (nargs >MAX_BPF_FUNC_ARGS ) {verbose (env,"Function %s has %d > %d args\n" , func_name, nargs, -MAX_BPF_FUNC_REG_ARGS ); +MAX_BPF_FUNC_ARGS );return -EINVAL ; }
@@ -13780,26 +13801,59 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 regno = i +1 , ref_id, type_size;bool is_ret_buf_sz = false; +bool is_stack_arg = false;int kf_arg_type; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/ +struct bpf_func_state *caller =cur_func (env); +int spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + + (i -MAX_BPF_FUNC_REG_ARGS ); + +if (!is_stack_arg_slot_initialized (caller, spi)) { +verbose (env,"stack arg#%d not properly initialized\n" , i); +return -EINVAL ; + } + + is_stack_arg = true; + regno =BPF_REG_1 ; + saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; +verbose (env,"Use reg %d for stack arg#%d\n" , regno, i); + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {verifier_bug (env,"Only 1 prog->aux argument supported per-kfunc" );return -EFAULT ; } +if (is_stack_arg) { +verbose (env,"arg#%d prog->aux cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->arg_prog = true;cur_aux (env)->arg_prog = regno; -continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i)) -continue ; + goto next_arg; t =btf_type_skip_modifiers (btf, args[i].type,NULL );
@@ -13818,9 +13872,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"R%d must be a known constant\n" , regno);return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (!is_stack_arg) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + } meta->arg_constant.found = true; meta->arg_constant.value = reg->var_off.value; }else if (is_kfunc_arg_scalar_with_name (btf, &args[i],"rdonly_buf_size" )) {
@@ -13842,11 +13898,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; - ret =mark_chain_precision (env, regno); -if (ret) -return ret; +if (!is_stack_arg) { + ret =mark_chain_precision (env, regno); +if (ret) +return ret; + } } -continue ; + goto next_arg; }if (!btf_type_is_ptr (t)) {
@@ -13868,8 +13926,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EFAULT ; } meta->ref_obj_id = reg->ref_obj_id; -if (is_kfunc_release (meta)) +if (is_kfunc_release (meta)) { +if (is_stack_arg) { +verbose (env,"arg#%d release arg cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->release_regno = regno; + } } ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);
@@ -13881,7 +13944,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_switch (kf_arg_type) {case KF_ARG_PTR_TO_NULL : -continue ; + goto next_arg;case KF_ARG_PTR_TO_MAP :if (!reg->map_ptr) {verbose (env,"pointer in R%d isn't map pointer\n" , regno);
@@ -14020,6 +14083,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ dynptr_arg_type |=DYNPTR_TYPE_FILE ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) { dynptr_arg_type |=DYNPTR_TYPE_FILE ; +if (is_stack_arg) { +verbose (env,"arg#%d release arg cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->release_regno = regno; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] && (dynptr_arg_type &MEM_UNINIT )) {
@@ -14169,8 +14236,18 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ {struct bpf_reg_state *buff_reg = ®s[regno];const struct btf_param *buff_arg = &args[i]; -struct bpf_reg_state *size_reg = ®s[regno +1 ]; +struct bpf_reg_state *size_reg;const struct btf_param *size_arg = &args[i +1 ]; +struct bpf_reg_state saved_size_reg = {}; +bool size_is_stack_arg = false; + +if (i >=MAX_BPF_FUNC_REG_ARGS ) { + size_is_stack_arg = true; + saved_size_reg = regs[regno +1 ]; + regs[regno +1 ] = *get_kfunc_arg_reg (env, i +1 ); +verbose (env,"Use reg %d to represent mem_size\n" , regno +1 ); + } + size_reg = ®s[regno +1 ];if (!register_is_null (buff_reg) || !is_kfunc_arg_nullable (meta->btf, buff_arg)) { ret =check_kfunc_mem_size_reg (env, size_reg, regno +1 );
@@ -14193,6 +14270,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->arg_constant.value = size_reg->var_off.value; } +if (size_is_stack_arg) + regs[regno +1 ] = saved_size_reg; +/* Skip next '__sz' or '__szk' argument */ i++;break ;
@@ -14294,6 +14374,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ; } } +next_arg: +if (is_stack_arg) { + regs[regno] = saved_reg; +verbose (env,"End of using reg %d for stack arg#%d\n" , regno, i); + } }if (is_kfunc_release (meta) && !meta->release_regno) {
@@ -15059,7 +15144,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, nargs =btf_type_vlen (meta.func_proto); args = (const struct btf_param *)(meta.func_proto +1 ); -for (i =0 ; i < nargs; i++) { +for (i =0 ; i < nargs && i <MAX_BPF_FUNC_REG_ARGS ; i++) {u32 regno = i +1 ; t =btf_type_skip_modifiers (desc_btf, args[i].type,NULL );
@@ -15070,6 +15155,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,mark_btf_func_reg_size (env, regno, t->size); } +/* Track outgoing stack arg depth for kfuncs with >5 args */ +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +struct bpf_func_state *caller =cur_func (env); +struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; +u16 kfunc_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + +if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth; + } +if (is_iter_next_kfunc (&meta)) { err =process_iter_next_call (env, insn_idx, &meta);if (err)
@@ -23975,6 +24070,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) };
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls
When check_kfunc_args() calls this function for a stack arg (argno >= 5), the stack arg state was placed into regs[BPF_REG_1], but this local reg reads regs[argno + 1] instead, which is a…
Quoted 3 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 16 lines
kernel/bpf/verifier.c +13
-0+static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + (argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_slots[spi].spilled_ptr; +} + static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,
[ ... ]
kernel/bpf/verifier.c
Quoted 4 lines
{
kernel/bpf/verifier.c +0
-0u32 regno = argno +1 ;struct bpf_reg_state *regs =cur_regs (env);struct bpf_reg_state *reg = ®s[regno];
When check_kfunc_args() calls this function for a stack arg (argno >= 5), the stack arg state was placed into regs[BPF_REG_1], but this local reg reads regs[argno + 1] instead, which is a callee-saved register (BPF_REG_6 through BPF_REG_9) or the frame pointer (BPF_REG_10).
For argno >= 10 (MAX_BPF_FUNC_ARGS is 12), this reads past regs[MAX_BPF_REG - 1] into adjacent struct fields, since bpf_func_state.regs[] only has 11 entries (BPF_REG_0..BPF_REG_10).
This reg is used later for register_is_null() and base_type(reg->type), which would cause wrong argument type classification for pointer-typed stack args. The is_kfunc_arg_mem_size() lookahead a few lines below was correctly updated to use get_kfunc_arg_reg() -- should this local reg use the same helper?
Quoted 6 lines
if (argno + 1 < nargs &&kernel/bpf/verifier.c +2
-2- (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]))) + (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) || + is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)))) arg_mem_size = true;
[ ... ]
Quoted 7 lines
case
KF_ARG_PTR_TO_MEM_SIZE:kernel/bpf/verifier.c +1
-1{struct bpf_reg_state *buff_reg = ®s[regno];const struct btf_param *buff_arg = &args[i]; -struct bpf_reg_state *size_reg = ®s[regno +1 ]; +struct bpf_reg_state *size_reg;const struct btf_param *size_arg = &args[i +1 ];
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 14250
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+struct bpf_reg_state saved_size_reg = {}; +bool size_is_stack_arg = false; + +if (i >=MAX_BPF_FUNC_REG_ARGS ) { + size_is_stack_arg = true; + saved_size_reg = regs[regno +1 ]; + regs[regno +1 ] = *get_kfunc_arg_reg (env, i +1 ); +verbose (env,"Use reg %d to represent mem_size\n" , regno +1 ); + } + size_reg = ®s[regno +1 ];
Is there a missing case when the pointer arg is the last register arg (i == MAX_BPF_FUNC_REG_ARGS - 1, i.e. i == 4) and the size arg is the first stack arg (i + 1 == 5)?
In that case i < MAX_BPF_FUNC_REG_ARGS, so is_stack_arg is false and regno is 5 (BPF_REG_5). The condition "i >= MAX_BPF_FUNC_REG_ARGS" is false, so size_is_stack_arg stays false. Then size_reg becomes ®s[6], which is BPF_REG_6 (a callee-saved register) instead of the stack arg slot for arg 5.
The get_kfunc_ptr_arg_type() lookahead correctly handles this boundary with get_kfunc_arg_reg(env, argno + 1), so the arg gets classified as KF_ARG_PTR_TO_MEM_SIZE, but then the actual size check reads from the wrong register.
Would something like "if (i + 1 >= MAX_BPF_FUNC_REG_ARGS)" be the right condition here?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Yonghong Song
reviewerRe: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls
Quoted 36 lines · On 4/5/26 11:20 AM, [email protected] wrote:
Quoted 3 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 16 lines
kernel/bpf/verifier.c +13
-0+static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + (argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_slots[spi].spilled_ptr; +} + static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,[ ... ]
Quoted 4 lines
{ u32 regno = argno + 1; struct bpf_reg_state *regs = cur_regs(env); struct bpf_reg_state *reg = ®s[regno];When
check_kfunc_args()calls this function for a stack arg (argno >= 5), the stack arg state was placed into regs[BPF_REG_1], but this local reg reads regs[argno + 1] instead, which is a callee-saved register (BPF_REG_6throughBPF_REG_9) or the frame pointer (BPF_REG_10).For argno >= 10 (
MAX_BPF_FUNC_ARGSis 12), this reads past regs[MAX_BPF_REG - 1] into adjacent struct fields, since bpf_func_state.regs[] only has 11 entries (BPF_REG_0..BPF_REG_10).
Quoted 5 lines
This reg is used later for
register_is_null()andbase_type(reg->type), which would cause wrong argument type classification for pointer-typed stack args. Theis_kfunc_arg_mem_size()lookahead a few lines below was correctly updated to useget_kfunc_arg_reg()-- should this local reg use the same helper?
To use BPF_REG_1 to represent the stack argument is a hack. It may confuse people and it may have some subtle issues due to this.
In the next revision, I will discard this patch and implement it properly. In most cases, for related functions w.r.t. stack argument validation, I will ensure functions has parameters like (reg_state, regno, argno). If the argument is a stack argument, regno will be -1 (not a register). In cases where argno is not really available, argno will be -1. But in any case, at least one of them is positive. The verification log will print out message properly. If regno is non-negative, log 'regno' is needed. Otherwise, log 'argno'.
Quoted 21 lines
Quoted 7 lines
if (argno + 1 < nargs &&kernel/bpf/verifier.c +2
-2- (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]))) + (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) || + is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)))) arg_mem_size = true;[ ... ]
Quoted 12 lines
case
KF_ARG_PTR_TO_MEM_SIZE:
{
structbpf_reg_state*buff_reg= ®s[regno]; const structbtf_param*buff_arg= &args[i];kernel/bpf/verifier.c +6
-1-struct bpf_reg_state *size_reg = ®s[regno +1 ]; +struct bpf_reg_state *size_reg;const struct btf_param *size_arg = &args[i +1 ]; +struct bpf_reg_state saved_size_reg = {}; +bool size_is_stack_arg = false; + +if (i >=MAX_BPF_FUNC_REG_ARGS ) { + size_is_stack_arg = true;
Quoted 8 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 14250
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+ saved_size_reg = regs[regno +1 ]; + regs[regno +1 ] = *get_kfunc_arg_reg (env, i +1 ); +verbose (env,"Use reg %d to represent mem_size\n" , regno +1 ); + } + size_reg = ®s[regno +1 ];Is there a missing case when the pointer arg is the last register arg (i ==
MAX_BPF_FUNC_REG_ARGS- 1, i.e. i == 4) and the size arg is the first stack arg (i + 1 == 5)?
Yes. This is something I intend to support. arg4 (starting from arg0) is a memory pointer at reg5, and arg5 represents the memory size at the first stack argument.
Quoted 6 lines
In that case i <
MAX_BPF_FUNC_REG_ARGS, sois_stack_argis false and regno is 5 (BPF_REG_5). The condition "i >=MAX_BPF_FUNC_REG_ARGS" is false, sosize_is_stack_argstays false. Thensize_regbecomes ®s[6], which isBPF_REG_6(a callee-saved register) instead of the stack arg slot for arg 5.
In this case, BPF_REG_6 will be saved first, and then BPF_REG_6 will have the first stack argument for future verification.
But in any case, this patch set is a hack. I will have proper implementation in the next revision.
Quoted 15 lines
The
get_kfunc_ptr_arg_type()lookahead correctly handles this boundary withget_kfunc_arg_reg(env, argno + 1), so the arg gets classified asKF_ARG_PTR_TO_MEM_SIZE, but then the actual size check reads from the wrong register.Would something like "if (i + 1 >=
MAX_BPF_FUNC_REG_ARGS)" be the right condition here?---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Re: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls
Quoted 10 lines · Yonghong Song
In the next revision, I will discard this patch and implement it properly. In most cases, for related functions w.r.t. stack argument validation, I will ensure functions has parameters like (
reg_state, regno, argno). If the argument is a stack argument, regno will be -1 (not a register). In cases where argno is not really available, argno will be -1. But in any case, at least one of them is positive. The verification log will print out message properly. If regno is non-negative, log 'regno' is needed. Otherwise, log 'argno'.
that also sounds quite hacky.
Just pass two reg_states for reg and for next reg,
and argno of the first reg. That's it.
argno is always valid for printing and argno+1 is also correct number. Both reg and next_reg are also always valid pointers
that get_kfunc_arg_reg() returned.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls
Quoted 10 lines · Alexei Starovoitov
Quoted 9 lines · Yonghong Song
In the next revision, I will discard this patch and implement it properly. In most cases, for related functions w.r.t. stack argument validation, I will ensure functions has parameters like (
reg_state, regno, argno). If the argument is a stack argument, regno will be -1 (not a register). In cases where argno is not really available, argno will be -1. But in any case, at least one of them is positive. The verification log will print out message properly. If regno is non-negative, log 'regno' is needed. Otherwise, log 'argno'.
Quoted 6 lines
that also sounds quite hacky.
Just pass tworeg_statesfor reg and for next reg,
and argno of the first reg. That's it.
argno is always valid for printing and argno+1 is also correct number. Both reg andnext_regare also always valid pointers
thatget_kfunc_arg_reg()returned.
I will give a try. In most cases, only one reg_state is needed. But there are a few cases (e.g. KF_ARG_PTR_TO_MEM_SIZE), two reg_states are needed.
Yonghong Song
reviewerPATCH v3 6/11 bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong SongThe interpreter does not understand the bpf register r12 (BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.
--- kernel/bpf/core.c | 3 ++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
kernel/bpf/core.c +2 -1
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || + fp->aux->stack_arg_depth) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/verifier.c +6 -0
@@ -23944,6 +23944,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)verbose (env,"calling kernel functions are not allowed in non-JITed programs\n" );return -EINVAL ; } +for (i =0 ; i < env->subprog_cnt; i++) { +if (env->subprog_info[i].incoming_stack_arg_depth) { +verbose (env,"stack args are not supported in non-JITed programs\n" ); +return -EINVAL ; + } + }if (env->subprog_cnt >1 && env->prog->aux->tail_call_reachable) {/* When JIT fails the progs with bpf2bpf calls and tail_calls * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 7/11 bpf: Enable stack argument support for x86_64
In reply to Yonghong SongAdd stack argument support for x86_64.
--- arch/x86/net/bpf_jit_comp.c | 5 +++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 9 ++++++++- kernel/bpf/core.c | 5 +++++ 4 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
include/linux/filter.h +1 -0
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void ); +bool bpf_jit_supports_stack_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );
kernel/bpf/btf.c +8 -1
@@ -20,6 +20,7 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/bpf.h> +#include <linux/filter.h> #include <linux/bpf_lsm.h> #include <linux/skmsg.h> #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tname, nargs, MAX_BPF_FUNC_REG_ARGS);return -EINVAL ; } -if (nargs >MAX_BPF_FUNC_REG_ARGS ) +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d args\n" , + tname, nargs); +return -ENOTSUPP ; + } sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + }/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);
kernel/bpf/core.c +5 -0
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)return false; } +bool __weak bpf_jit_supports_stack_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 8/11 bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongAdd x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.
The JIT follows the native x86_64 calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly where x86_64 expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.
The native x86_64 stack layout for a function with incoming and outgoing stack args:
high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth] │ ... │ (written via r12-relative STX/ST) │ outgoing stack arg M │ [rbp - prog_stack_depth - 8] ├─────────────────────────┤ │ callee-saved regs ... │ (pushed after sub rsp) └─────────────────────────┘ rsp
low address
BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <= incoming_depth):
native_off = 8 - bpf_off → [rbp + 16 + ...]
- Outgoing args (store: -off > incoming_depth):
native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
Since callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the x86_64 calling convention.
--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +129 -6
@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) { + off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + } + *pprog = prog; +return bytes; +} +static void emit_nops (u8 **pprog,int len) {u8 *prog = *pprog;
@@ -1664,16 +1686,33 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; -u32 stack_depth; +u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth; +u32 prog_stack_depth, stack_depth; +bool has_stack_args;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Save program stack depth before adding outgoing stack arg space. +* Incoming stack args are read directly from [rbp + 16 + ...]. +* Only the outgoing stack arg area is allocated below the +* program stack. Outgoing args written here become the callee's +* incoming args. +*/ + prog_stack_depth =round_up (stack_depth,8 ); +if (outgoing_stack_arg_depth) + stack_depth += outgoing_stack_arg_depth; + has_stack_args = stack_arg_depth >0 ; + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena);
@@ -1715,13 +1754,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +bool adjust_stack_arg_off = false;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;u8 b2 =0 , b3 =0 ;u8 *start_of_ldx;s64 jmp_offset; -s16 insn_off; +s32 insn_off;u8 jmp_cond;u8 *func;int nops;
@@ -1734,6 +1774,37 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image dst_reg =X86_REG_R9 ; } +if (has_stack_args) { +u8 class =BPF_CLASS (insn->code); + +if (class ==BPF_LDX && + src_reg ==BPF_REG_STACK_ARG_BASE ) { + src_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } +if ((class ==BPF_STX || class ==BPF_ST ) && + dst_reg ==BPF_REG_STACK_ARG_BASE ) { + dst_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } + } + +/* +* Translate BPF r12-relative offset to native RBP-relative: +* +* Incoming args (load: offset >= -incoming_depth): +* BPF: r12 + bpf_off = r12 - k * 8 (k = 1,2,...) for incoming arg k +* Native: [rbp + 8 + k * 8] +* Formula: native_off = 8 + k * 8 = 8 - bpf_off +* +* Outgoing args (store: offset < -incoming_depth): +* BPF: r12 + bpf_off = r12 - (incoming + k * 8) for outgoing arg k +* Native: [rbp - prog_stack_depth - outgoing + (k - 1) * 8] +* Formula: native_off = -(prog_stack_depth + outgoing) + (k - 1) * 8 +* = -(prog_stack_depth + outgoing + incoming + 8) - bpf_off +* = -(prog_stack_depth + stack_arg_depth + 8) - bpf_off +*/ +switch (insn->code) {/* ALU */ case BPF_ALU |BPF_ADD |BPF_X :
@@ -2131,10 +2202,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ST |BPF_MEM |BPF_DW :EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off = -(prog_stack_depth + stack_arg_depth +8 ) - insn_off; +if (is_imm8 (insn_off)) +EMIT2 (add_1reg (0x40 , dst_reg), insn_off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), insn_off);EMIT (imm32,bpf_size_to_x86_bytes (BPF_SIZE (insn->code)));break ;
@@ -2144,7 +2218,10 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W :case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off); + insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off = -(prog_stack_depth + stack_arg_depth +8 ) - insn_off; +emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);break ;case BPF_ST |BPF_PROBE_MEM32 |BPF_B :
@@ -2243,6 +2320,8 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off =8 - insn_off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ||BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {
@@ -2441,6 +2520,7 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : {u8 *ip = image + addrs[i -1 ]; +int stack_args =0 ; func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* +* BPF-to-BPF calls: push outgoing stack args from +* the outgoing area onto the stack before CALL. +* The outgoing area is at [rbp - prog_stack - outgoing], +* but rsp is below that due to callee-saved reg pushes, +* so we must explicitly push args for the callee. +*/ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int n_args = outgoing_stack_arg_depth /8 ; + + ip +=push_stack_args (&prog, outgoing_base, n_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { +/* Kfunc calls: arg 6 → R9, args 7+ → push. */ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int kfunc_stack_args = insn->off; + + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + +/* Push args 7+ in reverse order */ +if (stack_args >0 ) + ip +=push_stack_args (&prog, outgoing_base, kfunc_stack_args,2 ); + +/* mov r9, [rbp + outgoing_base] (arg 6) */ +if (is_imm8 (outgoing_base)) { +EMIT4 (0x4C ,0x8B ,0x4D , outgoing_base); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , outgoing_base); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;
@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* add rsp, outgoing_stack_arg_depth */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_arg_depth); + }break ; }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
When priv_frame_ptr is active (private stack in use), R9 holds the private frame pointer throughout the program. The existing push_r9 / pop_r9 around calls preserves R9 across the function call.
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 25 lines
arch/x86/net/bpf_jit_comp.c +21
-0@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int n_args = outgoing_stack_arg_depth /8 ; + + ip +=push_stack_args (&prog, outgoing_base, n_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { +/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int kfunc_stack_args = insn->off; + + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + +/* Push args 7+ in reverse order */ +if (stack_args >0 ) + ip +=push_stack_args (&prog, outgoing_base, kfunc_stack_args,2 ); + +/* mov r9, [rbp + outgoing_base] (arg 6) */ +if (is_imm8 (outgoing_base)) {
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2562
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +7
-0+EMIT4 (0x4C ,0x8B ,0x4D , outgoing_base); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , outgoing_base); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;
When priv_frame_ptr is active (private stack in use), R9 holds the private frame pointer throughout the program. The existing push_r9 / pop_r9 around calls preserves R9 across the function call.
With this new code, two issues arise when stack args and priv_frame_ptr are both active.
For kfunc calls with stack args: the MOV R9, [rbp + outgoing_base] overwrites the priv_frame_ptr in R9 with arg6 before push_r9 saves it. After the call, pop_r9 restores arg6 into R9 instead of the priv_frame_ptr, so subsequent BPF_REG_FP accesses (which map to R9 when priv_frame_ptr is set) would reference the wrong memory.
For both kfunc and BPF-to-BPF calls: push_r9 inserts an extra 8 bytes onto the stack between the pushed args and the CALL instruction. The callee then sees the wrong values for its stack arguments. For example, with a BPF-to-BPF call the callee's incoming arg formula expects [rbp + 16] to be the first stack arg, but instead it finds the caller's saved R9 value:
push_stack_args: [RSP] = arg1, [RSP+8] = arg2
push_r9: [RSP] = R9, [RSP+8] = arg1, [RSP+16] = arg2
CALL: [RSP] = ret, [RSP+8] = R9 ...
callee prologue: [rbp+16] = R9 (expected arg1)Both priv_stack_mode and stack_arg_depth are set independently per subprog in jit_subprogs(), so can this combination actually be reached?
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog);
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2585
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-0+if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* add rsp, outgoing_stack_arg_depth */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_arg_depth); + }break ; }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Yonghong Song
reviewerRe: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
Quoted 61 lines · On 4/5/26 11:20 AM, [email protected] wrote:
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 25 lines
arch/x86/net/bpf_jit_comp.c +21
-0@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int n_args = outgoing_stack_arg_depth /8 ; + + ip +=push_stack_args (&prog, outgoing_base, n_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { +/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int kfunc_stack_args = insn->off; + + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + +/* Push args 7+ in reverse order */ +if (stack_args >0 ) + ip +=push_stack_args (&prog, outgoing_base, kfunc_stack_args,2 ); + +/* mov r9, [rbp + outgoing_base] (arg 6) */ +if (is_imm8 (outgoing_base)) {arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2562
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +7
-0+EMIT4 (0x4C ,0x8B ,0x4D , outgoing_base); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , outgoing_base); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;When
priv_frame_ptris active (private stack in use), R9 holds the private frame pointer throughout the program. The existingpush_r9/pop_r9around calls preserves R9 across the function call.With this new code, two issues arise when stack args and
priv_frame_ptrare both active.For kfunc calls with stack args: the MOV R9, [rbp +
outgoing_base] overwrites thepriv_frame_ptrin R9 with arg6 beforepush_r9saves it. After the call,pop_r9restores arg6 into R9 instead of thepriv_frame_ptr, so subsequentBPF_REG_FPaccesses (which map to R9 whenpriv_frame_ptris set) would reference the wrong memory.For both kfunc and BPF-to-BPF calls:
push_r9inserts an extra 8 bytes onto the stack between the pushed args and the CALL instruction. The callee then sees the wrong values for its stack arguments. For example, with a BPF-to-BPF call the callee's incoming arg formula expects [rbp + 16] to be the first stack arg, but instead it finds the caller's saved R9 value:
Quoted 8 lines
push_stack_args: [RSP] = arg1, [RSP+8] = arg2 push_r9: [RSP] = R9, [RSP+8] = arg1, [RSP+16] = arg2 CALL: [RSP] = ret, [RSP+8] = R9 ... callee prologue: [rbp+16] = R9 (expected arg1)Both
priv_stack_modeandstack_arg_depthare set independently per subprog injit_subprogs(), so can this combination actually be reached?
Yes, it could. So in next revision, we will disable private stack if outgoing stack arguments is not empty.
Quoted 21 lines
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog);arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2585
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-0+if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* add rsp, outgoing_stack_arg_depth */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_arg_depth); + }break ; }---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Yonghong Song
reviewerRe: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
In reply to [email protected]Quoted 61 lines · On 4/5/26 11:20 AM, [email protected] wrote:
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 25 lines
arch/x86/net/bpf_jit_comp.c +21
-0@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int n_args = outgoing_stack_arg_depth /8 ; + + ip +=push_stack_args (&prog, outgoing_base, n_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { +/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int kfunc_stack_args = insn->off; + + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + +/* Push args 7+ in reverse order */ +if (stack_args >0 ) + ip +=push_stack_args (&prog, outgoing_base, kfunc_stack_args,2 ); + +/* mov r9, [rbp + outgoing_base] (arg 6) */ +if (is_imm8 (outgoing_base)) {arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2562
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +7
-0+EMIT4 (0x4C ,0x8B ,0x4D , outgoing_base); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , outgoing_base); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;When
priv_frame_ptris active (private stack in use), R9 holds the private frame pointer throughout the program. The existingpush_r9/pop_r9around calls preserves R9 across the function call.With this new code, two issues arise when stack args and
priv_frame_ptrare both active.For kfunc calls with stack args: the MOV R9, [rbp +
outgoing_base] overwrites thepriv_frame_ptrin R9 with arg6 beforepush_r9saves it. After the call,pop_r9restores arg6 into R9 instead of thepriv_frame_ptr, so subsequentBPF_REG_FPaccesses (which map to R9 whenpriv_frame_ptris set) would reference the wrong memory.For both kfunc and BPF-to-BPF calls:
push_r9inserts an extra 8 bytes onto the stack between the pushed args and the CALL instruction. The callee then sees the wrong values for its stack arguments. For example, with a BPF-to-BPF call the callee's incoming arg formula expects [rbp + 16] to be the first stack arg, but instead it finds the caller's saved R9 value:
Quoted 8 lines
push_stack_args: [RSP] = arg1, [RSP+8] = arg2 push_r9: [RSP] = R9, [RSP+8] = arg1, [RSP+16] = arg2 CALL: [RSP] = ret, [RSP+8] = R9 ... callee prologue: [rbp+16] = R9 (expected arg1)Both
priv_stack_modeandstack_arg_depthare set independently per subprog injit_subprogs(), so can this combination actually be reached?
You are right. Similar to my previous comments, private stack will be disabled if outgoing stack arguments are not empty.
Quoted 21 lines
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog);arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2585
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-0+if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* add rsp, outgoing_stack_arg_depth */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_arg_depth); + }break ; }---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024
Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongQuoted 70 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.The JIT follows the native
x86_64calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly wherex86_64expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth] │ ... │ (written via r12-relative STX/ST) │ outgoing stack arg M │ [rbp - prog_stack_depth - 8] ├─────────────────────────┤ │ callee-saved regs ... │ (pushed after sub rsp) └─────────────────────────┘ rsplow address
BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <=incoming_depth):native_off = 8 - bpf_off → [rbp + 16 + ...]- Outgoing args (store: -off >
incoming_depth):
native_off= -(bpf_prog_stack +stack_arg_depth+ 8) -bpf_offSince callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the
x86_64calling convention.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +8
-0@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) {
arch/x86/net/bpf_jit_comp.c static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · line 410
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + }
This is not any better than v1.
It is still a copy.
As I said earlier:
https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
It has to be zero overhead. Copy pasting:
"
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
"
Remapping in earlier patches is unnecessary.
These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into store of src_reg into %rbp-M slot.
Only shift by 8 bytes is necessary for N to become M.
where STX of 6th argument becomes 'mov' from one register to x86's r9.
The feature has to be zero overhead to pass these args from bpf to native and from struct_ops hooks into bpf progs.
All verifier considerations are secondary.
pw-bot: cr
Yonghong Song
reviewerRe: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
Quoted 102 lines · Alexei Starovoitov
Quoted 69 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.The JIT follows the native
x86_64calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly wherex86_64expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth] │ ... │ (written via r12-relative STX/ST) │ outgoing stack arg M │ [rbp - prog_stack_depth - 8] ├─────────────────────────┤ │ callee-saved regs ... │ (pushed after sub rsp) └─────────────────────────┘ rsplow address
BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <=incoming_depth):native_off = 8 - bpf_off → [rbp + 16 + ...]- Outgoing args (store: -off >
incoming_depth):
native_off= -(bpf_prog_stack +stack_arg_depth+ 8) -bpf_offSince callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the
x86_64calling convention.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +8
-0@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) {arch/x86/net/bpf_jit_comp.c static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · line 410
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + }This is not any better than v1.
It is still a copy.
As I said earlier:
https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
It has to be zero overhead. Copy pasting:"
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N],src_reginto 'mov r9,src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'movdst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
"
Quoted 5 lines
Remapping in earlier patches is unnecessary.
These STX [r12-N],src_regemitted by LLVM will be JITed as-is into store ofsrc_reginto %rbp-M slot.
Only shift by 8 bytes is necessary for N to become M.
where STX of 6th argument becomes 'mov' from one register to x86's r9.
Okay, I will do the following jit stack layout:
incoming stack arg N -> 1
return adderss
saved rbp
BPF program stack
tail call cnt <== if tail call reachable
callee-saved regs
r9 <== if priv_frame_ptr is not null
outgoing stack arg M -> 1
call ...
undo stack of outgoing stack arg + r9
In this case, insn *(u64 *)(r12 - off) = val can directly write 'val' into proper outgoing stack arg locations.
if the call is a kfunc, outgoing stack should remove the bottom one for the 6th argument before the call.
Quoted 6 lines
The feature has to be zero overhead to pass these args from bpf to native and from
struct_opshooks into bpf progs.
All verifier considerations are secondary.pw-bot: cr
Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
Quoted 114 lines · Yonghong Song
Quoted 102 lines · Alexei Starovoitov
Quoted 69 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.The JIT follows the native
x86_64calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly wherex86_64expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth] │ ... │ (written via r12-relative STX/ST) │ outgoing stack arg M │ [rbp - prog_stack_depth - 8] ├─────────────────────────┤ │ callee-saved regs ... │ (pushed after sub rsp) └─────────────────────────┘ rsplow address
BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <=incoming_depth):native_off = 8 - bpf_off → [rbp + 16 + ...]- Outgoing args (store: -off >
incoming_depth):
native_off= -(bpf_prog_stack +stack_arg_depth+ 8) -bpf_offSince callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the
x86_64calling convention.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +8
-0@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) {arch/x86/net/bpf_jit_comp.c static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · line 410
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + }This is not any better than v1.
It is still a copy.
As I said earlier:
https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
It has to be zero overhead. Copy pasting:"
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N],src_reginto 'mov r9,src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'movdst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
"Quoted 5 lines
Remapping in earlier patches is unnecessary.
These STX [r12-N],src_regemitted by LLVM will be JITed as-is into store ofsrc_reginto %rbp-M slot.
Only shift by 8 bytes is necessary for N to become M.
where STX of 6th argument becomes 'mov' from one register to x86's r9.Okay, I will do the following jit stack layout:
Quoted 10 lines
incoming stack arg N -> 1 return adderss saved rbp BPF program stack tail call cnt <== if tail call reachable callee-saved regs r9 <== if priv_frame_ptr is not null outgoing stack arg M -> 1 call ... undo stack of outgoing stack arg + r9
It looks like you're trying to preserve r9 as an auxiliary register. If it's in the way, rewrite JIT handling. The size of the diff doesn't matter.
r9 should be the 6th argument.
Yonghong Song
reviewerRe: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments
Quoted 123 lines · Alexei Starovoitov
Quoted 112 lines · Yonghong Song
Quoted 102 lines · Alexei Starovoitov
Quoted 69 lines · Yonghong Song
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.The JIT follows the native
x86_64calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly wherex86_64expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.The native
x86_64stack layout for a function with incoming and outgoing stack args:high address
┌─────────────────────────┐
│ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller)
│ ... │
│ incoming stack arg 1 │ [rbp + 16]
├─────────────────────────┤
│ return address │ [rbp + 8]
│ saved rbp │ [rbp]
├─────────────────────────┤
│ BPF program stack │ (stack_depthbytes)
├─────────────────────────┤
│ outgoing stack arg 1 │ [rbp -prog_stack_depth-outgoing_depth]
│ ... │ (written via r12-relative STX/ST)
│ outgoing stack arg M │ [rbp -prog_stack_depth- 8]
├─────────────────────────┤
│ callee-saved regs ... │ (pushed after sub rsp)
└─────────────────────────┘ rsp
low addressBPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <=incoming_depth):
native_off= 8 -bpf_off→ [rbp + 16 + ...]
- Outgoing args (store: -off >incoming_depth):
native_off= -(bpf_prog_stack +stack_arg_depth+ 8) -bpf_offSince callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the
x86_64calling convention.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +8
-0@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) {arch/x86/net/bpf_jit_comp.c static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · line 410
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+ off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + }This is not any better than v1.
It is still a copy.
As I said earlier:
https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
It has to be zero overhead. Copy pasting:"
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N],src_reginto 'mov r9,src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'movdst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.
"Quoted 5 lines
Remapping in earlier patches is unnecessary.
These STX [r12-N],src_regemitted by LLVM will be JITed as-is into store ofsrc_reginto %rbp-M slot.
Only shift by 8 bytes is necessary for N to become M.
where STX of 6th argument becomes 'mov' from one register to x86's r9.Okay, I will do the following jit stack layout:
Quoted 10 lines
incoming stack arg N -> 1 return adderss saved rbp BPF program stack tail call cnt <== if tail call reachable callee-saved regs r9 <== if priv_frame_ptr is not null outgoing stack arg M -> 1 call ... undo stack of outgoing stack arg + r9
Quoted 3 lines
It looks like you're trying to preserve r9 as an auxiliary register. If it's in the way, rewrite JIT handling. The size of the diff doesn't matter.
I actually will put r9 (priv_frame_ptr) into the stack. The following is stack layout:
incoming stack arg N -> 1
return adderss
saved rbp
BPF program stack
tail call cnt <== if tail call reachable
callee-saved regs
r9 <== if priv_frame_ptr is not null
outgoing stack arg M -> 1
Quoted 1 line
r9 should be the 6th argument.
Yonghong Song
reviewerPATCH v3 9/11 selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongAdd selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.
--- .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++++++++ tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++++++++++++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 65 ++++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 20 +- 5 files changed, 592 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +132 -0
@@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +} + +static void test_global_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_global_many_args,36 ); + +out: +stack_arg__destroy (skel); +} + +static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +} + +static void test_bpf2bpf (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_bpf2bpf_ptr_stack_arg,45 ); +run_subtest (skel->progs.test_bpf2bpf_mix_stack_args,51 ); +run_subtest (skel->progs.test_bpf2bpf_nesting_stack_arg,50 ); +run_subtest (skel->progs.test_bpf2bpf_dynptr_stack_arg,69 ); + +out: +stack_arg__destroy (skel); +} + +static void test_kfunc (void ) +{ +struct stack_arg_kfunc *skel; + + skel =stack_arg_kfunc__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg_kfunc__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_stack_arg_scalar,36 ); +run_subtest (skel->progs.test_stack_arg_ptr,45 ); +run_subtest (skel->progs.test_stack_arg_mix,51 ); +run_subtest (skel->progs.test_stack_arg_dynptr,69 ); +run_subtest (skel->progs.test_stack_arg_mem,151 ); +run_subtest (skel->progs.test_stack_arg_iter,115 ); +run_subtest (skel->progs.test_stack_arg_const_str,15 ); +run_subtest (skel->progs.test_stack_arg_timer,15 ); + +out: +stack_arg_kfunc__destroy (skel); +} + +void test_stack_arg (void ) +{ +if (test__start_subtest ("global_many_args" )) +test_global_many (); +if (test__start_subtest ("async_cb_many_args" )) +test_async_cb_many (); +if (test__start_subtest ("bpf2bpf" )) +test_bpf2bpf (); +if (test__start_subtest ("kfunc" )) +test_kfunc (); +}
tools/testing/selftests/bpf/progs/stack_arg.c new file +212 -0
212 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +164 -0
164 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +65 -0
77 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +19 -1
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc { };#endif +struct bpf_iter_testmod_seq ; +struct prog_test_pass1 {int x0;struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; - +__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, + __u64 e, __u64 f, __u64 g, __u64 h) __ksym; +__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p) __ksym; +__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p, __u64 f, + struct prog_test_pass1 *q) __ksym; +__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_dynptr *ptr) __ksym; +__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + void *mem, int mem__sz) __ksym; +__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_iter_testmod_seq *it__iter) __ksym; +__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + const char *str__str) __ksym; +__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_timer *timer) __ksym; void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 10/11 selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to Yonghong SongAdd a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).
--- .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 7 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 8 +++++ 4 files changed, 71 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "stack_arg_fail.skel.h" + +void test_stack_arg_fail (void ) +{ +struct stack_arg_fail *skel; + + skel =stack_arg_fail__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" ); + +out: +stack_arg_fail__destroy (skel); +}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" + +#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +const volatile bool has_stack_arg = true; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +struct prog_test_big_arg s = { .a =1 , .b =2 }; + +return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); +} + +#else + +const volatile bool has_stack_arg = false; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +return 0 ; +} + +#endif + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
@@ -881,6 +881,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 ereturn a + b + c + d + e; } +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_big_arg s) +{ +return a + b + c + d + e + s.a + s.b; +} +static struct prog_test_ref_kfunc prog_test_struct = { .a =42 , .b =108 ,
@@ -1352,6 +1358,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)BTF_ID_FLAGS (func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +8 -0
@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +struct prog_test_fail1 {void *p;int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u const char *str__str) __ksym; __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, struct bpf_timer *timer) __ksym; +__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_big_arg s) __ksym; + void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v3 11/11 selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongAdd inline-asm-based verifier tests that exercise the stack argument validation logic directly.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_stack_arg.c | 302 ++++++++++++++++++ 2 files changed, 304 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -90,6 +90,7 @@#include "verifier_sockmap_mutate.skel.h" #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" +#include "verifier_stack_arg.skel.h" #include "verifier_stack_ptr.skel.h" #include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h"
@@ -236,6 +237,7 @@ void test_verifier_sock_addr(void) { RUN(verifier_sock_addr); }void test_verifier_sockmap_mutate (void ) {RUN (verifier_sockmap_mutate); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }void test_verifier_spin_lock (void ) {RUN (verifier_spin_lock); } +void test_verifier_stack_arg (void ) {RUN (verifier_stack_arg); }void test_verifier_stack_ptr (void ) {RUN (verifier_stack_ptr); }void test_verifier_store_release (void ) {RUN (verifier_store_release); }void test_verifier_subprog_precision (void ) {RUN (verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +302 -0
302 lines not on this page · raw message
Signature · 1 line
2.52.0