bpf: Support stack arguments for BPF functions and kfuncs
Yonghong Song
reviewerv2 cover bpf: Support stack arguments for BPF functions and kfuncs
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.
The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL. The incoming parameters can directly get the value from pushed native stack from caller. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).
Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.
For the rest of patches, patches 1-6 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 7 enables x86_64 for stack arguments. Patch 8 implemented JIT for x86_64. Patches 9-11 are some selftests.
[1] https://github.com/llvm/llvm-project/pull/189060
Changelogs:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.
Yonghong Song (11):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Refactor process_iter_arg() to have proper argument index
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validation
arch/x86/net/bpf_jit_comp.c | 140 +++++- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 31 +- include/linux/filter.h | 4 +- kernel/bpf/btf.c | 21 +- kernel/bpf/core.c | 12 +- kernel/bpf/verifier.c | 474 ++++++++++++++++-- .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++ .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 + .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 ++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++ .../selftests/bpf/progs/verifier_stack_arg.c | 302 +++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 72 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 26 + 16 files changed, 1594 insertions(+), 60 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 1/11 bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf register R12 added by [1]. R12 is used as the base for stack arguments so it won't mess out R10 based stacks.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/filter.h | 3 ++- kernel/bpf/core.c | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/filter.h +2 -1
@@ -59,7 +59,8 @@ struct ctl_table_header;/* Kernel hidden auxiliary/helper register. */ #define BPF_REG_AX MAX_BPF_REG -#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) +#define BPF_REG_STACK_ARG_BASE (MAX_BPF_REG + 1) +#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG/* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,u32 imm_rnd =get_random_u32 ();s16 off; -BUILD_BUG_ON (BPF_REG_AX +1 !=MAX_BPF_JIT_REG ); -BUILD_BUG_ON (MAX_BPF_REG +1 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_AX +2 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_STACK_ARG_BASE +1 !=MAX_BPF_JIT_REG );/* Constraints on AX register: *
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 2/11 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongCurrently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.
--- include/linux/bpf.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 3/11 bpf: Support stack arguments for bpf functions
In reply to Yonghong SongCurrently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.
The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar(a1, a2, a3, a4, a5, a6, a7, a8);
...
}The following is a illustration of stack allocation:
Caller (foo) Callee (bar) ============ ============ r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe() to do pruning check.
If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar1(a1, a2, a3, a4, a5, a6, a7, a8);
...
bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo) Callee (bar1) ============ ============= r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
...
Back from bar1
... Callee (bar2)
=== =============
+---> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
|+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
|||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+||| ...
r12-32: [outgoing arg 7 to callee]-+||
r12-40: [outgoing arg 8 to callee]--+|r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 31 +++- kernel/bpf/btf.c | 14 +- kernel/bpf/verifier.c | 311 +++++++++++++++++++++++++++++++++-- 4 files changed, 334 insertions(+), 24 deletions(-)
include/linux/bpf.h +2 -0
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +29 -2
@@ -268,6 +268,11 @@ struct bpf_retval_range {bool return_32bit; }; +struct bpf_stack_arg_state { +struct bpf_reg_state spilled_ptr;/* for spilled scalar/pointer semantics */ +u8 slot_type[BPF_REG_SIZE ]; +}; +/* state of the program: * type of all registers and stack info */
@@ -319,6 +324,10 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +struct bpf_stack_arg_state *stack_arg_slots; };#define MAX_CALL_FRAMES 8
@@ -456,6 +465,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_slots[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \ + ? &frame->stack_arg_slots[slot].spilled_ptr : NULL) + +/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */ +#define bpf_for_each_spilled_stack_arg (iter, frame, reg, mask) \ + for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \ + iter < frame->stack_arg_depth / BPF_REG_SIZE; \ + iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask)) +#define bpf_for_each_reg_in_vstate_mask (__vst, __state, __reg, __mask, __expr) \ ({ \ struct bpf_verifier_state *___vstate = __vst; \
@@ -473,6 +493,11 @@ struct bpf_verifier_state {continue ; \ (void )(__expr); \ } \ + bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \ +if (!__reg) \ +continue ; \ + (void )(__expr); \ + } \ } \ })
@@ -686,10 +711,12 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; };struct bpf_verifier_env ;
kernel/bpf/btf.c +10 -4
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))
kernel/bpf/verifier.c +293 -18
424 lines not on this page · raw message
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 4/11 bpf: Refactor process_iter_arg() to have proper argument index
In reply to Yonghong SongIn the next patch for kfunc stack arguments, a faked register is used to do proper verification checking. For process_iter_arg(), the regno is passed in and the iterator assumes 'regno - 1' as the argument index. This is wrong as regno is fake. So refactor process_iter_arg() by adding actual argument index which is used inside the function.
--- kernel/bpf/verifier.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +7 -7
@@ -9259,14 +9259,14 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, }static int process_iter_arg (struct bpf_verifier_env *env,int regno,int insn_idx, -struct bpf_kfunc_call_arg_meta *meta) +int argno,struct bpf_kfunc_call_arg_meta *meta) {struct bpf_reg_state *reg =reg_state (env, regno);const struct btf_type *t;int spi, err, i, nr_slots, btf_id;if (reg->type !=PTR_TO_STACK ) { -verbose (env,"arg#%d expected pointer to an iterator on stack\n" , regno -1 ); +verbose (env,"arg#%d expected pointer to an iterator on stack\n" , argno);return -EINVAL ; }
@@ -9276,9 +9276,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id* to any kfunc, if arg has "__iter" suffix, we need to be a bit more * conservative here. */ - btf_id =btf_check_iter_arg (meta->btf, meta->func_proto, regno -1 ); + btf_id =btf_check_iter_arg (meta->btf, meta->func_proto, argno);if (btf_id <0 ) { -verbose (env,"expected valid iter pointer as arg #%d\n" , regno -1 ); +verbose (env,"expected valid iter pointer as arg #%d\n" , argno);return -EINVAL ; } t =btf_type_by_id (meta->btf, btf_id);
@@ -9288,7 +9288,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id/* bpf_iter_<type>_new() expects pointer to uninit iter state */ if (!is_iter_reg_valid_uninit (env, reg, nr_slots)) {verbose (env,"expected uninitialized iter_%s as arg #%d\n" , -iter_type_str (meta->btf, btf_id), regno -1 ); +iter_type_str (meta->btf, btf_id), argno);return -EINVAL ; }
@@ -9312,7 +9312,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idbreak ;case -EINVAL :verbose (env,"expected an initialized iter_%s as arg #%d\n" , -iter_type_str (meta->btf, btf_id), regno -1 ); +iter_type_str (meta->btf, btf_id), argno);return err;case -EPROTO :verbose (env,"expected an RCU CS when using %s\n" , meta->func_name);
@@ -14063,7 +14063,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EINVAL ; } } - ret =process_iter_arg (env, regno, insn_idx, meta); + ret =process_iter_arg (env, regno, insn_idx, i, meta);if (ret <0 )return ret;break ;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 5/11 bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongExtend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.
The following is an example to show how stack arguments are saved:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}The following is an illustration:
Caller (foo)
============
r12-relative stack arg area:
r12-8: [incoming arg 6]
r12-16: [incoming arg 7]
---- incoming/outgoing boundary (kfunc1)
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
...
Back from kfunc1
...
---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
r12-48: [outgoing arg 9 to callee]Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().
In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. The following is one of examples based on a later selftest:
13: (85) call bpf_kfunc_call_stack_arg_mem#152105 Use reg 1 for stack arg#5
Use reg 2 to represent mem_size
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2 End of using reg 1 for stack arg#6
14: R0=scalar()
14: (95) exit
The above example is for KF_ARG_PTR_TO_MEM_SIZE case. Registers 1 and 2 are used as an temporary register for argument checking. The verifier log will identify when the temporary register is used and when the temporary register is not used any more.
There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno' is also keep regno for later fixup. Since regno is a faked one such three cases are rejected for now if they are in stack arguments. If possible, new kfuncs could keep them in first 5 registers so there are no issues at all.
If faked register approach is not favored, the following is an alternative approach with the following data structure:
struct reg_or_arg_t {
struct bpf_reg_state *state;
union {
int regno;
int argno;
};
bool is_reg; /* distinguish between reg and arg */
};Such a struct can replace existing regno/argno/reg_state parameters to make it easy to distinguish reg vs. arg.
--- kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 19 deletions(-)
kernel/bpf/verifier.c +124 -19
@@ -3491,7 +3491,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)struct bpf_kfunc_meta kfunc;struct bpf_kfunc_desc *desc;unsigned long addr; -int err; +int i, err; prog_aux = env->prog->aux; tab = prog_aux->kfunc_tab;
@@ -3567,6 +3567,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)if (err)return err; +for (i =MAX_BPF_FUNC_REG_ARGS ; i < func_model.nr_args; i++) { +if (func_model.arg_size[i] >sizeof (u64)) { +verbose (env,"kfunc %s arg#%d size %d > %zu not supported for stack args\n" , + kfunc.name, i, func_model.arg_size[i],sizeof (u64)); +return -EINVAL ; + } + } + desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; desc->offset = offset;
@@ -13083,6 +13091,19 @@ static bool is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + (argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_slots[spi].spilled_ptr; +} + static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,
@@ -13101,8 +13122,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,return KF_ARG_PTR_TO_CTX ;if (argno +1 < nargs && - (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]) || -is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]))) + (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )) || +is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )))) arg_mem_size = true;/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -13770,9 +13791,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (nargs >MAX_BPF_FUNC_ARGS ) {verbose (env,"Function %s has %d > %d args\n" , func_name, nargs, -MAX_BPF_FUNC_REG_ARGS ); +MAX_BPF_FUNC_ARGS );return -EINVAL ; }
@@ -13780,26 +13801,59 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg; +struct bpf_reg_state saved_reg;const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 regno = i +1 , ref_id, type_size;bool is_ret_buf_sz = false; +bool is_stack_arg = false;int kf_arg_type; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* +* Retrieve the spilled reg state from the stack arg slot. +* Reuse the existing type checking infrastructure which +* reads from cur_regs(env)[regno], temporarily copy the +* stack arg reg state into regs[BPF_REG_1] and restore +* it after checking. +*/ +struct bpf_func_state *caller =cur_func (env); +int spi = caller->incoming_stack_arg_depth /BPF_REG_SIZE + + (i -MAX_BPF_FUNC_REG_ARGS ); + +if (!is_stack_arg_slot_initialized (caller, spi)) { +verbose (env,"stack arg#%d not properly initialized\n" , i); +return -EINVAL ; + } + + is_stack_arg = true; + regno =BPF_REG_1 ; + saved_reg = regs[BPF_REG_1 ]; + regs[BPF_REG_1 ] = caller->stack_arg_slots[spi].spilled_ptr; + reg = ®s[BPF_REG_1 ]; +verbose (env,"Use reg %d for stack arg#%d\n" , regno, i); + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {verifier_bug (env,"Only 1 prog->aux argument supported per-kfunc" );return -EFAULT ; } +if (is_stack_arg) { +verbose (env,"arg#%d prog->aux cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->arg_prog = true;cur_aux (env)->arg_prog = regno; -continue ; + goto next_arg; }if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i)) -continue ; + goto next_arg; t =btf_type_skip_modifiers (btf, args[i].type,NULL );
@@ -13818,9 +13872,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"R%d must be a known constant\n" , regno);return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (!is_stack_arg) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + } meta->arg_constant.found = true; meta->arg_constant.value = reg->var_off.value; }else if (is_kfunc_arg_scalar_with_name (btf, &args[i],"rdonly_buf_size" )) {
@@ -13842,11 +13898,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; - ret =mark_chain_precision (env, regno); -if (ret) -return ret; +if (!is_stack_arg) { + ret =mark_chain_precision (env, regno); +if (ret) +return ret; + } } -continue ; + goto next_arg; }if (!btf_type_is_ptr (t)) {
@@ -13868,8 +13926,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EFAULT ; } meta->ref_obj_id = reg->ref_obj_id; -if (is_kfunc_release (meta)) +if (is_kfunc_release (meta)) { +if (is_stack_arg) { +verbose (env,"arg#%d release arg cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->release_regno = regno; + } } ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);
@@ -13881,7 +13944,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_switch (kf_arg_type) {case KF_ARG_PTR_TO_NULL : -continue ; + goto next_arg;case KF_ARG_PTR_TO_MAP :if (!reg->map_ptr) {verbose (env,"pointer in R%d isn't map pointer\n" , regno);
@@ -14020,6 +14083,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ dynptr_arg_type |=DYNPTR_TYPE_FILE ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) { dynptr_arg_type |=DYNPTR_TYPE_FILE ; +if (is_stack_arg) { +verbose (env,"arg#%d release arg cannot be a stack argument\n" , i); +return -EINVAL ; + } meta->release_regno = regno; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] && (dynptr_arg_type &MEM_UNINIT )) {
@@ -14169,8 +14236,18 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ {struct bpf_reg_state *buff_reg = ®s[regno];const struct btf_param *buff_arg = &args[i]; -struct bpf_reg_state *size_reg = ®s[regno +1 ]; +struct bpf_reg_state *size_reg;const struct btf_param *size_arg = &args[i +1 ]; +struct bpf_reg_state saved_size_reg = {}; +bool size_is_stack_arg = false; + +if (i >=MAX_BPF_FUNC_REG_ARGS ) { + size_is_stack_arg = true; + saved_size_reg = regs[regno +1 ]; + regs[regno +1 ] = *get_kfunc_arg_reg (env, i +1 ); +verbose (env,"Use reg %d to represent mem_size\n" , regno +1 ); + } + size_reg = ®s[regno +1 ];if (!register_is_null (buff_reg) || !is_kfunc_arg_nullable (meta->btf, buff_arg)) { ret =check_kfunc_mem_size_reg (env, size_reg, regno +1 );
@@ -14193,6 +14270,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->arg_constant.value = size_reg->var_off.value; } +if (size_is_stack_arg) + regs[regno +1 ] = saved_size_reg; +/* Skip next '__sz' or '__szk' argument */ i++;break ;
@@ -14294,6 +14374,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ; } } +next_arg: +if (is_stack_arg) { + regs[regno] = saved_reg; +verbose (env,"End of using reg %d for stack arg#%d\n" , regno, i); + } }if (is_kfunc_release (meta) && !meta->release_regno) {
@@ -15059,7 +15144,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, nargs =btf_type_vlen (meta.func_proto); args = (const struct btf_param *)(meta.func_proto +1 ); -for (i =0 ; i < nargs; i++) { +for (i =0 ; i < nargs && i <MAX_BPF_FUNC_REG_ARGS ; i++) {u32 regno = i +1 ; t =btf_type_skip_modifiers (desc_btf, args[i].type,NULL );
@@ -15070,6 +15155,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,mark_btf_func_reg_size (env, regno, t->size); } +/* Track outgoing stack arg depth for kfuncs with >5 args */ +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +struct bpf_func_state *caller =cur_func (env); +struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; +u16 kfunc_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + +if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth; + } +if (is_iter_next_kfunc (&meta)) { err =process_iter_next_call (env, insn_idx, &meta);if (err)
@@ -23975,6 +24070,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) };
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 6/11 bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong SongThe interpreter does not understand the bpf register r12 (BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.
--- kernel/bpf/core.c | 3 ++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
kernel/bpf/core.c +2 -1
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || + fp->aux->stack_arg_depth) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/verifier.c +6 -0
@@ -23944,6 +23944,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)verbose (env,"calling kernel functions are not allowed in non-JITed programs\n" );return -EINVAL ; } +for (i =0 ; i < env->subprog_cnt; i++) { +if (env->subprog_info[i].incoming_stack_arg_depth) { +verbose (env,"stack args are not supported in non-JITed programs\n" ); +return -EINVAL ; + } + }if (env->subprog_cnt >1 && env->prog->aux->tail_call_reachable) {/* When JIT fails the progs with bpf2bpf calls and tail_calls * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 7/11 bpf: Enable stack argument support for x86_64
In reply to Yonghong SongAdd stack argument support for x86_64.
--- arch/x86/net/bpf_jit_comp.c | 5 +++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 9 ++++++++- kernel/bpf/core.c | 5 +++++ 4 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
include/linux/filter.h +1 -0
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void ); +bool bpf_jit_supports_stack_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );
kernel/bpf/btf.c +8 -1
@@ -20,6 +20,7 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/bpf.h> +#include <linux/filter.h> #include <linux/bpf_lsm.h> #include <linux/skmsg.h> #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tname, nargs, MAX_BPF_FUNC_REG_ARGS);return -EINVAL ; } -if (nargs >MAX_BPF_FUNC_REG_ARGS ) +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d args\n" , + tname, nargs); +return -ENOTSUPP ; + } sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + }/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);
kernel/bpf/core.c +5 -0
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)return false; } +bool __weak bpf_jit_supports_stack_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 8/11 bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongAdd x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.
The JIT follows the native x86_64 calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly where x86_64 expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.
The native x86_64 stack layout for a function with incoming and outgoing stack args:
high address
┌─────────────────────────┐ │ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller) │ ... │ │ incoming stack arg 1 │ [rbp + 16] ├─────────────────────────┤ │ return address │ [rbp + 8] │ saved rbp │ [rbp] ├─────────────────────────┤ │ BPF program stack │ (stack_depth bytes) ├─────────────────────────┤ │ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth] │ ... │ (written via r12-relative STX/ST) │ outgoing stack arg M │ [rbp - prog_stack_depth - 8] ├─────────────────────────┤ │ callee-saved regs ... │ (pushed after sub rsp) └─────────────────────────┘ rsp
low address
BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <= incoming_depth):
native_off = 8 - bpf_off → [rbp + 16 + ...]
- Outgoing args (store: -off > incoming_depth):
native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
Since callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the x86_64 calling convention.
--- arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++-- 1 file changed, 129 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +129 -6
@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */ +static int push_stack_args (u8 **pprog,s32 outgoing_base,int from,int to) +{ +u8 *prog = *pprog; +int k, bytes =0 ; +s32 off; + +for (k = from; k >= to; k--) { + off = outgoing_base + (k -1 ) *8 ; +/* push qword [rbp + off] */ +if (is_imm8 (off)) { +EMIT3 (0xFF ,0x75 , off); + bytes +=3 ; + }else { +EMIT2_off32 (0xFF ,0xB5 , off); + bytes +=6 ; + } + } + *pprog = prog; +return bytes; +} +static void emit_nops (u8 **pprog,int len) {u8 *prog = *pprog;
@@ -1664,16 +1686,33 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; -u32 stack_depth; +u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth; +u32 prog_stack_depth, stack_depth; +bool has_stack_args;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Save program stack depth before adding outgoing stack arg space. +* Incoming stack args are read directly from [rbp + 16 + ...]. +* Only the outgoing stack arg area is allocated below the +* program stack. Outgoing args written here become the callee's +* incoming args. +*/ + prog_stack_depth =round_up (stack_depth,8 ); +if (outgoing_stack_arg_depth) + stack_depth += outgoing_stack_arg_depth; + has_stack_args = stack_arg_depth >0 ; + arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena);
@@ -1715,13 +1754,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image prog = temp;for (i =1 ; i <= insn_cnt; i++, insn++) { +bool adjust_stack_arg_off = false;const s32 imm32 = insn->imm;u32 dst_reg = insn->dst_reg;u32 src_reg = insn->src_reg;u8 b2 =0 , b3 =0 ;u8 *start_of_ldx;s64 jmp_offset; -s16 insn_off; +s32 insn_off;u8 jmp_cond;u8 *func;int nops;
@@ -1734,6 +1774,37 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image dst_reg =X86_REG_R9 ; } +if (has_stack_args) { +u8 class =BPF_CLASS (insn->code); + +if (class ==BPF_LDX && + src_reg ==BPF_REG_STACK_ARG_BASE ) { + src_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } +if ((class ==BPF_STX || class ==BPF_ST ) && + dst_reg ==BPF_REG_STACK_ARG_BASE ) { + dst_reg =BPF_REG_FP ; + adjust_stack_arg_off = true; + } + } + +/* +* Translate BPF r12-relative offset to native RBP-relative: +* +* Incoming args (load: offset >= -incoming_depth): +* BPF: r12 + bpf_off = r12 - k * 8 (k = 1,2,...) for incoming arg k +* Native: [rbp + 8 + k * 8] +* Formula: native_off = 8 + k * 8 = 8 - bpf_off +* +* Outgoing args (store: offset < -incoming_depth): +* BPF: r12 + bpf_off = r12 - (incoming + k * 8) for outgoing arg k +* Native: [rbp - prog_stack_depth - outgoing + (k - 1) * 8] +* Formula: native_off = -(prog_stack_depth + outgoing) + (k - 1) * 8 +* = -(prog_stack_depth + outgoing + incoming + 8) - bpf_off +* = -(prog_stack_depth + stack_arg_depth + 8) - bpf_off +*/ +switch (insn->code) {/* ALU */ case BPF_ALU |BPF_ADD |BPF_X :
@@ -2131,10 +2202,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ST |BPF_MEM |BPF_DW :EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off = -(prog_stack_depth + stack_arg_depth +8 ) - insn_off; +if (is_imm8 (insn_off)) +EMIT2 (add_1reg (0x40 , dst_reg), insn_off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), insn_off);EMIT (imm32,bpf_size_to_x86_bytes (BPF_SIZE (insn->code)));break ;
@@ -2144,7 +2218,10 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W :case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off); + insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off = -(prog_stack_depth + stack_arg_depth +8 ) - insn_off; +emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);break ;case BPF_ST |BPF_PROBE_MEM32 |BPF_B :
@@ -2243,6 +2320,8 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (adjust_stack_arg_off) + insn_off =8 - insn_off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ||BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {
@@ -2441,6 +2520,7 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : {u8 *ip = image + addrs[i -1 ]; +int stack_args =0 ; func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off)) }if (!imm32)return -EINVAL ; + +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* +* BPF-to-BPF calls: push outgoing stack args from +* the outgoing area onto the stack before CALL. +* The outgoing area is at [rbp - prog_stack - outgoing], +* but rsp is below that due to callee-saved reg pushes, +* so we must explicitly push args for the callee. +*/ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int n_args = outgoing_stack_arg_depth /8 ; + + ip +=push_stack_args (&prog, outgoing_base, n_args,1 ); + } + +if (src_reg !=BPF_PSEUDO_CALL && insn->off >0 ) { +/* Kfunc calls: arg 6 → R9, args 7+ → push. */ +s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth); +int kfunc_stack_args = insn->off; + + stack_args = kfunc_stack_args >1 ? kfunc_stack_args -1 :0 ; + +/* Push args 7+ in reverse order */ +if (stack_args >0 ) + ip +=push_stack_args (&prog, outgoing_base, kfunc_stack_args,2 ); + +/* mov r9, [rbp + outgoing_base] (arg 6) */ +if (is_imm8 (outgoing_base)) { +EMIT4 (0x4C ,0x8B ,0x4D , outgoing_base); + ip +=4 ; + }else { +EMIT3_off32 (0x4C ,0x8B ,0x8D , outgoing_base); + ip +=7 ; + } + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;
@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (stack_args >0 ) { +/* add rsp, stack_args * 8 */ +EMIT4 (0x48 ,0x83 ,0xC4 , stack_args *8 ); + } +if (src_reg ==BPF_PSEUDO_CALL && outgoing_stack_arg_depth >0 ) { +/* add rsp, outgoing_stack_arg_depth */ +EMIT4 (0x48 ,0x83 ,0xC4 , outgoing_stack_arg_depth); + }break ; }
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 9/11 selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongAdd selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.
--- .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++++++++ tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++++++++++++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 65 ++++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 20 +- 5 files changed, 592 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +132 -0
@@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +} + +static void test_global_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_global_many_args,36 ); + +out: +stack_arg__destroy (skel); +} + +static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +} + +static void test_bpf2bpf (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_bpf2bpf_ptr_stack_arg,45 ); +run_subtest (skel->progs.test_bpf2bpf_mix_stack_args,51 ); +run_subtest (skel->progs.test_bpf2bpf_nesting_stack_arg,50 ); +run_subtest (skel->progs.test_bpf2bpf_dynptr_stack_arg,69 ); + +out: +stack_arg__destroy (skel); +} + +static void test_kfunc (void ) +{ +struct stack_arg_kfunc *skel; + + skel =stack_arg_kfunc__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg_kfunc__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_stack_arg_scalar,36 ); +run_subtest (skel->progs.test_stack_arg_ptr,45 ); +run_subtest (skel->progs.test_stack_arg_mix,51 ); +run_subtest (skel->progs.test_stack_arg_dynptr,69 ); +run_subtest (skel->progs.test_stack_arg_mem,151 ); +run_subtest (skel->progs.test_stack_arg_iter,115 ); +run_subtest (skel->progs.test_stack_arg_const_str,15 ); +run_subtest (skel->progs.test_stack_arg_timer,15 ); + +out: +stack_arg_kfunc__destroy (skel); +} + +void test_stack_arg (void ) +{ +if (test__start_subtest ("global_many_args" )) +test_global_many (); +if (test__start_subtest ("async_cb_many_args" )) +test_async_cb_many (); +if (test__start_subtest ("bpf2bpf" )) +test_bpf2bpf (); +if (test__start_subtest ("kfunc" )) +test_kfunc (); +}
tools/testing/selftests/bpf/progs/stack_arg.c new file +212 -0
212 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +164 -0
164 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +65 -0
77 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +19 -1
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc { };#endif +struct bpf_iter_testmod_seq ; +struct prog_test_pass1 {int x0;struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; - +__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, + __u64 e, __u64 f, __u64 g, __u64 h) __ksym; +__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p) __ksym; +__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p, __u64 f, + struct prog_test_pass1 *q) __ksym; +__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_dynptr *ptr) __ksym; +__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + void *mem, int mem__sz) __ksym; +__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_iter_testmod_seq *it__iter) __ksym; +__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + const char *str__str) __ksym; +__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_timer *timer) __ksym; void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 10/11 selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to Yonghong SongAdd a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).
--- .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 7 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 8 +++++ 4 files changed, 71 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "stack_arg_fail.skel.h" + +void test_stack_arg_fail (void ) +{ +struct stack_arg_fail *skel; + + skel =stack_arg_fail__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" ); + +out: +stack_arg_fail__destroy (skel); +}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" + +#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +const volatile bool has_stack_arg = true; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +struct prog_test_big_arg s = { .a =1 , .b =2 }; + +return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); +} + +#else + +const volatile bool has_stack_arg = false; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +return 0 ; +} + +#endif + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
@@ -881,6 +881,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 ereturn a + b + c + d + e; } +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_big_arg s) +{ +return a + b + c + d + e + s.a + s.b; +} +static struct prog_test_ref_kfunc prog_test_struct = { .a =42 , .b =108 ,
@@ -1352,6 +1358,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)BTF_ID_FLAGS (func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +8 -0
@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +struct prog_test_fail1 {void *p;int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u const char *str__str) __ksym; __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, struct bpf_timer *timer) __ksym; +__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_big_arg s) __ksym; + void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v2 11/11 selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongAdd inline-asm-based verifier tests that exercise the stack argument validation logic directly.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_stack_arg.c | 302 ++++++++++++++++++ 2 files changed, 304 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -90,6 +90,7 @@#include "verifier_sockmap_mutate.skel.h" #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" +#include "verifier_stack_arg.skel.h" #include "verifier_stack_ptr.skel.h" #include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h"
@@ -236,6 +237,7 @@ void test_verifier_sock_addr(void) { RUN(verifier_sock_addr); }void test_verifier_sockmap_mutate (void ) {RUN (verifier_sockmap_mutate); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }void test_verifier_spin_lock (void ) {RUN (verifier_spin_lock); } +void test_verifier_stack_arg (void ) {RUN (verifier_stack_arg); }void test_verifier_stack_ptr (void ) {RUN (verifier_stack_ptr); }void test_verifier_store_release (void ) {RUN (verifier_store_release); }void test_verifier_subprog_precision (void ) {RUN (verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +302 -0
302 lines not on this page · raw message
Signature · 1 line
2.52.0
Yonghong Song
reviewerQuoted 75 lines · Yonghong Song
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r12 (
BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped atMAX_BPF_FUNC_ARGS(12), which is sufficient for the vast majority of use cases.The
x86_64JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL. The incoming parameters can directly get the value from pushed native stack from caller. For kfunc calls, args are marshaled per thex86_64C calling convention (arg 6 in R9, args 7+ on the native stack).Global subprogs with >5 args are not yet supported. Only
x86_64is supported for now.For the rest of patches, patches 1-6 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 7 enables
x86_64for stack arguments. Patch 8 implemented JIT forx86_64. Patches 9-11 are some selftests.[1] https://github.com/llvm/llvm-project/pull/189060Changelogs:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Addstack_arg_safe()to do pruning for stack arguments.
- Fix an issue withKF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- Forx86_64JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.Yonghong Song (11):
bpf: Introduce bpf registerBPF_REG_STACK_ARG_BASE
bpf: ReuseMAX_BPF_FUNC_ARGSfor maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Refactorprocess_iter_arg()to have proper argument index
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support forx86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validationarch/x86/net/bpf_jit_comp.c | 140 +++++- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 31 +- include/linux/filter.h | 4 +- kernel/bpf/btf.c | 21 +- kernel/bpf/core.c | 12 +- kernel/bpf/verifier.c | 474 ++++++++++++++++-- .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++ .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 + .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 ++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++ .../selftests/bpf/progs/verifier_stack_arg.c | 302 +++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 72 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 26 + 16 files changed, 1594 insertions(+), 60 deletions(-)
Quoted 7 lines
create mode 100644
tools/testing/selftests/bpf/prog_tests/stack_arg.ccreate mode 100644tools/testing/selftests/bpf/prog_tests/stack_arg_fail.ccreate mode 100644tools/testing/selftests/bpf/progs/stack_arg.ccreate mode 100644tools/testing/selftests/bpf/progs/stack_arg_fail.ccreate mode 100644tools/testing/selftests/bpf/progs/stack_arg_kfunc.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Please ignore this patch set. Just discovered that there is a selftest failure and a couple of 'UTF-8' issues. I will fix these issues and send the next revision shortly.