bpf: Support stack arguments for BPF functions and kfuncs
Yonghong Song
reviewerv5 cover bpf: Support stack arguments for BPF functions and kfuncs
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.
In verifier, r11 based stores can survive bpf-to-bpf and kfunc calls. For example
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
call bar2; // reuses same arg6, arg7 without re-storing
If users want to have a different parameter for bar2(), it can have a store after bar1() and before bar2().
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
*(u64 *)(r11 - 16) = r8;
call bar2; // arg6 = r6, arg7 = r8The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.
To support kfunc stack arguments, before doing any stack arguments, existing codes are refactored/modified to use bpf_reg_state as much as possible instead of using regno, and to pass a non-negative argno, encoded to support both registers and stack arguments, as a single variable.
Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.
For the rest of patches, patches 1-4 make changes to make it easy for future stack arguments for kfuncs. Patches 5-8 supports bpf-to-bpf stack arguments. Patch 9 rejects interpreter for stack arguments. Patch 10 rejects subprogs if tailcall reachable. Patch 11 adds stack argument support for kfuncs. Patch 12 enables stack arguments for x86_64 and Patch 13 implements the x86_64 JIT. Patches 14-16 are some test cases.
[1] https://github.com/llvm/llvm-project/pull/189060
Note:
- The patch set is on top of the following commit:
1f5ffc672165 Fix mismerge of the arm64 / timer-core interrupt handling changes
- This patch set requires latest llvm23 compiler. It is possible that a build
failure may appear:
/home/yhs/work/bpf-next/scripts/mod/modpost.c:59:13: error: variable 'extra_warn' set but not used [-Werror,-Wunused-but-set-global]
59 | static bool extra_warn;
| ^
1 error generated.
In this case, the following hack can workaround the build issue:
--- a/Makefile
+++ b/Makefile
@@ -467,7 +467,7 @@ KERNELDOC = $(srctree)/tools/docs/kernel-doc
export KERNELDOC
KBUILD_USERHOSTCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \
- -O2 -fomit-frame-pointer -std=gnu11
+ -O2 -fomit-frame-pointer -std=gnu11 -Wno-unused-but-set-global
KBUILD_USERCFLAGS := $(KBUILD_USERHOSTCFLAGS) $(USERCFLAGS)
KBUILD_USERLDFLAGS := $(USERLDFLAGS)Changelogs:
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Use r11 instead of r12, llvm also updated with r11.
- Change int type 'reg_or_arg' to u32 'argno' where 'argno' encodes to support both bpf registers and stack arguments.
- Track per-state bitmask 'out_stack_arg_mask' for r11 based stores, so at any particular call, it knows what stores are available. This is important since stores may be in different basic block.
- Previously after each call, all store slots are invalidated. This patches disabled such invalidation.
- Ensure r11 reg only appearing in allowed insns. Also avoid r11 for reg tracking purpose.
- Make stack_arg_regs more similar to regular reg's (struct bpf_reg_state *)..
- Reorder r11 based stores from 'arg6:off:-24, arg7:off:-16, arg8:off:-8" to "arg6:off:-8, arg7:off:-16, arg8:off:-24".
- Add a few more tests, including e.g., two callee's with different number of stack arguments, shared r11-stores in different branches, etc.
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Refactor/Modify codes to make it easier for later kfunc stack argument support
- Invalidate outgoing slots immediately after the call to prevent reuse
- Fix interaction between stack argument PTR_TO_STACK and dead slot poisoning
- Reject stack arguments if tail call reachable
- Disable private stack if stack argument is used
- Allocate outgoing stack argument region after callee saved registers, and this simplifies the JITed code a lot.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Fix selftest stack_arg_gap_at_minus8().
- Fix a few 'UTF-8' issues.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.
Yonghong Song (16):
bpf: Remove unused parameter from check_map_kptr_access()
bpf: Refactor to avoid redundant calculation of bpf_reg_state
bpf: Refactor to handle memory and size together
bpf: Prepare verifier logs for upcoming kfunc stack arguments
bpf: Introduce bpf register BPF_REG_PARAMS
bpf: Limit the scope of BPF_REG_PARAMS usage
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Reject stack arguments in non-JITed programs
bpf: Reject stack arguments if tail call reachable
bpf: Support stack arguments for kfunc calls
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validation
arch/x86/net/bpf_jit_comp.c | 154 ++- include/linux/bpf.h | 6 + include/linux/bpf_verifier.h | 29 +- include/linux/filter.h | 6 +- kernel/bpf/btf.c | 20 +- kernel/bpf/const_fold.c | 9 +- kernel/bpf/core.c | 11 +- kernel/bpf/fixups.c | 32 +- kernel/bpf/liveness.c | 9 +- kernel/bpf/states.c | 41 + kernel/bpf/verifier.c | 1200 +++++++++++------ .../testing/selftests/bpf/prog_tests/bpf_nf.c | 22 +- .../selftests/bpf/prog_tests/cb_refs.c | 2 +- .../selftests/bpf/prog_tests/ctx_rewrite.c | 14 +- .../selftests/bpf/prog_tests/kfunc_call.c | 2 +- .../selftests/bpf/prog_tests/linked_list.c | 4 +- .../selftests/bpf/prog_tests/stack_arg.c | 133 ++ .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 + .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/cgrp_kfunc_failure.c | 14 +- .../selftests/bpf/progs/cpumask_failure.c | 10 +- .../testing/selftests/bpf/progs/dynptr_fail.c | 22 +- .../selftests/bpf/progs/file_reader_fail.c | 4 +- tools/testing/selftests/bpf/progs/irq.c | 4 +- tools/testing/selftests/bpf/progs/iters.c | 6 +- .../selftests/bpf/progs/iters_state_safety.c | 14 +- .../selftests/bpf/progs/iters_testmod.c | 4 +- .../selftests/bpf/progs/iters_testmod_seq.c | 4 +- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/percpu_alloc_fail.c | 4 +- .../testing/selftests/bpf/progs/rbtree_fail.c | 6 +- .../bpf/progs/refcounted_kptr_fail.c | 2 +- tools/testing/selftests/bpf/progs/stack_arg.c | 254 ++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 + .../selftests/bpf/progs/stack_arg_kfunc.c | 164 +++ .../testing/selftests/bpf/progs/stream_fail.c | 2 +- .../selftests/bpf/progs/task_kfunc_failure.c | 18 +- .../selftests/bpf/progs/task_work_fail.c | 6 +- .../selftests/bpf/progs/test_bpf_nf_fail.c | 8 +- .../bpf/progs/test_kfunc_dynptr_param.c | 2 +- .../bpf/progs/test_kfunc_param_nullable.c | 2 +- .../selftests/bpf/progs/verifier_bits_iter.c | 4 +- .../bpf/progs/verifier_bpf_fastcall.c | 24 +- .../selftests/bpf/progs/verifier_may_goto_1.c | 12 +- .../bpf/progs/verifier_ref_tracking.c | 6 +- .../selftests/bpf/progs/verifier_sdiv.c | 64 +- .../selftests/bpf/progs/verifier_stack_arg.c | 463 +++++++ .../selftests/bpf/progs/verifier_vfs_reject.c | 8 +- .../testing/selftests/bpf/progs/wq_failures.c | 2 +- .../selftests/bpf/test_kmods/bpf_testmod.c | 73 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 26 + tools/testing/selftests/bpf/verifier/calls.c | 14 +- 52 files changed, 2442 insertions(+), 558 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 1/16 bpf: Remove unused parameter from check_map_kptr_access()
The parameter 'regno' in check_map_kptr_access() is unused. Remove it.
--- kernel/bpf/verifier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +2 -2
@@ -4711,7 +4711,7 @@ static int mark_uptr_ld_reg(struct bpf_verifier_env *env, u32 regno,return 0 ; } -static int check_map_kptr_access (struct bpf_verifier_env *env,u32 regno, +static int check_map_kptr_access (struct bpf_verifier_env *env,int value_regno,int insn_idx,struct btf_field *kptr_field) {
@@ -6358,7 +6358,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn kptr_field =btf_record_find (reg->map_ptr->record, off + reg->var_off.value,BPF_KPTR |BPF_UPTR );if (kptr_field) { - err =check_map_kptr_access (env, regno, value_regno, insn_idx, kptr_field); + err =check_map_kptr_access (env, value_regno, insn_idx, kptr_field); }else if (t ==BPF_READ && value_regno >=0 ) {struct bpf_map *map = reg->map_ptr;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 2/16 bpf: Refactor to avoid redundant calculation of bpf_reg_state
In reply to Yonghong SongIn many cases, once a bpf_reg_state is defined, it can pass to callee's. Otherwise, callee will need to get bpf_reg_state again based on regno. More importantly, this is needed for later stack arguments for kfuncs since the register state for stack arguments does not have a corresponding regno. So it makes sense to pass reg state for callee's.
The following is the only change to avoid compilation warning:
static int sanitize_check_bounds(struct bpf_verifier_env *env,
const struct bpf_insn *insn,
- const struct bpf_reg_state *dst_reg)
+ struct bpf_reg_state *dst_reg)
--- kernel/bpf/verifier.c | 213 ++++++++++++++++++------------------------ 1 file changed, 93 insertions(+), 120 deletions(-)
kernel/bpf/verifier.c +93 -120
@@ -3933,13 +3933,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,static int check_stack_write_var_off (struct bpf_verifier_env *env,/* func where register points to */ struct bpf_func_state *state, -int ptr_regno,int off,int size, +struct bpf_reg_state *ptr_reg,int off,int size,int value_regno,int insn_idx) {struct bpf_func_state *cur;/* state of the current function */ int min_off, max_off;int i, err; -struct bpf_reg_state *ptr_reg =NULL , *value_reg =NULL ; +struct bpf_reg_state *value_reg =NULL ;struct bpf_insn *insn = &env->prog->insnsi[insn_idx];bool writing_zero = false;/* set if the fact that we're writing a zero is used to let any
@@ -3948,7 +3948,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,bool zero_used = false; cur = env->cur_state->frame[env->cur_state->curframe]; - ptr_reg = &cur->regs[ptr_regno]; min_off = ptr_reg->smin_value + off; max_off = ptr_reg->smax_value + off + size;if (value_regno >=0 )
@@ -4245,7 +4244,7 @@ enum bpf_access_src { ACCESS_HELPER = 2, /* the access is performed by a helper */ }; -static int check_stack_range_initialized(struct bpf_verifier_env *env, +static int check_stack_range_initialized(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int regno, int off, int access_size, bool zero_size_allowed, enum bpf_access_type type,
@@ -4269,18 +4268,16 @@ static struct bpf_reg_state *reg_state(struct bpf_verifier_env *env, int regno)* offset; for a fixed offset check_stack_read_fixed_off should be used * instead. */ -static int check_stack_read_var_off (struct bpf_verifier_env *env, +static int check_stack_read_var_off (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int ptr_regno,int off,int size,int dst_regno) { -/* The state of the source register. */ -struct bpf_reg_state *reg =reg_state (env, ptr_regno);struct bpf_func_state *ptr_state =bpf_func (env, reg);int err;int min_off, max_off;/* Note that we pass a NULL meta, so raw access will not be permitted. */ - err =check_stack_range_initialized (env, ptr_regno, off, size, + err =check_stack_range_initialized (env, reg, ptr_regno, off, size, false,BPF_READ ,NULL );if (err)return err;
@@ -4302,10 +4299,9 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env,* can be -1, meaning that the read value is not going to a register. */ static int check_stack_read (struct bpf_verifier_env *env, -int ptr_regno,int off,int size, +struct bpf_reg_state *reg,int ptr_regno,int off,int size,int dst_regno) { -struct bpf_reg_state *reg =reg_state (env, ptr_regno);struct bpf_func_state *state =bpf_func (env, reg);int err;/* Some accesses are only permitted with a static offset. */
@@ -4341,7 +4337,7 @@ static int check_stack_read(struct bpf_verifier_env *env,* than fixed offset ones. Note that dst_regno >= 0 on this * branch. */ - err =check_stack_read_var_off (env, ptr_regno, off, size, + err =check_stack_read_var_off (env, reg, ptr_regno, off, size, dst_regno); }return err;
@@ -4358,10 +4354,9 @@ static int check_stack_read(struct bpf_verifier_env *env,* The caller must ensure that the offset falls within the maximum stack size. */ static int check_stack_write (struct bpf_verifier_env *env, -int ptr_regno,int off,int size, +struct bpf_reg_state *reg,int off,int size,int value_regno,int insn_idx) { -struct bpf_reg_state *reg =reg_state (env, ptr_regno);struct bpf_func_state *state =bpf_func (env, reg);int err;
@@ -4374,16 +4369,15 @@ static int check_stack_write(struct bpf_verifier_env *env,* than fixed offset ones. */ err =check_stack_write_var_off (env, state, - ptr_regno, off, size, + reg, off, size, value_regno, insn_idx); }return err; } -static int check_map_access_type (struct bpf_verifier_env *env,u32 regno, +static int check_map_access_type (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,int off,int size,enum bpf_access_type type) { -struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_map *map = reg->map_ptr;u32 cap =bpf_map_flags_to_cap (map);
@@ -4403,17 +4397,15 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, }/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */ -static int __check_mem_access (struct bpf_verifier_env *env,int regno, +static int __check_mem_access (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int off,int size,u32 mem_size,bool zero_size_allowed) {bool size_ok = size >0 || (size ==0 && zero_size_allowed); -struct bpf_reg_state *reg;if (off >=0 && size_ok && (u64 )off + size <= mem_size)return 0 ; - reg = &cur_regs (env)[regno];switch (reg->type) {case PTR_TO_MAP_KEY : verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n",
@@ -4443,13 +4435,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, }/* check read/write into a memory region with possible variable offset */ -static int check_mem_region_access (struct bpf_verifier_env *env,u32 regno, +static int check_mem_region_access (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,int off,int size,u32 mem_size,bool zero_size_allowed) { -struct bpf_verifier_state *vstate = env->cur_state; -struct bpf_func_state *state = vstate->frame[vstate->curframe]; -struct bpf_reg_state *reg = &state->regs[regno];int err;/* We may have adjusted the register pointing to memory region, so we
@@ -4470,7 +4459,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, regno);return -EACCES ; } - err =__check_mem_access (env, regno, reg->smin_value + off, size, + err =__check_mem_access (env, reg, regno, reg->smin_value + off, size, mem_size, zero_size_allowed);if (err) { verbose(env, "R%d min value is outside of the allowed memory range\n",
@@ -4487,7 +4476,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, regno);return -EACCES ; } - err =__check_mem_access (env, regno, reg->umax_value + off, size, + err =__check_mem_access (env, reg, regno, reg->umax_value + off, size, mem_size, zero_size_allowed);if (err) { verbose(env, "R%d max value is outside of the allowed memory range\n",
@@ -4788,19 +4777,16 @@ static u32 map_mem_size(const struct bpf_map *map) }/* check read/write into a map element with possible variable offset */ -static int check_map_access (struct bpf_verifier_env *env,u32 regno, +static int check_map_access (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,int off,int size,bool zero_size_allowed,enum bpf_access_src src) { -struct bpf_verifier_state *vstate = env->cur_state; -struct bpf_func_state *state = vstate->frame[vstate->curframe]; -struct bpf_reg_state *reg = &state->regs[regno];struct bpf_map *map = reg->map_ptr;u32 mem_size =map_mem_size (map);struct btf_record *rec;int err, i; - err =check_mem_region_access (env, regno, off, size, mem_size, zero_size_allowed); + err =check_mem_region_access (env, reg, regno, off, size, mem_size, zero_size_allowed);if (err)return err;
@@ -4896,10 +4882,9 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, } } -static int check_packet_access (struct bpf_verifier_env *env,u32 regno,int off, +static int check_packet_access (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,int off,int size,bool zero_size_allowed) { -struct bpf_reg_state *reg =reg_state (env, regno);int err;if (reg->range <0 ) {
@@ -4907,7 +4892,7 @@ static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,return -EINVAL ; } - err =check_mem_region_access (env, regno, off, size, reg->range, zero_size_allowed); + err =check_mem_region_access (env, reg, regno, off, size, reg->range, zero_size_allowed);if (err)return err;
@@ -4962,7 +4947,7 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int ofreturn -EACCES ; } -static int check_ctx_access (struct bpf_verifier_env *env,int insn_idx,u32 regno, +static int check_ctx_access (struct bpf_verifier_env *env,int insn_idx,struct bpf_reg_state *reg,u32 regno,int off,int access_size,enum bpf_access_type t,struct bpf_insn_access_aux *info) {
@@ -4972,12 +4957,10 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn*/ bool var_off_ok =is_var_ctx_off_allowed (env->prog);bool fixed_off_ok = !env->ops->convert_ctx_access; -struct bpf_reg_state *regs =cur_regs (env); -struct bpf_reg_state *reg = regs + regno;int err;if (var_off_ok) - err =check_mem_region_access (env, regno, off, access_size,U16_MAX , false); + err =check_mem_region_access (env, reg, regno, off, access_size,U16_MAX , false);else err =__check_ptr_off_reg (env, reg, regno, fixed_off_ok);if (err)
@@ -5003,10 +4986,9 @@ static int check_flow_keys_access(struct bpf_verifier_env *env, int off, }static int check_sock_access (struct bpf_verifier_env *env,int insn_idx, -u32 regno,int off,int size, +struct bpf_reg_state *reg,u32 regno,int off,int size,enum bpf_access_type t) { -struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_insn_access_aux info = {};bool valid;
@@ -5969,12 +5951,11 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, }static int check_ptr_to_btf_access (struct bpf_verifier_env *env, -struct bpf_reg_state *regs, +struct bpf_reg_state *regs,struct bpf_reg_state *reg,int regno,int off,int size,enum bpf_access_type atype,int value_regno) { -struct bpf_reg_state *reg = regs + regno;const struct btf_type *t =btf_type_by_id (reg->btf, reg->btf_id);const char *tname =btf_name_by_offset (reg->btf, t->name_off);const char *field_name =NULL ;
@@ -6126,12 +6107,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, }static int check_ptr_to_map_access (struct bpf_verifier_env *env, -struct bpf_reg_state *regs, +struct bpf_reg_state *regs,struct bpf_reg_state *reg,int regno,int off,int size,enum bpf_access_type atype,int value_regno) { -struct bpf_reg_state *reg = regs + regno;struct bpf_map *map = reg->map_ptr;struct bpf_reg_state map_reg;enum bpf_type_flag flag =0 ;
@@ -6220,11 +6200,10 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,* 'off' includes `regno->offset`, but not its dynamic part (if any). */ static int check_stack_access_within_bounds ( -struct bpf_verifier_env *env, +struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int off,int access_size,enum bpf_access_type type) { -struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_func_state *state =bpf_func (env, reg);s64 min_off, max_off;int err;
@@ -6312,12 +6291,11 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)* if t==write && value_regno==-1, some unknown value is stored into memory * if t==read && value_regno==-1, don't care what we read from memory */ -static int check_mem_access (struct bpf_verifier_env *env,int insn_idx,u32 regno, +static int check_mem_access (struct bpf_verifier_env *env,int insn_idx,struct bpf_reg_state *reg,u32 regno,int off,int bpf_size,enum bpf_access_type t,int value_regno,bool strict_alignment_once,bool is_ldsx) {struct bpf_reg_state *regs =cur_regs (env); -struct bpf_reg_state *reg = regs + regno;int size, err =0 ; size =bpf_size_to_bytes (bpf_size);
@@ -6334,7 +6312,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnreturn -EACCES ; } - err =check_mem_region_access (env, regno, off, size, + err =check_mem_region_access (env, reg, regno, off, size, reg->map_ptr->key_size, false);if (err)return err;
@@ -6348,10 +6326,10 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnverbose (env,"R%d leaks addr into map\n" , value_regno);return -EACCES ; } - err =check_map_access_type (env, regno, off, size, t); + err =check_map_access_type (env, reg, regno, off, size, t);if (err)return err; - err =check_map_access (env, regno, off, size, false,ACCESS_DIRECT ); + err =check_map_access (env, reg, regno, off, size, false,ACCESS_DIRECT );if (err)return err;if (tnum_is_const (reg->var_off))
@@ -6420,7 +6398,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn* instructions, hence no need to check bounds in that case. */ if (!rdonly_untrusted) - err =check_mem_region_access (env, regno, off, size, + err =check_mem_region_access (env, reg, regno, off, size, reg->mem_size, false);if (!err && value_regno >=0 && (t ==BPF_READ || rdonly_mem))mark_reg_unknown (env, regs, value_regno);
@@ -6438,7 +6416,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnreturn -EACCES ; } - err =check_ctx_access (env, insn_idx, regno, off, size, t, &info); + err =check_ctx_access (env, insn_idx, reg, regno, off, size, t, &info);if (!err && t ==BPF_READ && value_regno >=0 ) {/* ctx access returns either a scalar, or a * PTR_TO_PACKET[_META,_END]. In the latter
@@ -6475,15 +6453,15 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn }else if (reg->type ==PTR_TO_STACK ) {/* Basic bounds checks. */ - err =check_stack_access_within_bounds (env, regno, off, size, t); + err =check_stack_access_within_bounds (env, reg, regno, off, size, t);if (err)return err;if (t ==BPF_READ ) - err =check_stack_read (env, regno, off, size, + err =check_stack_read (env, reg, regno, off, size, value_regno);else - err =check_stack_write (env, regno, off, size, + err =check_stack_write (env, reg, off, size, value_regno, insn_idx); }else if (reg_is_pkt_pointer (reg)) {if (t ==BPF_WRITE && !may_access_direct_pkt_data (env,NULL , t)) {
@@ -6496,7 +6474,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn value_regno);return -EACCES ; } - err =check_packet_access (env, regno, off, size, false); + err =check_packet_access (env, reg, regno, off, size, false);if (!err && t ==BPF_READ && value_regno >=0 )mark_reg_unknown (env, regs, value_regno); }else if (reg->type ==PTR_TO_FLOW_KEYS ) {
@@ -6516,7 +6494,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn regno, reg_type_str(env, reg->type));return -EACCES ; } - err =check_sock_access (env, insn_idx, regno, off, size, t); + err =check_sock_access (env, insn_idx, reg, regno, off, size, t);if (!err && value_regno >=0 )mark_reg_unknown (env, regs, value_regno); }else if (reg->type ==PTR_TO_TP_BUFFER ) {
@@ -6525,10 +6503,10 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnmark_reg_unknown (env, regs, value_regno); }else if (base_type (reg->type) ==PTR_TO_BTF_ID && !type_may_be_null (reg->type)) { - err =check_ptr_to_btf_access (env, regs, regno, off, size, t, + err =check_ptr_to_btf_access (env, regs, reg, regno, off, size, t, value_regno); }else if (reg->type ==CONST_PTR_TO_MAP ) { - err =check_ptr_to_map_access (env, regs, regno, off, size, t, + err =check_ptr_to_map_access (env, regs, reg, regno, off, size, t, value_regno); }else if (base_type (reg->type) ==PTR_TO_BUF && !type_may_be_null (reg->type)) {
@@ -6597,7 +6575,7 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,/* Check if (src_reg + off) is readable. The state of dst_reg will be * updated by this call. */ - err =check_mem_access (env, env->insn_idx, insn->src_reg, insn->off, + err =check_mem_access (env, env->insn_idx, regs + insn->src_reg, insn->src_reg, insn->off,BPF_SIZE (insn->code),BPF_READ , insn->dst_reg, strict_alignment_once, is_ldsx); err = err ?: save_aux_ptr_type(env, src_reg_type,
@@ -6627,7 +6605,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, dst_reg_type = regs[insn->dst_reg].type;/* Check if (dst_reg + off) is writeable. */ - err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off, + err =check_mem_access (env, env->insn_idx, regs + insn->dst_reg, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_WRITE , insn->src_reg, strict_alignment_once, false); err = err ?:save_aux_ptr_type (env, dst_reg_type, false);
@@ -6638,6 +6616,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,static int check_atomic_rmw (struct bpf_verifier_env *env,struct bpf_insn *insn) { +struct bpf_reg_state *dst_reg;int load_reg;int err;
@@ -6699,13 +6678,15 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, load_reg =-1 ; } + dst_reg =cur_regs (env) + insn->dst_reg; +/* Check whether we can read the memory, with second call for fetch * case to simulate the register fill. */ - err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off, + err =check_mem_access (env, env->insn_idx, dst_reg, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_READ ,-1 , true, false);if (!err && load_reg >=0 ) - err =check_mem_access (env, env->insn_idx, insn->dst_reg, + err =check_mem_access (env, env->insn_idx, dst_reg, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_READ , load_reg, true, false);if (err)
@@ -6717,7 +6698,7 @@ static int check_atomic_rmw(struct bpf_verifier_env *env,return err; }/* Check whether we can write into the same memory. */ - err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off, + err =check_mem_access (env, env->insn_idx, dst_reg, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_WRITE ,-1 , true, false);if (err)return err;
@@ -6806,11 +6787,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn)* read offsets are marked as read. */ static int check_stack_range_initialized ( -struct bpf_verifier_env *env,int regno,int off, +struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int off,int access_size,bool zero_size_allowed,enum bpf_access_type type,struct bpf_call_arg_meta *meta) { -struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_func_state *state =bpf_func (env, reg);int err, min_off, max_off, i, j, slot, spi;/* Some accesses can write anything into the stack, others are
@@ -6832,7 +6812,7 @@ static int check_stack_range_initialized(return -EACCES ; } - err =check_stack_access_within_bounds (env, regno, off, access_size, type); + err =check_stack_access_within_bounds (env, reg, regno, off, access_size, type);if (err)return err;
@@ -6963,7 +6943,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,switch (base_type (reg->type)) {case PTR_TO_PACKET :case PTR_TO_PACKET_META : -return check_packet_access (env, regno,0 , access_size, +return check_packet_access (env, reg, regno,0 , access_size, zero_size_allowed);case PTR_TO_MAP_KEY :if (access_type ==BPF_WRITE ) {
@@ -6971,12 +6951,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, reg_type_str(env, reg->type));return -EACCES ; } -return check_mem_region_access (env, regno,0 , access_size, +return check_mem_region_access (env, reg, regno,0 , access_size, reg->map_ptr->key_size, false);case PTR_TO_MAP_VALUE : -if (check_map_access_type (env, regno,0 , access_size, access_type)) +if (check_map_access_type (env, reg, regno,0 , access_size, access_type))return -EACCES ; -return check_map_access (env, regno,0 , access_size, +return check_map_access (env, reg, regno,0 , access_size, zero_size_allowed,ACCESS_HELPER );case PTR_TO_MEM :if (type_is_rdonly_mem (reg->type)) {
@@ -6986,7 +6966,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,return -EACCES ; } } -return check_mem_region_access (env, regno,0 , +return check_mem_region_access (env, reg, regno,0 , access_size, reg->mem_size, zero_size_allowed);case PTR_TO_BUF :
@@ -7006,16 +6986,16 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, max_access);case PTR_TO_STACK :return check_stack_range_initialized ( - env, + env, reg, regno,0 , access_size, zero_size_allowed, access_type, meta);case PTR_TO_BTF_ID : -return check_ptr_to_btf_access (env, regs, regno,0 , +return check_ptr_to_btf_access (env, regs, reg, regno,0 , access_size,BPF_READ ,-1 );case PTR_TO_CTX :/* Only permit reading or writing syscall context using helper calls. */ if (is_var_ctx_off_allowed (env->prog)) { -int err =check_mem_region_access (env, regno,0 , access_size,U16_MAX , +int err =check_mem_region_access (env, reg, regno,0 , access_size,U16_MAX , zero_size_allowed);if (err)return err;
@@ -7178,11 +7158,10 @@ enum {* env->cur_state->active_locks remembers which map value element or allocated * object got locked and clears it after bpf_spin_unlock. */ -static int process_spin_lock (struct bpf_verifier_env *env,int regno,int flags) +static int process_spin_lock (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int flags) {bool is_lock = flags &PROCESS_SPIN_LOCK , is_res_lock = flags &PROCESS_RES_LOCK ;const char *lock_str = is_res_lock ?"bpf_res_spin" :"bpf_spin" ; -struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_verifier_state *cur = env->cur_state;bool is_const =tnum_is_const (reg->var_off);bool is_irq = flags &PROCESS_LOCK_IRQ ;
@@ -7295,11 +7274,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) }/* Check if @regno is a pointer to a specific field in a map value */ -static int check_map_field_pointer (struct bpf_verifier_env *env,u32 regno, +static int check_map_field_pointer (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,enum btf_field_type field_type,struct bpf_map_desc *map_desc) { -struct bpf_reg_state *reg =reg_state (env, regno);bool is_const =tnum_is_const (reg->var_off);struct bpf_map *map = reg->map_ptr;u64 val = reg->var_off.value;
@@ -7349,26 +7327,26 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno,return 0 ; } -static int process_timer_func (struct bpf_verifier_env *env,int regno, +static int process_timer_func (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,struct bpf_map_desc *map) {if (IS_ENABLED (CONFIG_PREEMPT_RT )) {verbose (env,"bpf_timer cannot be used for PREEMPT_RT.\n" );return -EOPNOTSUPP ; } -return check_map_field_pointer (env, regno,BPF_TIMER , map); +return check_map_field_pointer (env, reg, regno,BPF_TIMER , map); } -static int process_timer_helper (struct bpf_verifier_env *env,int regno, +static int process_timer_helper (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,struct bpf_call_arg_meta *meta) { -return process_timer_func (env, regno, &meta->map); +return process_timer_func (env, reg, regno, &meta->map); } -static int process_timer_kfunc (struct bpf_verifier_env *env,int regno, +static int process_timer_kfunc (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,struct bpf_kfunc_call_arg_meta *meta) { -return process_timer_func (env, regno, &meta->map); +return process_timer_func (env, reg, regno, &meta->map); } static int process_kptr_func(struct bpf_verifier_env *env, int regno,
@@ -7444,10 +7422,9 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno,* Helpers which do not mutate the bpf_dynptr set MEM_RDONLY in their argument * type, and declare it as 'const struct bpf_dynptr *' in their prototype. */ -static int process_dynptr_func (struct bpf_verifier_env *env,int regno,int insn_idx, +static int process_dynptr_func (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int insn_idx,enum bpf_arg_type arg_type,int clone_ref_obj_id) { -struct bpf_reg_state *reg =reg_state (env, regno);int err;if (reg->type !=PTR_TO_STACK && reg->type !=CONST_PTR_TO_DYNPTR ) {
@@ -7490,7 +7467,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn/* we write BPF_DW bits (8 bytes) at a time */ for (i =0 ; i <BPF_DYNPTR_SIZE ; i +=8 ) { - err =check_mem_access (env, insn_idx, regno, + err =check_mem_access (env, insn_idx, reg, regno, i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err;
@@ -7560,10 +7537,9 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx,return btf_param_match_suffix (meta->btf, arg,"__iter" ); } -static int process_iter_arg (struct bpf_verifier_env *env,int regno,int insn_idx, +static int process_iter_arg (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int insn_idx,struct bpf_kfunc_call_arg_meta *meta) { -struct bpf_reg_state *reg =reg_state (env, regno);const struct btf_type *t;int spi, err, i, nr_slots, btf_id;
@@ -7595,7 +7571,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id }for (i =0 ; i < nr_slots *8 ; i +=BPF_REG_SIZE ) { - err =check_mem_access (env, insn_idx, regno, + err =check_mem_access (env, insn_idx, reg, regno, i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err;
@@ -8034,12 +8010,11 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_DYNPTR] = &dynptr_types, }; -static int check_reg_type (struct bpf_verifier_env *env,u32 regno, +static int check_reg_type (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,enum bpf_arg_type arg_type,const u32 *arg_btf_id,struct bpf_call_arg_meta *meta) { -struct bpf_reg_state *reg =reg_state (env, regno);enum bpf_reg_type expected, type = reg->type;const struct bpf_reg_types *compatible;int i, j, err;
@@ -8382,7 +8357,7 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return -EACCES ; } - err =check_map_access (env, regno,0 , + err =check_map_access (env, reg, regno,0 , map->value_size - reg->var_off.value, false,ACCESS_HELPER );if (err)
@@ -8518,7 +8493,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) arg_btf_id = fn->arg_btf_id[arg]; - err =check_reg_type (env, regno, arg_type, arg_btf_id, meta); + err =check_reg_type (env, reg, regno, arg_type, arg_btf_id, meta);if (err)return err;
@@ -8656,11 +8631,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,return -EACCES ; }if (meta->func_id == BPF_FUNC_spin_lock) { - err =process_spin_lock (env, regno,PROCESS_SPIN_LOCK ); + err =process_spin_lock (env, reg, regno,PROCESS_SPIN_LOCK );if (err)return err; }else if (meta->func_id == BPF_FUNC_spin_unlock) { - err =process_spin_lock (env, regno,0 ); + err =process_spin_lock (env, reg, regno,0 );if (err)return err; }else {
@@ -8669,7 +8644,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, }break ;case ARG_PTR_TO_TIMER : - err =process_timer_helper (env, regno, meta); + err =process_timer_helper (env, reg, regno, meta);if (err)return err;break ;
@@ -8704,7 +8679,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, true, meta);break ;case ARG_PTR_TO_DYNPTR : - err =process_dynptr_func (env, regno, insn_idx, arg_type,0 ); + err =process_dynptr_func (env, reg, regno, insn_idx, arg_type,0 );if (err)return err;break ;
@@ -9363,7 +9338,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,if (ret)return ret; - ret =process_dynptr_func (env, regno,-1 , arg->arg_type,0 ); + ret =process_dynptr_func (env, reg, regno,-1 , arg->arg_type,0 );if (ret)return ret; }else if (base_type (arg->arg_type) ==ARG_PTR_TO_BTF_ID ) {
@@ -9374,7 +9349,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,continue ;memset (&meta,0 ,sizeof (meta));/* leave func_id as zero */ - err =check_reg_type (env, regno, arg->arg_type, &arg->btf_id, &meta); + err =check_reg_type (env, reg, regno, arg->arg_type, &arg->btf_id, &meta); err = err ?:check_func_arg_reg_off (env, reg, regno, arg->arg_type);if (err)return err;
@@ -10332,18 +10307,18 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnif (err)return err; + regs =cur_regs (env); +/* Mark slots with STACK_MISC in case of raw mode, stack offset * is inferred from register state. */ for (i =0 ; i < meta.access_size; i++) { - err =check_mem_access (env, insn_idx, meta.regno, i,BPF_B , + err =check_mem_access (env, insn_idx, regs + meta.regno, meta.regno, i,BPF_B ,BPF_WRITE ,-1 , false, false);if (err)return err; } - regs =cur_regs (env); -if (meta.release_regno) { err = -EINVAL ;if (arg_type_is_dynptr (fn->arg_type[meta.release_regno -BPF_REG_1 ])) {
@@ -11347,11 +11322,10 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,struct bpf_kfunc_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, - int argno, int nargs) + int argno, int nargs, struct bpf_reg_state *reg) { u32 regno = argno + 1;struct bpf_reg_state *regs =cur_regs (env); -struct bpf_reg_state *reg = ®s[regno];bool arg_mem_size = false; if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11518,10 +11492,9 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,return 0 ; } -static int process_irq_flag (struct bpf_verifier_env *env,int regno, +static int process_irq_flag (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,struct bpf_kfunc_call_arg_meta *meta) { -struct bpf_reg_state *reg =reg_state (env, regno);int err, kfunc_class =IRQ_NATIVE_KFUNC ;bool irq_save;
@@ -11546,7 +11519,7 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno,return -EINVAL ; } - err =check_mem_access (env, env->insn_idx, regno,0 ,BPF_DW ,BPF_WRITE ,-1 , false, false); + err =check_mem_access (env, env->insn_idx, reg, regno,0 ,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err;
@@ -12134,7 +12107,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id); ref_tname =btf_name_by_offset (btf, ref_t->name_off); - kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs); + kf_arg_type =get_kfunc_ptr_arg_type (env, meta, t, ref_t, ref_tname, args, i, nargs, reg);if (kf_arg_type <0 )return kf_arg_type;
@@ -12299,7 +12272,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } } - ret =process_dynptr_func (env, regno, insn_idx, dynptr_arg_type, clone_ref_obj_id); + ret =process_dynptr_func (env, reg, regno, insn_idx, dynptr_arg_type, clone_ref_obj_id);if (ret <0 )return ret;
@@ -12324,7 +12297,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EINVAL ; } } - ret =process_iter_arg (env, regno, insn_idx, meta); + ret =process_iter_arg (env, reg, regno, insn_idx, meta);if (ret <0 )return ret;break ;
@@ -12501,7 +12474,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"arg#%d doesn't point to a map value\n" , i);return -EINVAL ; } - ret =check_map_field_pointer (env, regno,BPF_WORKQUEUE , &meta->map); + ret =check_map_field_pointer (env, reg, regno,BPF_WORKQUEUE , &meta->map);if (ret <0 )return ret;break ;
@@ -12510,7 +12483,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"arg#%d doesn't point to a map value\n" , i);return -EINVAL ; } - ret =process_timer_kfunc (env, regno, meta); + ret =process_timer_kfunc (env, reg, regno, meta);if (ret <0 )return ret;break ;
@@ -12519,7 +12492,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"arg#%d doesn't point to a map value\n" , i);return -EINVAL ; } - ret =check_map_field_pointer (env, regno,BPF_TASK_WORK , &meta->map); + ret =check_map_field_pointer (env, reg, regno,BPF_TASK_WORK , &meta->map);if (ret <0 )return ret;break ;
@@ -12528,7 +12501,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_verbose (env,"arg#%d doesn't point to an irq flag on stack\n" , i);return -EINVAL ; } - ret =process_irq_flag (env, regno, meta); + ret =process_irq_flag (env, reg, regno, meta);if (ret <0 )return ret;break ;
@@ -12549,7 +12522,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_if (meta->func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave] || meta->func_id == special_kfunc_list[KF_bpf_res_spin_unlock_irqrestore]) flags |=PROCESS_LOCK_IRQ ; - ret =process_spin_lock (env, regno, flags); + ret =process_spin_lock (env, reg, regno, flags);if (ret <0 )return ret;break ;
@@ -13683,7 +13656,7 @@ static int check_stack_access_for_ptr_arithmetic(static int sanitize_check_bounds (struct bpf_verifier_env *env,const struct bpf_insn *insn, -const struct bpf_reg_state *dst_reg) +struct bpf_reg_state *dst_reg) {u32 dst = insn->dst_reg;
@@ -13700,7 +13673,7 @@ static int sanitize_check_bounds(struct bpf_verifier_env *env,return -EACCES ;break ;case PTR_TO_MAP_VALUE : -if (check_map_access (env, dst,0 ,1 , false,ACCESS_HELPER )) { +if (check_map_access (env, dst_reg, dst,0 ,1 , false,ACCESS_HELPER )) {verbose (env,"R%d pointer arithmetic of map value goes out of range, " "prohibited for !root\n" , dst);return -EACCES ;
@@ -17584,7 +17557,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state) dst_reg_type =cur_regs (env)[insn->dst_reg].type; - err =check_mem_access (env, env->insn_idx, insn->dst_reg, + err =check_mem_access (env, env->insn_idx,cur_regs (env) + insn->dst_reg, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_WRITE ,-1 , false, false);if (err)
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 4/16 bpf: Prepare verifier logs for upcoming kfunc stack arguments
In reply to Yonghong SongThis change prepares verifier log reporting for upcoming kfunc stack argument support.
Today verifier log code mostly assumes that an argument can be described directly by a register number. That works for arguments passed in R1 to R5, but it does not work once kfunc arguments can also be passed on the stack.
Introduce an internal argno representation such that register-passed arguments keep using their real register numbers, while stack-passed arguments use an encoded value above a dedicated base. reg_arg_name() converts this representation into either R%d or *(R11-off) when emitting verifier logs. If a particular argno is corresponding to a stack argument, print *(R11-off). Otherwise, print R%d. Here R11 presents the base of stack arguments.
This keeps existing logs readable for register arguments and allows the same log sites to handle future stack arguments without open-coding special cases.
Update selftests accordingly.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 649 ++++++++++-------- .../testing/selftests/bpf/prog_tests/bpf_nf.c | 22 +- .../selftests/bpf/prog_tests/cb_refs.c | 2 +- .../selftests/bpf/prog_tests/kfunc_call.c | 2 +- .../selftests/bpf/prog_tests/linked_list.c | 4 +- .../selftests/bpf/progs/cgrp_kfunc_failure.c | 14 +- .../selftests/bpf/progs/cpumask_failure.c | 10 +- .../testing/selftests/bpf/progs/dynptr_fail.c | 22 +- .../selftests/bpf/progs/file_reader_fail.c | 4 +- tools/testing/selftests/bpf/progs/irq.c | 4 +- tools/testing/selftests/bpf/progs/iters.c | 6 +- .../selftests/bpf/progs/iters_state_safety.c | 14 +- .../selftests/bpf/progs/iters_testmod.c | 4 +- .../selftests/bpf/progs/iters_testmod_seq.c | 4 +- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/percpu_alloc_fail.c | 4 +- .../testing/selftests/bpf/progs/rbtree_fail.c | 6 +- .../bpf/progs/refcounted_kptr_fail.c | 2 +- .../testing/selftests/bpf/progs/stream_fail.c | 2 +- .../selftests/bpf/progs/task_kfunc_failure.c | 18 +- .../selftests/bpf/progs/task_work_fail.c | 6 +- .../selftests/bpf/progs/test_bpf_nf_fail.c | 8 +- .../bpf/progs/test_kfunc_dynptr_param.c | 2 +- .../bpf/progs/test_kfunc_param_nullable.c | 2 +- .../selftests/bpf/progs/verifier_bits_iter.c | 4 +- .../bpf/progs/verifier_ref_tracking.c | 6 +- .../selftests/bpf/progs/verifier_vfs_reject.c | 8 +- .../testing/selftests/bpf/progs/wq_failures.c | 2 +- tools/testing/selftests/bpf/verifier/calls.c | 14 +- 30 files changed, 474 insertions(+), 374 deletions(-)
include/linux/bpf_verifier.h +1 -0
@@ -912,6 +912,7 @@ struct bpf_verifier_env {* e.g., in reg_type_str() to generate reg_type string */ char tmp_str_buf[TMP_STR_BUF_LEN ]; +char tmp_reg_arg_name_buf[32 ];struct bpf_insn insn_buf[INSN_BUF_SIZE ];struct bpf_insn epilogue_buf[INSN_BUF_SIZE ];struct bpf_scc_callchain callchain_buf;
kernel/bpf/verifier.c +374 -275
1,731 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/bpf_nf.c +11 -11
@@ -11,18 +11,18 @@ struct {const char *prog_name;const char *err_msg; } test_bpf_nf_fail_tests[] = { - { "alloc_release", "kernel function bpf_ct_release args#0 expected pointer to STRUCT nf_conn but" }, - { "insert_insert", "kernel function bpf_ct_insert_entry args#0 expected pointer to STRUCT nf_conn___init but" }, - { "lookup_insert", "kernel function bpf_ct_insert_entry args#0 expected pointer to STRUCT nf_conn___init but" }, - { "set_timeout_after_insert", "kernel function bpf_ct_set_timeout args#0 expected pointer to STRUCT nf_conn___init but" }, - { "set_status_after_insert", "kernel function bpf_ct_set_status args#0 expected pointer to STRUCT nf_conn___init but" }, - { "change_timeout_after_alloc", "kernel function bpf_ct_change_timeout args#0 expected pointer to STRUCT nf_conn but" }, - { "change_status_after_alloc", "kernel function bpf_ct_change_status args#0 expected pointer to STRUCT nf_conn but" }, + { "alloc_release", "kernel function bpf_ct_release R1 expected pointer to STRUCT nf_conn but" }, + { "insert_insert", "kernel function bpf_ct_insert_entry R1 expected pointer to STRUCT nf_conn___init but" }, + { "lookup_insert", "kernel function bpf_ct_insert_entry R1 expected pointer to STRUCT nf_conn___init but" }, + { "set_timeout_after_insert", "kernel function bpf_ct_set_timeout R1 expected pointer to STRUCT nf_conn___init but" }, + { "set_status_after_insert", "kernel function bpf_ct_set_status R1 expected pointer to STRUCT nf_conn___init but" }, + { "change_timeout_after_alloc", "kernel function bpf_ct_change_timeout R1 expected pointer to STRUCT nf_conn but" }, + { "change_status_after_alloc", "kernel function bpf_ct_change_status R1 expected pointer to STRUCT nf_conn but" }, { "write_not_allowlisted_field", "no write support to nf_conn at off" }, - { "lookup_null_bpf_tuple", "Possibly NULL pointer passed to trusted arg1" }, - { "lookup_null_bpf_opts", "Possibly NULL pointer passed to trusted arg3" }, - { "xdp_lookup_null_bpf_tuple", "Possibly NULL pointer passed to trusted arg1" }, - { "xdp_lookup_null_bpf_opts", "Possibly NULL pointer passed to trusted arg3" }, + { "lookup_null_bpf_tuple", "Possibly NULL pointer passed to trusted R2" }, + { "lookup_null_bpf_opts", "Possibly NULL pointer passed to trusted R4" }, + { "xdp_lookup_null_bpf_tuple", "Possibly NULL pointer passed to trusted R2" }, + { "xdp_lookup_null_bpf_opts", "Possibly NULL pointer passed to trusted R4" }, }; enum {
tools/testing/selftests/bpf/prog_tests/cb_refs.c +1 -1
@@ -12,7 +12,7 @@ struct {const char *err_msg; } cb_refs_tests[] = { { "underflow_prog", "must point to scalar, or struct with scalar" }, - { "leak_prog", "Possibly NULL pointer passed to helper arg2" }, + { "leak_prog", "Possibly NULL pointer passed to helper R2" }, { "nested_cb", "Unreleased reference id=4 alloc_insn=2" }, /* alloc_insn=2{4,5} */ { "non_cb_transfer_ref", "Unreleased reference id=4 alloc_insn=1" }, /* alloc_insn=1{1,2} */ };
tools/testing/selftests/bpf/prog_tests/kfunc_call.c +1 -1
@@ -68,7 +68,7 @@ static struct kfunc_test_params kfunc_tests[] = {TC_FAIL (kfunc_call_test_get_mem_fail_oob,0 ,"min value is outside of the allowed memory range" ),TC_FAIL (kfunc_call_test_get_mem_fail_not_const,0 ,"is not a const" ),TC_FAIL (kfunc_call_test_mem_acquire_fail,0 ,"acquire kernel function does not return PTR_TO_BTF_ID" ), -TC_FAIL (kfunc_call_test_pointer_arg_type_mismatch,0 ,"arg#0 expected pointer to ctx, but got scalar" ), +TC_FAIL (kfunc_call_test_pointer_arg_type_mismatch,0 ,"R1 expected pointer to ctx, but got scalar" ),/* success cases */ TC_TEST(kfunc_call_test1, 12),
tools/testing/selftests/bpf/prog_tests/linked_list.c +2 -2
@@ -81,8 +81,8 @@ static struct { { "direct_write_node", "direct access to bpf_list_node is disallowed" }, { "use_after_unlock_push_front", "invalid mem access 'scalar'" }, { "use_after_unlock_push_back", "invalid mem access 'scalar'" }, - { "double_push_front", "arg#1 expected pointer to allocated object" }, - { "double_push_back", "arg#1 expected pointer to allocated object" }, + { "double_push_front", "R2 expected pointer to allocated object" }, + { "double_push_back", "R2 expected pointer to allocated object" }, { "no_node_value_type", "bpf_list_node not found at offset=0" }, { "incorrect_value_type","operation on bpf_list_head expects arg#1 bpf_list_node at offset=48 in struct foo, "
tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c +7 -7
@@ -29,7 +29,7 @@ static struct __cgrps_kfunc_map_value *insert_lookup_cgrp(struct cgroup *cgrp) } SEC("tp_btf/cgroup_mkdir") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(cgrp_kfunc_acquire_untrusted, struct cgroup *cgrp, const char *path) {struct cgroup *acquired;
@@ -48,7 +48,7 @@ int BPF_PROG(cgrp_kfunc_acquire_untrusted, struct cgroup *cgrp, const char *path } SEC("tp_btf/cgroup_mkdir") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(cgrp_kfunc_acquire_no_null_check, struct cgroup *cgrp, const char *path) {struct cgroup *acquired;
@@ -64,7 +64,7 @@ int BPF_PROG(cgrp_kfunc_acquire_no_null_check, struct cgroup *cgrp, const char * } SEC("tp_btf/cgroup_mkdir") -__failure __msg("arg#0 pointer type STRUCT cgroup must point") +__failure __msg("R1 pointer type STRUCT cgroup must point") int BPF_PROG(cgrp_kfunc_acquire_fp, struct cgroup *cgrp, const char *path) {struct cgroup *acquired, *stack_cgrp = (struct cgroup *)&path;
@@ -106,7 +106,7 @@ int BPF_PROG(cgrp_kfunc_acquire_trusted_walked, struct cgroup *cgrp, const char } SEC("tp_btf/cgroup_mkdir") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(cgrp_kfunc_acquire_null, struct cgroup *cgrp, const char *path) {struct cgroup *acquired;
@@ -175,7 +175,7 @@ int BPF_PROG(cgrp_kfunc_rcu_get_release, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(cgrp_kfunc_release_untrusted, struct cgroup *cgrp, const char *path) {struct __cgrps_kfunc_map_value *v;
@@ -191,7 +191,7 @@ int BPF_PROG(cgrp_kfunc_release_untrusted, struct cgroup *cgrp, const char *path } SEC("tp_btf/cgroup_mkdir") -__failure __msg("arg#0 pointer type STRUCT cgroup must point") +__failure __msg("R1 pointer type STRUCT cgroup must point") int BPF_PROG(cgrp_kfunc_release_fp, struct cgroup *cgrp, const char *path) {struct cgroup *acquired = (struct cgroup *)&path;
@@ -203,7 +203,7 @@ int BPF_PROG(cgrp_kfunc_release_fp, struct cgroup *cgrp, const char *path) } SEC("tp_btf/cgroup_mkdir") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(cgrp_kfunc_release_null, struct cgroup *cgrp, const char *path) {struct __cgrps_kfunc_map_value local, *v;
tools/testing/selftests/bpf/progs/cpumask_failure.c +5 -5
@@ -45,7 +45,7 @@ int BPF_PROG(test_alloc_no_release, struct task_struct *task, u64 clone_flags) } SEC("tp_btf/task_newtask") -__failure __msg("NULL pointer passed to trusted arg0") +__failure __msg("NULL pointer passed to trusted R1") int BPF_PROG(test_alloc_double_release, struct task_struct *task, u64 clone_flags) {struct bpf_cpumask *cpumask;
@@ -73,7 +73,7 @@ int BPF_PROG(test_acquire_wrong_cpumask, struct task_struct *task, u64 clone_fla } SEC("tp_btf/task_newtask") -__failure __msg("bpf_cpumask_set_cpu args#1 expected pointer to STRUCT bpf_cpumask") +__failure __msg("bpf_cpumask_set_cpu R2 expected pointer to STRUCT bpf_cpumask") int BPF_PROG(test_mutate_cpumask, struct task_struct *task, u64 clone_flags) {/* Can't set the CPU of a non-struct bpf_cpumask. */
@@ -107,7 +107,7 @@ int BPF_PROG(test_insert_remove_no_release, struct task_struct *task, u64 clone_ } SEC("tp_btf/task_newtask") -__failure __msg("NULL pointer passed to trusted arg0") +__failure __msg("NULL pointer passed to trusted R1") int BPF_PROG(test_cpumask_null, struct task_struct *task, u64 clone_flags) {/* NULL passed to kfunc. */
@@ -151,7 +151,7 @@ int BPF_PROG(test_global_mask_out_of_rcu, struct task_struct *task, u64 clone_fl } SEC("tp_btf/task_newtask") -__failure __msg("NULL pointer passed to trusted arg1") +__failure __msg("NULL pointer passed to trusted R2") int BPF_PROG(test_global_mask_no_null_check, struct task_struct *task, u64 clone_flags) {struct bpf_cpumask *local, *prev;
@@ -179,7 +179,7 @@ int BPF_PROG(test_global_mask_no_null_check, struct task_struct *task, u64 clone } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to helper arg2") +__failure __msg("Possibly NULL pointer passed to helper R2") int BPF_PROG(test_global_mask_rcu_no_null_check, struct task_struct *task, u64 clone_flags) {struct bpf_cpumask *prev, *curr;
tools/testing/selftests/bpf/progs/dynptr_fail.c +11 -11
@@ -149,7 +149,7 @@ int ringbuf_release_uninit_dynptr(void *ctx)/* A dynptr can't be used after it has been invalidated */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #2") +__failure __msg("Expected an initialized dynptr as R3")int use_after_invalid (void *ctx) {struct bpf_dynptr ptr;
@@ -448,7 +448,7 @@ int invalid_helper2(void *ctx)/* A bpf_dynptr is invalidated if it's been written into */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int invalid_write1 (void *ctx) {struct bpf_dynptr ptr;
@@ -1642,7 +1642,7 @@ int invalid_slice_rdwr_rdonly(struct __sk_buff *skb)/* bpf_dynptr_adjust can only be called on initialized dynptrs */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int dynptr_adjust_invalid (void *ctx) {struct bpf_dynptr ptr = {};
@@ -1655,7 +1655,7 @@ int dynptr_adjust_invalid(void *ctx)/* bpf_dynptr_is_null can only be called on initialized dynptrs */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int dynptr_is_null_invalid (void *ctx) {struct bpf_dynptr ptr = {};
@@ -1668,7 +1668,7 @@ int dynptr_is_null_invalid(void *ctx)/* bpf_dynptr_is_rdonly can only be called on initialized dynptrs */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int dynptr_is_rdonly_invalid (void *ctx) {struct bpf_dynptr ptr = {};
@@ -1681,7 +1681,7 @@ int dynptr_is_rdonly_invalid(void *ctx)/* bpf_dynptr_size can only be called on initialized dynptrs */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int dynptr_size_invalid (void *ctx) {struct bpf_dynptr ptr = {};
@@ -1694,7 +1694,7 @@ int dynptr_size_invalid(void *ctx)/* Only initialized dynptrs can be cloned */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #0") +__failure __msg("Expected an initialized dynptr as R1")int clone_invalid1 (void *ctx) {struct bpf_dynptr ptr1 = {};
@@ -1728,7 +1728,7 @@ int clone_invalid2(struct xdp_md *xdp)/* Invalidating a dynptr should invalidate its clones */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #2") +__failure __msg("Expected an initialized dynptr as R3")int clone_invalidate1 (void *ctx) {struct bpf_dynptr clone;
@@ -1749,7 +1749,7 @@ int clone_invalidate1(void *ctx)/* Invalidating a dynptr should invalidate its parent */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #2") +__failure __msg("Expected an initialized dynptr as R3")int clone_invalidate2 (void *ctx) {struct bpf_dynptr ptr;
@@ -1770,7 +1770,7 @@ int clone_invalidate2(void *ctx)/* Invalidating a dynptr should invalidate its siblings */ SEC("?raw_tp") -__failure __msg("Expected an initialized dynptr as arg #2") +__failure __msg("Expected an initialized dynptr as R3")int clone_invalidate3 (void *ctx) {struct bpf_dynptr ptr;
@@ -1981,7 +1981,7 @@ __noinline long global_call_bpf_dynptr(const struct bpf_dynptr *dynptr) } SEC("?raw_tp") -__failure __msg("arg#0 expected pointer to stack or const struct bpf_dynptr") +__failure __msg("R1 expected pointer to stack or const struct bpf_dynptr") int test_dynptr_reg_type(void *ctx) {struct task_struct *current =NULL ;
tools/testing/selftests/bpf/progs/file_reader_fail.c +2 -2
@@ -30,7 +30,7 @@ int on_nanosleep_unreleased_ref(void *ctx) SEC("xdp") __failure -__msg("Expected a dynptr of type file as arg #0") +__msg("Expected a dynptr of type file as R1")int xdp_wrong_dynptr_type (struct xdp_md *xdp) {struct bpf_dynptr dynptr;
@@ -42,7 +42,7 @@ int xdp_wrong_dynptr_type(struct xdp_md *xdp) SEC("xdp") __failure -__msg("Expected an initialized dynptr as arg #0") +__msg("Expected an initialized dynptr as R1")int xdp_no_dynptr_type (struct xdp_md *xdp) {struct bpf_dynptr dynptr;
tools/testing/selftests/bpf/progs/irq.c +2 -2
@@ -15,7 +15,7 @@ struct bpf_res_spin_lock lockA __hidden SEC(".data.A"); struct bpf_res_spin_lock lockB __hidden SEC(".data.B"); SEC("?tc") -__failure __msg("arg#0 doesn't point to an irq flag on stack") +__failure __msg("R1 doesn't point to an irq flag on stack") int irq_save_bad_arg(struct __sk_buff *ctx) {bpf_local_irq_save (&global_flags);
@@ -23,7 +23,7 @@ int irq_save_bad_arg(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("arg#0 doesn't point to an irq flag on stack") +__failure __msg("R1 doesn't point to an irq flag on stack") int irq_restore_bad_arg(struct __sk_buff *ctx) {bpf_local_irq_restore (&global_flags);
tools/testing/selftests/bpf/progs/iters.c +3 -3
@@ -1605,7 +1605,7 @@ int iter_subprog_check_stacksafe(const void *ctx)struct bpf_iter_num global_it; SEC("raw_tp") -__failure __msg("arg#0 expected pointer to an iterator on stack") +__failure __msg("R1 expected pointer to an iterator on stack") int iter_new_bad_arg(const void *ctx) {bpf_iter_num_new (&global_it,0 ,1 );
@@ -1613,7 +1613,7 @@ int iter_new_bad_arg(const void *ctx) } SEC("raw_tp") -__failure __msg("arg#0 expected pointer to an iterator on stack") +__failure __msg("R1 expected pointer to an iterator on stack") int iter_next_bad_arg(const void *ctx) {bpf_iter_num_next (&global_it);
@@ -1621,7 +1621,7 @@ int iter_next_bad_arg(const void *ctx) } SEC("raw_tp") -__failure __msg("arg#0 expected pointer to an iterator on stack") +__failure __msg("R1 expected pointer to an iterator on stack") int iter_destroy_bad_arg(const void *ctx) {bpf_iter_num_destroy (&global_it);
tools/testing/selftests/bpf/progs/iters_state_safety.c +7 -7
@@ -73,7 +73,7 @@ int create_and_forget_to_destroy_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int destroy_without_creating_fail(void *ctx) {/* init with zeros to stop verifier complaining about uninit stack */
@@ -91,7 +91,7 @@ int destroy_without_creating_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int compromise_iter_w_direct_write_fail(void *ctx) {struct bpf_iter_num iter;
@@ -143,7 +143,7 @@ int compromise_iter_w_direct_write_and_skip_destroy_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int compromise_iter_w_helper_write_fail(void *ctx) {struct bpf_iter_num iter;
@@ -230,7 +230,7 @@ int valid_stack_reuse(void *ctx) } SEC("?raw_tp") -__failure __msg("expected uninitialized iter_num as arg #0") +__failure __msg("expected uninitialized iter_num as R1") int double_create_fail(void *ctx) {struct bpf_iter_num iter;
@@ -258,7 +258,7 @@ int double_create_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int double_destroy_fail(void *ctx) {struct bpf_iter_num iter;
@@ -284,7 +284,7 @@ int double_destroy_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int next_without_new_fail(void *ctx) {struct bpf_iter_num iter;
@@ -305,7 +305,7 @@ int next_without_new_fail(void *ctx) } SEC("?raw_tp") -__failure __msg("expected an initialized iter_num as arg #0") +__failure __msg("expected an initialized iter_num as R1") int next_after_destroy_fail(void *ctx) {struct bpf_iter_num iter;
tools/testing/selftests/bpf/progs/iters_testmod.c +2 -2
@@ -29,7 +29,7 @@ int iter_next_trusted(const void *ctx) } SEC("raw_tp/sys_enter") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int iter_next_trusted_or_null(const void *ctx) {struct task_struct *cur_task =bpf_get_current_task_btf ();
@@ -67,7 +67,7 @@ int iter_next_rcu(const void *ctx) } SEC("raw_tp/sys_enter") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int iter_next_rcu_or_null(const void *ctx) {struct task_struct *cur_task =bpf_get_current_task_btf ();
tools/testing/selftests/bpf/progs/iters_testmod_seq.c +2 -2
@@ -79,7 +79,7 @@ int testmod_seq_truncated(const void *ctx) SEC("?raw_tp") __failure -__msg("expected an initialized iter_testmod_seq as arg #1") +__msg("expected an initialized iter_testmod_seq as R2")int testmod_seq_getter_before_bad (const void *ctx) {struct bpf_iter_testmod_seq it;
@@ -89,7 +89,7 @@ int testmod_seq_getter_before_bad(const void *ctx) SEC("?raw_tp") __failure -__msg("expected an initialized iter_testmod_seq as arg #1") +__msg("expected an initialized iter_testmod_seq as R2")int testmod_seq_getter_after_bad (const void *ctx) {struct bpf_iter_testmod_seq it;
tools/testing/selftests/bpf/progs/map_kptr_fail.c +1 -1
@@ -364,7 +364,7 @@ int kptr_xchg_ref_state(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Possibly NULL pointer passed to helper arg2") +__failure __msg("Possibly NULL pointer passed to helper R2") int kptr_xchg_possibly_null(struct __sk_buff *ctx) {struct prog_test_ref_kfunc *p;
tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +2 -2
@@ -110,7 +110,7 @@ int BPF_PROG(test_array_map_3) } SEC("?fentry.s/bpf_fentry_test1") -__failure __msg("arg#0 expected for bpf_percpu_obj_drop()") +__failure __msg("R1 expected for bpf_percpu_obj_drop()") int BPF_PROG(test_array_map_4) { struct val_t __percpu_kptr *p;
@@ -124,7 +124,7 @@ int BPF_PROG(test_array_map_4) } SEC("?fentry.s/bpf_fentry_test1") -__failure __msg("arg#0 expected for bpf_obj_drop()") +__failure __msg("R1 expected for bpf_obj_drop()") int BPF_PROG(test_array_map_5) {struct val_t *p;
tools/testing/selftests/bpf/progs/rbtree_fail.c +3 -3
@@ -134,7 +134,7 @@ long rbtree_api_remove_no_drop(void *ctx) } SEC("?tc") -__failure __msg("arg#1 expected pointer to allocated object") +__failure __msg("R2 expected pointer to allocated object") long rbtree_api_add_to_multiple_trees(void *ctx) {struct node_data *n;
@@ -153,7 +153,7 @@ long rbtree_api_add_to_multiple_trees(void *ctx) } SEC("?tc") -__failure __msg("Possibly NULL pointer passed to trusted arg1") +__failure __msg("Possibly NULL pointer passed to trusted R2") long rbtree_api_use_unchecked_remove_retval(void *ctx) {struct bpf_rb_node *res;
@@ -281,7 +281,7 @@ long add_with_cb(bool (cb)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) } SEC("?tc") -__failure __msg("arg#1 expected pointer to allocated object") +__failure __msg("R2 expected pointer to allocated object") long rbtree_api_add_bad_cb_bad_fn_call_add(void *ctx) {return add_with_cb (less__bad_fn_call_add);
tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +1 -1
@@ -54,7 +54,7 @@ long rbtree_refcounted_node_ref_escapes(void *ctx) } SEC("?tc") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") long refcount_acquire_maybe_null(void *ctx) {struct node_acquire *n, *m;
tools/testing/selftests/bpf/progs/stream_fail.c +1 -1
@@ -23,7 +23,7 @@ int stream_vprintk_scalar_arg(void *ctx) } SEC("syscall") -__failure __msg("arg#1 doesn't point to a const string") +__failure __msg("R2 doesn't point to a const string") int stream_vprintk_string_arg(void *ctx) {bpf_stream_vprintk (BPF_STDOUT , ctx,NULL ,0 );
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +9 -9
@@ -28,7 +28,7 @@ static struct __tasks_kfunc_map_value *insert_lookup_task(struct task_struct *ta } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_acquire_untrusted, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired;
@@ -49,7 +49,7 @@ int BPF_PROG(task_kfunc_acquire_untrusted, struct task_struct *task, u64 clone_f } SEC("tp_btf/task_newtask") -__failure __msg("arg#0 pointer type STRUCT task_struct must point") +__failure __msg("R1 pointer type STRUCT task_struct must point") int BPF_PROG(task_kfunc_acquire_fp, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired, *stack_task = (struct task_struct *)&clone_flags;
@@ -100,7 +100,7 @@ int BPF_PROG(task_kfunc_acquire_unsafe_kretprobe_rcu, struct task_struct *task, } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_acquire_null, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired;
@@ -149,7 +149,7 @@ int BPF_PROG(task_kfunc_xchg_unreleased, struct task_struct *task, u64 clone_fla } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_acquire_release_no_null_check, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired;
@@ -162,7 +162,7 @@ int BPF_PROG(task_kfunc_acquire_release_no_null_check, struct task_struct *task, } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_release_untrusted, struct task_struct *task, u64 clone_flags) {struct __tasks_kfunc_map_value *v;
@@ -178,7 +178,7 @@ int BPF_PROG(task_kfunc_release_untrusted, struct task_struct *task, u64 clone_f } SEC("tp_btf/task_newtask") -__failure __msg("arg#0 pointer type STRUCT task_struct must point") +__failure __msg("R1 pointer type STRUCT task_struct must point") int BPF_PROG(task_kfunc_release_fp, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired = (struct task_struct *)&clone_flags;
@@ -190,7 +190,7 @@ int BPF_PROG(task_kfunc_release_fp, struct task_struct *task, u64 clone_flags) } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_release_null, struct task_struct *task, u64 clone_flags) {struct __tasks_kfunc_map_value local, *v;
@@ -234,7 +234,7 @@ int BPF_PROG(task_kfunc_release_unacquired, struct task_struct *task, u64 clone_ } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_from_pid_no_null_check, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired;
@@ -248,7 +248,7 @@ int BPF_PROG(task_kfunc_from_pid_no_null_check, struct task_struct *task, u64 cl } SEC("tp_btf/task_newtask") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(task_kfunc_from_vpid_no_null_check, struct task_struct *task, u64 clone_flags) {struct task_struct *acquired;
tools/testing/selftests/bpf/progs/task_work_fail.c +3 -3
@@ -58,7 +58,7 @@ int mismatch_map(struct pt_regs *args) } SEC("perf_event") -__failure __msg("arg#1 doesn't point to a map value") +__failure __msg("R2 doesn't point to a map value") int no_map_task_work(struct pt_regs *args) {struct task_struct *task;
@@ -70,7 +70,7 @@ int no_map_task_work(struct pt_regs *args) } SEC("perf_event") -__failure __msg("Possibly NULL pointer passed to trusted arg1") +__failure __msg("Possibly NULL pointer passed to trusted R2") int task_work_null(struct pt_regs *args) {struct task_struct *task;
@@ -81,7 +81,7 @@ int task_work_null(struct pt_regs *args) } SEC("perf_event") -__failure __msg("Possibly NULL pointer passed to trusted arg2") +__failure __msg("Possibly NULL pointer passed to trusted R3") int map_null(struct pt_regs *args) {struct elem *work;
tools/testing/selftests/bpf/progs/test_bpf_nf_fail.c +4 -4
@@ -152,7 +152,7 @@ int change_status_after_alloc(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Possibly NULL pointer passed to trusted arg1") +__failure __msg("Possibly NULL pointer passed to trusted R2") int lookup_null_bpf_tuple(struct __sk_buff *ctx) {struct bpf_ct_opts___local opts = {};
@@ -165,7 +165,7 @@ int lookup_null_bpf_tuple(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Possibly NULL pointer passed to trusted arg3") +__failure __msg("Possibly NULL pointer passed to trusted R4") int lookup_null_bpf_opts(struct __sk_buff *ctx) {struct bpf_sock_tuple tup = {};
@@ -178,7 +178,7 @@ int lookup_null_bpf_opts(struct __sk_buff *ctx) } SEC("?xdp") -__failure __msg("Possibly NULL pointer passed to trusted arg1") +__failure __msg("Possibly NULL pointer passed to trusted R2") int xdp_lookup_null_bpf_tuple(struct xdp_md *ctx) {struct bpf_ct_opts___local opts = {};
@@ -191,7 +191,7 @@ int xdp_lookup_null_bpf_tuple(struct xdp_md *ctx) } SEC("?xdp") -__failure __msg("Possibly NULL pointer passed to trusted arg3") +__failure __msg("Possibly NULL pointer passed to trusted R4") int xdp_lookup_null_bpf_opts(struct xdp_md *ctx) {struct bpf_sock_tuple tup = {};
tools/testing/selftests/bpf/progs/test_kfunc_dynptr_param.c +1 -1
@@ -45,7 +45,7 @@ int BPF_PROG(not_valid_dynptr, int cmd, union bpf_attr *attr, unsigned int size, } SEC("?lsm.s/bpf") -__failure __msg("arg#0 expected pointer to stack or const struct bpf_dynptr") +__failure __msg("R1 expected pointer to stack or const struct bpf_dynptr") int BPF_PROG(not_ptr_to_stack, int cmd, union bpf_attr *attr, unsigned int size, bool kernel) {static struct bpf_dynptr val;
tools/testing/selftests/bpf/progs/test_kfunc_param_nullable.c +1 -1
@@ -29,7 +29,7 @@ int kfunc_dynptr_nullable_test2(struct __sk_buff *skb) } SEC("tc") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int kfunc_dynptr_nullable_test3(struct __sk_buff *skb) {struct bpf_dynptr data;
tools/testing/selftests/bpf/progs/verifier_bits_iter.c +2 -2
@@ -32,7 +32,7 @@ int BPF_PROG(no_destroy, struct bpf_iter_meta *meta, struct cgroup *cgrp) SEC("iter/cgroup") __description("uninitialized iter in ->next()") -__failure __msg("expected an initialized iter_bits as arg #0") +__failure __msg("expected an initialized iter_bits as R1") int BPF_PROG(next_uninit, struct bpf_iter_meta *meta, struct cgroup *cgrp) {struct bpf_iter_bits it = {};
@@ -43,7 +43,7 @@ int BPF_PROG(next_uninit, struct bpf_iter_meta *meta, struct cgroup *cgrp) SEC("iter/cgroup") __description("uninitialized iter in ->destroy()") -__failure __msg("expected an initialized iter_bits as arg #0") +__failure __msg("expected an initialized iter_bits as R1") int BPF_PROG(destroy_uninit, struct bpf_iter_meta *meta, struct cgroup *cgrp) {struct bpf_iter_bits it = {};
tools/testing/selftests/bpf/progs/verifier_ref_tracking.c +3 -3
@@ -263,7 +263,7 @@ l0_%=: r0 = 0; \ SEC("lsm.s/bpf") __description("reference tracking: release user key reference without check") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") __naked void user_key_reference_without_check(void) { asm volatile (" \
@@ -282,7 +282,7 @@ __naked void user_key_reference_without_check(void) SEC("lsm.s/bpf") __description("reference tracking: release system key reference without check") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") __naked void system_key_reference_without_check(void) { asm volatile (" \
@@ -300,7 +300,7 @@ __naked void system_key_reference_without_check(void) SEC("lsm.s/bpf") __description("reference tracking: release with NULL key pointer") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") __naked void release_with_null_key_pointer(void) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_vfs_reject.c +4 -4
@@ -13,7 +13,7 @@static char buf[PATH_MAX ]; SEC("lsm.s/file_open") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(get_task_exe_file_kfunc_null) {struct file *acquired;
@@ -28,7 +28,7 @@ int BPF_PROG(get_task_exe_file_kfunc_null) } SEC("lsm.s/inode_getxattr") -__failure __msg("arg#0 pointer type STRUCT task_struct must point to scalar, or struct with scalar") +__failure __msg("R1 pointer type STRUCT task_struct must point to scalar, or struct with scalar") int BPF_PROG(get_task_exe_file_kfunc_fp) {u64 x;
@@ -89,7 +89,7 @@ int BPF_PROG(put_file_kfunc_unacquired, struct file *file) } SEC("lsm.s/file_open") -__failure __msg("Possibly NULL pointer passed to trusted arg0") +__failure __msg("Possibly NULL pointer passed to trusted R1") int BPF_PROG(path_d_path_kfunc_null) {/* Can't pass NULL value to bpf_path_d_path() kfunc. */
@@ -128,7 +128,7 @@ int BPF_PROG(path_d_path_kfunc_untrusted_from_current) } SEC("lsm.s/file_open") -__failure __msg("kernel function bpf_path_d_path args#0 expected pointer to STRUCT path but R1 has a pointer to STRUCT file") +__failure __msg("kernel function bpf_path_d_path R1 expected pointer to STRUCT path but R1 has a pointer to STRUCT file") int BPF_PROG(path_d_path_kfunc_type_mismatch, struct file *file) {bpf_path_d_path ((struct path *)&file->f_task_work, buf,sizeof (buf));
tools/testing/selftests/bpf/progs/wq_failures.c +1 -1
@@ -98,7 +98,7 @@ __failure* is a correct bpf_wq pointer. */ __msg(": (85) call bpf_wq_set_callback#") /* anchor message */ -__msg("arg#0 doesn't point to a map value") +__msg("R1 doesn't point to a map value")long test_wrong_wq_pointer (void *ctx) {int key =0 ;
tools/testing/selftests/bpf/verifier/calls.c +7 -7
@@ -31,7 +31,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "arg#0 pointer type STRUCT prog_test_fail1 must point to scalar", + .errstr = "R1 pointer type STRUCT prog_test_fail1 must point to scalar", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail1", 2 }, },
@@ -46,7 +46,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "max struct nesting depth exceeded\narg#0 pointer type STRUCT prog_test_fail2", + .errstr = "max struct nesting depth exceeded\nR1 pointer type STRUCT prog_test_fail2", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail2", 2 }, },
@@ -61,7 +61,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "arg#0 pointer type STRUCT prog_test_fail3 must point to scalar", + .errstr = "R1 pointer type STRUCT prog_test_fail3 must point to scalar", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail3", 2 }, },
@@ -76,7 +76,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "arg#0 expected pointer to ctx, but got fp", + .errstr = "R1 expected pointer to ctx, but got fp", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_pass_ctx", 2 }, },
@@ -91,7 +91,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "arg#0 pointer type UNKNOWN must point to scalar", + .errstr = "R1 pointer type UNKNOWN must point to scalar", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_mem_len_fail1", 2 }, },
@@ -109,7 +109,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "Possibly NULL pointer passed to trusted arg0", + .errstr = "Possibly NULL pointer passed to trusted R1", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_acquire", 3 }, { "bpf_kfunc_call_test_release", 5 },
@@ -152,7 +152,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "kernel function bpf_kfunc_call_memb1_release args#0 expected pointer", + .errstr = "kernel function bpf_kfunc_call_memb1_release R1 expected pointer", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_memb_acquire", 1 }, { "bpf_kfunc_call_memb1_release", 5 },
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 5/16 bpf: Introduce bpf register BPF_REG_PARAMS
In reply to Yonghong SongThe newly-added register BPF_REG_PARAMS corresponds to bpf register R11 in llvm. R11 is used as the base for stack arguments so it won't mess out R10 based stacks. The kernel internal register BPF_REG_AX was R11 previously. With this change, BPF_REG_AX will be R12.
--- include/linux/filter.h | 5 +- kernel/bpf/core.c | 4 +- .../selftests/bpf/prog_tests/ctx_rewrite.c | 14 ++-- .../bpf/progs/verifier_bpf_fastcall.c | 24 +++---- .../selftests/bpf/progs/verifier_may_goto_1.c | 12 ++-- .../selftests/bpf/progs/verifier_sdiv.c | 64 +++++++++---------- 6 files changed, 62 insertions(+), 61 deletions(-)
include/linux/filter.h +3 -2
@@ -58,8 +58,9 @@ struct ctl_table_header;#define BPF_REG_H BPF_REG_9/* hlen, callee-saved */ /* Kernel hidden auxiliary/helper register. */ -#define BPF_REG_AX MAX_BPF_REG -#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) +#define BPF_REG_PARAMS MAX_BPF_REG +#define BPF_REG_AX (MAX_BPF_REG + 1) +#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG/* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,u32 imm_rnd =get_random_u32 ();s16 off; -BUILD_BUG_ON (BPF_REG_AX +1 !=MAX_BPF_JIT_REG ); -BUILD_BUG_ON (MAX_BPF_REG +1 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_PARAMS +2 !=MAX_BPF_JIT_REG ); +BUILD_BUG_ON (BPF_REG_AX +1 !=MAX_BPF_JIT_REG );/* Constraints on AX register: *
tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c +7 -7
@@ -69,19 +69,19 @@ static struct test_case test_cases[] = { #if defined(__x86_64__) || defined(__aarch64__) { N(SCHED_CLS, struct __sk_buff, tstamp), - .read = "r11 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);" - "if w11 & 0x4 goto pc+1;" + .read = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);" + "if w12 & 0x4 goto pc+1;" "goto pc+4;" - "if w11 & 0x3 goto pc+1;" + "if w12 & 0x3 goto pc+1;" "goto pc+2;" "$dst = 0;" "goto pc+1;" "$dst = *(u64 *)($ctx + sk_buff::tstamp);", - .write = "r11 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);" - "if w11 & 0x4 goto pc+1;" + .write = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);" + "if w12 & 0x4 goto pc+1;" "goto pc+2;" - "w11 &= -4;" - "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r11;" + "w12 &= -4;" + "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r12;" "*(u64 *)($ctx + sk_buff::tstamp) = $src;", }, #endif
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +12 -12
@@ -630,13 +630,13 @@ __xlated("...") __xlated("4: r0 = &(void __percpu *)(r0)")__xlated ("..." ) /* may_goto expansion starts */ -__xlated("6: r11 = *(u64 *)(r10 -24)") -__xlated("7: if r11 == 0x0 goto pc+6") -__xlated("8: r11 -= 1") -__xlated("9: if r11 != 0x0 goto pc+2") -__xlated("10: r11 = -24") +__xlated("6: r12 = *(u64 *)(r10 -24)") +__xlated("7: if r12 == 0x0 goto pc+6") +__xlated("8: r12 -= 1") +__xlated("9: if r12 != 0x0 goto pc+2") +__xlated("10: r12 = -24") __xlated("11: call unknown") -__xlated ("12: *(u64 *)(r10 -24) = r11" ) +__xlated ("12: *(u64 *)(r10 -24) = r12" ) /* may_goto expansion ends */ __xlated("13: *(u64 *)(r10 -8) = r1")__xlated ("14: exit" )
@@ -668,13 +668,13 @@ __xlated("1: *(u64 *)(r10 -16) =") __xlated("2: r1 = 1")__xlated ("3: call bpf_get_smp_processor_id" ) /* may_goto expansion starts */ -__xlated("4: r11 = *(u64 *)(r10 -24)") -__xlated("5: if r11 == 0x0 goto pc+6") -__xlated("6: r11 -= 1") -__xlated("7: if r11 != 0x0 goto pc+2") -__xlated("8: r11 = -24") +__xlated("4: r12 = *(u64 *)(r10 -24)") +__xlated("5: if r12 == 0x0 goto pc+6") +__xlated("6: r12 -= 1") +__xlated("7: if r12 != 0x0 goto pc+2") +__xlated("8: r12 = -24") __xlated("9: call unknown") -__xlated ("10: *(u64 *)(r10 -24) = r11" ) +__xlated ("10: *(u64 *)(r10 -24) = r12" ) /* may_goto expansion ends */ __xlated("11: *(u64 *)(r10 -8) = r1")__xlated ("12: exit" )
tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +6 -6
@@ -81,13 +81,13 @@ __arch_s390x __arch_arm64 __xlated("0: *(u64 *)(r10 -16) = 65535") __xlated("1: *(u64 *)(r10 -8) = 0") -__xlated("2: r11 = *(u64 *)(r10 -16)") -__xlated("3: if r11 == 0x0 goto pc+6") -__xlated("4: r11 -= 1") -__xlated("5: if r11 != 0x0 goto pc+2") -__xlated("6: r11 = -16") +__xlated("2: r12 = *(u64 *)(r10 -16)") +__xlated("3: if r12 == 0x0 goto pc+6") +__xlated("4: r12 -= 1") +__xlated("5: if r12 != 0x0 goto pc+2") +__xlated("6: r12 = -16") __xlated("7: call unknown") -__xlated("8: *(u64 *)(r10 -16) = r11") +__xlated("8: *(u64 *)(r10 -16) = r12") __xlated("9: r0 = 1") __xlated("10: r0 = 2")__xlated ("11: exit" )
tools/testing/selftests/bpf/progs/verifier_sdiv.c +32 -32
112 lines not on this page · raw message
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 3/16 bpf: Refactor to handle memory and size together
In reply to Yonghong SongSimilar to the previous patch, try to pass bpf_reg_state from caller to callee. Both mem_reg and size_reg are passed to helper functions. This is important for stack arguments as they may be beyond registers 1-5.
--- kernel/bpf/verifier.c | 59 ++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 29 deletions(-)
kernel/bpf/verifier.c +30 -29
@@ -6932,12 +6932,12 @@ static int check_stack_range_initialized(return 0 ; } -static int check_helper_mem_access (struct bpf_verifier_env *env,int regno, +static int check_helper_mem_access (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int regno,int access_size,enum bpf_access_type access_type,bool zero_size_allowed,struct bpf_call_arg_meta *meta) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[regno]; +struct bpf_reg_state *regs =cur_regs (env);u32 *max_access;switch (base_type (reg->type)) {
@@ -7020,15 +7020,17 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,/* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * -* @regno is the register containing the access size. regno-1 is the register -* containing the pointer. +* @mem_regno is the register containing the pointer, mem_regno+1 is the register +* containing the access size. */ static int check_mem_size_reg (struct bpf_verifier_env *env, -struct bpf_reg_state *reg,u32 regno, +struct bpf_reg_state *mem_reg, +struct bpf_reg_state *size_reg,u32 mem_regno,enum bpf_access_type access_type,bool zero_size_allowed,struct bpf_call_arg_meta *meta) { +int size_regno = mem_regno +1 ;int err;/* This is used to refine r0 return value bounds for helpers
@@ -7039,37 +7041,37 @@ static int check_mem_size_reg(struct bpf_verifier_env *env,* out. Only upper bounds can be learned because retval is an * int type and negative retvals are allowed. */ - meta->msize_max_value = reg->umax_value; + meta->msize_max_value = size_reg->umax_value;/* The register is SCALAR_VALUE; the access check happens using * its boundaries. For unprivileged variable accesses, disable * raw mode so that the program is required to initialize all * the memory that the helper could just partially fill up. */ -if (!tnum_is_const (reg->var_off)) +if (!tnum_is_const (size_reg->var_off)) meta =NULL ; -if (reg->smin_value <0 ) { +if (size_reg->smin_value <0 ) {verbose (env,"R%d min value is negative, either use unsigned or 'var &= const'\n" , - regno); + size_regno);return -EACCES ; } -if (reg->umin_value ==0 && !zero_size_allowed) { +if (size_reg->umin_value ==0 && !zero_size_allowed) {verbose (env,"R%d invalid zero-sized read: u64=[%lld,%lld]\n" , - regno, reg->umin_value, reg->umax_value); + size_regno, size_reg->umin_value, size_reg->umax_value);return -EACCES ; } -if (reg->umax_value >=BPF_MAX_VAR_SIZ ) { +if (size_reg->umax_value >=BPF_MAX_VAR_SIZ ) {verbose (env,"R%d unbounded memory access, use 'var &= const' or 'if (var < const)'\n" , - regno); + size_regno);return -EACCES ; } - err =check_helper_mem_access (env, regno -1 , reg->umax_value, + err =check_helper_mem_access (env, mem_reg, mem_regno, size_reg->umax_value, access_type, zero_size_allowed, meta);if (!err) - err =mark_chain_precision (env, regno); + err =mark_chain_precision (env, size_regno);return err; }
@@ -7094,8 +7096,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *regint size =base_type (reg->type) ==PTR_TO_STACK ? -(int )mem_size : mem_size; - err =check_helper_mem_access (env, regno, size,BPF_READ , true,NULL ); - err = err ?:check_helper_mem_access (env, regno, size,BPF_WRITE , true,NULL ); + err =check_helper_mem_access (env, reg, regno, size,BPF_READ , true,NULL ); + err = err ?:check_helper_mem_access (env, reg, regno, size,BPF_WRITE , true,NULL );if (may_be_null) *reg = saved_reg;
@@ -7103,16 +7105,15 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *regreturn err; } -static int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg, -u32 regno) +static int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *mem_reg, +struct bpf_reg_state *size_reg,u32 mem_regno) { -struct bpf_reg_state *mem_reg = &cur_regs (env)[regno -1 ];bool may_be_null =type_may_be_null (mem_reg->type);struct bpf_reg_state saved_reg;struct bpf_call_arg_meta meta;int err; -WARN_ON_ONCE (regno <BPF_REG_2 || regno >BPF_REG_5 ); +WARN_ON_ONCE (mem_regno >BPF_REG_4 );memset (&meta,0 ,sizeof (meta));
@@ -7121,8 +7122,8 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_regmark_ptr_not_null_reg (mem_reg); } - err =check_mem_size_reg (env, reg, regno,BPF_READ , true, &meta); - err = err ?:check_mem_size_reg (env, reg, regno,BPF_WRITE , true, &meta); + err =check_mem_size_reg (env, mem_reg, size_reg, mem_regno,BPF_READ , true, &meta); + err = err ?:check_mem_size_reg (env, mem_reg, size_reg, mem_regno,BPF_WRITE , true, &meta);if (may_be_null) *mem_reg = saved_reg;
@@ -8586,7 +8587,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,return -EFAULT ; } key_size = meta->map.ptr->key_size; - err =check_helper_mem_access (env, regno, key_size,BPF_READ , false,NULL ); + err =check_helper_mem_access (env, reg, regno, key_size,BPF_READ , false,NULL );if (err)return err;if (can_elide_value_nullness (meta->map.ptr->map_type)) {
@@ -8613,7 +8614,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,return -EFAULT ; } meta->raw_mode = arg_type &MEM_UNINIT ; - err =check_helper_mem_access (env, regno, meta->map.ptr->value_size, + err =check_helper_mem_access (env, reg, regno, meta->map.ptr->value_size, arg_type &MEM_WRITE ?BPF_WRITE :BPF_READ , false, meta);break ;
@@ -8657,7 +8658,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,*/ meta->raw_mode = arg_type &MEM_UNINIT ;if (arg_type &MEM_FIXED_SIZE ) { - err =check_helper_mem_access (env, regno, fn->arg_size[arg], + err =check_helper_mem_access (env, reg, regno, fn->arg_size[arg], arg_type &MEM_WRITE ?BPF_WRITE :BPF_READ , false, meta);if (err)
@@ -8667,13 +8668,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, }break ;case ARG_CONST_SIZE : - err =check_mem_size_reg (env, reg, regno, + err =check_mem_size_reg (env,reg_state (env, regno -1 ), reg, regno -1 , fn->arg_type[arg -1 ] &MEM_WRITE ?BPF_WRITE :BPF_READ , false, meta);break ;case ARG_CONST_SIZE_OR_ZERO : - err =check_mem_size_reg (env, reg, regno, + err =check_mem_size_reg (env,reg_state (env, regno -1 ), reg, regno -1 , fn->arg_type[arg -1 ] &MEM_WRITE ?BPF_WRITE :BPF_READ , true, meta);
@@ -12407,7 +12408,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_const struct btf_param *size_arg = &args[i +1 ];if (!bpf_register_is_null (buff_reg) || !is_kfunc_arg_nullable (meta->btf, buff_arg)) { - ret =check_kfunc_mem_size_reg (env, size_reg, regno +1 ); + ret =check_kfunc_mem_size_reg (env, buff_reg, size_reg, regno);if (ret <0 ) {verbose (env,"arg#%d arg#%d memory, len pair leads to invalid memory access\n" , i, i +1 );return ret;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 6/16 bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to Yonghong SongBPF_REG_PARAMS (r11) is used for stack argument accesses. The LLVM compiler [1] only emits BPF_REG_PARAMS in the following instruction forms:
- BPF_LDX | BPF_MEM (load incoming stack arg) - BPF_ST | BPF_MEM (store immediate to outgoing stack arg) - BPF_STX | BPF_MEM (store register to outgoing stack arg)
Reject any other use of BPF_REG_PARAMS in check_and_resolve_insns() to prevent misuse.
Since BPF_REG_PARAMS is beyond MAX_BPF_REG, array-based register tracking indexed by register number would cause out-of-bounds accesses. So do early return if needed.
[1] https://github.com/llvm/llvm-project/pull/189060
--- kernel/bpf/const_fold.c | 9 +++++++-- kernel/bpf/liveness.c | 9 +++++++-- kernel/bpf/verifier.c | 17 +++++++++++++---- 3 files changed, 27 insertions(+), 8 deletions(-)
kernel/bpf/const_fold.c +7 -2
@@ -51,13 +51,18 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * struct bpf_insn *insn, struct bpf_insn *insns, int idx) { struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 }; -struct const_arg_info *dst = &ci_out[insn->dst_reg]; -struct const_arg_info *src = &ci_out[insn->src_reg]; +struct const_arg_info *dst, *src;u8 class =BPF_CLASS (insn->code);u8 mode =BPF_MODE (insn->code);u8 opcode =BPF_OP (insn->code) |BPF_SRC (insn->code);int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ; + + dst = &ci_out[insn->dst_reg]; + src = &ci_out[insn->src_reg];switch (class) {case BPF_ALU :case BPF_ALU64 :
kernel/bpf/liveness.c +7 -2
@@ -1056,11 +1056,16 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,int depth = instance->depth;u8 class =BPF_CLASS (insn->code);u8 code =BPF_OP (insn->code); -struct arg_track *dst = &at_out[insn->dst_reg]; -struct arg_track *src = &at_out[insn->src_reg]; +struct arg_track *dst, *src;struct arg_track none = { .frame =ARG_NONE };int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ; + + dst = &at_out[insn->dst_reg]; + src = &at_out[insn->src_reg];if (class ==BPF_ALU64 &&BPF_SRC (insn->code) ==BPF_K ) {if (code ==BPF_MOV ) { *dst = none;
kernel/bpf/verifier.c +13 -4
@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + } }if (insn->src_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->src_reg); -return -EINVAL ; +if (insn->src_reg !=BPF_REG_PARAMS || class !=BPF_LDX || + mode !=BPF_MEM ) { +verbose (env,"R%d is invalid\n" , insn->src_reg); +return -EINVAL ; + } }if (insn[0 ].code == (BPF_LD |BPF_IMM |BPF_DW )) {struct bpf_insn_aux_data *aux;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 7/16 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongCurrently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.
--- include/linux/bpf.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 8/16 bpf: Support stack arguments for bpf functions
In reply to Yonghong SongCurrently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_PARAMS (r11), introduced in the previous patch.
The compiler uses positive r11 offsets for incoming (callee-side) args and negative r11 offsets for outgoing (caller-side) args, following the x86_64/arm64 calling convention direction. There is an 8-byte gap at offset 0 separating the two regions:
Incoming (callee reads): r11+8 (arg6), r11+16 (arg7), ...
Outgoing (caller writes): r11-8 (arg6), r11-16 (arg7), ...
The following is an example to show how stack arguments are saved and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar(a1, a2, a3, a4, a5, a6, a7, a8);
...
}Caller (foo) Callee (bar) ============ ============ Incoming (positive offsets): Incoming (positive offsets):
r11+8: [incoming arg 6] r11+8: [incoming arg 6] <-+
r11+16: [incoming arg 7] r11+16: [incoming arg 7] <-|+
r11+24: [incoming arg 8] <-||+
Outgoing (negative offsets): ||| r11-8: [outgoing arg 6 to bar] -------->-------------------------+||
r11-16: [outgoing arg 7 to bar] -------->--------------------------+|
r11-24: [outgoing arg 8 to bar] -------->---------------------------+
If the bpf function has more than one call:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar1(a1, a2, a3, a4, a5, a6, a7, a8);
...
bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}Caller (foo) Callee (bar2) ============ ============== Incoming (positive offsets): Incoming (positive offsets):
r11+8: [incoming arg 6] r11+8: [incoming arg 6] <+
r11+16: [incoming arg 7] r11+16: [incoming arg 7] <|+
r11+24: [incoming arg 8] <||+
Outgoing for bar2 (negative offsets): r11+32: [incoming arg 9] <|||+
r11-8: [outgoing arg 6] ---->----------->-------------------------+|||
r11-16: [outgoing arg 7] ---->----------->--------------------------+||
r11-24: [outgoing arg 8] ---->----------->---------------------------+|r11-32: [outgoing arg 9] ---->----------->----------------------------+
The verifier tracks stack arguments separately from the regular r10 stack. The stack_arg_regs are stored in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe() to do pruning check.
A per-state bitmask out_stack_arg_mask tracks which outgoing stack arg slots have been written on the current path. Each bit corresponds to an outgoing slot index (bit 0 = r11-8 = arg6, bit 1 = r11-16 = arg7, etc.). At a call site, the verifier checks that all slots required by the callee have their corresponding mask bits set. This enables precise per-path tracking: if one branch of a conditional writes arg6 but another does not, the mask correctly reflects the difference and the verifier rejects the uninitialized path. The mask is included in stack_arg_safe() so that states with different sets of initialized slots are not incorrectly pruned together.
Outgoing stack arg slots are not invalidated after a call. This allows the compiler to hoist shared stores above a call and reuse them for subsequent calls. The following are a few examples.
Example 1:
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
call bar2; // reuses same arg6, arg7 without re-storing
Example 2:
If the caller wants different values for a later call, it simply overwrites the slot before that call:
*(u64 *)(r11 - 8) = r6;
call bar1; // arg6 = r6 *(u64 *)(r11 - 8) = r8; // overwrite arg6 call bar2; // arg6 = r8
Example 3:
The compiler can hoist the shared stack arg stores above the branch:
*(u64 *)(r11 - 16) = r7;
...
if cond goto else;
*(u64 *)(r11 - 8) = r8;
call bar1; // arg6 = r8, arg7 = r7
goto end; else:
*(u64 *)(r11 - 8) = r9;
call bar2; // arg6 = r9, arg7 = r7
end:
Example 4:
The compiler hoists the store above the loop:
*(u64 *)(r11 - 8) = r6; // arg6, before loop
loop:
call bar; // reuses arg6 each iteration
if ... goto loop;
A separate max_out_stack_arg_depth field in bpf_subprog_info tracks the deepest outgoing offset actually written. This intends to reject programs that write to offsets beyond what any callee expects.
Callback functions with stack arguments need kernel setup parameter types (including stack parameters) properly and then callback function can retrieve such information for verification purpose.
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
--- include/linux/bpf.h | 2 + include/linux/bpf_verifier.h | 28 +++- kernel/bpf/btf.c | 14 +- kernel/bpf/fixups.c | 26 +++- kernel/bpf/states.c | 41 +++++ kernel/bpf/verifier.c | 284 ++++++++++++++++++++++++++++++++++- 6 files changed, 383 insertions(+), 12 deletions(-)
include/linux/bpf.h +2 -0
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {u32 max_pkt_offset;u32 max_tp_access;u32 stack_depth; +u16 incoming_stack_arg_depth; +u16 stack_arg_depth;/* both incoming and max outgoing of stack arguments */ u32 id;u32 func_cnt;/* used by non-func prog as the number of func progs */ u32 real_func_cnt;/* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +26 -2
@@ -372,6 +372,11 @@ struct bpf_func_state {* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. */ int allocated_stack; + +u16 stack_arg_depth;/* Size of incoming + max outgoing stack args in bytes. */ +u16 incoming_stack_arg_depth;/* Size of incoming stack args in bytes. */ +u16 out_stack_arg_mask;/* Bitmask of outgoing stack arg slots that have been written. */ +struct bpf_reg_state *stack_arg_regs;/* On-stack arguments */ };#define MAX_CALL_FRAMES 8
@@ -508,6 +513,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL) + +/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */ +#define bpf_for_each_spilled_stack_arg (iter, frame, reg, mask) \ + for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \ + iter < frame->stack_arg_depth / BPF_REG_SIZE; \ + iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask)) +#define bpf_for_each_reg_in_vstate_mask (__vst, __state, __reg, __mask, __expr) \ ({ \ struct bpf_verifier_state *___vstate = __vst; \
@@ -525,6 +541,11 @@ struct bpf_verifier_state {continue ; \ (void )(__expr); \ } \ + bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \ +if (!__reg) \ +continue ; \ + (void )(__expr); \ + } \ } \ })
@@ -738,10 +759,13 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; - u8 arg_cnt:3; + u8 arg_cnt:4;enum priv_stack_mode priv_stack_mode; -struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS ]; +struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS ]; +u16 incoming_stack_arg_depth; +u16 outgoing_stack_arg_depth; +u16 max_out_stack_arg_depth; };struct bpf_verifier_env ;
kernel/bpf/btf.c +10 -4
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) } args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { -if (!is_global) -return -EINVAL ; -bpf_log (log,"Global function %s() with %d > %d args. Buggy compiler.\n" , +if (nargs >MAX_BPF_FUNC_ARGS ) { +bpf_log (log,"Function %s() with %d > %d args not supported.\n" , + tname, nargs,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; + } +if (is_global && nargs >MAX_BPF_FUNC_REG_ARGS ) { +bpf_log (log,"Global function %s() with %d > %d args not supported.\n" , tname, nargs,MAX_BPF_FUNC_REG_ARGS );return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS ) + sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; +/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);while (btf_type_is_modifier (t))
kernel/bpf/fixups.c +23 -3
@@ -983,8 +983,14 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)int err, num_exentries;int old_len, subprog_start_adjustment =0 ; -if (env->subprog_cnt <=1 ) +if (env->subprog_cnt <=1 ) { +/* +* Even without subprogs, kfunc calls with >5 args need stack arg space +* allocated by the root program. +*/ + prog->aux->stack_arg_depth = env->subprog_info[0 ].outgoing_stack_arg_depth;return 0 ; + }for (i =0 , insn = prog->insnsi; i < prog->len; i++, insn++) {if (!bpf_pseudo_func (insn) && !bpf_pseudo_call (insn))
@@ -1074,6 +1080,9 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->name[0 ] ='F' ; func[i]->aux->stack_depth = env->subprog_info[i].stack_depth; + func[i]->aux->incoming_stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth; + func[i]->aux->stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth + + env->subprog_info[i].outgoing_stack_arg_depth;if (env->subprog_info[i].priv_stack_mode ==PRIV_STACK_ADAPTIVE ) func[i]->aux->jits_use_priv_stack = true;
@@ -1265,9 +1274,20 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)struct bpf_prog *prog = env->prog;struct bpf_insn *insn = prog->insnsi;bool has_kfunc_call =bpf_prog_has_kfunc_call (prog); -int i, depth; +int depth;#endif -int err =0 ; +int i, err =0 ; + +for (i =0 ; i < env->subprog_cnt; i++) { +struct bpf_subprog_info *subprog = &env->subprog_info[i]; + +if (subprog->max_out_stack_arg_depth > subprog->outgoing_stack_arg_depth) { +verbose (env, +"func#%d writes stack arg slot at depth %u, but calls only require %u bytes\n" , + i, subprog->max_out_stack_arg_depth, subprog->outgoing_stack_arg_depth); +return -EINVAL ; + } + }if (env->prog->jit_requested && !bpf_prog_is_offloaded (env->prog->aux)) {
kernel/bpf/states.c +41 -0
53 lines not on this page · raw message
kernel/bpf/verifier.c +281 -3
390 lines not on this page · raw message
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 9/16 bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong SongThe interpreter does not understand the bpf register r11 (BPF_REG_PARAMS) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.
--- kernel/bpf/core.c | 2 +- kernel/bpf/fixups.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
kernel/bpf/core.c +1 -1
@@ -2555,7 +2555,7 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || fp->aux->stack_arg_depth) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/fixups.c +6 -0
@@ -1302,6 +1302,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)verbose (env,"calling kernel functions are not allowed in non-JITed programs\n" );return -EINVAL ; } +for (i =0 ; i < env->subprog_cnt; i++) { +if (env->subprog_info[i].incoming_stack_arg_depth) { +verbose (env,"stack args are not supported in non-JITed programs\n" ); +return -EINVAL ; + } + }if (env->subprog_cnt >1 && env->prog->aux->tail_call_reachable) {/* When JIT fails the progs with bpf2bpf calls and tail_calls * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 10/16 bpf: Reject stack arguments if tail call reachable
In reply to Yonghong SongTailcalls are been deprecated. So reject stack arguments if tail call is in the way.
--- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + } subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 11/16 bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongExtend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r11-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at negative r11 offsets, following the same convention as BPF-to-BPF calls:
Outgoing: r11 - 8 (arg6), ..., r11 - N*8 (last arg)
The following is an example:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
} Caller (foo), generated by llvm
===============================
Incoming (positive offsets):
r11+8: [incoming arg 6]
r11+16: [incoming arg 7]Outgoing for kfunc1 (negative offsets):
r11-8: [outgoing arg 6]
r11-16: [outgoing arg 7]
r11-24: [outgoing arg 8]Outgoing for kfunc2 (negative offsets):
r11-8: [outgoing arg 6]
r11-16: [outgoing arg 7]
r11-24: [outgoing arg 8]
r11-32: [outgoing arg 9]Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().
There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno' is also keeping regno for later fixup. Since stack arguments don't have a valid register number (regno is set to -1), these three cases are rejected for now if the argument is on the stack.
--- kernel/bpf/verifier.c | 114 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 94 insertions(+), 20 deletions(-)
kernel/bpf/verifier.c +94 -20
@@ -4497,6 +4497,14 @@ static int check_stack_arg_access(struct bpf_verifier_env *env,return 0 ; } +/* Check that a stack arg slot has been properly initialized. */ +static bool is_stack_arg_slot_initialized (struct bpf_func_state *state,int spi) +{ +if (spi >= (int )(state->stack_arg_depth /BPF_REG_SIZE )) +return false; +return state->stack_arg_regs[spi].type !=NOT_INIT ; +} +static int out_arg_idx_from_off (int off) {return -off /BPF_REG_SIZE -1 ;
@@ -7355,8 +7363,6 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_regu32 argno =make_argno (mem_argno);int err; -WARN_ON_ONCE (mem_argno >BPF_REG_3 ); -memset (&meta,0 ,sizeof (meta));if (may_be_null) {
@@ -11653,6 +11659,19 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi =out_arg_spi (caller, argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_regs[spi]; +} + static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,
@@ -11660,8 +11679,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, const char *ref_tname, const struct btf_param *args, int argno, int nargs, struct bpf_reg_state *reg) { - u32 regno = argno + 1; -struct bpf_reg_state *regs =cur_regs (env); bool arg_mem_size = false; if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11670,8 +11687,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,return KF_ARG_PTR_TO_CTX ;if (argno +1 < nargs && - (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]) || -is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]))) + (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )) || +is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )))) arg_mem_size = true;/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -12344,9 +12361,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (nargs >MAX_BPF_FUNC_ARGS ) {verbose (env,"Function %s has %d > %d args\n" , func_name, nargs, -MAX_BPF_FUNC_REG_ARGS ); +MAX_BPF_FUNC_ARGS );return -EINVAL ; }
@@ -12354,20 +12371,44 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_* verifier sees. */ for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *regs =cur_regs (env), *reg = ®s[i +1 ]; +struct bpf_reg_state *regs =cur_regs (env), *reg;const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 argno =make_argno (i); -u32 regno = i +1 , ref_id, type_size; +struct bpf_reg_state tmp_reg; +int regno = i +1 ; +u32 ref_id, type_size;bool is_ret_buf_sz = false;int kf_arg_type; +if (i <MAX_BPF_FUNC_REG_ARGS ) { + reg = ®s[i +1 ]; + }else { +/* Retrieve the reg state from the outgoing stack arg slot. */ +struct bpf_func_state *caller =cur_func (env); +int spi =out_arg_spi (caller, i -MAX_BPF_FUNC_REG_ARGS ); + +if (!is_stack_arg_slot_initialized (caller, spi)) { +verbose (env,"stack %s not properly initialized\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } + + tmp_reg = caller->stack_arg_regs[spi]; + reg = &tmp_reg; + regno =-1 ; + } +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {verifier_bug (env,"Only 1 prog->aux argument supported per-kfunc" );return -EFAULT ; } +if (regno <0 ) { +verbose (env,"%s prog->aux cannot be a stack argument\n" ,reg_arg_name (env, argno)); +return -EINVAL ; + } meta->arg_prog = true;cur_aux (env)->arg_prog = regno;continue ;
@@ -12394,9 +12435,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ reg_arg_name(env, argno));return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (regno >0 ) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + } meta->arg_constant.found = true; meta->arg_constant.value = reg->var_off.value; }else if (is_kfunc_arg_scalar_with_name (btf, &args[i],"rdonly_buf_size" )) {
@@ -12419,9 +12462,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; - ret =mark_chain_precision (env, regno); -if (ret) -return ret; +if (regno >0 ) { + ret =mark_chain_precision (env, regno); +if (ret) +return ret; + } }continue ; }
@@ -12447,8 +12492,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_return -EFAULT ; } meta->ref_obj_id = reg->ref_obj_id; -if (is_kfunc_release (meta)) +if (is_kfunc_release (meta)) { +if (regno <0 ) { +verbose (env,"%s release arg cannot be a stack argument\n" ,reg_arg_name (env, argno)); +return -EINVAL ; + } meta->release_regno = regno; + } } ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);
@@ -12607,6 +12657,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ dynptr_arg_type |=DYNPTR_TYPE_FILE ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) { dynptr_arg_type |=DYNPTR_TYPE_FILE ; +if (regno <0 ) { +verbose (env,"%s release arg cannot be a stack argument\n" ,reg_arg_name (env, argno)); +return -EINVAL ; + } meta->release_regno = regno; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] && (dynptr_arg_type &MEM_UNINIT )) {
@@ -12761,9 +12815,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_break ;case KF_ARG_PTR_TO_MEM_SIZE : { -struct bpf_reg_state *buff_reg = ®s[regno]; +struct bpf_reg_state *buff_reg = reg;const struct btf_param *buff_arg = &args[i]; -struct bpf_reg_state *size_reg = ®s[regno +1 ]; +struct bpf_reg_state *size_reg =get_kfunc_arg_reg (env, i +1 );const struct btf_param *size_arg = &args[i +1 ];if (!bpf_register_is_null (buff_reg) || !is_kfunc_arg_nullable (meta->btf, buff_arg)) {
@@ -13667,7 +13721,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, nargs =btf_type_vlen (meta.func_proto); args = (const struct btf_param *)(meta.func_proto +1 ); -for (i =0 ; i < nargs; i++) { +for (i =0 ; i <min_t (int, nargs,MAX_BPF_FUNC_REG_ARGS ); i++) {u32 regno = i +1 ; t =btf_type_skip_modifiers (desc_btf, args[i].type,NULL );
@@ -13678,6 +13732,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,mark_btf_func_reg_size (env, regno, t->size); } +/* Track outgoing stack arg depth for kfuncs with >5 args */ +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +struct bpf_func_state *caller =cur_func (env); +struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; +u16 kfunc_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + +if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth) + caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth; + } +if (bpf_is_iter_next_kfunc (&meta)) { err =process_iter_next_call (env, insn_idx, &meta);if (err)
@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +if (is_bpf_obj_new_kfunc (desc->func_id) ||is_bpf_percpu_obj_new_kfunc (desc->func_id)) {struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;struct bpf_insn addr[2 ] = {BPF_LD_IMM64 (BPF_REG_2 , (long )kptr_struct_meta) };
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 12/16 bpf: Enable stack argument support for x86_64
In reply to Yonghong SongAdd stack argument support for x86_64.
--- arch/x86/net/bpf_jit_comp.c | 5 +++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 8 +++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 5 +++++ 5 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
include/linux/filter.h +1 -0
@@ -1161,6 +1161,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void ); +bool bpf_jit_supports_stack_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );
kernel/bpf/btf.c +7 -1
@@ -7897,8 +7897,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tname, nargs, MAX_BPF_FUNC_REG_ARGS);return -EINVAL ; } -if (nargs >MAX_BPF_FUNC_REG_ARGS ) +if (nargs >MAX_BPF_FUNC_REG_ARGS ) { +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d args\n" , + tname, nargs); +return -ENOTSUPP ; + } sub->incoming_stack_arg_depth = (nargs -MAX_BPF_FUNC_REG_ARGS ) *BPF_REG_SIZE ; + }/* check that function is void or returns int, exception cb also requires this */ t =btf_type_by_id (btf, t->type);
kernel/bpf/core.c +5 -0
@@ -3157,6 +3157,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)return false; } +bool __weak bpf_jit_supports_stack_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;
kernel/bpf/verifier.c +5 -0
@@ -12366,6 +12366,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ MAX_BPF_FUNC_ARGS);return -EINVAL ; } +if (nargs >MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args ()) { +verbose (env,"JIT does not support kfunc %s() with %d args\n" , + func_name, nargs); +return -ENOTSUPP ; + }/* Check that BTF function arguments match actual types that the * verifier sees.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 13/16 bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongAdd x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r11 (BPF_REG_PARAMS) in BPF bytecode, which the JIT translates to native code.
The JIT follows the x86-64 calling convention for both BPF-to-BPF and kfunc calls:
- Arg 6 is passed in the R9 register
- Args 7+ are passed on the stack
Incoming arg 6 (BPF r11+8) is translated to a MOV from R9 rather than a memory load. Incoming args 7+ (BPF r11+16, r11+24, ...) map directly to [rbp + 16], [rbp + 24], ..., matching the x86-64 stack layout after CALL + PUSH RBP, so no offset adjustment is needed.
The verifier guarantees that neither tail_call_reachable nor priv_stack is set when stack args exist, so R9 is always available. When BPF bytecode writes to the arg-6 stack slot (offset -8), the JIT emits a MOV into R9 instead of a memory store. Outgoing args 7+ are placed at [rsp] in a pre-allocated area below callee-saved registers, using:
native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16
The native x86_64 stack layout with stack arguments:
high address
+-------------------------+ | incoming stack arg N | [rbp + 16 + (N-7)*8] (from caller) | ... | | incoming stack arg 7 | [rbp + 16] +-------------------------+ | return address | [rbp + 8] | saved rbp | [rbp] +-------------------------+ | BPF program stack | (round_up(stack_depth, 8) bytes) +-------------------------+ | callee-saved regs | (r12, rbx, r13, r14, r15 as needed) +-------------------------+ | outgoing arg M | [rsp + (M-7)*8] | ... | | outgoing arg 7 | [rsp] +-------------------------+ rsp
low address
--- arch/x86/net/bpf_jit_comp.c | 149 ++++++++++++++++++++++++++++++++++-- 1 file changed, 143 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +143 -6
@@ -390,6 +390,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +/* add rsp, depth */ +static void emit_add_rsp (u8 **pprog,u16 depth) +{ +u8 *prog = *pprog; + +if (!depth) +return ; +if (is_imm8 (depth)) +EMIT4 (0x48 ,0x83 ,0xC4 , depth);/* add rsp, imm8 */ +else +EMIT3_off32 (0x48 ,0x81 ,0xC4 , depth);/* add rsp, imm32 */ + *pprog = prog; +} + +/* sub rsp, depth */ +static void emit_sub_rsp (u8 **pprog,u16 depth) +{ +u8 *prog = *pprog; + +if (!depth) +return ; +if (is_imm8 (depth)) +EMIT4 (0x48 ,0x83 ,0xEC , depth);/* sub rsp, imm8 */ +else +EMIT3_off32 (0x48 ,0x81 ,0xEC , depth);/* sub rsp, imm32 */ + *pprog = prog; +} +static void emit_nops (u8 **pprog,int len) {u8 *prog = *pprog;
@@ -1664,16 +1692,45 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageint i, excnt =0 ;int ilen, proglen =0 ;u8 *prog = temp; +u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth; +u16 outgoing_rsp;u32 stack_depth; +int callee_saved_size; +s32 outgoing_arg_base;int err; stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Follow x86-64 calling convention for both BPF-to-BPF and +* kfunc calls: +* - Arg 6 is passed in R9 register +* - Args 7+ are passed on the stack at [rsp] +* +* Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9). +* Incoming args 7+ are read from [rbp + 16], [rbp + 24], ... +* (BPF r11+16, r11+24, ... map directly with no offset change). +* +* The verifier guarantees that neither tail_call_reachable nor +* priv_stack is set when outgoing stack args exist, so R9 is +* always available. +* +* Stack layout (high to low): +* [rbp + 16 + ...] incoming stack args 7+ (from caller) +* [rbp + 8] return address +* [rbp] saved rbp +* [rbp - prog_stack] program stack +* [below] callee-saved regs +* [below] outgoing args 7+ (= rsp) +*/ arena_vm_start =bpf_arena_get_kern_vm_start (bpf_prog->aux->arena); user_vm_start =bpf_arena_get_user_vm_start (bpf_prog->aux->arena);
@@ -1700,6 +1757,42 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagepush_r12 (&prog);push_callee_regs (&prog, callee_regs_used); } + +/* Compute callee-saved register area size. */ + callee_saved_size =0 ; +if (bpf_prog->aux->exception_boundary || arena_vm_start) + callee_saved_size +=8 ;/* r12 */ +if (bpf_prog->aux->exception_boundary) { + callee_saved_size +=4 *8 ;/* rbx, r13, r14, r15 */ + }else { +int j; + +for (j =0 ; j <4 ; j++) +if (callee_regs_used[j]) + callee_saved_size +=8 ; + } +/* +* Base offset from rbp for translating BPF outgoing args 7+ +* to native offsets. BPF uses negative offsets from r11 +* (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses +* positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for +* arg8, ...). Arg 6 goes to R9 directly. +* +* The translation reverses direction: +* native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 +* +* Note that tail_call_reachable is guaranteed to be false when +* stack args exist, so tcc pushes need not be accounted for. +*/ + outgoing_arg_base = -(round_up (stack_depth,8 ) + callee_saved_size); + +/* +* Allocate outgoing stack arg area for args 7+ only. +* Arg 6 goes into r9 register, not on stack. +*/ + outgoing_rsp = outgoing_stack_arg_depth >8 ? outgoing_stack_arg_depth -8 :0 ; +emit_sub_rsp (&prog, outgoing_rsp); +if (arena_vm_start)emit_mov_imm64 (&prog,X86_REG_R12 , arena_vm_start >>32 , (u32 ) arena_vm_start);
@@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageu8 b2 =0 , b3 =0 ;u8 *start_of_ldx;s64 jmp_offset; -s16 insn_off; +s32 insn_off;u8 jmp_cond;u8 *func;int nops;
@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets. +* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + } +if (is_imm8 (insn_off)) +EMIT2 (add_1reg (0x40 , dst_reg), insn_off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), insn_off);EMIT (imm32,bpf_size_to_x86_bytes (BPF_SIZE (insn->code)));break ;
@@ -2144,7 +2251,17 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W :case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off); +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store register value in r9 */ +EMIT_mov (X86_REG_R9 , src_reg); +break ; + } + insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + } +emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);break ;case BPF_ST |BPF_PROBE_MEM32 |BPF_B :
@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */ +EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ; +/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ||BPF_MODE (insn->code) ==BPF_PROBE_MEMSX ) {
@@ -2734,6 +2864,8 @@ st: if (is_imm8(insn->off))if (emit_spectre_bhb_barrier (&prog, ip, bpf_prog))return -EINVAL ; } +/* Deallocate outgoing args 7+ area. */ +emit_add_rsp (&prog, outgoing_rsp);if (bpf_prog->aux->exception_boundary) {pop_callee_regs (&prog, all_callee_regs_used);pop_r12 (&prog);
@@ -3757,7 +3889,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog->aux->jit_data = jit_data; } priv_stack_ptr = prog->aux->priv_stack_ptr; -if (!priv_stack_ptr && prog->aux->jits_use_priv_stack) { +/* +* x86-64 uses R9 for both private stack frame pointer and arg 6, +* so disable private stack when stack args are present. +*/ +if (!priv_stack_ptr && prog->aux->jits_use_priv_stack && + prog->aux->stack_arg_depth ==0 ) {/* Allocate actual private stack size with verifier-calculated * stack size plus two memory guards to protect overflow and * underflow.
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 15/16 selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to Yonghong SongAdd a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).
--- .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++ .../selftests/bpf/progs/stack_arg_fail.c | 32 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 7 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 8 +++++ 4 files changed, 71 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "stack_arg_fail.skel.h" + +void test_stack_arg_fail (void ) +{ +struct stack_arg_fail *skel; + + skel =stack_arg_fail__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" ); + +out: +stack_arg_fail__destroy (skel); +}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" + +#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +const volatile bool has_stack_arg = true; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +struct prog_test_big_arg s = { .a =1 , .b =2 }; + +return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); +} + +#else + +const volatile bool has_stack_arg = false; + +SEC("tc") +int test_stack_arg_big (struct __sk_buff *skb) +{ +return 0 ; +} + +#endif + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
@@ -882,6 +882,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 ereturn a + b + c + d + e; } +__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e, +struct prog_test_big_arg s) +{ +return a + b + c + d + e + s.a + s.b; +} +static struct prog_test_ref_kfunc prog_test_struct = { .a =42 , .b =108 ,
@@ -1353,6 +1359,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer) +BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)BTF_ID_FLAGS (func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +8 -0
@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +struct prog_test_fail1 {void *p;int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u const char *str__str) __ksym; __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, struct bpf_timer *timer) __ksym; +__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_big_arg s) __ksym; + void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 14/16 selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongAdd selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.
BPF-to-BPF subprog call tests (stack_arg.c):
- Scalar stack args
- Pointer stack args
- Mixed pointer/scalar stack args
- Nested calls
- Dynptr stack arg
- Two callees with different stack arg counts
- Async callback
Kfunc call tests (stack_arg_kfunc.c, with bpf_testmod kfuncs):
- Scalar stack args
- Pointer stack args
- Mixed pointer/scalar stack args
- Dynptr stack arg
- Memory buffer + size pair
- Iterator
- Const string pointer
- Timer pointer
--- .../selftests/bpf/prog_tests/stack_arg.c | 133 +++++++++ tools/testing/selftests/bpf/progs/stack_arg.c | 254 ++++++++++++++++++ .../selftests/bpf/progs/stack_arg_kfunc.c | 164 +++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 66 +++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 20 +- 5 files changed, 636 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +133 -0
@@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +} + +static void test_global_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_global_many_args,36 ); + +out: +stack_arg__destroy (skel); +} + +static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +} + +static void test_bpf2bpf (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_bpf2bpf_ptr_stack_arg,45 ); +run_subtest (skel->progs.test_bpf2bpf_mix_stack_args,51 ); +run_subtest (skel->progs.test_bpf2bpf_nesting_stack_arg,50 ); +run_subtest (skel->progs.test_bpf2bpf_dynptr_stack_arg,69 ); +run_subtest (skel->progs.test_two_callees,91 ); + +out: +stack_arg__destroy (skel); +} + +static void test_kfunc (void ) +{ +struct stack_arg_kfunc *skel; + + skel =stack_arg_kfunc__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg_kfunc__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_stack_arg_scalar,36 ); +run_subtest (skel->progs.test_stack_arg_ptr,45 ); +run_subtest (skel->progs.test_stack_arg_mix,51 ); +run_subtest (skel->progs.test_stack_arg_dynptr,69 ); +run_subtest (skel->progs.test_stack_arg_mem,151 ); +run_subtest (skel->progs.test_stack_arg_iter,115 ); +run_subtest (skel->progs.test_stack_arg_const_str,15 ); +run_subtest (skel->progs.test_stack_arg_timer,15 ); + +out: +stack_arg_kfunc__destroy (skel); +} + +void test_stack_arg (void ) +{ +if (test__start_subtest ("global_many_args" )) +test_global_many (); +if (test__start_subtest ("async_cb_many_args" )) +test_async_cb_many (); +if (test__start_subtest ("bpf2bpf" )) +test_bpf2bpf (); +if (test__start_subtest ("kfunc" )) +test_kfunc (); +}
tools/testing/selftests/bpf/progs/stack_arg.c new file +254 -0
254 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +164 -0
164 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +66 -0
78 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +19 -1
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc { };#endif +struct bpf_iter_testmod_seq ; +struct prog_test_pass1 {int x0;struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; - +__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, + __u64 e, __u64 f, __u64 g, __u64 h) __ksym; +__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p) __ksym; +__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pass1 *p, __u64 f, + struct prog_test_pass1 *q) __ksym; +__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_dynptr *ptr) __ksym; +__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + void *mem, int mem__sz) __ksym; +__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_iter_testmod_seq *it__iter) __ksym; +__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + const char *str__str) __ksym; +__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct bpf_timer *timer) __ksym; void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym; void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym; void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
Yonghong Song
reviewerPATCH v5 16/16 selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongAdd inline-asm based verifier tests that exercise stack argument validation logic directly.
Positive tests:
- subprog call with 6 arg's
- Two sequential calls to different subprogs (6-arg and 7-arg)
- Share a r11 store for both branches
- Sequential calls reuse stack arg's
Negative tests — verifier rejection:
- Read from uninitialized incoming stack arg slot
- Gap in outgoing slots: only r11-16 written, r11-8 missing
- Sub-8-byte stack arg write (4 bytes instead of 8)
- Write at r11-80, exceeding max 7 stack args
- Missing store on one branch with a shared store
Negative tests — pointer/ref tracking:
- Pruning type mismatch: one branch stores PTR_TO_STACK, the other stores a scalar, callee dereferences — must not prune
- Release invalidation: bpf_sk_release invalidates a socket pointer stored in a stack arg slot
- Packet pointer invalidation: bpf_skb_pull_data invalidates a packet pointer stored in a stack arg slot
- Null propagation: PTR_TO_MAP_VALUE_OR_NULL stored in stack arg slot, null branch attempts dereference via callee
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_stack_arg.c | 463 ++++++++++++++++++ 2 files changed, 465 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -91,6 +91,7 @@#include "verifier_sockmap_mutate.skel.h" #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" +#include "verifier_stack_arg.skel.h" #include "verifier_stack_ptr.skel.h" #include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h"
@@ -238,6 +239,7 @@ void test_verifier_sock_addr(void) { RUN(verifier_sock_addr); }void test_verifier_sockmap_mutate (void ) {RUN (verifier_sockmap_mutate); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }void test_verifier_spin_lock (void ) {RUN (verifier_spin_lock); } +void test_verifier_stack_arg (void ) {RUN (verifier_stack_arg); }void test_verifier_stack_ptr (void ) {RUN (verifier_stack_ptr); }void test_verifier_store_release (void ) {RUN (verifier_store_release); }void test_verifier_subprog_precision (void ) {RUN (verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +463 -0
463 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to Yonghong SongSashiko AI review found 2 potential issue(s): - [Critical] False negative allowing tail calls and stack arguments together, leading to JIT memory corruption - [High] State leak in verifier DFS…
Sashiko AI review found 2 potential issue(s):
- [Critical] False negative allowing tail calls and stack arguments together, leading to JIT memory corruption
- [High] State leak in verifier DFS causing false positive rejections of valid BPF programs --
commit df24720ae892549300f1910df635621a3875dbdf
Author: Yonghong Song <[email protected]>
bpf: Reject stack arguments if tail call reachable
This commit updates the verifier to reject BPF programs if they contain both tail calls and stack arguments. This is enforced during the maximum stack depth check traversal.
Quoted 2 lines
Tailcalls are been deprecated. So reject stack arguments if tail call is in the way.
This isn't a bug, but there is a grammatical error here. "are been" should likely be "have been".
kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5669
Quoted 13 lines
kernel/bpf/verifier.c +5
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + }
Looking at the earlier parts of check_max_stack_depth_subprog(), the local variable tail_call_reachable only appears to be set to true when jumping into a callee that has a tail call:
if (subprog[idx].has_tail_call)
tail_call_reachable = true;If the main program (subprog[0]) natively contains a tail call, tail_call_reachable would remain false, causing this entire check block to be skipped.
Does this logic successfully catch the case where the main program itself contains a tail call and also uses stack arguments?
If this escapes the verifier, the JIT might emit prologue tail call instructions (pushing 24 bytes on x86) while calculating stack argument offsets assuming no tail call pushes, overwriting the BPF program stack.
Could this regression lead to memory corruption in the JIT?
When the traversal pops a frame to return to the caller and jumps back to continue_func, the tail_call_reachable variable is not reset to false:
/* pop caller idx from callee */
idx = dinfo[idx].caller;
...
goto continue_func;If the verifier traverses a subprogram with a tail call, returns, and then traverses a completely unrelated sibling subprogram that uses stack arguments, it seems it would incorrectly evaluate tail_call_reachable as true and reject the valid sibling.
Is there a state leak in this DFS traversal that could cause false positive rejections of valid BPF programs?
Quoted 3 lines
kernel/bpf/verifier.c +0
-0subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
[email protected]
AI reviewRe: [PATCH bpf-next v5 14/16] selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongSashiko AI review found 1 potential issue(s): - [Medium] The `test_async_cb_many` test fails to verify the execution and result of the asynchronous timer callback. --
Sashiko AI review found 1 potential issue(s):
- [Medium] The test_async_cb_many test fails to verify the execution and result of the asynchronous timer callback. --
commit ef139a03093697d9985422f1f900de9a12f9a0dc
Author: Yonghong Song <[email protected]>
selftests/bpf: Add tests for BPF function stack arguments
This commit adds selftests for BPF-to-BPF subprog and kfunc calls that utilize more than 5 arguments, verifying stack argument passing. It introduces test scenarios for various stack argument types, including scalars, pointers, and async callbacks.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +0
-0
[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +7
-0+static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ;
tools/testing/selftests/bpf/prog_tests/stack_arg.c line 62
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +10
-0+ +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 );
Does this test verify the asynchronous callback execution and its result?
Since run_subtest() only runs the program and asserts the return value is 0, the skeleton is immediately destroyed afterward. The timer callback might not have time to execute and could be canceled during map destruction.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +4
-0+ +out: +stack_arg__destroy (skel); +}
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/stack_arg.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/stack_arg.c line 188
Quoted 5 lines
tools/testing/selftests/bpf/progs/stack_arg.c new file +5
-0+static int timer_cb_many_args (void *map,int *key,struct bpf_timer *timer) +{ + timer_result =static_func_many_args (10 ,20 ,30 ,40 ,50 ,60 ,70 ,80 ); +return 0 ; +}
The result of the callback (timer_result) is never verified in userspace. Should there be a polling loop waiting for the timer to fire in the test, followed by an assertion checking that skel->bss->timer_result == 360?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
[email protected]
AI reviewRe: [PATCH bpf-next v5 15/16] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to Yonghong SongSashiko AI review found 2 potential issue(s): - [Medium] The `prog_test_big_arg` struct uses `long`, which fails to test the >8 byte limit on 32-bit architectures and causes BTF size mismatches.…
Sashiko AI review found 2 potential issue(s):
- [Medium] The prog_test_big_arg struct uses long, which fails to test the >8 byte limit on 32-bit architectures and causes BTF size mismatches.
- [Medium] Using ASSERT_ERR(stack_arg_fail__load(skel)) masks the actual verifier error reason, allowing the negative test to pass for unrelated reasons. --
commit 1ec644adfded3aa88befcb32fc73958c68b7e018
Author: Yonghong Song <[email protected]>
selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
This commit adds a selftest to ensure the verifier rejects kfunc calls where a stack argument exceeds the 8-byte register-sized slot limit. It introduces a new kfunc taking a struct argument and a BPF program that attempts to pass a large struct to it.
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_pass2 { · line 57
Quoted 13 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5
-0@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +
Does this structure achieve the intended >8 byte size on 32-bit architectures?
When the test kernel module is compiled on a 32-bit host, long is 32-bit, making the struct exactly 8 bytes. Since the BPF target always treats long as a 64-bit integer, this makes the struct 16 bytes in the BPF program.
This size discrepancy between the BPF program's BTF and the host kernel's BTF might cause the verifier to reject the load for a BTF size mismatch rather than the intended stack argument limit.
Could these fields use explicitly sized types like __u64 to ensure the struct is consistently larger than 8 bytes across all architectures?
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +0
-0@@ -0,0 +1,24 @@
[ ... ]
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c line 20
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +6
-0+if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" );
Could this generic error assertion mask the actual verifier rejection reason?
If the program fails to load due to an unrelated issue, such as a missing BTF, a syntax error, or the architecture-dependent size mismatch described above, the test will incorrectly report success because it only checks that the load operation failed.
Would it be better to refactor this test to use the standard test_loader infrastructure (via RUN_TESTS) with __failure and __msg annotations in the BPF program? That would ensure the rejection is explicitly due to the argument size limit.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +4
-0+ +out: +stack_arg_fail__destroy (skel); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
[email protected]
AI reviewRe: [PATCH bpf-next v5 07/16] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongIs this description of the calling convention accurate? The comment states "Args 1-5 are passed in registers, args 6-12 via stack arg slots," but doesn't argument 6 get passed in the R9 register…
include/linux/bpf.h struct bpf_prog_offload { · line 1157
Quoted 11 lines
include/linux/bpf.h +4
-0@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots.
Is this description of the calling convention accurate? The comment states "Args 1-5 are passed in registers, args 6-12 via stack arg slots," but doesn't argument 6 get passed in the R9 register rather than on the stack?
Looking at arch/x86/net/bpf_jit_comp.c, the existing comment states: "x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 are passed through regs, the remains are through stack."
The follow-up commit 0ef907c3ec2c in this patch series ("bpf,x86: Implement JIT support for stack arguments") documents the actual calling convention as:
"Follow x86-64 calling convention for both BPF-to-BPF and kfunc calls: - Arg 6 is passed in the R9 register - Args 7+ are passed on the stack at [rsp]"
Should the comment say "Args 1-5 are passed in registers, arg 6 in R9 register, args 7-12 via stack arg slots" instead?
include/linux/bpf.h struct bpf_prog_offload { · line 1159
Quoted 2 lines
include/linux/bpf.h +0
-0*/ #define MAX_BPF_FUNC_ARGS 12
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 09/16] bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong Songcommit 6f3447bea13f86a674e917f5f8ef4474cdd3b0bc Author: Yonghong Song <[email protected]>
commit 6f3447bea13f86a674e917f5f8ef4474cdd3b0bc
Author: Yonghong Song <[email protected]>
bpf: Reject stack arguments in non-JITed programs
The interpreter does not understand the bpf register r11 (BPF_REG_PARAMS) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.
This commit fixes a bug but lacks a Fixes: tag.
Suggested: Fixes: 21229535935384b01e3cef3c117233b23b2f3902 ("bpf: Support stack arguments for bpf functions")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to Yonghong SongThe commit message has a grammatical error: "Tailcalls are been deprecated" should be either "have been" or "are being".
Quoted 8 lines
commit
b77ef401ea17eab379f390ca5d89aeab4b271ca7Author: Yonghong Song <[email protected]> Date: Thu Apr 16 20:47:49 2026 -0700bpf: Reject stack arguments if tail call reachable
Tailcalls are been deprecated. So reject stack arguments if tail call is in the way.
The commit message has a grammatical error: "Tailcalls are been deprecated" should be either "have been" or "are being".
Also, is the claim that tail calls are deprecated accurate? Based on earlier mailing list discussion, the restriction exists because tail calls and stack arguments require incompatible stack frame handling, not because tail calls are being deprecated. Could the commit message be more precise about the actual technical reason for the restriction?
This also appears to be a bug fix for the stack arguments feature. Should this include a Fixes tag for the original stack arguments commit?
Suggested: Fixes: 21229535935384b01e3cef3c117233b23b2f3902 ("bpf: Support stack arguments for bpf functions")
Quoted 9 lines
Signed-off-by Yonghong Song <[email protected]>kernel/bpf/verifier.c +0
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );
kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5672
Quoted 10 lines
kernel/bpf/verifier.c +5
-0return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + } subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
When the traversal pops a frame to return to the caller and jumps back to continue_func, is the tail_call_reachable variable reset to false?
Looking at the broader context of check_max_stack_depth_subprog(), after processing a callee, the code pops back to the caller:
/* pop caller idx from callee */
idx = dinfo[idx].caller;
...
goto continue_func;If the verifier traverses a subprogram with a tail call, returns, and then traverses a completely unrelated sibling subprogram that uses stack arguments, would it incorrectly evaluate tail_call_reachable as true and reject the valid sibling?
Is there a state leak in this DFS traversal that could cause false positive rejections of valid BPF programs?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to Yonghong SongCan these instructions with BPF_REG_PARAMS trigger out-of-bounds accesses in subsequent verification functions? The regs array in struct bpf_func_state is declared as regs[MAX_BPF_REG] with…
Quoted 4 lines
kernel/bpf/const_fold.c +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/liveness.c +0
-0
[ ... ]
Quoted 21 lines
kernel/bpf/verifier.c +8
-2@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + } }if (insn->src_reg >=MAX_BPF_REG ) {
kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18508
Quoted 10 lines
kernel/bpf/verifier.c +5
-2-verbose (env,"R%d is invalid\n" , insn->src_reg); -return -EINVAL ; +if (insn->src_reg !=BPF_REG_PARAMS || class !=BPF_LDX || + mode !=BPF_MEM ) { +verbose (env,"R%d is invalid\n" , insn->src_reg); +return -EINVAL ; + } }if (insn[0 ].code == (BPF_LD |BPF_IMM |BPF_DW )) {struct bpf_insn_aux_data *aux;
Can these instructions with BPF_REG_PARAMS trigger out-of-bounds accesses in subsequent verification functions? The regs array in struct bpf_func_state is declared as regs[MAX_BPF_REG] with MAX_BPF_REG=11, giving valid indices 0-10.
When instructions with src_reg=BPF_REG_PARAMS (11) reach check_load_mem():
kernel/bpf/verifier.c:check_load_mem() {
...
src_reg_type = regs[insn->src_reg].type;
...
}This performs regs[11] which is one past the end of the array.
Similarly, when instructions with dst_reg=BPF_REG_PARAMS reach __check_reg_arg():
kernel/bpf/verifier.c:__check_reg_arg() {
...
reg = ®s[regno];
...
}The commit message states "array-based register tracking indexed by register number would cause out-of-bounds accesses. So do early return if needed." However, the patch adds early returns in const_fold.c and liveness.c but not in the main verification path functions like check_load_mem() or __check_reg_arg().
The fix appears to arrive in subsequent commit 212295359353 ("bpf:
Support stack arguments for bpf functions"). Between this commit and that fix, could user-provided BPF programs with BPF_REG_PARAMS instructions trigger out-of-bounds memory access in the kernel verifier?
This commit also lacks a Fixes: tag. The suggested tag would be:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 12/16] bpf: Enable stack argument support for x86_64
In reply to Yonghong Songcommit 96653dbc686aeae7abf4f70b15720267e083674d Author: Yonghong Song <[email protected]>
commit 96653dbc686aeae7abf4f70b15720267e083674d
Author: Yonghong Song <[email protected]>
bpf: Enable stack argument support for x86_64
Add stack argument support for x86_64.
This commit fixes a bug but lacks a Fixes: tag. The commit adds JIT capability checks (bpf_jit_supports_stack_args) before allowing stack arguments, which was missing when stack argument support was originally introduced.
Commit 212295359353 added code in btf.c that unconditionally set incoming_stack_arg_depth when nargs > MAX_BPF_FUNC_REG_ARGS without checking if the JIT actually supports stack arguments. Similarly, commit 173008b815e6 extended this to kfuncs in verifier.c without JIT capability checks. This could lead to programs with stack arguments being incorrectly accepted on architectures that don't support them.
Suggested: Fixes: 21229535935384 ("bpf: Support stack arguments for bpf functions")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 08/16] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongSashiko AI review found 5 potential issue(s): - [Critical] Verifier State Leakage in `copy_stack_state` when reusing states. - [Critical] Security / Liveness Tracking Bypass via `stack_arg_regs`.…
Sashiko AI review found 5 potential issue(s):
- [Critical] Verifier State Leakage in copy_stack_state when reusing states.
- [Critical] Security / Liveness Tracking Bypass via stack_arg_regs.
- [Critical] Use-After-Free via Type Mismatch in bpf_get_spilled_stack_arg().
- [Medium] Unnecessary Rejection of Valid Programs due to Dead Code Elimination.
- [Low] Missing Parentheses in bpf_get_spilled_stack_arg Macro. --
commit 6ec4a3fcb5e83a95626cd2bb13a60b2283e1c3e2
Author: Yonghong Song <[email protected]>
bpf: Support stack arguments for bpf functions
This commit adds support for passing more than 5 arguments to BPF subprograms by utilizing a dedicated stack area accessed via register r11. It introduces new fields to track incoming and outgoing stack arguments for bounds checking and precise verifier path tracking.
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 519
Quoted 11 lines
include/linux/bpf_verifier.h +4
-0@@ -508,6 +513,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL)
Is there a type mismatch in this check?
The mask parameter here is passed as 1 << STACK_SPILL, but frame->stack_arg_regs[slot].type is an enum bpf_reg_type, not an enum bpf_stack_slot_type.
For pointer types, this expression can evaluate to 0. This would cause the iterator to silently skip pointers stored in the stack argument area. If an acquired reference is dropped via release_reference(), the copy in the stack argument area wouldn't be invalidated, potentially allowing a use-after-free if a program subsequently reads the spilled pointer back from r11.
Also, this isn't a bug, but should the slot parameter be enclosed in parentheses to prevent operator precedence issues if a complex expression is passed in the future?
[ ... ]
Quoted 4 lines
kernel/bpf/fixups.c +0
-0@@ -1266,9 +1275,20 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
[ ... ]
kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1290
Quoted 10 lines
kernel/bpf/fixups.c +10
-0+for (i =0 ; i < env->subprog_cnt; i++) { +struct bpf_subprog_info *subprog = &env->subprog_info[i]; + +if (subprog->max_out_stack_arg_depth > subprog->outgoing_stack_arg_depth) { +verbose (env, +"func#%d writes stack arg slot at depth %u, but calls only require %u bytes\n" , + i, subprog->max_out_stack_arg_depth, subprog->outgoing_stack_arg_depth); +return -EINVAL ; + } + }
Could this reject valid programs if the compiler hoists a stack argument store above a branch that is later eliminated as dead code?
If the compiler optimizes a store above a conditional check, max_out_stack_arg_depth would be updated. However, if the configuration constants cause the branch containing the actual function call to be eliminated, outgoing_stack_arg_depth would remain 0, leading to a rejection of a safe verification path.
[ ... ]
Quoted 13 lines
kernel/bpf/verifier.c +6
-0@@ -1340,6 +1340,20 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_streturn -ENOMEM ; dst->allocated_stack = src->allocated_stack; + +/* copy stack args state */ + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, + sizeof(struct bpf_reg_state),
kernel/bpf/verifier.c static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · line 1358
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+ GFP_KERNEL_ACCOUNT); +if (!dst->stack_arg_regs) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + dst->out_stack_arg_mask = src->out_stack_arg_mask; + }return 0 ; }
Does this skip clearing the destination's stack argument fields if the source state has no stack arguments?
If src->stack_arg_depth is 0, dst->stack_arg_depth, out_stack_arg_mask, and incoming_stack_arg_depth are not updated. When the verifier reuses an existing state during pop_stack() via bpf_copy_verifier_state(), the destination state might incorrectly inherit a stale stack argument initialization mask from a previously evaluated path.
Could this allow a program to initialize stack arguments on one branch, and then safely perform a function call on a different branch without proper initialization?
[ ... ]
Quoted 1 line
kernel/bpf/verifier.c +0
-0@@ -4424,6 +4467,123 @@ static int check_stack_write(struct bpf_verifier_env *env,
[ ... ]
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{
[ ... ]
Quoted 22 lines
kernel/bpf/verifier.c +22
-0+ cur = env->cur_state->frame[env->cur_state->curframe]; +if (value_regno >=0 ) { + state->stack_arg_regs[spi] = cur->regs[value_regno]; + }else { +/* BPF_ST: store immediate, treat as scalar */ +struct bpf_reg_state *arg = &state->stack_arg_regs[spi]; + + arg->type =SCALAR_VALUE ; +__mark_reg_known (arg, env->prog->insnsi[env->insn_idx].imm); + } + state->out_stack_arg_mask |=BIT (idx); +return 0 ; +} + +/* +* Read a value from the incoming stack arg area. +* off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7). +* Callers ensure off > 0, 8-byte aligned, and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_read (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int dst_regno) +{
[ ... ]
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 4585
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+ arg = &state->stack_arg_regs[spi]; + cur = env->cur_state->frame[env->cur_state->curframe]; + +if (is_spillable_regtype (arg->type)) +copy_register_state (&cur->regs[dst_regno], arg); +else +mark_reg_unknown (env, cur->regs, dst_regno); +return 0 ; +}
Does this bypass liveness tracking for stack arguments?
Because the state is copied directly without calling mark_reg_read(), the live field of stack arguments is never marked as read.
During state pruning, stack_arg_safe() delegates to regsafe(), which contains a fast-path that unconditionally returns true if the old state's register was never marked as read. Could this cause the verifier to aggressively prune verification paths even if they contain completely different and potentially unsafe stack arguments?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
[email protected]
AI reviewRe: [PATCH bpf-next v5 16/16] selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongSashiko AI review found 3 potential issue(s): - [Medium] Negative verifier tests for stack arguments use inherently invalid memory accesses, causing them to pass even if the verifier has bugs.…
Sashiko AI review found 3 potential issue(s):
- [Medium] Negative verifier tests for stack arguments use inherently invalid memory accesses, causing them to pass even if the verifier has bugs.
- [Medium] stack_arg_read_uninitialized reads from r11 + 8 instead of a valid negative offset, failing to properly test uninitialized stack argument slots.
- [Low] Unused subprog_8args function defined in the test file.
--
commit 24b1eb352e7d04e7c35a9fbc88c6b91d08b17832
Author: Yonghong Song <[email protected]>
selftests/bpf: Add verifier tests for stack argument validation
This commit adds inline-asm based verifier tests to directly exercise stack argument validation logic.
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 33
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+__noinline __used +static int subprog_8args(int a, int b, int c, int d, int e, int f, int g, int h) +{ +return a + b + c + d + e + f + g + h; +}
Is subprog_8args intended to be used in any of the tests? It does not seem to be called anywhere in this file.
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +4
-0+SEC("tc") +__description("stack_arg: read from uninitialized stack arg slot") +__failure +__arch_x86_64
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 102
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+__msg("invalid read from stack arg") +__naked void stack_arg_read_uninitialized (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r11 + 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all + ); +}
Should this read from r11 - 8 instead? Since outgoing stack arguments are at negative offsets from r11, reading from r11 + 8 is out of bounds and might be rejected for the wrong reason, rather than testing an uninitialized but valid stack argument slot.
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+__noinline __used +static long subprog_deref_arg6(long a, long b, long c, long d, long e, long *f) +{ +return *f; +}
[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +8
-0+SEC("tc") +__description("stack_arg: release_reference invalidates stack arg slot") +__failure +__arch_x86_64 +__msg("R1 invalid sock access") +__naked void stack_arg_release_ref (void ) +{ + asm volatile (
[ ... ]
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 220
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* Release the reference — invalidates the stack arg slot */ +"r1 = r0;" +"call %[bpf_sk_release];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;"
Can this test pass even if the verifier fails to invalidate the stack slot?
subprog_deref_arg6() performs an 8-byte read on the argument. Since BPF sockets only allow up to 4-byte reads, an 8-byte read is always invalid. If the reference isn't correctly invalidated, the valid socket pointer will still be rejected with "R1 invalid sock access".
To properly test the invalidation, should the subprogram perform a valid 4-byte read instead?
[ ... ]
Quoted 18 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +18
-0+SEC("tc") +__description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data") +__failure +__arch_x86_64 +__msg("invalid access to packet") +__naked void stack_arg_stale_pkt_ptr (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = *(u32 *)(r6 + %[__sk_buff_data]);" +"r8 = *(u32 *)(r6 + %[__sk_buff_data_end]);" +/* check pkt has at least 1 byte */ +"r0 = r7;" +"r0 += 1;" +"if r0 > r8 goto l0_%=;" +/* Store valid pkt pointer to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r7;" + /* bpf_skb_pull_data invalidates all pkt pointers */
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 259
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+"r1 = r6;" +"r2 = 0;" +"call %[bpf_skb_pull_data];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;"
Similarly, does this properly test packet pointer invalidation? The test verifies 1 byte of the packet pointer, but subprog_deref_arg6() reads 8 bytes. If bpf_skb_pull_data() fails to invalidate the pointer, the 8-byte read would still be rejected as out-of-bounds, yielding the expected "invalid access to packet" message.
Should the packet bounds check verify at least 8 bytes so that a valid read is guaranteed if the pointer isn't invalidated?
[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +8
-0+SEC("tc") +__description("stack_arg: null propagation rejects deref on null branch") +__failure +__arch_x86_64 +__msg("R1 invalid mem access") +__naked void stack_arg_null_propagation_fail (void ) +{ + asm volatile (
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+"call %[bpf_map_lookup_elem];" +/* Store PTR_TO_MAP_VALUE_OR_NULL to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r0;" +/* null check on r0 */ +"if r0 != 0 goto l0_%=;"
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 299
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* +* On null branch, outgoing slot is SCALAR(0). +* Call subprog that dereferences arg6 — should fail. +*/ +"r1 = 0;" +"r2 = 0;" +"r3 = 0;" +"r4 = 0;" +"r5 = 0;" +"call subprog_deref_arg6;"
Does the __msg() substring match accept the wrong failure mode here? Dereferencing an unrefined PTR_TO_MAP_VALUE_OR_NULL yields "R1 invalid mem access 'map_value_or_null'", while dereferencing SCALAR yields "R1 invalid mem access 'scalar'".
The substring match "R1 invalid mem access" accepts both, allowing the test to pass even if null propagation fails and the register is still tracked as a map pointer. Should it match the exact scalar error, or perhaps test the non-null branch for success instead?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
[email protected]
AI reviewRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongSashiko AI review found 3 potential issue(s): - [Critical] Precision tracking (`mark_chain_precision`) is incorrectly bypassed or misapplied for kfunc scalar arguments passed via the stack, allowing…
Sashiko AI review found 3 potential issue(s):
- [Critical] Precision tracking (mark_chain_precision) is incorrectly bypassed or misapplied for kfunc scalar arguments passed via the stack, allowing malicious BPF programs to bypass verifier safety checks through state pruning.
- [High] get_kfunc_arg_reg() fetches lookahead stack arguments without bounds checking, leading to an out-of-bounds memory read or NULL pointer dereference when resolving memory sizes.
- [High] Overwriting insn->off in bpf_fixup_kfunc_call destroys the BTF file descriptor index, breaking module kfunc calls on architectures that use JIT far calls.
--
commit 15b2d48bd7c1080e972b83211cc9794ac2ba763f
Author: Yonghong Song <[email protected]>
bpf: Support stack arguments for kfunc calls
Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r11-based stack arg area.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 6 lines
kernel/bpf/verifier.c +2
-0@@ -11653,6 +11659,19 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{
kernel/bpf/verifier.c bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · line 11673
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi =out_arg_spi (caller, argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_regs[spi]; +}
Does get_kfunc_arg_reg() need to verify that spi is within bounds and the stack slot is initialized?
When verifying pointer arguments that have a trailing size argument (KF_ARG_PTR_TO_MEM_SIZE), the verifier looks ahead at argno + 1. If a BPF program maliciously omits the stack size argument, could spi exceed caller->stack_arg_depth and lead to an out-of-bounds memory read?
[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 12442
Quoted 12 lines
kernel/bpf/verifier.c +5
-3@@ -12394,9 +12435,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ reg_arg_name(env, argno));return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (regno >0 ) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + }
Stack arguments use regno = -1, so it seems mark_chain_precision() is bypassed for scalar constants and size arguments passed on the stack.
If precision tracking is bypassed, could the verifier's state equivalence logic consider states with different scalar values in these stack slots as equivalent and incorrectly prune them?
[ ... ]
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20220
Quoted 13 lines
kernel/bpf/verifier.c +9
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ;
For architectures where bpf_jit_supports_far_kfunc_call() returns true (such as s390x, loongarch, and powerpc), they resolve the address later using bpf_get_kfunc_addr(), which requires the BTF file descriptor index stored in insn->off.
If insn->off is overwritten here, could it destroy the BTF file descriptor index and break module kfunc calls during JIT compilation?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
[email protected]
AI reviewRe: [PATCH bpf-next v5 08/16] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongDoes this macro work correctly? The existing bpf_get_spilled_reg macro filters on frame->stack[slot].slot_type[], which is drawn from the STACK_* enum (STACK_SPILL, STACK_ITER, etc.) and callers…
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0
[ ... ]
Quoted 5 lines
include/linux/bpf_verifier.h +1
-0@@ -508,6 +513,17 @@ struct bpf_func_state {u32 callback_depth;u32 async_entry_cnt; +/* Stack argument tracking for current function. */
include/linux/bpf_verifier.h struct bpf_func_state { · line 526
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+struct bpf_reg_state *stack_arg_regs;/* Stack arg register states */ +u32 stack_arg_depth;/* Depth of stack_arg_regs array */ +u32 incoming_stack_arg_depth;/* Incoming stack arg count (callee) */ +u64 out_stack_arg_mask;/* Outgoing stack arg mask (caller) */ +}; + +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL)
Does this macro work correctly? The existing bpf_get_spilled_reg macro filters on frame->stack[slot].slot_type[], which is drawn from the STACK_* enum (STACK_SPILL, STACK_ITER, etc.) and callers pass masks like (1 << STACK_SPILL) | (1 << STACK_ITER).
This new macro filters on bpf_reg_state.type which is from a completely different enum bpf_reg_type (SCALAR_VALUE, PTR_TO_BTF_ID, etc.) with type modifier bits (PTR_MAYBE_NULL = BIT(8), PTR_TRUSTED = BIT(20), etc.).
For example, check_kfunc_call() in the rcu_unlock path uses:
u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
...
bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg,
clear_mask, ...);A stack-arg slot holding a MEM_RCU-tagged register would have reg->type including MEM_RCU (BIT(13)). Then (1 << 13) = 0x2000, and 0x2000 & clear_mask (0x22) = 0, so the slot is skipped and MEM_RCU is not cleared on rcu_unlock.
Additionally, when n >= 32 (for modifier bits like PTR_TRUSTED at BIT(20)), the shift (1 << n) invokes undefined behavior in C.
Should this use a different approach to filter stack arg slots, perhaps based on initialization status (type != NOT_INIT)?
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 8 lines
kernel/bpf/verifier.c +4
-0@@ -1340,6 +1340,20 @@ static int copy_stack_state(struct bpf_func_state *dst, goto out_free_stack; } + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, + sizeof(struct bpf_reg_state),
kernel/bpf/verifier.c static int copy_stack_state(struct bpf_func_state *dst, · line 1356
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ GFP_KERNEL_ACCOUNT); +if (!dst->stack_arg_regs) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + dst->out_stack_arg_mask = src->out_stack_arg_mask; + } + +return 0 ;
What happens when src->stack_arg_depth == 0 but dst has non-zero values from prior work?
The stack_arg_depth, incoming_stack_arg_depth, out_stack_arg_mask, and stack_arg_regs fields come after the 'stack' field in struct bpf_func_state, so the preceding memcpy(dst, src, offsetof(..., stack)) in copy_func_state does not overwrite them.
When n == 0, the assignments to dst->stack_arg_depth, dst->incoming_stack_arg_depth, and dst->out_stack_arg_mask are skipped, leaving stale values in dst.
This can be triggered via:
1. Frame starts with stack_arg_depth = 0, out_stack_arg_mask = 0
2. check_cond_jmp_op calls push_stack which saves this state
3. Verifier continues on one branch; BPF_ST/BPF_STX writes to r11-8, r11-16 cause check_stack_arg_write to call grow_stack_arg_slots, setting stack_arg_depth = 16 and out_stack_arg_mask = BIT(0)|BIT(1)
4. Branch ends; pop_stack calls bpf_copy_verifier_state which calls copy_stack_state with src->stack_arg_depth == 0
5. Because n == 0, the assignment block is skipped, and dst retains stack_arg_depth = 16 and out_stack_arg_mask = BIT(0)|BIT(1)
6. The other branch never writes r11-8 or r11-16, but when it calls a subprog, set_callee_state checks:
if ((caller->out_stack_arg_mask & req_mask) != req_mask) The stale mask makes this check pass incorrectly
Would the verifier accept a call with uninitialized stack args on that second branch? Should these fields be assigned unconditionally, similar to how allocated_stack is handled earlier in this function?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongIs insn->off really "no longer needed for BTF fd index" after this point? Looking at bpf_jit_find_kfunc_model() in kernel/bpf/fixups.c, the function is called during JIT compilation from every BPF…
Quoted 3 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr);
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20221
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +
Is insn->off really "no longer needed for BTF fd index" after this point? Looking at bpf_jit_find_kfunc_model() in kernel/bpf/fixups.c, the function is called during JIT compilation from every BPF JIT backend that supports kfunc calls (x86_32, s390, riscv64, ppc64, loongarch).
It performs a bsearch on the kfunc_tab keyed by (insn->imm, insn->off) using kfunc_desc_cmp_by_imm_off:
kernel/bpf/fixups.c:kfunc_desc_cmp_by_imm_off() {
...
if (a->imm != b->imm)
return a->imm < b->imm ? -1 : 1;
if (a->offset != b->offset)
return a->offset < b->offset ? -1 : 1;
return 0;
}The desc->offset was populated from the original insn->off during bpf_add_kfunc_call(), where for module kfuncs it is a non-zero BTF fd index.
The call order is:
bpf_check()
-> bpf_do_misc_fixups()
-> bpf_fixup_kfunc_call() (overwrites insn->off to 0 or stack_arg_count)
-> bpf_prog_select_runtime()
-> bpf_int_jit_compile()
-> arch JIT
-> bpf_jit_find_kfunc_model() (reads insn->off for bsearch key)After this patch, for a kfunc with 5 or fewer args that lives in a module (non-zero btf_fd_idx), insn->off is zeroed, and the bsearch will fail to find the entry that was stored with offset=btf_fd_idx.
For kfuncs with more than 5 args, the new value of insn->off (stack arg count) also will not match desc->offset (btf_fd_idx or 0), so module kfuncs with stack args are also broken, and vmlinux kfuncs with more than 5 args get a nonzero insn->off that no longer matches desc->offset=0.
Can the stack arg count be communicated to the JIT without clobbering the btf_fd_idx that bsearch still keys on?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
[email protected]
AI reviewRe: [PATCH bpf-next v5 13/16] bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongSashiko AI review found 4 potential issue(s): - [High] Data corruption of Argument 6 across function calls due to caller-saved register clobbering. - [High] Missing size handling for Argument 6…
Sashiko AI review found 4 potential issue(s):
- [High] Data corruption of Argument 6 across function calls due to caller-saved register clobbering.
- [High] Missing size handling for Argument 6 allows verifier bounds tracking bypass and out-of-bounds memory writes.
- [High] The offset translation formula for outgoing arguments 7+ reverses the memory layout, corrupting sub-slot accesses.
- [Medium] The JIT misroutes stack accesses because it rigidly assumes LDX only targets incoming arguments and STX/ST only target outgoing arguments. --
commit 41dcf99330b6b0a3804185f2a284a76be8340ed1
Author: Yonghong Song <[email protected]>
bpf,x86: Implement JIT support for stack arguments
Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r11 (BPF_REG_PARAMS) in BPF bytecode, which the JIT translates to native code.
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -1664,16 +1692,45 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1706
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +5
-0if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Follow x86-64 calling convention for both BPF-to-BPF and +* kfunc calls: +* - Arg 6 is passed in R9 register +* - Args 7+ are passed on the stack at [rsp]
Can the use of the R9 register cause data corruption across function calls?
The x86-64 C calling convention defines R9 as a caller-saved register. However, the BPF verifier models argument 6 as stack memory, which implies its value should be preserved across calls.
If a BPF program reads incoming argument 6 after making a helper or kfunc call, will it receive a clobbered value instead of the original argument? Similarly, could a helper call scheduled by LLVM between the setup of outgoing argument 6 and the target call corrupt the R9 register?
[ ... ]
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1786
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +13
-0+/* +* Base offset from rbp for translating BPF outgoing args 7+ +* to native offsets. BPF uses negative offsets from r11 +* (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses +* positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for +* arg8, ...). Arg 6 goes to R9 directly. +* +* The translation reverses direction: +* native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 +* +* Note that tail_call_reachable is guaranteed to be false when +* stack args exist, so tcc pushes need not be accounted for. +*/
Does this formula correctly handle multi-byte sub-slot accesses?
Since the formula subtracts the BPF offset, the memory layout within the 8-byte slots appears mathematically reversed. While this correctly maps the 8-byte slot bases (e.g., -16 maps to rsp + 0), it seems to point to the wrong memory for partial writes.
For example, if a program writes to the upper 32 bits of Arg 7 at BPF offset -12, the formula yields BASE - (-12) - 16 = BASE - 4. Will this map the access out-of-bounds below the stack pointer instead of to [rsp + 4]?
[ ... ]
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +5
-1@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off))
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2240
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +9
-1-EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets. +* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + }
Are the size constraints missing for Argument 6 here?
The check for Arg 6 (insn->off == -8) is only implemented under the BPF_DW case. For sizes B, H, and W, execution will fall through to the st: label. This would incorrectly translate the access as a normal stack operation using the formula above, writing out-of-bounds below the stack pointer at [rsp - 8] instead of populating R9.
Additionally, for BPF_LDX, a 1-byte load emits a full 64-bit move to R9. Since the upper 56 bits of R9 are undefined for 1-byte arguments, could they contain garbage data? This would bypass the verifier's strict value bounds tracking, which assumes the loaded value is strictly within [0, 255].
[ ... ]
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2375
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ; +/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }
Does this logic misroute stack accesses by assuming LDX always targets incoming arguments and STX/ST always target outgoing arguments?
If a program writes an outgoing argument ([r11 - 16]) and then reads it back via LDX, the JIT appears to apply the incoming argument translation mapping the read to [rbp - 16] (the local stack) instead of [rsp] (the outgoing area). This would return uninitialized data.
Conversely, if a program modifies an incoming argument ([r11 + 16]) using STX, the JIT would apply the outgoing translation formula, yielding BASE - 16 - 16 = BASE - 32, which writes out-of-bounds below the stack pointer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
[email protected]
AI reviewRe: [PATCH bpf-next v5 03/16] bpf: Refactor to handle memory and size together
In reply to Yonghong SongSashiko AI review found 1 potential issue(s): - [High] Incomplete refactoring for stack arguments breaks verifier precision tracking, marking incorrect registers. --
Sashiko AI review found 1 potential issue(s):
- [High] Incomplete refactoring for stack arguments breaks verifier precision tracking, marking incorrect registers. --
commit 7bebb6c179745d67fba0c807428d095b66738135
Author: Yonghong Song <[email protected]>
bpf: Refactor to handle memory and size together
This commit refactors memory and size register handling in BPF helper verification, passing bpf_reg_state directly to prepare for stack arguments.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 9 lines
kernel/bpf/verifier.c +2
-2@@ -7020,15 +7020,17 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,/* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * -* @regno is the register containing the access size. regno-1 is the register -* containing the pointer. +* @mem_regno is the register containing the pointer, mem_regno+1 is the register +* containing the access size. */
kernel/bpf/verifier.c static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · line 7034
Quoted 10 lines
kernel/bpf/verifier.c +3
-1static int check_mem_size_reg (struct bpf_verifier_env *env, -struct bpf_reg_state *reg,u32 regno, +struct bpf_reg_state *mem_reg, +struct bpf_reg_state *size_reg,u32 mem_regno,enum bpf_access_type access_type,bool zero_size_allowed,struct bpf_call_arg_meta *meta) { +int size_regno = mem_regno +1 ;int err;
Since this refactoring is intended to support stack arguments, does this hardcoded size_regno = mem_regno + 1 break verifier precision tracking?
If the memory pointer is passed in R5, mem_regno is 5, making size_regno 6.
This would cause mark_chain_precision() to incorrectly backtrack the callee-saved register BPF_REG_6 instead of the stack slot containing the size argument.
Furthermore, if mem_regno is later modified to represent a 0-based argument index, size_regno will evaluate to the wrong register entirely.
[ ... ]
kernel/bpf/verifier.c static int check_mem_size_reg(struct bpf_verifier_env *env, · line 7074
Quoted 6 lines
kernel/bpf/verifier.c +1
-1- err =check_helper_mem_access (env, regno -1 , reg->umax_value, + err =check_helper_mem_access (env, mem_reg, mem_regno, size_reg->umax_value, access_type, zero_size_allowed, meta);kernel/bpf/verifier.c +1
-1if (!err) - err =mark_chain_precision (env, regno); + err =mark_chain_precision (env, size_regno);
Does marking the wrong register leave the actual size argument imprecise, potentially corrupting verifier state pruning and allowing out-of-bounds memory accesses to bypass verification?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to Yonghong SongSashiko AI review found 2 potential issue(s): - [Critical] Early returns in dataflow tracking passes skip necessary state updates for the destination register during BPF_LDX using BPF_REG_PARAMS.…
Sashiko AI review found 2 potential issue(s):
- [Critical] Early returns in dataflow tracking passes skip necessary state updates for the destination register during BPF_LDX using BPF_REG_PARAMS.
- [Critical] Permitting BPF_REG_PARAMS causes severe out-of-bounds memory accesses to the regs array during the main verifier loop (do_check_main). --
commit d68722297de575b838cc2f76141cf9c052e63448
Author: Yonghong Song <[email protected]>
bpf: Limit the scope of BPF_REG_PARAMS usage
This commit restricts the usage of BPF_REG_PARAMS (r11) to specific memory load and store instruction forms. It also adds early returns in dataflow tracking passes to prevent out-of-bounds accesses for stack arguments.
Quoted 4 lines
kernel/bpf/const_fold.c +0
-0
[ ... ]
Quoted 5 lines
kernel/bpf/const_fold.c +0
-1@@ -51,13 +51,18 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * struct bpf_insn *insn, struct bpf_insn *insns, int idx) { struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 }; -struct const_arg_info *dst = &ci_out[insn->dst_reg];
kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 62
Quoted 10 lines
kernel/bpf/const_fold.c +4
-1-struct const_arg_info *src = &ci_out[insn->src_reg]; +struct const_arg_info *dst, *src;u8 class =BPF_CLASS (insn->code);u8 mode =BPF_MODE (insn->code);u8 opcode =BPF_OP (insn->code) |BPF_SRC (insn->code);int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ;
If we return early here during a BPF_LDX instruction where src_reg is BPF_REG_PARAMS (11), doesn't this skip necessary state updates for dst_reg (e.g., registers 0 through 9)?
Since dst_reg is not marked as unknown, won't it retain its stale tracking state? Could bpf_prune_dead_branches() rely on this stale constant to incorrectly rewrite conditional jumps, allowing verifier safety checks to be bypassed?
Quoted 3 lines
kernel/bpf/const_fold.c +3
-0+ + dst = &ci_out[insn->dst_reg]; + src = &ci_out[insn->src_reg];
[ ... ]
Quoted 7 lines
kernel/bpf/liveness.c +0
-0@@ -1068,11 +1068,16 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,int depth = instance->depth;u8 class =BPF_CLASS (insn->code);
kernel/bpf/liveness.c static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 1077
Quoted 10 lines
kernel/bpf/liveness.c +4
-2u8 code =BPF_OP (insn->code); -struct arg_track *dst = &at_out[insn->dst_reg]; -struct arg_track *src = &at_out[insn->src_reg]; +struct arg_track *dst, *src;struct arg_track none = { .frame =ARG_NONE };int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ;
Similarly, does this early return skip clearing the arg_track state for the destination register when loading from a stack argument?
Quoted 3 lines
kernel/bpf/liveness.c +3
-0+ + dst = &at_out[insn->dst_reg]; + src = &at_out[insn->src_reg];
[ ... ]
Quoted 10 lines
kernel/bpf/verifier.c +2
-0@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code);
kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18499
Quoted 10 lines
kernel/bpf/verifier.c +6
-2+if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + } }
By permitting BPF_REG_PARAMS (which equals MAX_BPF_REG) to pass this validation check, won't this cause out-of-bounds memory accesses to the regs array (e.g., struct bpf_reg_state regs[MAX_BPF_REG]) during the main verifier loop?
When do_check_main() processes these instructions, functions like check_reg_arg() will attempt to access regs[11], which overlays verifier internal state.
I note this regression is fixed later in the patch series in commit 6ec4a3fcb5e83 ("bpf: Support stack arguments for bpf functions"), but it appears to introduce an issue in this intermediate patch.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH bpf-next v5 12/16] bpf: Enable stack argument support for x86_64
In reply to Yonghong SongSashiko AI review found 1 potential issue(s): - [High] Enabling `bpf_jit_supports_stack_args()` for x86_64 without the corresponding JIT implementation introduces a bisection-breaking regression…
Sashiko AI review found 1 potential issue(s):
- [High] Enabling bpf_jit_supports_stack_args() for x86_64 without the corresponding JIT implementation introduces a bisection-breaking regression that can lead to arbitrary kernel memory writes and ABI violations. --
commit b40feee41e4472d0e27be38b4c99c940abe3c1ba
Author: Yonghong Song <[email protected]>
bpf: Enable stack argument support for x86_64
This commit enables stack argument support for the x86_64 JIT and verifier to allow BPF programs and kfuncs to use more than 5 arguments.
arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_kfunc_call(void) · line 3944
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +5
-0@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +
Does enabling this without the actual JIT implementation break bisection?
If bpf_jit_supports_stack_args() returns true, the verifier will allow instructions using BPF_REG_PARAMS (register 11) to handle stack arguments. However, since the JIT support isn't implemented in this commit, BPF_REG_PARAMS is not yet defined in the JIT's reg2hex array.
Could this cause reg2hex[11] to implicitly default to 0, which maps to RAX (BPF_REG_0)? If the verifier emits instructions using BPF_REG_PARAMS, the x86 JIT might miscompile them using RAX as the base register, potentially allowing an arbitrary kernel memory write since R0 is controllable by the BPF program.
Additionally, it appears kfunc calls with >5 arguments won't have their arguments correctly marshaled to the System V ABI (like moving the 6th argument to R9), which could lead to reading uninitialized garbage from registers and the stack.
I see the JIT implementation is added later in the series in commit 41dcf99330b6b ("bpf,x86: Implement JIT support for stack arguments"). Should the JIT implementation be merged before or in the same patch as this feature enablement to maintain bisection?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
[email protected]
AI reviewRe: [PATCH bpf-next v5 15/16] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to Yonghong SongCould this generic error assertion mask the actual verifier rejection reason?
Quoted 16 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +10
-0@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "stack_arg_fail.skel.h" + +void test_stack_arg_fail (void ) +{ +struct stack_arg_fail *skel; +
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c line 20
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +10
-0+ skel =stack_arg_fail__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" );
Could this generic error assertion mask the actual verifier rejection reason?
If the program fails to load due to an unrelated issue, such as a missing BTF, a syntax error, or the architecture-dependent size mismatch described below, the test will incorrectly report success because it only checks that the load operation failed.
Would it be better to refactor this test to use the standard test_loader infrastructure (via RUN_TESTS) with __failure and __msg annotations in the BPF program? That would ensure the rejection is explicitly due to the argument size limit.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +4
-0+ +out: +stack_arg_fail__destroy (skel); +}
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_pass2 { · line 56
Quoted 12 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +4
-0@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +};
Does this structure achieve the intended >8 byte size on 32-bit architectures?
When the test kernel module is compiled on a 32-bit host, long is 32-bit, making the struct exactly 8 bytes. Since the BPF target always treats long as a 64-bit integer, this makes the struct 16 bytes in the BPF program.
This size discrepancy between the BPF program's BTF and the host kernel's BTF might cause the verifier to reject the load for a BTF size mismatch rather than the intended stack argument limit.
Could these fields use explicitly sized types like __u64 to ensure the struct is consistently larger than 8 bytes across all architectures?
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_pass2 { · line 60
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1
-0+struct prog_test_fail1 {void *p;int x;
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 16/16] selftests/bpf: Add verifier tests for stack argument validation
In reply to Yonghong SongIs subprog_8args intended to be used in any of the tests? It does not seem to be called anywhere in this file.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +0
-0
[ ... ]
Quoted 12 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +6
-0@@ -0,0 +1,463 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h"
[ ... ]
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 33
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+__noinline __used +static int subprog_8args(int a, int b, int c, int d, int e, int f, int g, int h) +{ +return a + b + c + d + e + f + g + h; +}
Is subprog_8args intended to be used in any of the tests? It does not seem to be called anywhere in this file.
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +4
-0+SEC("tc") +__description("stack_arg: read from uninitialized stack arg slot") +__failure +__arch_x86_64
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 102
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+__msg("invalid read from stack arg") +__naked void stack_arg_read_uninitialized (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r11 + 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all + ); +}
Should this read from r11 - 8 instead? Since outgoing stack arguments are at negative offsets from r11, reading from r11 + 8 is out of bounds and might be rejected for the wrong reason, rather than testing an uninitialized but valid stack argument slot.
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +7
-0+SEC("tc") +__description("stack_arg: release_reference invalidates stack arg slot") +__failure +__arch_x86_64 +__msg("R1 invalid sock access") +__naked void stack_arg_release_ref (void ) +{
[ ... ]
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 220
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* Release the reference — invalidates the stack arg slot */ +"r1 = r0;" +"call %[bpf_sk_release];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;"
Can this test pass even if the verifier fails to invalidate the stack slot? subprog_deref_arg6() performs an 8-byte read on the argument. Since BPF sockets only allow up to 4-byte reads, an 8-byte read is always invalid.
If the reference isn't correctly invalidated, the valid socket pointer will still be rejected with "R1 invalid sock access".
To properly test the invalidation, should the subprogram perform a valid 4-byte read instead?
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 18 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +18
-0+SEC("tc") +__description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data") +__failure +__arch_x86_64 +__msg("invalid access to packet") +__naked void stack_arg_stale_pkt_ptr (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = *(u32 *)(r6 + %[__sk_buff_data]);" +"r8 = *(u32 *)(r6 + %[__sk_buff_data_end]);" +/* check pkt has at least 1 byte */ +"r0 = r7;" +"r0 += 1;" +"if r0 > r8 goto l0_%=;" +/* Store valid pkt pointer to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r7;" + /* bpf_skb_pull_data invalidates all pkt pointers */
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 259
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+"r1 = r6;" +"r2 = 0;" +"call %[bpf_skb_pull_data];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;"
Similarly, does this properly test packet pointer invalidation? The test verifies 1 byte of the packet pointer, but subprog_deref_arg6() reads 8 bytes.
If bpf_skb_pull_data() fails to invalidate the pointer, the 8-byte read would still be rejected as out-of-bounds, yielding the expected "invalid access to packet" message.
Should the packet bounds check verify at least 8 bytes so that a valid read is guaranteed if the pointer isn't invalidated?
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 19 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +19
-0+SEC("tc") +__description("stack_arg: null propagation rejects deref on null branch") +__failure +__arch_x86_64 +__msg("R1 invalid mem access") +__naked void stack_arg_null_propagation_fail (void ) +{ + asmvolatile ( +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +/* r0 = bpf_map_lookup_elem(&map_hash_8b, &key) */ +"r2 = r10;" +"r2 += -8;" +"r1 = %[map_hash_8b] ll;" +"call %[bpf_map_lookup_elem];" +/* Store PTR_TO_MAP_VALUE_OR_NULL to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r0;" +/* null check on r0 */ + "if r0 != 0 goto l0_%=;"
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 299
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* +* On null branch, outgoing slot is SCALAR(0). +* Call subprog that dereferences arg6 — should fail. +*/ +"r1 = 0;" +"r2 = 0;" +"r3 = 0;" +"r4 = 0;" +"r5 = 0;" +"call subprog_deref_arg6;"
Does the __msg() substring match accept the wrong failure mode here? Dereferencing an unrefined PTR_TO_MAP_VALUE_OR_NULL yields "R1 invalid mem access 'map_value_or_null'", while dereferencing SCALAR yields "R1 invalid mem access 'scalar'".
The substring match "R1 invalid mem access" accepts both, allowing the test to pass even if null propagation fails and the register is still tracked as a map pointer.
Should it match the exact scalar error, or perhaps test the non-null branch for success instead?
Reference: https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 09/16] bpf: Reject stack arguments in non-JITed programs
In reply to Yonghong SongShould this include a Fixes tag? The commit addresses a bug introduced in commit 13ee5d67a7c9 ("bpf: Support stack arguments for bpf functions"). Additionally, [email protected] noted in v5…
Quoted 8 lines
bpf: Reject stack arguments in non-JITed programs
The interpreter does not understand the bpf register r11 (
BPF_REG_PARAMS) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.Signed-off-by Yonghong Song <[email protected]>
Should this include a Fixes tag? The commit addresses a bug introduced in commit 13ee5d67a7c9 ("bpf: Support stack arguments for bpf functions"). Additionally, [email protected] noted in v5 review on lore that this commit fixes a bug but lacks the tag:
Suggested: Fixes: 13ee5d67a7c9 ("bpf: Support stack arguments for bpf functions")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 07/16] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to Yonghong SongThis isn't a bug, but the multi-line comment opening doesn't follow BPF subsystem style. Per bpf.md: 'Multi-line comments MUST have the opening /* on its own line'. The rule applies even if…
include/linux/bpf.h struct bpf_prog_offload { · line 1152
Quoted 7 lines
include/linux/bpf.h +1
-1@@ -1151,6 +1151,10 @@ struct bpf_prog_offload { -/* The longest tracepoint has 12 args. +/* The longest tracepoint has 12 args.
This isn't a bug, but the multi-line comment opening doesn't follow BPF subsystem style. Per bpf.md: 'Multi-line comments MUST have the opening /* on its own line'. The rule applies even if surrounding code uses the old style.
Should this be reformatted as:
/*
* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
* Also reuse this macro for maximum number of arguments a BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots.
*/
include/linux/bpf.h struct bpf_prog_offload { · line 1156
Quoted 4 lines
include/linux/bpf.h +3
-0* See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
Should this say "kfuncs" instead of "kfunc's"? The context is plural (referring to multiple functions), not possessive.
Additionally, [email protected] raised a concern in v5 about the calling convention description. The comment states "Args 1-5 are passed in registers, args 6-12 via stack arg slots," but argument 6 is actually passed in the R9 register, not on the stack.
Looking at arch/x86/net/bpf_jit_comp.c, the existing comment states:
"x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 are passed through regs, the remains are through stack."
The follow-up commit 0ef907c3ec2c in this patch series documents the actual calling convention as: "Follow x86-64 calling convention for both BPF-to-BPF and kfunc calls: - Arg 6 is passed in the R9 register - Args 7+ are passed on the stack at [rsp]"
Should the comment say "Args 1-5 are passed in registers, arg 6 in R9 register, args 7-12 via stack arg slots" instead?
include/linux/bpf.h struct bpf_prog_offload { · line 1160
Quoted 4 lines
include/linux/bpf.h +1
-0+* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 14/16] selftests/bpf: Add tests for BPF function stack arguments
In reply to Yonghong SongDoes this test adequately verify the asynchronous timer callback execution and its result?
Quoted 28 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +22
-0@@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +}
[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +11
-0+static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out;
tools/testing/selftests/bpf/prog_tests/stack_arg.c line 66
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +10
-0+ } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +}
Does this test adequately verify the asynchronous timer callback execution and its result?
Since run_subtest() only runs the program and asserts the return value is 0, the skeleton is immediately destroyed afterward. The timer callback might not have time to execute and could be canceled during map destruction.
The result of the callback (timer_result) is never verified in userspace. Should there be a polling loop waiting for the timer to fire, followed by an assertion checking that skel->bss->timer_result == 360?
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 03/16] bpf: Refactor to handle memory and size together
In reply to Yonghong SongThe refactoring passes size_reg as a parameter to prepare for stack arguments, but this code still hardcodes size_regno as mem_regno + 1. Does this assumption hold when stack arguments are used…
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 11 lines
kernel/bpf/verifier.c +2
-3@@ -7020,15 +7020,17 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,/* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * -* @regno is the register containing the access size. regno-1 is the register -* containing the pointer. +* @mem_regno is the register containing the pointer, mem_regno+1 is the register +* containing the access size. */ static int check_mem_size_reg(struct bpf_verifier_env *env, - struct bpf_reg_state *reg, u32 regno,
kernel/bpf/verifier.c static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · line 7036
Quoted 10 lines
kernel/bpf/verifier.c +3
-0+struct bpf_reg_state *mem_reg, + struct bpf_reg_state *size_reg, u32 mem_regno, enum bpf_access_type access_type, bool zero_size_allowed, struct bpf_call_arg_meta *meta) { + int size_regno = mem_regno + 1;int err;/* This is used to refine r0 return value bounds for helpers
The refactoring passes size_reg as a parameter to prepare for stack arguments, but this code still hardcodes size_regno as mem_regno + 1. Does this assumption hold when stack arguments are used, where the size might not be at mem_regno + 1?
Looking at how mark_chain_precision() is called later in this function:
kernel/bpf/verifier.c static int check_mem_size_reg(struct bpf_verifier_env *env, · line 7074
Quoted 5 lines
kernel/bpf/verifier.c +1
-1err =check_helper_mem_access (env, mem_reg, mem_regno, size_reg->umax_value, access_type, zero_size_allowed, meta);if (!err) - err =mark_chain_precision (env, regno); + err =mark_chain_precision (env, size_regno);
The incorrect size_regno value would cause mark_chain_precision() to mark the wrong register for precision tracking. For example, if the memory pointer is passed in R5 and the size is in a stack slot, mem_regno would be 5, making size_regno 6. This would incorrectly backtrack BPF_REG_6 instead of the stack slot containing the size argument.
Could this break verifier state pruning when the commit message's stated goal of handling "stack arguments as they may be beyond registers 1-5" is implemented?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 08/16] bpf: Support stack arguments for bpf functions
In reply to Yonghong SongThe shift count in this macro appears to be incorrect. The macro uses `1 << frame->stack_arg_regs[slot].type` where type is an `enum bpf_reg_type`.
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0
[ ... ]
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 519
Quoted 8 lines
include/linux/bpf_verifier.h +4
-0@@ -508,6 +513,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL)
^^^^
The shift count in this macro appears to be incorrect. The macro uses 1 << frame->stack_arg_regs[slot].type where type is an enum bpf_reg_type.
Looking at bpf_get_spilled_reg (which this macro is modeled after), it uses 1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1] where slot_type is an enum bpf_stack_slot_type with values in the range 0..7.
However, enum bpf_reg_type values frequently carry high bit flags like PTR_MAYBE_NULL (BIT(8)), PTR_UNTRUSTED (BIT(14)), PTR_TRUSTED (BIT(20)), and MEM_RCU (BIT(21)):
include/linux/bpf.h:
enum bpf_type_flag {
PTR_MAYBE_NULL = BIT(8),
...
PTR_UNTRUSTED = BIT(14),
...
PTR_TRUSTED = BIT(20),
MEM_RCU = BIT(21),
...
};When a stack arg has type PTR_TO_BTF_ID | MEM_RCU, the value is around 0x200010, and 1 << 0x200010 is undefined behavior (shift count >= width of int).
The concrete impact occurs where bpf_for_each_reg_in_vstate (line 541-554) walks all registers including stack args:
include/linux/bpf_verifier.h:
#define bpf_for_each_reg_in_vstate_mask(...)
bpf_for_each_spilled_reg(___j, __state, __reg, __mask) {
...
}
bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) {
if (!__reg)
continue;
(void)(__expr);
}At kernel/bpf/verifier.c:13438, this iterator is used with clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER) to strip MEM_RCU from all surviving pointers after rcu_read_unlock. For a stack arg with PTR_TO_BTF_ID | MEM_RCU, the mask test becomes(1 << (type & 31)) & 130 which does not match 130, so the macro returns NULL and the MEM_RCU stripping code never runs. MEM_RCU pointers passed as arguments 6+ remain tagged MEM_RCU across rcu_read_unlock, allowing later dereferences without PTR_UNTRUSTED semantics.
Could the mask test be rewritten to avoid bit-shifting an enum bpf_reg_type value? Perhaps using is_spillable_regtype() or a similar type predicate rather than a bit pattern?
Quoted 14 lines
kernel/bpf/verifier.c +6
-0@@ -1340,6 +1340,20 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_streturn -ENOMEM ; dst->allocated_stack = src->allocated_stack; + +/* copy stack args state */ + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, + sizeof(struct bpf_reg_state),
kernel/bpf/verifier.c static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · line 1358
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+ GFP_KERNEL_ACCOUNT); +if (!dst->stack_arg_regs) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + dst->out_stack_arg_mask = src->out_stack_arg_mask; + }return 0 ; }
Are dst->stack_arg_depth, dst->incoming_stack_arg_depth, and dst->out_stack_arg_mask updated when src->stack_arg_depth is 0?
copy_stack_state only updates those fields when n != 0. When src has no stack args, dst retains stale values from a prior copy.
This differs from the unconditional update of dst->allocated_stack a few lines above, and from copy_reference_state which always updates active_locks and acquired_refs.
The function copy_func_state() calls memcpy up to offsetof(struct bpf_func_state, stack). Since stack_arg_depth, incoming_stack_arg_depth, out_stack_arg_mask, and stack_arg_regs are all declared AFTER stack, memcpy does not copy them. copy_stack_state is the only place those fields are updated from src to dst.
The staleness is visible through pop_stack:
pop_stack() calls bpf_copy_verifier_state(cur, &head->st). cur->frame[i] may have out_stack_arg_mask/stack_arg_depth populated from a previously explored branch that called a stack-arg function. head->st is the saved state from before the branch, with mask=0 and stack_arg_depth=0.
For that frame, src->stack_arg_depth==0 so the copy is skipped, and cur keeps the prior branch's mask and stack_arg_regs.
The commit message promises "A per-state bitmask out_stack_arg_mask tracks which outgoing stack arg slots have been written on the current path... if one branch of a conditional writes arg6 but another does not, the mask correctly reflects the difference and the verifier rejects the uninitialized path."
After pop_stack, the other branch is processed on top of cur, where the mask still carries bits set by the taken branch.
Consequences in set_callee_state:
kernel/bpf/verifier.c:set_callee_state() {
...
if ((caller->out_stack_arg_mask & req_mask) != req_mask) {
for (i = 0; i < callee_incoming_slots; i++) {
if (caller->out_stack_arg_mask & BIT(i))
continue;
verbose(env, "stack arg#%d not properly initialized\n", ...);
return -EINVAL;
}
}
...
}With stale bits, this check can pass on a path where the current branch never wrote the required r11 offsets. The verifier then copies stale caller->stack_arg_regs[caller_spi] into the callee's incoming slot.
Similarly in btf_check_func_arg_match:
kernel/bpf/verifier.c:btf_check_func_arg_match() {
...
if (spi >= (caller->stack_arg_depth / BPF_REG_SIZE)) {
verbose(env, "stack %s not found in caller state\n", ...);
return -EINVAL;
}
...
}With stale caller->stack_arg_depth, this bound check can pass on paths where the actual live stack_arg_depth is 0.
Should these fields be updated unconditionally, even when n==0?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 12/16] bpf: Enable stack argument support for x86_64
In reply to Yonghong SongDoes enabling this without the actual JIT implementation break bisection?
arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_kfunc_call(void) · line 3944
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +5
-0@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +
Does enabling this without the actual JIT implementation break bisection?
If bpf_jit_supports_stack_args() returns true, the verifier will allow instructions using BPF_REG_PARAMS (register 11) to handle stack arguments. However, since the JIT support isn't implemented in this commit, BPF_REG_PARAMS is not yet defined in the JIT's reg2hex array.
Could this cause reg2hex[11] to implicitly default to 0, which maps to RAX (BPF_REG_0)? If the verifier emits instructions using BPF_REG_PARAMS, the x86 JIT might miscompile them using RAX as the base register, potentially allowing an arbitrary kernel memory write since R0 is controllable by the BPF program.
Additionally, it appears kfunc calls with >5 arguments won't have their arguments correctly marshaled to the System V ABI (like moving the 6th argument to R9), which could lead to reading uninitialized garbage from registers and the stack.
The JIT implementation appears in a later commit in the series. Should the JIT implementation be merged before or in the same patch as this feature enablement to maintain bisection?
This concern was raised by [email protected] in v5 review:
https://lore.kernel.org/bpf/[email protected]
arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_kfunc_call(void) · line 3947
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +0
-0void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to Yonghong SongCan the verifier crash here? Looking at get_kfunc_arg_reg, when argno >= MAX_BPF_FUNC_REG_ARGS, the function returns &caller->stack_arg_regs[spi] without checking whether stack_arg_regs is NULL.
Quoted 3 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · line 11670
Quoted 13 lines
kernel/bpf/verifier.c +12
-0@@ -11653,6 +11659,19 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi =out_arg_spi (caller, argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_regs[spi]; +}
Can the verifier crash here? Looking at get_kfunc_arg_reg, when argno >= MAX_BPF_FUNC_REG_ARGS, the function returns &caller->stack_arg_regs[spi] without checking whether stack_arg_regs is NULL.
The stack_arg_regs array is only allocated by grow_stack_arg_slots, which is called from check_stack_arg_write when the program writes outgoing stack args via r11. If a malformed BPF program invokes a kfunc without emitting any stack stores beforehand, stack_arg_regs will remain NULL.
Consider get_kfunc_ptr_arg_type calling get_kfunc_arg_reg to peek at argument i+1 when checking for buffer+__sz pairs:
kernel/bpf/verifier.c:get_kfunc_ptr_arg_type() {
if (argno + 1 < nargs &&
(is_kfunc_arg_mem_size(meta->btf, &args[argno + 1],
get_kfunc_arg_reg(env, argno + 1)) ||
...
}When argno=4 (pointer in r5) and argno+1=5 (size in first stack slot), get_kfunc_arg_reg(env, 5) enters the stack-arg branch and dereferences caller->stack_arg_regs even though it may be NULL. If the arg type is a scalar (which __sz/__szk suffixes imply), is_kfunc_arg_mem_size will try to read reg->type, causing a NULL pointer dereference.
The is_stack_arg_slot_initialized gate you added in check_kfunc_args only fires when the loop reaches i=5; by then the peek at i=4 has already dereferenced the NULL pointer.
Should get_kfunc_arg_reg verify that stack_arg_regs is non-NULL and spi < stack_arg_depth/BPF_REG_SIZE before returning the pointer? Alternatively, should check_kfunc_call add an out_stack_arg_mask pre-check mirroring the bpf-to-bpf gate that prevents check_func_arg_reg_off from reading uninitialized stack slots?
Quoted 18 lines
kernel/bpf/verifier.c +2
-4@@ -11660,8 +11679,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, const char *ref_tname, const struct btf_param *args, int argno, int nargs, struct bpf_reg_state *reg) { - u32 regno = argno + 1; -struct bpf_reg_state *regs =cur_regs (env); bool arg_mem_size = false; if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||@@ -11670,8 +11687,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,return KF_ARG_PTR_TO_CTX ;if (argno +1 < nargs && - (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]) || -is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]))) + (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )) || +is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )))) arg_mem_size = true;
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr);
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20221
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +
This concern was raised by the bpf-ci bot and sashiko-bot in the v5 review thread but does not appear to have been addressed.
Overwriting insn->off destroys the BTF file descriptor index required by bpf_jit_find_kfunc_model() for module kfunc lookups. Looking at the bsearch in kernel/bpf/fixups.c, it uses (insn->imm, insn->off) as the key:
kernel/bpf/fixups.c:bpf_jit_find_kfunc_model() {
key.addr = insn->imm;
key.offset = insn->off;
entry = bsearch(&key, tab, tab_count, sizeof(*tab), kfunc_addr_cmp);
}For module kfuncs, insn->off stores a non-zero btf_fd_idx set during find_kfunc_desc. After this patch overwrites it with the stack arg count (or 0), the bsearch will fail to find descriptors that were stored with offset=btf_fd_idx.
This breaks module kfunc calls on architectures using JIT far calls (s390x, loongarch, powerpc) for any module kfunc with 5 or fewer args (insn->off zeroed) or with more than 5 args (insn->off now contains the stack arg count instead of btf_fd_idx, still mismatching desc->offset). Vmlinux kfuncs with more than 5 args also get a nonzero insn->off that no longer matches desc->offset=0.
Can the stack arg count be communicated to the JIT without clobbering the btf_fd_idx that bsearch still keys on?
The original concern was raised here:
https://lore.kernel.org/bpf/e5a6e801c10994cc2e03b0cd57e76152eb234493a6b7395a9859feade4a6575b@mail.kernel.org/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to Yonghong SongIf we return early here during a BPF_LDX instruction where src_reg is BPF_REG_PARAMS (11), doesn't this skip necessary state updates for dst_reg (e.g., registers 0 through 9)?
Quoted 11 lines
kernel/bpf/const_fold.c +1
-2@@ -51,13 +51,18 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out, struct bpf_insn *insn, struct bpf_insn *insns, int idx) { struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 }; -struct const_arg_info *dst = &ci_out[insn->dst_reg]; -struct const_arg_info *src = &ci_out[insn->src_reg]; +struct const_arg_info *dst, *src;u8 class =BPF_CLASS (insn->code);
kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out, · line 65
Quoted 10 lines
kernel/bpf/const_fold.c +6
-0u8 mode =BPF_MODE (insn->code);u8 opcode =BPF_OP (insn->code) |BPF_SRC (insn->code);int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ; + + dst = &ci_out[insn->dst_reg]; + src = &ci_out[insn->src_reg];
If we return early here during a BPF_LDX instruction where src_reg is BPF_REG_PARAMS (11), doesn't this skip necessary state updates for dst_reg (e.g., registers 0 through 9)?
Since dst_reg is not marked as unknown, won't it retain its stale tracking state? Could bpf_prune_dead_branches() rely on this stale constant to incorrectly rewrite conditional jumps, allowing verifier safety checks to be bypassed?
A similar issue exists in liveness.c where the early return skips clearing the arg_track state for the destination register when loading from a stack argument.
Quoted 3 lines
kernel/bpf/const_fold.c +0
-0switch (class) {case BPF_ALU :case BPF_ALU64 :
[ ... ]
Quoted 18 lines
kernel/bpf/verifier.c +8
-2@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + }
kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18506
Quoted 10 lines
kernel/bpf/verifier.c +5
-2}if (insn->src_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->src_reg); -return -EINVAL ; +if (insn->src_reg !=BPF_REG_PARAMS || class !=BPF_LDX || + mode !=BPF_MEM ) { +verbose (env,"R%d is invalid\n" , insn->src_reg); +return -EINVAL ; + } }
Can these instructions with BPF_REG_PARAMS trigger out-of-bounds accesses in subsequent verification functions?
The regs array in struct bpf_func_state is declared as regs[MAX_BPF_REG] with MAX_BPF_REG=11, giving valid indices 0-10.
When instructions with src_reg=BPF_REG_PARAMS (11) reach check_load_mem():
kernel/bpf/verifier.c:check_load_mem() {
...
src_reg_type = regs[insn->src_reg].type;
...
}This performs regs[11] which is one past the end of the array.
Similarly, when instructions with dst_reg=BPF_REG_PARAMS reach __check_reg_arg():
kernel/bpf/verifier.c:__check_reg_arg() {
...
reg = ®s[regno];
...
}The commit message states "array-based register tracking indexed by register number would cause out-of-bounds accesses. So do early return if needed." However, the patch adds early returns in const_fold.c and liveness.c but not in the main verification path functions like check_load_mem() or __check_reg_arg().
The fix appears to arrive in subsequent commit 212295359353. Between this commit and that fix, could user-provided BPF programs with BPF_REG_PARAMS instructions trigger out-of-bounds memory access in the kernel verifier?
kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18508
Quoted 2 lines
kernel/bpf/verifier.c +0
-0if (insn[0 ].code == (BPF_LD |BPF_IMM |BPF_DW )) {struct bpf_insn_aux_data *aux;
[ ... ]
This commit fixes a bug but lacks a Fixes: tag. The commit prevents out-of-bounds array accesses when BPF_REG_PARAMS is used as an index in array-based register tracking. The bug was introduced when BPF_REG_PARAMS was added as MAX_BPF_REG without adding bounds checks.
Suggested: Fixes: fa38458af04a ("bpf: Introduce bpf register BPF_REG_PARAMS")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to Yonghong SongThis isn't a bug, but there's a grammar error: 'are been' should be 'have been' or 'are being'.
Quoted 3 lines
bpf: Reject stack arguments if tail call reachable
Tailcalls are been deprecated. So reject stack arguments
This isn't a bug, but there's a grammar error: 'are been' should be 'have been' or 'are being'.
The commit message states that "Tailcalls are been deprecated." Is this claim accurate?
Based on earlier mailing list discussion, the restriction exists because tail calls and stack arguments require incompatible stack frame handling, not because tail calls are being deprecated.
Could the commit message be more precise about the actual technical reason for the restriction?
This commit appears to fix a bug but lacks a Fixes: tag. The suggested tag would be:
Quoted 6 lines
kernel/bpf/verifier.c +0
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );
kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5672
Quoted 10 lines
kernel/bpf/verifier.c +5
-0return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + } subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
When the DFS traversal in check_max_stack_depth_subprog() pops a frame to return to the caller (moving from a child subprogram back to its parent), is the tail_call_reachable variable reset to false?
Looking at the code, tail_call_reachable is initialized at the function start but appears to persist across the entire traversal. Consider this scenario:
1. The verifier traverses a subprogram that contains a tail call, setting tail_call_reachable = true
2. The traversal returns to the parent (pops the frame via idx = dinfo[idx].caller)
3. The traversal then visits a sibling subprogram that uses stack arguments
In this scenario, wouldn't tail_call_reachable still be true from step 1, causing the check in the new code to incorrectly reject the sibling subprogram even though it has no tail calls in its own call chain?
This could cause false positive rejections of valid BPF programs where one subprogram uses tail calls and a completely unrelated sibling subprogram uses stack arguments.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
[email protected]
AI reviewRe: [PATCH bpf-next v5 13/16] bpf,x86: Implement JIT support for stack arguments
In reply to Yonghong SongThe special case for arg 6 at offset -8 is only implemented for BPF_ST | BPF_MEM | BPF_DW. What happens with smaller access sizes (BPF_B, BPF_H, BPF_W)?
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 17 lines
arch/x86/net/bpf_jit_comp.c +9
-2@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets.
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2244
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-1+* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + } +if (is_imm8 (insn_off)) +EMIT2 (add_1reg (0x40 , dst_reg), insn_off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), insn_off);
The special case for arg 6 at offset -8 is only implemented for BPF_ST | BPF_MEM | BPF_DW. What happens with smaller access sizes (BPF_B, BPF_H, BPF_W)?
Looking at the code flow, if a BPF_ST with size B/H/W targets dst_reg == BPF_REG_PARAMS at offset -8, execution would fall through to the st: label. The offset translation formula would then be applied:
insn_off = outgoing_arg_base - outgoing_rsp - (-8) - 16
= outgoing_arg_base - outgoing_rsp - 8
For example, with outgoing_arg_base = -40 and outgoing_rsp = 0, this yields -32, which would attempt to write at [rbp - 32], outside the designated outgoing argument area at [rsp].
Does this allow writing out-of-bounds below the stack pointer instead of updating R9?
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +0
-1@@ -2144,7 +2251,17 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W :case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off);
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2263
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store register value in r9 */ +EMIT_mov (X86_REG_R9 , src_reg); +break ; + } + insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + }
The offset translation formula subtracts the BPF offset, which reverses the addressing direction. While this correctly maps 8-byte aligned slot bases (e.g., -16 → rsp+0, -24 → rsp+8), can it handle partial writes within those slots?
For instance, if a program writes to the upper 32 bits of arg 7 at BPF offset -12:
insn_off = outgoing_arg_base - outgoing_rsp - (-12) - 16
= outgoing_arg_base - outgoing_rsp - 4
With outgoing_arg_base = -40 and outgoing_rsp = 0, this computes to -36, mapping the access to [rbp - 36] instead of [rsp + 4].
Does this formula produce out-of-bounds accesses below the stack pointer for partial writes?
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2375
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ; +/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }
When reading incoming arg 6 with smaller access sizes (BPF_LDX | BPF_MEM | BPF_B/H/W), the code performs a full 64-bit move from R9 to the destination register:
EMIT_mov(dst_reg, X86_REG_R9);
For sub-64-bit arguments, are the upper bits of R9 guaranteed to be zero? If not, could this move garbage data into the upper bits of dst_reg, bypassing the verifier's strict bounds tracking which expects values to stay within their declared ranges (e.g., [0, 255] for bytes)?
Reference: https://lore.kernel.org/bpf/[email protected]/
The JIT translation assumes that LDX operations target incoming arguments (positive offsets from r11) and STX/ST operations target outgoing arguments (negative offsets from r11). Is this assumption always valid?
Consider these scenarios:
Scenario 1: A program writes to an outgoing argument slot at [r11 - 16] via STX, then reads it back via LDX. The JIT would:
- Store: apply outgoing translation → [rsp + 0]
- Load: apply incoming translation → [rbp + 16] (different location)
The load would read from the BPF program stack area rather than the outgoing argument area, potentially returning uninitialized data.
Scenario 2: A program attempts to modify an incoming argument at [r11 + 16] via STX. The JIT would apply the outgoing translation:
insn_off = outgoing_arg_base - outgoing_rsp - 16 - 16
= outgoing_arg_base - outgoing_rsp - 32
With outgoing_arg_base = -40 and outgoing_rsp = 0, this yields -72, writing to [rbp - 72], which could be out-of-bounds below the allocated stack area.
Does the verifier enforce a constraint that reads can only target positive r11 offsets and writes can only target negative r11 offsets?
Reference: https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Quoted 14 lines · Yonghong Song
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (
BPF_REG_PARAMS) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped atMAX_BPF_FUNC_ARGS(12), which is sufficient for the vast majority of use cases.
Quoted 6 lines
In verifier, r11 based stores can survive bpf-to-bpf and kfunc calls. For example
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
call bar2; // reuses same arg6, arg7 without re-storing
Argument registers are caller saved, that means if the x86 JIT will use R9 for the arg6 and call bar1, it can clobber it (by calling a helper or another bpf-to-bpf call) and then bar2 will receive clobbered value for arg6 because it did not reload it.
This only works for caller saved registers like arg5 (R8 on x86-64) (R5 on bpf) because R5 (BPF) is caller saved and compiler will make sure to save R5 before the call. But for stacked arguments, compiler models them as stack memory and if the JIT models them as a register to follow calling convention, they need to be saved/reloaded before calls.
Let me explain with an example, if you apply this patch to the selftests:
-- >8 --
From 52924eb28056fe8a2321b1cadf9409f6ca90603d Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Sat, 18 Apr 2026 09:11:06 -0700
Subject: [PATCH] selftests/bpf: Add test for stack arg register clobbering
Add verifier test stack_arg_reuse_across_inner_call which uses inline asm to store stack args, call a subprog that internally calls another subprog with different stack arg values (100, 200 instead of 6, 7), then reuse the original stores for a second call without re-storing.
This exposes JIT bugs where stack arg registers (x5-x7 on arm64, R9 on x86) are clobbered by the inner call but not refreshed before the second call.
Correct result: 315 + 28 = 343.
Clobbered result: 315 + 315 = 630. (on arm64)
--- .../selftests/bpf/progs/verifier_stack_arg.c | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+)
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +56 -0
@@ -462,6 +462,62 @@ __naked void stack_arg_sequential_calls(void) ); } +/* +* subprog_7args_inner_call takes 7 args like subprog_7args, but +* internally calls subprog_7args with different stack arg values +* (100, 200). This clobbers whatever the JIT placed in the stack +* arg registers (x5-x7 on arm64, R9 on x86). +*/ +__noinline __used +static int subprog_7args_with_inner_call(int a, int b, int c, int d, +int e,int f,int g) +{ +return subprog_7args (a, b, c, d, e,100 ,200 ); +} + +SEC("tc") +__description("stack_arg: store reuse across inner call") +__success __retval(343) +__arch_x86_64 +__arch_arm64 +__naked void stack_arg_reuse_across_inner_call(void) +{ + asmvolatile ( +/* call1: subprog_7args_inner_call(1,2,3,4,5,6,7) +* internally calls subprog_7args(1,2,3,4,5,100,200)=315 +* which clobbers stack arg registers with 100, 200. +* Returns 315. +*/ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"*(u64 *)(r11 - 8) = 6;" +"*(u64 *)(r11 - 16) = 7;" +"call subprog_7args_with_inner_call;" +"r6 = r0;" /* 315 */ +/* call2: reuse r11 stores from call1 (no re-store). +* subprog_7args(1,2,3,4,5,6,7) should return 28. +* If clobbered: subprog_7args(1,2,3,4,5,100,200)=315. +*/ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +/* +* the 6th and 7th arg should be 6, 7 as set above. +* But the inner call in subprog_7args_inner_call clobbered +* them and reusing again without a re-store will be wrong. +*/ +"call subprog_7args;" /* should return 28, but will be 315 due to clobber */ +"r0 += r6;" /* should have 315 + 28 = 343 but will be 315 + 315 = 630 */ +"exit;" + ::: __clobber_all + ); +} +#else SEC ("socket" )
--
2.52.0
-- 8< --
If I run this arm64 (I have a patch to add support)
I get:
run_subtest:PASS:obj_open_mem 0 nsec run_subtest:PASS:unexpected_load_failure 0 nsec do_prog_test_run:PASS:bpf_prog_test_run 0 nsec run_subtest:FAIL:1313 Unexpected retval: 630 != 343 #631/14 verifier_stack_arg/stack_arg: store reuse across inner call:FAIL
And on x86 it will be:
- Second call gets R9 = 100 (clobbered), [rsp+0] = 7 (survives, because subprog get's own stack slot)
- subprog_7args(1,2,3,4,5, 100, 7) = 122 instead of 28
- Total: 315 + 122 = 437 instead of 343
^^ this is not tested just a theoretical looking at x86 JIT code added in this patchset that used R9 for sixth argument.
So, I think the fix it to make the JITs keep the 6th arg (x86) and (6,7,8)th arg (arm64) on the stack and re-load them before every call. Or the compiler should treat all r11 stack slots like caller saved stack and reload them.
Please let me know if my analysis is correct or maybe my arm64 jit implementation is broken.
Thanks,
Puranjay
Yonghong Song
reviewerRe: [PATCH bpf-next v5 03/16] bpf: Refactor to handle memory and size together
In reply to [email protected]Quoted 30 lines · On 4/16/26 9:49 PM, [email protected] wrote:
Sashiko AI review found 1 potential issue(s):
- [High] Incomplete refactoring for stack arguments breaks verifier precision tracking, marking incorrect registers. --commit
7bebb6c179745d67fba0c807428d095b66738135
Author: Yonghong Song <[email protected]>bpf: Refactor to handle memory and size together
This commit refactors memory and size register handling in BPF helper verification, passing
bpf_reg_statedirectly to prepare for stack arguments.Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 13 lines
kernel/bpf/verifier.c +4
-3@@ -7020,15 +7020,17 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,/* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * -* @regno is the register containing the access size. regno-1 is the register -* containing the pointer. +* @mem_regno is the register containing the pointer, mem_regno+1 is the register +* containing the access size. */ static int check_mem_size_reg(struct bpf_verifier_env *env, - struct bpf_reg_state *reg, u32 regno, + struct bpf_reg_state *mem_reg, + struct bpf_reg_state *size_reg, u32 mem_regno,
Quoted 8 lines
kernel/bpf/verifier.c static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · line 7034
Quoted 6 lines
kernel/bpf/verifier.c +1
-0enum bpf_access_type access_type, bool zero_size_allowed, struct bpf_call_arg_meta *meta) { + int size_regno = mem_regno + 1;int err;Since this refactoring is intended to support stack arguments, does this hardcoded
size_regno=mem_regno+ 1 break verifier precision tracking?
We are fine here. In function get_kfunc_ptr_arg_type(), we have
if (argno + 1 < nargs &&
(is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1])))
arg_mem_size = true;This ensures mem_size is at most at R5.
Quoted 21 lines
If the memory pointer is passed in R5,
mem_regnois 5, makingsize_regno6.This would cause
mark_chain_precision()to incorrectly backtrack the callee-saved registerBPF_REG_6instead of the stack slot containing the size argument.Furthermore, if
mem_regnois later modified to represent a 0-based argument index,size_regnowill evaluate to the wrong register entirely.[ ... ]
kernel/bpf/verifier.c static int check_mem_size_reg(struct bpf_verifier_env *env, · line 7074
Quoted 6 lines
kernel/bpf/verifier.c +1
-1- err =check_helper_mem_access (env, regno -1 , reg->umax_value, + err =check_helper_mem_access (env, mem_reg, mem_regno, size_reg->umax_value, access_type, zero_size_allowed, meta);if (!err)
kernel/bpf/verifier.c +1
-1- err =mark_chain_precision (env, regno); + err =mark_chain_precision (env, size_regno);Does marking the wrong register leave the actual size argument imprecise, potentially corrupting verifier state pruning and allowing out-of-bounds memory accesses to bypass verification?
Re: [PATCH bpf-next v5 13/16] bpf,x86: Implement JIT support for stack arguments
In reply to [email protected]Quoted 47 lines · [email protected] writes:
Sashiko AI review found 4 potential issue(s):
- [High] Data corruption of Argument 6 across function calls due to caller-saved register clobbering.
- [High] Missing size handling for Argument 6 allows verifier bounds tracking bypass and out-of-bounds memory writes.
- [High] The offset translation formula for outgoing arguments 7+ reverses the memory layout, corrupting sub-slot accesses.
- [Medium] The JIT misroutes stack accesses because it rigidly assumes LDX only targets incoming arguments and STX/ST only target outgoing arguments. --commit
41dcf99330b6b0a3804185f2a284a76be8340ed1
Author: Yonghong Song <[email protected]>bpf,x86: Implement JIT support for stack arguments
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r11 (BPF_REG_PARAMS) in BPF bytecode, which the JIT translates to native code.Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -1664,16 +1692,45 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1706
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +5
-0if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Follow x86-64 calling convention for both BPF-to-BPF and +* kfunc calls: +* - Arg 6 is passed in R9 register +* - Args 7+ are passed on the stack at [rsp] Can the use of the R9 register cause data corruption across function calls?
The x86-64 C calling convention defines R9 as a caller-saved register. However, the BPF verifier models argument 6 as stack memory, which implies its value should be preserved across calls.
Quoted 6 lines
If a BPF program reads incoming argument 6 after making a helper or kfunc call, will it receive a clobbered value instead of the original argument? Similarly, could a helper call scheduled by LLVM between the setup of outgoing argument 6 and the target call corrupt the R9 register?
[ ... ]
This is correct analysis, I just realized that sashiko has already found this!
Quoted 99 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +3
-0+/* +* Base offset from rbp for translating BPF outgoing args 7+ +* to native offsets. BPF uses negative offsets from r11 arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1786
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+* (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses +* positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for +* arg8, ...). Arg 6 goes to R9 directly. +* +* The translation reverses direction: +* native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 +* +* Note that tail_call_reachable is guaranteed to be false when +* stack args exist, so tcc pushes need not be accounted for. +*/ Does this formula correctly handle multi-byte sub-slot accesses?
Since the formula subtracts the BPF offset, the memory layout within the 8-byte slots appears mathematically reversed. While this correctly maps the 8-byte slot bases (e.g., -16 maps to rsp + 0), it seems to point to the wrong memory for partial writes.
For example, if a program writes to the upper 32 bits of Arg 7 at BPF offset -12, the formula yields BASE - (-12) - 16 = BASE - 4. Will this map the access out-of-bounds below the stack pointer instead of to [rsp + 4]?
[ ... ]
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +5
-1@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off))arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2240
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +9
-1-EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets. +* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + }Are the size constraints missing for Argument 6 here?
The check for Arg 6 (
insn->off== -8) is only implemented under theBPF_DWcase. For sizes B, H, and W, execution will fall through to the st: label. This would incorrectly translate the access as a normal stack operation using the formula above, writing out-of-bounds below the stack pointer at [rsp - 8] instead of populating R9.Additionally, for
BPF_LDX, a 1-byte load emits a full 64-bit move to R9. Since the upper 56 bits of R9 are undefined for 1-byte arguments, could they contain garbage data? This would bypass the verifier's strict value bounds tracking, which assumes the loaded value is strictly within [0, 255].[ ... ]
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */ arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2375
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ; +/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }Does this logic misroute stack accesses by assuming LDX always targets incoming arguments and STX/ST always target outgoing arguments?
If a program writes an outgoing argument ([r11 - 16]) and then reads it back via LDX, the JIT appears to apply the incoming argument translation mapping the read to [rbp - 16] (the local stack) instead of [rsp] (the outgoing area). This would return uninitialized data.
Conversely, if a program modifies an incoming argument ([r11 + 16]) using STX, the JIT would apply the outgoing translation formula, yielding BASE - 16 - 16 = BASE - 32, which writes out-of-bounds below the stack pointer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Yonghong Song
reviewerRe: [PATCH bpf-next v5 03/16] bpf: Refactor to handle memory and size together
In reply to [email protected]Quoted 26 lines · On 4/17/26 5:52 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 20 lines
kernel/bpf/verifier.c +5
-3@@ -7020,15 +7020,17 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,/* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * -* @regno is the register containing the access size. regno-1 is the register -* containing the pointer. +* @mem_regno is the register containing the pointer, mem_regno+1 is the register +* containing the access size. */ static int check_mem_size_reg (struct bpf_verifier_env *env, -struct bpf_reg_state *reg,u32 regno, +struct bpf_reg_state *mem_reg, +struct bpf_reg_state *size_reg,u32 mem_regno,enum bpf_access_type access_type,bool zero_size_allowed,struct bpf_call_arg_meta *meta) { +int size_regno = mem_regno +1 ;int err;
Quoted 5 lines
kernel/bpf/verifier.c static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · line 7036
Quoted 1 line
kernel/bpf/verifier.c +0
-0/* This is used to refine r0 return value bounds for helpers The refactoring passes
size_regas a parameter to prepare for stack arguments, but this code still hardcodessize_regnoasmem_regno+ 1. Does this assumption hold when stack arguments are used, where the size might not be atmem_regno+ 1?
When stack arguments are used, the 'mem_regno' will be replaced as 'mem_argno' and the size_regno will be size_argno which represents either a register or an argument idx (idx starts from 1). See next patch.
Quoted 13 lines
Looking at how
mark_chain_precision()is called later in this function:kernel/bpf/verifier.c static int check_mem_size_reg(struct bpf_verifier_env *env, · line 7074
Quoted 5 lines
err = check_helper_mem_access(env,
mem_reg,mem_regno,size_reg->umax_value,
access_type,zero_size_allowed, meta);
if (!err)kernel/bpf/verifier.c +1
-1- err =mark_chain_precision (env, regno); + err =mark_chain_precision (env, size_regno);The incorrect
size_regnovalue would causemark_chain_precision()to mark the wrong register for precision tracking. For example, if the memory pointer is passed in R5 and the size is in a stack slot,mem_regnowould be 5, makingsize_regno6. This would incorrectly backtrackBPF_REG_6instead of the stack slot containing the size argument.
This won't happen. There will be some check in the above if statement.
Quoted 4 lines
Could this break verifier state pruning when the commit message's stated goal of handling "stack arguments as they may be beyond registers 1-5" is implemented?
This will be addressed when stack arguments are supported and we can mark size_argno precision is true since the size itself will be a constant.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to [email protected]Quoted 73 lines · On 4/16/26 9:30 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/const_fold.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/liveness.c +0
-0[ ... ]
Quoted 21 lines
kernel/bpf/verifier.c +8
-2@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + } }if (insn->src_reg >=MAX_BPF_REG ) {kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18508
Quoted 10 lines
kernel/bpf/verifier.c +5
-2-verbose (env,"R%d is invalid\n" , insn->src_reg); -return -EINVAL ; +if (insn->src_reg !=BPF_REG_PARAMS || class !=BPF_LDX || + mode !=BPF_MEM ) { +verbose (env,"R%d is invalid\n" , insn->src_reg); +return -EINVAL ; + } }if (insn[0 ].code == (BPF_LD |BPF_IMM |BPF_DW )) {struct bpf_insn_aux_data *aux;Can these instructions with
BPF_REG_PARAMStrigger out-of-bounds accesses in subsequent verification functions? The regs array in structbpf_func_stateis declared as regs[MAX_BPF_REG] with MAX_BPF_REG=11, giving valid indices 0-10.When instructions with src_reg=BPF_REG_PARAMS (11) reach
check_load_mem():kernel/bpf/verifier.c:check_load_mem() { ... src_reg_type = regs[insn->src_reg].type; ... }This performs regs[11] which is one past the end of the array.
Similarly, when instructions with dst_reg=BPF_REG_PARAMS reach
__check_reg_arg():kernel/bpf/verifier.c:__check_reg_arg() { ... reg = ®s[regno]; ... }The commit message states "array-based register tracking indexed by register number would cause out-of-bounds accesses. So do early return if needed." However, the patch adds early returns in
const_fold.candliveness.cbut not in the main verification path functions likecheck_load_mem()or__check_reg_arg().
Quoted 4 lines
The fix appears to arrive in subsequent commit 212295359353 ("bpf:
Support stack arguments for bpf functions"). Between this commit and that fix, could user-provided BPF programs withBPF_REG_PARAMSinstructions trigger out-of-bounds memory access in the kernel verifier?
You are right, this patch should be after "bpf: Support stack arguments for bpf functions" to avoid such issues.
Quoted 11 lines
This commit also lacks a Fixes: tag. The suggested tag would be:
Fixes 595885eced33 ("bpf: Introduce bpf register BPF_REG_PARAMS")---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
Yonghong Song
reviewerRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to [email protected]Quoted 39 lines · On 4/16/26 9:50 PM, [email protected] wrote:
Sashiko AI review found 2 potential issue(s):
- [Critical] Early returns in dataflow tracking passes skip necessary state updates for the destination register duringBPF_LDXusingBPF_REG_PARAMS.
- [Critical] PermittingBPF_REG_PARAMScauses severe out-of-bounds memory accesses to theregsarray during the main verifier loop (do_check_main). --commit
d68722297de575b838cc2f76141cf9c052e63448
Author: Yonghong Song <[email protected]>bpf: Limit the scope of
BPF_REG_PARAMSusageThis commit restricts the usage of
BPF_REG_PARAMS(r11) to specific memory load and store instruction forms. It also adds early returns in dataflow tracking passes to prevent out-of-bounds accesses for stack arguments.Quoted 4 lines
kernel/bpf/const_fold.c +0
-0[ ... ]
Quoted 5 lines
kernel/bpf/const_fold.c +0
-1@@ -51,13 +51,18 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * struct bpf_insn *insn, struct bpf_insn *insns, int idx) { struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 }; -struct const_arg_info *dst = &ci_out[insn->dst_reg];kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · line 62
Quoted 10 lines
kernel/bpf/const_fold.c +4
-1-struct const_arg_info *src = &ci_out[insn->src_reg]; +struct const_arg_info *dst, *src;u8 class =BPF_CLASS (insn->code);u8 mode =BPF_MODE (insn->code);u8 opcode =BPF_OP (insn->code) |BPF_SRC (insn->code);int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ;If we return early here during a
BPF_LDXinstruction wheresrc_regisBPF_REG_PARAMS(11), doesn't this skip necessary state updates fordst_reg(e.g., registers 0 through 9)?
Quoted 4 lines
Since
dst_regis not marked as unknown, won't it retain its stale tracking state? Couldbpf_prune_dead_branches()rely on this stale constant to incorrectly rewrite conditional jumps, allowing verifier safety checks to be bypassed?
Good point. Will fix.
Quoted 20 lines
Quoted 4 lines
kernel/bpf/const_fold.c +3
-0+ + dst = &ci_out[insn->dst_reg]; + src = &ci_out[insn->src_reg];[ ... ]
Quoted 14 lines
kernel/bpf/liveness.c +1
-2@@ -1068,11 +1068,16 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,int depth = instance->depth;u8 class =BPF_CLASS (insn->code);u8 code =BPF_OP (insn->code); -struct arg_track *dst = &at_out[insn->dst_reg]; -struct arg_track *src = &at_out[insn->src_reg]; +struct arg_track *dst, *src;struct arg_track none = { .frame =ARG_NONE };int r;
Quoted 5 lines
kernel/bpf/liveness.c static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 1077
Quoted 3 lines
kernel/bpf/liveness.c +3
-0+/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ;Similarly, does this early return skip clearing the
arg_trackstate for the destination register when loading from a stack argument?
Same, will fix.
Quoted 30 lines
Quoted 4 lines
kernel/bpf/liveness.c +3
-0+ + dst = &at_out[insn->dst_reg]; + src = &at_out[insn->src_reg];[ ... ]
Quoted 10 lines
kernel/bpf/verifier.c +2
-0@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code);kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18499
Quoted 10 lines
kernel/bpf/verifier.c +6
-2+if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + } }By permitting
BPF_REG_PARAMS(which equalsMAX_BPF_REG) to pass this validation check, won't this cause out-of-bounds memory accesses to the regs array (e.g., structbpf_reg_stateregs[MAX_BPF_REG]) during the main verifier loop?
Quoted 7 lines
When
do_check_main()processes these instructions, functions likecheck_reg_arg()will attempt to access regs[11], which overlays verifier internal state.I note this regression is fixed later in the patch series in commit
6ec4a3fcb5e83("bpf: Support stack arguments for bpf functions"), but it appears to introduce an issue in this intermediate patch.
Yes, this patch should be after "bpf: Support stack arguments for bpf functions " patch.
Yonghong Song
reviewerRe: [PATCH bpf-next v5 06/16] bpf: Limit the scope of BPF_REG_PARAMS usage
In reply to [email protected]Quoted 106 lines · On 4/17/26 6:04 PM, [email protected] wrote:
Quoted 11 lines
kernel/bpf/const_fold.c +1
-2@@ -51,13 +51,18 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out, struct bpf_insn *insn, struct bpf_insn *insns, int idx) { struct const_arg_info unknown = { .state = CONST_ARG_UNKNOWN, .val = 0 }; -struct const_arg_info *dst = &ci_out[insn->dst_reg]; -struct const_arg_info *src = &ci_out[insn->src_reg]; +struct const_arg_info *dst, *src;u8 class =BPF_CLASS (insn->code);kernel/bpf/const_fold.c static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out, · line 65
Quoted 10 lines
kernel/bpf/const_fold.c +6
-0u8 mode =BPF_MODE (insn->code);u8 opcode =BPF_OP (insn->code) |BPF_SRC (insn->code);int r; +/* Stack arguments using BPF_REG_PARAMS are outside the tracked register set. */ +if (insn->dst_reg >=MAX_BPF_REG || insn->src_reg >=MAX_BPF_REG ) +return ; + + dst = &ci_out[insn->dst_reg]; + src = &ci_out[insn->src_reg];If we return early here during a
BPF_LDXinstruction wheresrc_regisBPF_REG_PARAMS(11), doesn't this skip necessary state updates fordst_reg(e.g., registers 0 through 9)?Since
dst_regis not marked as unknown, won't it retain its stale tracking state? Couldbpf_prune_dead_branches()rely on this stale constant to incorrectly rewrite conditional jumps, allowing verifier safety checks to be bypassed?A similar issue exists in
liveness.cwhere the early return skips clearing thearg_trackstate for the destination register when loading from a stack argument.Quoted 3 lines
kernel/bpf/const_fold.c +0
-0switch (class) {case BPF_ALU :case BPF_ALU64 :[ ... ]
Quoted 18 lines
kernel/bpf/verifier.c +8
-2@@ -18487,13 +18487,22 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)return err;for (i =0 ; i < insn_cnt; i++, insn++) { +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +if (insn->dst_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->dst_reg); -return -EINVAL ; +if (insn->dst_reg !=BPF_REG_PARAMS || + !((class ==BPF_ST || class ==BPF_STX ) && mode ==BPF_MEM )) { +verbose (env,"R%d is invalid\n" , insn->dst_reg); +return -EINVAL ; + }kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18506
Quoted 10 lines
kernel/bpf/verifier.c +5
-2}if (insn->src_reg >=MAX_BPF_REG ) { -verbose (env,"R%d is invalid\n" , insn->src_reg); -return -EINVAL ; +if (insn->src_reg !=BPF_REG_PARAMS || class !=BPF_LDX || + mode !=BPF_MEM ) { +verbose (env,"R%d is invalid\n" , insn->src_reg); +return -EINVAL ; + } }Can these instructions with
BPF_REG_PARAMStrigger out-of-bounds accesses in subsequent verification functions?The regs array in struct
bpf_func_stateis declared as regs[MAX_BPF_REG] with MAX_BPF_REG=11, giving valid indices 0-10.When instructions with src_reg=BPF_REG_PARAMS (11) reach
check_load_mem():kernel/bpf/verifier.c:check_load_mem() { ... src_reg_type = regs[insn->src_reg].type; ... }This performs regs[11] which is one past the end of the array.
Similarly, when instructions with dst_reg=BPF_REG_PARAMS reach
__check_reg_arg():kernel/bpf/verifier.c:__check_reg_arg() { ... reg = ®s[regno]; ... }The commit message states "array-based register tracking indexed by register number would cause out-of-bounds accesses. So do early return if needed." However, the patch adds early returns in
const_fold.candliveness.cbut not in the main verification path functions likecheck_load_mem()or__check_reg_arg().The fix appears to arrive in subsequent commit 212295359353. Between this commit and that fix, could user-provided BPF programs with
BPF_REG_PARAMSinstructions trigger out-of-bounds memory access in the kernel verifier?kernel/bpf/verifier.c static int check_and_resolve_insns(struct bpf_verifier_env *env) · line 18508
Quoted 2 lines
kernel/bpf/verifier.c +0
-0if (insn[0 ].code == (BPF_LD |BPF_IMM |BPF_DW )) {struct bpf_insn_aux_data *aux;
Quoted 8 lines
[ ... ]
This commit fixes a bug but lacks a Fixes: tag. The commit prevents out-of-bounds array accesses when
BPF_REG_PARAMSis used as an index in array-based register tracking. The bug was introduced whenBPF_REG_PARAMSwas added asMAX_BPF_REGwithout adding bounds checks.Suggested: Fixes:
fa38458af04a("bpf: Introduce bpf registerBPF_REG_PARAMS")
The same issues mentioned in previousSashiko-bot. Will fix.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 07/16] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to [email protected]Quoted 19 lines · On 4/16/26 9:30 PM, [email protected] wrote:
include/linux/bpf.h struct bpf_prog_offload { · line 1157
Quoted 11 lines
include/linux/bpf.h +4
-0@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {/* The longest tracepoint has 12 args. * See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via +* stack arg slots. Is this description of the calling convention accurate? The comment states "Args 1-5 are passed in registers, args 6-12 via stack arg slots," but doesn't argument 6 get passed in the R9 register rather than on the stack?
Looking at
arch/x86/net/bpf_jit_comp.c, the existing comment states: "x86-64 supports up toMAX_BPF_FUNC_ARGSarguments. 1-6 are passed through regs, the remains are through stack."
Quoted 7 lines
The follow-up commit
0ef907c3ec2cin this patch series ("bpf,x86: Implement JIT support for stack arguments") documents the actual calling convention as:
"Follow x86-64 calling convention for both BPF-to-BPF and kfunc calls: - Arg 6 is passed in the R9 register - Args 7+ are passed on the stack at [rsp]"Should the comment say "Args 1-5 are passed in registers, arg 6 in R9 register, args 7-12 via stack arg slots" instead?
I add something like:
The JIT may map some stack arg slots to registers based on the native calling convention (e.g., arg 6 to R9 on x86-64).
Quoted 9 lines
include/linux/bpf.h struct bpf_prog_offload { · line 1159
Quoted 3 lines
include/linux/bpf.h +0
-0*/ #define MAX_BPF_FUNC_ARGS 12---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
Yonghong Song
reviewerRe: [PATCH bpf-next v5 07/16] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
In reply to [email protected]Quoted 39 lines · On 4/17/26 5:52 PM, [email protected] wrote:
include/linux/bpf.h struct bpf_prog_offload { · line 1152
Quoted 7 lines
include/linux/bpf.h +1
-1@@ -1151,6 +1151,10 @@ struct bpf_prog_offload { -/* The longest tracepoint has 12 args. +/* The longest tracepoint has 12 args. This isn't a bug, but the multi-line comment opening doesn't follow BPF subsystem style. Per bpf.md: 'Multi-line comments MUST have the opening /* on its own line'. The rule applies even if surrounding code uses the old style.
Should this be reformatted as:
/*
* The longest tracepoint has 12 args.
* Seeinclude/trace/bpf_probe.h
*
* Also reuse this macro for maximum number of arguments a BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots.
*/include/linux/bpf.h struct bpf_prog_offload { · line 1156
Quoted 4 lines
include/linux/bpf.h +3
-0* See include/trace/bpf_probe.h +* +* Also reuse this macro for maximum number of arguments a BPF function +* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via Should this say "kfuncs" instead of "kfunc's"? The context is plural (referring to multiple functions), not possessive.
Additionally, [email protected] raised a concern in v5 about the calling convention description. The comment states "Args 1-5 are passed in registers, args 6-12 via stack arg slots," but argument 6 is actually passed in the R9 register, not on the stack.
Looking at
arch/x86/net/bpf_jit_comp.c, the existing comment states:
"x86-64 supports up toMAX_BPF_FUNC_ARGSarguments. 1-6 are passed through regs, the remains are through stack."
Quoted 7 lines
The follow-up commit
0ef907c3ec2cin this patch series documents the actual calling convention as: "Follow x86-64 calling convention for both BPF-to-BPF and kfunc calls: - Arg 6 is passed in the R9 register - Args 7+ are passed on the stack at [rsp]"Should the comment say "Args 1-5 are passed in registers, arg 6 in R9 register, args 7-12 via stack arg slots" instead?
See previous comment:
The JIT may map some stack arg slots to registers based on the native calling convention (e.g., arg 6 to R9 on x86-64).
This should clarify.
Quoted 13 lines
include/linux/bpf.h struct bpf_prog_offload { · line 1160
Quoted 4 lines
include/linux/bpf.h +1
-0+* stack arg slots. */ #define MAX_BPF_FUNC_ARGS 12---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Quoted 10 lines · Yonghong Song
Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.
This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (
BPF_REG_PARAMS) to pass arguments beyond
As an overall design of BPF as an architecture, shouldn't it map 1-1 with atleast one real architecture like x86-64? So it runs as fast a possible on that architecture?
x86 supports passing 6 arguments in registers, BPF currently supports only 5 (I am not aware of why it was designed like this to start with)
Now, if we add 6+ arguments support by using the stack, the 6th argument that could have been easily mapped to a caller saved register (R9) will need special handling.
I know changing these fundamental things about the ISA is not possible now, like making BPF R6 caller saved and passing the argument in that. Or add more registers to BPF so it runs faster on RISC architectures.
At least we can make the compiler treat these r11 based stack slots as caller saved, so they can be mapped to real registers by the JIT efficiently.
I would say atleast make the first 3 slots (arg 6-8) caller saved (reload them before every call), so they can be mapped to 3 arm64 registers. riscv64 also has the same layout and it will also map them to 3 registers directly.
Or to make it simpler, just make all the stack slots caller saved.
Thanks,
Puranjay
Yonghong Song
reviewerRe: [PATCH bpf-next v5 08/16] bpf: Support stack arguments for bpf functions
In reply to [email protected]Quoted 42 lines · On 4/16/26 9:35 PM, [email protected] wrote:
Sashiko AI review found 5 potential issue(s):
- [Critical] Verifier State Leakage incopy_stack_statewhen reusing states.
- [Critical] Security / Liveness Tracking Bypass viastack_arg_regs.
- [Critical] Use-After-Free via Type Mismatch inbpf_get_spilled_stack_arg().
- [Medium] Unnecessary Rejection of Valid Programs due to Dead Code Elimination.
- [Low] Missing Parentheses inbpf_get_spilled_stack_argMacro. --commit
6ec4a3fcb5e83a95626cd2bb13a60b2283e1c3e2
Author: Yonghong Song <[email protected]>
bpf: Support stack arguments for bpf functionsThis commit adds support for passing more than 5 arguments to BPF subprograms by utilizing a dedicated stack area accessed via register r11. It introduces new fields to track incoming and outgoing stack arguments for bounds checking and precise verifier path tracking.
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 519
Quoted 11 lines
include/linux/bpf_verifier.h +4
-0@@ -508,6 +513,17 @@ struct bpf_verifier_state { iter < frame->allocated_stack /BPF_REG_SIZE ; \ iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL)Is there a type mismatch in this check?
The mask parameter here is passed as 1 <<
STACK_SPILL, but frame->stack_arg_regs[slot].type is an enumbpf_reg_type, not an enumbpf_stack_slot_type.For pointer types, this expression can evaluate to 0. This would cause the iterator to silently skip pointers stored in the stack argument area. If an acquired reference is dropped via
release_reference(), the copy in the stack argument area wouldn't be invalidated, potentially allowing a use-after-free if a program subsequently reads the spilled pointer back from r11.
Quoted 3 lines
Also, this isn't a bug, but should the slot parameter be enclosed in parentheses to prevent operator precedence issues if a complex expression is passed in the future?
Yes, it is a bug. Will fix.
Quoted 20 lines
[ ... ]
Quoted 4 lines
kernel/bpf/fixups.c +0
-0@@ -1266,9 +1275,20 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)[ ... ]
kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1290
Quoted 10 lines
kernel/bpf/fixups.c +10
-0+for (i =0 ; i < env->subprog_cnt; i++) { +struct bpf_subprog_info *subprog = &env->subprog_info[i]; + +if (subprog->max_out_stack_arg_depth > subprog->outgoing_stack_arg_depth) { +verbose (env, +"func#%d writes stack arg slot at depth %u, but calls only require %u bytes\n" , + i, subprog->max_out_stack_arg_depth, subprog->outgoing_stack_arg_depth); +return -EINVAL ; + } + }Could this reject valid programs if the compiler hoists a stack argument store above a branch that is later eliminated as dead code?
Quoted 5 lines
If the compiler optimizes a store above a conditional check,
max_out_stack_arg_depthwould be updated. However, if the configuration constants cause the branch containing the actual function call to be eliminated,outgoing_stack_arg_depthwould remain 0, leading to a rejection of a safe verification path.
We should be okay here. btf_check_func_arg_match() ensures func prototype matches the number of outgoing parameters. Otherwise, it will fail:
if (i >= MAX_BPF_FUNC_REG_ARGS) {
struct bpf_func_state *caller = cur_func(env);
int spi = out_arg_spi(caller, i - MAX_BPF_FUNC_REG_ARGS);
/*
* The compiler may constant-fold stack arg values into the
* callee, eliminating the r11 stores. The BTF still declares
* these parameters, but no outgoing stack slots exist.
*/
if (spi >= (caller->stack_arg_depth / BPF_REG_SIZE)) {
verbose(env, "stack %s not found in caller state\n",
reg_arg_name(env, argno));
return -EINVAL;
}
}Quoted 34 lines
[ ... ]
Quoted 13 lines
kernel/bpf/verifier.c +6
-0@@ -1340,6 +1340,20 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_streturn -ENOMEM ; dst->allocated_stack = src->allocated_stack; + +/* copy stack args state */ + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, + sizeof(struct bpf_reg_state),kernel/bpf/verifier.c static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · line 1358
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+ GFP_KERNEL_ACCOUNT); +if (!dst->stack_arg_regs) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + dst->out_stack_arg_mask = src->out_stack_arg_mask; + }return 0 ; }Does this skip clearing the destination's stack argument fields if the source state has no stack arguments?
If
src->stack_arg_depthis 0,dst->stack_arg_depth,out_stack_arg_mask, andincoming_stack_arg_depthare not updated. When the verifier reuses an existing state duringpop_stack()viabpf_copy_verifier_state(), the destination state might incorrectly inherit a stale stack argument initialization mask from a previously evaluated path.
Quoted 3 lines
Could this allow a program to initialize stack arguments on one branch, and then safely perform a function call on a different branch without proper initialization?
Yes, you are right. dst->stack_arg_depth etc. should not under condition 'if (n)'.
Quoted 31 lines
[ ... ]
Quoted 1 line
kernel/bpf/verifier.c +0
-0@@ -4424,6 +4467,123 @@ static int check_stack_write(struct bpf_verifier_env *env,[ ... ]
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+static int check_stack_arg_write (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int value_regno) +{[ ... ]
Quoted 22 lines
kernel/bpf/verifier.c +22
-0+ cur = env->cur_state->frame[env->cur_state->curframe]; +if (value_regno >=0 ) { + state->stack_arg_regs[spi] = cur->regs[value_regno]; + }else { +/* BPF_ST: store immediate, treat as scalar */ +struct bpf_reg_state *arg = &state->stack_arg_regs[spi]; + + arg->type =SCALAR_VALUE ; +__mark_reg_known (arg, env->prog->insnsi[env->insn_idx].imm); + } + state->out_stack_arg_mask |=BIT (idx); +return 0 ; +} + +/* +* Read a value from the incoming stack arg area. +* off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7). +* Callers ensure off > 0, 8-byte aligned, and size is BPF_REG_SIZE. +*/ +static int check_stack_arg_read (struct bpf_verifier_env *env,struct bpf_func_state *state, +int off,int dst_regno) +{[ ... ]
Quoted 10 lines
kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 4585
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+ arg = &state->stack_arg_regs[spi]; + cur = env->cur_state->frame[env->cur_state->curframe]; + +if (is_spillable_regtype (arg->type)) +copy_register_state (&cur->regs[dst_regno], arg); +else +mark_reg_unknown (env, cur->regs, dst_regno); +return 0 ; +}Does this bypass liveness tracking for stack arguments?
This will be addressed in liveness.c.
Quoted 10 lines
Because the state is copied directly without calling
mark_reg_read(), the live field of stack arguments is never marked as read.During state pruning,
stack_arg_safe()delegates toregsafe(), which contains a fast-path that unconditionally returns true if the old state's register was never marked as read. Could this cause the verifier to aggressively prune verification paths even if they contain completely different and potentially unsafe stack arguments?
I think we should be okay. mark_reg_unknown will mark register as SCALAR_VALUE. It will do pruning properly in regsafe().
Yonghong Song
reviewerRe: [PATCH bpf-next v5 08/16] bpf: Support stack arguments for bpf functions
In reply to [email protected]Quoted 96 lines · On 4/16/26 9:43 PM, [email protected] wrote:
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0[ ... ]
Quoted 5 lines
include/linux/bpf_verifier.h +1
-0@@ -508,6 +513,17 @@ struct bpf_func_state {u32 callback_depth;u32 async_entry_cnt; +/* Stack argument tracking for current function. */ include/linux/bpf_verifier.h struct bpf_func_state { · line 526
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+struct bpf_reg_state *stack_arg_regs;/* Stack arg register states */ +u32 stack_arg_depth;/* Depth of stack_arg_regs array */ +u32 incoming_stack_arg_depth;/* Incoming stack arg count (callee) */ +u64 out_stack_arg_mask;/* Outgoing stack arg mask (caller) */ +}; + +#define bpf_get_spilled_stack_arg (slot, frame, mask) \ + (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \ + ((1 << frame->stack_arg_regs[slot].type) & (mask))) \ + ? &frame->stack_arg_regs[slot] : NULL)Does this macro work correctly? The existing
bpf_get_spilled_regmacro filters on frame->stack[slot].slot_type[], which is drawn from the STACK_* enum (STACK_SPILL,STACK_ITER, etc.) and callers pass masks like (1 <<STACK_SPILL) | (1 <<STACK_ITER).This new macro filters on bpf_reg_state.type which is from a completely different enum
bpf_reg_type(SCALAR_VALUE,PTR_TO_BTF_ID, etc.) with type modifier bits (PTR_MAYBE_NULL= BIT(8),PTR_TRUSTED= BIT(20), etc.).For example,
check_kfunc_call()in thercu_unlockpath uses:u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER); ... bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, clear_mask, ...);A stack-arg slot holding a MEM_RCU-tagged register would have
reg->typeincludingMEM_RCU(BIT(13)). Then (1 << 13) =0x2000, and0x2000&clear_mask(0x22) = 0, so the slot is skipped andMEM_RCUis not cleared onrcu_unlock.Additionally, when n >= 32 (for modifier bits like
PTR_TRUSTEDat BIT(20)), the shift (1 << n) invokes undefined behavior in C.Should this use a different approach to filter stack arg slots, perhaps based on initialization status (type !=
NOT_INIT)?Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 8 lines
kernel/bpf/verifier.c +4
-0@@ -1340,6 +1340,20 @@ static int copy_stack_state(struct bpf_func_state *dst, goto out_free_stack; } + n = src->stack_arg_depth /BPF_REG_SIZE ; +if (n) { + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, + sizeof(struct bpf_reg_state),kernel/bpf/verifier.c static int copy_stack_state(struct bpf_func_state *dst, · line 1356
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ GFP_KERNEL_ACCOUNT); +if (!dst->stack_arg_regs) +return -ENOMEM ; + + dst->stack_arg_depth = src->stack_arg_depth; + dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth; + dst->out_stack_arg_mask = src->out_stack_arg_mask; + } + +return 0 ;What happens when
src->stack_arg_depth== 0 but dst has non-zero values from prior work?The
stack_arg_depth,incoming_stack_arg_depth,out_stack_arg_mask, andstack_arg_regsfields come after the 'stack' field in structbpf_func_state, so the precedingmemcpy(dst, src, offsetof(..., stack))incopy_func_statedoes not overwrite them.When n == 0, the assignments to
dst->stack_arg_depth,dst->incoming_stack_arg_depth, anddst->out_stack_arg_maskare skipped, leaving stale values in dst.This can be triggered via:
1. Frame starts with
stack_arg_depth= 0,out_stack_arg_mask= 0
2.check_cond_jmp_opcallspush_stackwhich saves this state
3. Verifier continues on one branch; BPF_ST/BPF_STX writes to r11-8, r11-16 causecheck_stack_arg_writeto callgrow_stack_arg_slots, settingstack_arg_depth= 16 andout_stack_arg_mask= BIT(0)|BIT(1)
4. Branch ends;pop_stackcallsbpf_copy_verifier_statewhich callscopy_stack_state with src->stack_arg_depth == 0 5. Because n == 0, the assignment block is skipped, and dst retains stack_arg_depth = 16 and out_stack_arg_mask = BIT(0)|BIT(1)
Quoted 8 lines
6. The other branch never writes r11-8 or r11-16, but when it calls a subprog,
set_callee_statechecks:
if ((caller->out_stack_arg_mask&req_mask) !=req_mask) The stale mask makes this check pass incorrectlyWould the verifier accept a call with uninitialized stack args on that second branch? Should these fields be assigned unconditionally, similar to how
allocated_stackis handled earlier in this function?
The same issue as in the previous comment. Will fix.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
Yonghong Song
reviewerRe: [PATCH bpf-next v5 09/16] bpf: Reject stack arguments in non-JITed programs
In reply to [email protected]Quoted 5 lines · On 4/16/26 9:30 PM, [email protected] wrote:
commit
6f3447bea13f86a674e917f5f8ef4474cdd3b0bc
Author: Yonghong Song <[email protected]>bpf: Reject stack arguments in non-JITed programs
Quoted 7 lines
The interpreter does not understand the bpf register r11 (
BPF_REG_PARAMS) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.This commit fixes a bug but lacks a Fixes: tag.
Suggested: Fixes:
21229535935384b01e3cef3c117233b23b2f3902("bpf: Support stack arguments for bpf functions")
I will merge this patch and previous patch ("bpf: Support stack arguments for bpf functions") to avoid this issue.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
Yonghong Song
reviewerRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to [email protected]Quoted 39 lines · On 4/16/26 9:08 PM, [email protected] wrote:
Sashiko AI review found 2 potential issue(s):
- [Critical] False negative allowing tail calls and stack arguments together, leading to JIT memory corruption
- [High] State leak in verifier DFS causing false positive rejections of valid BPF programs --commit
df24720ae892549300f1910df635621a3875dbdf
Author: Yonghong Song <[email protected]>bpf: Reject stack arguments if tail call reachable
This commit updates the verifier to reject BPF programs if they contain both tail calls and stack arguments. This is enforced during the maximum stack depth check traversal.
Quoted 2 lines
Tailcalls are been deprecated. So reject stack arguments if tail call is in the way.
This isn't a bug, but there is a grammatical error here. "are been" should likely be "have been".
kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5669
Quoted 13 lines
kernel/bpf/verifier.c +5
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + }Looking at the earlier parts of
check_max_stack_depth_subprog(), the local variabletail_call_reachableonly appears to be set to true when jumping into a callee that has a tail call:if (subprog[idx].has_tail_call) tail_call_reachable = true;
Quoted 6 lines
If the main program (subprog[0]) natively contains a tail call,
tail_call_reachablewould remain false, causing this entire check block to be skipped.Does this logic successfully catch the case where the main program itself contains a tail call and also uses stack arguments?
Yes, we do have an issue here. Will fix.
Quoted 26 lines
If this escapes the verifier, the JIT might emit prologue tail call instructions (pushing 24 bytes on x86) while calculating stack argument offsets assuming no tail call pushes, overwriting the BPF program stack.
Could this regression lead to memory corruption in the JIT?
When the traversal pops a frame to return to the caller and jumps back to
continue_func, thetail_call_reachablevariable is not reset to false:/* pop caller idx from callee */ idx = dinfo[idx].caller; ... goto continue_func;If the verifier traverses a subprogram with a tail call, returns, and then traverses a completely unrelated sibling subprogram that uses stack arguments, it seems it would incorrectly evaluate
tail_call_reachableas true and reject the valid sibling.Is there a state leak in this DFS traversal that could cause false positive rejections of valid BPF programs?
Quoted 3 lines
kernel/bpf/verifier.c +0
-0subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
Yonghong Song
reviewerRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to [email protected]Quoted 9 lines · On 4/17/26 6:04 PM, [email protected] wrote:
Quoted 3 lines
bpf: Reject stack arguments if tail call reachable
Tailcalls are been deprecated. So reject stack arguments
This isn't a bug, but there's a grammar error: 'are been' should be 'have been' or 'are being'.
The commit message states that "Tailcalls are been deprecated." Is this claim accurate?
Quoted 6 lines
Based on earlier mailing list discussion, the restriction exists because tail calls and stack arguments require incompatible stack frame handling, not because tail calls are being deprecated.
Could the commit message be more precise about the actual technical reason for the restriction?
Stack arguments and tail calls CAN work together. However, since tail calls are in deprecation stage, it is not worthwhile to involve stack arguments and tail calls together.
I will keep commit message as is.
Quoted 49 lines
This commit appears to fix a bug but lacks a Fixes: tag. The suggested tag would be:
Fixes 13ee5d67a7c9 ("bpf: Support stack arguments for bpf functions")Quoted 6 lines
kernel/bpf/verifier.c +0
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5672
Quoted 10 lines
kernel/bpf/verifier.c +5
-0return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + } subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)When the DFS traversal in
check_max_stack_depth_subprog()pops a frame to return to the caller (moving from a child subprogram back to its parent), is thetail_call_reachablevariable reset to false?Looking at the code,
tail_call_reachableis initialized at the function start but appears to persist across the entire traversal. Consider this scenario:1. The verifier traverses a subprogram that contains a tail call, setting
tail_call_reachable= true
2. The traversal returns to the parent (pops the frame via idx = dinfo[idx].caller)
3. The traversal then visits a sibling subprogram that uses stack argumentsIn this scenario, wouldn't
tail_call_reachablestill be true from step 1, causing the check in the new code to incorrectly reject the sibling subprogram even though it has no tail calls in its own call chain?This could cause false positive rejections of valid BPF programs where one subprogram uses tail calls and a completely unrelated sibling subprogram uses stack arguments.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 10/16] bpf: Reject stack arguments if tail call reachable
In reply to [email protected]Quoted 60 lines · On 4/16/26 9:08 PM, [email protected] wrote:
Sashiko AI review found 2 potential issue(s):
- [Critical] False negative allowing tail calls and stack arguments together, leading to JIT memory corruption
- [High] State leak in verifier DFS causing false positive rejections of valid BPF programs --commit
df24720ae892549300f1910df635621a3875dbdf
Author: Yonghong Song <[email protected]>bpf: Reject stack arguments if tail call reachable
This commit updates the verifier to reject BPF programs if they contain both tail calls and stack arguments. This is enforced during the maximum stack depth check traversal.
Quoted 2 lines
Tailcalls are been deprecated. So reject stack arguments if tail call is in the way.
This isn't a bug, but there is a grammatical error here. "are been" should likely be "have been".
kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5669
Quoted 13 lines
kernel/bpf/verifier.c +5
-0@@ -5662,6 +5662,11 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,verbose (env,"cannot tail call within exception cb\n" );return -EINVAL ; } +if (subprog[tmp].incoming_stack_arg_depth || + subprog[tmp].outgoing_stack_arg_depth) { +verbose (env,"tail_calls are not allowed in programs with stack args\n" ); +return -EINVAL ; + }Looking at the earlier parts of
check_max_stack_depth_subprog(), the local variabletail_call_reachableonly appears to be set to true when jumping into a callee that has a tail call:if (subprog[idx].has_tail_call) tail_call_reachable = true;If the main program (subprog[0]) natively contains a tail call,
tail_call_reachablewould remain false, causing this entire check block to be skipped.Does this logic successfully catch the case where the main program itself contains a tail call and also uses stack arguments?
If this escapes the verifier, the JIT might emit prologue tail call instructions (pushing 24 bytes on x86) while calculating stack argument offsets assuming no tail call pushes, overwriting the BPF program stack.
Could this regression lead to memory corruption in the JIT?
When the traversal pops a frame to return to the caller and jumps back to
continue_func, thetail_call_reachablevariable is not reset to false:/* pop caller idx from callee */ idx = dinfo[idx].caller; ... goto continue_func;
Quoted 7 lines
If the verifier traverses a subprogram with a tail call, returns, and then traverses a completely unrelated sibling subprogram that uses stack arguments, it seems it would incorrectly evaluate
tail_call_reachableas true and reject the valid sibling.Is there a state leak in this DFS traversal that could cause false positive rejections of valid BPF programs?
You are right. This has been their for sometime. So I guess we can live with it even for stack arguments.
Quoted 4 lines
Quoted 4 lines
kernel/bpf/verifier.c +0
-0subprog[tmp].tail_call_reachable = true; }if (subprog[0 ].tail_call_reachable)
Yonghong Song
reviewerRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to [email protected]Quoted 25 lines · On 4/16/26 9:40 PM, [email protected] wrote:
Sashiko AI review found 3 potential issue(s):
- [Critical] Precision tracking (mark_chain_precision) is incorrectly bypassed or misapplied for kfunc scalar arguments passed via the stack, allowing malicious BPF programs to bypass verifier safety checks through state pruning.
- [High]get_kfunc_arg_reg()fetches lookahead stack arguments without bounds checking, leading to an out-of-bounds memory read or NULL pointer dereference when resolving memory sizes.
- [High] Overwritinginsn->offinbpf_fixup_kfunc_calldestroys the BTF file descriptor index, breaking module kfunc calls on architectures that use JIT far calls.
--commit
15b2d48bd7c1080e972b83211cc9794ac2ba763f
Author: Yonghong Song <[email protected]>bpf: Support stack arguments for kfunc calls
Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r11-based stack arg area.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -11653,6 +11659,19 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; }
Quoted 14 lines
kernel/bpf/verifier.c bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · line 11673
Quoted 12 lines
kernel/bpf/verifier.c +12
-0+static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi =out_arg_spi (caller, argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_regs[spi]; +}Does
get_kfunc_arg_reg()need to verify that spi is within bounds and the stack slot is initialized?
The checking is below:
static bool is_stack_arg_slot_initialized(struct bpf_func_state *state, int spi)
{
if (spi >= (int)(state->stack_arg_depth / BPF_REG_SIZE))
return false;
return state->stack_arg_regs[spi].type != NOT_INIT;
} if (i < MAX_BPF_FUNC_REG_ARGS) {
reg = ®s[i + 1];
} else {
/* Retrieve the reg state from the outgoing stack arg slot. */
struct bpf_func_state *caller = cur_func(env);
int spi = out_arg_spi(caller, i - MAX_BPF_FUNC_REG_ARGS);
if (!is_stack_arg_slot_initialized(caller, spi)) {
verbose(env, "stack %s not properly initialized\n",
reg_arg_name(env, argno));
return -EINVAL;
}
tmp_reg = caller->stack_arg_regs[spi];
reg = &tmp_reg;
regno = -1;
}spi has to be valid. Otherwise, verification will fail.
Quoted 5 lines
When verifying pointer arguments that have a trailing size argument (
KF_ARG_PTR_TO_MEM_SIZE), the verifier looks ahead at argno + 1. If a BPF program maliciously omits the stack size argument, could spi exceedcaller->stack_arg_depthand lead to an out-of-bounds memory read?
If the bpf program maliciously omits the stack size argument, btf_check_func_arg_match() should catch this and reject the program.
Quoted 18 lines
[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 12442
Quoted 12 lines
kernel/bpf/verifier.c +5
-3@@ -12394,9 +12435,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ reg_arg_name(env, argno));return -EINVAL ; } - ret =mark_chain_precision (env, regno); -if (ret <0 ) -return ret; +if (regno >0 ) { + ret =mark_chain_precision (env, regno); +if (ret <0 ) +return ret; + }Stack arguments use regno = -1, so it seems
mark_chain_precision()is bypassed for scalar constants and size arguments passed on the stack.
Quoted 3 lines
If precision tracking is bypassed, could the verifier's state equivalence logic consider states with different scalar values in these stack slots as equivalent and incorrectly prune them?
In this particular case, we know reg is a known constant, we can do reg->precise = true.
Will fix.
Quoted 21 lines
[ ... ]
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20220
Quoted 13 lines
kernel/bpf/verifier.c +9
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr); +/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ;For architectures where
bpf_jit_supports_far_kfunc_call()returns true (such as s390x, loongarch, and powerpc), they resolve the address later usingbpf_get_kfunc_addr(), which requires the BTF file descriptor index stored ininsn->off.
Quoted 3 lines
If
insn->offis overwritten here, could it destroy the BTF file descriptor index and break module kfunc calls during JIT compilation?
The JIT implementation has changed and the above is dead code. Will remove.
Yonghong Song
reviewerRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to [email protected]Quoted 54 lines · On 4/16/26 9:43 PM, [email protected] wrote:
Quoted 3 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr);kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20221
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +Is
insn->offreally "no longer needed for BTF fd index" after this point? Looking atbpf_jit_find_kfunc_model()inkernel/bpf/fixups.c, the function is called during JIT compilation from every BPF JIT backend that supports kfunc calls (x86_32, s390, riscv64, ppc64, loongarch).It performs a bsearch on the
kfunc_tabkeyed by (insn->imm,insn->off) usingkfunc_desc_cmp_by_imm_off:kernel/bpf/fixups.c:kfunc_desc_cmp_by_imm_off() { ... if (a->imm != b->imm) return a->imm < b->imm ? -1 : 1; if (a->offset != b->offset) return a->offset < b->offset ? -1 : 1; return 0; }The
desc->offsetwas populated from the originalinsn->offduringbpf_add_kfunc_call(), where for module kfuncs it is a non-zero BTF fd index.The call order is:
bpf_check()
->bpf_do_misc_fixups()-> bpf_fixup_kfunc_call() (overwrites insn->off to 0 or stack_arg_count) -> bpf_prog_select_runtime() -> bpf_int_jit_compile() -> arch JIT -> bpf_jit_find_kfunc_model() (reads insn->off for bsearch key)After this patch, for a kfunc with 5 or fewer args that lives in a module (non-zero
btf_fd_idx),insn->offis zeroed, and the bsearch will fail to find the entry that was stored with offset=btf_fd_idx.
Quoted 8 lines
For kfuncs with more than 5 args, the new value of
insn->off(stack arg count) also will not matchdesc->offset(btf_fd_idxor 0), so module kfuncs with stack args are also broken, and vmlinux kfuncs with more than 5 args get a nonzeroinsn->offthat no longer matches desc->offset=0.Can the stack arg count be communicated to the JIT without clobbering the
btf_fd_idxthat bsearch still keys on?
The above code is dead code. Will remove.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24546989054
Yonghong Song
reviewerRe: [PATCH bpf-next v5 11/16] bpf: Support stack arguments for kfunc calls
In reply to [email protected]Quoted 49 lines · On 4/17/26 6:04 PM, [email protected] wrote:
Quoted 3 lines
kernel/bpf/verifier.c +0
-0[ ... ]
kernel/bpf/verifier.c bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · line 11670
Quoted 13 lines
kernel/bpf/verifier.c +12
-0@@ -11653,6 +11659,19 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) +static struct bpf_reg_state *get_kfunc_arg_reg (struct bpf_verifier_env *env,int argno) +{ +struct bpf_func_state *caller; +int spi; + +if (argno <MAX_BPF_FUNC_REG_ARGS ) +return &cur_regs (env)[argno +1 ]; + + caller =cur_func (env); + spi =out_arg_spi (caller, argno -MAX_BPF_FUNC_REG_ARGS ); +return &caller->stack_arg_regs[spi]; +}Can the verifier crash here? Looking at
get_kfunc_arg_reg, when argno >=MAX_BPF_FUNC_REG_ARGS, the function returns &caller->stack_arg_regs[spi] without checking whetherstack_arg_regsis NULL.The
stack_arg_regsarray is only allocated bygrow_stack_arg_slots, which is called fromcheck_stack_arg_writewhen the program writes outgoing stack args via r11. If a malformed BPF program invokes a kfunc without emitting any stack stores beforehand,stack_arg_regswill remain NULL.Consider
get_kfunc_ptr_arg_typecallingget_kfunc_arg_regto peek at argument i+1 when checking for buffer+__sz pairs:kernel/bpf/verifier.c:get_kfunc_ptr_arg_type() { if (argno + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) || ... }When argno=4 (pointer in r5) and argno+1=5 (size in first stack slot),
get_kfunc_arg_reg(env, 5)enters the stack-arg branch and dereferencescaller->stack_arg_regseven though it may be NULL. If the arg type is a scalar (which __sz/__szk suffixes imply),is_kfunc_arg_mem_sizewill try to readreg->type, causing a NULL pointer dereference.The
is_stack_arg_slot_initializedgate you added incheck_kfunc_argsonly fires when the loop reaches i=5; by then the peek at i=4 has already dereferenced the NULL pointer.
Quoted 5 lines
Should
get_kfunc_arg_regverify thatstack_arg_regsis non-NULL and spi < stack_arg_depth/BPF_REG_SIZE before returning the pointer? Alternatively, shouldcheck_kfunc_calladd anout_stack_arg_maskpre-check mirroring the bpf-to-bpf gate that preventscheck_func_arg_reg_offfrom reading uninitialized stack slots?
Good points. Indeed we should check validity of spi in
spi = out_arg_spi(caller, argno - MAX_BPF_FUNC_REG_ARGS)
Will fix.
Quoted 73 lines
Quoted 19 lines
kernel/bpf/verifier.c +2
-4@@ -11660,8 +11679,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, const char *ref_tname, const struct btf_param *args, int argno, int nargs, struct bpf_reg_state *reg) { - u32 regno = argno + 1; -struct bpf_reg_state *regs =cur_regs (env); bool arg_mem_size = false; if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||@@ -11670,8 +11687,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,return KF_ARG_PTR_TO_CTX ;if (argno +1 < nargs && - (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]) || -is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ], ®s[regno +1 ]))) + (is_kfunc_arg_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )) || +is_kfunc_arg_const_mem_size (meta->btf, &args[argno +1 ],get_kfunc_arg_reg (env, argno +1 )))) arg_mem_size = true;[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -20145,6 +20209,16 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (!bpf_jit_supports_far_kfunc_call ()) insn->imm =BPF_CALL_IMM (desc->addr);kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 20221
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* +* After resolving the kfunc address, insn->off is no longer needed +* for BTF fd index. Repurpose it to store the number of stack args +* so the JIT can marshal them. +*/ +if (desc->func_model.nr_args >MAX_BPF_FUNC_REG_ARGS ) + insn->off = desc->func_model.nr_args -MAX_BPF_FUNC_REG_ARGS ; +else + insn->off =0 ; +This concern was raised by the bpf-ci bot and sashiko-bot in the v5 review thread but does not appear to have been addressed.
Overwriting
insn->offdestroys the BTF file descriptor index required bybpf_jit_find_kfunc_model()for module kfunc lookups. Looking at the bsearch inkernel/bpf/fixups.c, it uses (insn->imm,insn->off) as the key:kernel/bpf/fixups.c:bpf_jit_find_kfunc_model() { key.addr = insn->imm; key.offset = insn->off; entry = bsearch(&key, tab, tab_count, sizeof(*tab), kfunc_addr_cmp); }For module kfuncs,
insn->offstores a non-zerobtf_fd_idxset duringfind_kfunc_desc. After this patch overwrites it with the stack arg count (or 0), the bsearch will fail to find descriptors that were stored with offset=btf_fd_idx.This breaks module kfunc calls on architectures using JIT far calls (s390x, loongarch, powerpc) for any module kfunc with 5 or fewer args (
insn->offzeroed) or with more than 5 args (insn->offnow contains the stack arg count instead ofbtf_fd_idx, still mismatchingdesc->offset). Vmlinux kfuncs with more than 5 args also get a nonzeroinsn->offthat no longer matches desc->offset=0.Can the stack arg count be communicated to the JIT without clobbering the
btf_fd_idxthat bsearch still keys on?The original concern was raised here:
https://lore.kernel.org/bpf/e5a6e801c10994cc2e03b0cd57e76152eb234493a6b7395a9859feade4a6575b@mail.kernel.org/---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 12/16] bpf: Enable stack argument support for x86_64
In reply to [email protected]Quoted 43 lines · On 4/16/26 10:03 PM, [email protected] wrote:
Sashiko AI review found 1 potential issue(s):
- [High] Enablingbpf_jit_supports_stack_args()forx86_64without the corresponding JIT implementation introduces a bisection-breaking regression that can lead to arbitrary kernel memory writes and ABI violations. --commit
b40feee41e4472d0e27be38b4c99c940abe3c1ba
Author: Yonghong Song <[email protected]>bpf: Enable stack argument support for
x86_64This commit enables stack argument support for the
x86_64JIT and verifier to allow BPF programs and kfuncs to use more than 5 arguments.arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_kfunc_call(void) · line 3944
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +5
-0@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +Does enabling this without the actual JIT implementation break bisection?
If
bpf_jit_supports_stack_args()returns true, the verifier will allow instructions usingBPF_REG_PARAMS(register 11) to handle stack arguments. However, since the JIT support isn't implemented in this commit,BPF_REG_PARAMSis not yet defined in the JIT's reg2hex array.Could this cause reg2hex[11] to implicitly default to 0, which maps to RAX (
BPF_REG_0)? If the verifier emits instructions usingBPF_REG_PARAMS, the x86 JIT might miscompile them using RAX as the base register, potentially allowing an arbitrary kernel memory write since R0 is controllable by the BPF program.Additionally, it appears kfunc calls with >5 arguments won't have their arguments correctly marshaled to the System V ABI (like moving the 6th argument to R9), which could lead to reading uninitialized garbage from registers and the stack.
Quoted 4 lines
I see the JIT implementation is added later in the series in commit
41dcf99330b6b("bpf,x86: Implement JIT support for stack arguments"). Should the JIT implementation be merged before or in the same patch as this feature enablement to maintain bisection?
Good point. Will enable stack args in JIT patch.
Quoted 3 lines
[ ... ]
Yonghong Song
reviewerRe: [PATCH bpf-next v5 13/16] bpf,x86: Implement JIT support for stack arguments
In reply to [email protected]Quoted 41 lines · On 4/16/26 9:44 PM, [email protected] wrote:
Sashiko AI review found 4 potential issue(s):
- [High] Data corruption of Argument 6 across function calls due to caller-saved register clobbering.
- [High] Missing size handling for Argument 6 allows verifier bounds tracking bypass and out-of-bounds memory writes.
- [High] The offset translation formula for outgoing arguments 7+ reverses the memory layout, corrupting sub-slot accesses.
- [Medium] The JIT misroutes stack accesses because it rigidly assumes LDX only targets incoming arguments and STX/ST only target outgoing arguments. --commit
41dcf99330b6b0a3804185f2a284a76be8340ed1
Author: Yonghong Song <[email protected]>bpf,x86: Implement JIT support for stack arguments
Add
x86_64JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r11 (BPF_REG_PARAMS) in BPF bytecode, which the JIT translates to native code.Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -1664,16 +1692,45 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image stack_depth = bpf_prog->aux->stack_depth; + stack_arg_depth = bpf_prog->aux->stack_arg_depth; + incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth; + outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth; priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1706
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +5
-0if (priv_stack_ptr) { priv_frame_ptr = priv_stack_ptr +PRIV_STACK_GUARD_SZ +round_up (stack_depth,8 ); stack_depth =0 ; } +/* +* Follow x86-64 calling convention for both BPF-to-BPF and +* kfunc calls: +* - Arg 6 is passed in R9 register +* - Args 7+ are passed on the stack at [rsp] Can the use of the R9 register cause data corruption across function calls?
Quoted 8 lines
The x86-64 C calling convention defines R9 as a caller-saved register. However, the BPF verifier models argument 6 as stack memory, which implies its value should be preserved across calls.
If a BPF program reads incoming argument 6 after making a helper or kfunc call, will it receive a clobbered value instead of the original argument? Similarly, could a helper call scheduled by LLVM between the setup of outgoing argument 6 and the target call corrupt the R9 register?
Yes. This is correct. The verifier part needs to do proper invalidation to avoid this issue in JIT.
Quoted 18 lines
[ ... ]
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1786
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +13
-0+/* +* Base offset from rbp for translating BPF outgoing args 7+ +* to native offsets. BPF uses negative offsets from r11 +* (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses +* positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for +* arg8, ...). Arg 6 goes to R9 directly. +* +* The translation reverses direction: +* native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 +* +* Note that tail_call_reachable is guaranteed to be false when +* stack args exist, so tcc pushes need not be accounted for. +*/ Does this formula correctly handle multi-byte sub-slot accesses?
Quoted 8 lines
Since the formula subtracts the BPF offset, the memory layout within the 8-byte slots appears mathematically reversed. While this correctly maps the 8-byte slot bases (e.g., -16 maps to rsp + 0), it seems to point to the wrong memory for partial writes.
For example, if a program writes to the upper 32 bits of Arg 7 at BPF offset -12, the formula yields BASE - (-12) - 16 = BASE - 4. Will this map the access out-of-bounds below the stack pointer instead of to [rsp + 4]?
Currently, we do not support sub-slot access. All r11 load/store are 8 byte size. I will ensure this in verifier checking.
Quoted 33 lines
[ ... ]
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +5
-1@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off))arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2240
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +9
-1-EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets. +* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + }Are the size constraints missing for Argument 6 here?
The check for Arg 6 (
insn->off== -8) is only implemented under theBPF_DWcase. For sizes B, H, and W, execution will fall through to the st: label. This would incorrectly translate the access as a normal stack operation using the formula above, writing out-of-bounds below the stack pointer at [rsp - 8] instead of populating R9.
Quoted 6 lines
Additionally, for
BPF_LDX, a 1-byte load emits a full 64-bit move to R9. Since the upper 56 bits of R9 are undefined for 1-byte arguments, could they contain garbage data? This would bypass the verifier's strict value bounds tracking, which assumes the loaded value is strictly within [0, 255].[ ... ]
Will enforce 8-byte width for r11 based load/stores.
Quoted 12 lines
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +7
-0@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */ +EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ;
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2375
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }Does this logic misroute stack accesses by assuming LDX always targets incoming arguments and STX/ST always target outgoing arguments?
Yes.
Quoted 10 lines
If a program writes an outgoing argument ([r11 - 16]) and then reads it back via LDX, the JIT appears to apply the incoming argument translation mapping the read to [rbp - 16] (the local stack) instead of [rsp] (the outgoing area). This would return uninitialized data.
Conversely, if a program modifies an incoming argument ([r11 + 16]) using STX, the JIT would apply the outgoing translation formula, yielding BASE - 16 - 16 = BASE - 32, which writes out-of-bounds below the stack pointer.
In verifier, we have a check:
static int check_stack_arg_access(struct bpf_verifier_env *env,
struct bpf_insn *insn, const char *op)
{
int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
if (size != BPF_REG_SIZE) {
verbose(env, "stack arg %s must be %d bytes, got %d\n",
op, BPF_REG_SIZE, size);
return -EINVAL;
}
if (insn->off == 0 || insn->off % BPF_REG_SIZE) {
verbose(env, "stack arg %s offset %d not aligned to %d\n",
op, insn->off, BPF_REG_SIZE);
return -EINVAL;
}
/* Reads use positive offsets (incoming), writes use negative (outgoing) */
if (op[0] == 'r' && insn->off < 0) {
verbose(env, "stack arg read must use positive offset, got %d\n",
insn->off);
return -EINVAL;
}
if (op[0] == 'w' && insn->off > 0) {
verbose(env, "stack arg write must use negative offset, got %d\n",
insn->off);
return -EINVAL;
}
return 0;
}load must have positive offset and store must have negative offset. I will fix the above checking '< 0' => '<= 0" and '> 0' => '>= 0'.
Yonghong Song
reviewerRe: [PATCH bpf-next v5 13/16] bpf,x86: Implement JIT support for stack arguments
In reply to [email protected]Quoted 64 lines · On 4/17/26 6:20 PM, [email protected] wrote:
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 17 lines
arch/x86/net/bpf_jit_comp.c +9
-2@@ -2129,12 +2222,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imageEMIT1 (0xC7 ); goto st;case BPF_ST |BPF_MEM |BPF_DW : +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store immediate in r9 register */ +emit_mov_imm64 (&prog,X86_REG_R9 , imm32 >>31 , (u32 )imm32); +break ; + }EMIT2 (add_1mod (0x48 , dst_reg),0xC7 ); -st:if (is_imm8 (insn->off)) -EMIT2 (add_1reg (0x40 , dst_reg), insn->off); +st: insn_off = insn->off; +if (dst_reg ==BPF_REG_PARAMS ) { +/* Args 7+: reverse BPF negative offsets to +* x86 positive rsp offsets. arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2244
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-1+* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... +*/ + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + } +if (is_imm8 (insn_off)) +EMIT2 (add_1reg (0x40 , dst_reg), insn_off);else -EMIT1_off32 (add_1reg (0x80 , dst_reg), insn->off); +EMIT1_off32 (add_1reg (0x80 , dst_reg), insn_off);The special case for arg 6 at offset -8 is only implemented for
BPF_ST|BPF_MEM|BPF_DW. What happens with smaller access sizes (BPF_B,BPF_H,BPF_W)?Looking at the code flow, if a
BPF_STwith size B/H/W targetsdst_reg==BPF_REG_PARAMSat offset -8, execution would fall through to the st: label. The offset translation formula would then be applied:
insn_off=outgoing_arg_base-outgoing_rsp- (-8) - 16
=outgoing_arg_base-outgoing_rsp- 8For example, with
outgoing_arg_base= -40 andoutgoing_rsp= 0, this yields -32, which would attempt to write at [rbp - 32], outside the designated outgoing argument area at [rsp].Does this allow writing out-of-bounds below the stack pointer instead of updating R9?
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +6
-1@@ -2144,7 +2251,17 @@ st: if (is_imm8(insn->off))case BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_W :case BPF_STX |BPF_MEM |BPF_DW : -emit_stx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn->off); +if (dst_reg ==BPF_REG_PARAMS && insn->off ==-8 ) { +/* Arg 6: store register value in r9 */ +EMIT_mov (X86_REG_R9 , src_reg); +break ; + } + insn_off = insn->off;
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 2240
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+if (dst_reg ==BPF_REG_PARAMS ) { + insn_off = outgoing_arg_base - outgoing_rsp - insn_off -16 ; + dst_reg =BPF_REG_FP ; + }The offset translation formula subtracts the BPF offset, which reverses the addressing direction. While this correctly maps 8-byte aligned slot bases (e.g., -16 → rsp+0, -24 → rsp+8), can it handle partial writes within those slots?
Currently we do not support partial writes within those slots. In llvm, all stack arguments are promoted to 64bit.
Quoted 38 lines
For instance, if a program writes to the upper 32 bits of arg 7 at BPF offset -12:
insn_off=outgoing_arg_base-outgoing_rsp- (-12) - 16
=outgoing_arg_base-outgoing_rsp- 4With
outgoing_arg_base= -40 andoutgoing_rsp= 0, this computes to -36, mapping the access to [rbp - 36] instead of [rsp + 4].Does this formula produce out-of-bounds accesses below the stack pointer for partial writes?
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3
-0@@ -2243,6 +2360,19 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : insn_off = insn->off; +if (src_reg ==BPF_REG_PARAMS ) { +if (insn_off ==8 ) { +/* Incoming arg 6: read from r9 */ arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2375
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10
-0+EMIT_mov (dst_reg,X86_REG_R9 ); +break ; + } + src_reg =BPF_REG_FP ; +/* +* Incoming args 7+: native_off == bpf_off +* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) +* No offset adjustment needed. +*/ + }When reading incoming arg 6 with smaller access sizes (
BPF_LDX|BPF_MEM| BPF_B/H/W), the code performs a full 64-bit move from R9 to the destination register:
Quoted 1 line
EMIT_mov(dst_reg, X86_REG_R9);
Again, we only support 64bit operations.
Quoted 7 lines
For sub-64-bit arguments, are the upper bits of R9 guaranteed to be zero? If not, could this move garbage data into the upper bits of
dst_reg, bypassing the verifier's strict bounds tracking which expects values to stay within their declared ranges (e.g., [0, 255] for bytes)?
Quoted 6 lines
Reference: https://lore.kernel.org/bpf/[email protected]/
The JIT translation assumes that LDX operations target incoming arguments (positive offsets from r11) and STX/ST operations target outgoing arguments (negative offsets from r11). Is this assumption always valid?
Yes.
Quoted 7 lines
Consider these scenarios:
Scenario 1: A program writes to an outgoing argument slot at [r11 - 16] via STX, then reads it back via LDX. The JIT would:
- Store: apply outgoing translation → [rsp + 0]
- Load: apply incoming translation → [rbp + 16] (different location)
A LDX with [r11 - 16] will be rejected by verifier.
Quoted 6 lines
The load would read from the BPF program stack area rather than the outgoing argument area, potentially returning uninitialized data.
Scenario 2: A program attempts to modify an incoming argument at [r11 + 16] via STX. The JIT would apply the outgoing translation:
Again, this is illegal for STX with positive offset.
Quoted 19 lines
insn_off=outgoing_arg_base-outgoing_rsp- 16 - 16
=outgoing_arg_base-outgoing_rsp- 32With
outgoing_arg_base= -40 andoutgoing_rsp= 0, this yields -72, writing to [rbp - 72], which could be out-of-bounds below the allocated stack area.Does the verifier enforce a constraint that reads can only target positive r11 offsets and writes can only target negative r11 offsets?
Reference: https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 14/16] selftests/bpf: Add tests for BPF function stack arguments
In reply to [email protected]Quoted 54 lines · On 4/17/26 5:52 PM, [email protected] wrote:
Quoted 28 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +22
-0@@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include <network_helpers.h> +#include "stack_arg.skel.h" +#include "stack_arg_kfunc.skel.h" + +static void run_subtest (struct bpf_program *prog,int expected) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + + prog_fd =bpf_program__fd (prog); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval, expected,"retval" ); +}[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +11
-0+static void test_async_cb_many (void ) +{ +struct stack_arg *skel; + + skel =stack_arg__open (); +if (!ASSERT_OK_PTR (skel,"open" )) +return ; + +if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out;tools/testing/selftests/bpf/prog_tests/stack_arg.c line 66
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +10
-0+ } + +if (!ASSERT_OK (stack_arg__load (skel),"load" )) + goto out; + +run_subtest (skel->progs.test_async_cb_many_args,0 ); + +out: +stack_arg__destroy (skel); +}Does this test adequately verify the asynchronous timer callback execution and its result?
Quoted 8 lines
Since
run_subtest()only runs the program and asserts the return value is 0, the skeleton is immediately destroyed afterward. The timer callback might not have time to execute and could be canceled during map destruction.The result of the callback (
timer_result) is never verified in userspace. Should there be a polling loop waiting for the timer to fire, followed by an assertion checking thatskel->bss->timer_result== 360?
Okay, the timer test is incomplete. Let me add more meet to actually get a timer result.
Quoted 11 lines
Reference: https://lore.kernel.org/bpf/[email protected]/
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24592562564
Yonghong Song
reviewerRe: [PATCH bpf-next v5 15/16] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
In reply to [email protected]Quoted 34 lines · On 4/16/26 9:28 PM, [email protected] wrote:
Sashiko AI review found 2 potential issue(s):
- [Medium] Theprog_test_big_argstruct useslong, which fails to test the >8 byte limit on 32-bit architectures and causes BTF size mismatches.
- [Medium] UsingASSERT_ERR(stack_arg_fail__load(skel))masks the actual verifier error reason, allowing the negative test to pass for unrelated reasons. --commit
1ec644adfded3aa88befcb32fc73958c68b7e018
Author: Yonghong Song <[email protected]>selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
This commit adds a selftest to ensure the verifier rejects kfunc calls where a stack argument exceeds the 8-byte register-sized slot limit. It introduces a new kfunc taking a struct argument and a BPF program that attempts to pass a large struct to it.
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_pass2 { · line 57
Quoted 13 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5
-0@@ -50,6 +50,11 @@ struct prog_test_pass2 { } x; }; +struct prog_test_big_arg { +long a; +long b; +}; +Does this structure achieve the intended >8 byte size on 32-bit architectures?
When the test kernel module is compiled on a 32-bit host, long is 32-bit, making the struct exactly 8 bytes. Since the BPF target always treats long as a 64-bit integer, this makes the struct 16 bytes in the BPF program.
Quoted 6 lines
This size discrepancy between the BPF program's BTF and the host kernel's BTF might cause the verifier to reject the load for a BTF size mismatch rather than the intended stack argument limit.
Could these fields use explicitly sized types like
__u64to ensure the struct is consistently larger than 8 bytes across all architectures?
Good point, using __64 is much better.
Quoted 22 lines
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +0
-0@@ -0,0 +1,24 @@[ ... ]
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c line 20
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +6
-0+if (!skel->rodata->has_stack_arg) { +test__skip (); + goto out; + } + +ASSERT_ERR (stack_arg_fail__load (skel),"load_should_fail" );Could this generic error assertion mask the actual verifier rejection reason?
If the program fails to load due to an unrelated issue, such as a missing BTF, a syntax error, or the architecture-dependent size mismatch described above, the test will incorrectly report success because it only checks that the load operation failed.
Quoted 4 lines
Would it be better to refactor this test to use the standard
test_loaderinfrastructure (viaRUN_TESTS) with__failureand__msgannotations in the BPF program? That would ensure the rejection is explicitly due to the argument size limit.
will do RUN_TESTS style for this test.
Quoted 5 lines
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +4
-0+ +out: +stack_arg_fail__destroy (skel); +}
Yonghong Song
reviewerRe: [PATCH bpf-next v5 16/16] selftests/bpf: Add verifier tests for stack argument validation
In reply to [email protected]Quoted 21 lines · On 4/16/26 9:38 PM, [email protected] wrote:
Sashiko AI review found 3 potential issue(s):
- [Medium] Negative verifier tests for stack arguments use inherently invalid memory accesses, causing them to pass even if the verifier has bugs.
- [Medium]stack_arg_read_uninitializedreads fromr11 + 8instead of a valid negative offset, failing to properly test uninitialized stack argument slots.
- [Low] Unusedsubprog_8argsfunction defined in the test file.
--commit
24b1eb352e7d04e7c35a9fbc88c6b91d08b17832
Author: Yonghong Song <[email protected]>selftests/bpf: Add verifier tests for stack argument validation
This commit adds inline-asm based verifier tests to directly exercise stack argument validation logic.
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +0
-0[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 33
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+__noinline __used +static int subprog_8args(int a, int b, int c, int d, int e, int f, int g, int h) +{ +return a + b + c + d + e + f + g + h; +}Is
subprog_8argsintended to be used in any of the tests? It does not seem to be called anywhere in this file.
Dead code. Will remove.
Quoted 13 lines
[ ... ]
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+SEC("tc") +__description("stack_arg: read from uninitialized stack arg slot") +__failure +__arch_x86_64 +__msg("invalid read from stack arg") +__naked void stack_arg_read_uninitialized (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r11 + 8);" + "r0 = 0;"
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 58
Quoted 4 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +4
-0+"exit;" + ::: __clobber_all + ); +}Should this read from r11 - 8 instead? Since outgoing stack arguments are at negative offsets from r11, reading from r11 + 8 is out of bounds and might be rejected for the wrong reason, rather than testing an uninitialized but valid stack argument slot.
The read must be 'r11 + 8' and store will be 'r11 - <positive_off>'.
Quoted 31 lines
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+__noinline __used +static long subprog_deref_arg6(long a, long b, long c, long d, long e, long *f) +{ +return *f; +}[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +8
-0+SEC("tc") +__description("stack_arg: release_reference invalidates stack arg slot") +__failure +__arch_x86_64 +__msg("R1 invalid sock access") +__naked void stack_arg_release_ref (void ) +{ + asm volatile ([ ... ]
tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 220
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* Release the reference — invalidates the stack arg slot */ +"r1 = r0;" +"call %[bpf_sk_release];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;" Can this test pass even if the verifier fails to invalidate the stack slot?
Quoted 7 lines
subprog_deref_arg6()performs an 8-byte read on the argument. Since BPF sockets only allow up to 4-byte reads, an 8-byte read is always invalid. If the reference isn't correctly invalidated, the valid socket pointer will still be rejected with "R1 invalid sock access".To properly test the invalidation, should the subprogram perform a valid 4-byte read instead?
4-byte stack argument is not supported.
Quoted 37 lines
[ ... ]
Quoted 18 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +18
-0+SEC("tc") +__description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data") +__failure +__arch_x86_64 +__msg("invalid access to packet") +__naked void stack_arg_stale_pkt_ptr (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = *(u32 *)(r6 + %[__sk_buff_data]);" +"r8 = *(u32 *)(r6 + %[__sk_buff_data_end]);" +/* check pkt has at least 1 byte */ +"r0 = r7;" +"r0 += 1;" +"if r0 > r8 goto l0_%=;" +/* Store valid pkt pointer to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r7;" + /* bpf_skb_pull_data invalidates all pkt pointers */tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 259
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+"r1 = r6;" +"r2 = 0;" +"call %[bpf_skb_pull_data];" +/* Call subprog that dereferences arg6 — should fail */ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"call subprog_deref_arg6;" Similarly, does this properly test packet pointer invalidation? The test verifies 1 byte of the packet pointer, but
subprog_deref_arg6()reads 8 bytes. Ifbpf_skb_pull_data()fails to invalidate the pointer, the 8-byte read would still be rejected as out-of-bounds, yielding the expected "invalid access to packet" message.
Quoted 2 lines
Should the packet bounds check verify at least 8 bytes so that a valid read is guaranteed if the pointer isn't invalidated?
yes, I can do r0 += 8 instead.
Quoted 32 lines
[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +8
-0+SEC("tc") +__description("stack_arg: null propagation rejects deref on null branch") +__failure +__arch_x86_64 +__msg("R1 invalid mem access") +__naked void stack_arg_null_propagation_fail (void ) +{ + asm volatile ([ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +5
-0+"call %[bpf_map_lookup_elem];" +/* Store PTR_TO_MAP_VALUE_OR_NULL to outgoing arg6 slot */ +"*(u64 *)(r11 - 8) = r0;" +/* null check on r0 */ +"if r0 != 0 goto l0_%=;" tools/testing/selftests/bpf/progs/verifier_stack_arg.c line 299
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +10
-0+/* +* On null branch, outgoing slot is SCALAR(0). +* Call subprog that dereferences arg6 — should fail. +*/ +"r1 = 0;" +"r2 = 0;" +"r3 = 0;" +"r4 = 0;" +"r5 = 0;" +"call subprog_deref_arg6;" Does the
__msg()substring match accept the wrong failure mode here? Dereferencing an unrefinedPTR_TO_MAP_VALUE_OR_NULLyields "R1 invalid mem access 'map_value_or_null'", while dereferencing SCALAR yields "R1 invalid mem access 'scalar'".
Quoted 4 lines
The substring match "R1 invalid mem access" accepts both, allowing the test to pass even if null propagation fails and the register is still tracked as a map pointer. Should it match the exact scalar error, or perhaps test the non-null branch for success instead?
Will fix.
Quoted 19 lines · Puranjay Mohan
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +19
-0+/* call1: subprog_7args_inner_call(1,2,3,4,5,6,7) +* internally calls subprog_7args(1,2,3,4,5,100,200)=315 +* which clobbers stack arg registers with 100, 200. +* Returns 315. +*/ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"*(u64 *)(r11 - 8) = 6;" +"*(u64 *)(r11 - 16) = 7;" +"call subprog_7args_with_inner_call;" +"r6 = r0;" /* 315 */ +/* call2: reuse r11 stores from call1 (no re-store). +* subprog_7args(1,2,3,4,5,6,7) should return 28. +* If clobbered: subprog_7args(1,2,3,4,5,100,200)=315. +*/ +"r1 = 1;"
tools/testing/selftests/bpf/progs/verifier_stack_arg.c __naked void stack_arg_sequential_calls(void) · line 514
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +10
-0+"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +/* +* the 6th and 7th arg should be 6, 7 as set above. +* But the inner call in subprog_7args_inner_call clobbered +* them and reusing again without a re-store will be wrong. +*/ +"call subprog_7args;" /* should return 28, but will be 315 due to clobber */
Puranjay is correct.
The verifier should error here.
callee can modify 6 and 7th args in place in the stack. All args register or stack are caller saved.
I believe it matches other archs.
Yonghong Song
reviewerQuoted 30 lines · Alexei Starovoitov
Quoted 19 lines · Puranjay Mohan
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +19
-0+/* call1: subprog_7args_inner_call(1,2,3,4,5,6,7) +* internally calls subprog_7args(1,2,3,4,5,100,200)=315 +* which clobbers stack arg registers with 100, 200. +* Returns 315. +*/ +"r1 = 1;" +"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +"*(u64 *)(r11 - 8) = 6;" +"*(u64 *)(r11 - 16) = 7;" +"call subprog_7args_with_inner_call;" +"r6 = r0;" /* 315 */ +/* call2: reuse r11 stores from call1 (no re-store). +* subprog_7args(1,2,3,4,5,6,7) should return 28. +* If clobbered: subprog_7args(1,2,3,4,5,100,200)=315. +*/ +"r1 = 1;" tools/testing/selftests/bpf/progs/verifier_stack_arg.c __naked void stack_arg_sequential_calls(void) · line 514
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +10
-0+"r2 = 2;" +"r3 = 3;" +"r4 = 4;" +"r5 = 5;" +/* +* the 6th and 7th arg should be 6, 7 as set above. +* But the inner call in subprog_7args_inner_call clobbered +* them and reusing again without a re-store will be wrong. +*/ +"call subprog_7args;" /* should return 28, but will be 315 due to clobber */
Quoted 5 lines
Puranjay is correct.
The verifier should error here.
callee can modify 6 and 7th args in place in the stack. All args register or stack are caller saved.
I believe it matches other archs.
Agree. verifier should do invalidation of stack argument slots after copying states from caller to callee. I did have invalidation in v4: https://lore.kernel.org/bpf/[email protected]/ In v5, I tried to do some optimization. But sashiko-bot and bot+bpf-ci pointed the issue. I will bring back invalidation in verifier which should fix the issue.