Skip to contentSkip to conversation

bpf: Support stack arguments for BPF functions and kfuncs

PATCH Applied Yonghong Song 5 participants 55 messages May 13 – May 15

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems

v4 of 6 Applied 25/25 Latest is v6 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v4 cover bpf: Support stack arguments for BPF functions and kfuncs

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS), to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The current maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

All kfunc/bpf-function arguments are caller saved, including stack arguments. For register arguments (r1-r5), the verifier already marks them as clobbered after each call. For stack arguments, the verifier invalidates all outgoing stack arg slots immediately after a call, requiring the compiler to re-store them before any subsequent call. This follows the native calling convention where all function parameters are caller saved.

The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.

Global subprogs and freplace progs with >5 args are not yet supported. Only x86_64 and arm64 are supported for now. Same selftests are tested by both x86_64 and arm64. Please see each individual patch for details.

[1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Added no_stack_arg_load comparison in func_states_equal() to ensure correctness of pruning.
- Shrink bpf_jmp_history_entry.flags to 4bit to match the number of flags.
- Instead of passing bpf_subprog_info to JIT, use prog->aux->func_idx to
find corresponding bpf_subprog_info from 'env'.
- For patch 'bpf: Reject stack arguments if tail call reachable', use stack_arg_cnt instead of just incoming stack arg cnt.
- Tighten invalidate_outgoing_stack_args() for kfunc/helper/bpf-to-bpf calls.
- Disable private stack in verifier for x86_64 instead of in JIT.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- In do_check_common() and for main prog, if btf does not match with actual parameter, the verification will continue and will ignore arg_cnt. Make arg_cnt=1 explictly to prevent any incoming stack arguments.
- Remove the loop which clear current frame stack slot and set the upper level frame stack slot. This is not needed unless there is a bug. Add a verifier_bug
if the bug happens.
- For liveness, avoid r11 based load/stores mixing with r10 based stack tracking. Also, print out stack arguments properly.
- Pass bpf_subprog_info the JIT so we can avoid copy bpf_subprog_info fields to bpf_prog_aux.
- Fix the missed allocation free for test infra BTF fixup.
- Remove selftest result for precision backtracking test since the result would be change (two possible output).
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Several refactoring (convert bpf_get_spilled_reg macro to static inline func, Remove copy_register_state(), Refactor jmp history, Refactor record_call_access(), etc), suggested by Eduard.
- Use incoming_stack_arg_cnt/stack_arg_cnt instead of incoming_stack_arg_depth/stack_arg_depth, suggested by Eduard.
- Fix a stack arg pruning bug, from Eduard.
- Fix a bug for precision marking and backtracking, basically callee needs to get the stack arg value from callers, helped from Eduard.
- Set sub->arg_cnt earlier in btf_prepare_func_args(), this will avoid having incoming_stack_arg_cnt in bpf_subprog_info.
- Do stack-arg liveness analysis together with r10 based liveness analysis, suggested by Eduard.
- Fix a few tests to ensure that r11-based loads cannot be ahead of r11-based stores, and r11-based loads cannot be after kfunc/helper/bpf-function.

Puranjay Mohan (3):
bpf, arm64: Map BPF_REG_0 to x8 instead of x7 bpf, arm64: Add JIT support for stack arguments selftests/bpf: Enable stack argument tests for arm64

Yonghong Song (22):
bpf: Convert bpf_get_spilled_reg macro to static inline function
bpf: Remove copy_register_state wrapper function
bpf: Add helper functions for r11-based stack argument insns
bpf: Set sub->arg_cnt earlier in btf_prepare_func_args()
bpf: Support stack arguments for bpf functions
bpf: Refactor jmp history to use dedicated spi/frame fields
bpf: Add precision marking and backtracking for stack argument slots
bpf: Refactor record_call_access() to extract per-arg logic
bpf: Use arg_is_fp() in has_fp_args()
bpf: Extend liveness analysis to track stack argument slots
bpf: Reject stack arguments in non-JITed programs
bpf: Prepare architecture JIT support for stack arguments
bpf: Enable r11 based insns
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments if tail call reachable
bpf: Disable private stack for x86_64 if stack arguments used bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add tests for stack argument validation selftests/bpf: Add BTF fixup for __naked subprog parameter names selftests/bpf: Add verifier tests for stack argument validation selftests/bpf: Add precision backtracking test for stack arguments

 arch/arm64/net/bpf_jit_comp.c                 |  92 +++-
 arch/arm64/net/bpf_timed_may_goto.S           |   8 +-
 arch/x86/net/bpf_jit_comp.c                   | 149 +++++-
 include/linux/bpf.h                           |   1 +
 include/linux/bpf_verifier.h                  |  98 ++--
 include/linux/filter.h                        |  22 +
 kernel/bpf/backtrack.c                        |  82 +++-
 kernel/bpf/btf.c                              |  20 +-
 kernel/bpf/const_fold.c                       |   8 +
 kernel/bpf/core.c                             |  17 +-
 kernel/bpf/fixups.c                           |  22 +-
 kernel/bpf/liveness.c                         | 179 +++++--
 kernel/bpf/states.c                           |  34 +-
 kernel/bpf/verifier.c                         | 396 +++++++++++++---
 .../selftests/bpf/prog_tests/stack_arg.c      | 139 ++++++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  10 +
 .../bpf/prog_tests/stack_arg_precision.c      |  10 +
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   1 +
 .../bpf/progs/btf__stack_arg_precision.c      |  24 +
 .../bpf/progs/btf__verifier_stack_arg_order.c |  41 ++
 tools/testing/selftests/bpf/progs/stack_arg.c | 253 ++++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      | 114 +++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 +++++++
 .../selftests/bpf/progs/stack_arg_precision.c | 135 ++++++
 .../selftests/bpf/progs/verifier_jit_inline.c |   2 +-
 .../selftests/bpf/progs/verifier_ldsx.c       |   6 +-
 .../bpf/progs/verifier_private_stack.c        |  10 +-
 .../selftests/bpf/progs/verifier_stack_arg.c  | 445 ++++++++++++++++++
 .../bpf/progs/verifier_stack_arg_order.c      | 127 +++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  72 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  26 +
 tools/testing/selftests/bpf/test_loader.c     | 136 +++++-
 33 files changed, 2664 insertions(+), 183 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v4 1/25 bpf: Convert bpf_get_spilled_reg macro to static inline function

Convert the bpf_get_spilled_reg() macro to a static inline function for better type safety and readability. This also simplifies the macro definition in preparation for upcoming stack argument support which will introduce additional macros.

No functional change.

---
 include/linux/bpf_verifier.h | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
include/linux/bpf_verifier.h +8 -4
index 976e2b2f40e8..321b9d69cf9c 100644
@@ -552,10 +552,14 @@ struct bpf_verifier_state {
 	u32 may_goto_depth;
 };
 
-#define bpf_get_spilled_reg(slot, frame, mask)				\
-	(((slot < frame->allocated_stack / BPF_REG_SIZE) &&		\
-	  ((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
-	 ? &frame->stack[slot].spilled_ptr : NULL)
+static inline struct bpf_reg_state *
+bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
+{
+	if (slot < frame->allocated_stack / BPF_REG_SIZE &&
+	    (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
+		return &frame->stack[slot].spilled_ptr;
+	return NULL;
+}
 
 /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
 #define bpf_for_each_spilled_reg(iter, frame, reg, mask)			\
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH v4 2/25 bpf: Remove copy_register_state wrapper function

In reply to Yonghong Song

Remove the copy_register_state() helper which was just a plain struct assignment wrapper and replace all call sites with direct struct assignment. This simplifies the code in preparation for upcoming stack argument support.

No functional change.

---
 kernel/bpf/verifier.c | 44 +++++++++++++++++++------------------------
 1 file changed, 19 insertions(+), 25 deletions(-)
kernel/bpf/verifier.c +19 -25
index 11054ad89c14..3bafb7ad2ba7 100644
@@ -3410,12 +3410,6 @@ static void assign_scalar_id_before_mov(struct bpf_verifier_env *env,
 		src_reg->id = ++env->id_gen;
 }
 
-/* Copy src state preserving dst->parent and dst->live fields */
-static void copy_register_state(struct bpf_reg_state *dst, const struct bpf_reg_state *src)
-{
-	*dst = *src;
-}
-
 static void save_register_state(struct bpf_verifier_env *env,
 				struct bpf_func_state *state,
 				int spi, struct bpf_reg_state *reg,
@@ -3423,7 +3417,7 @@ static void save_register_state(struct bpf_verifier_env *env,
 {
 	int i;
 
-	copy_register_state(&state->stack[spi].spilled_ptr, reg);
+	state->stack[spi].spilled_ptr = *reg;
 
 	for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)
 		state->stack[spi].slot_type[i - 1] = STACK_SPILL;
@@ -3822,7 +3816,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 					 * with the destination register on fill.
 					 */
 					assign_scalar_id_before_mov(env, reg);
-				copy_register_state(&state->regs[dst_regno], reg);
+				state->regs[dst_regno] = *reg;
 				state->regs[dst_regno].subreg_def = subreg_def;
 
 				/* Break the relation on a narrowing fill.
@@ -3877,7 +3871,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 				 * with the destination register on fill.
 				 */
 				assign_scalar_id_before_mov(env, reg);
-			copy_register_state(&state->regs[dst_regno], reg);
+			state->regs[dst_regno] = *reg;
 			/* mark reg as written since spilled pointer state likely
 			 * has its liveness marks cleared by is_state_visited()
 			 * which resets stack/reg liveness for state transitions
@@ -6031,7 +6025,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 						     size);
 					return -EACCES;
 				}
-				copy_register_state(&regs[value_regno], reg);
+				regs[value_regno] = *reg;
 				add_scalar_to_reg(&regs[value_regno], off);
 				regs[value_regno].type = PTR_TO_INSN;
 			} else {
@@ -13248,7 +13242,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
 	 */
 	if (!ptr_is_dst_reg) {
 		tmp = *dst_reg;
-		copy_register_state(dst_reg, ptr_reg);
+		*dst_reg = *ptr_reg;
 	}
 	err = sanitize_speculative_path(env, NULL, env->insn_idx + 1, env->insn_idx);
 	if (err < 0)
@@ -14698,7 +14692,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 					 * copy register state to dest reg
 					 */
 					assign_scalar_id_before_mov(env, src_reg);
-					copy_register_state(dst_reg, src_reg);
+					*dst_reg = *src_reg;
 					dst_reg->subreg_def = DEF_NOT_SUBREG;
 				} else {
 					/* case: R1 = (s8, s16 s32)R2 */
@@ -14713,7 +14707,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 						no_sext = reg_umax(src_reg) < (1ULL << (insn->off - 1));
 						if (no_sext)
 							assign_scalar_id_before_mov(env, src_reg);
-						copy_register_state(dst_reg, src_reg);
+						*dst_reg = *src_reg;
 						if (!no_sext)
 							clear_scalar_id(dst_reg);
 						coerce_reg_to_size_sx(dst_reg, insn->off >> 3);
@@ -14735,7 +14729,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 
 						if (is_src_reg_u32)
 							assign_scalar_id_before_mov(env, src_reg);
-						copy_register_state(dst_reg, src_reg);
+						*dst_reg = *src_reg;
 						/* Make sure ID is cleared if src_reg is not in u32
 						 * range otherwise dst_reg min/max could be incorrectly
 						 * propagated into src_reg by sync_linked_regs()
@@ -14749,7 +14743,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 
 						if (no_sext)
 							assign_scalar_id_before_mov(env, src_reg);
-						copy_register_state(dst_reg, src_reg);
+						*dst_reg = *src_reg;
 						if (!no_sext)
 							clear_scalar_id(dst_reg);
 						dst_reg->subreg_def = env->insn_idx + 1;
@@ -15629,7 +15623,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
 		    reg->delta == known_reg->delta) {
 			s32 saved_subreg_def = reg->subreg_def;
 
-			copy_register_state(reg, known_reg);
+			*reg = *known_reg;
 			reg->subreg_def = saved_subreg_def;
 		} else {
 			s32 saved_subreg_def = reg->subreg_def;
@@ -15640,7 +15634,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
 			__mark_reg_known(&fake_reg, (s64)reg->delta - (s64)known_reg->delta);
 
 			/* reg = known_reg; reg += delta */
-			copy_register_state(reg, known_reg);
+			*reg = *known_reg;
 			/*
 			 * Must preserve off, id and subreg_def flag,
 			 * otherwise another sync_linked_regs() will be incorrect.
@@ -15743,10 +15737,10 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	}
 
 	is_jmp32 = BPF_CLASS(insn->code) == BPF_JMP32;
-	copy_register_state(&env->false_reg1, dst_reg);
-	copy_register_state(&env->false_reg2, src_reg);
-	copy_register_state(&env->true_reg1, dst_reg);
-	copy_register_state(&env->true_reg2, src_reg);
+	env->false_reg1 = *dst_reg;
+	env->false_reg2 = *src_reg;
+	env->true_reg1 = *dst_reg;
+	env->true_reg2 = *src_reg;
 	pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
 	if (pred >= 0) {
 		/* If we get here with a dst_reg pointer type it is because
@@ -15815,11 +15809,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	if (err)
 		return err;
 
-	copy_register_state(dst_reg, &env->false_reg1);
-	copy_register_state(src_reg, &env->false_reg2);
-	copy_register_state(&other_branch_regs[insn->dst_reg], &env->true_reg1);
+	*dst_reg = env->false_reg1;
+	*src_reg = env->false_reg2;
+	other_branch_regs[insn->dst_reg] = env->true_reg1;
 	if (BPF_SRC(insn->code) == BPF_X)
-		copy_register_state(&other_branch_regs[insn->src_reg], &env->true_reg2);
+		other_branch_regs[insn->src_reg] = env->true_reg2;
 
 	if (BPF_SRC(insn->code) == BPF_X &&
 	    src_reg->type == SCALAR_VALUE && src_reg->id &&
Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH v4 3/25 bpf: Add helper functions for r11-based stack argument insns

In reply to Yonghong Song

Add three static inline helper functions — is_stack_arg_ldx(), is_stack_arg_st(), and is_stack_arg_stx() — that identify r11-based (BPF_REG_PARAMS) instructions used for stack argument passing. These helpers encapsulate the detailed encoding requirements (operand size, register, offset alignment and sign) and hide raw BPF_REG_PARAMS usage from the verifier, making call sites more readable and explicit.

A later patch ("bpf: Enable r11 based insns") will wire these helpers into the verifier. Until then, check_and_resolve_insns() rejects any r11-based registers.

---
 include/linux/filter.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
include/linux/filter.h +21 -0
index b77d0b06db6e..918d9b34eac6 100644
@@ -749,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog,
 	return ret;
 }
 
+static inline bool is_stack_arg_ldx(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) &&
+	       insn->src_reg == BPF_REG_PARAMS &&
+	       insn->off > 0 && insn->off % 8 == 0;
+}
+
+static inline bool is_stack_arg_st(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_ST | BPF_MEM | BPF_DW) &&
+	       insn->dst_reg == BPF_REG_PARAMS &&
+	       insn->off < 0 && insn->off % 8 == 0;
+}
+
+static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
+{
+	return insn->code == (BPF_STX | BPF_MEM | BPF_DW) &&
+	       insn->dst_reg == BPF_REG_PARAMS &&
+	       insn->off < 0 && insn->off % 8 == 0;
+}
+
 #define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
 
 struct bpf_skb_data_end {
Signature · 1 line
2.53.0-Meta
author reviewer
#4 lore raw

PATCH v4 4/25 bpf: Set sub->arg_cnt earlier in btf_prepare_func_args()

In reply to Yonghong Song

Move the "sub->arg_cnt = nargs" assignment to immediately after nargs is computed from btf_type_vlen(), instead of at the end of btf_prepare_func_args().

btf_prepare_func_args() can return -EINVAL early in several cases, e.g. when a static function has some non-int/enum arguments. Since -EINVAL from btf_prepare_func_args() does not immediately reject verification, arg_cnt remains zero after the early return. This causes later stack argument based load/store insns to incorrectly assume the function has no arguments.

Setting arg_cnt right after nargs ensures it is available regardless of which path btf_prepare_func_args() takes.

---
 kernel/bpf/btf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/btf.c +1 -1
index 77af44d8a3ad..a33a5b4122f8 100644
@@ -7880,6 +7880,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	}
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
+	sub->arg_cnt = nargs;
 	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
 		if (!is_global)
 			return -EINVAL;
@@ -8067,7 +8068,6 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	sub->arg_cnt = nargs;
 	sub->args_cached = true;
 
 	return 0;
Signature · 1 line
2.53.0-Meta
author reviewer
#5 lore raw

PATCH v4 5/25 bpf: Support stack arguments for bpf functions

In reply to Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_PARAMS (r11), introduced in an earlier patch ([2]).

The compiler uses positive r11 offsets for incoming (callee-side) args and negative r11 offsets for outgoing (caller-side) args, following the x86_64/arm64 calling convention direction. There is an 8-byte gap at offset 0 separating two regions:
Incoming (callee reads): r11+8 (arg6), r11+16 (arg7), ...
Outgoing (caller writes): r11-8 (arg6), r11-16 (arg7), ...

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }
  Caller (foo)                           Callee (bar)
  ============                           ============
  Incoming (positive offsets):           Incoming (positive offsets):

r11+8: [incoming arg 6] r11+8: [incoming arg 6] <-+

  r11+16: [incoming arg 7]               r11+16: [incoming arg 7] <-|+
                                         r11+24: [incoming arg 8] <-||+
  Outgoing (negative offsets):                                      |||

r11-8: [outgoing arg 6 to bar] -------->-------------------------+||
r11-16: [outgoing arg 7 to bar] -------->--------------------------+|
r11-24: [outgoing arg 8 to bar] -------->---------------------------+

If the bpf function has more than one call:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }
  Caller (foo)                             Callee (bar2)
  ============                             ==============
  Incoming (positive offsets):             Incoming (positive offsets):

r11+8: [incoming arg 6] r11+8: [incoming arg 6] <+

  r11+16: [incoming arg 7]                 r11+16: [incoming arg 7] <|+
                                           r11+24: [incoming arg 8] <||+
  Outgoing for bar2 (negative offsets):    r11+32: [incoming arg 9] <|||+
  r11-8:  [outgoing arg 6] ---->----------->-------------------------+|||
  r11-16: [outgoing arg 7] ---->----------->--------------------------+||
  r11-24: [outgoing arg 8] ---->----------->---------------------------+|

r11-32: [outgoing arg 9] ---->----------->----------------------------+

The verifier tracks outgoing stack arguments in stack_arg_regs[] and out_stack_arg_cnt in bpf_func_state, separately from the regular r10 stack. The callee does not copy incoming args — it reads them directly from the caller's outgoing slots at positive r11 offsets. Similar to stacksafe(), introduce stack_arg_safe() to do pruning check.

Outgoing stack arg slots are invalidated when the callee returns (e.g. in prepare_func_exit), not at call time. This allows the callee to read incoming args from the caller's outgoing slots during verification. The following are a few examples.

Example 1:
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
call bar2; // expected with 2 stack arguments, failed

Example 2:
To fix the Example 1:
*(u64 *)(r11 - 8) = r6;
*(u64 *)(r11 - 16) = r7;
call bar1; // arg6 = r6, arg7 = r7
*(u64 *)(r11 - 8) = r8;
*(u64 *)(r11 - 16) = r9;
call bar2; // arg6 = r8, arg7 = r9

Example 3:
The compiler can hoist the shared stack arg stores above the branch:
*(u64 *)(r11 - 16) = r7;

  if cond goto else;
    *(u64 *)(r11 - 8) = r8;
    call bar1;               // arg6 = r8, arg7 = r7
    goto end;

else:
*(u64 *)(r11 - 8) = r9;
call bar2; // arg6 = r9, arg7 = r7
end:

Example 4:
Within a loop:
loop:

    *(u64 *)(r11 - 8) = r6;  // arg6, before loop
    call bar;                // reuses arg6 each iteration
    if ... goto loop;

A separate max_out_stack_arg_cnt field in bpf_subprog_info tracks the deepest outgoing slot actually written. This intends to reject programs that write to slots beyond what any callee expects. It is necessary for JIT.

Similar to typical compiler generated code, enforce the following orderings:
- all stack arg reads must be ahead of any stack arg write
- all stack arg reads must be before any bpf func, kfunc and helpers
This is needed as JIT may emit 'mov' insns for read/write with the same register and bpf function, kfunc and helper will invalidate all arguments immediately after the call.

Callback functions with stack arguments need kernel setup parameter types (including stack parameters) properly and then callback function can retrieve such information for verification purpose.

Global subprogs and freplace with >5 args are not yet supported.

[1] https://github.com/llvm/llvm-project/pull/189060
[2] https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf_verifier.h |  43 +++++++-
 kernel/bpf/btf.c             |  14 ++-
 kernel/bpf/fixups.c          |  16 ++-
 kernel/bpf/states.c          |  32 ++++++
 kernel/bpf/verifier.c        | 203 ++++++++++++++++++++++++++++++++++-
 5 files changed, 294 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +39 -4
index 321b9d69cf9c..13bb07699cb1 100644
@@ -402,6 +402,7 @@ struct bpf_func_state {
 	bool in_callback_fn;
 	bool in_async_callback_fn;
 	bool in_exception_callback_fn;
+	bool no_stack_arg_load;
 	/* For callback calling functions that limit number of possible
 	 * callback executions (e.g. bpf_loop) keeps track of current
 	 * simulated iteration number.
@@ -427,6 +428,9 @@ struct bpf_func_state {
 	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
 	 */
 	int allocated_stack;
+
+	u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */
+	struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */
 };
 
 #define MAX_CALL_FRAMES 8
@@ -465,8 +469,10 @@ struct bpf_jmp_history_entry {
 	u64 linked_regs;
 };
 
-/* Maximum number of register states that can exist at once */
-#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES)
+/* Maximum number of bpf_reg_state objects that can exist at once */
+#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
+			  MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
 struct bpf_verifier_state {
 	/* call stack tracking */
 	struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -561,12 +567,27 @@ bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
 	return NULL;
 }
 
+static inline struct bpf_reg_state *
+bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
+{
+	if (slot < frame->out_stack_arg_cnt &&
+	    frame->stack_arg_regs[slot].type != NOT_INIT)
+		return &frame->stack_arg_regs[slot];
+	return NULL;
+}
+
 /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
 #define bpf_for_each_spilled_reg(iter, frame, reg, mask)			\
 	for (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask);		\
 	     iter < frame->allocated_stack / BPF_REG_SIZE;		\
 	     iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
 
+/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
+#define bpf_for_each_spilled_stack_arg(iter, frame, reg)               \
+	for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame);   \
+	     iter < frame->out_stack_arg_cnt;                          \
+	     iter++, reg = bpf_get_spilled_stack_arg(iter, frame))
+
 #define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr)   \
 	({                                                               \
 		struct bpf_verifier_state *___vstate = __vst;            \
@@ -584,6 +605,11 @@ bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
 					continue;                        \
 				(void)(__expr);                          \
 			}                                                \
+			bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \
+				if (!__reg)                              \
+					continue;                        \
+				(void)(__expr);                          \
+			}						 \
 		}                                                        \
 	})
 
@@ -799,12 +825,21 @@ struct bpf_subprog_info {
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
 	bool might_sleep: 1;
-	u8 arg_cnt:3;
+	u8 arg_cnt:4;
 
 	enum priv_stack_mode priv_stack_mode;
-	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+	u16 stack_arg_cnt; /* incoming + max outgoing */
+	u16 max_out_stack_arg_cnt;
 };
 
+static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
+{
+	if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS)
+		return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS;
+	return 0;
+}
+
 struct bpf_verifier_env;
 
 struct backtrack_state {
kernel/bpf/btf.c +10 -4
index a33a5b4122f8..ec3fb8c8f4ee 100644
@@ -7881,10 +7881,16 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
 	sub->arg_cnt = nargs;
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-		if (!is_global)
-			return -EINVAL;
-		bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+	if (nargs > MAX_BPF_FUNC_ARGS) {
+		bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
+			MAX_BPF_FUNC_ARGS, tname, nargs);
+		return -EFAULT;
+	}
+	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+
+	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
kernel/bpf/fixups.c +14 -2
index fba9e8c00878..ba86039789fd 100644
@@ -1378,9 +1378,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 	struct bpf_prog *prog = env->prog;
 	struct bpf_insn *insn = prog->insnsi;
 	bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);
-	int i, depth;
+	int depth;
 #endif
-	int err = 0;
+	int i, err = 0;
+
+	for (i = 0; i < env->subprog_cnt; i++) {
+		struct bpf_subprog_info *subprog = &env->subprog_info[i];
+		u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog);
+
+		if (subprog->max_out_stack_arg_cnt > outgoing) {
+			verbose(env,
+				"func#%d writes %u stack arg slots, but calls only require %u\n",
+				i, subprog->max_out_stack_arg_cnt, outgoing);
+			return -EINVAL;
+		}
+	}
 
 	if (env->prog->jit_requested &&
 	    !bpf_prog_is_offloaded(env->prog->aux)) {
kernel/bpf/states.c +32 -0
index bd9c22945050..3ce6d2652b27 100644
@@ -833,6 +833,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 	return true;
 }
 
+/*
+ * Compare stack arg slots between old and current states.
+ * Outgoing stack args are path-local state and must agree for pruning.
+ */
+static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
+			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
+			   enum exact_level exact)
+{
+	int i, nslots;
+
+	nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
+	for (i = 0; i < nslots; i++) {
+		struct bpf_reg_state *old_arg, *cur_arg;
+		struct bpf_reg_state not_init = { .type = NOT_INIT };
+
+		old_arg = i < old->out_stack_arg_cnt ?
+			  &old->stack_arg_regs[i] : &not_init;
+		cur_arg = i < cur->out_stack_arg_cnt ?
+			  &cur->stack_arg_regs[i] : &not_init;
+		if (!regsafe(env, old_arg, cur_arg, idmap, exact))
+			return false;
+	}
+
+	return true;
+}
+
 static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
 		    struct bpf_idmap *idmap)
 {
@@ -915,6 +941,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (old->callback_depth > cur->callback_depth)
 		return false;
 
+	if (!old->no_stack_arg_load && cur->no_stack_arg_load)
+		return false;
+
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
 		    !regsafe(env, &old->regs[i], &cur->regs[i],
@@ -924,6 +953,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
 		return false;
 
+	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+		return false;
+
 	return true;
 }
 
kernel/bpf/verifier.c +199 -4

337 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#6 lore raw

Re: [PATCH bpf-next v4 05/25] bpf: Support stack arguments for bpf functions

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Bypass of caller stack argument initialization checks allows NULL pointer dereferences and…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Bypass of caller stack argument initialization checks allows NULL pointer dereferences and out-of-bounds heap reads, leading to verifier crashes or privilege escalation. --

commit 64c850bac22b15ca90af1ba361a4d764104b6ffb
Author: Yonghong Song <[email protected]>

bpf: Support stack arguments for bpf functions

This commit allows the compiler to pass additional arguments via a dedicated stack area through the BPF_REG_PARAMS (r11) register. It tracks outgoing stack arguments and introduces bounds checks for stack arguments in BPF subprograms.

Quoted 24 lines
kernel/bpf/verifier.c +16 -0
index 3bafb7ad2ba71..69965d2c5fdd5 100644
@@ -4050,6 +4080,103 @@ static int check_stack_write(struct bpf_verifier_env *env,
 	return err;
 }
 
+/*
+ * Read a value from the incoming stack arg area.
+ * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7).
+ */
+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				int off, int dst_regno)
+{
+	struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
+	struct bpf_verifier_state *vstate = env->cur_state;
+	int spi = off / BPF_REG_SIZE - 1;
+	struct bpf_func_state *caller, *cur;
+	struct bpf_reg_state *arg;
+
+	if (state->no_stack_arg_load) {
+		verbose(env, "r11 load must be before any r11 store or call insn\n");
+		return -EINVAL;

kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 4108

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+	}
+
+	if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) {
+		verbose(env, "invalid read from stack arg off %d depth %d\n",
+			off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE);
+		return -EACCES;
+	}
+
+	caller = vstate->frame[vstate->curframe - 1];
+	arg = &caller->stack_arg_regs[spi];

Could this lead to an out-of-bounds read or a NULL pointer dereference?

If check_outgoing_stack_args() is skipped (e.g. due to an error in btf_prepare_func_args()), caller->stack_arg_regs might be NULL or smaller than spi. Since check_func_call() or push_callback_call() can ignore btf_check_func_arg_match() failures for static subprograms or callbacks, the callee can still execute.

If sub->arg_cnt was updated before the failure, bpf_in_stack_arg_cnt(subprog) would be greater than 0, allowing this check to pass and potentially accessing unallocated memory.

[ ... ]

Quoted 9 lines
kernel/bpf/verifier.c +1 -0
@@ -8922,6 +9078,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 				    struct bpf_reg_state *regs)
 {
 	struct bpf_subprog_info *sub = subprog_info(env, subprog);
+	struct bpf_func_state *caller = cur_func(env);
 	struct bpf_verifier_log *log = &env->log;
 	u32 i;
 	int ret;

kernel/bpf/verifier.c static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · line 9093

Quoted 7 lines
kernel/bpf/verifier.c +4 -0
@@ -8930,13 +9087,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	if (ret)
 		return ret;
 
+	ret = check_outgoing_stack_args(env, caller, sub->arg_cnt);
+	if (ret)
+		return ret;
+

Is it possible for btf_prepare_func_args() to return an error after modifying sub->arg_cnt?

For instance, if a BTF tag is unsupported in a static subprogram, btf_prepare_func_args() will return an error after already assigning sub->arg_cnt. This skips check_outgoing_stack_args(), which is responsible for verifying that the caller actually allocated and initialized caller->stack_arg_regs. This could lead to the unverified memory access mentioned above.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#7 lore raw

Re: [PATCH bpf-next v4 05/25] bpf: Support stack arguments for bpf functions

Quoted 55 lines · On 5/14/26 12:46 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Bypass of caller stack argument initialization checks allows NULL pointer dereferences and out-of-bounds heap reads, leading to verifier crashes or privilege escalation. --

commit 64c850bac22b15ca90af1ba361a4d764104b6ffb
Author: Yonghong Song <[email protected]>

bpf: Support stack arguments for bpf functions

This commit allows the compiler to pass additional arguments via a dedicated stack area through the BPF_REG_PARAMS (r11) register. It tracks outgoing stack arguments and introduces bounds checks for stack arguments in BPF subprograms.

Quoted 24 lines
kernel/bpf/verifier.c +16 -0
index 3bafb7ad2ba71..69965d2c5fdd5 100644
@@ -4050,6 +4080,103 @@ static int check_stack_write(struct bpf_verifier_env *env,
  	return err;
  }
 
+/*
+ * Read a value from the incoming stack arg area.
+ * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7).
+ */
+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				int off, int dst_regno)
+{
+	struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
+	struct bpf_verifier_state *vstate = env->cur_state;
+	int spi = off / BPF_REG_SIZE - 1;
+	struct bpf_func_state *caller, *cur;
+	struct bpf_reg_state *arg;
+
+	if (state->no_stack_arg_load) {
+		verbose(env, "r11 load must be before any r11 store or call insn\n");
+		return -EINVAL;

kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 4108

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+	}
+
+	if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) {
+		verbose(env, "invalid read from stack arg off %d depth %d\n",
+			off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE);
+		return -EACCES;
+	}
+
+	caller = vstate->frame[vstate->curframe - 1];
+	arg = &caller->stack_arg_regs[spi];

Could this lead to an out-of-bounds read or a NULL pointer dereference?

If check_outgoing_stack_args() is skipped (e.g. due to an error in btf_prepare_func_args()), caller->stack_arg_regs might be NULL or smaller than spi. Since check_func_call() or push_callback_call() can ignore btf_check_func_arg_match() failures for static subprograms or callbacks, the callee can still execute.

Quoted 3 lines

If sub->arg_cnt was updated before the failure, bpf_in_stack_arg_cnt(subprog) would be greater than 0, allowing this check to pass and potentially accessing unallocated memory.

Yes, it is possible for an out-of-bound read or a NULL pointer dereference. This can be reproduced with inline asm where the caller does not have any outgoing stack arguments but callee has incoming stack arguments.

Will fix.

Quoted 29 lines

[ ... ]

Quoted 9 lines
kernel/bpf/verifier.c +1 -0
@@ -8922,6 +9078,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
  				    struct bpf_reg_state *regs)
  {
  	struct bpf_subprog_info *sub = subprog_info(env, subprog);
+	struct bpf_func_state *caller = cur_func(env);
  	struct bpf_verifier_log *log = &env->log;
  	u32 i;
  	int ret;

kernel/bpf/verifier.c static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · line 9093

Quoted 7 lines
kernel/bpf/verifier.c +4 -0
@@ -8930,13 +9087,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
  	if (ret)
  		return ret;
 
+	ret = check_outgoing_stack_args(env, caller, sub->arg_cnt);
+	if (ret)
+		return ret;
+

Is it possible for btf_prepare_func_args() to return an error after modifying sub->arg_cnt?

For instance, if a BTF tag is unsupported in a static subprogram, btf_prepare_func_args() will return an error after already assigning sub->arg_cnt. This skips check_outgoing_stack_args(), which is responsible for verifying that the caller actually allocated and initialized caller->stack_arg_regs. This could lead to the unverified memory access mentioned above.

author reviewer
#8 lore raw

PATCH v4 6/25 bpf: Refactor jmp history to use dedicated spi/frame fields

In reply to Yonghong Song

Move stack slot index (spi) and frame number out of the flags field in bpf_jmp_history_entry into dedicated bitfields. This simplifies the encoding and makes room for new flags.

Previously, spi and frame were packed into the lower 9 bits of the 12-bit flags field (3 bits frame + 6 bits spi), with INSN_F_STACK_ACCESS at BIT(9) and INSN_F_DST/SRC_REG_STACK at BIT(10)/BIT(11). But this has no room for an INSN_F_* flag for stack arguments.

To resolve this issue, bpf_jmp_history_entry field idx is narrowed to 20 bits (sufficient for insn indices up to 1M), and the freed bits hold spi (6 bits) and frame (3 bits) as dedicated struct fields. The flags enum is simplified accordingly:
INSN_F_STACK_ACCESS -> BIT(0)
INSN_F_DST_REG_STACK -> BIT(1)
INSN_F_SRC_REG_STACK -> BIT(2)
which allows more room for additional INSN_F_* flags.

bpf_push_jmp_history() now takes explicit spi and frame parameters instead of encoding them into flags. The insn_stack_access_flags(), insn_stack_access_spi(), and insn_stack_access_frameno() helpers are removed.

No functional change.

---
 include/linux/bpf_verifier.h | 37 ++++++++++++++++--------------------
 kernel/bpf/backtrack.c       | 24 +++++++++--------------
 kernel/bpf/states.c          |  2 +-
 kernel/bpf/verifier.c        | 23 +++++++++++-----------
 4 files changed, 37 insertions(+), 49 deletions(-)
include/linux/bpf_verifier.h +16 -21
index 13bb07699cb1..a8685886f915 100644
@@ -435,40 +435,35 @@ struct bpf_func_state {
 
 #define MAX_CALL_FRAMES 8
 
-/* instruction history flags, used in bpf_jmp_history_entry.flags field */
+/* instruction history flags, used in bpf_jmp_history_entry.flags field.
+ * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
+ */
 enum {
-	/* instruction references stack slot through PTR_TO_STACK register;
-	 * we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8)
-	 * and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512,
-	 * 8 bytes per slot, so slot index (spi) is [0, 63])
-	 */
-	INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */
-
-	INSN_F_SPI_MASK = 0x3f, /* 6 bits */
-	INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */
+	INSN_F_STACK_ACCESS = BIT(0),
 
-	INSN_F_STACK_ACCESS = BIT(9),
-
-	INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */
-	INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */
-	/* total 12 bits are used now. */
+	INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
+	INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
 };
 
-static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES);
-static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8);
-
 struct bpf_jmp_history_entry {
-	u32 idx;
 	/* insn idx can't be bigger than 1 million */
+	u32 idx : 20;
+	u32 frame : 3;	/* stack access frame number */
+	u32 spi : 6;	/* stack slot index (0..63) */
+	u32 : 3;
 	u32 prev_idx : 20;
 	/* special INSN_F_xxx flags */
-	u32 flags : 12;
+	u32 flags : 4;
+	u32 : 8;
 	/* additional registers that need precision tracking when this
 	 * jump is backtracked, vector of six 10-bit records
 	 */
 	u64 linked_regs;
 };
 
+static_assert(MAX_CALL_FRAMES <= (1 << 3));
+static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
+
 /* Maximum number of bpf_reg_state objects that can exist at once */
 #define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
 #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
@@ -1182,7 +1177,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
 void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
 void bpf_free_backedges(struct bpf_scc_visit *visit);
 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, u64 linked_regs);
+			 int insn_flags, int spi, int frame, u64 linked_regs);
 void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg);
kernel/bpf/backtrack.c +9 -15
index 854731dc93fe..5e93e57fb7ae 100644
@@ -9,7 +9,7 @@
 
 /* for any branch, call, exit record the history of jmps in the given state */
 int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
-			 int insn_flags, u64 linked_regs)
+			 int insn_flags, int spi, int frame, u64 linked_regs)
 {
 	u32 cnt = cur->jmp_history_cnt;
 	struct bpf_jmp_history_entry *p;
@@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
 				env, "insn history: insn_idx %d cur flags %x new flags %x",
 				env->insn_idx, env->cur_hist_ent->flags, insn_flags);
 		env->cur_hist_ent->flags |= insn_flags;
+		env->cur_hist_ent->spi = spi;
+		env->cur_hist_ent->frame = frame;
 		verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
 				"insn history: insn_idx %d linked_regs: %#llx",
 				env->insn_idx, env->cur_hist_ent->linked_regs);
@@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
 	p->idx = env->insn_idx;
 	p->prev_idx = env->prev_insn_idx;
 	p->flags = insn_flags;
+	p->spi = spi;
+	p->frame = frame;
 	p->linked_regs = linked_regs;
 	cur->jmp_history_cnt = cnt;
 	env->cur_hist_ent = p;
@@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
 	       (insn->imm & BPF_FETCH);
 }
 
-static int insn_stack_access_spi(int insn_flags)
-{
-	return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK;
-}
-
-static int insn_stack_access_frameno(int insn_flags)
-{
-	return insn_flags & INSN_F_FRAMENO_MASK;
-}
-
 /* Backtrack one insn at a time. If idx is not at the top of recorded
  * history then previous instruction came from straight line execution.
  * Return -ENOENT if we exhausted all instructions within given state.
@@ -353,8 +347,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 		 * that [fp - off] slot contains scalar that needs to be
 		 * tracked with precision
 		 */
-		spi = insn_stack_access_spi(hist->flags);
-		fr = insn_stack_access_frameno(hist->flags);
+		spi = hist->spi;
+		fr = hist->frame;
 		bpf_bt_set_frame_slot(bt, fr, spi);
 	} else if (class == BPF_STX || class == BPF_ST) {
 		if (bt_is_reg_set(bt, dreg))
@@ -366,8 +360,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 		/* scalars can only be spilled into stack */
 		if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
 			return 0;
-		spi = insn_stack_access_spi(hist->flags);
-		fr = insn_stack_access_frameno(hist->flags);
+		spi = hist->spi;
+		fr = hist->frame;
 		if (!bt_is_frame_slot_set(bt, fr, spi))
 			return 0;
 		bt_clear_frame_slot(bt, fr, spi);
kernel/bpf/states.c +1 -1
index 3ce6d2652b27..877338136009 100644
@@ -1403,7 +1403,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 			 */
 			err = 0;
 			if (bpf_is_jmp_point(env, env->insn_idx))
-				err = bpf_push_jmp_history(env, cur, 0, 0);
+				err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
 			err = err ? : propagate_precision(env, &sl->state, cur, NULL);
 			if (err)
 				return err;
kernel/bpf/verifier.c +11 -12
index 69965d2c5fdd..14cbb38aa0e0 100644
@@ -3198,11 +3198,6 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno,
 	return __check_reg_arg(env, state->regs, regno, t);
 }
 
-static int insn_stack_access_flags(int frameno, int spi)
-{
-	return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno;
-}
-
 static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
 {
 	env->insn_aux_data[idx].indirect_target = true;
@@ -3517,7 +3512,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	struct bpf_reg_state *reg = NULL;
-	int insn_flags = insn_stack_access_flags(state->frameno, spi);
+	int insn_flags = INSN_F_STACK_ACCESS;
+	int hist_spi = spi, hist_frame = state->frameno;
 
 	/* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
 	 * so it's aligned access and [off, off + size) are within stack limits
@@ -3613,7 +3609,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 	}
 
 	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);
+		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
+					    hist_spi, hist_frame, 0);
 	return 0;
 }
 
@@ -3809,7 +3806,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
 	struct bpf_reg_state *reg;
 	u8 *stype, type;
-	int insn_flags = insn_stack_access_flags(reg_state->frameno, spi);
+	int insn_flags = INSN_F_STACK_ACCESS;
+	int hist_spi = spi, hist_frame = reg_state->frameno;
 
 	stype = reg_state->stack[spi].slot_type;
 	reg = &reg_state->stack[spi].spilled_ptr;
@@ -3940,7 +3938,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 		insn_flags = 0; /* we are not restoring spilled register */
 	}
 	if (insn_flags)
-		return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0);
+		return bpf_push_jmp_history(env, env->cur_state, insn_flags,
+					    hist_spi, hist_frame, 0);
 	return 0;
 }
 
@@ -15907,7 +15906,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	}
 
 	if (insn_flags) {
-		err = bpf_push_jmp_history(env, this_branch, insn_flags, 0);
+		err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, 0);
 		if (err)
 			return err;
 	}
@@ -15971,7 +15970,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 	if (dst_reg->type == SCALAR_VALUE && dst_reg->id)
 		collect_linked_regs(env, this_branch, dst_reg->id, &linked_regs);
 	if (linked_regs.cnt > 1) {
-		err = bpf_push_jmp_history(env, this_branch, 0, linked_regs_pack(&linked_regs));
+		err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, linked_regs_pack(&linked_regs));
 		if (err)
 			return err;
 	}
@@ -17278,7 +17277,7 @@ static int do_check(struct bpf_verifier_env *env)
 		}
 
 		if (bpf_is_jmp_point(env, env->insn_idx)) {
-			err = bpf_push_jmp_history(env, state, 0, 0);
+			err = bpf_push_jmp_history(env, state, 0, 0, 0, 0);
 			if (err)
 				return err;
 		}
Signature · 1 line
2.53.0-Meta
author reviewer
#9 lore raw

PATCH v4 7/25 bpf: Add precision marking and backtracking for stack argument slots

In reply to Yonghong Song

Extend the precision marking and backtracking infrastructure to support stack argument slots (r11-based accesses). Without this, precision demands for scalar values passed through stack arguments are silently dropped, which could allow the verifier to incorrectly prune states with different constant values in stack arg slots.

---
 include/linux/bpf_verifier.h |  8 +++++
 kernel/bpf/backtrack.c       | 58 +++++++++++++++++++++++++++++++++++-
 kernel/bpf/verifier.c        | 32 ++++++++++++++++----
 3 files changed, 92 insertions(+), 6 deletions(-)
include/linux/bpf_verifier.h +8 -0
index a8685886f915..24fa1d4f9201 100644
@@ -443,6 +443,8 @@ enum {
 
 	INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
 	INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
+
+	INSN_F_STACK_ARG_ACCESS = BIT(3),
 };
 
 struct bpf_jmp_history_entry {
@@ -842,6 +844,7 @@ struct backtrack_state {
 	u32 frame;
 	u32 reg_masks[MAX_CALL_FRAMES];
 	u64 stack_masks[MAX_CALL_FRAMES];
+	u8 stack_arg_masks[MAX_CALL_FRAMES];
 };
 
 struct bpf_id_pair {
@@ -1240,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame,
 	bt->stack_masks[frame] |= 1ull << slot;
 }
 
+static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
+{
+	bt->stack_arg_masks[frame] |= 1 << slot;
+}
+
 static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
 {
 	return bt->reg_masks[frame] & (1 << reg);
kernel/bpf/backtrack.c +57 -1

98 lines not on this page · raw message

kernel/bpf/verifier.c +27 -5
index 14cbb38aa0e0..30143bea6a86 100644
@@ -292,6 +292,11 @@ static int arg_from_argno(argno_t a)
 	return -1;
 }
 
+static int arg_idx_from_argno(argno_t a)
+{
+	return arg_from_argno(a) - 1;
+}
+
 static const char *btf_type_name(const struct btf *btf, u32 id)
 {
 	return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off);
@@ -4115,7 +4120,8 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
 		__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
 	}
 	state->no_stack_arg_load = true;
-	return 0;
+	return bpf_push_jmp_history(env, env->cur_state,
+				    INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
 }
 
 /*
@@ -4146,7 +4152,17 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
 	arg = &caller->stack_arg_regs[spi];
 	cur = vstate->frame[vstate->curframe];
 	cur->regs[dst_regno] = *arg;
-	return 0;
+	return bpf_push_jmp_history(env, env->cur_state,
+				    INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
+}
+
+static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
+{
+	struct bpf_func_state *caller = cur_func(env);
+	int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
+
+	bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
+	return mark_chain_precision_batch(env, env->cur_state);
 }
 
 static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
@@ -6875,8 +6891,14 @@ static int check_mem_size_reg(struct bpf_verifier_env *env,
 	}
 	err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg),
 				      access_type, zero_size_allowed, meta);
-	if (!err)
-		err = mark_chain_precision(env, reg_from_argno(size_argno));
+	if (!err) {
+		int regno = reg_from_argno(size_argno);
+
+		if (regno >= 0)
+			err = mark_chain_precision(env, regno);
+		else
+			err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno));
+	}
 	return err;
 }
 
@@ -7325,7 +7347,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
 			    struct bpf_kfunc_call_arg_meta *meta)
 {
 	const struct btf_type *t;
-	u32 arg_idx = arg_from_argno(argno) - 1;
+	u32 arg_idx = arg_idx_from_argno(argno);
 	int spi, err, i, nr_slots, btf_id;
 
 	if (reg->type != PTR_TO_STACK) {
Signature · 1 line
2.53.0-Meta
#10 lore raw

Re: [PATCH bpf-next v4 07/25] bpf: Add precision marking and backtracking for stack argument slots

This looks like a bug fix for stack argument support. Should this include:
Quoted 6 lines

commit 8578e3b9e780699b52da3c2fdeeac2cfbc51f3fe

Author: Yonghong Song <[email protected]>
Date:   Tue May 12 21:50:25 2026 -0700

bpf: Add precision marking and backtracking for stack argument slots

Quoted 7 lines

Extend the precision marking and backtracking infrastructure to support stack argument slots (r11-based accesses). Without this, precision demands for scalar values passed through stack arguments are silently dropped, which could allow the verifier to incorrectly prune states with different constant values in stack arg slots.

This looks like a bug fix for stack argument support. Should this include:

Fixes cc06010ca1b8 ("bpf: Support stack arguments for bpf functions")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

author reviewer
#11 lore raw

PATCH v4 8/25 bpf: Refactor record_call_access() to extract per-arg logic

In reply to Yonghong Song

Extract the per-argument FP-derived pointer handling from record_call_access() into a new record_arg_access() helper.

The existing loop body — checking arg_is_fp, querying stack access bytes, and calling record_stack_access/record_imprecise — will be reused for stack argument slots in the next patch. Factoring it out now avoids duplicating the logic.

No functional change.

---
 kernel/bpf/liveness.c | 65 +++++++++++++++++++++++++------------------
 1 file changed, 38 insertions(+), 27 deletions(-)
kernel/bpf/liveness.c +38 -27
index 58197d73b120..c81337dfbfc7 100644
@@ -1343,6 +1343,42 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 	return 0;
 }
 
+static int record_arg_access(struct bpf_verifier_env *env,
+			     struct func_instance *instance,
+			     struct bpf_insn *insn,
+			     struct arg_track *at, int arg_idx,
+			     int insn_idx)
+{
+	int depth = instance->depth;
+	int frame = at->frame;
+	int err = 0;
+	s64 bytes;
+
+	if (!arg_is_fp(at))
+		return 0;
+
+	if (bpf_helper_call(insn)) {
+		bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
+	} else if (bpf_pseudo_kfunc_call(insn)) {
+		bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
+	} else {
+		for (int f = 0; f <= depth; f++) {
+			err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
+			if (err)
+				return err;
+		}
+		return 0;
+	}
+	if (bytes == 0)
+		return 0;
+
+	if (frame >= 0 && frame <= depth)
+		err = record_stack_access(instance, at, bytes, frame, insn_idx);
+	else if (frame == ARG_IMPRECISE)
+		err = record_imprecise(instance, at->mask, insn_idx);
+	return err;
+}
+
 /* Record stack access for a given 'at' state of helper/kfunc 'insn' */
 static int record_call_access(struct bpf_verifier_env *env,
 			      struct func_instance *instance,
@@ -1350,9 +1386,8 @@ static int record_call_access(struct bpf_verifier_env *env,
 			      int insn_idx)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
-	int depth = instance->depth;
 	struct bpf_call_summary cs;
-	int r, err = 0, num_params = 5;
+	int r, err, num_params = 5;
 
 	if (bpf_pseudo_call(insn))
 		return 0;
@@ -1361,31 +1396,7 @@ static int record_call_access(struct bpf_verifier_env *env,
 		num_params = cs.num_params;
 
 	for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) {
-		int frame = at[r].frame;
-		s64 bytes;
-
-		if (!arg_is_fp(&at[r]))
-			continue;
-
-		if (bpf_helper_call(insn)) {
-			bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx);
-		} else if (bpf_pseudo_kfunc_call(insn)) {
-			bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx);
-		} else {
-			for (int f = 0; f <= depth; f++) {
-				err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
-				if (err)
-					return err;
-			}
-			return 0;
-		}
-		if (bytes == 0)
-			continue;
-
-		if (frame >= 0 && frame <= depth)
-			err = record_stack_access(instance, &at[r], bytes, frame, insn_idx);
-		else if (frame == ARG_IMPRECISE)
-			err = record_imprecise(instance, at[r].mask, insn_idx);
+		err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx);
 		if (err)
 			return err;
 	}
Signature · 1 line
2.53.0-Meta
author reviewer
#12 lore raw

PATCH v4 9/25 bpf: Use arg_is_fp() in has_fp_args()

In reply to Yonghong Song

Replace "frame != ARG_NONE" with arg_is_fp() in has_fp_args(). The function's purpose is to check whether any argument is derived from a frame pointer, which is exactly what arg_is_fp() tests (frame >= 0 || frame == ARG_IMPRECISE). Using the dedicated predicate is clearer and more consistent with the rest of the file.

---
 kernel/bpf/liveness.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/liveness.c +1 -1
index c81337dfbfc7..13dc5ae44d2b 100644
@@ -1689,7 +1689,7 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
 static bool has_fp_args(struct arg_track *args)
 {
 	for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
-		if (args[r].frame != ARG_NONE)
+		if (arg_is_fp(&args[r]))
 			return true;
 	return false;
 }
Signature · 1 line
2.53.0-Meta
author reviewer
#13 lore raw

PATCH v4 10/25 bpf: Extend liveness analysis to track stack argument slots

In reply to Yonghong Song

BPF_REG_PARAMS (R11) is at index MAX_BPF_REG, which is beyond the register tracking arrays in const_fold.c and liveness.c. Handle it explicitly to avoid out-of-bounds accesses.

Extend the arg tracking dataflow to cover stack arg slots. Otherwise, pointers passed through stack args are invisible to liveness, causing the pointed-to stack slots to be incorrectly poisoned.

Extend the at_out tracking array to MAX_AT_TRACK_REGS (registers plus stack arg slots) so that outgoing stack arg stores are tracked alongside registers. Add a separate at_stack_arg_entry array in compute_subprog_args(), passed to arg_track_xfer(), to restore FP-derived values on incoming stack arg reads.

Extend record_call_access() to check stack arg slots for FP-derived pointers at kfunc call sites, reusing the record_arg_access() helper extracted in the previous patch. Pass stack arg state from caller to callee in analyze_subprog() so that callees can track pointers received through stack args, hence avoid poisoning.

Skip stack arg instructions in record_load_store_access(). Stack arg STX uses dst_reg=BPF_REG_PARAMS (index 11), but at[11] is repurposed to track the value stored in stack arg slot 0. Without the skip, if a prior stack arg STX stored an FP-derived pointer (e.g., fp-64) into slot 0, a subsequent stack arg STX would read that FP-derived value as the base pointer and spuriously mark a regular stack slot (e.g., fp-72 from -64 + -8) as accessed in the liveness bitmap.

Extend arg_track_log() to log state transitions for outgoing stack arg slots at indices MAX_BPF_REG through MAX_AT_TRACK_REGS-1. Without this, changes to at_out[11..17] caused by stack arg store instructions are silently omitted from BPF_LOG_LEVEL2 output. For example, when a caller passes fp-64 through a stack argument:

subprog#0:
10: (bf) r6 = r10
11: (07) r6 += -64
12: (7b) *(u64 *)(r11 -8) = r6

	sa0: none -> fp0-64

13: (85) call pc+5

Without the fix, the "sa0: none -> fp0-64" transition at insn 12 would not appear.

Extend print_subprog_arg_access() to include stack arg slots in the per-instruction FP-derived state dump. For example:

subprog#0:
12: (7b) *(u64 *)(r11 - 8) = r6 // r6=fp0-64
13: (85) call pc+5 // r6=fp0-64 sa0=fp0-64

Without the fix, the "sa0=fp0-64" annotation at insn 13 would not appear, making it harder to debug liveness analysis for programs that pass FP-derived pointers through stack arguments.

Extend has_fp_args() to also check stack arg slots for FP-derived pointers, so that callees receiving pointers only through stack args are still recursively analyzed.

---
 kernel/bpf/const_fold.c |   8 +++
 kernel/bpf/liveness.c   | 114 ++++++++++++++++++++++++++++++++++------
 2 files changed, 106 insertions(+), 16 deletions(-)
kernel/bpf/const_fold.c +8 -0
index db73c4740b1e..b2a19acadb91 100644
@@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 	u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code);
 	int r;
 
+	/* Stack arg stores (r11-based) are outside the tracked register set. */
+	if (is_stack_arg_st(insn) || is_stack_arg_stx(insn))
+		return;
+	if (is_stack_arg_ldx(insn)) {
+		ci_out[insn->dst_reg] = unknown;
+		return;
+	}
+
 	switch (class) {
 	case BPF_ALU:
 	case BPF_ALU64:
kernel/bpf/liveness.c +98 -16

252 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#14 lore raw

Re: [PATCH bpf-next v4 10/25] bpf: Extend liveness analysis to track stack argument slots

This looks like a bug fix for stack argument support. Should this include:
Quoted 57 lines

bpf: Extend liveness analysis to track stack argument slots

BPF_REG_PARAMS (R11) is at index MAX_BPF_REG, which is beyond the register tracking arrays in const_fold.c and liveness.c. Handle it explicitly to avoid out-of-bounds accesses.

Extend the arg tracking dataflow to cover stack arg slots. Otherwise, pointers passed through stack args are invisible to liveness, causing the pointed-to stack slots to be incorrectly poisoned.

Extend the at_out tracking array to MAX_AT_TRACK_REGS (registers plus stack arg slots) so that outgoing stack arg stores are tracked alongside registers. Add a separate at_stack_arg_entry array in compute_subprog_args(), passed to arg_track_xfer(), to restore FP-derived values on incoming stack arg reads.

Extend record_call_access() to check stack arg slots for FP-derived pointers at kfunc call sites, reusing the record_arg_access() helper extracted in the previous patch. Pass stack arg state from caller to callee in analyze_subprog() so that callees can track pointers received through stack args, hence avoid poisoning.

Skip stack arg instructions in record_load_store_access(). Stack arg STX uses dst_reg=BPF_REG_PARAMS (index 11), but at[11] is repurposed to track the value stored in stack arg slot 0. Without the skip, if a prior stack arg STX stored an FP-derived pointer (e.g., fp-64) into slot 0, a subsequent stack arg STX would read that FP-derived value as the base pointer and spuriously mark a regular stack slot (e.g., fp-72 from -64 + -8) as accessed in the liveness bitmap.

Extend arg_track_log() to log state transitions for outgoing stack arg slots at indices MAX_BPF_REG through MAX_AT_TRACK_REGS-1. Without this, changes to at_out[11..17] caused by stack arg store instructions are silently omitted from BPF_LOG_LEVEL2 output. For example, when a caller passes fp-64 through a stack argument:

subprog#0:
10: (bf) r6 = r10
11: (07) r6 += -64
12: (7b) *(u64 *)(r11 -8) = r6
sa0: none -> fp0-64
13: (85) call pc+5

Without the fix, the "sa0: none -> fp0-64" transition at insn 12 would not appear.

Extend print_subprog_arg_access() to include stack arg slots in the per-instruction FP-derived state dump. For example:

subprog#0:
12: (7b) *(u64 *)(r11 - 8) = r6 // r6=fp0-64
13: (85) call pc+5 // r6=fp0-64 sa0=fp0-64

Without the fix, the "sa0=fp0-64" annotation at insn 13 would not appear, making it harder to debug liveness analysis for programs that pass FP-derived pointers through stack arguments.

Quoted 5 lines

Extend has_fp_args() to also check stack arg slots for FP-derived pointers, so that callees receiving pointers only through stack args are still recursively analyzed.

This looks like a bug fix for stack argument support. Should this include:

Fixes cc06010ca1b8 ("bpf: Support stack arguments for bpf functions")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

#15 lore raw

Re: [PATCH bpf-next v4 10/25] bpf: Extend liveness analysis to track stack argument slots

In reply to Yonghong Song
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Incorrect logging of stack argument slots in `arg_track_join`. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Incorrect logging of stack argument slots in arg_track_join.
--

commit 5139934569dff44b05e1ee083e4eb084fdca15e3
Author: Yonghong Song <[email protected]>

bpf: Extend liveness analysis to track stack argument slots

This commit extends the BPF verifier's liveness analysis to correctly track stack argument slots. It expands tracking arrays to cover MAX_AT_TRACK_REGS and propagates the tracked state through function calls to prevent incorrectly poisoning stack slots derived from frame pointers passed through stack arguments. It also updates logging functions to expose these new stack argument states.

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
index 13dc5ae44d2b5..7f4a0e4c2c49d 100644

[ ... ]

kernel/bpf/liveness.c static int compute_subprog_args(struct bpf_verifier_env *env, · line 1710

Quoted 8 lines
kernel/bpf/liveness.c +1 -1
@@ -1630,7 +1705,7 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
 				continue;
 			ti = target - start;
 
-			for (r = 0; r < MAX_BPF_REG; r++)
+			for (r = 0; r < MAX_AT_TRACK_REGS; r++)
 				changed |= arg_track_join(env, idx, target, r,
 							  &at_in[ti][r], at_out[r]);

Is it possible this will misleadingly print stack argument slots as non-existent registers?

Since compute_subprog_args() now iterates r up to MAX_AT_TRACK_REGS, arg_track_join() will receive r >= MAX_BPF_REG.

Looking at arg_track_join():

kernel/bpf/liveness.c:arg_track_join() {
    ...
	if (r >= 0)
		verbose(env, "r%d: ", r);
	else
		verbose(env, "fp%+d: ", r * 8);
    ...
}

It seems this will log stack argument slots as r11, r12, etc., in the BPF_LOG_LEVEL2 verifier log, rather than their proper designations (sa0 to sa5) like arg_track_log() and print_subprog_arg_access() do.

Should arg_track_join() be updated to handle r >= MAX_BPF_REG to maintain consistent formatting?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author reviewer
#16 lore raw

Re: [PATCH bpf-next v4 10/25] bpf: Extend liveness analysis to track stack argument slots

Quoted 47 lines · On 5/14/26 12:53 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Incorrect logging of stack argument slots in arg_track_join.
--

commit 5139934569dff44b05e1ee083e4eb084fdca15e3
Author: Yonghong Song <[email protected]>

bpf: Extend liveness analysis to track stack argument slots

This commit extends the BPF verifier's liveness analysis to correctly track stack argument slots. It expands tracking arrays to cover MAX_AT_TRACK_REGS and propagates the tracked state through function calls to prevent incorrectly poisoning stack slots derived from frame pointers passed through stack arguments. It also updates logging functions to expose these new stack argument states.

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
index 13dc5ae44d2b5..7f4a0e4c2c49d 100644

[ ... ]

kernel/bpf/liveness.c static int compute_subprog_args(struct bpf_verifier_env *env, · line 1710

Quoted 8 lines
kernel/bpf/liveness.c +1 -1
@@ -1630,7 +1705,7 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
  				continue;
  			ti = target - start;
 
-			for (r = 0; r < MAX_BPF_REG; r++)
+			for (r = 0; r < MAX_AT_TRACK_REGS; r++)
  				changed |= arg_track_join(env, idx, target, r,
  							  &at_in[ti][r], at_out[r]);

Is it possible this will misleadingly print stack argument slots as non-existent registers?

Since compute_subprog_args() now iterates r up to MAX_AT_TRACK_REGS, arg_track_join() will receive r >= MAX_BPF_REG.

Looking at arg_track_join():

kernel/bpf/liveness.c:arg_track_join() {
     ...
	if (r >= 0)
		verbose(env, "r%d: ", r);
	else
		verbose(env, "fp%+d: ", r * 8);
     ...
}
Quoted 6 lines

It seems this will log stack argument slots as r11, r12, etc., in the BPF_LOG_LEVEL2 verifier log, rather than their proper designations (sa0 to sa5) like arg_track_log() and print_subprog_arg_access() do.

Should arg_track_join() be updated to handle r >= MAX_BPF_REG to maintain consistent formatting?

Yes, you are right, for 'r >= 0' case, we should have r[0-10] and for r >= 11 we should use 'sa*' to indicate it is for stack arguments.
Will fix.

author reviewer
#17 lore raw

PATCH v4 11/25 bpf: Reject stack arguments in non-JITed programs

In reply to Yonghong Song

The interpreter does not understand the bpf register r11 (BPF_REG_PARAMS) used for stack arguments. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.

---
 kernel/bpf/core.c   | 2 +-
 kernel/bpf/fixups.c | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)
kernel/bpf/core.c +1 -1
index ae10b9ca018d..958d86f0beac 100644
@@ -2599,7 +2599,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || (env && env->subprog_info[0].stack_arg_cnt))
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/fixups.c +6 -0
index ba86039789fd..19056016eed8 100644
@@ -1407,6 +1407,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}
+	for (i = 1; i < env->subprog_cnt; i++) {
+		if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
 	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
 		/* When JIT fails the progs with bpf2bpf calls and tail_calls
 		 * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.53.0-Meta
#18 lore raw

Re: [PATCH bpf-next v4 11/25] bpf: Reject stack arguments in non-JITed programs

This looks like a bug fix for stack argument support. The interpreter does not understand register r11 (BPF_REG_PARAMS) used for stack arguments, but the original stack argument implementation did…
Quoted 8 lines

bpf: Reject stack arguments in non-JITed programs

The interpreter does not understand the bpf register r11 (BPF_REG_PARAMS) used for stack arguments. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.

This looks like a bug fix for stack argument support. The interpreter does not understand register r11 (BPF_REG_PARAMS) used for stack arguments, but the original stack argument implementation did not add checks to prevent interpreter usage.

Should this include:

Fixes cc06010ca1b8 ("bpf: Support stack arguments for bpf functions")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

#19 lore raw

Re: [PATCH bpf-next v4 11/25] bpf: Reject stack arguments in non-JITed programs

In reply to Yonghong Song
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] The patch fails to prevent interpreter fallback if stack arguments are used only in nested…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] The patch fails to prevent interpreter fallback if stack arguments are used only in nested subprograms (e.g., Main -> Subprog A -> Subprog B). This can lead to a kernel crash via an uninitialized R11 dereference.
- [Low] Standalone global functions with stack arguments are rejected with -ENOTSUPP without a verifier log, instead of -EINVAL with a descriptive message.
--

commit 851a484e2b5b68983f11bc3764118f2bcba5acb0
Author: Yonghong Song <[email protected]>

bpf: Reject stack arguments in non-JITed programs

This patch attempts to reject programs that use stack arguments when running in the interpreter, as the interpreter does not support the BPF_REG_PARAMS (r11) register. It does this by checking for stack arguments in __bpf_prog_select_runtime() and bpf_fixup_call_args().

kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2603

Quoted 11 lines
kernel/bpf/core.c +1 -1
index ae10b9ca018df..958d86f0beac3 100644
@@ -2599,7 +2599,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || (env && env->subprog_info[0].stack_arg_cnt))
 		jit_needed = true;

Does this condition correctly catch stack arguments in nested subprograms?

env->subprog_info[0].stack_arg_cnt only tracks the main program's incoming and direct outgoing stack arguments. If the main program calls Subprog A with no stack arguments, and Subprog A calls Subprog B with more than 5 arguments, the main program's stack_arg_cnt remains 0.

If bpf_prog_jit_compile() subsequently fails for the main program, jit_needed evaluates to false and it falls back to the interpreter.

kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1409

Quoted 10 lines
kernel/bpf/core.c +0 -0
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/fixups.c +0 -0
index ba86039789fdf..19056016eed8b 100644
@@ -1407,6 +1407,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}

Is it possible for bpf_jit_subprogs() to succeed earlier in this function and return 0, bypassing these new checks entirely?

If jit_requested is true, bpf_jit_subprogs() is called before this point:

kernel/bpf/fixups.c:bpf_fixup_call_args() {
    ...
	if (env->prog->jit_requested &&
	    !bpf_prog_is_offloaded(env->prog->aux)) {
		err = bpf_jit_subprogs(env);
		if (err == 0)
			return 0;
    ...
}

If it succeeds, the function returns 0 early, skipping the new loop.

Combined with the issue in __bpf_prog_select_runtime() above, the interpreter could end up executing the main program and processing BPF_PSEUDO_CALL instructions by jumping to the un-JITted instructions of the subprograms.

When Subprog A writes to outgoing stack slots via BPF_REG_PARAMS (r11), the interpreter will use an uninitialized r11, potentially corrupting memory.

kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1410

Quoted 1 line
kernel/bpf/fixups.c +1 -0
+	for (i = 1; i < env->subprog_cnt; i++) {

Should this loop start at i = 0 to properly handle standalone global functions?

For standalone global functions that are verified independently, env->subprog_cnt is 1. Starting the loop at i = 1 skips the global function entirely.

The program is still safely rejected later in __bpf_prog_select_runtime() because env->subprog_info[0].stack_arg_cnt > 0 correctly sets jit_needed = true, leading to an -ENOTSUPP return. However, starting at i = 0 would emit the expected verifier log and return -EINVAL instead of failing silently with -ENOTSUPP.

Quoted 6 lines
kernel/bpf/fixups.c +5 -0
+		if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
 	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author reviewer
#20 lore raw

Re: [PATCH bpf-next v4 11/25] bpf: Reject stack arguments in non-JITed programs

Quoted 22 lines · On 5/15/26 1:59 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] The patch fails to prevent interpreter fallback if stack arguments are used only in nested subprograms (e.g., Main -> Subprog A -> Subprog B). This can lead to a kernel crash via an uninitialized R11 dereference.
- [Low] Standalone global functions with stack arguments are rejected with -ENOTSUPP without a verifier log, instead of -EINVAL with a descriptive message.
--

commit 851a484e2b5b68983f11bc3764118f2bcba5acb0
Author: Yonghong Song <[email protected]>

bpf: Reject stack arguments in non-JITed programs

This patch attempts to reject programs that use stack arguments when running in the interpreter, as the interpreter does not support the BPF_REG_PARAMS (r11) register. It does this by checking for stack arguments in __bpf_prog_select_runtime() and bpf_fixup_call_args().

Quoted 7 lines
kernel/bpf/core.c +0 -0
index ae10b9ca018df..958d86f0beac3 100644
@@ -2599,7 +2599,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
  		goto finalize;
 
Quoted 5 lines

kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2603

Quoted 4 lines
kernel/bpf/core.c +1 -1
  	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || (env && env->subprog_info[0].stack_arg_cnt))
  		jit_needed = true;

Does this condition correctly catch stack arguments in nested subprograms?

I think the above condition '(env && env->subprog_info[0].stack_arg_cnt)' is unnecessary. This check is for the *main* prog. main prog should not call kfunc, otherwise jit_needed must be true. helper does not have stack arguments.

If bpf-to-bpf calls are used in this prog, then bpf_fixup_call_args() will do some checking for all subprogs.

Quoted 35 lines

env->subprog_info[0].stack_arg_cnt only tracks the main program's incoming and direct outgoing stack arguments. If the main program calls Subprog A with no stack arguments, and Subprog A calls Subprog B with more than 5 arguments, the main program's stack_arg_cnt remains 0.

If bpf_prog_jit_compile() subsequently fails for the main program, jit_needed evaluates to false and it falls back to the interpreter.

kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1409

Quoted 10 lines
kernel/bpf/core.c +0 -0
  	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/fixups.c +0 -0
index ba86039789fdf..19056016eed8b 100644
@@ -1407,6 +1407,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
  		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
  		return -EINVAL;
  	}

Is it possible for bpf_jit_subprogs() to succeed earlier in this function and return 0, bypassing these new checks entirely?

If jit_requested is true, bpf_jit_subprogs() is called before this point:

kernel/bpf/fixups.c:bpf_fixup_call_args() {
     ...
	if (env->prog->jit_requested &&
	    !bpf_prog_is_offloaded(env->prog->aux)) {
		err = bpf_jit_subprogs(env);
		if (err == 0)
			return 0;
     ...
}
Quoted 1 line

If it succeeds, the function returns 0 early, skipping the new loop.

But if bpf_jit_subprogs() succeed (with more than one subprog), then we will have

           prog->bpf_func = func[0]->bpf_func;

In __bpf_prog_select_runtime(), we have

	   if (fp->bpf_func)
                goto finalize;

So interpreter will be ignored since jit has done successfully.

Quoted 11 lines

Combined with the issue in __bpf_prog_select_runtime() above, the interpreter could end up executing the main program and processing BPF_PSEUDO_CALL instructions by jumping to the un-JITted instructions of the subprograms.

When Subprog A writes to outgoing stack slots via BPF_REG_PARAMS (r11), the interpreter will use an uninitialized r11, potentially corrupting memory.

kernel/bpf/fixups.c int bpf_fixup_call_args(struct bpf_verifier_env *env) · line 1410

Quoted 1 line
kernel/bpf/fixups.c +1 -0
+	for (i = 1; i < env->subprog_cnt; i++) {

Should this loop start at i = 0 to properly handle standalone global functions?

We can do this. Previously I skip 'i = 0' due to

                if (env->prog->aux->func_info_aux) {
                        ret = btf_prepare_func_args(env, 0);
                        if (ret || sub->arg_cnt != 1 || sub->args[0].arg_type != ARG_PTR_TO_CTX) {
                                env->prog->aux->func_info_aux[0].unreliable = true;

                }

since the main prog may have a btf which have more than 5 arguments although in reality it will have just one argument. If I have i = 0, this will reject interpreter which is not correct.

Now we have the following change

                if (env->prog->aux->func_info_aux) {
                        ret = btf_prepare_func_args(env, 0);
                        if (ret || sub->arg_cnt != 1 || sub->args[0].arg_type != ARG_PTR_TO_CTX) {
                                env->prog->aux->func_info_aux[0].unreliable = true;
                                sub->arg_cnt = 1;
                                sub->stack_arg_cnt = 0;
                        }
                }

So we can safely start from i = 0. Note that i = 0 will definitely succeed since we have sub->arg_cnt = 1 above. So change to i = 0 won't change anything but can avoid confusion.

Quoted 17 lines

For standalone global functions that are verified independently, env->subprog_cnt is 1. Starting the loop at i = 1 skips the global function entirely.

The program is still safely rejected later in __bpf_prog_select_runtime() because env->subprog_info[0].stack_arg_cnt > 0 correctly sets jit_needed = true, leading to an -ENOTSUPP return. However, starting at i = 0 would emit the expected verifier log and return -EINVAL instead of failing silently with -ENOTSUPP.

Quoted 6 lines
kernel/bpf/fixups.c +5 -0
+		if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
  	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
author reviewer
#21 lore raw

PATCH v4 12/25 bpf: Prepare architecture JIT support for stack arguments

In reply to Yonghong Song

Add bpf_jit_supports_stack_args() as a weak function defaulting to false. Architectures that implement JIT support for stack arguments override it to return true.

Reject BPF functions with more than 5 parameters at verification time if the architecture does not support stack arguments.

---
 include/linux/filter.h | 1 +
 kernel/bpf/btf.c       | 8 +++++++-
 kernel/bpf/core.c      | 5 +++++
 3 files changed, 13 insertions(+), 1 deletion(-)
include/linux/filter.h +1 -0
index 918d9b34eac6..a515a9769078 100644
@@ -1184,6 +1184,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_ptr_xchg(void);
kernel/bpf/btf.c +7 -1
index ec3fb8c8f4ee..3d8080eba544 100644
@@ -7886,8 +7886,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			MAX_BPF_FUNC_ARGS, tname, nargs);
 		return -EFAULT;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -EFAULT;
+		}
 		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+	}
 
 	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
 		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
kernel/bpf/core.c +5 -0
index 958d86f0beac..e6b836f846eb 100644
@@ -3217,6 +3217,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_stack_args(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_far_kfunc_call(void)
 {
 	return false;
Signature · 1 line
2.53.0-Meta
#22 lore raw

Re: [PATCH bpf-next v4 12/25] bpf: Prepare architecture JIT support for stack arguments

Is there an issue with the error code ordering here? Looking at btf_prepare_func_args(), the code checks JIT stack argument support before checking if the function is global.
Quoted 4 lines
include/linux/filter.h +0 -0
index 918d9b34eac6..a515a9769078 100644

[ ... ]

Quoted 10 lines
kernel/bpf/btf.c +1 -1
index 4fb8641546b8..17d4ab0a8206 100644
@@ -7870,8 +7870,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			MAX_BPF_FUNC_ARGS, tname, nargs);
 		return -EFAULT;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {

kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 7883

Quoted 10 lines
kernel/bpf/btf.c +6 -0
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -EFAULT;
+		}
 		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+	}
 
 	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
 		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",

Is there an issue with the error code ordering here? Looking at btf_prepare_func_args(), the code checks JIT stack argument support before checking if the function is global.

For global functions with more than 5 parameters on architectures without JIT stack argument support, the code returns -EFAULT at the first check instead of -EINVAL at the second check.

According to the function's documentation comment, EFAULT means "there is a verifier bug" while EINVAL means "cannot convert BTF" (invalid program).

Global functions with more than 5 arguments are always invalid regardless of JIT capabilities, as stated in the second check's error message: "stack args not supported". This is a user error, not a verifier bug.

Should the global function check come first to ensure it returns -EINVAL?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

#23 lore raw

Re: [PATCH bpf-next v4 12/25] bpf: Prepare architecture JIT support for stack arguments

In reply to Yonghong Song
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Returning `-EFAULT` for an unsupported JIT feature causes a misleading syscall error. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Returning -EFAULT for an unsupported JIT feature causes a misleading syscall error. --

commit 468549a72b25192a0a4df32e6475b6634931dde6
Author: Yonghong Song <[email protected]>

bpf: Prepare architecture JIT support for stack arguments

This commit adds bpf_jit_supports_stack_args() to allow architectures to indicate support for stack arguments. It rejects BPF functions with more than 5 parameters at verification time if the architecture lacks this support.

Quoted 4 lines
kernel/bpf/btf.c +0 -0
index ec3fb8c8f4eef..3d8080eba5445 100644

[ ... ]

kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 7893

Quoted 10 lines
kernel/bpf/btf.c +5 -1
@@ -7886,8 +7886,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			MAX_BPF_FUNC_ARGS, tname, nargs);
 		return -EFAULT;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -EFAULT;

Does returning -EFAULT here cause a misleading error for userspace?

The function documentation states that -EFAULT is reserved for internal verifier bugs to abort verification, while unsupported features or invalid arguments typically return -EINVAL or -EOPNOTSUPP (which is used elsewhere in this function for unsupported type tags).

Since returning -EFAULT propagates to the bpf() system call and typically indicates a bad address, it might confuse userspace loaders like libbpf. Would -EINVAL or -EOPNOTSUPP be more appropriate here?

Quoted 6 lines
kernel/bpf/btf.c +2 -0
+		}
 		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+	}
 
 	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
 		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author reviewer
#24 lore raw

Re: [PATCH bpf-next v4 12/25] bpf: Prepare architecture JIT support for stack arguments

Quoted 31 lines · On 5/15/26 2:30 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Returning -EFAULT for an unsupported JIT feature causes a misleading syscall error. --

commit 468549a72b25192a0a4df32e6475b6634931dde6
Author: Yonghong Song <[email protected]>

bpf: Prepare architecture JIT support for stack arguments

This commit adds bpf_jit_supports_stack_args() to allow architectures to indicate support for stack arguments. It rejects BPF functions with more than 5 parameters at verification time if the architecture lacks this support.

Quoted 4 lines
kernel/bpf/btf.c +0 -0
index ec3fb8c8f4eef..3d8080eba5445 100644

[ ... ]

kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 7893

Quoted 10 lines
kernel/bpf/btf.c +5 -1
@@ -7886,8 +7886,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
  			MAX_BPF_FUNC_ARGS, tname, nargs);
  		return -EFAULT;
  	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -EFAULT;

Does returning -EFAULT here cause a misleading error for userspace?

Quoted 8 lines

The function documentation states that -EFAULT is reserved for internal verifier bugs to abort verification, while unsupported features or invalid arguments typically return -EINVAL or -EOPNOTSUPP (which is used elsewhere in this function for unsupported type tags).

Since returning -EFAULT propagates to the bpf() system call and typically indicates a bad address, it might confuse userspace loaders like libbpf. Would -EINVAL or -EOPNOTSUPP be more appropriate here?

-EFAULT here is necessary. Otherwise, later on, the verification/jit will proceed and it may cause incorrect results. Note that -EINVAL/-EOPNOTSUPP won't prevent further verification/jit.

Quoted 7 lines
Quoted 7 lines
kernel/bpf/btf.c +2 -0
+		}
  		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+	}
 
  	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
  		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
author reviewer
#25 lore raw

PATCH v4 13/25 bpf: Enable r11 based insns

In reply to Yonghong Song

BPF_REG_PARAMS (r11) is used for stack argument accesses and the following are only insns with r11 presence:
- load incoming stack arg
- store register to outgoing stack arg
- store immediate to outgoing stack arg

The detailed insn format can be found in is_stack_arg_ldx/st/stx() helpers. After this patch, stack arg ldx/st/stx insns become valid for kernel and these insns can be properly checked by verifier.

The LLVM compiler [1] implemented the above BPF_REG_PARAMS insns.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 kernel/bpf/verifier.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +3 -2
index 30143bea6a86..18bbc5b6c2d2 100644
@@ -18006,11 +18006,12 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
 		return err;
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
-		if (insn->dst_reg >= MAX_BPF_REG) {
+		if (insn->dst_reg >= MAX_BPF_REG &&
+		    !is_stack_arg_st(insn) && !is_stack_arg_stx(insn)) {
 			verbose(env, "R%d is invalid\n", insn->dst_reg);
 			return -EINVAL;
 		}
-		if (insn->src_reg >= MAX_BPF_REG) {
+		if (insn->src_reg >= MAX_BPF_REG && !is_stack_arg_ldx(insn)) {
 			verbose(env, "R%d is invalid\n", insn->src_reg);
 			return -EINVAL;
 		}
Signature · 1 line
2.53.0-Meta
author reviewer
#26 lore raw

PATCH v4 14/25 bpf: Support stack arguments for kfunc calls

In reply to Yonghong Song

Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r11-based stack arg area.

For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at negative r11 offsets, following the same convention as BPF-to-BPF calls:

Outgoing: r11 - 8 (arg6), ..., r11 - N*8 (last arg)

The following is an example:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

Caller (foo), generated by llvm
===============================
Incoming (positive offsets):

     r11+8:  [incoming arg 6]
     r11+16: [incoming arg 7]

Outgoing for kfunc1 (negative offsets):

     r11-8:  [outgoing arg 6]
     r11-16: [outgoing arg 7]
     r11-24: [outgoing arg 8]

Outgoing for kfunc2 (negative offsets):

     r11-8:  [outgoing arg 6]
     r11-16: [outgoing arg 7]
     r11-24: [outgoing arg 8]
     r11-32: [outgoing arg 9]

Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().

For kfunc calls where stack args are used as constant or size parameters, a mark_stack_arg_precision() helper is used to propagate precision and do proper backtracking.

There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno' is also keeping regno for later fixup. Since stack arguments don't have a valid register number (regno is negative), these three cases are rejected for now if the argument is on the stack.

---
 kernel/bpf/verifier.c | 77 +++++++++++++++++++++++++++++++++----------
 1 file changed, 60 insertions(+), 17 deletions(-)
kernel/bpf/verifier.c +60 -17
index 18bbc5b6c2d2..a363811aec7a 100644
@@ -11160,14 +11160,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)
 }
 
 static enum kfunc_ptr_arg_type
-get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
-		       struct bpf_kfunc_call_arg_meta *meta,
+get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller,
+		       struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta,
 		       const struct btf_type *t, const struct btf_type *ref_t,
 		       const char *ref_tname, const struct btf_param *args,
 		       int arg, int nargs, argno_t argno, struct bpf_reg_state *reg)
 {
-	u32 regno = arg + 1;
-	struct bpf_reg_state *regs = cur_regs(env);
 	bool arg_mem_size = false;
 
 	if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11176,8 +11174,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		return KF_ARG_PTR_TO_CTX;
 
 	if (arg + 1 < nargs &&
-	    (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], &regs[regno + 1]) ||
-	     is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], &regs[regno + 1])))
+	    (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) ||
+	     is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1))))
 		arg_mem_size = true;
 
 	/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -11842,6 +11840,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			    int insn_idx)
 {
 	const char *func_name = meta->func_name, *ref_tname;
+	struct bpf_func_state *caller = cur_func(env);
+	struct bpf_reg_state *regs = cur_regs(env);
 	const struct btf *btf = meta->btf;
 	const struct btf_param *args;
 	struct btf_record *rec;
@@ -11850,21 +11850,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+	if (nargs > MAX_BPF_FUNC_ARGS) {
 		verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
-			MAX_BPF_FUNC_REG_ARGS);
+			MAX_BPF_FUNC_ARGS);
 		return -EINVAL;
 	}
+	if (nargs > MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args()) {
+		verbose(env, "JIT does not support kfunc %s() with %d args\n",
+			func_name, nargs);
+		return -ENOTSUPP;
+	}
+
+	ret = check_outgoing_stack_args(env, caller, nargs);
+	if (ret)
+		return ret;
 
 	/* Check that BTF function arguments match actual types that the
 	 * verifier sees.
 	 */
 	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
 		argno_t argno = argno_from_arg(i + 1);
-		u32 regno = i + 1, ref_id, type_size;
+		int regno = reg_from_argno(argno);
+		u32 ref_id, type_size;
 		bool is_ret_buf_sz = false;
 		int kf_arg_type;
 
@@ -11874,6 +11884,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc");
 				return -EFAULT;
 			}
+			if (regno < 0) {
+				verbose(env, "%s prog->aux cannot be a stack argument\n",
+					reg_arg_name(env, argno));
+				return -EINVAL;
+			}
 			meta->arg_prog = true;
 			cur_aux(env)->arg_prog = regno;
 			continue;
@@ -11900,7 +11915,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 						reg_arg_name(env, argno));
 					return -EINVAL;
 				}
-				ret = mark_chain_precision(env, regno);
+				if (regno >= 0)
+					ret = mark_chain_precision(env, regno);
+				else
+					ret = mark_stack_arg_precision(env, i);
 				if (ret < 0)
 					return ret;
 				meta->arg_constant.found = true;
@@ -11925,7 +11943,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				}
 
 				meta->r0_size = reg->var_off.value;
-				ret = mark_chain_precision(env, regno);
+				if (regno >= 0)
+					ret = mark_chain_precision(env, regno);
+				else
+					ret = mark_stack_arg_precision(env, i);
 				if (ret)
 					return ret;
 			}
@@ -11953,14 +11974,21 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				return -EFAULT;
 			}
 			meta->ref_obj_id = reg->ref_obj_id;
-			if (is_kfunc_release(meta))
+			if (is_kfunc_release(meta)) {
+				if (regno < 0) {
+					verbose(env, "%s release arg cannot be a stack argument\n",
+						reg_arg_name(env, argno));
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
+			}
 		}
 
 		ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
 		ref_tname = btf_name_by_offset(btf, ref_t->name_off);
 
-		kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs, argno, reg);
+		kf_arg_type = get_kfunc_ptr_arg_type(env, caller, regs, meta, t, ref_t, ref_tname,
+						     args, i, nargs, argno, reg);
 		if (kf_arg_type < 0)
 			return kf_arg_type;
 
@@ -12110,6 +12138,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				dynptr_arg_type |= DYNPTR_TYPE_FILE;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) {
 				dynptr_arg_type |= DYNPTR_TYPE_FILE | OBJ_RELEASE;
+				if (regno < 0) {
+					verbose(env, "%s release arg cannot be a stack argument\n",
+						reg_arg_name(env, argno));
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] &&
 				   (dynptr_arg_type & MEM_UNINIT)) {
@@ -12264,9 +12297,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			break;
 		case KF_ARG_PTR_TO_MEM_SIZE:
 		{
-			struct bpf_reg_state *buff_reg = &regs[regno];
+			struct bpf_reg_state *buff_reg = reg;
 			const struct btf_param *buff_arg = &args[i];
-			struct bpf_reg_state *size_reg = &regs[regno + 1];
+			struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1);
 			const struct btf_param *size_arg = &args[i + 1];
 			argno_t next_argno = argno_from_arg(i + 2);
 
@@ -13171,8 +13204,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_all_pkt_pointers(env);
 
 	nargs = btf_type_vlen(meta.func_proto);
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		struct bpf_func_state *caller = cur_func(env);
+		struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+		u16 out_stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+		u16 stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + out_stack_arg_cnt;
+
+		if (stack_arg_cnt > caller_info->stack_arg_cnt)
+			caller_info->stack_arg_cnt = stack_arg_cnt;
+	}
+
 	args = (const struct btf_param *)(meta.func_proto + 1);
-	for (i = 0; i < nargs; i++) {
+	for (i = 0; i < min_t(int, nargs, MAX_BPF_FUNC_REG_ARGS); i++) {
 		u32 regno = i + 1;
 
 		t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
Signature · 1 line
2.53.0-Meta
author reviewer
#27 lore raw

PATCH v4 15/25 bpf: Reject stack arguments if tail call reachable

In reply to Yonghong Song

Tail calls are deprecated and will be replaced by indirect calls in the future. Reject programs that combine tail calls with stack arguments rather than adding complexity for a deprecated feature.

---
 kernel/bpf/verifier.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +10 -1
index a363811aec7a..47df3f40a71c 100644
@@ -5267,14 +5267,23 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
 	 * this info will be utilized by JIT so that we will be preserving the
 	 * tail call counter throughout bpf2bpf calls combined with tailcalls
 	 */
-	if (tail_call_reachable)
+	if (tail_call_reachable) {
 		for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
 			if (subprog[tmp].is_exception_cb) {
 				verbose(env, "cannot tail call within exception cb\n");
 				return -EINVAL;
 			}
+			if (subprog[tmp].stack_arg_cnt) {
+				verbose(env, "tail_calls are not allowed in programs with stack args\n");
+				return -EINVAL;
+			}
 			subprog[tmp].tail_call_reachable = true;
 		}
+	} else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {
+		verbose(env, "tail_calls are not allowed in programs with stack args\n");
+		return -EINVAL;
+	}
+
 	if (subprog[0].tail_call_reachable)
 		env->prog->aux->tail_call_reachable = true;
 
Signature · 1 line
2.53.0-Meta
#28 lore raw

Re: [PATCH bpf-next v4 15/25] bpf: Reject stack arguments if tail call reachable

Multiple reviewers across v4, v5, v6, v1, v2, and v3 raised concerns that the commit message inaccurately states 'Tail calls are deprecated' when the actual technical reason is that tail calls and…
Quoted 11 lines

commit 45936037fdce86edc48f07b4ce94daed4efac78e

Author: Yonghong Song <[email protected]>
Date:   Tue May 12 21:51:09 2026 -0700

bpf: Reject stack arguments if tail call reachable

Tail calls are deprecated and will be replaced by indirect calls in the future. Reject programs that combine tail calls with stack arguments rather than adding complexity for a deprecated feature.

Multiple reviewers across v4, v5, v6, v1, v2, and v3 raised concerns that the commit message inaccurately states 'Tail calls are deprecated' when the actual technical reason is that tail calls and stack arguments require incompatible stack frame handling. The author acknowledged this feedback but chose to keep the commit message as-is, stating that tail calls are 'in deprecation stage'.

Would it be more precise if the commit message explained the technical
constraint: 'tail calls and stack arguments require incompatible stack frame handling'?

This looks like a bug fix for stack argument support. Should this include:

Fixes cc06010ca1b8 ("bpf: Support stack arguments for bpf functions")
Quoted 18 lines
kernel/bpf/verifier.c +4 -1
index 1a734ab91a31..a10cc045057d 100644
@@ -5267,14 +5267,23 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
 	 * this info will be utilized by JIT so that we will be preserving the
 	 * tail call counter throughout bpf2bpf calls combined with tailcalls
 	 */
-	if (tail_call_reachable)
+	if (tail_call_reachable) {
 		for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
 			if (subprog[tmp].is_exception_cb) {
 				verbose(env, "cannot tail call within exception cb\n");
 				return -EINVAL;
 			}
+			if (subprog[tmp].stack_arg_cnt) {
+				verbose(env, "tail_calls are not allowed in programs with stack args\n");
+				return -EINVAL;

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5288

Quoted 10 lines
kernel/bpf/verifier.c +6 -0
+			}
 			subprog[tmp].tail_call_reachable = true;
 		}
+	} else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {
+		verbose(env, "tail_calls are not allowed in programs with stack args\n");
+		return -EINVAL;
+	}
+
 	if (subprog[0].tail_call_reachable)
 		env->prog->aux->tail_call_reachable = true;

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

#29 lore raw

Re: [PATCH bpf-next v4 15/25] bpf: Reject stack arguments if tail call reachable

In reply to Yonghong Song
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Order-dependent verification allows invalid programs combining tail calls and stack arguments to…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Order-dependent verification allows invalid programs combining tail calls and stack arguments to bypass checks.
- [High] Exception and async callbacks bypass the new stack argument and tail call restriction.
--

commit 807349b213e40b7418c45dc1ae2cee5695bf101d
Author: Yonghong Song <[email protected]>

bpf: Reject stack arguments if tail call reachable

This patch updates the BPF verifier to reject programs that combine tail calls with stack arguments, as tail calls are deprecated. It hooks into the call depth traversal to fail verification if stack arguments are present in a call chain that can reach a tail call.

Quoted 6 lines
kernel/bpf/verifier.c +0 -0
index a363811aec7a0..47df3f40a71cf 100644
@@ -5267,14 +5267,23 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
 	 * this info will be utilized by JIT so that we will be preserving the

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5276

Quoted 10 lines
kernel/bpf/verifier.c +2 -1
 	 * tail call counter throughout bpf2bpf calls combined with tailcalls
 	 */
-	if (tail_call_reachable)
+	if (tail_call_reachable) {
 		for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
 			if (subprog[tmp].is_exception_cb) {
 				verbose(env, "cannot tail call within exception cb\n");
 				return -EINVAL;
 			}
+			if (subprog[tmp].stack_arg_cnt) {

Can this check be bypassed depending on the order of subprogram calls?

If a main program calls subprogram A (which uses stack args) and then subprogram B (which has a tail call), it seems subprogram A might be evaluated first when tail_call_reachable is false, bypassing this check.

When subprogram B is evaluated, it sets tail_call_reachable to true, which propagates to the main program.

When the main program finishes, only the main program's stack_arg_cnt is checked, allowing the program to pass verification.

If the calls were reversed, subprogram A would inherit the true state and be correctly rejected.

Does this sequence allow invalid programs to bypass the safety check?

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5282

Quoted 6 lines
kernel/bpf/verifier.c +4 -0
+				verbose(env, "tail_calls are not allowed in programs with stack args\n");
+				return -EINVAL;
+			}
 			subprog[tmp].tail_call_reachable = true;
 		}
+	} else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {

Does this fallback check miss exception and async callbacks?

By explicitly checking !idx and subprog[0], it appears to assume the main program is the only call graph root.

Since exception and async callbacks are verified independently as their own root programs (where idx > 0), would a callback that contains a tail call, uses stack arguments, and makes no BPF-to-BPF calls bypass both the primary tail_call_reachable check and this fallback check?

Quoted 6 lines
kernel/bpf/verifier.c +4 -0
+		verbose(env, "tail_calls are not allowed in programs with stack args\n");
+		return -EINVAL;
+	}
+
 	if (subprog[0].tail_call_reachable)
 		env->prog->aux->tail_call_reachable = true;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author reviewer
#30 lore raw

Re: [PATCH bpf-next v4 15/25] bpf: Reject stack arguments if tail call reachable

Quoted 24 lines · On 5/15/26 5:23 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Order-dependent verification allows invalid programs combining tail calls and stack arguments to bypass checks.
- [High] Exception and async callbacks bypass the new stack argument and tail call restriction.
--

commit 807349b213e40b7418c45dc1ae2cee5695bf101d
Author: Yonghong Song <[email protected]>

bpf: Reject stack arguments if tail call reachable

This patch updates the BPF verifier to reject programs that combine tail calls with stack arguments, as tail calls are deprecated. It hooks into the call depth traversal to fail verification if stack arguments are present in a call chain that can reach a tail call.

Quoted 9 lines
kernel/bpf/verifier.c +0 -1
index a363811aec7a0..47df3f40a71cf 100644
@@ -5267,14 +5267,23 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
  	 * this info will be utilized by JIT so that we will be preserving the
  	 * tail call counter throughout bpf2bpf calls combined with tailcalls
  	 */
-	if (tail_call_reachable)
Quoted 8 lines

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5276

Quoted 7 lines
kernel/bpf/verifier.c +2 -0
+	if (tail_call_reachable) {
  		for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
  			if (subprog[tmp].is_exception_cb) {
  				verbose(env, "cannot tail call within exception cb\n");
  				return -EINVAL;
  			}
+			if (subprog[tmp].stack_arg_cnt) {

Can this check be bypassed depending on the order of subprogram calls?

No.

Quoted 10 lines

If a main program calls subprogram A (which uses stack args) and then subprogram B (which has a tail call), it seems subprogram A might be evaluated first when tail_call_reachable is false, bypassing this check.

When subprogram B is evaluated, it sets tail_call_reachable to true, which propagates to the main program.

When the main program finishes, only the main program's stack_arg_cnt is checked, allowing the program to pass verification.

But in this case, main program stack_arg_cnt is not 0, so -EINVAL will return.

Quoted 5 lines

If the calls were reversed, subprogram A would inherit the true state and be correctly rejected.

Does this sequence allow invalid programs to bypass the safety check?

Quoted 7 lines

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5282

Quoted 6 lines
kernel/bpf/verifier.c +4 -0
+				verbose(env, "tail_calls are not allowed in programs with stack args\n");
+				return -EINVAL;
+			}
  			subprog[tmp].tail_call_reachable = true;
  		}
+	} else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) {

Does this fallback check miss exception and async callbacks?

This check does not deal with exception and async callbacks.

Quoted 8 lines

By explicitly checking !idx and subprog[0], it appears to assume the main program is the only call graph root.

Since exception and async callbacks are verified independently as their own root programs (where idx > 0), would a callback that contains a tail call, uses stack arguments, and makes no BPF-to-BPF calls bypass both the primary tail_call_reachable check and this fallback check?

As far as I know, callback and exception prog are not allowing tail call. For example:

                if (subprog[sidx].is_async_cb) {
                        if (subprog[sidx].has_tail_call) {
                                verifier_bug(env, "subprog has tail_call and async cb");
                                return -EFAULT;
                        }
			...
		}

...

        if (tail_call_reachable) {
                for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
                        if (subprog[tmp].is_exception_cb) {
                                verbose(env, "cannot tail call within exception cb\n");
                                return -EINVAL;
                        }
			...
		}
	}
Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +4 -0
+		verbose(env, "tail_calls are not allowed in programs with stack args\n");
+		return -EINVAL;
+	}
+
  	if (subprog[0].tail_call_reachable)
  		env->prog->aux->tail_call_reachable = true;
author reviewer
#31 lore raw

PATCH v4 16/25 bpf: Disable private stack for x86_64 if stack arguments used

In reply to Yonghong Song

Other architectures like arm64, riscv, etc. have enough register and for them private stack can be used together with stack arguments.

---
 kernel/bpf/verifier.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +4 -1
index 47df3f40a71c..745e13cfb54d 100644
@@ -5163,7 +5163,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
 	}
 
 	subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);
-	if (priv_stack_supported) {
+	if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) {
+		/* x86-64 uses R9 for both private stack frame pointer and arg6. */
+		subprog[idx].priv_stack_mode = NO_PRIV_STACK;
+	} else if (priv_stack_supported) {
 		/* Request private stack support only if the subprog stack
 		 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to
 		 * avoid jit penalty if the stack usage is small.
Signature · 1 line
2.53.0-Meta
#32 lore raw

Re: [PATCH bpf-next v4 16/25] bpf: Disable private stack for x86_64 if stack arguments used

This looks like a bug fix for stack argument support. Should this include:
Quoted 11 lines

commit 86c79df525268be6b8e148743cb68e4d7a0a2956

Author: Yonghong Song <[email protected]>
Date:   Tue May 12 21:51:14 2026 -0700

bpf: Disable private stack for x86_64 if stack arguments used

Other architectures like arm64, riscv, etc. have enough register and for them private stack can be used together with stack arguments.

This looks like a bug fix for stack argument support. Should this include:

Fixes cc06010ca1b8 ("bpf: Support stack arguments for bpf functions")

The issue appears to have been introduced when stack arguments support was added, but did not account for the x86_64 specific register conflict where R9 is used for both the private stack frame pointer and arg6.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25779371388

#33 lore raw

Re: [PATCH bpf-next v4 16/25] bpf: Disable private stack for x86_64 if stack arguments used

In reply to Yonghong Song
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Hardcoding architecture-specific JIT register limitations into the generic BPF verifier. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Hardcoding architecture-specific JIT register limitations into the generic BPF verifier. --

commit d388190c12c82279caa212e6edeefb82d873dd00
Author: Yonghong Song <[email protected]>

bpf: Disable private stack for x86_64 if stack arguments used

This commit disables private stack support on x86_64 when stack arguments are used, because x86-64 uses R9 for both the private stack frame pointer and arg6. Other architectures have enough registers to support both.

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5166

Quoted 10 lines
kernel/bpf/verifier.c +1 -1
index 47df3f40a71cf..745e13cfb54d6 100644
@@ -5163,7 +5163,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
 	}
 
 	subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);
-	if (priv_stack_supported) {
+	if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) {

Is it appropriate to hardcode host architecture macros like IS_ENABLED(CONFIG_X86_64) directly into the core verifier logic?

This isn't a bug, but it introduces architecture-specific constraints into the generic BPF verifier. Would it be better to handle this by querying JIT backend capabilities via a generic API (e.g., passing stack argument requirements to a JIT capability checker) so the verifier remains architecture-agnostic?

Quoted 6 lines
kernel/bpf/verifier.c +3 -0
+		/* x86-64 uses R9 for both private stack frame pointer and arg6. */
+		subprog[idx].priv_stack_mode = NO_PRIV_STACK;
+	} else if (priv_stack_supported) {
 		/* Request private stack support only if the subprog stack
 		 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to
 		 * avoid jit penalty if the stack usage is small.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
author reviewer
#34 lore raw

Re: [PATCH bpf-next v4 16/25] bpf: Disable private stack for x86_64 if stack arguments used

Quoted 26 lines · On 5/15/26 7:28 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Hardcoding architecture-specific JIT register limitations into the generic BPF verifier. --

commit d388190c12c82279caa212e6edeefb82d873dd00
Author: Yonghong Song <[email protected]>

bpf: Disable private stack for x86_64 if stack arguments used

This commit disables private stack support on x86_64 when stack arguments are used, because x86-64 uses R9 for both the private stack frame pointer and arg6. Other architectures have enough registers to support both.

kernel/bpf/verifier.c static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, · line 5166

Quoted 10 lines
kernel/bpf/verifier.c +1 -1
index 47df3f40a71cf..745e13cfb54d6 100644
@@ -5163,7 +5163,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
  	}
 
  	subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth);
-	if (priv_stack_supported) {
+	if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) {

Is it appropriate to hardcode host architecture macros like IS_ENABLED(CONFIG_X86_64) directly into the core verifier logic?

Quoted 4 lines

This isn't a bug, but it introduces architecture-specific constraints into the generic BPF verifier. Would it be better to handle this by querying JIT backend capabilities via a generic API (e.g., passing stack argument requirements to a JIT capability checker) so the verifier remains architecture-agnostic?

Disabling private stack support in verifier sounds better. We do have two cases of '#ifdef CONFIG_X86_64' in verifier.

Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +3 -0
+		/* x86-64 uses R9 for both private stack frame pointer and arg6. */
+		subprog[idx].priv_stack_mode = NO_PRIV_STACK;
+	} else if (priv_stack_supported) {
  		/* Request private stack support only if the subprog stack
  		 * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to
  		 * avoid jit penalty if the stack usage is small.
author reviewer
#35 lore raw

PATCH v4 17/25 bpf,x86: Implement JIT support for stack arguments

In reply to Yonghong Song

Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r11 (BPF_REG_PARAMS) in BPF bytecode, which the JIT translates to native code.

The JIT follows the x86-64 calling convention for both BPF-to-BPF and kfunc calls:
- Arg 6 is passed in the R9 register
- Args 7+ are passed on the stack

Incoming arg 6 (BPF r11+8) is translated to a MOV from R9 rather than a memory load. Incoming args 7+ (BPF r11+16, r11+24, ...) map directly to [rbp + 16], [rbp + 24], ..., matching the x86-64 stack layout after CALL + PUSH RBP, so no offset adjustment is needed.

tail_call_reachable is rejected by the verifier and priv_stack is disabled by the JIT when stack args exist, so R9 is always available. When BPF bytecode writes to the arg-6 stack slot (offset -8), the JIT emits a MOV into R9 instead of a memory store. Outgoing args 7+ are placed at [rsp] in a pre-allocated area below callee-saved registers, using:
native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16

The native x86_64 stack layout with stack arguments:

high address

  +-------------------------+
  | incoming stack arg N    |  [rbp + 16 + (N-7)*8]  (from caller)
  | ...                     |
  | incoming stack arg 7    |  [rbp + 16]
  +-------------------------+
  | return address          |  [rbp + 8]
  | saved rbp               |  [rbp]
  +-------------------------+
  | BPF program stack       |  (round_up(stack_depth, 8) bytes)
  +-------------------------+
  | callee-saved regs       |  (r12, rbx, r13, r14, r15 as needed)
  +-------------------------+
  | outgoing arg M          |  [rsp + (M-7)*8]
  | ...                     |
  | outgoing arg 7          |  [rsp]
  +-------------------------+  rsp

low address

---
 arch/x86/net/bpf_jit_comp.c | 149 ++++++++++++++++++++++++++++++++++--
 include/linux/bpf.h         |   1 +
 kernel/bpf/core.c           |  10 +++
 3 files changed, 154 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +143 -6
index ea9e707e8abf..ceefefb4da21 100644
@@ -10,6 +10,7 @@
 #include <linux/if_vlan.h>
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
 #include <asm/extable.h>
@@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
 	*pprog = prog;
 }
 
+/* add rsp, depth */
+static void emit_add_rsp(u8 **pprog, u16 depth)
+{
+	u8 *prog = *pprog;
+
+	if (!depth)
+		return;
+	if (is_imm8(depth))
+		EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
+	else
+		EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
+	*pprog = prog;
+}
+
+/* sub rsp, depth */
+static void emit_sub_rsp(u8 **pprog, u16 depth)
+{
+	u8 *prog = *pprog;
+
+	if (!depth)
+		return;
+	if (is_imm8(depth))
+		EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
+	else
+		EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
+	*pprog = prog;
+}
+
 static void emit_nops(u8 **pprog, int len)
 {
 	u8 *prog = *pprog;
@@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	bool seen_exit = false;
 	u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
 	void __percpu *priv_frame_ptr = NULL;
+	u16 out_stack_arg_cnt, outgoing_rsp;
 	u64 arena_vm_start, user_vm_start;
 	void __percpu *priv_stack_ptr;
 	int i, excnt = 0;
 	int ilen, proglen = 0;
 	u8 *ip, *prog = temp;
 	u32 stack_depth;
+	int callee_saved_size;
+	s32 outgoing_arg_base;
 	int err;
 
 	stack_depth = bpf_prog->aux->stack_depth;
+	out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
 	if (priv_stack_ptr) {
 		priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
 		stack_depth = 0;
 	}
 
+	/*
+	 * Follow x86-64 calling convention for both BPF-to-BPF and
+	 * kfunc calls:
+	 *   - Arg 6 is passed in R9 register
+	 *   - Args 7+ are passed on the stack at [rsp]
+	 *
+	 * Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9).
+	 * Incoming args 7+ are read from [rbp + 16], [rbp + 24], ...
+	 * (BPF r11+16, r11+24, ... map directly with no offset change).
+	 *
+	 * tail_call_reachable is rejected by the verifier and priv_stack
+	 * is disabled by the JIT when stack args exist, so R9 is always
+	 * available.
+	 *
+	 * Stack layout (high to low):
+	 *   [rbp + 16 + ...]    incoming stack args 7+ (from caller)
+	 *   [rbp + 8]           return address
+	 *   [rbp]               saved rbp
+	 *   [rbp - prog_stack]  program stack
+	 *   [below]             callee-saved regs
+	 *   [below]             outgoing args 7+ (= rsp)
+	 */
 	arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
 	user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
@@ -1700,6 +1755,42 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			push_r12(&prog);
 		push_callee_regs(&prog, callee_regs_used);
 	}
+
+	/* Compute callee-saved register area size. */
+	callee_saved_size = 0;
+	if (bpf_prog->aux->exception_boundary || arena_vm_start)
+		callee_saved_size += 8; /* r12 */
+	if (bpf_prog->aux->exception_boundary) {
+		callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
+	} else {
+		int j;
+
+		for (j = 0; j < 4; j++)
+			if (callee_regs_used[j])
+				callee_saved_size += 8;
+	}
+	/*
+	 * Base offset from rbp for translating BPF outgoing args 7+
+	 * to native offsets. BPF uses negative offsets from r11
+	 * (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses
+	 * positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for
+	 * arg8, ...). Arg 6 goes to R9 directly.
+	 *
+	 * The translation reverses direction:
+	 *   native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16
+	 *
+	 * Note that tail_call_reachable is guaranteed to be false when
+	 * stack args exist, so tcc pushes need not be accounted for.
+	 */
+	outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
+
+	/*
+	 * Allocate outgoing stack arg area for args 7+ only.
+	 * Arg 6 goes into r9 register, not on stack.
+	 */
+	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	emit_sub_rsp(&prog, outgoing_rsp);
+
 	if (arena_vm_start)
 		emit_mov_imm64(&prog, X86_REG_R12,
 			       arena_vm_start >> 32, (u32) arena_vm_start);
@@ -1721,7 +1812,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		u8 b2 = 0, b3 = 0;
 		u8 *start_of_ldx;
 		s64 jmp_offset;
-		s16 insn_off;
+		s32 insn_off;
 		u8 jmp_cond;
 		u8 *func;
 		int nops;
@@ -2134,12 +2225,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 				EMIT1(0xC7);
 			goto st;
 		case BPF_ST | BPF_MEM | BPF_DW:
+			if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+				/* Arg 6: store immediate in r9 register */
+				emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
+				break;
+			}
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
 
-st:			if (is_imm8(insn->off))
-				EMIT2(add_1reg(0x40, dst_reg), insn->off);
+st:			insn_off = insn->off;
+			if (dst_reg == BPF_REG_PARAMS) {
+				/*
+				 * Args 7+: reverse BPF negative offsets to
+				 * x86 positive rsp offsets.
+				 * BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
+				 */
+				insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
+				dst_reg = BPF_REG_FP;
+			}
+			if (is_imm8(insn_off))
+				EMIT2(add_1reg(0x40, dst_reg), insn_off);
 			else
-				EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
+				EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
 
 			EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
 			break;
@@ -2149,7 +2255,17 @@ st:			if (is_imm8(insn->off))
 		case BPF_STX | BPF_MEM | BPF_H:
 		case BPF_STX | BPF_MEM | BPF_W:
 		case BPF_STX | BPF_MEM | BPF_DW:
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+			if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+				/* Arg 6: store register value in r9 */
+				EMIT_mov(X86_REG_R9, src_reg);
+				break;
+			}
+			insn_off = insn->off;
+			if (dst_reg == BPF_REG_PARAMS) {
+				insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
+				dst_reg = BPF_REG_FP;
+			}
+			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			break;
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2248,6 +2364,19 @@ st:			if (is_imm8(insn->off))
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
 			insn_off = insn->off;
+			if (src_reg == BPF_REG_PARAMS) {
+				if (insn_off == 8) {
+					/* Incoming arg 6: read from r9 */
+					EMIT_mov(dst_reg, X86_REG_R9);
+					break;
+				}
+				src_reg = BPF_REG_FP;
+				/*
+				 * Incoming args 7+: native_off == bpf_off
+				 * (r11+16 → [rbp+16], r11+24 → [rbp+24], ...)
+				 * No offset adjustment needed.
+				 */
+			}
 
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2736,6 +2865,8 @@ st:			if (is_imm8(insn->off))
 				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
 					return -EINVAL;
 			}
+			/* Deallocate outgoing args 7+ area. */
+			emit_add_rsp(&prog, outgoing_rsp);
 			if (bpf_prog->aux->exception_boundary) {
 				pop_callee_regs(&prog, all_callee_regs_used);
 				pop_r12(&prog);
@@ -3793,7 +3924,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	for (pass = 0; pass < MAX_PASSES || image; pass++) {
 		if (!padding && pass >= PADDING_PASSES)
 			padding = true;
-		proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);
+		proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen,
+				 &ctx, padding);
 		if (proglen <= 0) {
 out_image:
 			image = NULL;
@@ -3910,6 +4042,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_stack_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (text_poke_copy(dst, src, len) == NULL)
include/linux/bpf.h +1 -0
index 14759972f148..40c333484d54 100644
@@ -1548,6 +1548,7 @@ void bpf_jit_uncharge_modmem(u32 size);
 bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
 bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
 				 int insn_idx);
+u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog);
 #else
 static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
 					   struct bpf_trampoline *tr,
kernel/bpf/core.c +10 -0
index e6b836f846eb..427a6d828e01 100644
@@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc
 	insn_idx += prog->aux->subprog_start;
 	return env->insn_aux_data[insn_idx].indirect_target;
 }
+
+u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog)
+{
+	const struct bpf_subprog_info *sub;
+
+	if (!env)
+		return 0;
+	sub = &env->subprog_info[prog->aux->func_idx];
+	return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub);
+}
 #endif /* CONFIG_BPF_JIT */
 
 /* Base function for offset calculation. Needs to go into .text section,
Signature · 1 line
2.53.0-Meta
#36 lore raw

Re: [PATCH bpf-next v4 17/25] bpf,x86: Implement JIT support for stack arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] When a BPF program with an exception boundary calls a function with more than 5 arguments…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] When a BPF program with an exception boundary calls a function with more than 5 arguments, unwinding an exception via bpf_throw() restores callee-saved registers from the wrong stack offset, corrupting kernel registers. --

commit 86dff956386bae1d3e030c84363f5b56f2e023a7
Author: Yonghong Song <[email protected]>

bpf,x86: Implement JIT support for stack arguments

This commit implements x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed on the stack, maintaining compatibility with the x86-64 calling convention. It introduces logic to handle pushing outgoing arguments to the stack area below the callee-saved registers.

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index ea9e707e8abff..ceefefb4da217 100644

[ ... ]

Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3 -0
@@ -1700,6 +1755,42 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			push_r12(&prog);
 		push_callee_regs(&prog, callee_regs_used);
 	}
+
+	/* Compute callee-saved register area size. */
+	callee_saved_size = 0;

[ ... ]

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 1792

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6 -0
+	/*
+	 * Allocate outgoing stack arg area for args 7+ only.
+	 * Arg 6 goes into r9 register, not on stack.
+	 */
+	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	emit_sub_rsp(&prog, outgoing_rsp);

Does allocating the outgoing stack arguments area (outgoing_rsp) after pushing the callee-saved registers cause an issue when unwinding an exception via bpf_throw()?

When bpf_throw() is called, the stack unwinder (arch_bpf_stack_walk()) captures the current stack pointer (sp) of the BPF frame, which points to the bottom of this outgoing_rsp area.

bpf_throw() passes this shifted sp to the exception callback in the rsi register.

The JIT-generated exception callback executes mov rsp, rsi followed by pop_callee_regs(). Because rsp is offset by outgoing_rsp bytes below the actual saved registers, the pop instructions will read from the outgoing arguments area instead of the saved registers.

Could returning to the kernel with corrupted callee-saved registers cause unpredictable behavior or panics?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author reviewer
#37 lore raw

Re: [PATCH bpf-next v4 17/25] bpf,x86: Implement JIT support for stack arguments

Quoted 29 lines · On 5/15/26 8:02 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] When a BPF program with an exception boundary calls a function with more than 5 arguments, unwinding an exception via bpf_throw() restores callee-saved registers from the wrong stack offset, corrupting kernel registers. --

commit 86dff956386bae1d3e030c84363f5b56f2e023a7
Author: Yonghong Song <[email protected]>

bpf,x86: Implement JIT support for stack arguments

This commit implements x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed on the stack, maintaining compatibility with the x86-64 calling convention. It introduces logic to handle pushing outgoing arguments to the stack area below the callee-saved registers.

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index ea9e707e8abff..ceefefb4da217 100644

[ ... ]

Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3 -0
@@ -1700,6 +1755,42 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  			push_r12(&prog);
  		push_callee_regs(&prog, callee_regs_used);
  	}
+
+	/* Compute callee-saved register area size. */
+	callee_saved_size = 0;

[ ... ]

Quoted 9 lines

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 1792

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6 -0
+	/*
+	 * Allocate outgoing stack arg area for args 7+ only.
+	 * Arg 6 goes into r9 register, not on stack.
+	 */
+	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+	emit_sub_rsp(&prog, outgoing_rsp);

Does allocating the outgoing stack arguments area (outgoing_rsp) after pushing the callee-saved registers cause an issue when unwinding an exception via bpf_throw()?

Yes, this is indeed an issue, we need to restore 'rsp' between

        if (is_exception_cb) {
                EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
                EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */

and

                /* The main frame must have exception_boundary as true, so we
                 * first restore those callee-saved regs from stack, before
                 * reusing the stack frame.
                 */
                pop_callee_regs(&prog, all_callee_regs_used);
                pop_r12(&prog);
                /* Reset the stack frame. */
                EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
	}

due to stack argument.

But I would like to know whether we want to support this exception + stack_args combination or not.

Quoted 16 lines

When bpf_throw() is called, the stack unwinder (arch_bpf_stack_walk()) captures the current stack pointer (sp) of the BPF frame, which points to the bottom of this outgoing_rsp area.

bpf_throw() passes this shifted sp to the exception callback in the rsi register.

The JIT-generated exception callback executes mov rsp, rsi followed by pop_callee_regs(). Because rsp is offset by outgoing_rsp bytes below the actual saved registers, the pop instructions will read from the outgoing arguments area instead of the saved registers.

Could returning to the kernel with corrupted callee-saved registers cause unpredictable behavior or panics?

author reviewer
#38 lore raw

PATCH v4 18/25 selftests/bpf: Add tests for BPF function stack arguments

In reply to Yonghong Song

Add selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.

BPF-to-BPF subprog call tests (stack_arg.c):
- Scalar stack args
- Pointer stack args
- Mixed pointer/scalar stack args
- Nested calls
- Dynptr stack arg
- Two callees with different stack arg counts
- Async callback

Kfunc call tests (stack_arg_kfunc.c, with bpf_testmod kfuncs):
- Scalar stack args
- Pointer stack args
- Mixed pointer/scalar stack args
- Dynptr stack arg
- Memory buffer + size pair
- Iterator
- Const string pointer
- Timer pointer

---
 .../selftests/bpf/prog_tests/stack_arg.c      | 139 ++++++++++
 tools/testing/selftests/bpf/progs/stack_arg.c | 252 ++++++++++++++++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 163 +++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  65 +++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  20 +-
 5 files changed, 638 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +139 -0
new file mode 100644
index 000000000000..d61bac33f809
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "stack_arg.skel.h"
+#include "stack_arg_kfunc.skel.h"
+
+static void run_subtest(struct bpf_program *prog, int expected)
+{
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = &pkt_v4,
+		.data_size_in = sizeof(pkt_v4),
+		.repeat = 1,
+	);
+
+	prog_fd = bpf_program__fd(prog);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, expected, "retval");
+}
+
+static void test_global_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_global_many_args, 36);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_async_cb_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+	/* Wait for the timer callback to fire and verify the result.
+	 * 10+20+30+40+50+60+70+80 = 360
+	 */
+	usleep(50);
+	ASSERT_EQ(skel->bss->timer_result, 360, "timer_result");
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_bpf2bpf(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_bpf2bpf_ptr_stack_arg, 45);
+	run_subtest(skel->progs.test_bpf2bpf_mix_stack_args, 51);
+	run_subtest(skel->progs.test_bpf2bpf_nesting_stack_arg, 50);
+	run_subtest(skel->progs.test_bpf2bpf_dynptr_stack_arg, 69);
+	run_subtest(skel->progs.test_two_callees, 91);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_kfunc(void)
+{
+	struct stack_arg_kfunc *skel;
+
+	skel = stack_arg_kfunc__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg_kfunc__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_stack_arg_scalar, 36);
+	run_subtest(skel->progs.test_stack_arg_ptr, 45);
+	run_subtest(skel->progs.test_stack_arg_mix, 51);
+	run_subtest(skel->progs.test_stack_arg_dynptr, 69);
+	run_subtest(skel->progs.test_stack_arg_mem, 151);
+	run_subtest(skel->progs.test_stack_arg_iter, 115);
+	run_subtest(skel->progs.test_stack_arg_const_str, 15);
+	run_subtest(skel->progs.test_stack_arg_timer, 15);
+
+out:
+	stack_arg_kfunc__destroy(skel);
+}
+
+void test_stack_arg(void)
+{
+	if (test__start_subtest("global_many_args"))
+		test_global_many();
+	if (test__start_subtest("async_cb_many_args"))
+		test_async_cb_many();
+	if (test__start_subtest("bpf2bpf"))
+		test_bpf2bpf();
+	if (test__start_subtest("kfunc"))
+		test_kfunc();
+}
tools/testing/selftests/bpf/progs/stack_arg.c new file +252 -0

252 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +163 -0

163 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +65 -0

77 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +19 -1
index aa0b8d41e71b..2c1cb118f886 100644
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc {
 };
 #endif
 
+struct bpf_iter_testmod_seq;
+
 struct prog_test_pass1 {
 	int x0;
 	struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
-
+__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
+			       __u64 e, __u64 f, __u64 g, __u64 h) __ksym;
+__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p, __u64 f,
+				   struct prog_test_pass1 *q) __ksym;
+__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				      struct bpf_dynptr *ptr) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   void *mem, int mem__sz) __ksym;
+__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				    struct bpf_iter_testmod_seq *it__iter) __ksym;
+__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+					 const char *str__str) __ksym;
+__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				     struct bpf_timer *timer) __ksym;
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.53.0-Meta
#39 lore raw

Re: [PATCH bpf-next v4 18/25] selftests/bpf: Add tests for BPF function stack arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The test `test_async_cb_many` uses a hardcoded `usleep(50)` to wait for an asynchronous BPF…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test test_async_cb_many uses a hardcoded usleep(50) to wait for an asynchronous BPF timer to execute, which is prone to race conditions and flakiness on loaded CI systems. --

commit 12812727f17076398fed8cfc802771db232582f8
Author: Yonghong Song <[email protected]>

selftests/bpf: Add tests for BPF function stack arguments

This commit adds selftests for stack argument passing in BPF-to-BPF subprog and kfunc calls with more than 5 arguments. It covers scalar, pointer, mixed, dynptr, memory buffer, iterator, const string, and timer pointers.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +0 -0
new file mode 100644
index 0000000000000..d61bac33f8092

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +2 -0
+static void test_async_cb_many(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/stack_arg.c line 67

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +6 -0
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+	/* Wait for the timer callback to fire and verify the result.
+	 * 10+20+30+40+50+60+70+80 = 360
+	 */
+	usleep(50);

Could this short sleep lead to flaky test failures on loaded CI systems?

Timer callbacks are executed in softirq context, and under load, scheduling delays could easily exceed 50 microseconds. This might cause the test process to wake up and read timer_result before the callback has actually executed.

Would a polling loop with a reasonable timeout be more reliable here?

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +5 -0
+	ASSERT_EQ(skel->bss->timer_result, 360, "timer_result");
+
+out:
+	stack_arg__destroy(skel);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
author reviewer
#40 lore raw

Re: [PATCH bpf-next v4 18/25] selftests/bpf: Add tests for BPF function stack arguments

Quoted 30 lines · On 5/15/26 8:16 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test test_async_cb_many uses a hardcoded usleep(50) to wait for an asynchronous BPF timer to execute, which is prone to race conditions and flakiness on loaded CI systems. --

commit 12812727f17076398fed8cfc802771db232582f8
Author: Yonghong Song <[email protected]>

selftests/bpf: Add tests for BPF function stack arguments

This commit adds selftests for stack argument passing in BPF-to-BPF subprog and kfunc calls with more than 5 arguments. It covers scalar, pointer, mixed, dynptr, memory buffer, iterator, const string, and timer pointers.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +0 -0
new file mode 100644
index 0000000000000..d61bac33f8092

[ ... ]

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +2 -0
+static void test_async_cb_many(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/stack_arg.c line 67

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +6 -0
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+	/* Wait for the timer callback to fire and verify the result.
+	 * 10+20+30+40+50+60+70+80 = 360
+	 */
+	usleep(50);

Could this short sleep lead to flaky test failures on loaded CI systems?

Quoted 5 lines

Timer callbacks are executed in softirq context, and under load, scheduling delays could easily exceed 50 microseconds. This might cause the test process to wake up and read timer_result before the callback has actually executed.

Would a polling loop with a reasonable timeout be more reliable here?

The prog_tests/tiemr.c uses 'usleep(50);' which seems working fine. So I used it too.

Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +5 -0
+	ASSERT_EQ(skel->bss->timer_result, 360, "timer_result");
+
+out:
+	stack_arg__destroy(skel);
+}
author reviewer
#41 lore raw

PATCH v4 19/25 selftests/bpf: Add tests for stack argument validation

In reply to Yonghong Song

Add negative tests that verify the kfunc (rejecting kfunc call with >8 byte struct as stack argument) and the verifier (rejecting invalid uses of r11 for stack arguments).

---
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  10 ++
 .../selftests/bpf/progs/stack_arg_fail.c      | 114 ++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |   7 ++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |   8 ++
 4 files changed, 139 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +10 -0
new file mode 100644
index 000000000000..090af1330953
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_fail.skel.h"
+
+void test_stack_arg_fail(void)
+{
+	RUN_TESTS(stack_arg_fail);
+}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +114 -0

114 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
index aef2f68b7e83..0be918fe3021 100644
@@ -882,6 +882,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e
 	return a + b + c + d + e;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_big_arg s)
+{
+	return a + b + c + d + e + s.a + s.b;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1353,6 +1359,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_acquire, KF_ACQUIRE | KF_RET_NULL)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +8 -0
index 2c1cb118f886..2edc36b66de9 100644
@@ -50,6 +50,11 @@ struct prog_test_pass2 {
 	} x;
 };
 
+struct prog_test_big_arg {
+	__u64 a;
+	__u64 b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u
 					 const char *str__str) __ksym;
 __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				     struct bpf_timer *timer) __ksym;
+__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_big_arg s) __ksym;
+
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.53.0-Meta
author reviewer
#42 lore raw

PATCH v4 20/25 selftests/bpf: Add BTF fixup for __naked subprog parameter names

In reply to Yonghong Song

When __naked subprogs are used in verifier tests, clang drops parameter names from their BTF FUNC_PROTO entries. This prevents the verifier from resolving stack argument slots by name.

Add a __btf_func_path(path) annotation that points to a separate BTF file containing properly-named FUNC entries. The test_loader matches FUNC entries by name, detects anonymous parameters, and replaces the FUNC_PROTO with a new one that carries parameter names from the custom file while preserving the original type IDs.

The custom BTF file also serves as btf_custom_path for kfunc resolution when no separate btf_custom_path is specified.

---
 tools/testing/selftests/bpf/progs/bpf_misc.h |   1 +
 tools/testing/selftests/bpf/test_loader.c    | 136 ++++++++++++++++++-
 2 files changed, 136 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/bpf_misc.h +1 -0
index a0d7b15a24b1..9eeb5b0b63d6 100644
@@ -152,6 +152,7 @@
 #define __auxiliary		__test_tag("test_auxiliary")
 #define __auxiliary_unpriv	__test_tag("test_auxiliary_unpriv")
 #define __btf_path(path)	__test_tag("test_btf_path=" path)
+#define __btf_func_path(path)	__test_tag("test_btf_func_path=" path)
 #define __arch(arch)		__test_tag("test_arch=" arch)
 #define __arch_x86_64		__arch("X86_64")
 #define __arch_arm64		__arch("ARM64")
tools/testing/selftests/bpf/test_loader.c +135 -1

166 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#43 lore raw

PATCH v4 21/25 selftests/bpf: Add verifier tests for stack argument validation

In reply to Yonghong Song

Add inline-asm based verifier tests that exercise stack argument validation logic directly.

Positive tests:
- subprog call with 6 arg's
- Two sequential calls to different subprogs (6-arg and 7-arg)
- Share a r11 store for both branches

Negative tests — verifier rejection:
- Read from uninitialized incoming stack arg slot
- Gap in outgoing slots: only r11-16 written, r11-8 missing
- Write at r11-80, exceeding max 7 stack args
- Missing store on one branch with a shared store
- First call has proper stack arguments and the second call intends to inherit stack arguments but not working
- r11 load ordering issue

Negative tests — pointer/ref tracking:
- Pruning type mismatch: one branch stores PTR_TO_STACK, the other stores a scalar, callee dereferences — must not prune
- Release invalidation: bpf_sk_release invalidates a socket pointer stored in a stack arg slot
- Packet pointer invalidation: bpf_skb_pull_data invalidates a packet pointer stored in a stack arg slot
- Null propagation: PTR_TO_MAP_VALUE_OR_NULL stored in stack arg slot, null branch attempts dereference via callee

---
 .../selftests/bpf/prog_tests/verifier.c       |   4 +
 .../bpf/progs/btf__verifier_stack_arg_order.c |  40 ++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 444 ++++++++++++++++++
 .../bpf/progs/verifier_stack_arg_order.c      | 126 +++++
 4 files changed, 614 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
tools/testing/selftests/bpf/prog_tests/verifier.c +4 -0
index a96b25ebff23..ee3d929fac8a 100644
@@ -91,6 +91,8 @@
 #include "verifier_sockmap_mutate.skel.h"
 #include "verifier_spill_fill.skel.h"
 #include "verifier_spin_lock.skel.h"
+#include "verifier_stack_arg.skel.h"
+#include "verifier_stack_arg_order.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
 #include "verifier_subprog_precision.skel.h"
@@ -238,6 +240,8 @@ void test_verifier_sock_addr(void)            { RUN(verifier_sock_addr); }
 void test_verifier_sockmap_mutate(void)       { RUN(verifier_sockmap_mutate); }
 void test_verifier_spill_fill(void)           { RUN(verifier_spill_fill); }
 void test_verifier_spin_lock(void)            { RUN(verifier_spin_lock); }
+void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
+void test_verifier_stack_arg_order(void)      { RUN(verifier_stack_arg_order); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c new file +40 -0
new file mode 100644
index 000000000000..83692570d5bc
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+int subprog_bad_order_6args(int a, int b, int c, int d, int e, int f)
+{
+	return a + b + c + d + e + f;
+}
+
+int subprog_call_before_load_6args(int a, int b, int c, int d, int e, int f)
+{
+	return a + b + c + d + e + f;
+}
+
+int subprog_pruning_call_before_load_6args(int a, int b, int c, int d, int e, int f)
+{
+	return a + b + c + d + e + f;
+}
+
+#else
+
+int subprog_bad_order_6args(void)
+{
+	return 0;
+}
+
+int subprog_call_before_load_6args(void)
+{
+	return 0;
+}
+
+int subprog_pruning_call_before_load_6args(void)
+{
+	return 0;
+}
+
+#endif
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +444 -0

444 lines not on this page · raw message

tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c new file +126 -0

126 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#44 lore raw

PATCH v4 22/25 selftests/bpf: Add precision backtracking test for stack arguments

In reply to Yonghong Song

Add a test that verifies precision backtracking works correctly across BPF-to-BPF calls when stack arguments are involved.

The test passes a size value as incoming stack arg (arg6) to a subprog, which forwards it as the mem__sz parameter (outgoing arg7) to bpf_kfunc_call_stack_arg_mem. The expected __msg annotations verify that precision propagates from the kfunc's mem__sz argument back through the subprog frame to the caller's outgoing stack arg store.

A companion BTF file (btf__stack_arg_precision.c) provides named parameter BTF for the __naked subprog via __btf_func_path.

---
 .../bpf/prog_tests/stack_arg_precision.c      |  10 ++
 .../bpf/progs/btf__stack_arg_precision.c      |  23 +++
 .../selftests/bpf/progs/stack_arg_precision.c | 134 ++++++++++++++++++
 3 files changed, 167 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_precision.c
tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c new file +10 -0
new file mode 100644
index 000000000000..1ab041d66de3
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_precision.skel.h"
+
+void test_stack_arg_precision(void)
+{
+	RUN_TESTS(stack_arg_precision);
+}
tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c new file +23 -0
new file mode 100644
index 000000000000..296fddfe6804
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long size)
+{
+	char buf[8] = {};
+
+	return bpf_kfunc_call_stack_arg_mem(a, b, c, d, e, buf, size);
+}
+
+#else
+
+long subprog_call_mem_kfunc(void)
+{
+	return 0;
+}
+
+#endif
tools/testing/selftests/bpf/progs/stack_arg_precision.c new file +134 -0

134 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#45 lore raw

PATCH v4 23/25 bpf, arm64: Map BPF_REG_0 to x8 instead of x7

In reply to Yonghong Song

From: Puranjay Mohan <[email protected]>

Move the BPF return value register from x7 to x8, freeing x7 for use as an argument register. AAPCS64 designates x8 as the indirect result location register; it is caller-saved and not used for argument passing, making it a suitable home for BPF_REG_0.

This is a prerequisite for stack argument support, which needs x5-x7 to pass arguments 6-8 to native kfuncs following the AAPCS64 calling convention.

---
 arch/arm64/net/bpf_jit_comp.c                          |  4 ++--
 arch/arm64/net/bpf_timed_may_goto.S                    |  8 ++++----
 .../testing/selftests/bpf/progs/verifier_jit_inline.c  |  2 +-
 tools/testing/selftests/bpf/progs/verifier_ldsx.c      |  6 +++---
 .../selftests/bpf/progs/verifier_private_stack.c       | 10 +++++-----
 5 files changed, 15 insertions(+), 15 deletions(-)
arch/arm64/net/bpf_jit_comp.c +2 -2
index 0816c40fc7af..085e650662e3 100644
@@ -47,7 +47,7 @@
 /* Map BPF registers to A64 registers */
 static const int bpf2a64[] = {
 	/* return value from in-kernel function, and exit value from eBPF */
-	[BPF_REG_0] = A64_R(7),
+	[BPF_REG_0] = A64_R(8),
 	/* arguments from eBPF program to in-kernel function */
 	[BPF_REG_1] = A64_R(0),
 	[BPF_REG_2] = A64_R(1),
@@ -1048,7 +1048,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
 	/* Restore FP/LR registers */
 	emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
 
-	/* Move the return value from bpf:r0 (aka x7) to x0 */
+	/* Move the return value from bpf:r0 (aka x8) to x0 */
 	emit(A64_MOV(1, A64_R(0), r0), ctx);
 
 	/* Authenticate lr */
arch/arm64/net/bpf_timed_may_goto.S +4 -4
index 894cfcd7b241..a9a802711a7f 100644
@@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
 	stp     x29, x30, [sp, #-64]!
 	mov     x29, sp
 
-	/* Save BPF registers R0 - R5 (x7, x0-x4)*/
-	stp	x7, x0, [sp, #16]
+	/* Save BPF registers R0 - R5 (x8, x0-x4)*/
+	stp	x8, x0, [sp, #16]
 	stp	x1, x2, [sp, #32]
 	stp	x3, x4, [sp, #48]
 
@@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
 	/* BPF_REG_AX(x9) will be stored into count, so move return value to it. */
 	mov	x9, x0
 
-	/* Restore BPF registers R0 - R5 (x7, x0-x4) */
-	ldp	x7, x0, [sp, #16]
+	/* Restore BPF registers R0 - R5 (x8, x0-x4) */
+	ldp	x8, x0, [sp, #16]
 	ldp	x1, x2, [sp, #32]
 	ldp	x3, x4, [sp, #48]
 
tools/testing/selftests/bpf/progs/verifier_jit_inline.c +1 -1
index 4ea254063646..885ff69a3a62 100644
@@ -9,7 +9,7 @@ __success __retval(0)
 __arch_x86_64
 __jited("	addq	%gs:{{.*}}, %rax")
 __arch_arm64
-__jited("	mrs	x7, SP_EL0")
+__jited("	mrs	x8, SP_EL0")
 int inline_bpf_get_current_task(void)
 {
 	bpf_get_current_task();
tools/testing/selftests/bpf/progs/verifier_ldsx.c +3 -3
index 1026524a1983..41340877dc9d 100644
@@ -274,11 +274,11 @@ __jited("movslq	0x10(%rdi,%r12), %r15")
 __jited("movswq	0x18(%rdi,%r12), %r15")
 __jited("movsbq	0x20(%rdi,%r12), %r15")
 __arch_arm64
-__jited("add	x11, x7, x28")
+__jited("add	x11, x8, x28")
 __jited("ldrsw	x21, [x11, #0x10]")
-__jited("add	x11, x7, x28")
+__jited("add	x11, x8, x28")
 __jited("ldrsh	x21, [x11, #0x18]")
-__jited("add	x11, x7, x28")
+__jited("add	x11, x8, x28")
 __jited("ldrsb	x21, [x11, #0x20]")
 __jited("add	x11, x0, x28")
 __jited("ldrsw	x22, [x11, #0x10]")
tools/testing/selftests/bpf/progs/verifier_private_stack.c +5 -5
index 646e8ef82051..c5078face38d 100644
@@ -170,12 +170,12 @@ __jited("	mrs	x10, TPIDR_EL{{[0-1]}}")
 __jited("	add	x27, x27, x10")
 __jited("	add	x25, x27, {{.*}}")
 __jited("	bl	0x{{.*}}")
-__jited("	mov	x7, x0")
+__jited("	mov	x8, x0")
 __jited("	mov	x0, #0x2a")
 __jited("	str	x0, [x27]")
 __jited("	bl	0x{{.*}}")
-__jited("	mov	x7, x0")
-__jited("	mov	x7, #0x0")
+__jited("	mov	x8, x0")
+__jited("	mov	x8, #0x0")
 __jited("	ldp	x25, x27, [sp], {{.*}}")
 __naked void private_stack_callback(void)
 {
@@ -220,7 +220,7 @@ __jited("	mov	x0, #0x2a")
 __jited("	str	x0, [x27]")
 __jited("	mov	x0, #0x0")
 __jited("	bl	0x{{.*}}")
-__jited("	mov	x7, x0")
+__jited("	mov	x8, x0")
 __jited("	ldp	x27, x28, [sp], #0x10")
 int private_stack_exception_main_prog(void)
 {
@@ -258,7 +258,7 @@ __jited("	add	x25, x27, {{.*}}")
 __jited("	mov	x0, #0x2a")
 __jited("	str	x0, [x27]")
 __jited("	bl	0x{{.*}}")
-__jited("	mov	x7, x0")
+__jited("	mov	x8, x0")
 __jited("	ldp	x27, x28, [sp], #0x10")
 int private_stack_exception_sub_prog(void)
 {
Signature · 1 line
2.53.0-Meta
author reviewer
#46 lore raw

PATCH v4 24/25 bpf, arm64: Add JIT support for stack arguments

In reply to Yonghong Song

From: Puranjay Mohan <[email protected]>

Implement stack argument passing for BPF-to-BPF and kfunc calls with more than 5 parameters on arm64, following the AAPCS64 calling convention.

BPF R1-R5 already map to x0-x4. With BPF_REG_0 moved to x8 by the previous commit, x5-x7 are free for arguments 6-8. Arguments 9-12 spill onto the stack at [SP+0], [SP+8], ... and the callee reads them from [FP+16], [FP+24], ... (above the saved FP/LR pair).

BPF convention uses fixed offsets from BPF_REG_PARAMS (r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.

---
 arch/arm64/net/bpf_jit_comp.c | 88 ++++++++++++++++++++++++++++++++++-
 1 file changed, 87 insertions(+), 1 deletion(-)
arch/arm64/net/bpf_jit_comp.c +87 -1
index 085e650662e3..e3bbeaa94590 100644
@@ -86,6 +86,7 @@ struct jit_ctx {
 	__le32 *image;
 	__le32 *ro_image;
 	u32 stack_size;
+	u16 stack_arg_size;
 	u64 user_vm_start;
 	u64 arena_vm_start;
 	bool fp_used;
@@ -533,13 +534,19 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	 *                        |     |
 	 *                        +-----+ <= (BPF_FP - prog->aux->stack_depth)
 	 *                        |RSVD | padding
-	 * current A64_SP =>      +-----+ <= (BPF_FP - ctx->stack_size)
+	 *                        +-----+ <= (BPF_FP - ctx->stack_size)
+	 *                        |     |
+	 *                        | ... | outgoing stack args (9+, if any)
+	 *                        |     |
+	 * current A64_SP =>      +-----+
 	 *                        |     |
 	 *                        | ... | Function call stack
 	 *                        |     |
 	 *                        +-----+
 	 *                          low
 	 *
+	 * Stack args 6-8 are passed in x5-x7, args 9+ at [SP].
+	 * Incoming args 9+ are at [FP + 16], [FP + 24], ...
 	 */
 
 	emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
@@ -613,6 +620,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
 	if (ctx->stack_size && !ctx->priv_sp_used)
 		emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
 
+	if (ctx->stack_arg_size)
+		emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
+
 	if (ctx->arena_vm_start)
 		emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx);
 
@@ -673,6 +683,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)
 	/* Update tail_call_cnt if the slot is populated. */
 	emit(A64_STR64I(tcc, ptr, 0), ctx);
 
+	if (ctx->stack_arg_size)
+		emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
+
 	/* restore SP */
 	if (ctx->stack_size && !ctx->priv_sp_used)
 		emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1034,6 +1047,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
 	const u8 r0 = bpf2a64[BPF_REG_0];
 	const u8 ptr = bpf2a64[TCCNT_PTR];
 
+	if (ctx->stack_arg_size)
+		emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
+
 	/* We're done with BPF stack */
 	if (ctx->stack_size && !ctx->priv_sp_used)
 		emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,
 	return 0;
 }
 
+static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
+
+#define NR_STACK_ARG_REGS	ARRAY_SIZE(stack_arg_reg)
+
+static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
+{
+	int idx = bpf_off / sizeof(u64) - 1;
+
+	if (idx < NR_STACK_ARG_REGS)
+		emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);
+	else
+		emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx);
+}
+
+static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx)
+{
+	int idx = -bpf_off / sizeof(u64) - 1;
+
+	if (idx < NR_STACK_ARG_REGS)
+		emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx);
+	else
+		emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
+}
+
+static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx)
+{
+	int idx = -bpf_off / sizeof(u64) - 1;
+
+	emit_a64_mov_i(1, tmp, imm, ctx);
+	if (idx < NR_STACK_ARG_REGS)
+		emit(A64_MOV(1, stack_arg_reg[idx], tmp), ctx);
+	else
+		emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
+}
+
 /* JITs an eBPF instruction.
  * Returns:
  * 0  - successfully JITed an 8-byte eBPF instruction.
@@ -1646,6 +1697,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	case BPF_LDX | BPF_MEM | BPF_H:
 	case BPF_LDX | BPF_MEM | BPF_B:
 	case BPF_LDX | BPF_MEM | BPF_DW:
+		if (insn->src_reg == BPF_REG_PARAMS) {
+			emit_stack_arg_load(dst, off, ctx);
+			break;
+		}
+		fallthrough;
 	case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
 	case BPF_LDX | BPF_PROBE_MEM | BPF_W:
 	case BPF_LDX | BPF_PROBE_MEM | BPF_H:
@@ -1672,6 +1728,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (src == fp) {
 			src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
 			off_adj = off + ctx->stack_size;
+			if (!ctx->priv_sp_used)
+				off_adj += ctx->stack_arg_size;
 		} else {
 			src_adj = src;
 			off_adj = off;
@@ -1752,6 +1810,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	case BPF_ST | BPF_MEM | BPF_H:
 	case BPF_ST | BPF_MEM | BPF_B:
 	case BPF_ST | BPF_MEM | BPF_DW:
+		if (insn->dst_reg == BPF_REG_PARAMS) {
+			emit_stack_arg_store_imm(imm, off, tmp, ctx);
+			break;
+		}
+		fallthrough;
 	case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
 	case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
 	case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
@@ -1763,6 +1826,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (dst == fp) {
 			dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
 			off_adj = off + ctx->stack_size;
+			if (!ctx->priv_sp_used)
+				off_adj += ctx->stack_arg_size;
 		} else {
 			dst_adj = dst;
 			off_adj = off;
@@ -1814,6 +1879,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	case BPF_STX | BPF_MEM | BPF_H:
 	case BPF_STX | BPF_MEM | BPF_B:
 	case BPF_STX | BPF_MEM | BPF_DW:
+		if (insn->dst_reg == BPF_REG_PARAMS) {
+			emit_stack_arg_store(src, off, ctx);
+			break;
+		}
+		fallthrough;
 	case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
 	case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
 	case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
@@ -1825,6 +1895,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (dst == fp) {
 			dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
 			off_adj = off + ctx->stack_size;
+			if (!ctx->priv_sp_used)
+				off_adj += ctx->stack_arg_size;
 		} else {
 			dst_adj = dst;
 			off_adj = off;
@@ -2018,6 +2090,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	u8 *ro_image_ptr;
 	int body_idx;
 	int exentry_idx;
+	int out_cnt;
 
 	if (!prog->jit_requested)
 		return prog;
@@ -2065,6 +2138,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
 
+	out_cnt = bpf_out_stack_arg_cnt(env, prog);
+	if (out_cnt) {
+		int nr_on_stack = out_cnt - NR_STACK_ARG_REGS;
+
+		if (nr_on_stack > 0)
+			ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16);
+	}
+
 	if (priv_stack_ptr)
 		ctx.priv_sp_used = true;
 
@@ -2229,6 +2310,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_stack_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (!aarch64_insn_copy(dst, src, len))
Signature · 1 line
2.53.0-Meta
#47 lore raw

Re: [PATCH bpf-next v4 24/25] bpf, arm64: Add JIT support for stack arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Incoming stack arguments 6-8 are mapped to caller-saved registers `x5-x7` without being…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Incoming stack arguments 6-8 are mapped to caller-saved registers x5-x7 without being spilled to the stack, allowing them to be corrupted by function calls. This bypasses verifier bounds checks and enables arbitrary kernel memory read/write. --

commit 20fea8a10865fdb2efc7f0e237022c9e2e575fd7
Author: Puranjay Mohan <[email protected]>

bpf, arm64: Add JIT support for stack arguments

Implement stack argument passing for BPF-to-BPF and kfunc calls with more than 5 parameters on arm64, following the AAPCS64 calling convention.

Quoted 4 lines

BPF convention uses fixed offsets from BPF_REG_PARAMS (r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.

While it is true that the verifier invalidates outgoing stack arg slots after each call, does this logic safely apply to incoming stack arguments?

Incoming arguments are modeled as memory accesses relative to BPF_REG_PARAMS and the verifier assumes these memory slots are preserved across helper and subprog calls.

Since x5-x7 are caller-saved registers per AAPCS64, what happens if a BPF program makes a call (which clobbers x5-x7) and subsequently reads its incoming stack arguments 6-8?

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 085e650662e3d..e3bbeaa94590c 100644

[ ... ]

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,
 	return 0;
 }
 

arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1219

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
+
+#define NR_STACK_ARG_REGS	ARRAY_SIZE(stack_arg_reg)
+
+static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
+{
+	int idx = bpf_off / sizeof(u64) - 1;
+
+	if (idx < NR_STACK_ARG_REGS)
+		emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);

Could emit_stack_arg_load() read corrupted values if the BPF program reads an incoming argument after making a helper or subprog call that clobbered the x5-x7 registers?

If LLVM treats the incoming arguments as preserved memory and doesn't spill them locally, a mismatch could occur where the verifier applies its bounds tracking to the original argument but the program actually reads a clobbered value, potentially leading to arbitrary memory reads or writes.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
author reviewer
#48 lore raw

Re: [PATCH bpf-next v4 24/25] bpf, arm64: Add JIT support for stack arguments

Quoted 20 lines · On 5/15/26 10:20 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Incoming stack arguments 6-8 are mapped to caller-saved registers x5-x7 without being spilled to the stack, allowing them to be corrupted by function calls. This bypasses verifier bounds checks and enables arbitrary kernel memory read/write. --

commit 20fea8a10865fdb2efc7f0e237022c9e2e575fd7
Author: Puranjay Mohan <[email protected]>

bpf, arm64: Add JIT support for stack arguments

Implement stack argument passing for BPF-to-BPF and kfunc calls with more than 5 parameters on arm64, following the AAPCS64 calling convention.

Quoted 4 lines

BPF convention uses fixed offsets from BPF_REG_PARAMS (r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.

While it is true that the verifier invalidates outgoing stack arg slots after each call, does this logic safely apply to incoming stack arguments?

Quoted 7 lines

Incoming arguments are modeled as memory accesses relative to BPF_REG_PARAMS and the verifier assumes these memory slots are preserved across helper and subprog calls.

Since x5-x7 are caller-saved registers per AAPCS64, what happens if a BPF program makes a call (which clobbers x5-x7) and subsequently reads its incoming stack arguments 6-8?

Verifier will reject this. The call will clear all stack arg slots, so subsequent read of incoming stack arguments will be rejected by verifier.

Quoted 11 lines
Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 085e650662e3d..e3bbeaa94590c 100644

[ ... ]

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,
  	return 0;
  }
 
Quoted 13 lines

arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1219

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
+
+#define NR_STACK_ARG_REGS	ARRAY_SIZE(stack_arg_reg)
+
+static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
+{
+	int idx = bpf_off / sizeof(u64) - 1;
+
+	if (idx < NR_STACK_ARG_REGS)
+		emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);

Could emit_stack_arg_load() read corrupted values if the BPF program reads an incoming argument after making a helper or subprog call that clobbered the x5-x7 registers?

Verifier will reject this as load (read) cannot be behind helper/subprog-call/kfunc and store (write).

Quoted 6 lines

If LLVM treats the incoming arguments as preserved memory and doesn't spill them locally, a mismatch could occur where the verifier applies its bounds tracking to the original argument but the program actually reads a clobbered value, potentially leading to arbitrary memory reads or writes.

author reviewer
#49 lore raw

PATCH v4 25/25 selftests/bpf: Enable stack argument tests for arm64

In reply to Yonghong Song

From: Puranjay Mohan <[email protected]>

Now that arm64 supports stack arguments, enable the existing stack_arg, stack_arg_kfunc and verifier_stack_arg tests for __TARGET_ARCH_arm64.

---
 tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c   | 3 ++-
 .../selftests/bpf/progs/btf__verifier_stack_arg_order.c        | 3 ++-
 tools/testing/selftests/bpf/progs/stack_arg.c                  | 3 ++-
 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c            | 3 ++-
 tools/testing/selftests/bpf/progs/stack_arg_precision.c        | 3 ++-
 tools/testing/selftests/bpf/progs/verifier_stack_arg.c         | 3 ++-
 tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c   | 3 ++-
 7 files changed, 14 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c +2 -1
index 296fddfe6804..8d38aafe66a2 100644
@@ -4,7 +4,8 @@
 #include <bpf/bpf_helpers.h>
 #include "../test_kmods/bpf_testmod_kfunc.h"
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long size)
 {
tools/testing/selftests/bpf/progs/btf__verifier_stack_arg_order.c +2 -1
index 83692570d5bc..da34e8456b6c 100644
@@ -3,7 +3,8 @@
 #include <vmlinux.h>
 #include <bpf/bpf_helpers.h>
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 int subprog_bad_order_6args(int a, int b, int c, int d, int e, int f)
 {
tools/testing/selftests/bpf/progs/stack_arg.c +2 -1
index ab6240b997c5..b5e9929a4d63 100644
@@ -21,7 +21,8 @@ struct {
 
 int timer_result;
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 const volatile bool has_stack_arg = true;
 
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +2 -1
index fa9def876ea5..da0d4f91d273 100644
@@ -6,7 +6,8 @@
 #include "bpf_kfuncs.h"
 #include "../test_kmods/bpf_testmod_kfunc.h"
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 const volatile bool has_stack_arg = true;
 
tools/testing/selftests/bpf/progs/stack_arg_precision.c +2 -1
index 2a0a344c83ca..bee2eeec021d 100644
@@ -6,7 +6,8 @@
 #include "../test_kmods/bpf_testmod_kfunc.h"
 #include "bpf_misc.h"
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 /* Force kfunc extern BTF generation for inline asm call below.
  * Uses its own SEC so it's not included as a .text subprog.
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +2 -1
index 6587bf912bc0..d43a9b42034c 100644
@@ -12,7 +12,8 @@ struct {
 	__type(value, long long);
 } map_hash_8b SEC(".maps");
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 __noinline __used
 static int subprog_6args(int a, int b, int c, int d, int e, int f)
tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +2 -1
index 938f4a2f5482..1240cf8a40d6 100644
@@ -5,7 +5,8 @@
 #include <bpf/bpf_helpers.h>
 #include "bpf_misc.h"
 
-#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \
+	defined(__BPF_FEATURE_STACK_ARGUMENT)
 
 __noinline __used __naked
 static int subprog_bad_order_6args(int a, int b, int c, int d, int e, int f)
Signature · 1 line
2.53.0-Meta
#50 lore raw
In reply to Yonghong Song
Quoted 34 lines · Yonghong Song

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS), to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The current maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

All kfunc/bpf-function arguments are caller saved, including stack arguments. For register arguments (r1-r5), the verifier already marks them as clobbered after each call. For stack arguments, the verifier invalidates all outgoing stack arg slots immediately after a call, requiring the compiler to re-store them before any subsequent call. This follows the native calling convention where all function parameters are caller saved.

The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.

Quoted 8 lines

Global subprogs and freplace progs with >5 args are not yet supported. Only x86_64 and arm64 are supported for now. Same selftests are tested by both x86_64 and arm64. Please see each individual patch for details.

[1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:
v3 -> v4:

Applied, but see veristat failures:

|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_cu_mem_address_reserve_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_cu_mem_create_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_malloc_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_mtia_allocate_exit
|success -> failure (!!)|-83.67 %

I couldn't see what might have caused it. Hopefully something minor. Pls follow up.

author reviewer
#51 lore raw
Quoted 53 lines · Alexei Starovoitov
Quoted 33 lines · Yonghong Song

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS), to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The current maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

All kfunc/bpf-function arguments are caller saved, including stack arguments. For register arguments (r1-r5), the verifier already marks them as clobbered after each call. For stack arguments, the verifier invalidates all outgoing stack arg slots immediately after a call, requiring the compiler to re-store them before any subsequent call. This follows the native calling convention where all function parameters are caller saved.

The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.

Quoted 8 lines

Global subprogs and freplace progs with >5 args are not yet supported. Only x86_64 and arm64 are supported for now. Same selftests are tested by both x86_64 and arm64. Please see each individual patch for details.

   [1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:

   v3 -> v4:

Applied, but see veristat failures:

|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_cu_mem_address_reserve_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_cu_mem_create_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_malloc_ret
|success -> failure (!!)|-100.00 % |
|strobelight-server-monitors-gpu_mem_snapshot-gpu_mem_snapshot_bpf-gpu_mem_snapshot_bpf.o|handle_mtia_allocate_exit
|success -> failure (!!)|-83.67 %
Quoted 2 lines

I couldn't see what might have caused it. Hopefully something minor. Pls follow up.

I found the reason, for gpu_mem_snapshot/bpf/gpu_mem_snapshot.bpf.c, the source code:

static int alloc_exit(

    struct pt_regs* ctx,
    uint8_t alloc_type,
    uint64_t direct_addr,
    uint16_t extra_arg_cnt,
    uint64_t extra_arg0,
    uint64_t extra_arg1) {

uint64_t pid_tgid = bpf_get_current_pid_tgid();
...
} // 6 arguments

SEC("uretprobe")
int BPF_KPROBE(handle_malloc_ret) {
return alloc_exit(ctx, STROBELIGHT_GPU_MEM_ALLOC_SAMPLE, 0, 0, 0, 0);
}

But actually, the 'struct pt_regs* ctx' argument is a dead argument and llvm compiler removed it. So for final code, the alloc_exit() has 5 arguments and this is what bpf backend see.

Note that function call signature still has 6 arguments.

I am using llvm21 to test which is CI current using.

Without kernel stack argument support
=====================================

we have

btf_check_func_arg_match()
btf_prepare_func_args()

     ...
     nargs = btf_type_vlen(t); // nargs = 6
     if (nargs > MAX_BPF_FUNC_REG_ARGS) {
        if (!is_global)
             return -EINVAL;
        bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
             tname, nargs, MAX_BPF_FUNC_REG_ARGS);
        return -EINVAL;
     }

So bpf_prepare_func_args() return -EINVAL. Eventually error code propagated back to check_func_call:

        err = btf_check_subprog_call(env, subprog, caller->regs);
        if (err == -EFAULT)
                return err;
        ...

Since the return value is not -EFAULT, verification continues although btf type does not agree with the actual parameter number.

With kernel stack argument support
==================================

we have

btf_check_func_arg_match()
btf_prepare_func_args()

    ...
    nargs = btf_type_vlen(t); // nargs = 6
    ... // all arguments validated properly

return 0

// error message: callee expects 6 args, stack arg1 is not initialized

This is expected since optimized function signature only has 5 parameters in llvm bpf backend.

How to resolve this issue
=========================

I have merged a patch internally with the following manual change:

static int alloc_exit(
- struct pt_regs* ctx,

     uint8_t alloc_type,
     uint64_t direct_addr,
     uint16_t extra_arg_cnt,

@@ -728,7 +727,7 @@

SEC("uretprobe")
int BPF_KPROBE(handle_malloc_ret) {
- return alloc_exit(ctx, STROBELIGHT_GPU_MEM_ALLOC_SAMPLE, 0, 0, 0, 0);
+ return alloc_exit(STROBELIGHT_GPU_MEM_ALLOC_SAMPLE, 0, 0, 0, 0); }

SEC("uprobe")
@@ -768,7 +767,7 @@

SEC("uretprobe")
int BPF_KPROBE(handle_cu_mem_create_ret) {
- return alloc_exit(ctx, STROBELIGHT_CU_MEM_CREATE_SAMPLE, 0, 0, 0, 0);
+ return alloc_exit(STROBELIGHT_CU_MEM_CREATE_SAMPLE, 0, 0, 0, 0); }

...

This should fix the issue and next CI gets meta bpf progs and the error should be gone.

To really fix this issue, we should encode btf with true signatures, i.e., btf should match the actual function parameters. This should be done in llvm.

#52 lore raw
Quoted 4 lines · Yonghong Song

To really fix this issue, we should encode btf with true signatures, i.e., btf should match the actual function parameters. This should be done in llvm.

Yeah. That's a real issue and llvm fix is must-have.
I recall there were many other cases where funcs with 6 args collapsed into 5. Like katran and others. We have them in selftests.
I suspect they're passing only because they're static functions instead of global.

author reviewer
#53 lore raw
Quoted 8 lines · Alexei Starovoitov
Quoted 3 lines · Yonghong Song

To really fix this issue, we should encode btf with true signatures, i.e., btf should match the actual function parameters. This should be done in llvm.

Yeah. That's a real issue and llvm fix is must-have.
I recall there were many other cases where funcs with 6 args collapsed into 5. Like katran and others. We have them in selftests.
I suspect they're passing only because they're static functions instead of global.

Ack. Will work on this. This should help compiler-generated vmlinux BTF having true signatures too.

#54 lore raw
In reply to Yonghong Song
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 8 lines · On Tue, 12 May 2026 21:49:49 -0700 you wrote:

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

[...]

Here is the summary with links:
- [bpf-next,v4,01/25] bpf: Convert bpf_get_spilled_reg macro to static inline function

    https://git.kernel.org/bpf/bpf-next/c/ede2dc5c6b57
  - [bpf-next,v4,02/25] bpf: Remove copy_register_state wrapper function
    https://git.kernel.org/bpf/bpf-next/c/1cb229a54af1
  - [bpf-next,v4,03/25] bpf: Add helper functions for r11-based stack argument insns
    https://git.kernel.org/bpf/bpf-next/c/78bbe61632f1
  - [bpf-next,v4,04/25] bpf: Set sub->arg_cnt earlier in btf_prepare_func_args()
    https://git.kernel.org/bpf/bpf-next/c/3ab5bd317ee2
  - [bpf-next,v4,05/25] bpf: Support stack arguments for bpf functions
    https://git.kernel.org/bpf/bpf-next/c/0f6bd5e7a804
  - [bpf-next,v4,06/25] bpf: Refactor jmp history to use dedicated spi/frame fields
    https://git.kernel.org/bpf/bpf-next/c/3a656670fd6d
  - [bpf-next,v4,07/25] bpf: Add precision marking and backtracking for stack argument slots
    https://git.kernel.org/bpf/bpf-next/c/0a0fdc64b68c
  - [bpf-next,v4,08/25] bpf: Refactor record_call_access() to extract per-arg logic
    https://git.kernel.org/bpf/bpf-next/c/84dd7df76efe
  - [bpf-next,v4,09/25] bpf: Use arg_is_fp() in has_fp_args()
    https://git.kernel.org/bpf/bpf-next/c/f44e815e65a3
  - [bpf-next,v4,10/25] bpf: Extend liveness analysis to track stack argument slots
    https://git.kernel.org/bpf/bpf-next/c/2af4e792773f
  - [bpf-next,v4,11/25] bpf: Reject stack arguments in non-JITed programs
    https://git.kernel.org/bpf/bpf-next/c/dc8f1cf6787c
  - [bpf-next,v4,12/25] bpf: Prepare architecture JIT support for stack arguments
    https://git.kernel.org/bpf/bpf-next/c/848d624acf66
  - [bpf-next,v4,13/25] bpf: Enable r11 based insns
    https://git.kernel.org/bpf/bpf-next/c/9fae4cba3bfd
  - [bpf-next,v4,14/25] bpf: Support stack arguments for kfunc calls
    https://git.kernel.org/bpf/bpf-next/c/e0b7b91c72db
  - [bpf-next,v4,15/25] bpf: Reject stack arguments if tail call reachable
    https://git.kernel.org/bpf/bpf-next/c/35b78733160c
  - [bpf-next,v4,16/25] bpf: Disable private stack for x86_64 if stack arguments used
    https://git.kernel.org/bpf/bpf-next/c/cb6af5314056
  - [bpf-next,v4,17/25] bpf,x86: Implement JIT support for stack arguments
    https://git.kernel.org/bpf/bpf-next/c/324c3ca6eed6
  - [bpf-next,v4,18/25] selftests/bpf: Add tests for BPF function stack arguments
    https://git.kernel.org/bpf/bpf-next/c/79e7ec00634e
  - [bpf-next,v4,19/25] selftests/bpf: Add tests for stack argument validation
    https://git.kernel.org/bpf/bpf-next/c/9f42204c62d5
  - [bpf-next,v4,20/25] selftests/bpf: Add BTF fixup for __naked subprog parameter names
    https://git.kernel.org/bpf/bpf-next/c/5b31de88920b
  - [bpf-next,v4,21/25] selftests/bpf: Add verifier tests for stack argument validation
    https://git.kernel.org/bpf/bpf-next/c/00c3ac4292a6
  - [bpf-next,v4,22/25] selftests/bpf: Add precision backtracking test for stack arguments
    https://git.kernel.org/bpf/bpf-next/c/6e277efbb19d
  - [bpf-next,v4,23/25] bpf, arm64: Map BPF_REG_0 to x8 instead of x7
    https://git.kernel.org/bpf/bpf-next/c/68e5627579d7
  - [bpf-next,v4,24/25] bpf, arm64: Add JIT support for stack arguments
    https://git.kernel.org/bpf/bpf-next/c/235b2fe772f5
  - [bpf-next,v4,25/25] selftests/bpf: Enable stack argument tests for arm64
    https://git.kernel.org/bpf/bpf-next/c/90e43f1b4753

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html