Skip to contentSkip to conversation

bpf: Support stack arguments for BPF functions and kfuncs

PATCH Superseded Yonghong Song 1 participant 13 messages Apr 5

Subsystems (7)
x86 Kselftest BPF Arm STM32 +3 more subsystems

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v2 cover bpf: Support stack arguments for BPF functions and kfuncs

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL. The incoming parameters can directly get the value from pushed native stack from caller. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).

Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.

For the rest of patches, patches 1-6 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 7 enables x86_64 for stack arguments. Patch 8 implemented JIT for x86_64. Patches 9-11 are some selftests.

[1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.

Yonghong Song (11):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Refactor process_iter_arg() to have proper argument index
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validation

 arch/x86/net/bpf_jit_comp.c                   | 140 +++++-
 include/linux/bpf.h                           |   6 +
 include/linux/bpf_verifier.h                  |  31 +-
 include/linux/filter.h                        |   4 +-
 kernel/bpf/btf.c                              |  21 +-
 kernel/bpf/core.c                             |  12 +-
 kernel/bpf/verifier.c                         | 474 ++++++++++++++++--
 .../selftests/bpf/prog_tests/stack_arg.c      | 132 +++++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  24 +
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      |  32 ++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 ++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 302 +++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  72 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  26 +
 16 files changed, 1594 insertions(+), 60 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH v2 1/11 bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE

The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf register R12 added by [1]. R12 is used as the base for stack arguments so it won't mess out R10 based stacks.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/filter.h | 3 ++-
 kernel/bpf/core.c      | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/filter.h +2 -1
index e40d4071a345..68f018dd4b9c 100644
@@ -59,7 +59,8 @@ struct ctl_table_header;
 
 /* Kernel hidden auxiliary/helper register. */
 #define BPF_REG_AX		MAX_BPF_REG
-#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 1)
+#define BPF_REG_STACK_ARG_BASE	(MAX_BPF_REG + 1)
+#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 2)
 #define MAX_BPF_JIT_REG		MAX_BPF_EXT_REG
 
 /* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
index 1af5fb3f21d9..3520337a1c0e 100644
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
 	u32 imm_rnd = get_random_u32();
 	s16 off;
 
-	BUILD_BUG_ON(BPF_REG_AX  + 1 != MAX_BPF_JIT_REG);
-	BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_AX + 2 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_STACK_ARG_BASE + 1 != MAX_BPF_JIT_REG);
 
 	/* Constraints on AX register:
 	 *
Signature · 1 line
2.52.0
author reviewer
#2 lore raw

PATCH v2 2/11 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments

In reply to Yonghong Song

Currently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.

Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.

---
 include/linux/bpf.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
index 35b1e25bd104..99701c4cc34e 100644
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {
 
 /* The longest tracepoint has 12 args.
  * See include/trace/bpf_probe.h
+ *
+ * Also reuse this macro for maximum number of arguments a BPF function
+ * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
+ * stack arg slots.
  */
 #define MAX_BPF_FUNC_ARGS 12
 
Signature · 1 line
2.52.0
author reviewer
#3 lore raw

PATCH v2 3/11 bpf: Support stack arguments for bpf functions

In reply to Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary

r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe() to do pruning check.

If the bpf function has more than one calls, e.g.,

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

The following is an illustration:

   Caller (foo)                           Callee (bar1)
   ============                           =============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||  ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+
   ...
   Back from bar1
   ...                                     Callee (bar2)
   ===                                     =============
                                     +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                     |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                     |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
   ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+|||  ...
   r12-32: [outgoing arg 7 to callee]-+||
   r12-40: [outgoing arg 8 to callee]--+|

r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |  31 +++-
 kernel/bpf/btf.c             |  14 +-
 kernel/bpf/verifier.c        | 311 +++++++++++++++++++++++++++++++++--
 4 files changed, 334 insertions(+), 24 deletions(-)
include/linux/bpf.h +2 -0
index 99701c4cc34e..72f990054d07 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
 	u32 max_pkt_offset;
 	u32 max_tp_access;
 	u32 stack_depth;
+	u16 incoming_stack_arg_depth;
+	u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
 	u32 id;
 	u32 func_cnt; /* used by non-func prog as the number of func progs */
 	u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +29 -2
index 36bfd96d4563..75fe67baa833 100644
@@ -268,6 +268,11 @@ struct bpf_retval_range {
 	bool return_32bit;
 };
 
+struct bpf_stack_arg_state {
+	struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+	u8 slot_type[BPF_REG_SIZE];
+};
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -319,6 +324,10 @@ struct bpf_func_state {
 	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
 	 */
 	int allocated_stack;
+
+	u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+	u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+	struct bpf_stack_arg_state *stack_arg_slots;
 };
 
 #define MAX_CALL_FRAMES 8
@@ -456,6 +465,17 @@ struct bpf_verifier_state {
 	     iter < frame->allocated_stack / BPF_REG_SIZE;		\
 	     iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
 
+#define bpf_get_spilled_stack_arg(slot, frame, mask)			\
+	(((slot < frame->stack_arg_depth / BPF_REG_SIZE) &&		\
+	  ((1 << frame->stack_arg_slots[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
+	 ? &frame->stack_arg_slots[slot].spilled_ptr : NULL)
+
+/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
+#define bpf_for_each_spilled_stack_arg(iter, frame, reg, mask)		\
+	for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \
+	     iter < frame->stack_arg_depth / BPF_REG_SIZE;		\
+	     iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask))
+
 #define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr)   \
 	({                                                               \
 		struct bpf_verifier_state *___vstate = __vst;            \
@@ -473,6 +493,11 @@ struct bpf_verifier_state {
 					continue;                        \
 				(void)(__expr);                          \
 			}                                                \
+			bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \
+				if (!__reg)                              \
+					continue;                        \
+				(void)(__expr);                          \
+			}                                                \
 		}                                                        \
 	})
 
@@ -686,10 +711,12 @@ struct bpf_subprog_info {
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
 	bool might_sleep: 1;
-	u8 arg_cnt:3;
+	u8 arg_cnt:4;
 
 	enum priv_stack_mode priv_stack_mode;
-	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+	u16 incoming_stack_arg_depth;
+	u16 outgoing_stack_arg_depth;
 };
 
 struct bpf_verifier_env;
kernel/bpf/btf.c +10 -4
index a62d78581207..c5f3aa05d5a3 100644
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	}
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-		if (!is_global)
-			return -EINVAL;
-		bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+	if (nargs > MAX_BPF_FUNC_ARGS) {
+		bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+			tname, nargs, MAX_BPF_FUNC_ARGS);
+		return -EINVAL;
+	}
+	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+		bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
+	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
 	while (btf_type_is_modifier(t))
kernel/bpf/verifier.c +293 -18

424 lines not on this page · raw message

Signature · 1 line
2.52.0
author reviewer
#4 lore raw

PATCH v2 4/11 bpf: Refactor process_iter_arg() to have proper argument index

In reply to Yonghong Song

In the next patch for kfunc stack arguments, a faked register is used to do proper verification checking. For process_iter_arg(), the regno is passed in and the iterator assumes 'regno - 1' as the argument index. This is wrong as regno is fake. So refactor process_iter_arg() by adding actual argument index which is used inside the function.

---
 kernel/bpf/verifier.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/bpf/verifier.c +7 -7
index ebf164036849..fd13309d9006 100644
@@ -9259,14 +9259,14 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx,
 }
 
 static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx,
-			    struct bpf_kfunc_call_arg_meta *meta)
+			    int argno, struct bpf_kfunc_call_arg_meta *meta)
 {
 	struct bpf_reg_state *reg = reg_state(env, regno);
 	const struct btf_type *t;
 	int spi, err, i, nr_slots, btf_id;
 
 	if (reg->type != PTR_TO_STACK) {
-		verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1);
+		verbose(env, "arg#%d expected pointer to an iterator on stack\n", argno);
 		return -EINVAL;
 	}
 
@@ -9276,9 +9276,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 	 * to any kfunc, if arg has "__iter" suffix, we need to be a bit more
 	 * conservative here.
 	 */
-	btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1);
+	btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, argno);
 	if (btf_id < 0) {
-		verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1);
+		verbose(env, "expected valid iter pointer as arg #%d\n", argno);
 		return -EINVAL;
 	}
 	t = btf_type_by_id(meta->btf, btf_id);
@@ -9288,7 +9288,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 		/* bpf_iter_<type>_new() expects pointer to uninit iter state */
 		if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {
 			verbose(env, "expected uninitialized iter_%s as arg #%d\n",
-				iter_type_str(meta->btf, btf_id), regno - 1);
+				iter_type_str(meta->btf, btf_id), argno);
 			return -EINVAL;
 		}
 
@@ -9312,7 +9312,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 			break;
 		case -EINVAL:
 			verbose(env, "expected an initialized iter_%s as arg #%d\n",
-				iter_type_str(meta->btf, btf_id), regno - 1);
+				iter_type_str(meta->btf, btf_id), argno);
 			return err;
 		case -EPROTO:
 			verbose(env, "expected an RCU CS when using %s\n", meta->func_name);
@@ -14063,7 +14063,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					return -EINVAL;
 				}
 			}
-			ret = process_iter_arg(env, regno, insn_idx, meta);
+			ret = process_iter_arg(env, regno, insn_idx, i, meta);
 			if (ret < 0)
 				return ret;
 			break;
Signature · 1 line
2.52.0
author reviewer
#5 lore raw

PATCH v2 5/11 bpf: Support stack arguments for kfunc calls

In reply to Yonghong Song

Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.

For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.

The following is an example to show how stack arguments are saved:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

Caller (foo)
============

       r12-relative stack arg area:

       r12-8:  [incoming arg 6]
       r12-16: [incoming arg 7]

       ---- incoming/outgoing boundary (kfunc1)
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       ...
       Back from kfunc1
       ...

       ---- incoming/outgoing boundary
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       r12-48: [outgoing arg 9 to callee]

Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().

In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. The following is one of examples based on a later selftest:

13: (85) call bpf_kfunc_call_stack_arg_mem#152105 Use reg 1 for stack arg#5
Use reg 2 to represent mem_size
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2 End of using reg 1 for stack arg#6
14: R0=scalar()
14: (95) exit

The above example is for KF_ARG_PTR_TO_MEM_SIZE case. Registers 1 and 2 are used as an temporary register for argument checking. The verifier log will identify when the temporary register is used and when the temporary register is not used any more.

There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno' is also keep regno for later fixup. Since regno is a faked one such three cases are rejected for now if they are in stack arguments. If possible, new kfuncs could keep them in first 5 registers so there are no issues at all.

If faked register approach is not favored, the following is an alternative approach with the following data structure:

   struct reg_or_arg_t {
     struct bpf_reg_state *state;
     union {
       int regno;
       int argno;
     };
     bool is_reg; /* distinguish between reg and arg */
   };

Such a struct can replace existing regno/argno/reg_state parameters to make it easy to distinguish reg vs. arg.

---
 kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 124 insertions(+), 19 deletions(-)
kernel/bpf/verifier.c +124 -19
index fd13309d9006..f19398f628ba 100644
@@ -3491,7 +3491,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	struct bpf_kfunc_meta kfunc;
 	struct bpf_kfunc_desc *desc;
 	unsigned long addr;
-	int err;
+	int i, err;
 
 	prog_aux = env->prog->aux;
 	tab = prog_aux->kfunc_tab;
@@ -3567,6 +3567,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	if (err)
 		return err;
 
+	for (i = MAX_BPF_FUNC_REG_ARGS; i < func_model.nr_args; i++) {
+		if (func_model.arg_size[i] > sizeof(u64)) {
+			verbose(env, "kfunc %s arg#%d size %d > %zu not supported for stack args\n",
+				kfunc.name, i, func_model.arg_size[i], sizeof(u64));
+			return -EINVAL;
+		}
+	}
+
 	desc = &tab->descs[tab->nr_descs++];
 	desc->func_id = func_id;
 	desc->offset = offset;
@@ -13083,6 +13091,19 @@ static bool is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)
 	return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data];
 }
 
+static struct bpf_reg_state *get_kfunc_arg_reg(struct bpf_verifier_env *env, int argno)
+{
+	struct bpf_func_state *caller;
+	int spi;
+
+	if (argno < MAX_BPF_FUNC_REG_ARGS)
+		return &cur_regs(env)[argno + 1];
+
+	caller = cur_func(env);
+	spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE + (argno - MAX_BPF_FUNC_REG_ARGS);
+	return &caller->stack_arg_slots[spi].spilled_ptr;
+}
+
 static enum kfunc_ptr_arg_type
 get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		       struct bpf_kfunc_call_arg_meta *meta,
@@ -13101,8 +13122,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		return KF_ARG_PTR_TO_CTX;
 
 	if (argno + 1 < nargs &&
-	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
-	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
+	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) ||
+	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1))))
 		arg_mem_size = true;
 
 	/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -13770,9 +13791,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+	if (nargs > MAX_BPF_FUNC_ARGS) {
 		verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
-			MAX_BPF_FUNC_REG_ARGS);
+			MAX_BPF_FUNC_ARGS);
 		return -EINVAL;
 	}
 
@@ -13780,26 +13801,59 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 	 * verifier sees.
 	 */
 	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *regs = cur_regs(env), *reg;
+		struct bpf_reg_state saved_reg;
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
 		u32 regno = i + 1, ref_id, type_size;
 		bool is_ret_buf_sz = false;
+		bool is_stack_arg = false;
 		int kf_arg_type;
 
+		if (i < MAX_BPF_FUNC_REG_ARGS) {
+			reg = &regs[i + 1];
+		} else {
+			/*
+			 * Retrieve the spilled reg state from the stack arg slot.
+			 * Reuse the existing type checking infrastructure which
+			 * reads from cur_regs(env)[regno], temporarily copy the
+			 * stack arg reg state into regs[BPF_REG_1] and restore
+			 * it after checking.
+			 */
+			struct bpf_func_state *caller = cur_func(env);
+			int spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE +
+				  (i - MAX_BPF_FUNC_REG_ARGS);
+
+			if (!is_stack_arg_slot_initialized(caller, spi)) {
+				verbose(env, "stack arg#%d not properly initialized\n", i);
+				return -EINVAL;
+			}
+
+			is_stack_arg = true;
+			regno = BPF_REG_1;
+			saved_reg = regs[BPF_REG_1];
+			regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+			reg = &regs[BPF_REG_1];
+			verbose(env, "Use reg %d for stack arg#%d\n", regno, i);
+		}
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
 			/* Reject repeated use bpf_prog_aux */
 			if (meta->arg_prog) {
 				verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc");
 				return -EFAULT;
 			}
+			if (is_stack_arg) {
+				verbose(env, "arg#%d prog->aux cannot be a stack argument\n", i);
+				return -EINVAL;
+			}
 			meta->arg_prog = true;
 			cur_aux(env)->arg_prog = regno;
-			continue;
+			goto next_arg;
 		}
 
 		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
-			continue;
+			goto next_arg;
 
 		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
 
@@ -13818,9 +13872,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					verbose(env, "R%d must be a known constant\n", regno);
 					return -EINVAL;
 				}
-				ret = mark_chain_precision(env, regno);
-				if (ret < 0)
-					return ret;
+				if (!is_stack_arg) {
+					ret = mark_chain_precision(env, regno);
+					if (ret < 0)
+						return ret;
+				}
 				meta->arg_constant.found = true;
 				meta->arg_constant.value = reg->var_off.value;
 			} else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) {
@@ -13842,11 +13898,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				}
 
 				meta->r0_size = reg->var_off.value;
-				ret = mark_chain_precision(env, regno);
-				if (ret)
-					return ret;
+				if (!is_stack_arg) {
+					ret = mark_chain_precision(env, regno);
+					if (ret)
+						return ret;
+				}
 			}
-			continue;
+			goto next_arg;
 		}
 
 		if (!btf_type_is_ptr(t)) {
@@ -13868,8 +13926,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				return -EFAULT;
 			}
 			meta->ref_obj_id = reg->ref_obj_id;
-			if (is_kfunc_release(meta))
+			if (is_kfunc_release(meta)) {
+				if (is_stack_arg) {
+					verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
+			}
 		}
 
 		ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13881,7 +13944,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 		switch (kf_arg_type) {
 		case KF_ARG_PTR_TO_NULL:
-			continue;
+			goto next_arg;
 		case KF_ARG_PTR_TO_MAP:
 			if (!reg->map_ptr) {
 				verbose(env, "pointer in R%d isn't map pointer\n", regno);
@@ -14020,6 +14083,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				dynptr_arg_type |= DYNPTR_TYPE_FILE;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) {
 				dynptr_arg_type |= DYNPTR_TYPE_FILE;
+				if (is_stack_arg) {
+					verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] &&
 				   (dynptr_arg_type & MEM_UNINIT)) {
@@ -14169,8 +14236,18 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		{
 			struct bpf_reg_state *buff_reg = &regs[regno];
 			const struct btf_param *buff_arg = &args[i];
-			struct bpf_reg_state *size_reg = &regs[regno + 1];
+			struct bpf_reg_state *size_reg;
 			const struct btf_param *size_arg = &args[i + 1];
+			struct bpf_reg_state saved_size_reg = {};
+			bool size_is_stack_arg = false;
+
+			if (i >= MAX_BPF_FUNC_REG_ARGS) {
+				size_is_stack_arg = true;
+				saved_size_reg = regs[regno + 1];
+				regs[regno + 1] = *get_kfunc_arg_reg(env, i + 1);
+				verbose(env, "Use reg %d to represent mem_size\n", regno + 1);
+			}
+			size_reg = &regs[regno + 1];
 
 			if (!register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) {
 				ret = check_kfunc_mem_size_reg(env, size_reg, regno + 1);
@@ -14193,6 +14270,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				meta->arg_constant.value = size_reg->var_off.value;
 			}
 
+			if (size_is_stack_arg)
+				regs[regno + 1] = saved_size_reg;
+
 			/* Skip next '__sz' or '__szk' argument */
 			i++;
 			break;
@@ -14294,6 +14374,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			break;
 		}
 		}
+next_arg:
+		if (is_stack_arg) {
+			regs[regno] = saved_reg;
+			verbose(env, "End of using reg %d for stack arg#%d\n", regno, i);
+		}
 	}
 
 	if (is_kfunc_release(meta) && !meta->release_regno) {
@@ -15059,7 +15144,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	nargs = btf_type_vlen(meta.func_proto);
 	args = (const struct btf_param *)(meta.func_proto + 1);
-	for (i = 0; i < nargs; i++) {
+	for (i = 0; i < nargs && i < MAX_BPF_FUNC_REG_ARGS; i++) {
 		u32 regno = i + 1;
 
 		t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
@@ -15070,6 +15155,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			mark_btf_func_reg_size(env, regno, t->size);
 	}
 
+	/* Track outgoing stack arg depth for kfuncs with >5 args */
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		struct bpf_func_state *caller = cur_func(env);
+		struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+		u16 kfunc_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
+		if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth)
+			caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth;
+	}
+
 	if (is_iter_next_kfunc(&meta)) {
 		err = process_iter_next_call(env, insn_idx, &meta);
 		if (err)
@@ -23975,6 +24070,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	if (!bpf_jit_supports_far_kfunc_call())
 		insn->imm = BPF_CALL_IMM(desc->addr);
 
+	/*
+	 * After resolving the kfunc address, insn->off is no longer needed
+	 * for BTF fd index. Repurpose it to store the number of stack args
+	 * so the JIT can marshal them.
+	 */
+	if (desc->func_model.nr_args > MAX_BPF_FUNC_REG_ARGS)
+		insn->off = desc->func_model.nr_args - MAX_BPF_FUNC_REG_ARGS;
+	else
+		insn->off = 0;
+
 	if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
 		struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
 		struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };
Signature · 1 line
2.52.0
author reviewer
#6 lore raw

PATCH v2 6/11 bpf: Reject stack arguments in non-JITed programs

In reply to Yonghong Song

The interpreter does not understand the bpf register r12 (BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.

---
 kernel/bpf/core.c     | 3 ++-
 kernel/bpf/verifier.c | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)
kernel/bpf/core.c +2 -1
index 3520337a1c0e..a04b31eb4c49 100644
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) ||
+	    fp->aux->stack_arg_depth)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +6 -0
index f19398f628ba..010ef7fc6c72 100644
@@ -23944,6 +23944,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}
+	for (i = 0; i < env->subprog_cnt; i++) {
+		if (env->subprog_info[i].incoming_stack_arg_depth) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
 	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
 		/* When JIT fails the progs with bpf2bpf calls and tail_calls
 		 * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
author reviewer
#7 lore raw

PATCH v2 7/11 bpf: Enable stack argument support for x86_64

In reply to Yonghong Song

Add stack argument support for x86_64.

---
 arch/x86/net/bpf_jit_comp.c | 5 +++++
 include/linux/filter.h      | 1 +
 kernel/bpf/btf.c            | 9 ++++++++-
 kernel/bpf/core.c           | 5 +++++
 4 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
index e9b78040d703..32864dbc2c4e 100644
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_stack_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (text_poke_copy(dst, src, len) == NULL)
include/linux/filter.h +1 -0
index 68f018dd4b9c..a5035fb80a6b 100644
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_ptr_xchg(void);
kernel/bpf/btf.c +8 -1
index c5f3aa05d5a3..1cbe0f2b0e41 100644
@@ -20,6 +20,7 @@
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
 #include <linux/bpf.h>
+#include <linux/filter.h>
 #include <linux/bpf_lsm.h>
 #include <linux/skmsg.h>
 #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -ENOTSUPP;
+		}
 		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+	}
 
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
kernel/bpf/core.c +5 -0
index a04b31eb4c49..01de4f7f3a82 100644
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_stack_args(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_far_kfunc_call(void)
 {
 	return false;
Signature · 1 line
2.52.0
author reviewer
#8 lore raw

PATCH v2 8/11 bpf,x86: Implement JIT support for stack arguments

In reply to Yonghong Song

Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.

The JIT follows the native x86_64 calling convention for stack argument placement. Incoming stack args from the caller sit above the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly where x86_64 expects them after CALL + PUSH RBP. Only the outgoing stack arg area is allocated below the program stack in the prologue.

The native x86_64 stack layout for a function with incoming and outgoing stack args:

high address

  ┌─────────────────────────┐
  │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
  │ ...                     │
  │ incoming stack arg 1    │  [rbp + 16]
  ├─────────────────────────┤
  │ return address          │  [rbp + 8]
  │ saved rbp               │  [rbp]
  ├─────────────────────────┤
  │ BPF program stack       │  (stack_depth bytes)
  ├─────────────────────────┤
  │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
  │ ...                     │   (written via r12-relative STX/ST)
  │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
  ├─────────────────────────┤
  │ callee-saved regs ...   │  (pushed after sub rsp)
  └─────────────────────────┘  rsp

low address

BPF r12-relative offsets are translated to native RBP-relative offsets with two formulas:
- Incoming args (load: -off <= incoming_depth):

      native_off = 8 - bpf_off  →  [rbp + 16 + ...]

- Outgoing args (store: -off > incoming_depth):
native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off

Since callee-saved registers are pushed below the outgoing area, outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF calls and kfunc calls, outgoing args are explicitly pushed from the outgoing area onto the stack before CALL and rsp is restored after return.

For kfunc calls specifically, arg 6 is loaded into R9 and args 7+ are pushed onto the native stack, per the x86_64 calling convention.

---
 arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
 1 file changed, 129 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +129 -6
index 32864dbc2c4e..206f342a0ca0 100644
@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
 	*pprog = prog;
 }
 
+/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
+static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
+{
+	u8 *prog = *pprog;
+	int k, bytes = 0;
+	s32 off;
+
+	for (k = from; k >= to; k--) {
+		off = outgoing_base + (k - 1) * 8;
+		/* push qword [rbp + off] */
+		if (is_imm8(off)) {
+			EMIT3(0xFF, 0x75, off);
+			bytes += 3;
+		} else {
+			EMIT2_off32(0xFF, 0xB5, off);
+			bytes += 6;
+		}
+	}
+	*pprog = prog;
+	return bytes;
+}
+
 static void emit_nops(u8 **pprog, int len)
 {
 	u8 *prog = *pprog;
@@ -1664,16 +1686,33 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	int i, excnt = 0;
 	int ilen, proglen = 0;
 	u8 *prog = temp;
-	u32 stack_depth;
+	u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth;
+	u32 prog_stack_depth, stack_depth;
+	bool has_stack_args;
 	int err;
 
 	stack_depth = bpf_prog->aux->stack_depth;
+	stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+	incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
+	outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth;
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
 	if (priv_stack_ptr) {
 		priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
 		stack_depth = 0;
 	}
 
+	/*
+	 * Save program stack depth before adding outgoing stack arg space.
+	 * Incoming stack args are read directly from [rbp + 16 + ...].
+	 * Only the outgoing stack arg area is allocated below the
+	 * program stack. Outgoing args written here become the callee's
+	 * incoming args.
+	 */
+	prog_stack_depth = round_up(stack_depth, 8);
+	if (outgoing_stack_arg_depth)
+		stack_depth += outgoing_stack_arg_depth;
+	has_stack_args = stack_arg_depth > 0;
+
 	arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
 	user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
@@ -1715,13 +1754,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
+		bool adjust_stack_arg_off = false;
 		const s32 imm32 = insn->imm;
 		u32 dst_reg = insn->dst_reg;
 		u32 src_reg = insn->src_reg;
 		u8 b2 = 0, b3 = 0;
 		u8 *start_of_ldx;
 		s64 jmp_offset;
-		s16 insn_off;
+		s32 insn_off;
 		u8 jmp_cond;
 		u8 *func;
 		int nops;
@@ -1734,6 +1774,37 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 				dst_reg = X86_REG_R9;
 		}
 
+		if (has_stack_args) {
+			u8 class = BPF_CLASS(insn->code);
+
+			if (class == BPF_LDX &&
+			    src_reg == BPF_REG_STACK_ARG_BASE) {
+				src_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+			if ((class == BPF_STX || class == BPF_ST) &&
+			    dst_reg == BPF_REG_STACK_ARG_BASE) {
+				dst_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+		}
+
+		/*
+		 * Translate BPF r12-relative offset to native RBP-relative:
+		 *
+		 * Incoming args (load: offset >= -incoming_depth):
+		 *   BPF: r12 + bpf_off = r12 - k * 8 (k = 1,2,...) for incoming arg k
+		 *   Native: [rbp + 8 + k * 8]
+		 *   Formula: native_off = 8 + k * 8 = 8 - bpf_off
+		 *
+		 * Outgoing args (store: offset < -incoming_depth):
+		 *   BPF: r12 + bpf_off = r12 - (incoming + k * 8) for outgoing arg k
+		 *   Native: [rbp - prog_stack_depth - outgoing + (k - 1) * 8]
+		 *   Formula: native_off = -(prog_stack_depth + outgoing) + (k - 1) * 8
+		 *          = -(prog_stack_depth + outgoing + incoming + 8) - bpf_off
+		 *          = -(prog_stack_depth + stack_arg_depth + 8) - bpf_off
+		 */
+
 		switch (insn->code) {
 			/* ALU */
 		case BPF_ALU | BPF_ADD | BPF_X:
@@ -2131,10 +2202,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 		case BPF_ST | BPF_MEM | BPF_DW:
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
 
-st:			if (is_imm8(insn->off))
-				EMIT2(add_1reg(0x40, dst_reg), insn->off);
+st:			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+			if (is_imm8(insn_off))
+				EMIT2(add_1reg(0x40, dst_reg), insn_off);
 			else
-				EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
+				EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
 
 			EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
 			break;
@@ -2144,7 +2218,10 @@ st:			if (is_imm8(insn->off))
 		case BPF_STX | BPF_MEM | BPF_H:
 		case BPF_STX | BPF_MEM | BPF_W:
 		case BPF_STX | BPF_MEM | BPF_DW:
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			break;
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2243,6 +2320,8 @@ st:			if (is_imm8(insn->off))
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
 			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = 8 - insn_off;
 
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2441,6 +2520,7 @@ st:			if (is_imm8(insn->off))
 			/* call */
 		case BPF_JMP | BPF_CALL: {
 			u8 *ip = image + addrs[i - 1];
+			int stack_args = 0;
 
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -2449,6 +2529,41 @@ st:			if (is_imm8(insn->off))
 			}
 			if (!imm32)
 				return -EINVAL;
+
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+				/*
+				 * BPF-to-BPF calls: push outgoing stack args from
+				 * the outgoing area onto the stack before CALL.
+				 * The outgoing area is at [rbp - prog_stack - outgoing],
+				 * but rsp is below that due to callee-saved reg pushes,
+				 * so we must explicitly push args for the callee.
+				*/
+				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+				int n_args = outgoing_stack_arg_depth / 8;
+
+				ip += push_stack_args(&prog, outgoing_base, n_args, 1);
+			}
+
+			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+				/* Kfunc calls: arg 6 → R9, args 7+ → push. */
+				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+				int kfunc_stack_args = insn->off;
+
+				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+
+				/* Push args 7+ in reverse order */
+				if (stack_args > 0)
+					ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
+
+				/* mov r9, [rbp + outgoing_base] (arg 6) */
+				if (is_imm8(outgoing_base)) {
+					EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
+					ip += 4;
+				} else {
+					EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
+					ip += 7;
+				}
+			}
 			if (priv_frame_ptr) {
 				push_r9(&prog);
 				ip += 2;
@@ -2458,6 +2573,14 @@ st:			if (is_imm8(insn->off))
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (stack_args > 0) {
+				/* add rsp, stack_args * 8 */
+				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+			}
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+				/* add rsp, outgoing_stack_arg_depth */
+				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
+			}
 			break;
 		}
 
Signature · 1 line
2.52.0
author reviewer
#9 lore raw

PATCH v2 9/11 selftests/bpf: Add tests for BPF function stack arguments

In reply to Yonghong Song

Add selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.

---
 .../selftests/bpf/prog_tests/stack_arg.c      | 132 +++++++++++
 tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++++++++++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 ++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  65 ++++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  20 +-
 5 files changed, 592 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +132 -0
new file mode 100644
index 000000000000..1af5e5c91e0d
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "stack_arg.skel.h"
+#include "stack_arg_kfunc.skel.h"
+
+static void run_subtest(struct bpf_program *prog, int expected)
+{
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = &pkt_v4,
+		.data_size_in = sizeof(pkt_v4),
+		.repeat = 1,
+	);
+
+	prog_fd = bpf_program__fd(prog);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, expected, "retval");
+}
+
+static void test_global_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_global_many_args, 36);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_async_cb_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_bpf2bpf(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_bpf2bpf_ptr_stack_arg, 45);
+	run_subtest(skel->progs.test_bpf2bpf_mix_stack_args, 51);
+	run_subtest(skel->progs.test_bpf2bpf_nesting_stack_arg, 50);
+	run_subtest(skel->progs.test_bpf2bpf_dynptr_stack_arg, 69);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_kfunc(void)
+{
+	struct stack_arg_kfunc *skel;
+
+	skel = stack_arg_kfunc__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg_kfunc__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_stack_arg_scalar, 36);
+	run_subtest(skel->progs.test_stack_arg_ptr, 45);
+	run_subtest(skel->progs.test_stack_arg_mix, 51);
+	run_subtest(skel->progs.test_stack_arg_dynptr, 69);
+	run_subtest(skel->progs.test_stack_arg_mem, 151);
+	run_subtest(skel->progs.test_stack_arg_iter, 115);
+	run_subtest(skel->progs.test_stack_arg_const_str, 15);
+	run_subtest(skel->progs.test_stack_arg_timer, 15);
+
+out:
+	stack_arg_kfunc__destroy(skel);
+}
+
+void test_stack_arg(void)
+{
+	if (test__start_subtest("global_many_args"))
+		test_global_many();
+	if (test__start_subtest("async_cb_many_args"))
+		test_async_cb_many();
+	if (test__start_subtest("bpf2bpf"))
+		test_bpf2bpf();
+	if (test__start_subtest("kfunc"))
+		test_kfunc();
+}
tools/testing/selftests/bpf/progs/stack_arg.c new file +212 -0

212 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +164 -0

164 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +65 -0

77 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +19 -1
index aa0b8d41e71b..2c1cb118f886 100644
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc {
 };
 #endif
 
+struct bpf_iter_testmod_seq;
+
 struct prog_test_pass1 {
 	int x0;
 	struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
-
+__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
+			       __u64 e, __u64 f, __u64 g, __u64 h) __ksym;
+__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p, __u64 f,
+				   struct prog_test_pass1 *q) __ksym;
+__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				      struct bpf_dynptr *ptr) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   void *mem, int mem__sz) __ksym;
+__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				    struct bpf_iter_testmod_seq *it__iter) __ksym;
+__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+					 const char *str__str) __ksym;
+__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				     struct bpf_timer *timer) __ksym;
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
author reviewer
#10 lore raw

PATCH v2 10/11 selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument

In reply to Yonghong Song

Add a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).

---
 .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      | 32 +++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  7 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  8 +++++
 4 files changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
new file mode 100644
index 000000000000..328a79edee45
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_fail.skel.h"
+
+void test_stack_arg_fail(void)
+{
+	struct stack_arg_fail *skel;
+
+	skel = stack_arg_fail__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	ASSERT_ERR(stack_arg_fail__load(skel), "load_should_fail");
+
+out:
+	stack_arg_fail__destroy(skel);
+}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
new file mode 100644
index 000000000000..caa63b6f6a80
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	struct prog_test_big_arg s = { .a = 1, .b = 2 };
+
+	return bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
index 4fcdb13e4a9a..861c564cc330 100644
@@ -881,6 +881,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e
 	return a + b + c + d + e;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_big_arg s)
+{
+	return a + b + c + d + e + s.a + s.b;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1352,6 +1358,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +8 -0
index 2c1cb118f886..2a40f80b074a 100644
@@ -50,6 +50,11 @@ struct prog_test_pass2 {
 	} x;
 };
 
+struct prog_test_big_arg {
+	long a;
+	long b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u
 					 const char *str__str) __ksym;
 __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				     struct bpf_timer *timer) __ksym;
+__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_big_arg s) __ksym;
+
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
Signature · 1 line
2.52.0
author reviewer
#11 lore raw

PATCH v2 11/11 selftests/bpf: Add verifier tests for stack argument validation

In reply to Yonghong Song

Add inline-asm-based verifier tests that exercise the stack argument validation logic directly.

---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_stack_arg.c  | 302 ++++++++++++++++++
 2 files changed, 304 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index 1ac366fd4dae..f39ed1f28a7b 100644
@@ -90,6 +90,7 @@
 #include "verifier_sockmap_mutate.skel.h"
 #include "verifier_spill_fill.skel.h"
 #include "verifier_spin_lock.skel.h"
+#include "verifier_stack_arg.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
 #include "verifier_subprog_precision.skel.h"
@@ -236,6 +237,7 @@ void test_verifier_sock_addr(void)            { RUN(verifier_sock_addr); }
 void test_verifier_sockmap_mutate(void)       { RUN(verifier_sockmap_mutate); }
 void test_verifier_spill_fill(void)           { RUN(verifier_spill_fill); }
 void test_verifier_spin_lock(void)            { RUN(verifier_spin_lock); }
+void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +302 -0

302 lines not on this page · raw message

Signature · 1 line
2.52.0
author reviewer
#12 lore raw
In reply to Yonghong Song
Quoted 75 lines · Yonghong Song

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL. The incoming parameters can directly get the value from pushed native stack from caller. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).

Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.

For the rest of patches, patches 1-6 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 7 enables x86_64 for stack arguments. Patch 8 implemented JIT for x86_64. Patches 9-11 are some selftests.

   [1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.

Yonghong Song (11):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Refactor process_iter_arg() to have proper argument index
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
argument
selftests/bpf: Add verifier tests for stack argument validation

  arch/x86/net/bpf_jit_comp.c                   | 140 +++++-
  include/linux/bpf.h                           |   6 +
  include/linux/bpf_verifier.h                  |  31 +-
  include/linux/filter.h                        |   4 +-
  kernel/bpf/btf.c                              |  21 +-
  kernel/bpf/core.c                             |  12 +-
  kernel/bpf/verifier.c                         | 474 ++++++++++++++++--
  .../selftests/bpf/prog_tests/stack_arg.c      | 132 +++++
  .../selftests/bpf/prog_tests/stack_arg_fail.c |  24 +
  .../selftests/bpf/prog_tests/verifier.c       |   2 +
  tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++
  .../selftests/bpf/progs/stack_arg_fail.c      |  32 ++
  .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 ++++++
  .../selftests/bpf/progs/verifier_stack_arg.c  | 302 +++++++++++
  .../selftests/bpf/test_kmods/bpf_testmod.c    |  72 +++
  .../bpf/test_kmods/bpf_testmod_kfunc.h        |  26 +
  16 files changed, 1594 insertions(+), 60 deletions(-)
Quoted 7 lines

create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c

Please ignore this patch set. Just discovered that there is a selftest failure and a couple of 'UTF-8' issues. I will fix these issues and send the next revision shortly.