Skip to contentSkip to conversation

bpf: Support stack arguments for BPF functions and kfuncs

PATCH Superseded Yonghong Song 5 participants 33 messages Apr 2 – Apr 6

Subsystems (7)
x86 Kselftest BPF Arm STM32 +3 more subsystems

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

cover bpf: Support stack arguments for BPF functions and kfuncs

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits stores/loads through a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

The x86_64 JIT translates r12-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by (incoming + max_outgoing) bytes to hold the stack arg area below the program stack. This makes implementation easier as the r10 can be reused for stack argument access. At BPF-to-BPF call sites, outgoing args are pushed onto the native stack before CALL and popped after return. For kfunc calls, args are marshaled per the x86_64 C calling convention (arg 6 in R9, args 7+ on the native stack).

Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.

For the rest of patches, patches 1-5 added verifier support of stack arguments for bpf-to-bpf functions and kfunc's. Patch 6 enables x86_64 for stack arguments. Patch 7 implemented JIT for x86_64. Patches 8-10 are some selftests.

[1] https://github.com/llvm/llvm-project/pull/189060

Yonghong Song (10):
bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments in non-JITed programs
bpf: Enable stack argument support for x86_64
bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add negative test for oversized kfunc stack argument selftests/bpf: Add verifier tests for stack argument validation

 arch/x86/net/bpf_jit_comp.c                   | 150 +++++++-
 include/linux/bpf.h                           |   6 +
 include/linux/bpf_verifier.h                  |  15 +-
 include/linux/filter.h                        |   4 +-
 kernel/bpf/btf.c                              |  21 +-
 kernel/bpf/core.c                             |  12 +-
 kernel/bpf/verifier.c                         | 351 ++++++++++++++++--
 .../selftests/bpf/prog_tests/stack_arg.c      | 143 +++++++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  24 ++
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/stack_arg.c | 111 ++++++
 .../selftests/bpf/progs/stack_arg_fail.c      |  32 ++
 .../selftests/bpf/progs/stack_arg_kfunc.c     |  59 +++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 122 ++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  29 ++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  14 +
 16 files changed, 1045 insertions(+), 50 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH 1/10 bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE

The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf register R12 added by [1]. R12 is used as the base for stack arguments so it won't mess out R10 based stacks.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/filter.h | 3 ++-
 kernel/bpf/core.c      | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/filter.h +2 -1
index e40d4071a345..68f018dd4b9c 100644
@@ -59,7 +59,8 @@ struct ctl_table_header;
 
 /* Kernel hidden auxiliary/helper register. */
 #define BPF_REG_AX		MAX_BPF_REG
-#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 1)
+#define BPF_REG_STACK_ARG_BASE	(MAX_BPF_REG + 1)
+#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 2)
 #define MAX_BPF_JIT_REG		MAX_BPF_EXT_REG
 
 /* unused opcode to mark special call to bpf_tail_call() helper */
kernel/bpf/core.c +2 -2
index 1af5fb3f21d9..3520337a1c0e 100644
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
 	u32 imm_rnd = get_random_u32();
 	s16 off;
 
-	BUILD_BUG_ON(BPF_REG_AX  + 1 != MAX_BPF_JIT_REG);
-	BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_AX + 2 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_STACK_ARG_BASE + 1 != MAX_BPF_JIT_REG);
 
 	/* Constraints on AX register:
 	 *
Signature · 1 line
2.52.0
author reviewer
#2 lore raw

PATCH 2/10 bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments

In reply to Yonghong Song

Currently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.

Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS is 12 which should be sufficient for majority of bpf functions and kfunc's.

---
 include/linux/bpf.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/bpf.h +4 -0
index 05b34a6355b0..e24c4a2e95f7 100644
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {
 
 /* The longest tracepoint has 12 args.
  * See include/trace/bpf_probe.h
+ *
+ * Also reuse this macro for maximum number of arguments a BPF function
+ * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
+ * stack arg slots.
  */
 #define MAX_BPF_FUNC_ARGS 12
 
Signature · 1 line
2.52.0
author reviewer
#3 lore raw

PATCH 3/10 bpf: Support stack arguments for bpf functions

In reply to Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary

r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.

If the bpf function has more than one calls, e.g.,

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

The following is an illustration:

   Caller (foo)                           Callee (bar1)
   ============                           =============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||  ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+
   ...
   Back from bar1
   ...                                     Callee (bar2)
   ===                                     =============
                                     +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                     |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                     |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
   ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+|||  ...
   r12-32: [outgoing arg 7 to callee]-+||
   r12-40: [outgoing arg 8 to callee]--+|

r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |  15 ++-
 kernel/bpf/btf.c             |  14 +-
 kernel/bpf/verifier.c        | 248 ++++++++++++++++++++++++++++++++---
 4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
index e24c4a2e95f7..a0a1e14e4394 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
 	u32 max_pkt_offset;
 	u32 max_tp_access;
 	u32 stack_depth;
+	u16 incoming_stack_arg_depth;
+	u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
 	u32 id;
 	u32 func_cnt; /* used by non-func prog as the number of func progs */
 	u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +13 -2
index 090aa26d1c98..a260610cd1c1 100644
@@ -268,6 +268,11 @@ struct bpf_retval_range {
 	bool return_32bit;
 };
 
+struct bpf_stack_arg_state {
+	struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+	u8 slot_type[BPF_REG_SIZE];
+};
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -319,6 +324,10 @@ struct bpf_func_state {
 	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
 	 */
 	int allocated_stack;
+
+	u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+	u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+	struct bpf_stack_arg_state *stack_arg_slots;
 };
 
 #define MAX_CALL_FRAMES 8
@@ -674,10 +683,12 @@ struct bpf_subprog_info {
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
 	bool might_sleep: 1;
-	u8 arg_cnt:3;
+	u8 arg_cnt:4;
 
 	enum priv_stack_mode priv_stack_mode;
-	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+	u16 incoming_stack_arg_depth;
+	u16 outgoing_stack_arg_depth;
 };
 
 struct bpf_verifier_env;
kernel/bpf/btf.c +10 -4
index a62d78581207..c5f3aa05d5a3 100644
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	}
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-		if (!is_global)
-			return -EINVAL;
-		bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+	if (nargs > MAX_BPF_FUNC_ARGS) {
+		bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+			tname, nargs, MAX_BPF_FUNC_ARGS);
+		return -EINVAL;
+	}
+	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+		bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
+	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
 	while (btf_type_is_modifier(t))
kernel/bpf/verifier.c +232 -16

340 lines not on this page · raw message

Signature · 1 line
2.52.0
#4 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

The new stack_arg_slots state is part of bpf_func_state and is properly copied in copy_stack_state() and freed in free_func_state(), but stacksafe() does not compare it during state pruning.
Quoted 4 lines
include/linux/bpf_verifier.h +0 -0
index 090aa26d1c98..a260610cd1c1 100644

[ ... ]

include/linux/bpf_verifier.h struct bpf_func_state { · line 331

Quoted 9 lines
include/linux/bpf_verifier.h +4 -0
@@ -319,6 +324,10 @@ struct bpf_func_state {
 	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
 	 */
 	int allocated_stack;
+
+	u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+	u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+	struct bpf_stack_arg_state *stack_arg_slots;
 };

The new stack_arg_slots state is part of bpf_func_state and is properly copied in copy_stack_state() and freed in free_func_state(), but stacksafe() does not compare it during state pruning.

stacksafe() only iterates the regular stack:

kernel/bpf/verifier.c:stacksafe() {
    for (i = 0; i < old->allocated_stack; i++) {
        ...
        old->stack[spi].slot_type[...]
        ...
    }
    return true;
}

It never examines old->stack_arg_slots or cur->stack_arg_slots. This means func_states_equal() -> stacksafe() treats two states as equivalent even when they differ in their stack arg slot contents.

Could this allow the verifier to incorrectly prune a path where outgoing stack arg slots are uninitialized? For example, if a BPF program has two paths to a call instruction -- one that writes the outgoing stack args and one that does not -- the verifier could explore the initialized path first, save the state at a checkpoint, then prune the uninitialized path as equivalent. The initialization check in set_callee_state():

kernel/bpf/verifier.c:set_callee_state() {
    for (i = 0; i < callee_incoming_slots; i++) {
        int caller_spi = i + caller_incoming_slots;

        if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
            verbose(env, "stack arg#%d not properly initialized\n",
                i + 1 + MAX_BPF_FUNC_REG_ARGS);
            return -EINVAL;
        }
        callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
    }
}

would never run on the pruned path, potentially allowing the callee to receive uninitialized data.

It seems like stacksafe() needs a companion loop for stack_arg_slots, similar to how it walks the regular stack, to maintain verifier soundness for this new state.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767

author reviewer
#5 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 55 lines · On 4/1/26 8:18 PM, [email protected] wrote:
Quoted 4 lines
include/linux/bpf_verifier.h +0 -0
index 090aa26d1c98..a260610cd1c1 100644

[ ... ]

include/linux/bpf_verifier.h struct bpf_func_state { · line 331

Quoted 9 lines
include/linux/bpf_verifier.h +4 -0
@@ -319,6 +324,10 @@ struct bpf_func_state {
  	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
  	 */
  	int allocated_stack;
+
+	u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+	u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+	struct bpf_stack_arg_state *stack_arg_slots;
  };

The new stack_arg_slots state is part of bpf_func_state and is properly copied in copy_stack_state() and freed in free_func_state(), but stacksafe() does not compare it during state pruning.

stacksafe() only iterates the regular stack:

kernel/bpf/verifier.c:stacksafe() {
     for (i = 0; i < old->allocated_stack; i++) {
         ...
         old->stack[spi].slot_type[...]
         ...
     }
     return true;
}

It never examines old->stack_arg_slots or cur->stack_arg_slots. This means func_states_equal() -> stacksafe() treats two states as equivalent even when they differ in their stack arg slot contents.

Could this allow the verifier to incorrectly prune a path where outgoing stack arg slots are uninitialized? For example, if a BPF program has two paths to a call instruction -- one that writes the outgoing stack args and one that does not -- the verifier could explore the initialized path first, save the state at a checkpoint, then prune the uninitialized path as equivalent. The initialization check in set_callee_state():

kernel/bpf/verifier.c:set_callee_state() {
     for (i = 0; i < callee_incoming_slots; i++) {
         int caller_spi = i + caller_incoming_slots;

         if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
             verbose(env, "stack arg#%d not properly initialized\n",
                 i + 1 + MAX_BPF_FUNC_REG_ARGS);
             return -EINVAL;
         }
         callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
     }
}
Quoted 6 lines

would never run on the pruned path, potentially allowing the callee to receive uninitialized data.

It seems like stacksafe() needs a companion loop for stack_arg_slots, similar to how it walks the regular stack, to maintain verifier soundness for this new state.

You are absolutely correct. Indeed, we should do some checking for stack_arg_slots along side stacksafe(). I will fix it in the next revision.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767

#6 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

In reply to Yonghong Song
Quoted 277 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary

r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.

If the bpf function has more than one calls, e.g.,

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

The following is an illustration:

   Caller (foo)                           Callee (bar1)
   ============                           =============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||  ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+
   ...
   Back from bar1
   ...                                     Callee (bar2)
   ===                                     =============
                                     +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                     |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                     |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
   ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+|||  ...
   r12-32: [outgoing arg 7 to callee]-+||
   r12-40: [outgoing arg 8 to callee]--+|

r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |  15 ++-
 kernel/bpf/btf.c             |  14 +-
 kernel/bpf/verifier.c        | 248 ++++++++++++++++++++++++++++++++---
 4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
index e24c4a2e95f7..a0a1e14e4394 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
        u32 max_pkt_offset;
        u32 max_tp_access;
        u32 stack_depth;
+       u16 incoming_stack_arg_depth;
+       u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
        u32 id;
        u32 func_cnt; /* used by non-func prog as the number of func progs */
        u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +13 -2
index 090aa26d1c98..a260610cd1c1 100644
@@ -268,6 +268,11 @@ struct bpf_retval_range {
        bool return_32bit;
 };
 
+struct bpf_stack_arg_state {
+       struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+       u8 slot_type[BPF_REG_SIZE];
+};
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -319,6 +324,10 @@ struct bpf_func_state {
         * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
         */
        int allocated_stack;
+
+       u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+       u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+       struct bpf_stack_arg_state *stack_arg_slots;
 };
 
 #define MAX_CALL_FRAMES 8
@@ -674,10 +683,12 @@ struct bpf_subprog_info {
        bool keep_fastcall_stack: 1;
        bool changes_pkt_data: 1;
        bool might_sleep: 1;
-       u8 arg_cnt:3;
+       u8 arg_cnt:4;
 
        enum priv_stack_mode priv_stack_mode;
-       struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+       struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+       u16 incoming_stack_arg_depth;
+       u16 outgoing_stack_arg_depth;
 };
 
 struct bpf_verifier_env;
kernel/bpf/btf.c +10 -4
index a62d78581207..c5f3aa05d5a3 100644
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
        }
        args = (const struct btf_param *)(t + 1);
        nargs = btf_type_vlen(t);
-       if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-               if (!is_global)
-                       return -EINVAL;
-               bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+       if (nargs > MAX_BPF_FUNC_ARGS) {
+               bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+                       tname, nargs, MAX_BPF_FUNC_ARGS);
+               return -EINVAL;
+       }
+       if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+               bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
                        tname, nargs, MAX_BPF_FUNC_REG_ARGS);
                return -EINVAL;
        }
+       if (nargs > MAX_BPF_FUNC_REG_ARGS)
+               sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
        /* check that function is void or returns int, exception cb also requires this */
        t = btf_type_by_id(btf, t->type);
        while (btf_type_is_modifier(t))
kernel/bpf/verifier.c +76 -0
index 8c1cf2eb6cbb..d424fe611ef8 100644
@@ -1488,6 +1488,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
                return -ENOMEM;
 
        dst->allocated_stack = src->allocated_stack;
+
+       /* copy stack_arg_slots state */
+       n = src->stack_arg_depth / BPF_REG_SIZE;
+       if (n) {
+               dst->stack_arg_slots = copy_array(dst->stack_arg_slots, src->stack_arg_slots, n,
+                                                 sizeof(struct bpf_stack_arg_state),
+                                                 GFP_KERNEL_ACCOUNT);
+               if (!dst->stack_arg_slots)
+                       return -ENOMEM;
+
+               dst->stack_arg_depth = src->stack_arg_depth;
+               dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth;
+       }
        return 0;
 }
 
@@ -1529,6 +1542,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state
        return 0;
 }
 
+static int grow_stack_arg_slots(struct bpf_verifier_env *env,
+                               struct bpf_func_state *state, int size)
+{
+       size_t old_n = state->stack_arg_depth / BPF_REG_SIZE, n;
+
+       size = round_up(size, BPF_REG_SIZE);
+       n = size / BPF_REG_SIZE;
+       if (old_n >= n)
+               return 0;
+
+       state->stack_arg_slots = realloc_array(state->stack_arg_slots, old_n, n,
+                                              sizeof(struct bpf_stack_arg_state));
+       if (!state->stack_arg_slots)
+               return -ENOMEM;
+
+       state->stack_arg_depth = size;
+       return 0;
+}
+
 /* Acquire a pointer id from the env and update the state->refs to include
  * this new pointer reference.
  * On success, returns a valid pointer id to associate with the register
@@ -1699,6 +1731,7 @@ static void free_func_state(struct bpf_func_state *state)
 {
        if (!state)
                return;
+       kfree(state->stack_arg_slots);
        kfree(state->stack);
        kfree(state);
 }
@@ -5848,6 +5881,101 @@ static int check_stack_write(struct bpf_verifier_env *env,
        return err;
 }
 
+/* Validate that a stack arg access is 8-byte sized and aligned. */
+static int check_stack_arg_access(struct bpf_verifier_env *env,
+                                 struct bpf_insn *insn, const char *op)
+{
+       int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+
+       if (size != BPF_REG_SIZE) {
+               verbose(env, "stack arg %s must be %d bytes, got %d\n",
+                       op, BPF_REG_SIZE, size);
+               return -EINVAL;
+       }
+       if (insn->off % BPF_REG_SIZE) {
+               verbose(env, "stack arg %s offset %d not aligned to %d\n",
+                       op, insn->off, BPF_REG_SIZE);
+               return -EINVAL;
+       }
+       return 0;
+}
+
+/* Check that a stack arg slot has been properly initialized. */
+static bool is_stack_arg_slot_initialized(struct bpf_func_state *state, int spi)
+{
+       u8 type;
+
+       if (spi >= (int)(state->stack_arg_depth / BPF_REG_SIZE))
+               return false;
+       type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1];
+       return type == STACK_SPILL || type == STACK_MISC;
+}
+
+/*
+ * Write a value to the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                int off, int value_regno)
+{
+       int spi = (-off - 1) / BPF_REG_SIZE;
+       struct bpf_func_state *cur;
+       struct bpf_reg_state *reg;
+       int i, err;
+       u8 type;

kernel/bpf/verifier.c static int check_stack_write(struct bpf_verifier_env *env, · line 5936

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+
+       err = grow_stack_arg_slots(env, state, -off);
+       if (err)
+               return err;
+
+       cur = env->cur_state->frame[env->cur_state->curframe];
+       if (value_regno >= 0) {
+               reg = &cur->regs[value_regno];
+               state->stack_arg_slots[spi].spilled_ptr = *reg;
+               type = is_spillable_regtype(reg->type) ? STACK_SPILL : STACK_MISC;

It seems any spillable register can be passed to the callee, so reg containing ref_obj_id can be spilled to stack_arg_slots. However, release_reference() does not invalidate ref_obj_id in stack_arg_slots. Can this cause UAF like below?

a6 = bpf_task_acquire(t);
if (!a6)

        goto err;

// a6 now has a valid ref_obj_id
// foo1 calls bpf_task_release(a6);
foo1(a1, a2, a3, a4, a5, a6);

// a6 still has a valid ref_obj_id
// foo2 dereference a6 -> UAF
foo2(a1, a2, a3, a4, a5, a6);

Since stack_arg_slots is separated from the normal stack slots, other types of stale registers may exist in the outgoing stack slots. For example:
- stale pkt pointer after calling clear_all_pkt_pointers() in callee
- register with inprecise nullness after calling mark_ptr_or_null_regs() in callee

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+               for (i = 0; i < BPF_REG_SIZE; i++)
+                       state->stack_arg_slots[spi].slot_type[i] = type;
+       } else {
+               /* BPF_ST: store immediate, treat as scalar */
author reviewer
#7 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 109 lines · Amery Hung
Quoted 108 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
   }

The following is a illustration of stack allocation:

    Caller (foo)                           Callee (bar)
    ============                           ============
    r12-relative stack arg area:           r12-relative stack arg area:
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||   ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.

If the bpf function has more than one calls, e.g.,

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

    Caller (foo)                           Callee (bar1)
    ============                           =============
    r12-relative stack arg area:           r12-relative stack arg area:
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||  ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+
    ...
    Back from bar1
    ...                                     Callee (bar2)
    ===                                     =============
                                      +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                      |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                      |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
    ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+|||  ...
    r12-32: [outgoing arg 7 to callee]-+||
    r12-40: [outgoing arg 8 to callee]--+|
    r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

   [1] https://github.com/llvm/llvm-project/pull/189060
---
  include/linux/bpf.h          |   2 +
  include/linux/bpf_verifier.h |  15 ++-
  kernel/bpf/btf.c             |  14 +-
  kernel/bpf/verifier.c        | 248 ++++++++++++++++++++++++++++++++---
  4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
index e24c4a2e95f7..a0a1e14e4394 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
         u32 max_pkt_offset;
         u32 max_tp_access;
         u32 stack_depth;
+       u16 incoming_stack_arg_depth;
+       u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
         u32 id;
         u32 func_cnt; /* used by non-func prog as the number of func progs */
         u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +0 -0
index 090aa26d1c98..a260610cd1c1 100644
@@ -268,6 +268,11 @@ struct bpf_retval_range {
         bool return_32bit;
  };

...

Quoted 41 lines
Quoted 14 lines
include/linux/bpf_verifier.h +14 -0
+
+/*
+ * Write a value to the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                int off, int value_regno)
+{
+       int spi = (-off - 1) / BPF_REG_SIZE;
+       struct bpf_func_state *cur;
+       struct bpf_reg_state *reg;
+       int i, err;
+       u8 type;

include/linux/bpf_verifier.h

Quoted 10 lines
include/linux/bpf_verifier.h +10 -0
+
+       err = grow_stack_arg_slots(env, state, -off);
+       if (err)
+               return err;
+
+       cur = env->cur_state->frame[env->cur_state->curframe];
+       if (value_regno >= 0) {
+               reg = &cur->regs[value_regno];
+               state->stack_arg_slots[spi].spilled_ptr = *reg;
+               type = is_spillable_regtype(reg->type) ? STACK_SPILL : STACK_MISC;

It seems any spillable register can be passed to the callee, so reg containing ref_obj_id can be spilled to stack_arg_slots. However, release_reference() does not invalidate ref_obj_id in stack_arg_slots. Can this cause UAF like below?

a6 = bpf_task_acquire(t);
if (!a6)

         goto err;

// a6 now has a valid ref_obj_id
// foo1 calls bpf_task_release(a6);
foo1(a1, a2, a3, a4, a5, a6);

// a6 still has a valid ref_obj_id
// foo2 dereference a6 -> UAF
foo2(a1, a2, a3, a4, a5, a6);

Quoted 6 lines

Since stack_arg_slots is separated from the normal stack slots, other types of stale registers may exist in the outgoing stack slots. For example:
- stale pkt pointer after calling clear_all_pkt_pointers() in callee
- register with inprecise nullness after calling mark_ptr_or_null_regs() in callee

Thanks for pointing this out. Indeed, a lot of checking are needed for stack arguments. Looks like I need to add something like
bpf_for_each_spilled_stack_arg
in bpf_for_each_reg_in_vstate_mask for full coverage.

Will fix in the next resivion.

Quoted 6 lines
Quoted 6 lines
include/linux/bpf_verifier.h +4 -0
+               for (i = 0; i < BPF_REG_SIZE; i++)
+                       state->stack_arg_slots[spi].slot_type[i] = type;
+       } else {
+               /* BPF_ST: store immediate, treat as scalar */
#8 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

In reply to Yonghong Song
Quoted 408 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary

r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.

If the bpf function has more than one calls, e.g.,

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

The following is an illustration:

   Caller (foo)                           Callee (bar1)
   ============                           =============
   r12-relative stack arg area:           r12-relative stack arg area:
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||  ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+
   ...
   Back from bar1
   ...                                     Callee (bar2)
   ===                                     =============
                                     +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                     |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                     |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
   ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+|||  ...
   r12-32: [outgoing arg 7 to callee]-+||
   r12-40: [outgoing arg 8 to callee]--+|

r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

[1] https://github.com/llvm/llvm-project/pull/189060

---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |  15 ++-
 kernel/bpf/btf.c             |  14 +-
 kernel/bpf/verifier.c        | 248 ++++++++++++++++++++++++++++++++---
 4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
index e24c4a2e95f7..a0a1e14e4394 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
        u32 max_pkt_offset;
        u32 max_tp_access;
        u32 stack_depth;
+       u16 incoming_stack_arg_depth;
+       u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
        u32 id;
        u32 func_cnt; /* used by non-func prog as the number of func progs */
        u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
include/linux/bpf_verifier.h +13 -2
index 090aa26d1c98..a260610cd1c1 100644
@@ -268,6 +268,11 @@ struct bpf_retval_range {
        bool return_32bit;
 };
 
+struct bpf_stack_arg_state {
+       struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+       u8 slot_type[BPF_REG_SIZE];
+};
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -319,6 +324,10 @@ struct bpf_func_state {
         * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
         */
        int allocated_stack;
+
+       u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+       u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+       struct bpf_stack_arg_state *stack_arg_slots;
 };
 
 #define MAX_CALL_FRAMES 8
@@ -674,10 +683,12 @@ struct bpf_subprog_info {
        bool keep_fastcall_stack: 1;
        bool changes_pkt_data: 1;
        bool might_sleep: 1;
-       u8 arg_cnt:3;
+       u8 arg_cnt:4;
 
        enum priv_stack_mode priv_stack_mode;
-       struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+       struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+       u16 incoming_stack_arg_depth;
+       u16 outgoing_stack_arg_depth;
 };
 
 struct bpf_verifier_env;
kernel/bpf/btf.c +10 -4
index a62d78581207..c5f3aa05d5a3 100644
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
        }
        args = (const struct btf_param *)(t + 1);
        nargs = btf_type_vlen(t);
-       if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-               if (!is_global)
-                       return -EINVAL;
-               bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+       if (nargs > MAX_BPF_FUNC_ARGS) {
+               bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+                       tname, nargs, MAX_BPF_FUNC_ARGS);
+               return -EINVAL;
+       }
+       if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+               bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
                        tname, nargs, MAX_BPF_FUNC_REG_ARGS);
                return -EINVAL;
        }
+       if (nargs > MAX_BPF_FUNC_REG_ARGS)
+               sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
        /* check that function is void or returns int, exception cb also requires this */
        t = btf_type_by_id(btf, t->type);
        while (btf_type_is_modifier(t))
kernel/bpf/verifier.c +165 -0
index 8c1cf2eb6cbb..d424fe611ef8 100644
@@ -1488,6 +1488,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
                return -ENOMEM;
 
        dst->allocated_stack = src->allocated_stack;
+
+       /* copy stack_arg_slots state */
+       n = src->stack_arg_depth / BPF_REG_SIZE;
+       if (n) {
+               dst->stack_arg_slots = copy_array(dst->stack_arg_slots, src->stack_arg_slots, n,
+                                                 sizeof(struct bpf_stack_arg_state),
+                                                 GFP_KERNEL_ACCOUNT);
+               if (!dst->stack_arg_slots)
+                       return -ENOMEM;
+
+               dst->stack_arg_depth = src->stack_arg_depth;
+               dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth;
+       }
        return 0;
 }
 
@@ -1529,6 +1542,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state
        return 0;
 }
 
+static int grow_stack_arg_slots(struct bpf_verifier_env *env,
+                               struct bpf_func_state *state, int size)
+{
+       size_t old_n = state->stack_arg_depth / BPF_REG_SIZE, n;
+
+       size = round_up(size, BPF_REG_SIZE);
+       n = size / BPF_REG_SIZE;
+       if (old_n >= n)
+               return 0;
+
+       state->stack_arg_slots = realloc_array(state->stack_arg_slots, old_n, n,
+                                              sizeof(struct bpf_stack_arg_state));
+       if (!state->stack_arg_slots)
+               return -ENOMEM;
+
+       state->stack_arg_depth = size;
+       return 0;
+}
+
 /* Acquire a pointer id from the env and update the state->refs to include
  * this new pointer reference.
  * On success, returns a valid pointer id to associate with the register
@@ -1699,6 +1731,7 @@ static void free_func_state(struct bpf_func_state *state)
 {
        if (!state)
                return;
+       kfree(state->stack_arg_slots);
        kfree(state->stack);
        kfree(state);
 }
@@ -5848,6 +5881,101 @@ static int check_stack_write(struct bpf_verifier_env *env,
        return err;
 }
 
+/* Validate that a stack arg access is 8-byte sized and aligned. */
+static int check_stack_arg_access(struct bpf_verifier_env *env,
+                                 struct bpf_insn *insn, const char *op)
+{
+       int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+
+       if (size != BPF_REG_SIZE) {
+               verbose(env, "stack arg %s must be %d bytes, got %d\n",
+                       op, BPF_REG_SIZE, size);
+               return -EINVAL;
+       }
+       if (insn->off % BPF_REG_SIZE) {
+               verbose(env, "stack arg %s offset %d not aligned to %d\n",
+                       op, insn->off, BPF_REG_SIZE);
+               return -EINVAL;
+       }
+       return 0;
+}
+
+/* Check that a stack arg slot has been properly initialized. */
+static bool is_stack_arg_slot_initialized(struct bpf_func_state *state, int spi)
+{
+       u8 type;
+
+       if (spi >= (int)(state->stack_arg_depth / BPF_REG_SIZE))
+               return false;
+       type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1];
+       return type == STACK_SPILL || type == STACK_MISC;
+}
+
+/*
+ * Write a value to the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                int off, int value_regno)
+{
+       int spi = (-off - 1) / BPF_REG_SIZE;
+       struct bpf_func_state *cur;
+       struct bpf_reg_state *reg;
+       int i, err;
+       u8 type;
+
+       err = grow_stack_arg_slots(env, state, -off);
+       if (err)
+               return err;
+
+       cur = env->cur_state->frame[env->cur_state->curframe];
+       if (value_regno >= 0) {
+               reg = &cur->regs[value_regno];
+               state->stack_arg_slots[spi].spilled_ptr = *reg;
+               type = is_spillable_regtype(reg->type) ? STACK_SPILL : STACK_MISC;
+               for (i = 0; i < BPF_REG_SIZE; i++)
+                       state->stack_arg_slots[spi].slot_type[i] = type;
+       } else {
+               /* BPF_ST: store immediate, treat as scalar */
+               reg = &state->stack_arg_slots[spi].spilled_ptr;
+               reg->type = SCALAR_VALUE;
+               __mark_reg_known(reg, (u32)env->prog->insnsi[env->insn_idx].imm);
+               for (i = 0; i < BPF_REG_SIZE; i++)
+                       state->stack_arg_slots[spi].slot_type[i] = STACK_MISC;
+       }
+       return 0;
+}
+
+/*
+ * Read a value from the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                               int off, int dst_regno)
+{
+       int spi = (-off - 1) / BPF_REG_SIZE;
+       struct bpf_func_state *cur;
+       u8 *stype;
+
+       if (-off > state->stack_arg_depth) {
+               verbose(env, "invalid read from stack arg off %d depth %d\n",
+                       off, state->stack_arg_depth);
+               return -EACCES;
+       }
+
+       stype = state->stack_arg_slots[spi].slot_type;
+       cur = env->cur_state->frame[env->cur_state->curframe];
+
+       if (stype[BPF_REG_SIZE - 1] == STACK_SPILL)
+               copy_register_state(&cur->regs[dst_regno],
+                                   &state->stack_arg_slots[spi].spilled_ptr);
+       else
+               mark_reg_unknown(env, cur->regs, dst_regno);
+       return 0;
+}
+
 static int check_map_access_type(struct bpf_verifier_env *env, u32 regno,
                                 int off, int size, enum bpf_access_type type)
 {
@@ -8022,10 +8150,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
                          bool strict_alignment_once, bool is_ldsx,
                          bool allow_trust_mismatch, const char *ctx)
 {
+       struct bpf_verifier_state *vstate = env->cur_state;
+       struct bpf_func_state *state = vstate->frame[vstate->curframe];
        struct bpf_reg_state *regs = cur_regs(env);
        enum bpf_reg_type src_reg_type;
        int err;
 
+       /* Handle stack arg access */
+       if (insn->src_reg == BPF_REG_STACK_ARG_BASE) {
+               err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+               if (err)
+                       return err;
+               err = check_stack_arg_access(env, insn, "read");
+               if (err)
+                       return err;
+               return check_stack_arg_read(env, state, insn->off, insn->dst_reg);
+       }
+
        /* check src operand */
        err = check_reg_arg(env, insn->src_reg, SRC_OP);
        if (err)
@@ -8054,10 +8195,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
 static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
                           bool strict_alignment_once)
 {
+       struct bpf_verifier_state *vstate = env->cur_state;
+       struct bpf_func_state *state = vstate->frame[vstate->curframe];
        struct bpf_reg_state *regs = cur_regs(env);
        enum bpf_reg_type dst_reg_type;
        int err;
 
+       /* Handle stack arg write */
+       if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+               err = check_reg_arg(env, insn->src_reg, SRC_OP);
+               if (err)
+                       return err;
+               err = check_stack_arg_access(env, insn, "write");
+               if (err)
+                       return err;
+               return check_stack_arg_write(env, state, insn->off, insn->src_reg);
+       }
+
        /* check src1 operand */
        err = check_reg_arg(env, insn->src_reg, SRC_OP);
        if (err)
@@ -10940,8 +11094,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                           int *insn_idx)
 {
        struct bpf_verifier_state *state = env->cur_state;
+       struct bpf_subprog_info *caller_info;
        struct bpf_func_state *caller;
        int err, subprog, target_insn;
+       u16 callee_incoming;
 
        target_insn = *insn_idx + insn->imm + 1;
        subprog = find_subprog(env, target_insn);
@@ -10993,6 +11149,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                return 0;
        }
 
+       /*
+        * Track caller's outgoing stack arg depth (max across all callees).
+        * This is needed so the JIT knows how much stack arg space to allocate.
+        */
+       caller_info = &env->subprog_info[caller->subprogno];
+       callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth;
+       if (callee_incoming > caller_info->outgoing_stack_arg_depth)
+               caller_info->outgoing_stack_arg_depth = callee_incoming;
+
        /* for regular function entry setup new frame and continue
         * from that frame.
         */

kernel/bpf/verifier.c static int set_callee_state(struct bpf_verifier_env *env, · line 11215

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
@@ -11048,13 +11213,41 @@ static int set_callee_state(struct bpf_verifier_env *env,
                            struct bpf_func_state *caller,
                            struct bpf_func_state *callee, int insn_idx)
 {

Taking note when reading the change to set_callee_state():

The function is not called when handling callback function, which uses push_callback_call() -> setup_func_entry() -> callback specific set_callee_state_cb. So caller stack argument will not be transferred.

This should be fine as callee's stack_arg_depth will remain zero and then when callee tries to do r12 based load, check_stack_arg_read() should reject the program. Not sure if this needs a selftest since callbacks' set_callee_state_cb will also transfer register state very intentionally.

Quoted 121 lines
kernel/bpf/verifier.c +67 -17
-       int i;
+       struct bpf_subprog_info *callee_info;
+       int i, err;
 
        /* copy r1 - r5 args that callee can access.  The copy includes parent
         * pointers, which connects us up to the liveness chain
         */
        for (i = BPF_REG_1; i <= BPF_REG_5; i++)
                callee->regs[i] = caller->regs[i];
+
+       /*
+        * Transfer stack args from caller's outgoing area to callee's incoming area.
+        * Caller wrote outgoing args at offsets '-(incoming + 8)', '-(incoming + 16)', ...
+        * These outgoing args will go to callee's incoming area.
+        */
+       callee_info = &env->subprog_info[callee->subprogno];
+       if (callee_info->incoming_stack_arg_depth) {
+               int caller_incoming_slots = caller->incoming_stack_arg_depth / BPF_REG_SIZE;
+               int callee_incoming_slots = callee_info->incoming_stack_arg_depth / BPF_REG_SIZE;
+
+               callee->incoming_stack_arg_depth = callee_info->incoming_stack_arg_depth;
+               err = grow_stack_arg_slots(env, callee, callee_info->incoming_stack_arg_depth);
+               if (err)
+                       return err;
+
+               for (i = 0; i < callee_incoming_slots; i++) {
+                       int caller_spi = i + caller_incoming_slots;
+
+                       if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
+                               verbose(env, "stack arg#%d not properly initialized\n",
+                                       i + 1 + MAX_BPF_FUNC_REG_ARGS);
+                               return -EINVAL;
+                       }
+                       callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
+               }
+       }
        return 0;
 }
 
@@ -21262,23 +21455,37 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
                        verbose(env, "BPF_ST uses reserved fields\n");
                        return -EINVAL;
                }
-               /* check src operand */
-               err = check_reg_arg(env, insn->dst_reg, SRC_OP);
-               if (err)
-                       return err;
 
-               dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+               /* Handle stack arg write (store immediate) */
+               if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+                       struct bpf_verifier_state *vstate = env->cur_state;
+                       struct bpf_func_state *state = vstate->frame[vstate->curframe];
 
-               /* check that memory (dst_reg + off) is writeable */
-               err = check_mem_access(env, env->insn_idx, insn->dst_reg,
-                                      insn->off, BPF_SIZE(insn->code),
-                                      BPF_WRITE, -1, false, false);
-               if (err)
-                       return err;
+                       err = check_stack_arg_access(env, insn, "write");
+                       if (err)
+                               return err;
+                       err = check_stack_arg_write(env, state, insn->off, -1);
+                       if (err)
+                               return err;
+               } else {
+                       /* check src operand */
+                       err = check_reg_arg(env, insn->dst_reg, SRC_OP);
+                       if (err)
+                               return err;
 
-               err = save_aux_ptr_type(env, dst_reg_type, false);
-               if (err)
-                       return err;
+                       dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+
+                       /* check that memory (dst_reg + off) is writeable */
+                       err = check_mem_access(env, env->insn_idx, insn->dst_reg,
+                                              insn->off, BPF_SIZE(insn->code),
+                                              BPF_WRITE, -1, false, false);
+                       if (err)
+                               return err;
+
+                       err = save_aux_ptr_type(env, dst_reg_type, false);
+                       if (err)
+                               return err;
+               }
        } else if (class == BPF_JMP || class == BPF_JMP32) {
                u8 opcode = BPF_OP(insn->code);
 
@@ -22974,8 +23181,14 @@ static int jit_subprogs(struct bpf_verifier_env *env)
        int err, num_exentries;
        int old_len, subprog_start_adjustment = 0;
 
-       if (env->subprog_cnt <= 1)
+       if (env->subprog_cnt <= 1) {
+               /*
+                * Even without subprogs, kfunc calls with >5 args need stack arg space
+                * allocated by the root program.
+                */
+               prog->aux->stack_arg_depth = env->subprog_info[0].outgoing_stack_arg_depth;
                return 0;
+       }
 
        for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
                if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
@@ -23065,6 +23278,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 
                func[i]->aux->name[0] = 'F';
                func[i]->aux->stack_depth = env->subprog_info[i].stack_depth;
+               func[i]->aux->incoming_stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth;
+               func[i]->aux->stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth +
+                                               env->subprog_info[i].outgoing_stack_arg_depth;
                if (env->subprog_info[i].priv_stack_mode == PRIV_STACK_ADAPTIVE)
                        func[i]->aux->jits_use_priv_stack = true;
 
--

2.52.0

author reviewer
#9 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 102 lines · Amery Hung
Quoted 101 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
   }

The following is a illustration of stack allocation:

    Caller (foo)                           Callee (bar)
    ============                           ============
    r12-relative stack arg area:           r12-relative stack arg area:
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||   ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+

The caller writes outgoing args past its own incoming area. At the call site, the verifier transfers the caller's outgoing slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10 stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated 'stack_arg_slots' array in bpf_func_state. This separation keeps the stack arg area from interfering with the normal stack and frame pointer (r10) bookkeeping.

If the bpf function has more than one calls, e.g.,

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

    Caller (foo)                           Callee (bar1)
    ============                           =============
    r12-relative stack arg area:           r12-relative stack arg area:
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||  ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+
    ...
    Back from bar1
    ...                                     Callee (bar2)
    ===                                     =============
                                      +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                      |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                      |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
    ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+|||  ...
    r12-32: [outgoing arg 7 to callee]-+||
    r12-40: [outgoing arg 8 to callee]--+|
    r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

   [1] https://github.com/llvm/llvm-project/pull/189060
---
  include/linux/bpf.h          |   2 +
  include/linux/bpf_verifier.h |  15 ++-
  kernel/bpf/btf.c             |  14 +-
  kernel/bpf/verifier.c        | 248 ++++++++++++++++++++++++++++++++---
  4 files changed, 257 insertions(+), 22 deletions(-)
include/linux/bpf.h +2 -0
index e24c4a2e95f7..a0a1e14e4394 100644
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
         u32 max_pkt_offset;
         u32 max_tp_access;
         u32 stack_depth;
+       u16 incoming_stack_arg_depth;
+       u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
         u32 id;
         u32 func_cnt; /* used by non-func prog as the number of func progs */
         u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */

[...]

Quoted 61 lines
Quoted 52 lines
include/linux/bpf.h +24 -0
@@ -8054,10 +8195,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
  static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
                            bool strict_alignment_once)
  {
+       struct bpf_verifier_state *vstate = env->cur_state;
+       struct bpf_func_state *state = vstate->frame[vstate->curframe];
         struct bpf_reg_state *regs = cur_regs(env);
         enum bpf_reg_type dst_reg_type;
         int err;
 
+       /* Handle stack arg write */
+       if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+               err = check_reg_arg(env, insn->src_reg, SRC_OP);
+               if (err)
+                       return err;
+               err = check_stack_arg_access(env, insn, "write");
+               if (err)
+                       return err;
+               return check_stack_arg_write(env, state, insn->off, insn->src_reg);
+       }
+
         /* check src1 operand */
         err = check_reg_arg(env, insn->src_reg, SRC_OP);
         if (err)
@@ -10940,8 +11094,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                            int *insn_idx)
  {
         struct bpf_verifier_state *state = env->cur_state;
+       struct bpf_subprog_info *caller_info;
         struct bpf_func_state *caller;
         int err, subprog, target_insn;
+       u16 callee_incoming;
 
         target_insn = *insn_idx + insn->imm + 1;
         subprog = find_subprog(env, target_insn);
@@ -10993,6 +11149,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                 return 0;
         }
 
+       /*
+        * Track caller's outgoing stack arg depth (max across all callees).
+        * This is needed so the JIT knows how much stack arg space to allocate.
+        */
+       caller_info = &env->subprog_info[caller->subprogno];
+       callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth;
+       if (callee_incoming > caller_info->outgoing_stack_arg_depth)
+               caller_info->outgoing_stack_arg_depth = callee_incoming;
+
         /* for regular function entry setup new frame and continue
          * from that frame.
          */

include/linux/bpf.h static int set_callee_state(struct bpf_verifier_env *env, · line 11215

Quoted 3 lines
include/linux/bpf.h +0 -0
@@ -11048,13 +11213,41 @@ static int set_callee_state(struct bpf_verifier_env *env,
                             struct bpf_func_state *caller,
                             struct bpf_func_state *callee, int insn_idx)
  {

Taking note when reading the change to set_callee_state():

The function is not called when handling callback function, which uses push_callback_call() -> setup_func_entry() -> callback specific set_callee_state_cb. So caller stack argument will not be transferred.

Quoted 5 lines

This should be fine as callee's stack_arg_depth will remain zero and then when callee tries to do r12 based load, check_stack_arg_read() should reject the program. Not sure if this needs a selftest since callbacks' set_callee_state_cb will also transfer register state very intentionally.

All callback functions are carefully designed in kernel. So far all callback functions are within 5 register parameters. I ignore them for now. If in the future, there is a need for callback functions with more than 5 arguments, we can deal with them at that time.

Quoted 10 lines
Quoted 10 lines
include/linux/bpf.h +2 -1
-       int i;
+       struct bpf_subprog_info *callee_info;
+       int i, err;
 
         /* copy r1 - r5 args that callee can access.  The copy includes parent
          * pointers, which connects us up to the liveness chain
          */
         for (i = BPF_REG_1; i <= BPF_REG_5; i++)
                 callee->regs[i] = caller->regs[i];

[...]

#10 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

In reply to Yonghong Song
Quoted 21 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:
Quoted 7 lines
   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary

r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+

I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?

author reviewer
#11 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 28 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
   }

The following is a illustration of stack allocation:

    Caller (foo)                           Callee (bar)
    ============                           ============
    r12-relative stack arg area:           r12-relative stack arg area:
Quoted 7 lines
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||   ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+
Quoted 6 lines

I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?

This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.

#12 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 38 lines · Yonghong Song
Quoted 28 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     bar(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
   }

The following is a illustration of stack allocation:

    Caller (foo)                           Callee (bar)
    ============                           ============
    r12-relative stack arg area:           r12-relative stack arg area:
Quoted 7 lines
    r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
    r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                      ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
    ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
    r12-24: [outgoing arg 6 to callee]+||   ...
    r12-32: [outgoing arg 7 to callee]-+|
    r12-40: [outgoing arg 8 to callee]--+
Quoted 6 lines

I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?

Quoted 3 lines

This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.

There shouldn't be any extra copy.
For 7th and higher argument:
the caller does 'stx [r12 - N]' and the callee does 'ldx [r12 + M]' and JIT emits them as-is only adjusting N and M constants.

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

author reviewer
#13 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 38 lines · Alexei Starovoitov
Quoted 37 lines · Yonghong Song
Quoted 34 lines · Alexei Starovoitov
Quoted 20 lines · Yonghong Song

Currently BPF functions (subprogs) are limited to 5 register arguments. With [1], the compiler can emit code that passes additional arguments via a dedicated stack area through bpf register BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved and transferred between caller and callee:

    int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
      ...
      bar(a1, a2, a3, a4, a5, a6, a7, a8);
      ...
    }

The following is a illustration of stack allocation:

     Caller (foo)                           Callee (bar)
     ============                           ============
     r12-relative stack arg area:           r12-relative stack arg area:
Quoted 7 lines
     r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
     r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                       ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
     ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
     r12-24: [outgoing arg 6 to callee]+||   ...
     r12-32: [outgoing arg 7 to callee]-+|
     r12-40: [outgoing arg 8 to callee]--+

I haven't looked at the patch itself only at this diagram. How does it suppose to map to x86 calling convention? The shift is unusual.
x86 is using fp-N for outgoing and fp+N for incoming. Why can't we use the same?

Quoted 7 lines
Quoted 3 lines

This is not for jit. The above transfer is for verification purpose. For example, for callee, a load 'rX = *(u64 *)(r12 - 8)' can easily get the value rX in callee since the value is copied from caller to callee.

There shouldn't be any extra copy.
For 7th and higher argument:
the caller does 'stx [r12 - N]' and the callee does 'ldx [r12 + M]' and JIT emits them as-is only adjusting N and M constants.

Indeed, for 7th and higher arguments, ldx and stx and directly get or store values in expected calling-convention places.

Quoted 2 lines

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

For ldx case, this is a bpf callee to access a bpf caller. In such cases, r9 is not involved.

#14 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 7 lines · Yonghong Song
Quoted 4 lines

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.

author reviewer
#15 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 8 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
Quoted 2 lines

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.

The following stack layout can avoid the above push/pop issue:

   incoming stack arg N -> 1
   return adderss
   saved rbp
   BPF program stack
   tail call cnt <== if tail call reachable
   callee-saved regs
   r9 <== if priv_frame_ptr is not null
outgoing stack arg 1
   outgoing stack arg M -> 2

After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.

#16 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 13 lines · Yonghong Song
Quoted 8 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
Quoted 2 lines

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.

Quoted 7 lines

The following stack layout can avoid the above push/pop issue:

    incoming stack arg N -> 1
    return adderss
    saved rbp
    BPF program stack
    tail call cnt <== if tail call reachable

let's disallow mixing 6+ args and tailcalls.

Quoted 2 lines
    callee-saved regs
    r9 <== if priv_frame_ptr is not null

I guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N], src_reg // store of outgoing 6th arg call

For priv stack JIT will emit push_r9 while JITing call insn. But it should push_r9 before stx converts to 'mov %r9 <- %src_reg'. Maybe we can reload r9 after the call instead of push/pop.

Quoted 6 lines

outgoing stack arg 1
outgoing stack arg M -> 2

After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.

Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',

so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.

It feels to me that it's easier to disallow priv stack for now.

author reviewer
#17 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 11 lines · Alexei Starovoitov
Quoted 10 lines · Yonghong Song
Quoted 7 lines · Alexei Starovoitov
Quoted 3 lines · Yonghong Song
Quoted 1 line

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.

Quoted 8 lines
Quoted 7 lines

The following stack layout can avoid the above push/pop issue:

     incoming stack arg N -> 1
     return adderss
     saved rbp
     BPF program stack
     tail call cnt <== if tail call reachable

let's disallow mixing 6+ args and tailcalls.

Okay.

Quoted 11 lines
Quoted 3 lines
     callee-saved regs
     r9 <== if priv_frame_ptr is not null

I guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N], src_reg // store of outgoing 6th arg call

For priv stack JIT will emit push_r9 while JITing call insn. But it should push_r9 before stx converts to 'mov %r9 <- %src_reg'.

I guess this is not enough. we can push_r9 in prologue. But for priv stack, r9 is still used in the normal bpf load/store codes. This will interfere with jitting 'mov %r9 <- %src_reg'.

As you mentioned later, disable priv_stack would work.

Quoted 13 lines

Maybe we can reload r9 after the call instead of push/pop.

Quoted 6 lines

outgoing stack arg 1
outgoing stack arg M -> 2

After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.

Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',

Quoted 7 lines

so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.

It feels to me that it's easier to disallow priv stack for now.

So if we disallow priv_stack, we can do
stx[r12 - first_arg_off] = val/reg => mov %r9, val/reg
other arg_off will be pushed to stack in reverse orders.

So for kfunc, 'mov %r9, val' already there, so we should be okay. For bpf-to-bpf, do 'push %r9'.

Is this correct?

#18 lore raw

Re: [PATCH bpf-next 03/10] bpf: Support stack arguments for bpf functions

Quoted 63 lines · Yonghong Song
Quoted 11 lines · Alexei Starovoitov
Quoted 10 lines · Yonghong Song
Quoted 7 lines · Alexei Starovoitov
Quoted 3 lines · Yonghong Song
Quoted 1 line

For 6th argument JIT emits these two stx/ldx as moves to/from x86's r9.

For stx case, we should move the bottom stack value (6th argument) to r9 and pop the bottom stack slot (8 bytes).

That doesn't sound right.
Passing 6th argument should not involve stack manipulation at all. No push and no pop.

Quoted 8 lines
Quoted 7 lines

The following stack layout can avoid the above push/pop issue:

     incoming stack arg N -> 1
     return adderss
     saved rbp
     BPF program stack
     tail call cnt <== if tail call reachable

let's disallow mixing 6+ args and tailcalls.

Okay.

Quoted 11 lines
Quoted 3 lines
     callee-saved regs
     r9 <== if priv_frame_ptr is not null

I guess we can also disable private stack and 6+ args. Looks like it's a bit in the way.
The compilers will emit
stx [r12 - N], src_reg // store of outgoing 6th arg call

For priv stack JIT will emit push_r9 while JITing call insn. But it should push_r9 before stx converts to 'mov %r9 <- %src_reg'.

I guess this is not enough. we can push_r9 in prologue. But for priv stack, r9 is still used in the normal bpf load/store codes. This will interfere with jitting 'mov %r9 <- %src_reg'.

As you mentioned later, disable priv_stack would work.

Quoted 19 lines

Maybe we can reload r9 after the call instead of push/pop.

Quoted 6 lines

outgoing stack arg 1
outgoing stack arg M -> 2

After the above pushing (outgoing stack arg M -> 2), if bpf-to-bpf, push the outgoing stack arg 1. If kfunc, move outgoing stack arg 1 to r9.

Ideally JIT treats subprog calls and kfunc calls the same way. Why should they be different ?
In the callee, in both cases, the first insn in the prologue can be 'mov [rbp + ] <- %r9',

so that later 'ldx [r12 + ' can be JITed as-is. Here priv stack is not in the way.
Without priv stack we can avoid this first 'mov [rbp + ] <- %r9' in the prologue and instead JIT 'ldx [r12 + ' as 'mov %dst_reg <- %r9' since nothing will be overwriting %r9.

Quoted 4 lines
Quoted 1 line

It feels to me that it's easier to disallow priv stack for now.

So if we disallow priv_stack, we can do

    stx[r12 - first_arg_off] = val/reg => mov %r9, val/reg

yes.

Quoted 1 line

other arg_off will be pushed to stack in reverse orders.

no. Not pushed. they will be JITed in place stx[r12 -off] -> mov[%rbp], val Probably needs add %rsp somewhere to grow the stack.

Quoted 2 lines

So for kfunc, 'mov %r9, val' already there, so we should be okay. For bpf-to-bpf, do 'push %r9'.

Don't follow. why should they be different?

author reviewer
#19 lore raw

PATCH 4/10 bpf: Support stack arguments for kfunc calls

In reply to Yonghong Song

Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.

For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.

The following is an example to show how stack arguments are saved:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

Caller (foo)
============

       r12-relative stack arg area:

       r12-8:  [incoming arg 6]
       r12-16: [incoming arg 7]

       ---- incoming/outgoing boundary (kfunc1)
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       ...
       Back from kfunc1
       ...

       ---- incoming/outgoing boundary
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       r12-48: [outgoing arg 9 to callee]

Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().

In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. Also in fixup_kfunc_call(), repurpose insn->off (no longer needed after kfunc address resolution) to store the number of stack args, so the JIT knows how many args to marshal.

---
 kernel/bpf/verifier.c | 97 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 80 insertions(+), 17 deletions(-)
kernel/bpf/verifier.c +80 -17
index d424fe611ef8..6579156486b8 100644
@@ -3502,7 +3502,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	struct bpf_kfunc_meta kfunc;
 	struct bpf_kfunc_desc *desc;
 	unsigned long addr;
-	int err;
+	int i, err;
 
 	prog_aux = env->prog->aux;
 	tab = prog_aux->kfunc_tab;
@@ -3578,6 +3578,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	if (err)
 		return err;
 
+	for (i = MAX_BPF_FUNC_REG_ARGS; i < func_model.nr_args; i++) {
+		if (func_model.arg_size[i] > sizeof(u64)) {
+			verbose(env, "kfunc %s arg#%d size %d > %zu not supported for stack args\n",
+				kfunc.name, i, func_model.arg_size[i], sizeof(u64));
+			return -EINVAL;
+		}
+	}
+
 	desc = &tab->descs[tab->nr_descs++];
 	desc->func_id = func_id;
 	desc->offset = offset;
@@ -12995,9 +13003,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		       struct bpf_kfunc_call_arg_meta *meta,
 		       const struct btf_type *t, const struct btf_type *ref_t,
 		       const char *ref_tname, const struct btf_param *args,
-		       int argno, int nargs)
+		       int argno, int nargs, u32 regno)
 {
-	u32 regno = argno + 1;
 	struct bpf_reg_state *regs = cur_regs(env);
 	struct bpf_reg_state *reg = &regs[regno];
 	bool arg_mem_size = false;
@@ -13677,9 +13684,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+	if (nargs > MAX_BPF_FUNC_ARGS) {
 		verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
-			MAX_BPF_FUNC_REG_ARGS);
+			MAX_BPF_FUNC_ARGS);
 		return -EINVAL;
 	}
 
@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 	 * verifier sees.
 	 */
 	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *regs = cur_regs(env), *reg;
+		struct bpf_reg_state saved_reg;
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
 		u32 regno = i + 1, ref_id, type_size;
 		bool is_ret_buf_sz = false;
+		bool is_stack_arg = false;
 		int kf_arg_type;
 
+		if (i < MAX_BPF_FUNC_REG_ARGS) {
+			reg = &regs[i + 1];
+		} else {
+			/*
+			 * Retrieve the spilled reg state from the stack arg slot.
+			 * Reuse the existing type checking infrastructure which
+			 * reads from cur_regs(env)[regno], temporarily copy the
+			 * stack arg reg state into regs[BPF_REG_1] and restore
+			 * it after checking.
+			 */
+			struct bpf_func_state *caller = cur_func(env);
+			int spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE +
+				  (i - MAX_BPF_FUNC_REG_ARGS);
+
+			if (!is_stack_arg_slot_initialized(caller, spi)) {
+				verbose(env, "stack arg#%d not properly initialized\n", i);
+				return -EINVAL;
+			}
+
+			is_stack_arg = true;
+			regno = BPF_REG_1;
+			saved_reg = regs[BPF_REG_1];
+			regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+			reg = &regs[BPF_REG_1];
+		}
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
 			/* Reject repeated use bpf_prog_aux */
 			if (meta->arg_prog) {
@@ -13702,7 +13737,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			}
 			meta->arg_prog = true;
 			cur_aux(env)->arg_prog = regno;
-			continue;
+			goto next_arg;
 		}
 
 		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
@@ -13725,9 +13760,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					verbose(env, "R%d must be a known constant\n", regno);
 					return -EINVAL;
 				}
-				ret = mark_chain_precision(env, regno);
-				if (ret < 0)
-					return ret;
+				if (i < MAX_BPF_FUNC_REG_ARGS) {
+					ret = mark_chain_precision(env, regno);
+					if (ret < 0)
+						return ret;
+				}
 				meta->arg_constant.found = true;
 				meta->arg_constant.value = reg->var_off.value;
 			} else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) {
@@ -13749,11 +13786,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				}
 
 				meta->r0_size = reg->var_off.value;
-				ret = mark_chain_precision(env, regno);
-				if (ret)
-					return ret;
+				if (i < MAX_BPF_FUNC_REG_ARGS) {
+					ret = mark_chain_precision(env, regno);
+					if (ret)
+						return ret;
+				}
 			}
-			continue;
+			goto next_arg;
 		}
 
 		if (!btf_type_is_ptr(t)) {
@@ -13782,13 +13821,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
 		ref_tname = btf_name_by_offset(btf, ref_t->name_off);
 
-		kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs);
+		kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs,
+						     regno);
 		if (kf_arg_type < 0)
 			return kf_arg_type;
 
 		switch (kf_arg_type) {
 		case KF_ARG_PTR_TO_NULL:
-			continue;
+			goto next_arg;
 		case KF_ARG_PTR_TO_MAP:
 			if (!reg->map_ptr) {
 				verbose(env, "pointer in R%d isn't map pointer\n", regno);
@@ -14201,6 +14241,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			break;
 		}
 		}
+next_arg:
+		if (is_stack_arg)
+			regs[BPF_REG_1] = saved_reg;
 	}
 
 	if (is_kfunc_release(meta) && !meta->release_regno) {
@@ -14778,7 +14821,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	nargs = btf_type_vlen(meta.func_proto);
 	args = (const struct btf_param *)(meta.func_proto + 1);
-	for (i = 0; i < nargs; i++) {
+	for (i = 0; i < nargs && i < MAX_BPF_FUNC_REG_ARGS; i++) {
 		u32 regno = i + 1;
 
 		t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
@@ -14789,6 +14832,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			mark_btf_func_reg_size(env, regno, t->size);
 	}
 
+	/* Track outgoing stack arg depth for kfuncs with >5 args */
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		struct bpf_func_state *caller = cur_func(env);
+		struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+		u16 kfunc_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
+		if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth)
+			caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth;
+	}
+
 	if (is_iter_next_kfunc(&meta)) {
 		err = process_iter_next_call(env, insn_idx, &meta);
 		if (err)
@@ -23615,6 +23668,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	if (!bpf_jit_supports_far_kfunc_call())
 		insn->imm = BPF_CALL_IMM(desc->addr);
 
+	/*
+	 * After resolving the kfunc address, insn->off is no longer needed
+	 * for BTF fd index. Repurpose it to store the number of stack args
+	 * so the JIT can marshal them.
+	 */
+	if (desc->func_model.nr_args > MAX_BPF_FUNC_REG_ARGS)
+		insn->off = desc->func_model.nr_args - MAX_BPF_FUNC_REG_ARGS;
+	else
+		insn->off = 0;
+
 	if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
 		struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
 		struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };
Signature · 1 line
2.52.0
#20 lore raw

Re: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls

The is_kfunc_arg_prog_aux path above was converted from continue to goto next_arg so that regs[BPF_REG_1] gets restored via the next_arg label. The scalar path and the KF_ARG_PTR_TO_NULL path were…
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index d424fe611ef8..6579156486b8 100644

[ ... ]

Quoted 7 lines
kernel/bpf/verifier.c +2 -1
@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 	 * verifier sees.
 	 */
 	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *regs = cur_regs(env), *reg;
+		struct bpf_reg_state saved_reg;

[ ... ]

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		if (i < MAX_BPF_FUNC_REG_ARGS) {
+			reg = &regs[i + 1];
+		} else {
+			/*
+			 * Retrieve the spilled reg state from the stack arg slot.
+			 * Reuse the existing type checking infrastructure which
+			 * reads from cur_regs(env)[regno], temporarily copy the
+			 * stack arg reg state into regs[BPF_REG_1] and restore
+			 * it after checking.
+			 */

[ ... ]

Quoted 8 lines
kernel/bpf/verifier.c +7 -0
+			is_stack_arg = true;
+			regno = BPF_REG_1;
+			saved_reg = regs[BPF_REG_1];
+			regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+			reg = &regs[BPF_REG_1];
+		}
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {

[ ... ]

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
-			continue;
+			goto next_arg;
 		}
 
 		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
 			continue;

The is_kfunc_arg_prog_aux path above was converted from continue to goto next_arg so that regs[BPF_REG_1] gets restored via the next_arg label. The scalar path and the KF_ARG_PTR_TO_NULL path were also converted the same way.

Should this continue for is_kfunc_arg_ignore/is_kfunc_arg_implicit also be goto next_arg? When a stack arg (i >= MAX_BPF_FUNC_REG_ARGS) hits this continue, is_stack_arg is true but the next_arg restore is skipped, so regs[BPF_REG_1] keeps the spilled pointer from the current iteration. On the next iteration, saved_reg then captures that stale value instead of the original register state.

The comment above says "restore it after checking", which this continue path does not do. The impact is currently masked because each iteration independently overwrites regs[BPF_REG_1] and the caller_saved clearing in check_kfunc_call() resets R1 after the loop, but it breaks the save/restore invariant the rest of this patch establishes.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767

author reviewer
#21 lore raw

Re: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls

Quoted 48 lines · On 4/1/26 8:18 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index d424fe611ef8..6579156486b8 100644

[ ... ]

Quoted 7 lines
kernel/bpf/verifier.c +2 -1
@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
  	 * verifier sees.
  	 */
  	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *regs = cur_regs(env), *reg;
+		struct bpf_reg_state saved_reg;

[ ... ]

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		if (i < MAX_BPF_FUNC_REG_ARGS) {
+			reg = &regs[i + 1];
+		} else {
+			/*
+			 * Retrieve the spilled reg state from the stack arg slot.
+			 * Reuse the existing type checking infrastructure which
+			 * reads from cur_regs(env)[regno], temporarily copy the
+			 * stack arg reg state into regs[BPF_REG_1] and restore
+			 * it after checking.
+			 */

[ ... ]

Quoted 8 lines
kernel/bpf/verifier.c +7 -0
+			is_stack_arg = true;
+			regno = BPF_REG_1;
+			saved_reg = regs[BPF_REG_1];
+			regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+			reg = &regs[BPF_REG_1];
+		}
+
  		if (is_kfunc_arg_prog_aux(btf, &args[i])) {

[ ... ]

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
-			continue;
+			goto next_arg;
  		}
 
  		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
  			continue;

The is_kfunc_arg_prog_aux path above was converted from continue to goto next_arg so that regs[BPF_REG_1] gets restored via the next_arg label. The scalar path and the KF_ARG_PTR_TO_NULL path were also converted the same way.

Quoted 2 lines

Should this continue for is_kfunc_arg_ignore/is_kfunc_arg_implicit also be goto next_arg? When a stack arg (i >= MAX_BPF_FUNC_REG_ARGS)

You are right. My initial version with a early bpf-next is okay. But after rebase on top of master, I missed the above as it is newly introduced. Will fix in the next revision.

Quoted 17 lines

hits this continue, is_stack_arg is true but the next_arg restore is skipped, so regs[BPF_REG_1] keeps the spilled pointer from the current iteration. On the next iteration, saved_reg then captures that stale value instead of the original register state.

The comment above says "restore it after checking", which this continue path does not do. The impact is currently masked because each iteration independently overwrites regs[BPF_REG_1] and the caller_saved clearing in check_kfunc_call() resets R1 after the loop, but it breaks the save/restore invariant the rest of this patch establishes.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23879588767

#22 lore raw

Re: [PATCH bpf-next 04/10] bpf: Support stack arguments for kfunc calls

In reply to Yonghong Song
Quoted 96 lines · Yonghong Song

Extend the stack argument mechanism to kfunc calls, allowing kfuncs with more than 5 parameters to receive additional arguments via the r12-based stack arg area.

For kfuncs, the caller is a BPF program and the callee is a kernel function. The BPF program writes outgoing args at r12-relative offsets past its own incoming area.

The following is an example to show how stack arguments are saved:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

   Caller (foo)
   ============
       r12-relative stack arg area:

       r12-8:  [incoming arg 6]
       r12-16: [incoming arg 7]

       ---- incoming/outgoing boundary (kfunc1)
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       ...
       Back from kfunc1
       ...

       ---- incoming/outgoing boundary
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       r12-48: [outgoing arg 9 to callee]

Later JIT will marshal outgoing arguments to the native calling convention for kfunc1() and kfunc2().

In check_kfunc_args(), for args beyond the 5th, retrieve the spilled register state from the caller's stack arg slots. Temporarily copy it into regs[BPF_REG_1] to reuse the existing type checking infrastructure, then restore after checking. Also in fixup_kfunc_call(), repurpose insn->off (no longer needed after kfunc address resolution) to store the number of stack args, so the JIT knows how many args to marshal.

---
 kernel/bpf/verifier.c | 97 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 80 insertions(+), 17 deletions(-)
kernel/bpf/verifier.c +10 -3
index d424fe611ef8..6579156486b8 100644
@@ -3502,7 +3502,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
        struct bpf_kfunc_meta kfunc;
        struct bpf_kfunc_desc *desc;
        unsigned long addr;
-       int err;
+       int i, err;
 
        prog_aux = env->prog->aux;
        tab = prog_aux->kfunc_tab;
@@ -3578,6 +3578,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
        if (err)
                return err;
 
+       for (i = MAX_BPF_FUNC_REG_ARGS; i < func_model.nr_args; i++) {
+               if (func_model.arg_size[i] > sizeof(u64)) {
+                       verbose(env, "kfunc %s arg#%d size %d > %zu not supported for stack args\n",
+                               kfunc.name, i, func_model.arg_size[i], sizeof(u64));
+                       return -EINVAL;
+               }
+       }
+
        desc = &tab->descs[tab->nr_descs++];
        desc->func_id = func_id;
        desc->offset = offset;
@@ -12995,9 +13003,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
                       struct bpf_kfunc_call_arg_meta *meta,
                       const struct btf_type *t, const struct btf_type *ref_t,
                       const char *ref_tname, const struct btf_param *args,
-                      int argno, int nargs)
+                      int argno, int nargs, u32 regno)
 {
-       u32 regno = argno + 1;
        struct bpf_reg_state *regs = cur_regs(env);
        struct bpf_reg_state *reg = &regs[regno];
        bool arg_mem_size = false;

[...]

Quoted 37 lines
kernel/bpf/verifier.c +16 -3
@@ -13677,9 +13684,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
        args = (const struct btf_param *)(meta->func_proto + 1);
        nargs = btf_type_vlen(meta->func_proto);
-       if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+       if (nargs > MAX_BPF_FUNC_ARGS) {
                verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
-                       MAX_BPF_FUNC_REG_ARGS);
+                       MAX_BPF_FUNC_ARGS);
                return -EINVAL;
        }
 
@@ -13687,13 +13694,41 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
         * verifier sees.
         */
        for (i = 0; i < nargs; i++) {
-               struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+               struct bpf_reg_state *regs = cur_regs(env), *reg;
+               struct bpf_reg_state saved_reg;
                const struct btf_type *t, *ref_t, *resolve_ret;
                enum bpf_arg_type arg_type = ARG_DONTCARE;
                u32 regno = i + 1, ref_id, type_size;
                bool is_ret_buf_sz = false;
+               bool is_stack_arg = false;
                int kf_arg_type;
 
+               if (i < MAX_BPF_FUNC_REG_ARGS) {
+                       reg = &regs[i + 1];
+               } else {
+                       /*
+                        * Retrieve the spilled reg state from the stack arg slot.
+                        * Reuse the existing type checking infrastructure which
+                        * reads from cur_regs(env)[regno], temporarily copy the
+                        * stack arg reg state into regs[BPF_REG_1] and restore
+                        * it after checking.
+                        */
+                       struct bpf_func_state *caller = cur_func(env);

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13726

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+                       int spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE +
+                                 (i - MAX_BPF_FUNC_REG_ARGS);
+
+                       if (!is_stack_arg_slot_initialized(caller, spi)) {
+                               verbose(env, "stack arg#%d not properly initialized\n", i);
+                               return -EINVAL;
+                       }
+
+                       is_stack_arg = true;
+                       regno = BPF_REG_1;

is_kfunc_arg_mem_size() and is_kfunc_arg_const_mem_size() will not work properly in get_kfunc_ptr_arg_type(). Since they assume regno + 1 to be the size, R2 would be accidentally treated as size.

Maybe consider creating a helper to retrieve bpf_reg_state of a specific arg and use it in get_kfunc_ptr_arg_type() instead of copying and restoring?

Quoted 124 lines
kernel/bpf/verifier.c +44 -12
+                       saved_reg = regs[BPF_REG_1];
+                       regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+                       reg = &regs[BPF_REG_1];
+               }
+
                if (is_kfunc_arg_prog_aux(btf, &args[i])) {
                        /* Reject repeated use bpf_prog_aux */
                        if (meta->arg_prog) {
@@ -13702,7 +13737,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                        }
                        meta->arg_prog = true;
                        cur_aux(env)->arg_prog = regno;
-                       continue;
+                       goto next_arg;
                }
 
                if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
@@ -13725,9 +13760,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                                        verbose(env, "R%d must be a known constant\n", regno);
                                        return -EINVAL;
                                }
-                               ret = mark_chain_precision(env, regno);
-                               if (ret < 0)
-                                       return ret;
+                               if (i < MAX_BPF_FUNC_REG_ARGS) {
+                                       ret = mark_chain_precision(env, regno);
+                                       if (ret < 0)
+                                               return ret;
+                               }
                                meta->arg_constant.found = true;
                                meta->arg_constant.value = reg->var_off.value;
                        } else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) {
@@ -13749,11 +13786,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                                }
 
                                meta->r0_size = reg->var_off.value;
-                               ret = mark_chain_precision(env, regno);
-                               if (ret)
-                                       return ret;
+                               if (i < MAX_BPF_FUNC_REG_ARGS) {
+                                       ret = mark_chain_precision(env, regno);
+                                       if (ret)
+                                               return ret;
+                               }
                        }
-                       continue;
+                       goto next_arg;
                }
 
                if (!btf_type_is_ptr(t)) {
@@ -13782,13 +13821,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
                ref_tname = btf_name_by_offset(btf, ref_t->name_off);
 
-               kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs);
+               kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs,
+                                                    regno);
                if (kf_arg_type < 0)
                        return kf_arg_type;
 
                switch (kf_arg_type) {
                case KF_ARG_PTR_TO_NULL:
-                       continue;
+                       goto next_arg;
                case KF_ARG_PTR_TO_MAP:
                        if (!reg->map_ptr) {
                                verbose(env, "pointer in R%d isn't map pointer\n", regno);
@@ -14201,6 +14241,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                        break;
                }
                }
+next_arg:
+               if (is_stack_arg)
+                       regs[BPF_REG_1] = saved_reg;
        }
 
        if (is_kfunc_release(meta) && !meta->release_regno) {
@@ -14778,7 +14821,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
        nargs = btf_type_vlen(meta.func_proto);
        args = (const struct btf_param *)(meta.func_proto + 1);
-       for (i = 0; i < nargs; i++) {
+       for (i = 0; i < nargs && i < MAX_BPF_FUNC_REG_ARGS; i++) {
                u32 regno = i + 1;
 
                t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
@@ -14789,6 +14832,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                        mark_btf_func_reg_size(env, regno, t->size);
        }
 
+       /* Track outgoing stack arg depth for kfuncs with >5 args */
+       if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+               struct bpf_func_state *caller = cur_func(env);
+               struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+               u16 kfunc_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
+               if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth)
+                       caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth;
+       }
+
        if (is_iter_next_kfunc(&meta)) {
                err = process_iter_next_call(env, insn_idx, &meta);
                if (err)
@@ -23615,6 +23668,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
        if (!bpf_jit_supports_far_kfunc_call())
                insn->imm = BPF_CALL_IMM(desc->addr);
 
+       /*
+        * After resolving the kfunc address, insn->off is no longer needed
+        * for BTF fd index. Repurpose it to store the number of stack args
+        * so the JIT can marshal them.
+        */
+       if (desc->func_model.nr_args > MAX_BPF_FUNC_REG_ARGS)
+               insn->off = desc->func_model.nr_args - MAX_BPF_FUNC_REG_ARGS;
+       else
+               insn->off = 0;
+
        if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
                struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
                struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };
--

2.52.0

author reviewer
#23 lore raw

PATCH 5/10 bpf: Reject stack arguments in non-JITed programs

In reply to Yonghong Song

The interpreter does not understand the bpf register r12 (BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So reject interpreter usage if stack arguments are used either in the main program or any subprogram.

---
 kernel/bpf/core.c     | 3 ++-
 kernel/bpf/verifier.c | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)
kernel/bpf/core.c +2 -1
index 3520337a1c0e..a04b31eb4c49 100644
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) ||
+	    fp->aux->stack_arg_depth)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +6 -0
index 6579156486b8..6d5a764c3460 100644
@@ -23542,6 +23542,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}
+	for (i = 0; i < env->subprog_cnt; i++) {
+		if (env->subprog_info[i].incoming_stack_arg_depth) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
 	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
 		/* When JIT fails the progs with bpf2bpf calls and tail_calls
 		 * have to be rejected, since interpreter doesn't support them yet.
Signature · 1 line
2.52.0
author reviewer
#24 lore raw

PATCH 6/10 bpf: Enable stack argument support for x86_64

In reply to Yonghong Song

Add stack argument support for x86_64.

---
 arch/x86/net/bpf_jit_comp.c | 5 +++++
 include/linux/filter.h      | 1 +
 kernel/bpf/btf.c            | 9 ++++++++-
 kernel/bpf/core.c           | 5 +++++
 4 files changed, 19 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +5 -0
index e9b78040d703..32864dbc2c4e 100644
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_stack_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (text_poke_copy(dst, src, len) == NULL)
include/linux/filter.h +1 -0
index 68f018dd4b9c..a5035fb80a6b 100644
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_ptr_xchg(void);
kernel/bpf/btf.c +8 -1
index c5f3aa05d5a3..1cbe0f2b0e41 100644
@@ -20,6 +20,7 @@
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
 #include <linux/bpf.h>
+#include <linux/filter.h>
 #include <linux/bpf_lsm.h>
 #include <linux/skmsg.h>
 #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -ENOTSUPP;
+		}
 		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+	}
 
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
kernel/bpf/core.c +5 -0
index a04b31eb4c49..01de4f7f3a82 100644
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_stack_args(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_far_kfunc_call(void)
 {
 	return false;
Signature · 1 line
2.52.0
author reviewer
#25 lore raw

PATCH 7/10 bpf,x86: Implement JIT support for stack arguments

In reply to Yonghong Song

Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.

There are two possible approaches to allocate the stack arg area:

Option 1: Allocate a single combined region (incoming + max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp - prog_stack_depth - offset] where the 'offset' is the offset value in (incoming + max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.

Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.

This patch uses Option 1 for simplicity.

The native x86_64 stack layout for a function with incoming and outgoing stack args:

high address

  ┌─────────────────────────┐
  │ incoming stack arg N    │  [rbp + 16 + (N - 1) * 8]  (pushed by caller)
  │ ...                     │
  │ incoming stack arg 1    │  [rbp + 16]
  ├─────────────────────────┤
  │ return address          │  [rbp + 8]
  │ saved rbp               │  [rbp]
  ├─────────────────────────┤
  │ callee-saved regs       │
  │ BPF program stack       │  (stack_depth bytes)
  ├─────────────────────────┤
  │ incoming stack arg 1    │  [rbp - prog_stack_depth - 8]
  │ ...   (copied from      │   (copied in prologue)
  │        caller's push)   │
  │ incoming stack arg N    │  [rbp - prog_stack_depth - N * 8]
  ├─────────────────────────┤
  │ outgoing stack arg 1    │  (written via r12-relative STX/ST,
  │ ...                     │   JIT translates to RBP-relative)
  │ outgoing stack arg M    │
  └─────────────────────────┘
    ...                        Other stack usage
  ┌─────────────────────────┐
  │ incoming stack arg M    │ (copy from outgoing stack arg to
  │ ...                     │  incoming stack arg)
  │ incoming stack arg 1    │
  ├─────────────────────────┤
  │ return address          │
  │ saved rbp               │
  ├─────────────────────────┤
  │ ...                     │
  └─────────────────────────┘

low address

In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.

For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.

For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the x86_64 calling convention, then clean up RSP after return.

---
 arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++--
 1 file changed, 138 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +138 -7
index 32864dbc2c4e..807493f109e5 100644
@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
 	*pprog = prog;
 }
 
+static int push_stack_args(u8 **pprog, s32 base_off, int from, int to)
+{
+	u8 *prog = *pprog;
+	int j, off, cnt = 0;
+
+	for (j = from; j >= to; j--) {
+		off = base_off - j * 8;
+
+		/* push qword [rbp + off] */
+		if (is_imm8(off)) {
+			EMIT3(0xFF, 0x75, off);
+			cnt += 3;
+		} else {
+			EMIT2_off32(0xFF, 0xB5, off);
+			cnt += 6;
+		}
+	}
+	*pprog = prog;
+	return cnt;
+}
+
 static void pop_r12(u8 **pprog)
 {
 	u8 *prog = *pprog;
@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	int i, excnt = 0;
 	int ilen, proglen = 0;
 	u8 *prog = temp;
-	u32 stack_depth;
+	u16 stack_arg_depth, incoming_stack_arg_depth;
+	u32 prog_stack_depth, stack_depth;
+	bool has_stack_args;
 	int err;
 
 	stack_depth = bpf_prog->aux->stack_depth;
+	stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+	incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
 	if (priv_stack_ptr) {
 		priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
 		stack_depth = 0;
 	}
 
+	/*
+	 * Save program stack depth before adding stack arg space.
+	 * Each function allocates its own stack arg space
+	 * (incoming + outgoing) below its BPF stack.
+	 * Stack args are accessed via RBP-based addressing.
+	 */
+	prog_stack_depth = round_up(stack_depth, 8);
+	if (stack_arg_depth)
+		stack_depth += stack_arg_depth;
+	has_stack_args = stack_arg_depth > 0;
+
 	arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
 	user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
+
 	detect_reg_usage(insn, insn_cnt, callee_regs_used);
 
 	emit_prologue(&prog, image, stack_depth,
@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 		emit_mov_imm64(&prog, X86_REG_R12,
 			       arena_vm_start >> 32, (u32) arena_vm_start);
 
+	if (incoming_stack_arg_depth && bpf_is_subprog(bpf_prog)) {
+		int n = incoming_stack_arg_depth / 8;
+
+		/*
+		 * Caller pushed stack args before CALL, so after prologue
+		 * (CALL saves ret addr, then PUSH saves old RBP) they sit
+		 * above RBP:
+		 *
+		 *   [rbp + 16 + (n - 1) * 8]  stack_arg n
+		 *   ...
+		 *   [rbp + 24]                stack_arg 2
+		 *   [rbp + 16]                stack_arg 1
+		 *   [rbp +  8]                return address
+		 *   [rbp +  0]                saved rbp
+		 *
+		 * Copy each into callee's own region below the program stack:
+		 *   [rbp - prog_stack_depth - i * 8]
+		 */
+		for (i = 0; i < n; i++) {
+			s32 src = 16 + i * 8;
+			s32 dst = -prog_stack_depth - (i + 1) * 8;
+
+			/* mov rax, [rbp + src] */
+			EMIT4(0x48, 0x8B, 0x45, src);
+			/* mov [rbp + dst], rax */
+			if (is_imm8(dst))
+				EMIT4(0x48, 0x89, 0x45, dst);
+			else
+				EMIT3_off32(0x48, 0x89, 0x85, dst);
+		}
+	}
+
 	if (priv_frame_ptr)
 		emit_priv_frame_ptr(&prog, priv_frame_ptr);
 
@@ -1715,13 +1784,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
+		bool adjust_stack_arg_off = false;
 		const s32 imm32 = insn->imm;
 		u32 dst_reg = insn->dst_reg;
 		u32 src_reg = insn->src_reg;
 		u8 b2 = 0, b3 = 0;
 		u8 *start_of_ldx;
 		s64 jmp_offset;
-		s16 insn_off;
+		s32 insn_off;
 		u8 jmp_cond;
 		u8 *func;
 		int nops;
@@ -1734,6 +1804,21 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 				dst_reg = X86_REG_R9;
 		}
 
+		if (has_stack_args) {
+			u8 class = BPF_CLASS(insn->code);
+
+			if (class == BPF_LDX &&
+			    src_reg == BPF_REG_STACK_ARG_BASE) {
+				src_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+			if ((class == BPF_STX || class == BPF_ST) &&
+			    dst_reg == BPF_REG_STACK_ARG_BASE) {
+				dst_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+		}
+
 		switch (insn->code) {
 			/* ALU */
 		case BPF_ALU | BPF_ADD | BPF_X:
@@ -2131,10 +2216,16 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 		case BPF_ST | BPF_MEM | BPF_DW:
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
 
-st:			if (is_imm8(insn->off))
-				EMIT2(add_1reg(0x40, dst_reg), insn->off);
+st:		{
+			int off = insn->off;
+
+			if (adjust_stack_arg_off)
+				off -= prog_stack_depth;
+			if (is_imm8(off))
+				EMIT2(add_1reg(0x40, dst_reg), off);
 			else
-				EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
+				EMIT1_off32(add_1reg(0x80, dst_reg), off);
+		}
 
 			EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
 			break;
@@ -2143,9 +2234,14 @@ st:			if (is_imm8(insn->off))
 		case BPF_STX | BPF_MEM | BPF_B:
 		case BPF_STX | BPF_MEM | BPF_H:
 		case BPF_STX | BPF_MEM | BPF_W:
-		case BPF_STX | BPF_MEM | BPF_DW:
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+		case BPF_STX | BPF_MEM | BPF_DW: {
+			int off = insn->off;
+
+			if (adjust_stack_arg_off)
+				off -= prog_stack_depth;
+			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, off);
 			break;
+		}
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
@@ -2243,6 +2339,8 @@ st:			if (is_imm8(insn->off))
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
 			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off -= prog_stack_depth;
 
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2440,6 +2538,8 @@ st:			if (is_imm8(insn->off))
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			int off, base_off, n_stack_args, kfunc_stack_args = 0, stack_args = 0;
+			u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;
 			u8 *ip = image + addrs[i - 1];
 
 			func = (u8 *) __bpf_call_base + imm32;
@@ -2449,6 +2549,29 @@ st:			if (is_imm8(insn->off))
 			}
 			if (!imm32)
 				return -EINVAL;
+
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+				n_stack_args = outgoing_stack_args / 8;
+				base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+				ip += push_stack_args(&prog, base_off, n_stack_args, 1);
+			}
+
+			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+				kfunc_stack_args = insn->off;
+				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+				base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+				ip += push_stack_args(&prog, base_off, kfunc_stack_args, 2);
+
+				/* mov r9, [rbp + base_off - 8] */
+				off = base_off - 8;
+				if (is_imm8(off)) {
+					EMIT4(0x4C, 0x8B, 0x4D, off);
+					ip += 4;
+				} else {
+					EMIT3_off32(0x4C, 0x8B, 0x8D, off);
+					ip += 7;
+				}
+			}
 			if (priv_frame_ptr) {
 				push_r9(&prog);
 				ip += 2;
@@ -2458,6 +2581,14 @@ st:			if (is_imm8(insn->off))
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (stack_args > 0) {
+				/* add rsp, stack_args * 8 */
+				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+			}
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+				/* add rsp, outgoing_stack_args */
+				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_args);
+			}
 			break;
 		}
 
Signature · 1 line
2.52.0
#26 lore raw

Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments

Quoted 1 line · Yonghong Song

[...]

Quoted 23 lines
                case BPF_JMP | BPF_CALL: {
arch/x86/net/bpf_jit_comp.c +15 -0
+                       int off, base_off, n_stack_args, kfunc_stack_args = 0, stack_args = 0;
+                       u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;
                        u8 *ip = image + addrs[i - 1];
 
                        func = (u8 *) __bpf_call_base + imm32;
@@ -2449,6 +2549,29 @@ st:                      if (is_imm8(insn->off))
                        }
                        if (!imm32)
                                return -EINVAL;
+
+                       if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+                               n_stack_args = outgoing_stack_args / 8;
+                               base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+                               ip += push_stack_args(&prog, base_off, n_stack_args, 1);
+                       }
+
+                       if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+                               kfunc_stack_args = insn->off;
+                               stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+                               base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+                               ip += push_stack_args(&prog, base_off, kfunc_stack_args, 2);
+

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2574

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+                               /* mov r9, [rbp + base_off - 8] */
+                               off = base_off - 8;
+                               if (is_imm8(off)) {
+                                       EMIT4(0x4C, 0x8B, 0x4D, off);
+                                       ip += 4;
+                               } else {
+                                       EMIT3_off32(0x4C, 0x8B, 0x8D, off);
+                                       ip += 7;
+                               }
+                       }

Do we need to make sure RSP is 16-byte aligned before passing arg through stack per x86-64 ABI?

Quoted 22 lines
arch/x86/net/bpf_jit_comp.c +8 -1
                        if (priv_frame_ptr) {
                                push_r9(&prog);
                                ip += 2;
@@ -2458,6 +2581,14 @@ st:                      if (is_imm8(insn->off))
                                return -EINVAL;
                        if (priv_frame_ptr)
                                pop_r9(&prog);
+                       if (stack_args > 0) {
+                               /* add rsp, stack_args * 8 */
+                               EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+                       }
+                       if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+                               /* add rsp, outgoing_stack_args */
+                               EMIT4(0x48, 0x83, 0xC4, outgoing_stack_args);
+                       }
                        break;
                }
 
--

2.52.0

author reviewer
#27 lore raw

Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments

Quoted 30 lines · Amery Hung

On Wed, Apr 1, 2026 at 6:28 PM Yonghong Song <[email protected]> wrote:
[...]

Quoted 27 lines
                 case BPF_JMP | BPF_CALL: {
arch/x86/net/bpf_jit_comp.c +19 -0
+                       int off, base_off, n_stack_args, kfunc_stack_args = 0, stack_args = 0;
+                       u16 outgoing_stack_args = stack_arg_depth - incoming_stack_arg_depth;
                         u8 *ip = image + addrs[i - 1];
 
                         func = (u8 *) __bpf_call_base + imm32;
@@ -2449,6 +2549,29 @@ st:                      if (is_imm8(insn->off))
                         }
                         if (!imm32)
                                 return -EINVAL;
+
+                       if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+                               n_stack_args = outgoing_stack_args / 8;
+                               base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+                               ip += push_stack_args(&prog, base_off, n_stack_args, 1);
+                       }
+
+                       if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+                               kfunc_stack_args = insn->off;
+                               stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+                               base_off = -(prog_stack_depth + incoming_stack_arg_depth);
+                               ip += push_stack_args(&prog, base_off, kfunc_stack_args, 2);
+
+                               /* mov r9, [rbp + base_off - 8] */
+                               off = base_off - 8;
+                               if (is_imm8(off)) {
+                                       EMIT4(0x4C, 0x8B, 0x4D, off);
Quoted 8 lines

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2574

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6 -0
+                                       ip += 4;
+                               } else {
+                                       EMIT3_off32(0x4C, 0x8B, 0x8D, off);
+                                       ip += 7;
+                               }
+                       }

Do we need to make sure RSP is 16-byte aligned before passing arg through stack per x86-64 ABI?

Good question. Without this patch set, looks like bpf jit here does not enforce 16 byte aligned. IIUC, 16 byte aligned requirement will be necessary for some special SSE/AVX/128-bit XMM etc. The bpf jit does not really emit them, so I guess it is okay?

Quoted 23 lines
Quoted 23 lines
arch/x86/net/bpf_jit_comp.c +8 -1
                         if (priv_frame_ptr) {
                                 push_r9(&prog);
                                 ip += 2;
@@ -2458,6 +2581,14 @@ st:                      if (is_imm8(insn->off))
                                 return -EINVAL;
                         if (priv_frame_ptr)
                                 pop_r9(&prog);
+                       if (stack_args > 0) {
+                               /* add rsp, stack_args * 8 */
+                               EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+                       }
+                       if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_args > 0) {
+                               /* add rsp, outgoing_stack_args */
+                               EMIT4(0x48, 0x83, 0xC4, outgoing_stack_args);
+                       }
                         break;
                 }
 
--

2.52.0

#28 lore raw

Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments

In reply to Yonghong Song
Quoted 172 lines · Yonghong Song

Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.

There are two possible approaches to allocate the stack arg area:

Option 1: Allocate a single combined region (incoming + max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp - prog_stack_depth - offset] where the 'offset' is the offset value in (incoming + max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.

Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.

This patch uses Option 1 for simplicity.

The native x86_64 stack layout for a function with incoming and outgoing stack args:

high address

  ┌─────────────────────────┐
  │ incoming stack arg N    │  [rbp + 16 + (N - 1) * 8]  (pushed by caller)
  │ ...                     │
  │ incoming stack arg 1    │  [rbp + 16]
  ├─────────────────────────┤
  │ return address          │  [rbp + 8]
  │ saved rbp               │  [rbp]
  ├─────────────────────────┤
  │ callee-saved regs       │
  │ BPF program stack       │  (stack_depth bytes)
  ├─────────────────────────┤
  │ incoming stack arg 1    │  [rbp - prog_stack_depth - 8]
  │ ...   (copied from      │   (copied in prologue)
  │        caller's push)   │
  │ incoming stack arg N    │  [rbp - prog_stack_depth - N * 8]
  ├─────────────────────────┤
  │ outgoing stack arg 1    │  (written via r12-relative STX/ST,
  │ ...                     │   JIT translates to RBP-relative)
  │ outgoing stack arg M    │
  └─────────────────────────┘
    ...                        Other stack usage
  ┌─────────────────────────┐
  │ incoming stack arg M    │ (copy from outgoing stack arg to
  │ ...                     │  incoming stack arg)
  │ incoming stack arg 1    │
  ├─────────────────────────┤
  │ return address          │
  │ saved rbp               │
  ├─────────────────────────┤
  │ ...                     │
  └─────────────────────────┘

low address

In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.

For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.

For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the x86_64 calling convention, then clean up RSP after return.

---
 arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++--
 1 file changed, 138 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +58 -1
index 32864dbc2c4e..807493f109e5 100644
@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
        *pprog = prog;
 }
 
+static int push_stack_args(u8 **pprog, s32 base_off, int from, int to)
+{
+       u8 *prog = *pprog;
+       int j, off, cnt = 0;
+
+       for (j = from; j >= to; j--) {
+               off = base_off - j * 8;
+
+               /* push qword [rbp + off] */
+               if (is_imm8(off)) {
+                       EMIT3(0xFF, 0x75, off);
+                       cnt += 3;
+               } else {
+                       EMIT2_off32(0xFF, 0xB5, off);
+                       cnt += 6;
+               }
+       }
+       *pprog = prog;
+       return cnt;
+}
+
 static void pop_r12(u8 **pprog)
 {
        u8 *prog = *pprog;
@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
        int i, excnt = 0;
        int ilen, proglen = 0;
        u8 *prog = temp;
-       u32 stack_depth;
+       u16 stack_arg_depth, incoming_stack_arg_depth;
+       u32 prog_stack_depth, stack_depth;
+       bool has_stack_args;
        int err;
 
        stack_depth = bpf_prog->aux->stack_depth;
+       stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+       incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
        priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
        if (priv_stack_ptr) {
                priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
                stack_depth = 0;
        }
 
+       /*
+        * Save program stack depth before adding stack arg space.
+        * Each function allocates its own stack arg space
+        * (incoming + outgoing) below its BPF stack.
+        * Stack args are accessed via RBP-based addressing.
+        */
+       prog_stack_depth = round_up(stack_depth, 8);
+       if (stack_arg_depth)
+               stack_depth += stack_arg_depth;
+       has_stack_args = stack_arg_depth > 0;
+
        arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
        user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
+
        detect_reg_usage(insn, insn_cnt, callee_regs_used);
 
        emit_prologue(&prog, image, stack_depth,
@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
                emit_mov_imm64(&prog, X86_REG_R12,
                               arena_vm_start >> 32, (u32) arena_vm_start);
 
+       if (incoming_stack_arg_depth && bpf_is_subprog(bpf_prog)) {
+               int n = incoming_stack_arg_depth / 8;
+
+               /*
+                * Caller pushed stack args before CALL, so after prologue
+                * (CALL saves ret addr, then PUSH saves old RBP) they sit
+                * above RBP:
+                *
+                *   [rbp + 16 + (n - 1) * 8]  stack_arg n
+                *   ...
+                *   [rbp + 24]                stack_arg 2
+                *   [rbp + 16]                stack_arg 1
+                *   [rbp +  8]                return address
+                *   [rbp +  0]                saved rbp
+                *
+                * Copy each into callee's own region below the program stack:
+                *   [rbp - prog_stack_depth - i * 8]
+                */
+               for (i = 0; i < n; i++) {
+                       s32 src = 16 + i * 8;

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1773

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+                       s32 dst = -prog_stack_depth - (i + 1) * 8;
+
+                       /* mov rax, [rbp + src] */
+                       EMIT4(0x48, 0x8B, 0x45, src);
+                       /* mov [rbp + dst], rax */
+                       if (is_imm8(dst))
+                               EMIT4(0x48, 0x89, 0x45, dst);
+                       else
+                               EMIT3_off32(0x48, 0x89, 0x85, dst);
+               }

This is really suboptimal.
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.

pw-bot: cr

author reviewer
#29 lore raw

Re: [PATCH bpf-next 07/10] bpf,x86: Implement JIT support for stack arguments

Quoted 188 lines · Alexei Starovoitov
Quoted 171 lines · Yonghong Song

Add x86_64 JIT support for BPF functions and kfuncs with more than 5 arguments. The extra arguments are passed through a stack area addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode, which the JIT translates to RBP-relative accesses in native code.

There are two possible approaches to allocate the stack arg area:

Option 1: Allocate a single combined region (incoming + max_outgoing)
below the program stack in the function prologue. All r12-relative accesses become [rbp - prog_stack_depth - offset] where the 'offset' is the offset value in (incoming + max_outgoing) region. This is simple because the area is always at a fixed offset from RBP. The tradeoff is slightly higher stack usage when multiple callees have different stack arg counts — the area is sized to the maximum.

Option 2: Allocate each outgoing area individually at the call
site, sized exactly to the callee's needs. This minimizes stack usage but significantly complicates the JIT: each call site must dynamically adjust RSP, and addresses of stack args would shift depending on context, making the offset calculations harder.

This patch uses Option 1 for simplicity.

The native x86_64 stack layout for a function with incoming and outgoing stack args:

high address

   ┌─────────────────────────┐
   │ incoming stack arg N    │  [rbp + 16 + (N - 1) * 8]  (pushed by caller)
   │ ...                     │
   │ incoming stack arg 1    │  [rbp + 16]
   ├─────────────────────────┤
   │ return address          │  [rbp + 8]
   │ saved rbp               │  [rbp]
   ├─────────────────────────┤
   │ callee-saved regs       │
   │ BPF program stack       │  (stack_depth bytes)
   ├─────────────────────────┤
   │ incoming stack arg 1    │  [rbp - prog_stack_depth - 8]
   │ ...   (copied from      │   (copied in prologue)
   │        caller's push)   │
   │ incoming stack arg N    │  [rbp - prog_stack_depth - N * 8]
   ├─────────────────────────┤
   │ outgoing stack arg 1    │  (written via r12-relative STX/ST,
   │ ...                     │   JIT translates to RBP-relative)
   │ outgoing stack arg M    │
   └─────────────────────────┘
     ...                        Other stack usage
   ┌─────────────────────────┐
   │ incoming stack arg M    │ (copy from outgoing stack arg to
   │ ...                     │  incoming stack arg)
   │ incoming stack arg 1    │
   ├─────────────────────────┤
   │ return address          │
   │ saved rbp               │
   ├─────────────────────────┤
   │ ...                     │
   └─────────────────────────┘

low address

In prologue, the caller's incoming stack arguments are copied to callee's incoming stack arguments, which will be fetched by later load insns. The outgoing stack arguments are written by JIT RBP-relative STX or ST.

For each bpf-to-bpf call, push outgoing stack args onto the native stack before CALL, pop them after return. So the same 'outgoing stack arg' area is used by all bpf-to-bpf functions.

For kfunc calls, push stack args (arg 7+) onto the native stack and load arg 6 into R9 per the x86_64 calling convention, then clean up RSP after return.

---
  arch/x86/net/bpf_jit_comp.c | 145 ++++++++++++++++++++++++++++++++++--
  1 file changed, 138 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +58 -1
index 32864dbc2c4e..807493f109e5 100644
@@ -367,6 +367,27 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
         *pprog = prog;
  }
 
+static int push_stack_args(u8 **pprog, s32 base_off, int from, int to)
+{
+       u8 *prog = *pprog;
+       int j, off, cnt = 0;
+
+       for (j = from; j >= to; j--) {
+               off = base_off - j * 8;
+
+               /* push qword [rbp + off] */
+               if (is_imm8(off)) {
+                       EMIT3(0xFF, 0x75, off);
+                       cnt += 3;
+               } else {
+                       EMIT2_off32(0xFF, 0xB5, off);
+                       cnt += 6;
+               }
+       }
+       *pprog = prog;
+       return cnt;
+}
+
  static void pop_r12(u8 **pprog)
  {
         u8 *prog = *pprog;
@@ -1664,19 +1685,35 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
         int i, excnt = 0;
         int ilen, proglen = 0;
         u8 *prog = temp;
-       u32 stack_depth;
+       u16 stack_arg_depth, incoming_stack_arg_depth;
+       u32 prog_stack_depth, stack_depth;
+       bool has_stack_args;
         int err;
 
         stack_depth = bpf_prog->aux->stack_depth;
+       stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+       incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
         priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
         if (priv_stack_ptr) {
                 priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
                 stack_depth = 0;
         }
 
+       /*
+        * Save program stack depth before adding stack arg space.
+        * Each function allocates its own stack arg space
+        * (incoming + outgoing) below its BPF stack.
+        * Stack args are accessed via RBP-based addressing.
+        */
+       prog_stack_depth = round_up(stack_depth, 8);
+       if (stack_arg_depth)
+               stack_depth += stack_arg_depth;
+       has_stack_args = stack_arg_depth > 0;
+
         arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
         user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
+
         detect_reg_usage(insn, insn_cnt, callee_regs_used);
 
         emit_prologue(&prog, image, stack_depth,
@@ -1704,6 +1741,38 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
                 emit_mov_imm64(&prog, X86_REG_R12,
                                arena_vm_start >> 32, (u32) arena_vm_start);
 
+       if (incoming_stack_arg_depth && bpf_is_subprog(bpf_prog)) {
+               int n = incoming_stack_arg_depth / 8;
+
+               /*
+                * Caller pushed stack args before CALL, so after prologue
+                * (CALL saves ret addr, then PUSH saves old RBP) they sit
+                * above RBP:
+                *
+                *   [rbp + 16 + (n - 1) * 8]  stack_arg n
+                *   ...
+                *   [rbp + 24]                stack_arg 2
+                *   [rbp + 16]                stack_arg 1
+                *   [rbp +  8]                return address
+                *   [rbp +  0]                saved rbp
+                *
+                * Copy each into callee's own region below the program stack:
+                *   [rbp - prog_stack_depth - i * 8]
+                */
+               for (i = 0; i < n; i++) {
+                       s32 src = 16 + i * 8;

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1773

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+                       s32 dst = -prog_stack_depth - (i + 1) * 8;
+
+                       /* mov rax, [rbp + src] */
+                       EMIT4(0x48, 0x8B, 0x45, src);
+                       /* mov [rbp + dst], rax */
+                       if (is_imm8(dst))
+                               EMIT4(0x48, 0x89, 0x45, dst);
+                       else
+                               EMIT3_off32(0x48, 0x89, 0x85, dst);
+               }

This is really suboptimal.
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg

Quoted 8 lines

into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86. The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9. Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s because arm64 has more registers for arguments.

Good point. I will try to simplify the JIT by following x86_64 calling convention.

Quoted 2 lines

pw-bot: cr

author reviewer
#30 lore raw

PATCH 8/10 selftests/bpf: Add tests for BPF function stack arguments

In reply to Yonghong Song

Add selftests covering stack argument passing for both BPF-to-BPF subprog calls and kfunc calls with more than 5 arguments. All tests are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.

---
 .../selftests/bpf/prog_tests/stack_arg.c      | 143 ++++++++++++++++++
 tools/testing/selftests/bpf/progs/stack_arg.c | 111 ++++++++++++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     |  59 ++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  22 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |   7 +
 5 files changed, 342 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/stack_arg.c new file +143 -0
new file mode 100644
index 000000000000..86b8025f5541
@@ -0,0 +1,143 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "stack_arg.skel.h"
+#include "stack_arg_kfunc.skel.h"
+
+static void test_nesting(void)
+{
+	struct stack_arg *skel;
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = &pkt_v4,
+		.data_size_in = sizeof(pkt_v4),
+		.repeat = 1,
+	);
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	err = stack_arg__load(skel);
+	if (!ASSERT_OK(err, "load"))
+		goto out;
+
+	skel->bss->a = 0;
+	skel->bss->b = 0;
+	skel->bss->c = 0;
+	skel->bss->d = 0;
+	skel->bss->e = 0;
+	skel->bss->f = 6;
+	skel->bss->g = 7;
+	skel->bss->i = 8;
+
+	prog_fd = bpf_program__fd(skel->progs.test);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, 29, "retval");
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void run_subtest(struct bpf_program *prog, int expected)
+{
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = &pkt_v4,
+		.data_size_in = sizeof(pkt_v4),
+		.repeat = 1,
+	);
+
+	prog_fd = bpf_program__fd(prog);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, expected, "retval");
+}
+
+static void test_global_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_global_many_args, 36);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_async_cb_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_kfunc(void)
+{
+	struct stack_arg_kfunc *skel;
+
+	skel = stack_arg_kfunc__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg_kfunc__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_stack_arg_scalar, 36);
+	run_subtest(skel->progs.test_stack_arg_ptr, 45);
+	run_subtest(skel->progs.test_stack_arg_mix, 51);
+
+out:
+	stack_arg_kfunc__destroy(skel);
+}
+
+void test_stack_arg(void)
+{
+	if (test__start_subtest("nesting"))
+		test_nesting();
+	if (test__start_subtest("global_many_args"))
+		test_global_many();
+	if (test__start_subtest("async_cb_many_args"))
+		test_async_cb_many();
+	if (test__start_subtest("kfunc"))
+		test_kfunc();
+}
tools/testing/selftests/bpf/progs/stack_arg.c new file +111 -0

111 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stack_arg_kfunc.c new file +59 -0

59 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +22 -0
index 061356f10093..d88ab1dc5106 100644
@@ -824,6 +824,25 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
+					 u64 e, u64 f, u64 g, u64 h)
+{
+	return a + b + c + d + e + f + g + h;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_ptr(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_pass1 *p)
+{
+	return a + b + c + d + e + p->x0 + p->x1;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_pass1 *p, u64 f,
+					     struct prog_test_pass1 *q)
+{
+	return a + b + c + d + e + p->x0 + f + q->x1;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1287,6 +1306,9 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +7 -0
index aa0b8d41e71b..f93cf2db12b2 100644
@@ -111,6 +111,13 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
+			       __u64 e, __u64 f, __u64 g, __u64 h) __ksym;
+__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p, __u64 f,
+				   struct prog_test_pass1 *q) __ksym;
 
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
Signature · 1 line
2.52.0
author reviewer
#31 lore raw

PATCH 9/10 selftests/bpf: Add negative test for oversized kfunc stack argument

In reply to Yonghong Song

Add a test that the verifier rejects kfunc calls where a stack argument exceeds 8 bytes (the register-sized slot limit).

---
 .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      | 32 +++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  7 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  7 ++++
 4 files changed, 70 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c new file +24 -0
new file mode 100644
index 000000000000..328a79edee45
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_fail.skel.h"
+
+void test_stack_arg_fail(void)
+{
+	struct stack_arg_fail *skel;
+
+	skel = stack_arg_fail__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	ASSERT_ERR(stack_arg_fail__load(skel), "load_should_fail");
+
+out:
+	stack_arg_fail__destroy(skel);
+}
tools/testing/selftests/bpf/progs/stack_arg_fail.c new file +32 -0
new file mode 100644
index 000000000000..caa63b6f6a80
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	struct prog_test_big_arg s = { .a = 1, .b = 2 };
+
+	return bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +7 -0
index d88ab1dc5106..00d0b238219d 100644
@@ -843,6 +843,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e,
 	return a + b + c + d + e + p->x0 + f + q->x1;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_big_arg s)
+{
+	return a + b + c + d + e + s.a + s.b;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1309,6 +1315,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +7 -0
index f93cf2db12b2..b142b87e9f60 100644
@@ -48,6 +48,11 @@ struct prog_test_pass2 {
 	} x;
 };
 
+struct prog_test_big_arg {
+	long a;
+	long b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -118,6 +123,8 @@ __u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 __u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				   struct prog_test_pass1 *p, __u64 f,
 				   struct prog_test_pass1 *q) __ksym;
+__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_big_arg s) __ksym;
 
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
Signature · 1 line
2.52.0
author reviewer
#32 lore raw

PATCH 10/10 selftests/bpf: Add verifier tests for stack argument validation

In reply to Yonghong Song

Add inline-asm-based verifier tests that exercise the stack argument validation logic directly.

---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_stack_arg.c  | 122 ++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index bcf01cb4cfe4..cbdbcb913bae 100644
@@ -90,6 +90,7 @@
 #include "verifier_sockmap_mutate.skel.h"
 #include "verifier_spill_fill.skel.h"
 #include "verifier_spin_lock.skel.h"
+#include "verifier_stack_arg.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
 #include "verifier_subprog_precision.skel.h"
@@ -235,6 +236,7 @@ void test_verifier_sock_addr(void)            { RUN(verifier_sock_addr); }
 void test_verifier_sockmap_mutate(void)       { RUN(verifier_sockmap_mutate); }
 void test_verifier_spill_fill(void)           { RUN(verifier_spill_fill); }
 void test_verifier_spin_lock(void)            { RUN(verifier_spin_lock); }
+void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
tools/testing/selftests/bpf/progs/verifier_stack_arg.c new file +122 -0

122 lines not on this page · raw message

Signature · 1 line
2.52.0