Skip to contentSkip to conversation

bpf: Support by-value struct and __int128 arguments

PATCH Superseded Yonghong Song 4 participants 32 messages Sep 11 – Sep 12

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v3 cover bpf: Support by-value struct and __int128 arguments

A global function or a kfunc taking a struct or union by value is rejected today:

Arg#1 type STRUCT in tar() is not supported yet. Unrecognized R2 type STRUCT

and an __int128 argument is accepted but mis-counted: the compiler passes it in two registers while the verifier gives it one, so every argument after it is checked against the wrong register and the program is rejected for a register its source never names.

The compiler passes such a value in one argument register per eightbyte, so a value of at most 16 bytes arrives in one or two consecutive registers. This series teaches the verifier to describe that: sub->args[] gains one entry per argument slot rather than one per BTF parameter, and the kfunc argument walk counts slots the same way. A value may only be composed of scalars, as a pointer would reach the callee as an opaque scalar, losing the provenance and reference tracking that make it safe to use. A global function still has no stack arguments, so all of its slots have to fit in the argument registers.

For a kfunc the two calling conventions have to agree, and they do not always. Where the BPF convention splits a value between the last argument register and the stack, x86_64 moves the whole of it to the stack and gives the register to the argument that follows, while arm64 rounds the register number up to an even one for a 16-byte aligned value. For stack arguments, both x86_64 and arm64 will have 16-byte alignment for those 16-byte aligned parameters. Otherwise the alignment will be 8-byte. This is implemented in x86_64 and arm64 jit.

Patch 1 fixed a pre-existing bug in bpf_kfunc_stack_access_bytes(). Patch 2 records the __int128 failure as it stands today, which patch 6 turns into a success. Patches 3 to 6 rename some fields, index the arguments of a global function by slot and accept a by-value struct and an __int128. Patches 8 and 9 have the same for a kfunc call. Patches 10 and 11 place the arguments per the kernel calling convention in the x86-64 and arm64 JITs. Patches 12 to 14 are the tests. Patch 15 removes some tests which requires pahole patch:
https://lore.kernel.org/bpf/[email protected]/
Once pahole patch is merged, the patch 15 can be removed.

Tested on x86-64 and arm64 with test_progs. The by-value struct tests are built with clang, as GCC passes an aggregate by invisible reference, and some of them need __BPF_FEATURE_STACK_ARGUMENT.

Changelog:
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Rename test name from verifier_int_arg to verifier_aggregate_arg.
- Rename from arg_cnt to arg_slot_cnt to make codes easy to understand.
- Fix an issue for by-value struct arguments up to 16 bytes where the bound check, e.g. number of slots is more than 5 or not, etc, is missing after all arguments are processed.
- Increase local slot size from MAX_BPF_FUNC_ARGS to 2 * MAX_BPF_FUNC_ARGS to avoid out-of-bound write.
- One codegen improvement for arm64 jit.
- Added a not-to-merge patch (patch 15) to workaround some tests. The pahole patch
https://lore.kernel.org/bpf/[email protected]/
is pending.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Fix a few issues due to potential out of bound access.
- Fix a few missed cases for kfunc with '> 8 byte' arguments.
- Add jit support for x86_64 and arm64. Previously certain functions are rejected, and now these functions can succeed.

Yonghong Song (15):
bpf: Read a kfunc's __sz argument only when it is in a register selftests/bpf: Add a test for an __int128 by-value argument
bpf: Rename bpf_subprog_info::arg_cnt to arg_slot_cnt
bpf: Index global function arguments by argument slot
bpf: Support by-value struct arguments up to 16 bytes
bpf: Support __int128 as a by-value function argument
bpf: Rename bpf_call_summary::num_params to arg_slot_cnt
bpf: Recognize by-value struct and __int128 kfunc arguments
bpf: Prepare kfunc arguments for the JIT from an ABI description bpf, x86: Move kfunc arguments into the x86-64 calling convention bpf, arm64: Move kfunc arguments into the arm64 calling convention selftests/bpf: Add C tests for by-value arguments up to 16 bytes selftests/bpf: Add inline-asm tests for by-value arguments selftests/bpf: Add tests for by-value kfunc arguments selftests/bpf: Temporary hack to disable register mismatch in arm64

 arch/arm64/net/bpf_jit_comp.c                 |  92 +++++-
 arch/x86/net/bpf_jit_comp.c                   |  73 ++++-
 include/linux/bpf.h                           |  15 +
 include/linux/bpf_verifier.h                  |   9 +-
 include/linux/filter.h                        |  32 ++
 kernel/bpf/btf.c                              | 124 ++++++--
 kernel/bpf/core.c                             |  94 ++++++
 kernel/bpf/liveness.c                         |  10 +-
 kernel/bpf/verifier.c                         | 287 ++++++++++++++----
 .../selftests/bpf/prog_tests/aggregate_arg.c  |  11 +
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/aggregate_arg_func.c  | 187 ++++++++++++
 .../selftests/bpf/progs/aggregate_arg_kfunc.c | 235 ++++++++++++++
 .../testing/selftests/bpf/progs/arena_kfunc.c |  15 +
 .../selftests/bpf/progs/stack_arg_fail.c      |  10 -
 .../bpf/progs/verifier_aggregate_arg.c        | 200 ++++++++++++
 .../bpf/progs/verifier_stack_arg_order.c      |   4 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  81 +++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  34 +++
 19 files changed, 1396 insertions(+), 119 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c
Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH v3 1/15 bpf: Read a kfunc's __sz argument only when it is in a register

Commit e0b7b91c72db ("bpf: Support stack arguments for kfunc calls") supported stack arguments for kfunc's. In bpf_kfunc_stack_access_bytes(), the size of a ptr + __sz pair is read from const_reg_vals[] at index 'BPF_REG_1 + arg + 1'. Past the fifth argument that index leaves the argument registers and reaches 6 through 9, which are the callee saved registers R6 through R9. The verifier does record constants for those, so a __sz argument passed on the stack can take the value of an unrelated register as its size.

Fix it by guard size_reg which has to be less than or equal to MAX_BPF_FUNC_REG_ARGS.

Fixes e0b7b91c72db ("bpf: Support stack arguments for kfunc calls")
---
 kernel/bpf/verifier.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +3 -2
index 9e79750e2480..fed576b8f7fe 100644
@@ -13771,13 +13771,14 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 		goto out;
 	}
 
-	/* ptr + __sz/__szk pair: size is in the next register */
+	/* ptr + __sz/__szk pair: the size follows the pointer */
 	if (arg + 1 < nargs &&
 	    (btf_param_match_suffix(btf, &args[arg + 1], "__sz") ||
 	     btf_param_match_suffix(btf, &args[arg + 1], "__szk"))) {
 		int size_reg = BPF_REG_1 + arg + 1;
 
-		if (aux->const_reg_mask & BIT(size_reg)) {
+		if (size_reg <= MAX_BPF_FUNC_REG_ARGS &&
+		    (aux->const_reg_mask & BIT(size_reg))) {
 			size = (s64)aux->const_reg_vals[size_reg];
 			goto out;
 		}
Signature · 1 line
2.52.0
author reviewer
#2 lore raw

PATCH v3 2/15 selftests/bpf: Add a test for an __int128 by-value argument

In reply to Yonghong Song

A 128-bit integer is passed in two consecutive argument registers, but the verifier counts one argument register per parameter whatever its size. For

__u64 take_i128_global(int a, u128 v, int c)

the compiler passes a in R1, v in R2:R3 and c in R4, while the verifier marks only R1 through R3 at the entry of the global function. The callee then reads its own third parameter out of a register the verifier considers uninitialized, and the program is rejected for a register the source never names:

Validating take_i128_global() func#1...
20: R1=scalar() R2=scalar() R3=scalar() R10=fp0
; __noinline __u64 take_i128_global(int a, u128 v, int c) @ verifier_aggregate_arg.c:12
20: (bf) r0 = r2 ; R0=scalar(id=4) R2=scalar(id=4)
; return (__u64)a + (__u64)(v >> 64) + (__u64)v + c; @ verifier_aggregate_arg.c:14

  21: (bc) w1 = w1                      ; R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
  22: (67) r1 <<= 32                    ; R1=scalar(smax=0x7fffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
  23: (c7) r1 s>>= 32                   ; R1=scalar(smin=0xffffffff80000000,smax=0x7fffffff)
  24: (0f) r0 += r1                     ; R0=scalar() R1=scalar(smin=0xffffffff80000000,smax=0x7fffffff)
  25: (0f) r0 += r3                     ; R0=scalar() R3=scalar()

26: (bc) w1 = w4
R4 !read_ok

The log is from clang 23. LLVM 21/22 place the argument in the same registers.

Add the test with the failure it produces now. A later patch, "bpf:
Support __int128 as a by-value function argument", places the two slots and flips this test to __success.

---
 .../selftests/bpf/prog_tests/verifier.c       |  2 +
 .../bpf/progs/verifier_aggregate_arg.c        | 40 +++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index f7f94ccebce2..3c7ded537314 100644
@@ -5,6 +5,7 @@
 #include "arena_kfunc.skel.h"
 #include "arena_kfunc_jit.skel.h"
 #include "cap_helpers.h"
+#include "verifier_aggregate_arg.skel.h"
 #include "verifier_aggregate_ret.skel.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
@@ -171,6 +172,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
 void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
 
+void test_verifier_aggregate_arg(void)        { RUN_TESTS(verifier_aggregate_arg); }
 void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c new file +40 -0
new file mode 100644
index 000000000000..d90f754396d0
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+#ifdef __SIZEOF_INT128__
+
+typedef unsigned __int128 u128;
+
+__noinline __u64 take_i128_global(int a, u128 v, int c)
+{
+	return (__u64)a + (__u64)(v >> 64) + (__u64)v + c;
+}
+
+SEC("tc")
+/*
+ * The verifier counts one argument register for the __int128 and marks only
+ * R1 through R3 at the entry of take_i128_global(), while the compiler passed
+ * a in R1, v in R2:R3 and c in R4.
+ */
+__failure __msg("R4 !read_ok")
+int aggregate_arg_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v = ((u128)a << 64) | b;
+
+	if (take_i128_global(1, v, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+#endif /* __SIZEOF_INT128__ */
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.52.0
#3 lore raw

Re: [PATCH bpf-next v3 02/15] selftests/bpf: Add a test for an __int128 by-value argument

A subsystem pattern flags this as potentially concerning: every program in the new file is wrapped in '#ifdef __SIZEOF_INT128__' with no '#else' branch.
Quoted 56 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index f7f94ccebce2..3c7ded537314 100644
@@ -5,6 +5,7 @@
 #include "arena_kfunc.skel.h"
 #include "arena_kfunc_jit.skel.h"
 #include "cap_helpers.h"
+#include "verifier_aggregate_arg.skel.h"
 #include "verifier_aggregate_ret.skel.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
@@ -171,6 +172,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
 void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
 
+void test_verifier_aggregate_arg(void)        { RUN_TESTS(verifier_aggregate_arg); }
 void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c new file +30 -0
new file mode 100644
index 000000000000..d90f754396d0
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+#ifdef __SIZEOF_INT128__
+
+typedef unsigned __int128 u128;
+
+__noinline __u64 take_i128_global(int a, u128 v, int c)
+{
+	return (__u64)a + (__u64)(v >> 64) + (__u64)v + c;
+}
+
+SEC("tc")
+/*
+ * The verifier counts one argument register for the __int128 and marks only
+ * R1 through R3 at the entry of take_i128_global(), while the compiler passed
+ * a in R1, v in R2:R3 and c in R4.
+ */
+__failure __msg("R4 !read_ok")
+int aggregate_arg_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v = ((u128)a << 64) | b;

tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c line 40

Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c new file +10 -0
+
+	if (take_i128_global(1, v, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+#endif /* __SIZEOF_INT128__ */
+
+char _license[] SEC("license") = "GPL";

A subsystem pattern flags this as potentially concerning: every program in the new file is wrapped in '#ifdef __SIZEOF_INT128__' with no '#else' branch.

When a BPF compiler does not define __SIZEOF_INT128__ (for example, when building with bpf-gcc via BPF_GCC=1), the generated verifier_aggregate_arg.bpf.o contains no programs at all. The skeleton still compiles because bpftool guards maps/progs/links members on their counts, but test_loader__run_subtests() then iterates zero programs and test_progs.c reports the test as passing with no indication that the __int128 coverage was compiled out.

Every sibling prog file in progs/ that wraps all of its tests in a feature #ifdef adds an '#else' dummy carrying __description()/__skip() so the result is a visible SKIP instead. For instance, verifier_aggregate_ret.c (from the same series) includes such a dummy at lines 167-175 with the message "needs LLVM 23, dummy test". The same pattern appears in verifier_gotol.c, verifier_bswap.c, verifier_ldsx.c, verifier_movsx.c, verifier_sdiv.c, verifier_private_stack.c, verifier_stack_arg_order.c, verifier_load_acquire.c, verifier_store_release.c, arena_kfunc.c, stack_arg_precision.c, and stack_arg_fail.c.

Should this file include an '#else' dummy with __skip() to match the pattern used by its sibling tests, so that builds without __SIZEOF_INT128__ report SKIP rather than silently passing?

Note: __SIZEOF_INT128__ is defined by every clang for the bpf target, so the normal CI matrix (clang test_progs, no_alu32, cpuv4) does compile and run the test. Only the optional bpf-gcc flavor would be affected.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#4 lore raw

Re: [PATCH bpf-next v3 02/15] selftests/bpf: Add a test for an __int128 by-value argument

Quoted 87 lines · On 9/11/26 9:47 AM, [email protected] wrote:
Quoted 56 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index f7f94ccebce2..3c7ded537314 100644
@@ -5,6 +5,7 @@
  #include "arena_kfunc.skel.h"
  #include "arena_kfunc_jit.skel.h"
  #include "cap_helpers.h"
+#include "verifier_aggregate_arg.skel.h"
  #include "verifier_aggregate_ret.skel.h"
  #include "verifier_align.skel.h"
  #include "verifier_and.skel.h"
@@ -171,6 +172,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
  void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
 
+void test_verifier_aggregate_arg(void)        { RUN_TESTS(verifier_aggregate_arg); }
  void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }
  void test_verifier_align(void)                { RUN(verifier_align); }
  void test_verifier_and(void)                  { RUN(verifier_and); }
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c new file +30 -0
new file mode 100644
index 000000000000..d90f754396d0
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+#ifdef __SIZEOF_INT128__
+
+typedef unsigned __int128 u128;
+
+__noinline __u64 take_i128_global(int a, u128 v, int c)
+{
+	return (__u64)a + (__u64)(v >> 64) + (__u64)v + c;
+}
+
+SEC("tc")
+/*
+ * The verifier counts one argument register for the __int128 and marks only
+ * R1 through R3 at the entry of take_i128_global(), while the compiler passed
+ * a in R1, v in R2:R3 and c in R4.
+ */
+__failure __msg("R4 !read_ok")
+int aggregate_arg_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v = ((u128)a << 64) | b;

tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c line 40

Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c new file +10 -0
+
+	if (take_i128_global(1, v, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+#endif /* __SIZEOF_INT128__ */
+
+char _license[] SEC("license") = "GPL";

A subsystem pattern flags this as potentially concerning: every program in the new file is wrapped in '#ifdef __SIZEOF_INT128__' with no '#else' branch.

When a BPF compiler does not define __SIZEOF_INT128__ (for example, when building with bpf-gcc via BPF_GCC=1), the generated verifier_aggregate_arg.bpf.o contains no programs at all. The skeleton still compiles because bpftool guards maps/progs/links members on their counts, but test_loader__run_subtests() then iterates zero programs and test_progs.c reports the test as passing with no indication that the __int128 coverage was compiled out.

Every sibling prog file in progs/ that wraps all of its tests in a feature #ifdef adds an '#else' dummy carrying __description()/__skip() so the result is a visible SKIP instead. For instance, verifier_aggregate_ret.c (from the same series) includes such a dummy at lines 167-175 with the message "needs LLVM 23, dummy test". The same pattern appears in verifier_gotol.c, verifier_bswap.c, verifier_ldsx.c, verifier_movsx.c, verifier_sdiv.c, verifier_private_stack.c, verifier_stack_arg_order.c, verifier_load_acquire.c, verifier_store_release.c, arena_kfunc.c, stack_arg_precision.c, and stack_arg_fail.c.

Quoted 7 lines

Should this file include an '#else' dummy with __skip() to match the pattern used by its sibling tests, so that builds without __SIZEOF_INT128__ report SKIP rather than silently passing?

Note: __SIZEOF_INT128__ is defined by every clang for the bpf target, so the normal CI matrix (clang test_progs, no_alu32, cpuv4) does compile and run the test. Only the optional bpf-gcc flavor would be affected.

This should be okay. Later selftest will have more contents in file verifier_aggregate_arg.c which has some not-guarded bpf programs.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#5 lore raw

PATCH v3 3/15 bpf: Rename bpf_subprog_info::arg_cnt to arg_slot_cnt

In reply to Yonghong Song

Rename arg_cnt to arg_slot_cnt, as a later patch gives a parameter that takes two argument registers, an __int128 or a 16-byte aggregate, two slots. No functional change.

---
 include/linux/bpf_verifier.h                  |  6 +++---
 kernel/bpf/btf.c                              |  2 +-
 kernel/bpf/verifier.c                         | 19 +++++++++++--------
 .../bpf/progs/verifier_stack_arg_order.c      |  4 ++--
 4 files changed, 17 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +3 -3
index 9727df5af83a..edb904424aba 100644
@@ -828,7 +828,7 @@ struct bpf_subprog_info {
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
 	bool might_sleep: 1;
-	u8 arg_cnt:4;
+	u8 arg_slot_cnt:4;
 
 	enum priv_stack_mode priv_stack_mode;
 	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
@@ -838,8 +838,8 @@ struct bpf_subprog_info {
 
 static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
 {
-	if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS)
-		return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS;
+	if (sub->arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS)
+		return sub->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS;
 	return 0;
 }
 
kernel/bpf/btf.c +1 -1
index 31057c8f3a7c..01ec2b40f376 100644
@@ -8081,7 +8081,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	}
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
-	sub->arg_cnt = nargs;
+	sub->arg_slot_cnt = nargs;
 	if (nargs > MAX_BPF_FUNC_ARGS) {
 		bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
 			MAX_BPF_FUNC_ARGS, tname, nargs);
kernel/bpf/verifier.c +11 -8
index fed576b8f7fe..e8f4c17fb27d 100644
@@ -9772,7 +9772,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	ret = btf_prepare_func_args(env, subprog);
 	if (ret) {
 		if (bpf_in_stack_arg_cnt(sub) > 0) {
-			err = check_outgoing_stack_args(env, caller, sub->arg_cnt,
+			err = check_outgoing_stack_args(env, caller, sub->arg_slot_cnt,
 							bpf_subprog_name(env, subprog),
 							NULL, NULL);
 			if (err)
@@ -9784,7 +9784,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
 	func_proto = btf_type_by_id(btf, func->type);
 	args = btf_params(func_proto);
-	ret = check_outgoing_stack_args(env, caller, sub->arg_cnt,
+	ret = check_outgoing_stack_args(env, caller, sub->arg_slot_cnt,
 					bpf_subprog_name(env, subprog), btf, args);
 	if (ret)
 		return ret;
@@ -9792,7 +9792,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	/* check that BTF function arguments match actual types that the
 	 * verifier sees.
 	 */
-	for (i = 0; i < sub->arg_cnt; i++) {
+	for (i = 0; i < sub->arg_slot_cnt; i++) {
 		argno_t argno = argno_from_arg(i + 1);
 		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
 		struct bpf_subprog_arg_info *arg = &sub->args[i];
@@ -19777,13 +19777,14 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog)
 			}
 
 			/* Also ensure the callback only has a single scalar argument. */
-			if (sub->arg_cnt != 1 || sub->args[0].arg_type != ARG_ANYTHING) {
+			if (sub->arg_slot_cnt != 1 || sub->args[0].arg_type != ARG_ANYTHING) {
 				verbose(env, "exception cb only supports single integer argument\n");
 				ret = -EINVAL;
 				goto out;
 			}
 		}
-		for (i = BPF_REG_1; i <= min_t(u32, sub->arg_cnt, MAX_BPF_FUNC_REG_ARGS); i++) {
+		for (i = BPF_REG_1;
+		     i <= min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++) {
 			arg = &sub->args[i - BPF_REG_1];
 			reg = &regs[i];
 
@@ -19826,7 +19827,8 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog)
 				goto out;
 			}
 		}
-		if (env->prog->type == BPF_PROG_TYPE_EXT && sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS) {
+		if (env->prog->type == BPF_PROG_TYPE_EXT &&
+		    sub->arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS) {
 			verbose(env, "freplace programs with >%d args not supported yet\n",
 				MAX_BPF_FUNC_REG_ARGS);
 			ret = -EINVAL;
@@ -19839,9 +19841,10 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog)
 		 */
 		if (env->prog->aux->func_info_aux) {
 			ret = btf_prepare_func_args(env, 0);
-			if (ret || sub->arg_cnt != 1 || sub->args[0].arg_type != ARG_PTR_TO_CTX) {
+			if (ret || sub->arg_slot_cnt != 1 ||
+			    sub->args[0].arg_type != ARG_PTR_TO_CTX) {
 				env->prog->aux->func_info_aux[0].unreliable = true;
-				sub->arg_cnt = 1;
+				sub->arg_slot_cnt = 1;
 				sub->stack_arg_cnt = 0;
 			}
 		}
tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c +2 -2
index 57f22691744a..ab1955852233 100644
@@ -116,8 +116,8 @@ __naked void stack_arg_pruning_load_after_call(void)
 /*
  * "bad_ptr": the first arg is 'long *', which is not a recognized pointer
  * type for static subprogs (not ctx, dynptr, or tagged).  btf_prepare_func_args()
- * sets arg_cnt = 7 / stack_arg_cnt = 2, then fails with -EINVAL.  The subprog
- * is marked unreliable but the call still proceeds for static subprogs.
+ * sets arg_slot_cnt = 7 / stack_arg_cnt = 2, then fails with -EINVAL.  The
+ * subprog is marked unreliable but the call still proceeds for static subprogs.
  */
 __noinline __used __naked
 static void subprog_bad_ptr_7args(long *a, int b, int c, int d, int e, int f, int g)
Signature · 1 line
2.52.0
author reviewer
#6 lore raw

PATCH v3 4/15 bpf: Index global function arguments by argument slot

In reply to Yonghong Song

btf_prepare_func_args() indexes sub->args[] by BTF parameter, so a parameter can only ever stand for a single argument register. Give the loop a second index: 'i' keeps walking the BTF parameters while 'slots_used' walks the argument slots, and sub->arg_slot_cnt becomes the number of slots, so that a later patch can give a parameter two of them.

No functional change: every parameter still takes exactly one slot.

---
 kernel/bpf/btf.c | 40 ++++++++++++++++++++++++----------------
 1 file changed, 24 insertions(+), 16 deletions(-)
kernel/bpf/btf.c +24 -16
index 01ec2b40f376..35eaa28c85b5 100644
@@ -8037,7 +8037,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	const struct btf_param *args;
 	const struct btf_type *t, *ref_t, *fn_t;
 	int err;
-	u32 i, nargs, btf_id;
+	u32 i, slots_used, nargs, btf_id;
 	const char *tname;
 
 	if (sub->args_cached)
@@ -8122,8 +8122,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	/* Convert BTF function arguments into verifier types.
 	 * Only PTR_TO_CTX and SCALAR are supported atm.
 	 */
-	for (i = 0; i < nargs; i++) {
+	for (i = 0, slots_used = 0; i < nargs; i++) {
 		u32 tags = 0;
+
 		err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags);
 		if (err)
 			return err;
@@ -8145,7 +8146,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			    btf_validate_prog_ctx_type(log, btf, t, i, prog_type,
 						       prog->expected_attach_type))
 				return -EINVAL;
-			sub->args[i].arg_type = ARG_PTR_TO_CTX;
+			sub->args[slots_used++].arg_type = ARG_PTR_TO_CTX;
 			continue;
 		}
 		if (btf_is_dynptr_ptr(btf, t)) {
@@ -8153,7 +8154,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 				bpf_log(log, "arg#%d has invalid combination of tags\n", i);
 				return -EINVAL;
 			}
-			sub->args[i].arg_type = ARG_PTR_TO_DYNPTR;
+			sub->args[slots_used++].arg_type = ARG_PTR_TO_DYNPTR;
 			continue;
 		}
 		if (tags & ARG_TAG_TRUSTED) {
@@ -8168,10 +8169,11 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			if (kern_type_id < 0)
 				return kern_type_id;
 
-			sub->args[i].arg_type = ARG_PTR_TO_BTF_ID | PTR_TRUSTED;
+			sub->args[slots_used].arg_type = ARG_PTR_TO_BTF_ID | PTR_TRUSTED;
 			if (tags & ARG_TAG_NULLABLE)
-				sub->args[i].arg_type |= PTR_MAYBE_NULL;
-			sub->args[i].btf_id = kern_type_id;
+				sub->args[slots_used].arg_type |= PTR_MAYBE_NULL;
+			sub->args[slots_used].btf_id = kern_type_id;
+			slots_used++;
 			continue;
 		}
 		if (tags & ARG_TAG_UNTRUSTED) {
@@ -8185,8 +8187,10 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 
 			ref_t = btf_type_skip_modifiers(btf, t->type, NULL);
 			if (btf_type_is_void(ref_t) || btf_type_is_primitive(ref_t)) {
-				sub->args[i].arg_type = ARG_PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED;
-				sub->args[i].mem_size = 0;
+				sub->args[slots_used].arg_type = ARG_PTR_TO_MEM | MEM_RDONLY |
+								 PTR_UNTRUSTED;
+				sub->args[slots_used].mem_size = 0;
+				slots_used++;
 				continue;
 			}
 
@@ -8202,8 +8206,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 					i, btf_type_str(ref_t), tname);
 				return -EINVAL;
 			}
-			sub->args[i].arg_type = ARG_PTR_TO_BTF_ID | PTR_UNTRUSTED;
-			sub->args[i].btf_id = kern_type_id;
+			sub->args[slots_used].arg_type = ARG_PTR_TO_BTF_ID | PTR_UNTRUSTED;
+			sub->args[slots_used].btf_id = kern_type_id;
+			slots_used++;
 			continue;
 		}
 		if (tags & ARG_TAG_ARENA) {
@@ -8211,7 +8216,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 				bpf_log(log, "arg#%d arena cannot be combined with any other tags\n", i);
 				return -EINVAL;
 			}
-			sub->args[i].arg_type = ARG_PTR_TO_ARENA;
+			sub->args[slots_used++].arg_type = ARG_PTR_TO_ARENA;
 			continue;
 		}
 		if (is_global) { /* generic user data pointer */
@@ -8231,10 +8236,11 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 				return -EINVAL;
 			}
 
-			sub->args[i].arg_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL;
+			sub->args[slots_used].arg_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL;
 			if (tags & ARG_TAG_NONNULL)
-				sub->args[i].arg_type &= ~PTR_MAYBE_NULL;
-			sub->args[i].mem_size = mem_size;
+				sub->args[slots_used].arg_type &= ~PTR_MAYBE_NULL;
+			sub->args[slots_used].mem_size = mem_size;
+			slots_used++;
 			continue;
 		}
 
@@ -8244,7 +8250,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			return -EINVAL;
 		}
 		if (btf_type_is_int(t) || btf_is_any_enum(t)) {
-			sub->args[i].arg_type = ARG_ANYTHING;
+			sub->args[slots_used++].arg_type = ARG_ANYTHING;
 			continue;
 		}
 		if (!is_global)
@@ -8254,6 +8260,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
+	sub->arg_slot_cnt = slots_used;
+
 	sub->args_cached = true;
 
 	return 0;
Signature · 1 line
2.52.0
author reviewer
#7 lore raw

PATCH v3 5/15 bpf: Support by-value struct arguments up to 16 bytes

In reply to Yonghong Song

A global function taking a struct or union by value is rejected today:

Arg#1 type STRUCT in tar() is not supported yet.

Accept one of at most 16 bytes, which arrives in one or two consecutive argument registers. Only structs composed entirely of scalars are taken for now; btf_struct_is_composed_of() enforces that.

The slots of a value are independent of each other, so the compiler may split one across the last argument register and the stack, as in

static void f(int a, int b, int c, int d, struct pair p);

or place it wholly past the registers, and the verifier describes either the same way. What has to follow the slots is stack_arg_cnt, recomputed from the slots consumed, together with the "no stack args in global functions" and JIT support checks, and the MAX_BPF_FUNC_ARGS bound, which now has to account for a parameter that takes two slots at once.

---
 kernel/bpf/btf.c      | 80 +++++++++++++++++++++++++++++++++++--------
 kernel/bpf/verifier.c |  2 ++
 2 files changed, 68 insertions(+), 14 deletions(-)
kernel/bpf/btf.c +66 -14
index 35eaa28c85b5..5bbc1ba00ae2 100644
@@ -8018,6 +8018,29 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	return -EOPNOTSUPP;
 }
 
+static int btf_check_arg_slots(struct bpf_verifier_log *log, const char *tname,
+			       bool is_global, u32 slot_cnt,
+			       struct bpf_subprog_info *sub)
+{
+	if (slot_cnt <= MAX_BPF_FUNC_REG_ARGS)
+		return 0;
+
+	if (is_global) {
+		bpf_log(log,
+			"global function %s() needs %d > %d argument slots, "
+			"stack args not supported\n",
+			tname, slot_cnt, MAX_BPF_FUNC_REG_ARGS);
+		return -EINVAL;
+	}
+	if (!bpf_jit_supports_stack_args()) {
+		bpf_log(log, "JIT does not support function %s() with %d argument slots\n",
+			tname, slot_cnt);
+		return -EFAULT;
+	}
+	sub->stack_arg_cnt = slot_cnt - MAX_BPF_FUNC_REG_ARGS;
+	return 0;
+}
+
 /* Process BTF of a function to produce high-level expectation of function
  * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information
  * is cached in subprog info for reuse.
@@ -8087,20 +8110,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			MAX_BPF_FUNC_ARGS, tname, nargs);
 		return -EFAULT;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-		if (!bpf_jit_supports_stack_args()) {
-			bpf_log(log, "JIT does not support function %s() with %d args\n",
-				tname, nargs);
-			return -EFAULT;
-		}
-		sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
-	}
-
-	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
-		bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
-			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
-		return -EINVAL;
-	}
+	err = btf_check_arg_slots(log, tname, is_global, nargs, sub);
+	if (err)
+		return err;
 
 	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
@@ -8125,6 +8137,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	for (i = 0, slots_used = 0; i < nargs; i++) {
 		u32 tags = 0;
 
+		if (slots_used >= MAX_BPF_FUNC_ARGS)
+			goto too_many_slots;
+
 		err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags);
 		if (err)
 			return err;
@@ -8253,6 +8268,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			sub->args[slots_used++].arg_type = ARG_ANYTHING;
 			continue;
 		}
+		if (btf_type_is_struct(t)) {
+			u32 nslots;
+
+			if (!t->size || t->size > 2 * BPF_REG_SIZE) {
+				if (!is_global)
+					return -EINVAL;
+				bpf_log(log,
+					"Arg#%d type %s in %s() has size %u, only 1 to %d bytes "
+					"can be passed by value\n",
+					i, btf_type_str(t), tname, t->size, 2 * BPF_REG_SIZE);
+				return -EINVAL;
+			}
+			if (!btf_struct_is_composed_of(env, btf, t, BTF_MEMBER_SCALAR)) {
+				if (!is_global)
+					return -EINVAL;
+				bpf_log(log, "Arg#%d type %s in %s() is not composed of scalars\n",
+					i, btf_type_str(t), tname);
+				return -EINVAL;
+			}
+
+			nslots = (t->size + BPF_REG_SIZE - 1) / BPF_REG_SIZE;
+			if (slots_used + nslots > MAX_BPF_FUNC_ARGS)
+				goto too_many_slots;
+			while (nslots--)
+				sub->args[slots_used++].arg_type = ARG_ANYTHING;
+			continue;
+		}
 		if (!is_global)
 			return -EINVAL;
 		bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",
@@ -8260,11 +8302,21 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
+	err = btf_check_arg_slots(log, tname, is_global, slots_used, sub);
+	if (err)
+		return err;
 	sub->arg_slot_cnt = slots_used;
 
 	sub->args_cached = true;
 
 	return 0;
+
+too_many_slots:
+	if (!is_global)
+		return -EINVAL;
+	bpf_log(log, "Arguments of %s() need more than %d argument slots\n",
+		tname, MAX_BPF_FUNC_ARGS);
+	return -EINVAL;
 }
 
 static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
kernel/bpf/verifier.c +2 -0
index e8f4c17fb27d..500092a2c924 100644
@@ -9784,6 +9784,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
 	func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
 	func_proto = btf_type_by_id(btf, func->type);
 	args = btf_params(func_proto);
+	if (sub->arg_slot_cnt != btf_type_vlen(func_proto))
+		args = NULL;
 	ret = check_outgoing_stack_args(env, caller, sub->arg_slot_cnt,
 					bpf_subprog_name(env, subprog), btf, args);
 	if (ret)
Signature · 1 line
2.52.0
author reviewer
#8 lore raw

PATCH v3 6/15 bpf: Support __int128 as a by-value function argument

In reply to Yonghong Song

A 128-bit integer follows the same calling convention as a 16-byte by-value struct: LLVM emits it as a 16-byte BTF_KIND_INT and passes it in two consecutive argument registers. So a __int128 gets its two slots.

The test added at the start of the series, which recorded the wrong register map as an R4 !read_ok rejection, now passes and is flipped to __success.

---
 kernel/bpf/btf.c                                         | 9 +++------
 .../testing/selftests/bpf/progs/verifier_aggregate_arg.c | 7 +------
 2 files changed, 4 insertions(+), 12 deletions(-)
kernel/bpf/btf.c +3 -6
index 5bbc1ba00ae2..088788e29b03 100644
@@ -8264,11 +8264,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			bpf_log(log, "arg#%d has pointer tag, but is not a pointer type\n", i);
 			return -EINVAL;
 		}
-		if (btf_type_is_int(t) || btf_is_any_enum(t)) {
-			sub->args[slots_used++].arg_type = ARG_ANYTHING;
-			continue;
-		}
-		if (btf_type_is_struct(t)) {
+		if (btf_type_is_int(t) || btf_is_any_enum(t) || btf_type_is_struct(t)) {
 			u32 nslots;
 
 			if (!t->size || t->size > 2 * BPF_REG_SIZE) {
@@ -8280,7 +8276,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 					i, btf_type_str(t), tname, t->size, 2 * BPF_REG_SIZE);
 				return -EINVAL;
 			}
-			if (!btf_struct_is_composed_of(env, btf, t, BTF_MEMBER_SCALAR)) {
+			if (btf_type_is_struct(t) &&
+			    !btf_struct_is_composed_of(env, btf, t, BTF_MEMBER_SCALAR)) {
 				if (!is_global)
 					return -EINVAL;
 				bpf_log(log, "Arg#%d type %s in %s() is not composed of scalars\n",
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c +1 -6
index d90f754396d0..fc7c1b18bc40 100644
@@ -17,12 +17,7 @@ __noinline __u64 take_i128_global(int a, u128 v, int c)
 }
 
 SEC("tc")
-/*
- * The verifier counts one argument register for the __int128 and marks only
- * R1 through R3 at the entry of take_i128_global(), while the compiler passed
- * a in R1, v in R2:R3 and c in R4.
- */
-__failure __msg("R4 !read_ok")
+__success __retval(0)
 int aggregate_arg_int128_c_test(struct __sk_buff *skb)
 {
 	__u64 a = skb->len ^ MIX_A;
Signature · 1 line
2.52.0
author reviewer
#9 lore raw

PATCH v3 7/15 bpf: Rename bpf_call_summary::num_params to arg_slot_cnt

In reply to Yonghong Song

The field 'num_params' counts the argument registers and outgoing stack slots a helper or kfunc call takes. The next patch gives a 16-byte parameter two slots, so the name stops describing what the field holds.

Rename 'num_params' to 'arg_slot_cnt'. No functional change.

---
 include/linux/bpf_verifier.h |  2 +-
 kernel/bpf/liveness.c        | 10 +++++-----
 kernel/bpf/verifier.c        |  8 ++++----
 3 files changed, 10 insertions(+), 10 deletions(-)
include/linux/bpf_verifier.h +1 -1
index edb904424aba..64dfa29f6b45 100644
@@ -1066,7 +1066,7 @@ static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 }
 
 struct bpf_call_summary {
-	u8 num_params;
+	u8 arg_slot_cnt;
 	bool is_void;
 	bool fastcall;
 };
kernel/bpf/liveness.c +5 -5
index 7165ea325961..44ecdc5b4ec2 100644
@@ -1434,21 +1434,21 @@ static int record_call_access(struct bpf_verifier_env *env,
 {
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	struct bpf_call_summary cs;
-	int r, err, num_params = 5;
+	int r, err, arg_slot_cnt = 5;
 
 	if (bpf_pseudo_call(insn))
 		return 0;
 
 	if (bpf_get_call_summary(env, insn, &cs))
-		num_params = cs.num_params;
+		arg_slot_cnt = cs.arg_slot_cnt;
 
-	for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) {
+	for (r = BPF_REG_1; r < BPF_REG_1 + min(arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); r++) {
 		err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx);
 		if (err)
 			return err;
 	}
 
-	for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) {
+	for (r = 0; r < MAX_STACK_ARG_SLOTS && r < arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS; r++) {
 		err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r],
 					r + MAX_BPF_FUNC_REG_ARGS, insn_idx);
 		if (err)
@@ -2199,7 +2199,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			def = ALL_CALLER_SAVED_REGS;
 			use = def & ~BIT(BPF_REG_0);
 			if (bpf_get_call_summary(env, insn, &cs))
-				use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
+				use = GENMASK(min_t(u8, cs.arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS), 1);
 			def = mask_widen(def);
 			use = mask_widen(use);
 			break;
kernel/bpf/verifier.c +4 -4
index 500092a2c924..c520c37bb3c9 100644
@@ -17973,11 +17973,11 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			       (bpf_verifier_inlines_helper_call(env, call->imm) ||
 				bpf_jit_inlines_helper_call(call->imm));
 		cs->is_void = fn->ret_type == RET_VOID;
-		cs->num_params = 0;
+		cs->arg_slot_cnt = 0;
 		for (i = 0; i < ARRAY_SIZE(fn->arg_type); ++i) {
 			if (fn->arg_type[i] == ARG_DONTCARE)
 				break;
-			cs->num_params++;
+			cs->arg_slot_cnt++;
 		}
 		return true;
 	}
@@ -17989,7 +17989,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 		if (err < 0)
 			/* error would be reported later */
 			return false;
-		cs->num_params = btf_type_vlen(meta.func_proto);
+		cs->arg_slot_cnt = btf_type_vlen(meta.func_proto);
 		cs->fastcall = meta.kfunc_flags & KF_FASTCALL;
 		cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type));
 		return true;
@@ -18098,7 +18098,7 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
 	 * - includes R1-R5 if corresponding parameter has is described
 	 *   in the function prototype.
 	 */
-	clobbered_regs_mask = GENMASK(cs.num_params, cs.is_void ? 1 : 0);
+	clobbered_regs_mask = GENMASK(cs.arg_slot_cnt, cs.is_void ? 1 : 0);
 	/* e.g. if helper call clobbers r{0,1}, expect r{2,3,4,5} in the pattern */
 	expected_regs_mask = ~clobbered_regs_mask & ALL_CALLER_SAVED_REGS;
 
Signature · 1 line
2.52.0
author reviewer
#10 lore raw

PATCH v3 8/15 bpf: Recognize by-value struct and __int128 kfunc arguments

In reply to Yonghong Song

A kfunc taking a struct or union by value is rejected today, and one taking an __int128 is accepted but mis-counted:

Unrecognized R2 type STRUCT

The kfunc arguments walk the same slot as a BPF-to-BPF call:
one argument register per eightbyte, and a 16-byte value takes two. The outgoing stack argument count at the call site follows the slots for the same reason. Similar to BPF-to-BPF aggregate handling, a kfunc aggregate argument is only supported when it is composed of scalars. Everything that maps a kfunc argument to a register has to follow the slots too.

An argument of a single eightbyte lands in the same place under every calling convention, so those are taken. The conventions the JIT has to reconcile do not agree on where a larger one goes, so refuse it for now with

Function f arg#1 type INT cannot be passed at argument slot 1 on this architecture

which the next patch turns into an answer from the JIT. The paths that handle a two-slot argument are therefore unreachable until then.

---
 kernel/bpf/verifier.c | 177 +++++++++++++++++++++++++++++++++---------
 1 file changed, 141 insertions(+), 36 deletions(-)
kernel/bpf/verifier.c +141 -36
index c520c37bb3c9..9e70995c763f 100644
@@ -12149,12 +12149,36 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta)
 	return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data];
 }
 
+/*
+ * The slots a parameter takes, from its BTF type. This has to agree with
+ * btf_func_model_arg_slots(), which answers the same from the size the
+ * func model recorded, or the verifier would check an argument at a slot
+ * the JIT does not place it at.
+ */
+static u32 kfunc_arg_slots(const struct btf_type *t)
+{
+	if (btf_type_is_int(t) || btf_type_is_struct(t))
+		return (t->size + BPF_REG_SIZE - 1) / BPF_REG_SIZE;
+	return 1;
+}
+
+static u32 kfunc_proto_slots(const struct btf *btf, const struct btf_type *func_proto)
+{
+	const struct btf_param *args = btf_params(func_proto);
+	u32 i, nargs = btf_type_vlen(func_proto), slots_used = 0;
+
+	for (i = 0; i < nargs; i++)
+		slots_used += kfunc_arg_slots(btf_type_skip_modifiers(btf, args[i].type, NULL));
+
+	return slots_used;
+}
+
 static int
 get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
-		   const struct btf_param *args, int arg, int nargs)
+		   const struct btf_param *args, int arg, int nargs, u32 slot)
 {
 	const struct btf_type *t, *ref_t = NULL;
-	argno_t argno = argno_from_arg(arg + 1);
+	argno_t argno = argno_from_arg(slot + 1);
 	const char *ref_tname = NULL;
 	int arg_type;
 
@@ -12174,6 +12198,23 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		return KF_ARG_ANYTHING;
 	}
 
+	if (btf_type_is_struct(t)) {
+		if (!t->size || t->size > 2 * BPF_REG_SIZE) {
+			verbose(env,
+				"%s type %s has size %u, only 1 to %d bytes "
+				"can be passed by value\n",
+				reg_arg_name(env, argno), btf_type_str(t), t->size,
+				2 * BPF_REG_SIZE);
+			return -EINVAL;
+		}
+		if (!btf_type_is_scalar_struct(env, meta->btf, t)) {
+			verbose(env, "%s type %s is not composed of scalars\n",
+				reg_arg_name(env, argno), btf_type_str(t));
+			return -EINVAL;
+		}
+		return KF_ARG_ANYTHING;
+	}
+
 	if (!btf_type_is_ptr(t)) {
 		verbose(env, "Unrecognized %s type %s\n",
 			reg_arg_name(env, argno), btf_type_str(t));
@@ -12289,7 +12330,7 @@ static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg
 {
 	const struct btf *btf = meta->btf;
 	const struct btf_param *args;
-	u32 i, nargs;
+	u32 i, nargs, slots_used;
 	int arg_type;
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
@@ -12305,19 +12346,44 @@ static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg
 		return -ENOTSUPP;
 	}
 
-	for (i = 0; i < nargs; i++) {
+	for (i = 0, slots_used = 0; i < nargs; i++) {
+		const struct btf_type *t;
+		u32 nslots;
+
+		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
+		nslots = kfunc_arg_slots(t);
+		/*
+		 * The calling conventions the JIT has to reconcile do not
+		 * agree on where an argument of more than one eightbyte goes,
+		 * so refuse one until the JIT can say where this arch puts it.
+		 */
+		if (nslots > 1) {
+			verbose(env,
+				"Function %s arg#%d type %s cannot be passed at "
+				"argument slot %d on this architecture\n",
+				meta->func_name, i, btf_type_str(t), slots_used);
+			return -EINVAL;
+		}
+		slots_used += nslots;
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i]) ||
 		    is_kfunc_arg_ignore(btf, &args[i]) ||
 		    is_kfunc_arg_implicit(meta, i))
 			continue;
 
-		arg_type = get_kfunc_arg_type(env, meta, args, i, nargs);
+		arg_type = get_kfunc_arg_type(env, meta, args, i, nargs, slots_used - nslots);
 		if (arg_type < 0)
 			return arg_type;
 
 		proto->arg_type[i] = arg_type;
 	}
 
+	if (slots_used > MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args()) {
+		verbose(env, "JIT does not support kfunc %s() with %d argument slots\n",
+			meta->func_name, slots_used);
+		return -ENOTSUPP;
+	}
+
 	return 0;
 }
 
@@ -12883,6 +12949,26 @@ static bool check_css_task_iter_allowlist(struct bpf_verifier_env *env)
 	}
 }
 
+/* The argument slot @slot of the call has to hold a scalar. */
+static int check_kfunc_scalar_arg(struct bpf_verifier_env *env, struct bpf_func_state *caller,
+				  struct bpf_reg_state *regs, u32 slot, int insn_idx,
+				  const char *func_name)
+{
+	struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, slot);
+	argno_t argno = argno_from_arg(slot + 1);
+
+	if (reg->type == SCALAR_VALUE)
+		return 0;
+
+	verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno));
+	bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name,
+			      "Pass an integer scalar value for this argument, not a pointer or resource object.",
+			      "the kfunc expects an integer scalar, but %s is %s",
+			      reg_arg_name(env, argno),
+			      bpf_diag_reg_type_plain(env, reg->type));
+	return -EINVAL;
+}
+
 static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 			    int insn_idx)
 {
@@ -12892,29 +12978,35 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 	const struct btf *btf = meta->btf;
 	const struct btf_param *args;
 	struct btf_record *rec;
-	u32 i, nargs;
+	u32 i, k, nargs, proto_slots, slots_used, prev_slot = 0, nslots = 0;
 	int ret;
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
+	proto_slots = kfunc_proto_slots(btf, meta->func_proto);
 
-	ret = check_outgoing_stack_args(env, caller, nargs, func_name, btf, args);
+	ret = check_outgoing_stack_args(env, caller, proto_slots, func_name, btf,
+					proto_slots == nargs ? args : NULL);
 	if (ret)
 		return ret;
 
 	/* Check that BTF function arguments match actual types that the
 	 * verifier sees.
 	 */
-	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
+	for (i = 0, slots_used = 0; i < nargs;
+	     i++, prev_slot = slots_used, slots_used += nslots) {
+		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, slots_used);
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
-		argno_t argno = argno_from_arg(i + 1);
+		argno_t argno = argno_from_arg(slots_used + 1);
 		int regno = reg_from_argno(argno);
 		bool btf_id_fixed_off_ok = true;
 		u32 ref_id = args[i].type, type_size;
 		int kf_arg_type = meta->fn->arg_type[i];
 
+		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
+		nslots = kfunc_arg_slots(t);
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
 			/* Reject repeated use bpf_prog_aux */
 			if (meta->arg_prog) {
@@ -12934,8 +13026,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
 			continue;
 
-		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
-
 		if (btf_type_is_ptr(t)) {
 			ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
 			ref_tname = btf_name_by_offset(btf, ref_t->name_off);
@@ -12986,6 +13076,17 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 			ref_tname = btf_name_by_offset(btf, ref_t->name_off);
 		}
 
+		/*
+		 * The first slot is checked in below KF_ARG_ANYTHING.
+		 * The rest of it has to be a scalar.
+		 */
+		for (k = 1; k < nslots; k++) {
+			ret = check_kfunc_scalar_arg(env, caller, regs, slots_used + k,
+						     insn_idx, func_name);
+			if (ret)
+				return ret;
+		}
+
 		switch (base_type(kf_arg_type)) {
 		case KF_ARG_CONST:
 		case KF_ARG_CONST_MEM_SIZE:
@@ -13055,15 +13156,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 			}
 			break;
 		case KF_ARG_ANYTHING:
-			if (reg->type != SCALAR_VALUE) {
-				verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno));
-				bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name,
-						      "Pass an integer scalar value for this argument, not a pointer or resource object.",
-						      "the kfunc expects an integer scalar, but %s is %s",
-						      reg_arg_name(env, argno),
-						      bpf_diag_reg_type_plain(env, reg->type));
-				return -EINVAL;
-			}
+			ret = check_kfunc_scalar_arg(env, caller, regs, slots_used,
+						     insn_idx, func_name);
+			if (ret)
+				return ret;
 			break;
 		case KF_ARG_CONST_ALLOC_SIZE_OR_ZERO:
 			if (reg->type != SCALAR_VALUE) {
@@ -13403,9 +13499,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 			fallthrough;
 		case KF_ARG_MEM_SIZE:
 		{
-			struct bpf_reg_state *buff_reg = get_func_arg_reg(caller, regs, i - 1);
+			struct bpf_reg_state *buff_reg = get_func_arg_reg(caller, regs, prev_slot);
 			struct bpf_reg_state *size_reg = reg;
-			argno_t buff_argno = argno_from_arg(i);
+			argno_t buff_argno = argno_from_arg(prev_slot + 1);
 			enum bpf_mem_size_failure failure;
 
 			if (reg->type != SCALAR_VALUE) {
@@ -13751,7 +13847,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 	const struct btf_param *args;
 	const struct btf_type *t, *ref_t;
 	const struct btf *btf;
-	u32 nargs, type_size;
+	u32 i, slot, nargs, type_size;
 	s64 size;
 
 	if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
@@ -13760,23 +13856,32 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 	btf = meta.btf;
 	args = btf_params(meta.func_proto);
 	nargs = btf_type_vlen(meta.func_proto);
-	if (arg >= nargs)
+
+	/*
+	 * @arg is an argument slot and a 16-byte parameter takes two of them,
+	 * so walk the parameters to find the one that starts at this slot. A
+	 * slot holding the upper eightbyte of such a parameter belongs to no
+	 * pointer, and neither does a slot past the last parameter.
+	 */
+	for (i = 0, slot = 0; i < nargs && slot < arg; i++)
+		slot += kfunc_arg_slots(btf_type_skip_modifiers(btf, args[i].type, NULL));
+	if (i >= nargs || slot != arg)
 		return 0;
 
-	t = btf_type_skip_modifiers(btf, args[arg].type, NULL);
+	t = btf_type_skip_modifiers(btf, args[i].type, NULL);
 	if (!btf_type_is_ptr(t))
 		return 0;
 
 	/* dynptr: fixed 16-byte on-stack representation */
-	if (is_kfunc_arg_dynptr(btf, &args[arg])) {
+	if (is_kfunc_arg_dynptr(btf, &args[i])) {
 		size = BPF_DYNPTR_SIZE;
 		goto out;
 	}
 
 	/* ptr + __sz/__szk pair: the size follows the pointer */
-	if (arg + 1 < nargs &&
-	    (btf_param_match_suffix(btf, &args[arg + 1], "__sz") ||
-	     btf_param_match_suffix(btf, &args[arg + 1], "__szk"))) {
+	if (i + 1 < nargs &&
+	    (btf_param_match_suffix(btf, &args[i + 1], "__sz") ||
+	     btf_param_match_suffix(btf, &args[i + 1], "__szk"))) {
 		int size_reg = BPF_REG_1 + arg + 1;
 
 		if (size_reg <= MAX_BPF_FUNC_REG_ARGS &&
@@ -13799,7 +13904,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 	/* KF_ITER_NEW kfuncs initialize the iterator state at arg 0 */
 	if (arg == 0 && meta.kfunc_flags & KF_ITER_NEW)
 		return -size;
-	if (is_kfunc_arg_uninit(btf, &args[arg]))
+	if (is_kfunc_arg_uninit(btf, &args[i]))
 		return -size;
 	return size;
 }
@@ -13992,7 +14097,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	struct bpf_insn_aux_data *insn_aux;
 	const char *operation;
 	int err, insn_idx = *insn_idx_p;
-	u32 i, nargs, ptr_type_id, ret_nregs = 1;
+	u32 i, proto_slots, ptr_type_id, ret_nregs = 1;
 	struct bpf_kfunc_desc *desc;
 	struct btf *desc_btf;
 	int id;
@@ -14418,11 +14523,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	if (bpf_is_kfunc_pkt_changing(&meta))
 		clear_all_pkt_pointers(env);
 
-	nargs = btf_type_vlen(meta.func_proto);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+	proto_slots = kfunc_proto_slots(desc_btf, meta.func_proto);
+	if (proto_slots > MAX_BPF_FUNC_REG_ARGS) {
 		struct bpf_func_state *caller = cur_func(env);
 		struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
-		u16 out_stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+		u16 out_stack_arg_cnt = proto_slots - MAX_BPF_FUNC_REG_ARGS;
 		u16 stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + out_stack_arg_cnt;
 
 		if (stack_arg_cnt > caller_info->stack_arg_cnt)
@@ -17989,7 +18094,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 		if (err < 0)
 			/* error would be reported later */
 			return false;
-		cs->arg_slot_cnt = btf_type_vlen(meta.func_proto);
+		cs->arg_slot_cnt = kfunc_proto_slots(meta.btf, meta.func_proto);
 		cs->fastcall = meta.kfunc_flags & KF_FASTCALL;
 		cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type));
 		return true;
Signature · 1 line
2.52.0
#11 lore raw

Re: [PATCH bpf-next v3 08/15] bpf: Recognize by-value struct and __int128 kfunc arguments

This looks like a bug fix for kfunc struct argument support. Should this include:
Quoted 20 lines

bpf: Recognize by-value struct and __int128 kfunc arguments

A kfunc taking a struct or union by value is rejected today, and one taking an __int128 is accepted but mis-counted:

Unrecognized R2 type STRUCT

The kfunc arguments walk the same slot as a BPF-to-BPF call:
one argument register per eightbyte, and a 16-byte value takes two. The outgoing stack argument count at the call site follows the slots for the same reason. Similar to BPF-to-BPF aggregate handling, a kfunc aggregate argument is only supported when it is composed of scalars. Everything that maps a kfunc argument to a register has to follow the slots too.

An argument of a single eightbyte lands in the same place under every calling convention, so those are taken. The conventions the JIT has to reconcile do not agree on where a larger one goes, so refuse it for now with

Quoted 7 lines

Function f arg#1 type INT cannot be passed at argument slot 1 on this architecture

which the next patch turns into an answer from the JIT. The paths that handle a two-slot argument are therefore unreachable until then.

This looks like a bug fix for kfunc struct argument support. Should this include:

Fixes 0ab8eb34abe4 ("bpf: Support by-value struct arguments up to 16 bytes")
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index c520c37bb3c9..9e70995c763f 100644

[ ... ]

Quoted 41 lines
kernel/bpf/verifier.c +11 -5
@@ -12892,29 +12978,35 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 	const struct btf *btf = meta->btf;
 	const struct btf_param *args;
 	struct btf_record *rec;
-	u32 i, nargs;
+	u32 i, k, nargs, proto_slots, slots_used, prev_slot = 0, nslots = 0;
 	int ret;
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
+	proto_slots = kfunc_proto_slots(btf, meta->func_proto);
 
-	ret = check_outgoing_stack_args(env, caller, nargs, func_name, btf, args);
+	ret = check_outgoing_stack_args(env, caller, proto_slots, func_name, btf,
+					proto_slots == nargs ? args : NULL);
 	if (ret)
 		return ret;
 
 	/* Check that BTF function arguments match actual types that the
 	 * verifier sees.
 	 */
-	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
+	for (i = 0, slots_used = 0; i < nargs;
+	     i++, prev_slot = slots_used, slots_used += nslots) {
+		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, slots_used);
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
-		argno_t argno = argno_from_arg(i + 1);
+		argno_t argno = argno_from_arg(slots_used + 1);
 		int regno = reg_from_argno(argno);
 		bool btf_id_fixed_off_ok = true;
 		u32 ref_id = args[i].type, type_size;
 		int kf_arg_type = meta->fn->arg_type[i];
 
+		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
+		nslots = kfunc_arg_slots(t);
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
 			/* Reject repeated use bpf_prog_aux */
 			if (meta->arg_prog) {

[ ... ]

Quoted 5 lines
kernel/bpf/verifier.c +1 -0
@@ -12986,6 +13076,17 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 			ref_tname = btf_name_by_offset(btf, ref_t->name_off);
 		}
 
+		/*

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 13089

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		 * The first slot is checked in below KF_ARG_ANYTHING.
+		 * The rest of it has to be a scalar.
+		 */
+		for (k = 1; k < nslots; k++) {
+			ret = check_kfunc_scalar_arg(env, caller, regs, slots_used + k,
+						     insn_idx, func_name);
+			if (ret)
+				return ret;
+		}
+

This isn't a bug, but would it be worth rewording this comment to name the KF_ARG_ANYTHING case explicitly and speak of the remaining slots in the plural? The phrase "checked in below KF_ARG_ANYTHING" is a bit hard to parse, and "the rest of it" refers to slots while the sentence subject is singular.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#12 lore raw

Re: [PATCH bpf-next v3 08/15] bpf: Recognize by-value struct and __int128 kfunc arguments

Quoted 85 lines · On 9/11/26 9:47 AM, [email protected] wrote:
Quoted 20 lines

bpf: Recognize by-value struct and __int128 kfunc arguments

A kfunc taking a struct or union by value is rejected today, and one taking an __int128 is accepted but mis-counted:

Unrecognized R2 type STRUCT

The kfunc arguments walk the same slot as a BPF-to-BPF call:
one argument register per eightbyte, and a 16-byte value takes two. The outgoing stack argument count at the call site follows the slots for the same reason. Similar to BPF-to-BPF aggregate handling, a kfunc aggregate argument is only supported when it is composed of scalars. Everything that maps a kfunc argument to a register has to follow the slots too.

An argument of a single eightbyte lands in the same place under every calling convention, so those are taken. The conventions the JIT has to reconcile do not agree on where a larger one goes, so refuse it for now with

Quoted 7 lines

Function f arg#1 type INT cannot be passed at argument slot 1 on this architecture

which the next patch turns into an answer from the JIT. The paths that handle a two-slot argument are therefore unreachable until then.

This looks like a bug fix for kfunc struct argument support. Should this include:

Fixes 0ab8eb34abe4 ("bpf: Support by-value struct arguments up to 16 bytes")
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index c520c37bb3c9..9e70995c763f 100644

[ ... ]

Quoted 41 lines
kernel/bpf/verifier.c +11 -5
@@ -12892,29 +12978,35 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
  	const struct btf *btf = meta->btf;
  	const struct btf_param *args;
  	struct btf_record *rec;
-	u32 i, nargs;
+	u32 i, k, nargs, proto_slots, slots_used, prev_slot = 0, nslots = 0;
  	int ret;
 
  	args = (const struct btf_param *)(meta->func_proto + 1);
  	nargs = btf_type_vlen(meta->func_proto);
+	proto_slots = kfunc_proto_slots(btf, meta->func_proto);
 
-	ret = check_outgoing_stack_args(env, caller, nargs, func_name, btf, args);
+	ret = check_outgoing_stack_args(env, caller, proto_slots, func_name, btf,
+					proto_slots == nargs ? args : NULL);
  	if (ret)
  		return ret;
 
  	/* Check that BTF function arguments match actual types that the
  	 * verifier sees.
  	 */
-	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
+	for (i = 0, slots_used = 0; i < nargs;
+	     i++, prev_slot = slots_used, slots_used += nslots) {
+		struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, slots_used);
  		const struct btf_type *t, *ref_t, *resolve_ret;
  		enum bpf_arg_type arg_type = ARG_DONTCARE;
-		argno_t argno = argno_from_arg(i + 1);
+		argno_t argno = argno_from_arg(slots_used + 1);
  		int regno = reg_from_argno(argno);
  		bool btf_id_fixed_off_ok = true;
  		u32 ref_id = args[i].type, type_size;
  		int kf_arg_type = meta->fn->arg_type[i];
 
+		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
+		nslots = kfunc_arg_slots(t);
+
  		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
  			/* Reject repeated use bpf_prog_aux */
  			if (meta->arg_prog) {

[ ... ]

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
@@ -12986,6 +13076,17 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
  			ref_tname = btf_name_by_offset(btf, ref_t->name_off);
  		}
 
Quoted 16 lines

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 13089

Quoted 11 lines
kernel/bpf/verifier.c +11 -0
+		/*
+		 * The first slot is checked in below KF_ARG_ANYTHING.
+		 * The rest of it has to be a scalar.
+		 */
+		for (k = 1; k < nslots; k++) {
+			ret = check_kfunc_scalar_arg(env, caller, regs, slots_used + k,
+						     insn_idx, func_name);
+			if (ret)
+				return ret;
+		}
+

This isn't a bug, but would it be worth rewording this comment to name the KF_ARG_ANYTHING case explicitly and speak of the remaining slots in the plural? The phrase "checked in below KF_ARG_ANYTHING" is a bit hard to parse, and "the rest of it" refers to slots while the sentence subject is singular.

I will update above comments to be clear about 'below KF_ARG_ANYTHING'.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#13 lore raw

PATCH v3 9/15 bpf: Prepare kfunc arguments for the JIT from an ABI description

In reply to Yonghong Song

The previous patch refuses a kfunc argument of more than one eightbyte. This patch allows up to 16 byte kfunc arguments.

But different architectures have different ways to map the BPF calling convention (no gap, no backfill) to the native one. Rather than have each arch open-code where it wants an argument, describe the convention with a register count and four booleans, and let each arch set what applies to it:

  struct bpf_jit_arg_abi {
        u8   nr_arg_regs;
        bool even_reg_align;
        bool even_stack_align;
        bool split_at_boundary;
        bool backfill_after_stack;
  };

The four booleans are meant to cover x86-64, arm64, RISC-V LP64 and PowerPC64 ELFv2, although only x86-64 and arm64 fill the struct in here. bpf_jit_place_args() and bpf_jit_plan_arg_moves() use that description to work out where each argument belongs and which slots the JIT then has to move, to be used in the JIT later on.

---
 include/linux/bpf.h          | 15 ++++++
 include/linux/bpf_verifier.h |  1 +
 include/linux/filter.h       | 32 ++++++++++++
 kernel/bpf/btf.c             |  3 ++
 kernel/bpf/core.c            | 94 ++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        | 98 +++++++++++++++++++++++++++++++-----
 6 files changed, 230 insertions(+), 13 deletions(-)
include/linux/bpf.h +15 -0
index e80963971f68..e939fb448197 100644
@@ -977,6 +977,13 @@ static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
  */
 #define MAX_BPF_FUNC_REG_ARGS 5
 
+/* A by-value argument takes two eightbytes at most, so the maximum number of
+ * argument slots of any function is 2 * MAX_BPF_FUNC_ARGS. A local array may
+ * need that size for processing, although eventually the maximum slots will
+ * be capped at MAX_BPF_FUNC_ARGS.
+ */
+#define MAX_BPF_FUNC_ARG_SLOTS (2 * MAX_BPF_FUNC_ARGS)
+
 /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
  * to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
  * instructions after verifying
@@ -1194,6 +1201,9 @@ struct bpf_prog_offload {
 	u32			jited_len;
 };
 
+/* The argument is aligned to 16 bytes. */
+#define BTF_FMODEL_ALIGN16_ARG		BIT(0)
+
 /* The argument is signed. */
 #define BTF_FMODEL_SIGNED_ARG		BIT(1)
 
@@ -1211,6 +1221,11 @@ struct btf_func_model {
 	u8 arg_flags[MAX_BPF_FUNC_ARGS];
 };
 
+static inline u32 btf_func_model_arg_slots(const struct btf_func_model *m, u32 arg)
+{
+	return (m->arg_size[arg] + sizeof(u64) - 1) / sizeof(u64);
+}
+
 /* Restore arguments before returning from trampoline to let original function
  * continue executing. This flag is used for fentry progs when there are no
  * fexit progs.
include/linux/bpf_verifier.h +1 -0
index 64dfa29f6b45..65d6d5444dc4 100644
@@ -1512,6 +1512,7 @@ enum btf_member_kind {
 
 bool btf_struct_is_composed_of(struct bpf_verifier_env *env, const struct btf *btf,
 			       const struct btf_type *t, u32 member_kinds);
+u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
include/linux/filter.h +32 -0
index 00ad8b63aa47..b17222db2efc 100644
@@ -1248,6 +1248,38 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
 bool bpf_jit_supports_private_stack(void);
 bool bpf_jit_supports_timed_may_goto(void);
 bool bpf_jit_supports_fsession(void);
+
+struct bpf_jit_arg_abi {
+	/* Argument registers of the kernel convention. */
+	u8 nr_arg_regs;
+	/* Round the register number up to an even one for 16-byte alignment. */
+	bool even_reg_align;
+	/* Round the stack slot up to an even one for 16-byte alignment. */
+	bool even_stack_align;
+	/* An argument may straddle the last register and the stack. */
+	bool split_at_boundary;
+	/* A later argument may reuse a register a stack-passed one skipped. */
+	bool backfill_after_stack;
+};
+
+const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void);
+u32 bpf_jit_place_args(const struct bpf_jit_arg_abi *abi,
+		       const struct btf_func_model *fm, u8 *pos_of_slot);
+
+/* The JIT's scratch register, in place of an argument slot. */
+#define BPF_JIT_ARG_TMP		0xff
+
+/* Every argument slot moves at most once, and the scratch goes out and back. */
+#define BPF_JIT_MAX_ARG_MOVES	(MAX_BPF_FUNC_ARG_SLOTS + 2)
+
+struct bpf_jit_arg_move {
+	u8 dst;
+	u8 src;
+};
+
+u32 bpf_jit_plan_arg_moves(const struct bpf_jit_arg_abi *abi,
+			   const struct btf_func_model *fm,
+			   struct bpf_jit_arg_move *moves);
 u64 bpf_arch_uaddress_limit(void);
 void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie);
 u64 arch_bpf_timed_may_goto(void);
kernel/bpf/btf.c +3 -0
index 088788e29b03..12f216515447 100644
@@ -7579,6 +7579,9 @@ static u8 __get_arg_fmodel_flags(const struct btf *btf,
 {
 	u8 flags = __get_type_fmodel_flags(t);
 
+	if (btf_func_arg_align(btf, t) > sizeof(u64))
+		flags |= BTF_FMODEL_ALIGN16_ARG;
+
 	if (btf_param_match_suffix(btf, arg, "__arena__nullable"))
 		flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG;
 	else if (btf_param_match_suffix(btf, arg, "__arena"))
kernel/bpf/core.c +94 -0

100 lines not on this page · raw message

kernel/bpf/verifier.c +85 -13

139 lines not on this page · raw message

Signature · 1 line
2.52.0
author reviewer
#14 lore raw

PATCH v3 10/15 bpf, x86: Move kfunc arguments into the x86-64 calling convention

In reply to Yonghong Song

Do the proper move from the BPF calling convention to the x86-64 calling convention to satisfy the native requirement.

In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.

---
 arch/x86/net/bpf_jit_comp.c | 73 +++++++++++++++++++++++++++++++++++--
 1 file changed, 70 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +70 -3
index bba351944202..0496607a7003 100644
@@ -1839,6 +1839,60 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 	return 0;
 }
 
+static const struct bpf_jit_arg_abi x86_arg_abi = {
+	.nr_arg_regs		= 6,
+	.backfill_after_stack	= true,
+	.even_stack_align	= true,
+};
+
+static const u8 x86_arg_reg[] = {
+	BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5, X86_REG_R9,
+};
+
+/*
+ * Move the arguments the x86-64 ABI places somewhere other than the argument
+ * slot the BPF calling convention gave them. @stack_base addresses the
+ * outgoing stack argument area from RBP. Return the number of emitted bytes.
+ */
+static int emit_kfunc_arg_moves(const struct btf_func_model *fm, s32 stack_base, u8 **pprog)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 nreg = x86_arg_abi.nr_arg_regs;
+	u8 *prog = *pprog, *start = prog;
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&x86_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+		bool dst_mem = dst != BPF_JIT_ARG_TMP && dst >= nreg;
+		bool src_mem = src != BPF_JIT_ARG_TMP && src >= nreg;
+
+		/* Take the value into a register. */
+		if (src == BPF_JIT_ARG_TMP) {
+			reg = AUX_REG;
+		} else if (src_mem) {
+			reg = dst_mem || dst == BPF_JIT_ARG_TMP ? BPF_REG_AX : x86_arg_reg[dst];
+			emit_ldx(&prog, BPF_DW, reg, BPF_REG_FP,
+				 stack_base + (src - nreg) * 8);
+		} else {
+			reg = x86_arg_reg[src];
+		}
+
+		/* And leave it where the argument belongs. */
+		if (dst == BPF_JIT_ARG_TMP)
+			emit_mov_reg(&prog, true, AUX_REG, reg);
+		else if (dst_mem)
+			emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
+				 stack_base + (dst - nreg) * 8);
+		else if (reg != x86_arg_reg[dst])
+			emit_mov_reg(&prog, true, x86_arg_reg[dst], reg);
+	}
+
+	*pprog = prog;
+	return prog - start;
+}
+
 /*
  * Rebase the __arena args of a kfunc call to arena kernel addresses,
  * rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
@@ -1850,11 +1904,17 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
 {
 	u8 *prog = *pprog;
 	u8 *start = prog;
-	int i;
+	int i, slot;
 
-	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
+	for (i = 0, slot = 0; i < fm->nr_args; i++) {
+		u32 arg_regs = (fm->arg_size[i] + 7) / 8;
 		u8 flags = fm->arg_flags[i];
-		u32 reg = BPF_REG_1 + i;
+		u32 reg;
+
+		if (slot + arg_regs > MAX_BPF_FUNC_REG_ARGS)
+			break;
+		reg = BPF_REG_1 + slot;
+		slot += arg_regs;
 
 		if (!(flags & BTF_FMODEL_ARENA_ARG))
 			continue;
@@ -2837,6 +2897,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 				if (err < 0)
 					return err;
 				ip += err;
+				ip += emit_kfunc_arg_moves(fm, outgoing_arg_base -
+							   outgoing_rsp, &prog);
 			}
 			if (priv_frame_ptr) {
 				push_r9(&prog);
@@ -4351,6 +4413,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
 	return true;
 }
 
+const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
+{
+	return &x86_arg_abi;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
Signature · 1 line
2.52.0
#15 lore raw

Re: [PATCH bpf-next v3 10/15] bpf, x86: Move kfunc arguments into the x86-64 calling convention

This isn't a bug, but for the memory-source case, could reg be AUX_REG when dst == BPF_JIT_ARG_TMP so the load lands directly in the scratch instead of going through BPF_REG_AX?
Quoted 41 lines
arch/x86/net/bpf_jit_comp.c +33 -0
index bba351944202..0496607a7003 100644
@@ -1839,6 +1839,60 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 	return 0;
 }
 
+static const struct bpf_jit_arg_abi x86_arg_abi = {
+	.nr_arg_regs		= 6,
+	.backfill_after_stack	= true,
+	.even_stack_align	= true,
+};
+
+static const u8 x86_arg_reg[] = {
+	BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5, X86_REG_R9,
+};
+
+/*
+ * Move the arguments the x86-64 ABI places somewhere other than the argument
+ * slot the BPF calling convention gave them. @stack_base addresses the
+ * outgoing stack argument area from RBP. Return the number of emitted bytes.
+ */
+static int emit_kfunc_arg_moves(const struct btf_func_model *fm, s32 stack_base, u8 **pprog)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 nreg = x86_arg_abi.nr_arg_regs;
+	u8 *prog = *pprog, *start = prog;
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&x86_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+		bool dst_mem = dst != BPF_JIT_ARG_TMP && dst >= nreg;
+		bool src_mem = src != BPF_JIT_ARG_TMP && src >= nreg;
+
+		/* Take the value into a register. */
+		if (src == BPF_JIT_ARG_TMP) {
+			reg = AUX_REG;
+		} else if (src_mem) {

arch/x86/net/bpf_jit_comp.c static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip, · line 1884

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+			reg = dst_mem || dst == BPF_JIT_ARG_TMP ? BPF_REG_AX : x86_arg_reg[dst];
+			emit_ldx(&prog, BPF_DW, reg, BPF_REG_FP,
+				 stack_base + (src - nreg) * 8);
+		} else {
+			reg = x86_arg_reg[src];
+		}
+
+		/* And leave it where the argument belongs. */
+		if (dst == BPF_JIT_ARG_TMP)
+			emit_mov_reg(&prog, true, AUX_REG, reg);

This isn't a bug, but for the memory-source case, could reg be AUX_REG when dst == BPF_JIT_ARG_TMP so the load lands directly in the scratch instead of going through BPF_REG_AX?

The case is reachable: a prototype such as (u64 a..e, struct {u64; u64;} s, u64 f) backfills f into R9 and hands the planner a down-move whose source is a stack slot. With the current code, emit_kfunc_arg_moves() generates:

    mov r10, [rbp+off]
    mov r11, r10

when this would do:

    mov r11, [rbp+off]

Picking AUX_REG when dst == BPF_JIT_ARG_TMP (and BPF_REG_AX only for the memory-to-memory case) would drop the extra instruction from every such call site.

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+		else if (dst_mem)
+			emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
+				 stack_base + (dst - nreg) * 8);
+		else if (reg != x86_arg_reg[dst])
+			emit_mov_reg(&prog, true, x86_arg_reg[dst], reg);
+	}
+
+	*pprog = prog;
+	return prog - start;
+}

[ ... ]

Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +2 -0
@@ -2837,6 +2897,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 				if (err < 0)
 					return err;
 				ip += err;
+				ip += emit_kfunc_arg_moves(fm, outgoing_arg_base -
+							   outgoing_rsp, &prog);
 			}

[ ... ]

Quoted 4 lines

bpf, x86: Move kfunc arguments into the x86-64 calling convention

Do the proper move from the BPF calling convention to the x86-64 calling convention to satisfy the native requirement.

This isn't a bug, but could the changelog name the case that actually needs a move (a by-value argument spilling to the stack followed by a backfilled register argument) and note that at most one argument moves down, so the scratch register carry is safe?

The current wording says what the patch does without saying which arguments actually end up somewhere other than their BPF slot, or why a reviewer has to reconstruct that from bpf_jit_place_args() and bpf_jit_plan_arg_moves().

Quoted 2 lines

In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.

(This second paragraph reads well and is concrete.)

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#16 lore raw

Re: [PATCH bpf-next v3 10/15] bpf, x86: Move kfunc arguments into the x86-64 calling convention

Quoted 63 lines · On 9/11/26 9:47 AM, [email protected] wrote:
Quoted 41 lines
arch/x86/net/bpf_jit_comp.c +33 -0
index bba351944202..0496607a7003 100644
@@ -1839,6 +1839,60 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  	return 0;
  }
 
+static const struct bpf_jit_arg_abi x86_arg_abi = {
+	.nr_arg_regs		= 6,
+	.backfill_after_stack	= true,
+	.even_stack_align	= true,
+};
+
+static const u8 x86_arg_reg[] = {
+	BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5, X86_REG_R9,
+};
+
+/*
+ * Move the arguments the x86-64 ABI places somewhere other than the argument
+ * slot the BPF calling convention gave them. @stack_base addresses the
+ * outgoing stack argument area from RBP. Return the number of emitted bytes.
+ */
+static int emit_kfunc_arg_moves(const struct btf_func_model *fm, s32 stack_base, u8 **pprog)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 nreg = x86_arg_abi.nr_arg_regs;
+	u8 *prog = *pprog, *start = prog;
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&x86_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+		bool dst_mem = dst != BPF_JIT_ARG_TMP && dst >= nreg;
+		bool src_mem = src != BPF_JIT_ARG_TMP && src >= nreg;
+
+		/* Take the value into a register. */
+		if (src == BPF_JIT_ARG_TMP) {
+			reg = AUX_REG;
+		} else if (src_mem) {

arch/x86/net/bpf_jit_comp.c static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip, · line 1884

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+			reg = dst_mem || dst == BPF_JIT_ARG_TMP ? BPF_REG_AX : x86_arg_reg[dst];
+			emit_ldx(&prog, BPF_DW, reg, BPF_REG_FP,
+				 stack_base + (src - nreg) * 8);
+		} else {
+			reg = x86_arg_reg[src];
+		}
+
+		/* And leave it where the argument belongs. */
+		if (dst == BPF_JIT_ARG_TMP)
+			emit_mov_reg(&prog, true, AUX_REG, reg);

This isn't a bug, but for the memory-source case, could reg be AUX_REG when dst == BPF_JIT_ARG_TMP so the load lands directly in the scratch instead of going through BPF_REG_AX?

The case is reachable: a prototype such as (u64 a..e, struct {u64; u64;} s, u64 f) backfills f into R9 and hands the planner a down-move whose source is a stack slot. With the current code, emit_kfunc_arg_moves() generates:

     mov r10, [rbp+off]
     mov r11, r10
Quoted 7 lines

when this would do:

     mov r11, [rbp+off]

Picking AUX_REG when dst == BPF_JIT_ARG_TMP (and BPF_REG_AX only for the memory-to-memory case) would drop the extra instruction from every such call site.

Sounds good. A little bit optimization. Will do.

Quoted 30 lines
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+		else if (dst_mem)
+			emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
+				 stack_base + (dst - nreg) * 8);
+		else if (reg != x86_arg_reg[dst])
+			emit_mov_reg(&prog, true, x86_arg_reg[dst], reg);
+	}
+
+	*pprog = prog;
+	return prog - start;
+}

[ ... ]

Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +2 -0
@@ -2837,6 +2897,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  				if (err < 0)
  					return err;
  				ip += err;
+				ip += emit_kfunc_arg_moves(fm, outgoing_arg_base -
+							   outgoing_rsp, &prog);
  			}

[ ... ]

Quoted 4 lines

bpf, x86: Move kfunc arguments into the x86-64 calling convention

Do the proper move from the BPF calling convention to the x86-64 calling convention to satisfy the native requirement.

This isn't a bug, but could the changelog name the case that actually needs a move (a by-value argument spilling to the stack followed by a backfilled register argument) and note that at most one argument moves down, so the scratch register carry is safe?

Quoted 4 lines

The current wording says what the patch does without saying which arguments actually end up somewhere other than their BPF slot, or why a reviewer has to reconstruct that from bpf_jit_place_args() and bpf_jit_plan_arg_moves().

Indeed, the above description is not enough. I will add more things like you mentioned above.

Quoted 11 lines
Quoted 3 lines

In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.

(This second paragraph reads well and is concrete.)

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#17 lore raw

PATCH v3 11/15 bpf, arm64: Move kfunc arguments into the arm64 calling convention

In reply to Yonghong Song

Do the proper move from the BPF calling convention to the arm64 calling convention to satisfy the native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.

In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers. The walk takes the func model from the caller now, as the moves need it too, and runs first so that they carry the rebased value.

btf_distill_func_proto() only bounds the argument count although a by-value argument could take two slots. Similar to x86-64, let us support up to MAX_BPF_FUNC_ARGS argument slots.

---
 arch/arm64/net/bpf_jit_comp.c | 92 ++++++++++++++++++++++++++++++-----
 1 file changed, 81 insertions(+), 11 deletions(-)
arch/arm64/net/bpf_jit_comp.c +81 -11
index 3aa3ea0bc30b..bbde7c0836ae 100644
@@ -1220,6 +1220,12 @@ static int add_exception_handler(const struct bpf_insn *insn,
 	return 0;
 }
 
+static const struct bpf_jit_arg_abi arm64_arg_abi = {
+	.nr_arg_regs		= 8,
+	.even_reg_align		= true,
+	.even_stack_align	= true,
+};
+
 static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
 
 #define NR_STACK_ARG_REGS	ARRAY_SIZE(stack_arg_reg)
@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct
  * kern_vm_start. A nullable arg preserves NULL by skipping the add, tested
  * on the truncated value as arena NULL is offset 0.
  */
-static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn)
+static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct btf_func_model *fm)
 {
 	const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-	const struct btf_func_model *fm;
-	int i;
-
-	fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
-	if (!fm)
-		return -EINVAL;
+	int i, slot;
 
-	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
-		const u8 reg = bpf2a64[BPF_REG_1 + i];
+	for (i = 0, slot = 0; i < fm->nr_args; i++) {
+		u32 arg_regs = (fm->arg_size[i] + 7) / 8;
 		u8 flags = fm->arg_flags[i];
+		u8 reg;
+
+		if (slot + arg_regs > MAX_BPF_FUNC_REG_ARGS)
+			break;
+		reg = bpf2a64[BPF_REG_1 + slot];
+		slot += arg_regs;
 
 		if (!(flags & BTF_FMODEL_ARENA_ARG))
 			continue;
@@ -1293,6 +1300,52 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins
 	return 0;
 }
 
+static bool a64_arg_on_stack(u8 slot)
+{
+	return slot >= arm64_arg_abi.nr_arg_regs;
+}
+
+static s32 a64_arg_stack_off(u8 slot)
+{
+	return (slot - arm64_arg_abi.nr_arg_regs) * sizeof(u64);
+}
+
+/*
+ * Move the arguments AAPCS64 places somewhere other than the argument slot the
+ * BPF calling convention gave them. Slot N is X(N) up to the eighth, and the
+ * outgoing stack argument area from SP beyond it, both for the slot an
+ * argument comes from and for the one it goes to.
+ *
+ * AAPCS64 only ever moves an argument to a higher slot, so no move here ever
+ * takes BPF_JIT_ARG_TMP: bpf_jit_plan_arg_moves() hands out the scratch only
+ * for a convention that moves one down, which needs a register to carry the
+ * value past its own destination.
+ */
+static void emit_kfunc_arg_moves(struct jit_ctx *ctx, const struct btf_func_model *fm)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 tmp = bpf2a64[TMP_REG_1];
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&arm64_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+
+		if (a64_arg_on_stack(src)) {
+			reg = tmp;
+			emit(A64_LDR64I(reg, A64_SP, a64_arg_stack_off(src)), ctx);
+		} else {
+			reg = src;
+		}
+
+		if (a64_arg_on_stack(dst))
+			emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);
+		else if (reg != dst)
+			emit(A64_MOV(1, dst, reg), ctx);
+	}
+}
+
 /* JITs an eBPF instruction.
  * Returns:
  * 0  - successfully JITed an 8-byte eBPF instruction.
@@ -1716,9 +1769,15 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (ret < 0)
 			return ret;
 		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
-			ret = emit_kfunc_arena_args(ctx, insn);
+			const struct btf_func_model *fm;
+
+			fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
+			if (!fm)
+				return -EINVAL;
+			ret = emit_kfunc_arena_args(ctx, fm);
 			if (ret < 0)
 				return ret;
+			emit_kfunc_arg_moves(ctx, fm);
 		}
 		emit_call(func_addr, ctx);
 		/*
@@ -2393,6 +2452,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)
 	return true;
 }
 
+const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
+{
+	return &arm64_arg_abi;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
@@ -2534,7 +2598,13 @@ struct arg_aux {
 static int calc_arg_aux(const struct btf_func_model *m,
 			 struct arg_aux *a)
 {
-	int stack_slots, nregs, slots, i;
+	int stack_slots, nregs, slots, i, total;
+
+	/* arm64 supports up to MAX_BPF_FUNC_ARGS argument slots */
+	for (i = 0, total = 0; i < m->nr_args; i++)
+		total += (m->arg_size[i] + 7) / 8;
+	if (total > MAX_BPF_FUNC_ARGS)
+		return -ENOTSUPP;
 
 	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
 	for (i = 0, nregs = 0; i < m->nr_args; i++) {
Signature · 1 line
2.52.0
#18 lore raw

Re: [PATCH bpf-next v3 11/15] bpf, arm64: Move kfunc arguments into the arm64 calling convention

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The ARM64 BPF trampoline incorrectly saves and restores 16-byte aligned arguments, violating AAPCS64 alignment rules. --

commit cd40aa31f4bf1158416008f7fcb9c0b7717dda3f
Author: Yonghong Song <[email protected]>

bpf, arm64: Move kfunc arguments into the arm64 calling convention

Do the proper move from the BPF calling convention to the arm64 calling convention to satisfy the native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 3aa3ea0bc30bb..bbde7c0836ae5 100644

[ ... ]

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0 -1
@@ -2534,7 +2598,13 @@ struct arg_aux {
 static int calc_arg_aux(const struct btf_func_model *m,
 			 struct arg_aux *a)
 {
-	int stack_slots, nregs, slots, i;

arch/arm64/net/bpf_jit_comp.c struct arg_aux { · line 2610

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +7 -0
+	int stack_slots, nregs, slots, i, total;
+
+	/* arm64 supports up to MAX_BPF_FUNC_ARGS argument slots */
+	for (i = 0, total = 0; i < m->nr_args; i++)
+		total += (m->arg_size[i] + 7) / 8;
+	if (total > MAX_BPF_FUNC_ARGS)
+		return -ENOTSUPP;
 
 	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
 	for (i = 0, nregs = 0; i < m->nr_args; i++) {

[Severity: High]
This isn't a bug introduced by this patch, but since the series introduces support for by-value 16-byte arguments, this previously dormant ABI violation in the BPF trampoline appears to now be actively reachable.

Does calc_arg_aux() fail to account for skipped registers due to alignment requirements?

If a user attaches a BPF program to a kernel function (or struct_ops callback) that takes a 16-byte aligned argument (e.g., __int128 or by-value struct) preceded by an odd number of register-sized arguments, the native AAPCS64 caller places the 16-byte argument in an even/odd register pair (like x2 and x3, skipping x1).

However, the nregs calculation here simply increments by slots without checking for skipped odd registers:

arch/arm64/net/bpf_jit_comp.c:calc_arg_aux() {
    ...
    for (i = 0, nregs = 0; i < m->nr_args; i++) {
        slots = (m->arg_size[i] + 7) / 8;
        if (nregs + slots <= 8) /* passed through register ? */
            nregs += slots;
    ...
}

This strictly sequential register handling seems to propagate to both save_args() and restore_args() in the trampoline, where arguments are read and written sequentially without skipping odd registers:

arch/arm64/net/bpf_jit_comp.c:save_args() {
    ...
    slots = (m->arg_size[i] + 7) / 8;
    while (slots-- > 0) {
        ...
        emit(A64_STR64I(reg, A64_SP, bargs_off), ctx);
        reg++;
        bargs_off += 8;
    }
    ...
}
arch/arm64/net/bpf_jit_comp.c:restore_args() {
    ...
    for (reg = 0; reg < nregs; reg++) {
        emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);
        bargs_off += 8;
    }
    ...
}

Could this cause data corruption for arguments passed to BPF programs attached via fentry, fexit, or struct_ops when 16-byte arguments are involved?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author reviewer
#19 lore raw

Re: [PATCH bpf-next v3 11/15] bpf, arm64: Move kfunc arguments into the arm64 calling convention

Quoted 89 lines · On 9/11/26 9:19 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The ARM64 BPF trampoline incorrectly saves and restores 16-byte aligned arguments, violating AAPCS64 alignment rules. --

commit cd40aa31f4bf1158416008f7fcb9c0b7717dda3f
Author: Yonghong Song <[email protected]>

bpf, arm64: Move kfunc arguments into the arm64 calling convention

Do the proper move from the BPF calling convention to the arm64 calling convention to satisfy the native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 3aa3ea0bc30bb..bbde7c0836ae5 100644

[ ... ]

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0 -1
@@ -2534,7 +2598,13 @@ struct arg_aux {
  static int calc_arg_aux(const struct btf_func_model *m,
  			 struct arg_aux *a)
  {
-	int stack_slots, nregs, slots, i;

arch/arm64/net/bpf_jit_comp.c struct arg_aux { · line 2610

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +7 -0
+	int stack_slots, nregs, slots, i, total;
+
+	/* arm64 supports up to MAX_BPF_FUNC_ARGS argument slots */
+	for (i = 0, total = 0; i < m->nr_args; i++)
+		total += (m->arg_size[i] + 7) / 8;
+	if (total > MAX_BPF_FUNC_ARGS)
+		return -ENOTSUPP;
 
  	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
  	for (i = 0, nregs = 0; i < m->nr_args; i++) {

[Severity: High]
This isn't a bug introduced by this patch, but since the series introduces support for by-value 16-byte arguments, this previously dormant ABI violation in the BPF trampoline appears to now be actively reachable.

Does calc_arg_aux() fail to account for skipped registers due to alignment requirements?

If a user attaches a BPF program to a kernel function (or struct_ops callback) that takes a 16-byte aligned argument (e.g., __int128 or by-value struct) preceded by an odd number of register-sized arguments, the native AAPCS64 caller places the 16-byte argument in an even/odd register pair (like x2 and x3, skipping x1).

However, the nregs calculation here simply increments by slots without checking for skipped odd registers:

arch/arm64/net/bpf_jit_comp.c:calc_arg_aux() {
     ...
     for (i = 0, nregs = 0; i < m->nr_args; i++) {
         slots = (m->arg_size[i] + 7) / 8;
         if (nregs + slots <= 8) /* passed through register ? */
             nregs += slots;
     ...
}

This strictly sequential register handling seems to propagate to both save_args() and restore_args() in the trampoline, where arguments are read and written sequentially without skipping odd registers:

arch/arm64/net/bpf_jit_comp.c:save_args() {
     ...
     slots = (m->arg_size[i] + 7) / 8;
     while (slots-- > 0) {
         ...
         emit(A64_STR64I(reg, A64_SP, bargs_off), ctx);
         reg++;
         bargs_off += 8;
     }
     ...
}
arch/arm64/net/bpf_jit_comp.c:restore_args() {
     ...
     for (reg = 0; reg < nregs; reg++) {
         emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);
         bargs_off += 8;
     }
     ...
}
Quoted 4 lines

Could this cause data corruption for arguments passed to BPF programs attached via fentry, fexit, or struct_ops when 16-byte arguments are involved?

Yes, this is a bug. I will need to consider 16-byte alignment. Will fix.

#20 lore raw

Re: [PATCH bpf-next v3 11/15] bpf, arm64: Move kfunc arguments into the arm64 calling convention

In reply to Yonghong Song
This isn't a bug, but would it be worth expressing emit_stack_arg_load()/emit_stack_arg_store() in terms of a64_arg_on_stack()/a64_arg_stack_off() (or noting in a comment that the two forms describe…
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 3aa3ea0bc30b..bbde7c0836ae 100644

[ ... ]

Quoted 33 lines
arch/arm64/net/bpf_jit_comp.c +10 -9
@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct
  * kern_vm_start. A nullable arg preserves NULL by skipping the add, tested
  * on the truncated value as arena NULL is offset 0.
  */
-static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn)
+static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct btf_func_model *fm)
 {
 	const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-	const struct btf_func_model *fm;
-	int i;
-
-	fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
-	if (!fm)
-		return -EINVAL;
+	int i, slot;
 
-	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
-		const u8 reg = bpf2a64[BPF_REG_1 + i];
+	for (i = 0, slot = 0; i < fm->nr_args; i++) {
+		u32 arg_regs = (fm->arg_size[i] + 7) / 8;
 		u8 flags = fm->arg_flags[i];
+		u8 reg;
+
+		if (slot + arg_regs > MAX_BPF_FUNC_REG_ARGS)
+			break;
+		reg = bpf2a64[BPF_REG_1 + slot];
+		slot += arg_regs;
 
 		if (!(flags & BTF_FMODEL_ARENA_ARG))
 			continue;
@@ -1293,6 +1300,52 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins
 	return 0;
 }

arch/arm64/net/bpf_jit_comp.c static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins · line 1311

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +9 -0
 
+static bool a64_arg_on_stack(u8 slot)
+{
+	return slot >= arm64_arg_abi.nr_arg_regs;
+}
+
+static s32 a64_arg_stack_off(u8 slot)
+{
+	return (slot - arm64_arg_abi.nr_arg_regs) * sizeof(u64);
+}

This isn't a bug, but would it be worth expressing emit_stack_arg_load()/emit_stack_arg_store() in terms of a64_arg_on_stack()/a64_arg_stack_off() (or noting in a comment that the two forms describe the same slot-to-location map), so the layout lives in one place? The file now carries two independent encodings of the same physical outgoing-argument layout: the pre-existing helpers express it as 'idx = bpf slot - 5', register when idx < NR_STACK_ARG_REGS (x5/x6/x7) and memory at SP + (idx - NR_STACK_ARG_REGS) * 8 otherwise; the new helpers express it as 'slot', register when slot < arm64_arg_abi.nr_arg_regs and memory at SP + (slot - nr_arg_regs) * 8.

Quoted 36 lines
arch/arm64/net/bpf_jit_comp.c +36 -0
+
+/*
+ * Move the arguments AAPCS64 places somewhere other than the argument slot the
+ * BPF calling convention gave them. Slot N is X(N) up to the eighth, and the
+ * outgoing stack argument area from SP beyond it, both for the slot an
+ * argument comes from and for the one it goes to.
+ *
+ * AAPCS64 only ever moves an argument to a higher slot, so no move here ever
+ * takes BPF_JIT_ARG_TMP: bpf_jit_plan_arg_moves() hands out the scratch only
+ * for a convention that moves one down, which needs a register to carry the
+ * value past its own destination.
+ */
+static void emit_kfunc_arg_moves(struct jit_ctx *ctx, const struct btf_func_model *fm)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 tmp = bpf2a64[TMP_REG_1];
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&arm64_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+
+		if (a64_arg_on_stack(src)) {
+			reg = tmp;
+			emit(A64_LDR64I(reg, A64_SP, a64_arg_stack_off(src)), ctx);
+		} else {
+			reg = src;
+		}
+
+		if (a64_arg_on_stack(dst))
+			emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);
+		else if (reg != dst)
+			emit(A64_MOV(1, dst, reg), ctx);
+	}
+}

[ ... ]

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0 -1
@@ -2534,7 +2598,13 @@ struct arg_aux {
 static int calc_arg_aux(const struct btf_func_model *m,
 			 struct arg_aux *a)
 {
-	int stack_slots, nregs, slots, i;

arch/arm64/net/bpf_jit_comp.c struct arg_aux { · line 2610

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +7 -0
+	int stack_slots, nregs, slots, i, total;
+
+	/* arm64 supports up to MAX_BPF_FUNC_ARGS argument slots */
+	for (i = 0, total = 0; i < m->nr_args; i++)
+		total += (m->arg_size[i] + 7) / 8;
+	if (total > MAX_BPF_FUNC_ARGS)
+		return -ENOTSUPP;
 
 	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
 	for (i = 0; i < m->nr_args; i++) {

This isn't a bug, but could the changelog say explicitly that arm64's trampoline path now rejects models whose arguments exceed MAX_BPF_FUNC_ARGS eightbytes (previously it accepted up to twice that), so the narrowing is visible to anyone bisecting an fentry attach failure? The third changelog paragraph describes the calc_arg_aux() hunk as adding support ('let us support up to MAX_BPF_FUNC_ARGS argument slots'), while on the trampoline path the hunk is a new restriction:
calc_arg_aux() previously accepted any model btf_distill_func_proto() admitted (up to 12 arguments of up to 16 bytes, i.e. up to 24 eightbytes), and now returns -ENOTSUPP for anything above 12 eightbytes, so arch_prepare_bpf_trampoline()/arch_bpf_trampoline_size() fail for those targets.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#21 lore raw

Re: [PATCH bpf-next v3 11/15] bpf, arm64: Move kfunc arguments into the arm64 calling convention

Quoted 51 lines · On 9/11/26 9:47 AM, [email protected] wrote:
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 3aa3ea0bc30b..bbde7c0836ae 100644

[ ... ]

Quoted 33 lines
arch/arm64/net/bpf_jit_comp.c +10 -9
@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct
   * kern_vm_start. A nullable arg preserves NULL by skipping the add, tested
   * on the truncated value as arena NULL is offset 0.
   */
-static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn)
+static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct btf_func_model *fm)
  {
  	const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
-	const struct btf_func_model *fm;
-	int i;
-
-	fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
-	if (!fm)
-		return -EINVAL;
+	int i, slot;
 
-	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
-		const u8 reg = bpf2a64[BPF_REG_1 + i];
+	for (i = 0, slot = 0; i < fm->nr_args; i++) {
+		u32 arg_regs = (fm->arg_size[i] + 7) / 8;
  		u8 flags = fm->arg_flags[i];
+		u8 reg;
+
+		if (slot + arg_regs > MAX_BPF_FUNC_REG_ARGS)
+			break;
+		reg = bpf2a64[BPF_REG_1 + slot];
+		slot += arg_regs;
 
  		if (!(flags & BTF_FMODEL_ARENA_ARG))
  			continue;
@@ -1293,6 +1300,52 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins
  	return 0;
  }

arch/arm64/net/bpf_jit_comp.c static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins · line 1311

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +9 -0
 
+static bool a64_arg_on_stack(u8 slot)
+{
+	return slot >= arm64_arg_abi.nr_arg_regs;
+}
+
+static s32 a64_arg_stack_off(u8 slot)
+{
+	return (slot - arm64_arg_abi.nr_arg_regs) * sizeof(u64);
+}

This isn't a bug, but would it be worth expressing emit_stack_arg_load()/emit_stack_arg_store() in terms of

Quoted 8 lines

a64_arg_on_stack()/a64_arg_stack_off() (or noting in a comment that the two forms describe the same slot-to-location map), so the layout lives in one place? The file now carries two independent encodings of the same physical outgoing-argument layout: the pre-existing helpers express it as 'idx = bpf slot - 5', register when idx < NR_STACK_ARG_REGS (x5/x6/x7) and memory at SP + (idx - NR_STACK_ARG_REGS) * 8 otherwise; the new helpers express it as 'slot', register when slot < arm64_arg_abi.nr_arg_regs and memory at SP + (slot - nr_arg_regs) * 8.

I think the code is self explained.

Quoted 52 lines
Quoted 37 lines
arch/arm64/net/bpf_jit_comp.c +36 -0
+
+/*
+ * Move the arguments AAPCS64 places somewhere other than the argument slot the
+ * BPF calling convention gave them. Slot N is X(N) up to the eighth, and the
+ * outgoing stack argument area from SP beyond it, both for the slot an
+ * argument comes from and for the one it goes to.
+ *
+ * AAPCS64 only ever moves an argument to a higher slot, so no move here ever
+ * takes BPF_JIT_ARG_TMP: bpf_jit_plan_arg_moves() hands out the scratch only
+ * for a convention that moves one down, which needs a register to carry the
+ * value past its own destination.
+ */
+static void emit_kfunc_arg_moves(struct jit_ctx *ctx, const struct btf_func_model *fm)
+{
+	struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
+	const u8 tmp = bpf2a64[TMP_REG_1];
+	u32 i, n;
+
+	n = bpf_jit_plan_arg_moves(&arm64_arg_abi, fm, moves);
+
+	for (i = 0; i < n; i++) {
+		u8 dst = moves[i].dst, src = moves[i].src, reg;
+
+		if (a64_arg_on_stack(src)) {
+			reg = tmp;
+			emit(A64_LDR64I(reg, A64_SP, a64_arg_stack_off(src)), ctx);
+		} else {
+			reg = src;
+		}
+
+		if (a64_arg_on_stack(dst))
+			emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);
+		else if (reg != dst)
+			emit(A64_MOV(1, dst, reg), ctx);
+	}
+}

[ ... ]

Quoted 13 lines
arch/arm64/net/bpf_jit_comp.c +7 -1
@@ -2534,7 +2598,13 @@ struct arg_aux {
  static int calc_arg_aux(const struct btf_func_model *m,
  			 struct arg_aux *a)
  {
-	int stack_slots, nregs, slots, i;
+	int stack_slots, nregs, slots, i, total;
+
+	/* arm64 supports up to MAX_BPF_FUNC_ARGS argument slots */
+	for (i = 0, total = 0; i < m->nr_args; i++)
+		total += (m->arg_size[i] + 7) / 8;
+	if (total > MAX_BPF_FUNC_ARGS)
+		return -ENOTSUPP;
 
Quoted 14 lines

arch/arm64/net/bpf_jit_comp.c struct arg_aux { · line 2610

Quoted 2 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
  	/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
  	for (i = 0; i < m->nr_args; i++) {

This isn't a bug, but could the changelog say explicitly that arm64's trampoline path now rejects models whose arguments exceed MAX_BPF_FUNC_ARGS eightbytes (previously it accepted up to twice that), so the narrowing is visible to anyone bisecting an fentry attach failure? The third changelog paragraph describes the calc_arg_aux() hunk as adding support ('let us support up to MAX_BPF_FUNC_ARGS argument slots'), while on the trampoline path the hunk is a new restriction:
calc_arg_aux() previously accepted any model btf_distill_func_proto() admitted (up to 12 arguments of up to 16 bytes, i.e. up to 24 eightbytes), and now returns -ENOTSUPP for anything above 12 eightbytes, so arch_prepare_bpf_trampoline()/arch_bpf_trampoline_size() fail for those targets.

You are right. We shouldn't just reject. I will do proper implementation to fix the issue.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#22 lore raw

PATCH v3 12/15 selftests/bpf: Add C tests for by-value arguments up to 16 bytes

In reply to Yonghong Song

Extend the by-value argument test with the aggregate cases, written in C so that they depend on the compiler lowering the argument into a pair of argument registers rather than on a hand-written register layout.

The programs cover a struct and a union that fill two registers, a smaller struct that fills one, and two struct arguments in a row, alongside the __int128 already there. Each has an int argument around it so that a wrong slot count shows up as a wrong value in the parameters beside it; two pairs leave room for only one, which follows them. A global function taking a struct with a pointer member is rejected: the callee would receive the pointer as an opaque scalar.

A struct the argument registers cannot hold reaches the callee partly on the stack, which the interpreter does not implement, so that case is loaded only when the JIT is on.

---
 .../bpf/progs/verifier_aggregate_arg.c        | 165 ++++++++++++++++++
 1 file changed, 165 insertions(+)
tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c +165 -0
index fc7c1b18bc40..b0ecccede47e 100644
@@ -7,6 +7,131 @@
 #define MIX_A	0xdeadbeefcafef00dULL
 #define MIX_B	0x0123456789abcdefULL
 
+struct pair {
+	__u64 lo;
+	__u64 hi;
+};
+
+struct small {
+	__u32 a;
+	__u32 b;
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+static __noinline __u64 take_pair(int a, struct pair p, int c)
+{
+	return (__u64)a + p.lo + p.hi + c;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_arg_static_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p = { .lo = a, .hi = b };
+
+	if (take_pair(1, p, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+#if defined(__clang__)
+
+__noinline __u64 take_pair_global(int a, struct pair p, int c)
+{
+	return (__u64)a + p.lo + p.hi + c;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_arg_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p = { .lo = a, .hi = b };
+
+	if (take_pair_global(1, p, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+__noinline __u64 take_two_pairs_global(struct pair p, struct pair q, int d)
+{
+	return p.lo + p.hi + q.lo + q.hi + d;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_arg_two_structs_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p = { .lo = a, .hi = b };
+	struct pair q = { .lo = a + 1, .hi = b + 2 };
+
+	if (take_two_pairs_global(p, q, 3) != 2 * a + 2 * b + 6)
+		return 1;
+
+	return 0;
+}
+
+__noinline __u64 take_small_global(int a, struct small s, int c)
+{
+	return (__u64)a + s.a + s.b + c;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_arg_small_struct_c_test(struct __sk_buff *skb)
+{
+	__u32 a = skb->len ^ (__u32)MIX_A;
+	__u32 b = skb->len ^ (__u32)MIX_B;
+	struct small s = { .a = a, .b = b };
+
+	if (take_small_global(1, s, 2) != (__u64)a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+__noinline __u64 take_upair_global(int a, union upair u, int c)
+{
+	return (__u64)a + u.parts.lo + u.parts.hi + c;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_arg_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union upair u;
+
+	u.halves[0] = a;
+	u.halves[1] = b;
+	if (take_upair_global(1, u, 2) != a + b + 3)
+		return 1;
+
+	return 0;
+}
+
+#endif
+
 #ifdef __SIZEOF_INT128__
 
 typedef unsigned __int128 u128;
@@ -32,4 +157,44 @@ int aggregate_arg_int128_c_test(struct __sk_buff *skb)
 
 #endif /* __SIZEOF_INT128__ */
 
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+static __noinline __u64 take_spilled_pair(int a, int b, int c, int d, struct pair p)
+{
+	return (__u64)a + b + c + d + p.lo + p.hi;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_arg_spilled_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p = { .lo = a, .hi = b };
+	int n = skb->len;
+
+	if (take_spilled_pair(n, n + 1, n + 2, n + 3, p) != a + b + 4 * n + 6)
+		return 1;
+
+	return 0;
+}
+
+#endif
+
+__noinline __u64 take_with_ptr_global(struct with_ptr s)
+{
+	return s.x;
+}
+
+SEC("tc")
+__failure __msg("type STRUCT in take_with_ptr_global() is not composed of scalars")
+int aggregate_arg_ptr_member_fail(struct __sk_buff *skb)
+{
+	struct with_ptr s = { .p = skb, .x = skb->len };
+
+	return take_with_ptr_global(s);
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.52.0
author reviewer
#23 lore raw

PATCH v3 13/15 selftests/bpf: Add inline-asm tests for by-value arguments

In reply to Yonghong Song

The tests cover a struct passed in a register pair, a pointer in one half of it refused at the call site, a struct too large to pass by value, and four placements a global function cannot have: six scalars, a struct split between the last argument register and the stack, one wholly past the registers, and an __int128 whose two slots push the last parameter out.

The six-scalar case is the one whose slot count is known from the parameters alone, so it takes the check btf_prepare_func_args() makes before it walks them, while the other three take the one it makes after. Both report the same way.

GCC passes an aggregate by invisible reference, so a callee it compiles expects a pointer where BTF says the halves of the struct are, and those tests are left to clang.

---
 .../selftests/bpf/prog_tests/aggregate_arg.c  |   9 +
 .../selftests/bpf/progs/aggregate_arg_func.c  | 187 ++++++++++++++++++
 2 files changed, 196 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_func.c
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c new file +9 -0
new file mode 100644
index 000000000000..b230f3bd3b2a
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_arg_func.skel.h"
+
+void test_aggregate_arg(void)
+{
+	RUN_TESTS(aggregate_arg_func);
+}
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +187 -0

187 lines not on this page · raw message

Signature · 1 line
2.52.0
#24 lore raw

Re: [PATCH bpf-next v3 13/15] selftests/bpf: Add inline-asm tests for by-value arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Multi-line comment does not follow the required kernel BPF comment style. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Multi-line comment does not follow the required kernel BPF comment style. --

commit d1826c8bebe9c02679f7bde2ef6a1536675d8bf2
Author: Yonghong Song <[email protected]> selftests/bpf: Add inline-asm tests for by-value arguments

This commit adds inline-asm tests to cover struct argument passing, specifically for structs passed in register pairs, refused pointers, and structs too large for by-value passing. It also checks various placements that global functions cannot have.

Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +0 -0
new file mode 100644
index 0000000000000..ae3faa8a4a940

[ ... ]

tools/testing/selftests/bpf/progs/aggregate_arg_func.c line 89

Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +6 -0
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+/* One slot per parameter, so the slot count is settled before the parameters
+ * are walked. The cases below reach the same count only once they have been.
+ */
+__noinline __u64 global_arg_six_scalars(int a, int b, int c, int d, int e, int f)

[Severity: Low]
This isn't a bug, but does this multi-line comment follow the preferred BPF subsystem comment style? The guidelines require multi-line comments to have the opening /* on its own line, with the text starting on the next line.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
author reviewer
#25 lore raw

PATCH v3 14/15 selftests/bpf: Add tests for by-value kfunc arguments

In reply to Yonghong Song

Add kfuncs taking a 16-byte struct and an __int128 by value, with combinations of <= 8 byte arguments and '> 8 && <= 16' byte arguments. Every test checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly.

There are some cases with __int128 as the register argument. For such arguments, arm64 requires the argument to start at an even slot while x86-64 has no such requirement.

Two more cover a rejection. An aggregate holding a pointer is refused everywhere, and in arena_kfunc.c a two-slot struct pushes an arena pointer past the argument registers, which is refused too. An aggregate too large to pass by value is already covered in aggregate_arg_func.c.

test_stack_arg_big() in stack_arg_fail.c passed a 16-byte struct as the sixth argument and asserted the unrecognized stack argument type it used to be reported as. The JIT places that argument now, so the test is removed.

---
 .../selftests/bpf/prog_tests/aggregate_arg.c  |   2 +
 .../selftests/bpf/progs/aggregate_arg_kfunc.c | 231 ++++++++++++++++++
 .../testing/selftests/bpf/progs/arena_kfunc.c |  15 ++
 .../selftests/bpf/progs/stack_arg_fail.c      |  10 -
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  77 ++++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  32 +++
 6 files changed, 357 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c +2 -0
index b230f3bd3b2a..aa7562f48737 100644
@@ -2,8 +2,10 @@
 /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
 #include <test_progs.h>
 #include "aggregate_arg_func.skel.h"
+#include "aggregate_arg_kfunc.skel.h"
 
 void test_aggregate_arg(void)
 {
 	RUN_TESTS(aggregate_arg_func);
+	RUN_TESTS(aggregate_arg_kfunc);
 }
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +231 -0

231 lines not on this page · raw message

tools/testing/selftests/bpf/progs/arena_kfunc.c +15 -0
index 50609f3b0564..7302d279ead2 100644
@@ -228,6 +228,21 @@ int arena_arg_stack(void *ctx)
 	bpf_kfunc_arena_stack_arg_test(1, 2, 3, 4, 5, (u64 *)1);
 	return 0;
 }
+
+#if defined(__clang__)
+/* The struct takes two slots, so the arena pointer is the sixth. */
+SEC("syscall")
+__arch_x86_64 __arch_arm64
+__failure __msg("arena pointer cannot be a stack argument")
+int arena_arg_stack_after_pair(void *ctx)
+{
+	struct prog_test_pair_arg s = { .lo = 1, .hi = 2 };
+
+	bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
+	bpf_kfunc_call_test_pair_arena_arg(1, 2, 3, s, (u64 *)1);
+	return 0;
+}
+#endif
 #else
 SEC("syscall")
 __arch_x86_64
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0 -10
index eed97d582515..fff2e947ea33 100644
@@ -3,20 +3,10 @@
 
 #include <vmlinux.h>
 #include <bpf/bpf_helpers.h>
-#include "../test_kmods/bpf_testmod_kfunc.h"
 #include "bpf_misc.h"
 
 #if defined(__BPF_FEATURE_STACK_ARGUMENT)
 
-SEC("tc")
-__failure __msg("Unrecognized *(R11-8) type STRUCT")
-int test_stack_arg_big(struct __sk_buff *skb)
-{
-	struct prog_test_big_arg s = { .a = 1, .b = 2 };
-
-	return bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s);
-}
-
 SEC("socket")
 __description("r11 in ALU instruction")
 __failure __msg("R11 is invalid")
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +77 -0

89 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +32 -0
index b213ef14848b..195ec37d5bbc 100644
@@ -61,6 +61,16 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+struct prog_test_pair_arg {	/* 16 bytes: two argument registers */
+	__u64 lo;
+	__u64 hi;
+};
+
+struct prog_test_ptr_arg {	/* 16 bytes, but holds a pointer */
+	void *p;
+	__u64 x;
+};
+
 struct prog_test_ret_pair {	/* 16 bytes: R0:R2 */
 	__u64 lo;
 	__u64 hi;
@@ -221,6 +231,28 @@ __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
 struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
 struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
 struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+__u64 bpf_kfunc_call_test_pair_arg(__u64 a, struct prog_test_pair_arg s, __u64 b) __ksym;
+#ifdef __SIZEOF_INT128__
+__u64 bpf_kfunc_call_test_i128_arg(__u64 a, __u64 b, __int128 v) __ksym;
+__u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
+__u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
+					 __u64 d) __ksym;
+__u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
+				       __u64 d, __u64 e, __u64 f) __ksym;
+__u64 bpf_kfunc_call_test_i128_arg_pad(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				       __u64 f, __u64 g, __int128 v) __ksym;
+#endif
+__u64 bpf_kfunc_call_test_pair_arg_nofit(__u64 a, __u64 b, __u64 c, __u64 d,
+					 struct prog_test_pair_arg s) __ksym;
+__u64 bpf_kfunc_call_test_pair_arg_tail(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+					struct prog_test_pair_arg s, __u64 f) __ksym;
+__u64 bpf_kfunc_call_test_pair_arg_split8(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+					  __u64 f, __u64 g,
+					  struct prog_test_pair_arg s) __ksym;
+__u64 bpf_kfunc_call_test_ptr_arg(struct prog_test_ptr_arg s) __ksym;
+__u64 bpf_kfunc_call_test_pair_arena_arg(__u64 a, __u64 b, __u64 c,
+					 struct prog_test_pair_arg s,
+					 __u64 *f__arena) __ksym;
 struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
 struct prog_test_ret_nested bpf_kfunc_call_test_ret_nested(__u64 tag) __ksym;
 struct prog_test_ret_ptr_arr bpf_kfunc_call_test_ret_ptr_arr(void) __ksym;
Signature · 1 line
2.52.0
#26 lore raw

Re: [PATCH bpf-next v3 14/15] selftests/bpf: Add tests for by-value kfunc arguments

Do these tests actually verify that arguments end up in the correct slots?
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +0 -0
new file mode 100644
index 000000000000..8b0e7aed1ff1

[ ... ]

tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c line 130

Quoted 13 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +13 -0
+SEC("tc")
+__arch_x86_64 __arch_arm64 __load_if_JITed()
+__description("aggregate kfunc argument: struct at the tail")
+__success __retval(0)
+int aggregate_arg_kfunc_tail(struct __sk_buff *skb)
+{
+	u64 a = 1, b = 2;
+	struct prog_test_pair_arg s = { .lo = a, .hi = b };
+
+	if (bpf_kfunc_call_test_pair_arg_tail(1, 2, 3, 4, 5, s, 6) != a + b + 21)
+		return 1;
+	return 0;
+}

Do these tests actually verify that arguments end up in the correct slots?

Looking at the test pattern: bpf_kfunc_call_test_pair_arg_tail(1, 2, 3, 4, 5, s, 6) expects 'a + b + 21', but the kfunc returns the sum of all its arguments unweighted:

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c:

    __bpf_kfunc u64 bpf_kfunc_call_test_pair_arg_tail(
        u64 a, u64 b, u64 c, u64 d, u64 e,
        struct prog_test_pair_arg s, u64 f)
    {
        return a + b + c + d + e + s.lo + s.hi + f;
    }

The check is invariant under any permutation of values across argument slots. It will catch a value that is lost, duplicated or garbage, but not a value that ends up in the wrong slot.

Concrete case in this test: aggregate_arg_kfunc_tail is the one case where x86-64 moves an argument down into a still-live register (BPF slots a..e -> 0..4, s -> 6,7, f -> 5, so f travels through BPF_JIT_ARG_TMP). If the scratch handling swapped f with s.lo, the callee would read s.lo = 6, f = a, and 1+2+3+4+5 + 6 + b + a still equals a + b + 21, so the test passes.

The same holds for the two halves of every 16-byte argument:
'(u64)(v >> 64) + (u64)v' and 's.lo + s.hi' are symmetric, so a hi/lo swap of an __int128 or of prog_test_pair_arg is invisible in all nine value tests.

The commit message states "Every test checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly." Does this hold if the values are commutative?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#27 lore raw

Re: [PATCH bpf-next v3 14/15] selftests/bpf: Add tests for by-value kfunc arguments

Quoted 45 lines · On 9/11/26 9:47 AM, [email protected] wrote:
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +0 -0
new file mode 100644
index 000000000000..8b0e7aed1ff1

[ ... ]

tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c line 130

Quoted 13 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +13 -0
+SEC("tc")
+__arch_x86_64 __arch_arm64 __load_if_JITed()
+__description("aggregate kfunc argument: struct at the tail")
+__success __retval(0)
+int aggregate_arg_kfunc_tail(struct __sk_buff *skb)
+{
+	u64 a = 1, b = 2;
+	struct prog_test_pair_arg s = { .lo = a, .hi = b };
+
+	if (bpf_kfunc_call_test_pair_arg_tail(1, 2, 3, 4, 5, s, 6) != a + b + 21)
+		return 1;
+	return 0;
+}

Do these tests actually verify that arguments end up in the correct slots?

Looking at the test pattern: bpf_kfunc_call_test_pair_arg_tail(1, 2, 3, 4, 5, s, 6) expects 'a + b + 21', but the kfunc returns the sum of all its arguments unweighted:

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c:

     __bpf_kfunc u64 bpf_kfunc_call_test_pair_arg_tail(
         u64 a, u64 b, u64 c, u64 d, u64 e,
         struct prog_test_pair_arg s, u64 f)
     {
         return a + b + c + d + e + s.lo + s.hi + f;
     }

The check is invariant under any permutation of values across argument slots. It will catch a value that is lost, duplicated or garbage, but not a value that ends up in the wrong slot.

Concrete case in this test: aggregate_arg_kfunc_tail is the one case where x86-64 moves an argument down into a still-live register (BPF slots a..e -> 0..4, s -> 6,7, f -> 5, so f travels through BPF_JIT_ARG_TMP). If the scratch handling swapped f with s.lo, the callee would read s.lo = 6, f = a, and 1+2+3+4+5 + 6 + b + a still equals a + b + 21, so the test passes.

Quoted 8 lines

The same holds for the two halves of every 16-byte argument:
'(u64)(v >> 64) + (u64)v' and 's.lo + s.hi' are symmetric, so a hi/lo swap of an __int128 or of prog_test_pair_arg is invisible in all nine value tests.

The commit message states "Every test checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly." Does this hold if the values are commutative?

I will change kfunc to have different values with different multiplier for each argument to make the test more robust.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

author reviewer
#28 lore raw

PATCH v3 15/15 selftests/bpf: Temporary hack to disable register mismatch in arm64

In reply to Yonghong Song

There are 3 kfuncs like below:
__u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym; __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,

                                            __u64 d) __ksym;
   __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
                                          __u64 d, __u64 e, __u64 f) __ksym;

which requires that '__int128 v' must be 16-byte align on arm64.

Current pahole will reject BTF generation since pahole expects '__int128 v' has start register 'x1' while arm64 abi requires 'x2'. Hence, btf generation will fail.

This patch is a hack to disable a few related tests to satisfy CI. The following is the fix in pahole:
https://lore.kernel.org/bpf/[email protected]/
Once pahole patch is merged, this patch can be discarded.

---
 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c    | 4 ++++
 tools/testing/selftests/bpf/test_kmods/bpf_testmod.c       | 4 ++++
 tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h | 2 ++
 3 files changed, 10 insertions(+)
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c +4 -0
index 2100f0ab1abe..59f56ab39413 100644
@@ -50,6 +50,7 @@ int aggregate_arg_kfunc_int128(struct __sk_buff *skb)
 	return 0;
 }
 
+#if 0
 /*
  * arm64 rounds the register number up to an even one for an argument
  * aligned to 16 bytes, so it wants this __int128 in x2 and x3.
@@ -70,6 +71,7 @@ int aggregate_arg_kfunc_int128_odd(struct __sk_buff *skb)
 
 	return 0;
 }
+#endif
 
 #endif /* __SIZEOF_INT128__ */
 
@@ -152,6 +154,7 @@ int aggregate_arg_kfunc_split8(struct __sk_buff *skb)
 
 #ifdef __SIZEOF_INT128__
 
+#if 0
 /*
  * The same hole, with enough arguments after the __int128 that the shift
  * reaches the registers the BPF convention counts as stack slots: arm64
@@ -193,6 +196,7 @@ int aggregate_arg_kfunc_int128_ovf(struct __sk_buff *skb)
 
 	return 0;
 }
+#endif
 
 /*
  * Both conventions pad the stack to align this __int128, and the BPF
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4 -0
index baccee6fdad4..c367442ce086 100644
@@ -1039,6 +1039,7 @@ __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg(u64 a, u64 b, __int128 v)
 	return a + b + (u64)((unsigned __int128)v >> 64) + (u64)v;
 }
 
+#if 0
 __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_odd(u64 a, __int128 v, u64 b)
 {
 	return a + b + (u64)((unsigned __int128)v >> 64) + (u64)v;
@@ -1056,6 +1057,7 @@ __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_ovf(u64 a, __int128 v, u64 b, u64 c
 	return a + b + c + d + e + f +
 	       (u64)((unsigned __int128)v >> 64) + (u64)v;
 }
+#endif
 
 __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_pad(u64 a, u64 b, u64 c, u64 d, u64 e,
 						 u64 f, u64 g, __int128 v)
@@ -1735,9 +1737,11 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_deep)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg)
+#if 0
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_odd)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_shift)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_ovf)
+#endif
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_pad)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg_nofit)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg_tail)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +2 -0
index 195ec37d5bbc..0f098045856e 100644
@@ -234,11 +234,13 @@ struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
 __u64 bpf_kfunc_call_test_pair_arg(__u64 a, struct prog_test_pair_arg s, __u64 b) __ksym;
 #ifdef __SIZEOF_INT128__
 __u64 bpf_kfunc_call_test_i128_arg(__u64 a, __u64 b, __int128 v) __ksym;
+#if 0
 __u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
 __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
 					 __u64 d) __ksym;
 __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
 				       __u64 d, __u64 e, __u64 f) __ksym;
+#endif
 __u64 bpf_kfunc_call_test_i128_arg_pad(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				       __u64 f, __u64 g, __int128 v) __ksym;
 #endif
Signature · 1 line
2.52.0
#29 lore raw

Re: [PATCH bpf-next v3 15/15] selftests/bpf: Temporary hack to disable register mismatch in arm64

The changelog states the patch is a temporary hack that 'can be discarded' once the pahole fix is merged, but the patch is submitted as the last patch of the series with no marker keeping it out of…
Quoted 14 lines

selftests/bpf: Temporary hack to disable register mismatch in arm64

There are 3 kfuncs like below:

   __u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
   __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
                                            __u64 d) __ksym;
   __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
                                          __u64 d, __u64 e, __u64 f) __ksym;

which requires that '__int128 v' must be 16-byte align on arm64.

Current pahole will reject BTF generation since pahole expects '__int128 v' has start register 'x1' while arm64 abi requires 'x2'. Hence, btf generation will fail.

Quoted 6 lines

This patch is a hack to disable a few related tests to satisfy CI. The following is the fix in pahole:
https://lore.kernel.org/bpf/[email protected]/
Once pahole patch is merged, this patch can be discarded.

The changelog states the patch is a temporary hack that 'can be discarded' once the pahole fix is merged, but the patch is submitted as the last patch of the series with no marker keeping it out of a merge. Should this have [DO NOT APPLY] or RFC in the subject, or a note under the --- line where non-applied notes belong?

A few smaller notes: 'must be 16-byte align' should read '16-byte aligned', and the pahole reference would normally be a Link: tag.

tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c int aggregate_arg_kfunc_int128(struct __sk_buff *skb) · line 56

Quoted 12 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c +1 -0
index 2100f0ab1abe..59f56ab39413 100644
@@ -50,6 +50,7 @@ int aggregate_arg_kfunc_int128(struct __sk_buff *skb)
 	return 0;
 }
 
+#if 0
 /*
  * arm64 rounds the register number up to an even one for an argument
  * aligned to 16 bytes, so it wants this __int128 in x2 and x3.

Five #if 0 regions are added and none carries an in-tree comment explaining the disable. The reason (the pahole dependency), the fact that it's meant to be reverted, and the link to the pahole fix exist only in the commit message. Once applied, the tree contains unexplained dead code with nothing that would prompt anyone to re-enable it after the pahole fix lands. Should each #if 0 have a short comment or TODO naming the toolchain dependency, or could these use a version/arch-conditional guard rather than a blanket #if 0?

Quoted 9 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c +1 -0
@@ -70,6 +71,7 @@ int aggregate_arg_kfunc_int128_odd(struct __sk_buff *skb)
 
 	return 0;
 }
+#endif
 
 #endif /* __SIZEOF_INT128__ */
 

tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c int aggregate_arg_kfunc_split8(struct __sk_buff *skb) · line 160

Quoted 7 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c +1 -0
@@ -152,6 +154,7 @@ int aggregate_arg_kfunc_split8(struct __sk_buff *skb)
 
 #ifdef __SIZEOF_INT128__
 
+#if 0
 /*
  * The same hole, with enough arguments after the __int128 that the shift
  * reaches the registers the BPF convention counts as stack slots: arm64

The three programs disabled here (aggregate_arg_kfunc_int128_odd, aggregate_arg_kfunc_int128_shift, and aggregate_arg_kfunc_int128_ovf) are the only tests that observably exercise the arm64 even-register round-up added by 7189ec8b9091 in this same series.

Looking at bpf_jit_place_args() in kernel/bpf/core.c, it applies the round-up only for args flagged BTF_FMODEL_ALIGN16_ARG, i.e. __int128 by value:

    if (align16 && abi->even_reg_align)
            nregs_used = round_up(nregs_used, 2);

arm64_arg_abi sets .even_reg_align = true. Of the __int128 kfunc tests that remain enabled:

- aggregate_arg_kfunc_int128 -> bpf_kfunc_call_test_i128_arg(u64, u64, __int128): nregs_used is 2 at the __int128, round_up(2, 2) == 2, so placement is the identity and emit_kfunc_arg_moves() emits nothing.

- aggregate_arg_kfunc_int128_pad ->

    bpf_kfunc_call_test_i128_arg_pad(7 x u64, __int128): nregs_used is
    7, round_up(7, 2) == 8, but 7 + 2 > 8 already forces the argument
    wholly onto the stack, so the round-up changes nothing observable;
    only the even_stack_align path is covered.

After this commit, does the arm64 register-shift path (the headline behaviour of the arm64 patch in this series) and the alignment-hole-pushes-an-argument-to-the-stack path have any test coverage on any architecture?

The surrounding code already uses #if defined(__x86_64__) || defined(__aarch64__) (bpf_testmod.c and earlier in the same file), and BPF progs are built with -D__TARGET_ARCH_$(SRCARCH). Would an arm64-only guard (plus dropping __arch_arm64 from the affected programs) keep the x86-64 arm of these tests running?

Quoted 20 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c +1 -0
@@ -193,6 +196,7 @@ int aggregate_arg_kfunc_int128_ovf(struct __sk_buff *skb)
 
 	return 0;
 }
+#endif
 
 /*
  * Both conventions pad the stack to align this __int128, and the BPF
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 -0
index baccee6fdad4..c367442ce086 100644
@@ -1039,6 +1039,7 @@ __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg(u64 a, u64 b, __int128 v)
 	return a + b + (u64)((unsigned __int128)v >> 64) + (u64)v;
 }
 
+#if 0
 __bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_odd(u64 a, __int128 v, u64 b)
 {
 	return a + b + (u64)((unsigned __int128)v >> 64) + (u64)v;

[ ... ]

Quoted 20 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +2 -0
@@ -1735,9 +1737,11 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_deep)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg)
+#if 0
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_odd)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_shift)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_ovf)
+#endif
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_pad)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg_nofit)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_pair_arg_tail)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0 -0
index 195ec37d5bbc..0f098045856e 100644
@@ -234,11 +234,13 @@ struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
 __u64 bpf_kfunc_call_test_pair_arg(__u64 a, struct prog_test_pair_arg s, __u64 b) __ksym;
 #ifdef __SIZEOF_INT128__
 __u64 bpf_kfunc_call_test_i128_arg(__u64 a, __u64 b, __int128 v) __ksym;

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; · line 246

Quoted 10 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +2 -0
+#if 0
 __u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
 __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
 					 __u64 d) __ksym;
 __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
 				       __u64 d, __u64 e, __u64 f) __ksym;
+#endif
 __u64 bpf_kfunc_call_test_i128_arg_pad(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				       __u64 f, __u64 g, __int128 v) __ksym;
 #endif

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34620351527

#30 lore raw

Re: [PATCH bpf-next v3 15/15] selftests/bpf: Temporary hack to disable register mismatch in arm64

In reply to Yonghong Song
Quoted 8 lines · Yonghong Song

There are 3 kfuncs like below:

   __u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
   __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
                                            __u64 d) __ksym;
   __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
                                          __u64 d, __u64 e, __u64 f) __ksym;

which requires that '__int128 v' must be 16-byte align on arm64.

Quoted 8 lines

Current pahole will reject BTF generation since pahole expects '__int128 v' has start register 'x1' while arm64 abi requires 'x2'. Hence, btf generation will fail.

This patch is a hack to disable a few related tests to satisfy CI. The following is the fix in pahole:
https://lore.kernel.org/bpf/[email protected]/
Once pahole patch is merged, this patch can be discarded.

You're adding them in patch 14 only to disable them in patch 15? sure, pahole needs to be fixed, but let's carry such selftests for pahole out of tree of the time being.
Once it is fixed and we have a mechanism to detect that it is fixed then we will add these tests.
I'm not sure what would be such pahole detection mechanism. We don't want to bump the version just for that.
So imo just drop such tests.

Also rebase to bpf-next is needed.

pw-bot: cr

author reviewer
#31 lore raw

Re: [PATCH bpf-next v3 15/15] selftests/bpf: Temporary hack to disable register mismatch in arm64

Quoted 26 lines · Alexei Starovoitov
Quoted 8 lines · Yonghong Song

There are 3 kfuncs like below:

    __u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym;
    __u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c,
                                             __u64 d) __ksym;
    __u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c,
                                           __u64 d, __u64 e, __u64 f) __ksym;

which requires that '__int128 v' must be 16-byte align on arm64.

Quoted 8 lines

Current pahole will reject BTF generation since pahole expects '__int128 v' has start register 'x1' while arm64 abi requires 'x2'. Hence, btf generation will fail.

This patch is a hack to disable a few related tests to satisfy CI. The following is the fix in pahole:

   https://lore.kernel.org/bpf/[email protected]/

Once pahole patch is merged, this patch can be discarded.

You're adding them in patch 14 only to disable them in patch 15? sure, pahole needs to be fixed, but let's carry such selftests for pahole out of tree of the time being.
Once it is fixed and we have a mechanism to detect that it is fixed then we will add these tests.
I'm not sure what would be such pahole detection mechanism. We don't want to bump the version just for that.
So imo just drop such tests.

Quoted 1 line

Also rebase to bpf-next is needed.

Okay, will remove patch 15 and remove the tests mentioned in patch 15. Also address with other comments.

And will rebase and re-submit.

Quoted 2 lines

pw-bot: cr