bpf: Support by-value struct and __int128 arguments
Yonghong Song
reviewerv2 cover bpf: Support by-value struct and __int128 arguments
A global function or a kfunc taking a struct or union by value is rejected today:
Arg#1 type STRUCT in tar() is not supported yet. Unrecognized R2 type STRUCT
and an __int128 argument is accepted but mis-counted: the compiler passes it in two registers while the verifier gives it one, so every argument after it is checked against the wrong register and the program is rejected for a register its source never names.
The compiler passes such a value in one argument register per eightbyte, so a value of at most 16 bytes arrives in one or two consecutive registers. This series teaches the verifier to describe that: sub->args[] gains one entry per argument slot rather than one per BTF parameter, and the kfunc argument walk counts slots the same way. A value may only be composed of scalars, as a pointer would reach the callee as an opaque scalar, losing the provenance and reference tracking that make it safe to use. A global function still has no stack arguments, so all of its slots have to fit in the argument registers.
For a kfunc the two calling conventions have to agree, and they do not always. Where the BPF convention splits a value between the last argument register and the stack, x86_64 moves the whole of it to the stack and gives the register to the argument that follows, while arm64 rounds the register number up to an even one for a 16-byte aligned value. For stack arguments, both x86_64 and arm64 will have 16-byte alignment for those 16-byte aligned parameters. Otherwise the alignment will be 8-byte. This is implemented in x86_64 and arm64 jit.
Patch 1 records the __int128 failure as it stands today, which patch 4 turns into a success. Patches 2 to 4 index the arguments of a global function by slot and accept a by-value struct and an __int128. Patches 6 and 7 have the same for a kfunc call. Patches 8 and 9 place the arguments per the kernel calling convention in the x86-64 and arm64 JITs. Patches 10 to 12 are the tests.
Tested on x86-64 and arm64 with test_progs. The by-value struct tests are built with clang, as GCC passes an aggregate by invisible reference, and some of them need __BPF_FEATURE_STACK_ARGUMENT.
Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Fix a few issues due to potential out of bound access.
- Fix a few missed cases for kfunc with '> 8 byte' arguments.
- Add jit support for x86_64 and arm64. Previously certain functions are rejected, and now these functions can succeed.
Yonghong Song (12):
selftests/bpf: Add a test for an __int128 by-value argument
bpf: Index global function arguments by argument slot
bpf: Support by-value struct arguments up to 16 bytes
bpf: Support __int128 as a by-value function argument
bpf: Rename bpf_call_summary::num_params to arg_slot_cnt
bpf: Recognize by-value struct and __int128 kfunc arguments
bpf: Prepare kfunc arguments for the JIT from an ABI description bpf, x86: Move kfunc arguments into the x86-64 calling convention bpf, arm64: Move kfunc arguments into the arm64 calling convention selftests/bpf: Add C tests for by-value arguments up to 16 bytes selftests/bpf: Add inline-asm tests for by-value arguments selftests/bpf: Add tests for by-value kfunc arguments
arch/arm64/net/bpf_jit_comp.c | 77 +++++- arch/x86/net/bpf_jit_comp.c | 73 +++++- include/linux/bpf.h | 8 + include/linux/bpf_verifier.h | 3 +- include/linux/filter.h | 32 +++ kernel/bpf/btf.c | 94 +++++-- kernel/bpf/core.c | 94 +++++++ kernel/bpf/liveness.c | 10 +- kernel/bpf/verifier.c | 239 +++++++++++++++--- .../selftests/bpf/prog_tests/aggregate_arg.c | 11 + .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/aggregate_arg_func.c | 155 ++++++++++++ .../selftests/bpf/progs/aggregate_arg_kfunc.c | 221 ++++++++++++++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 16 ++ .../selftests/bpf/progs/stack_arg_fail.c | 10 - .../selftests/bpf/progs/verifier_int128_arg.c | 196 ++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 77 ++++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 32 +++ 18 files changed, 1271 insertions(+), 79 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_arg.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_func.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_int128_arg.c
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 1/12 selftests/bpf: Add a test for an __int128 by-value argument
A 128-bit integer is passed in two consecutive argument registers, but the verifier counts one argument register per parameter whatever its size. For
__u64 take_i128_global(int a, u128 v, int c)
the compiler passes a in R1, v in R2:R3 and c in R4, while the verifier marks only R1 through R3 at the entry of the global function. The callee then reads its own third parameter out of a register the verifier considers uninitialized, and the program is rejected for a register the source never names:
Validating take_i128_global() func#1...
20: R1=scalar() R2=scalar() R3=scalar() R10=fp0
; __noinline __u64 take_i128_global(int a, u128 v, int c) @ verifier_int128_arg.c:12
20: (bf) r0 = r2 ; R0=scalar(id=4) R2=scalar(id=4)
; return (__u64)a + (__u64)(v >> 64) + (__u64)v + c; @ verifier_int128_arg.c:14
21: (bc) w1 = w1 ; R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) 22: (67) r1 <<= 32 ; R1=scalar(smax=0x7fffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000)) 23: (c7) r1 s>>= 32 ; R1=scalar(smin=0xffffffff80000000,smax=0x7fffffff) 24: (0f) r0 += r1 ; R0=scalar() R1=scalar(smin=0xffffffff80000000,smax=0x7fffffff) 25: (0f) r0 += r3 ; R0=scalar() R3=scalar()
26: (bc) w1 = w4
R4 !read_ok
The log is from clang 23. LLVM 21/22 place the argument in the same registers.
Add the test with the failure it produces now. A later patch will fix this issue so this test should succeed.
--- .../selftests/bpf/prog_tests/verifier.c | 2 ++ .../selftests/bpf/progs/verifier_int128_arg.c | 36 +++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_int128_arg.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -50,6 +50,7 @@#include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h" #include "verifier_helper_value_access.skel.h" +#include "verifier_int128_arg.skel.h" #include "verifier_int_ptr.skel.h" #include "verifier_iterating_callbacks.skel.h" #include "verifier_jeq_infer_not_null.skel.h"
@@ -214,6 +215,7 @@ void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }void test_verifier_helper_value_access (void ) {RUN (verifier_helper_value_access); } +void test_verifier_int128_arg (void ) {RUN_TESTS (verifier_int128_arg); }void test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }void test_verifier_iterating_callbacks (void ) {RUN (verifier_iterating_callbacks); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }
tools/testing/selftests/bpf/progs/verifier_int128_arg.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128 ; + +__noinline __u64 take_i128_global(int a, u128 v, int c) +{ +return (__u64 )a + (__u64)(v >>64 ) + (__u64 )v + c; +} + +SEC ("tc" ) +/* +* The verifier counts one argument register for the __int128 and marks only +* R1 through R3 at the entry of take_i128_global(), while the compiler passed +* a in R1, v in R2:R3 and c in R4. + */ +__failure __msg("R4 !read_ok") +int aggregate_arg_int128_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +u128 v = ((u128 )a <<64 ) | b; + +if (take_i128_global (1 , v,2 ) != a + b +3 ) +return 1 ; + +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 01/12] selftests/bpf: Add a test for an __int128 by-value argument
This isn't a bug, but the name verifier_int128_arg describes only the one case progs/verifier_int128_arg.c holds at this commit. Two patches later in the same series, f4da9830f5a9 adds by-value…
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +1
-0@@ -50,6 +50,7 @@#include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h" #include "verifier_helper_value_access.skel.h" +#include "verifier_int128_arg.skel.h"
[ ... ]
tools/testing/selftests/bpf/prog_tests/verifier.c void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_ · line 219
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +1
-0@@ -214,6 +215,7 @@ void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }void test_verifier_helper_value_access (void ) {RUN (verifier_helper_value_access); } +void test_verifier_int128_arg (void ) {RUN_TESTS (verifier_int128_arg); }void test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }
This isn't a bug, but the name verifier_int128_arg describes only the one case progs/verifier_int128_arg.c holds at this commit. Two patches later in the same series, f4da9830f5a9 adds by-value struct argument tests (struct pair, struct small, struct with_ptr) to the same file, none of which are __int128. The sibling entry two lines up is test_verifier_aggregate_ret for the aggregate-return family.
Since progs/verifier_int128_arg.c later grows to cover by-value struct arguments too, would a name in the same family as verifier_aggregate_ret -- say verifier_aggregate_arg -- fit the file better and save a rename later in the series?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 01/12] selftests/bpf: Add a test for an __int128 by-value argument
Quoted 24 lines · On 9/9/26 12:13 AM, [email protected] wrote:
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +1
-0@@ -50,6 +50,7 @@#include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h" #include "verifier_helper_value_access.skel.h" +#include "verifier_int128_arg.skel.h" [ ... ]
tools/testing/selftests/bpf/prog_tests/verifier.c void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_ · line 219
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +1
-0@@ -214,6 +215,7 @@ void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }void test_verifier_helper_value_access (void ) {RUN (verifier_helper_value_access); } +void test_verifier_int128_arg (void ) {RUN_TESTS (verifier_int128_arg); }void test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }This isn't a bug, but the name
verifier_int128_argdescribes only the one caseprogs/verifier_int128_arg.cholds at this commit. Two patches later in the same series,f4da9830f5a9adds by-value struct argument tests (struct pair, struct small, structwith_ptr) to the same file, none of which are__int128. The sibling entry two lines up istest_verifier_aggregate_retfor the aggregate-return family.
Quoted 4 lines
Since
progs/verifier_int128_arg.clater grows to cover by-value struct arguments too, would a name in the same family asverifier_aggregate_ret-- sayverifier_aggregate_arg-- fit the file better and save a rename later in the series?
Correct, will use verifier_aggregate_arg.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 2/12 bpf: Index global function arguments by argument slot
In reply to Yonghong Songbtf_prepare_func_args() indexes sub->args[] by BTF parameter, so a parameter can only ever stand for a single argument register. Give the loop a second index: 'i' keeps walking the BTF parameters while 'slots_used' walks the argument slots, and sub->arg_cnt becomes the number of slots, so that a later patch can give a parameter two of them.
No functional change: every parameter still takes exactly one slot.
--- kernel/bpf/btf.c | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-)
kernel/bpf/btf.c +24 -16
@@ -8037,7 +8037,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)const struct btf_param *args;const struct btf_type *t, *ref_t, *fn_t;int err; -u32 i, nargs, btf_id; +u32 i, slots_used, nargs, btf_id;const char *tname;if (sub->args_cached)
@@ -8122,8 +8122,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)/* Convert BTF function arguments into verifier types. * Only PTR_TO_CTX and SCALAR are supported atm. */ -for (i =0 ; i < nargs; i++) { +for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; + err =btf_scan_decl_tags (env, btf, fn_t, i, is_global, &tags);if (err)return err;
@@ -8145,7 +8146,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)btf_validate_prog_ctx_type (log, btf, t, i, prog_type, prog->expected_attach_type)) return -EINVAL; - sub->args[i].arg_type =ARG_PTR_TO_CTX ; + sub->args[slots_used++].arg_type =ARG_PTR_TO_CTX ;continue ; }if (btf_is_dynptr_ptr (btf, t)) {
@@ -8153,7 +8154,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)bpf_log (log,"arg#%d has invalid combination of tags\n" , i);return -EINVAL ; } - sub->args[i].arg_type =ARG_PTR_TO_DYNPTR ; + sub->args[slots_used++].arg_type =ARG_PTR_TO_DYNPTR ;continue ; }if (tags &ARG_TAG_TRUSTED ) {
@@ -8168,10 +8169,11 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)if (kern_type_id <0 )return kern_type_id; - sub->args[i].arg_type =ARG_PTR_TO_BTF_ID |PTR_TRUSTED ; + sub->args[slots_used].arg_type =ARG_PTR_TO_BTF_ID |PTR_TRUSTED ;if (tags &ARG_TAG_NULLABLE ) - sub->args[i].arg_type |=PTR_MAYBE_NULL ; - sub->args[i].btf_id = kern_type_id; + sub->args[slots_used].arg_type |=PTR_MAYBE_NULL ; + sub->args[slots_used].btf_id = kern_type_id; + slots_used++;continue ; }if (tags &ARG_TAG_UNTRUSTED ) {
@@ -8185,8 +8187,10 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) ref_t =btf_type_skip_modifiers (btf, t->type,NULL );if (btf_type_is_void (ref_t) ||btf_type_is_primitive (ref_t)) { - sub->args[i].arg_type =ARG_PTR_TO_MEM |MEM_RDONLY |PTR_UNTRUSTED ; - sub->args[i].mem_size =0 ; + sub->args[slots_used].arg_type =ARG_PTR_TO_MEM |MEM_RDONLY | +PTR_UNTRUSTED ; + sub->args[slots_used].mem_size =0 ; + slots_used++;continue ; }
@@ -8202,8 +8206,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) i, btf_type_str(ref_t), tname);return -EINVAL ; } - sub->args[i].arg_type =ARG_PTR_TO_BTF_ID |PTR_UNTRUSTED ; - sub->args[i].btf_id = kern_type_id; + sub->args[slots_used].arg_type =ARG_PTR_TO_BTF_ID |PTR_UNTRUSTED ; + sub->args[slots_used].btf_id = kern_type_id; + slots_used++;continue ; }if (tags &ARG_TAG_ARENA ) {
@@ -8211,7 +8216,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)bpf_log (log,"arg#%d arena cannot be combined with any other tags\n" , i);return -EINVAL ; } - sub->args[i].arg_type =ARG_PTR_TO_ARENA ; + sub->args[slots_used++].arg_type =ARG_PTR_TO_ARENA ;continue ; }if (is_global) {/* generic user data pointer */
@@ -8231,10 +8236,11 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } - sub->args[i].arg_type =ARG_PTR_TO_MEM |PTR_MAYBE_NULL ; + sub->args[slots_used].arg_type =ARG_PTR_TO_MEM |PTR_MAYBE_NULL ;if (tags &ARG_TAG_NONNULL ) - sub->args[i].arg_type &= ~PTR_MAYBE_NULL ; - sub->args[i].mem_size = mem_size; + sub->args[slots_used].arg_type &= ~PTR_MAYBE_NULL ; + sub->args[slots_used].mem_size = mem_size; + slots_used++;continue ; }
@@ -8244,7 +8250,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; }if (btf_type_is_int (t) ||btf_is_any_enum (t)) { - sub->args[i].arg_type =ARG_ANYTHING ; + sub->args[slots_used++].arg_type =ARG_ANYTHING ;continue ; }if (!is_global)
@@ -8254,6 +8260,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } + sub->arg_cnt = slots_used; + sub->args_cached = true;return 0 ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 02/12] bpf: Index global function arguments by argument slot
This isn't a bug, but would a short comment on the early 'sub->arg_cnt = nargs;' noting that it exists only so the error path in btf_check_func_arg_match() has a stack-arg count, and that the…
Quoted 11 lines
kernel/bpf/btf.c +1
-1@@ -8037,7 +8037,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)const struct btf_param *args;const struct btf_type *t, *ref_t, *fn_t;int err; -u32 i, nargs, btf_id; +u32 i, slots_used, nargs, btf_id;const char *tname;
[ ... ]
Quoted 3 lines
kernel/bpf/btf.c +0
-0args = (const struct btf_param *)(t +1 ); nargs =btf_type_vlen (t); sub->arg_cnt = nargs;
[ ... ]
Quoted 8 lines
kernel/bpf/btf.c +2
-1@@ -8122,8 +8122,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)/* Convert BTF function arguments into verifier types. * Only PTR_TO_CTX and SCALAR are supported atm. */ -for (i =0 ; i < nargs; i++) { +for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; +
[ ... ]
kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8264
Quoted 6 lines
kernel/bpf/btf.c +2
-0@@ -8254,6 +8260,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } + sub->arg_cnt = slots_used; +
This isn't a bug, but would a short comment on the early 'sub->arg_cnt = nargs;' noting that it exists only so the error path in btf_check_func_arg_match() has a stack-arg count, and that the authoritative slot count is set after the loop, make the two meanings of arg_cnt easier to keep straight in the later patches?
kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8267
Quoted 3 lines
kernel/bpf/btf.c +0
-0sub->args_cached = true;return 0 ;
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 02/12] bpf: Index global function arguments by argument slot
Quoted 32 lines · On 9/9/26 12:13 AM, [email protected] wrote:
Quoted 11 lines
kernel/bpf/btf.c +1
-1@@ -8037,7 +8037,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)const struct btf_param *args;const struct btf_type *t, *ref_t, *fn_t;int err; -u32 i, nargs, btf_id; +u32 i, slots_used, nargs, btf_id;const char *tname;[ ... ]
Quoted 3 lines
args = (const struct btf_param *)(t + 1); nargs = btf_type_vlen(t); sub->arg_cnt = nargs;[ ... ]
Quoted 8 lines
kernel/bpf/btf.c +2
-1@@ -8122,8 +8122,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)/* Convert BTF function arguments into verifier types. * Only PTR_TO_CTX and SCALAR are supported atm. */ -for (i =0 ; i < nargs; i++) { +for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; +[ ... ]
Quoted 4 lines
kernel/bpf/btf.c +0
-0@@ -8254,6 +8260,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; }
Quoted 7 lines
kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8264
Quoted 2 lines
kernel/bpf/btf.c +2
-0+ sub->arg_cnt = slots_used; +This isn't a bug, but would a short comment on the early '
sub->arg_cnt= nargs;' noting that it exists only so the error path inbtf_check_func_arg_match()has a stack-arg count, and that the authoritative slot count is set after the loop, make the two meanings ofarg_cnteasier to keep straight in the later patches?
I will rename arg_cnt to arg_slot_cnt to make it clear.
Quoted 10 lines
kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8267
Quoted 4 lines
kernel/bpf/btf.c +0
-0sub->args_cached = true;return 0 ;---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 3/12 bpf: Support by-value struct arguments up to 16 bytes
In reply to Yonghong SongA global function taking a struct or union by value is rejected today:
Arg#1 type STRUCT in tar() is not supported yet.
Accept one of at most 16 bytes, which arrives in one or two consecutive argument registers. Only structs composed entirely of scalars are taken for now; btf_struct_is_composed_of() enforces that.
The slots of a value are independent of each other, so the compiler may split one across the last argument register and the stack, as in
static void f(int a, int b, int c, int d, struct pair p);
or place it wholly past the registers, and the verifier describes either the same way. What has to follow the slots is stack_arg_cnt, recomputed from the slots consumed, together with the "no stack args in global functions" and JIT support checks, and the MAX_BPF_FUNC_ARGS bound, which now has to account for a parameter that takes two slots at once.
The previous patch made sub->arg_cnt a count of argument slots; this is the first patch where that count can exceed the number of BTF parameters, so a slot index no longer indexes the parameter array. Have btf_check_func_arg_match() hand the parameters to check_outgoing_stack_args(), which uses them only to name a stack argument in its diagnostic, just when the two counts still agree.
--- kernel/bpf/btf.c | 52 +++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 7 ++++++ 2 files changed, 59 insertions(+)
kernel/bpf/btf.c +52 -0
@@ -8125,6 +8125,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; +if (slots_used >=MAX_BPF_FUNC_ARGS ) + goto too_many_slots; + err =btf_scan_decl_tags (env, btf, fn_t, i, is_global, &tags);if (err)return err;
@@ -8253,6 +8256,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) sub->args[slots_used++].arg_type =ARG_ANYTHING ;continue ; } +if (btf_type_is_struct (t)) { +u32 nslots; + +if (!t->size || t->size >2 *BPF_REG_SIZE ) { +if (!is_global) +return -EINVAL ; +bpf_log (log, +"Arg#%d type %s in %s() has size %u, only 1 to %d bytes " +"can be passed by value\n" , + i,btf_type_str (t), tname, t->size,2 *BPF_REG_SIZE ); +return -EINVAL ; + } +if (!btf_struct_is_composed_of (env, btf, t,BTF_MEMBER_SCALAR )) { +if (!is_global) +return -EINVAL ; +bpf_log (log,"Arg#%d type %s in %s() is not composed of scalars\n" , + i,btf_type_str (t), tname); +return -EINVAL ; + } + + nslots = (t->size +BPF_REG_SIZE -1 ) /BPF_REG_SIZE ; +if (slots_used + nslots >MAX_BPF_FUNC_ARGS ) + goto too_many_slots; +while (nslots--) + sub->args[slots_used++].arg_type =ARG_ANYTHING ; +continue ; + }if (!is_global)return -EINVAL ; bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",
@@ -8260,11 +8290,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } +if (slots_used >MAX_BPF_FUNC_REG_ARGS ) { +if (is_global) { +bpf_log (log, +"global function %s() needs %d > %d argument slots, " +"stack args not supported\n" , + tname, slots_used,MAX_BPF_FUNC_REG_ARGS ); +return -EINVAL ; + } +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d argument slots\n" , + tname, slots_used); +return -EFAULT ; + } + sub->stack_arg_cnt = slots_used -MAX_BPF_FUNC_REG_ARGS ; + } sub->arg_cnt = slots_used; sub->args_cached = true;return 0 ; + +too_many_slots: +if (!is_global) +return -EINVAL ; +bpf_log (log,"Arguments of %s() need more than %d argument slots\n" , + tname,MAX_BPF_FUNC_ARGS ); +return -EINVAL ; } static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
kernel/bpf/verifier.c +7 -0
@@ -9784,6 +9784,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, func =btf_type_by_id (btf, env->prog->aux->func_info[subprog].type_id); func_proto =btf_type_by_id (btf, func->type); args =btf_params (func_proto); +/* +* An argument slot index can run past the end of the parameter array, +* so only pass the parameters, for the names of the stack arguments, +* if the two match. +*/ +if (sub->arg_cnt !=btf_type_vlen (func_proto)) + args =NULL ; ret =check_outgoing_stack_args (env, caller, sub->arg_cnt,bpf_subprog_name (env, subprog), btf, args);if (ret)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 03/12] bpf: Support by-value struct arguments up to 16 bytes
This isn't a bug, but now that slots_used is always >= nargs, is the pre-loop nargs-based block still needed? Looking earlier in the function:
Quoted 14 lines
kernel/bpf/btf.c +3
-0@@ -8125,6 +8125,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; +if (slots_used >=MAX_BPF_FUNC_ARGS ) + goto too_many_slots; + err =btf_scan_decl_tags (env, btf, fn_t, i, is_global, &tags);if (err)return err;
[ ... ]
Quoted 44 lines
kernel/bpf/btf.c +33
-0@@ -8253,6 +8256,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) sub->args[slots_used++].arg_type =ARG_ANYTHING ;continue ; } +if (btf_type_is_struct (t)) { +u32 nslots; + +if (!t->size || t->size >2 *BPF_REG_SIZE ) { +if (!is_global) +return -EINVAL ; +bpf_log (log, +"Arg#%d type %s in %s() has size %u, only 1 to %d bytes " +"can be passed by value\n" , + i,btf_type_str (t), tname, t->size,2 *BPF_REG_SIZE ); +return -EINVAL ; + } +if (!btf_struct_is_composed_of (env, btf, t,BTF_MEMBER_SCALAR )) { +if (!is_global) +return -EINVAL ; +bpf_log (log,"Arg#%d type %s in %s() is not composed of scalars\n" , + i,btf_type_str (t), tname); +return -EINVAL ; + } + + nslots = (t->size +BPF_REG_SIZE -1 ) /BPF_REG_SIZE ; +if (slots_used + nslots >MAX_BPF_FUNC_ARGS ) + goto too_many_slots; +while (nslots--) + sub->args[slots_used++].arg_type =ARG_ANYTHING ; +continue ; + }if (!is_global)return -EINVAL ; bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",@@ -8260,11 +8290,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } +if (slots_used >MAX_BPF_FUNC_REG_ARGS ) { +if (is_global) { +bpf_log (log, +"global function %s() needs %d > %d argument slots, " +"stack args not supported\n" , + tname, slots_used,MAX_BPF_FUNC_REG_ARGS );
kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8308
Quoted 10 lines
kernel/bpf/btf.c +9
-0+return -EINVAL ; + } +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d argument slots\n" , + tname, slots_used); +return -EFAULT ; + } + sub->stack_arg_cnt = slots_used -MAX_BPF_FUNC_REG_ARGS ; + } sub->arg_cnt = slots_used;
This isn't a bug, but now that slots_used is always >= nargs, is the pre-loop nargs-based block still needed? Looking earlier in the function:
if (nargs > MAX_BPF_FUNC_REG_ARGS) {
if (!bpf_jit_supports_stack_args()) {
bpf_log(log, "JIT does not support function %s() with %d args\n",
tname, nargs);
return -EFAULT;
}
sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
}
if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}The new block above recomputes stack_arg_cnt from slots_used. Since every parameter consumes at least one slot, whenever nargs > MAX_BPF_FUNC_REG_ARGS, slots_used > MAX_BPF_FUNC_REG_ARGS as well, so the assignment sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS in the earlier block is always overwritten by the new sub->stack_arg_cnt = slots_used - MAX_BPF_FUNC_REG_ARGS assignment.
Also, the two blocks order their tests differently. The earlier block tests bpf_jit_supports_stack_args() before is_global, while the new block tests is_global first. On an architecture without stack-arg JIT support, a global function needing stack args therefore fails with -EFAULT and "JIT does not support function f() with 6 args" when the sixth slot comes from a sixth parameter, but with -EINVAL and "global function f() needs 6 > 5 argument slots, stack args not supported" when it comes from a two-slot struct.
Could this new block be the only place that sets stack_arg_cnt and applies the global/JIT checks, or does the earlier nargs-based block serve a purpose I'm missing?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 03/12] bpf: Support by-value struct arguments up to 16 bytes
Quoted 103 lines · On 9/9/26 12:13 AM, [email protected] wrote:
Quoted 14 lines
kernel/bpf/btf.c +3
-0@@ -8125,6 +8125,9 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)for (i =0 , slots_used =0 ; i < nargs; i++) {u32 tags =0 ; +if (slots_used >=MAX_BPF_FUNC_ARGS ) + goto too_many_slots; + err =btf_scan_decl_tags (env, btf, fn_t, i, is_global, &tags);if (err)return err;[ ... ]
Quoted 44 lines
kernel/bpf/btf.c +33
-0@@ -8253,6 +8256,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) sub->args[slots_used++].arg_type =ARG_ANYTHING ;continue ; } +if (btf_type_is_struct (t)) { +u32 nslots; + +if (!t->size || t->size >2 *BPF_REG_SIZE ) { +if (!is_global) +return -EINVAL ; +bpf_log (log, +"Arg#%d type %s in %s() has size %u, only 1 to %d bytes " +"can be passed by value\n" , + i,btf_type_str (t), tname, t->size,2 *BPF_REG_SIZE ); +return -EINVAL ; + } +if (!btf_struct_is_composed_of (env, btf, t,BTF_MEMBER_SCALAR )) { +if (!is_global) +return -EINVAL ; +bpf_log (log,"Arg#%d type %s in %s() is not composed of scalars\n" , + i,btf_type_str (t), tname); +return -EINVAL ; + } + + nslots = (t->size +BPF_REG_SIZE -1 ) /BPF_REG_SIZE ; +if (slots_used + nslots >MAX_BPF_FUNC_ARGS ) + goto too_many_slots; +while (nslots--) + sub->args[slots_used++].arg_type =ARG_ANYTHING ; +continue ; + }if (!is_global)return -EINVAL ; bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",@@ -8260,11 +8290,33 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)return -EINVAL ; } +if (slots_used >MAX_BPF_FUNC_REG_ARGS ) { +if (is_global) { +bpf_log (log, +"global function %s() needs %d > %d argument slots, " +"stack args not supported\n" , + tname, slots_used,MAX_BPF_FUNC_REG_ARGS );kernel/bpf/btf.c int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · line 8308
Quoted 10 lines
kernel/bpf/btf.c +9
-0+return -EINVAL ; + } +if (!bpf_jit_supports_stack_args ()) { +bpf_log (log,"JIT does not support function %s() with %d argument slots\n" , + tname, slots_used); +return -EFAULT ; + } + sub->stack_arg_cnt = slots_used -MAX_BPF_FUNC_REG_ARGS ; + } sub->arg_cnt = slots_used;This isn't a bug, but now that
slots_usedis always >= nargs, is the pre-loop nargs-based block still needed? Looking earlier in the function:if (nargs > MAX_BPF_FUNC_REG_ARGS) { if (!bpf_jit_supports_stack_args()) { bpf_log(log, "JIT does not support function %s() with %d args\n", tname, nargs); return -EFAULT; } sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS; } if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) { bpf_log(log, "global function %s has %d > %d args, stack args not supported\n", tname, nargs, MAX_BPF_FUNC_REG_ARGS); return -EINVAL; }The new block above recomputes
stack_arg_cntfromslots_used. Since every parameter consumes at least one slot, whenever nargs >MAX_BPF_FUNC_REG_ARGS,slots_used>MAX_BPF_FUNC_REG_ARGSas well, so the assignmentsub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGSin the earlier block is always overwritten by the newsub->stack_arg_cnt = slots_used - MAX_BPF_FUNC_REG_ARGSassignment.Also, the two blocks order their tests differently. The earlier block tests
bpf_jit_supports_stack_args()beforeis_global, while the new block testsis_globalfirst. On an architecture without stack-arg JIT support, a global function needing stack args therefore fails with-EFAULTand "JIT does not support functionf()with 6 args" when the sixth slot comes from a sixth parameter, but with-EINVALand "global functionf()needs 6 > 5 argument slots, stack args not supported" when it comes from a two-slot struct.
Quoted 3 lines
Could this new block be the only place that sets
stack_arg_cntand applies the global/JIT checks, or does the earlier nargs-based block serve a purpose I'm missing?
As you mentioned, actually we do have some issues. The above nargs and is_global chcking is before going through all arguments. So they works. But we should check again after all arguments processing since some arguments may have two registers.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 4/12 bpf: Support __int128 as a by-value function argument
In reply to Yonghong SongA 128-bit integer follows the same calling convention as a 16-byte by-value struct: LLVM emits it as a 16-byte BTF_KIND_INT and passes it in two consecutive argument registers. So a __int128 gets its two slots.
The test added at the start of the series, which recorded the wrong register map as an R4 !read_ok rejection, now passes and is flipped to __success.
--- kernel/bpf/btf.c | 9 +++------ tools/testing/selftests/bpf/progs/verifier_int128_arg.c | 7 +------ 2 files changed, 4 insertions(+), 12 deletions(-)
kernel/bpf/btf.c +3 -6
@@ -8252,11 +8252,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)bpf_log (log,"arg#%d has pointer tag, but is not a pointer type\n" , i);return -EINVAL ; } -if (btf_type_is_int (t) ||btf_is_any_enum (t)) { - sub->args[slots_used++].arg_type =ARG_ANYTHING ; -continue ; - } -if (btf_type_is_struct (t)) { +if (btf_type_is_int (t) ||btf_is_any_enum (t) ||btf_type_is_struct (t)) {u32 nslots;if (!t->size || t->size >2 *BPF_REG_SIZE ) {
@@ -8268,7 +8264,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) i, btf_type_str(t), tname, t->size, 2 * BPF_REG_SIZE);return -EINVAL ; } -if (!btf_struct_is_composed_of (env, btf, t,BTF_MEMBER_SCALAR )) { +if (btf_type_is_struct (t) && + !btf_struct_is_composed_of (env, btf, t,BTF_MEMBER_SCALAR )) {if (!is_global)return -EINVAL ; bpf_log(log, "Arg#%d type %s in %s() is not composed of scalars\n",
tools/testing/selftests/bpf/progs/verifier_int128_arg.c +1 -6
@@ -15,12 +15,7 @@ __noinline __u64 take_i128_global(int a, u128 v, int c) } SEC("tc") -/* -* The verifier counts one argument register for the __int128 and marks only -* R1 through R3 at the entry of take_i128_global(), while the compiler passed -* a in R1, v in R2:R3 and c in R4. - */ -__failure __msg("R4 !read_ok") +__success __retval(0) int aggregate_arg_int128_c_test(struct __sk_buff *skb) {__u64 a = skb->len ^MIX_A ;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 5/12 bpf: Rename bpf_call_summary::num_params to arg_slot_cnt
In reply to Yonghong SongThe field 'num_params' counts the argument registers and outgoing stack slots a helper or kfunc call takes. The next patch gives a 16-byte parameter two slots, so the name stops describing what the field holds.
Rename 'num_params' to 'arg_slot_cnt'. No functional change.
--- include/linux/bpf_verifier.h | 2 +- kernel/bpf/liveness.c | 10 +++++----- kernel/bpf/verifier.c | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-)
include/linux/bpf_verifier.h +1 -1
@@ -1066,7 +1066,7 @@ static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) }struct bpf_call_summary { -u8 num_params; +u8 arg_slot_cnt;bool is_void;bool fastcall; };
kernel/bpf/liveness.c +5 -5
@@ -1434,21 +1434,21 @@ static int record_call_access(struct bpf_verifier_env *env, {struct bpf_insn *insn = &env->prog->insnsi[insn_idx];struct bpf_call_summary cs; -int r, err, num_params =5 ; +int r, err, arg_slot_cnt =5 ;if (bpf_pseudo_call (insn))return 0 ;if (bpf_get_call_summary (env, insn, &cs)) - num_params = cs.num_params; + arg_slot_cnt = cs.arg_slot_cnt; -for (r =BPF_REG_1 ; r <BPF_REG_1 +min (num_params,MAX_BPF_FUNC_REG_ARGS ); r++) { +for (r =BPF_REG_1 ; r <BPF_REG_1 +min (arg_slot_cnt,MAX_BPF_FUNC_REG_ARGS ); r++) { err =record_arg_access (env, instance, insn, &at[r], r -1 , insn_idx);if (err)return err; } -for (r =0 ; r <MAX_STACK_ARG_SLOTS && r < num_params -MAX_BPF_FUNC_REG_ARGS ; r++) { +for (r =0 ; r <MAX_STACK_ARG_SLOTS && r < arg_slot_cnt -MAX_BPF_FUNC_REG_ARGS ; r++) { err =record_arg_access (env, instance, insn, &at[MAX_BPF_REG + r], r +MAX_BPF_FUNC_REG_ARGS , insn_idx);if (err)
@@ -2199,7 +2199,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, def =ALL_CALLER_SAVED_REGS ; use = def & ~BIT (BPF_REG_0 );if (bpf_get_call_summary (env, insn, &cs)) - use =GENMASK (min_t (u8, cs.num_params,MAX_BPF_FUNC_REG_ARGS ),1 ); + use =GENMASK (min_t (u8, cs.arg_slot_cnt,MAX_BPF_FUNC_REG_ARGS ),1 ); def =mask_widen (def); use =mask_widen (use);break ;
kernel/bpf/verifier.c +4 -4
@@ -17977,11 +17977,11 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, (bpf_verifier_inlines_helper_call (env, call->imm) ||bpf_jit_inlines_helper_call (call->imm)); cs->is_void = fn->ret_type ==RET_VOID ; - cs->num_params =0 ; + cs->arg_slot_cnt =0 ;for (i =0 ; i <ARRAY_SIZE (fn->arg_type); ++i) {if (fn->arg_type[i] ==ARG_DONTCARE )break ; - cs->num_params++; + cs->arg_slot_cnt++; }return true; }
@@ -17993,7 +17993,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,if (err <0 )/* error would be reported later */ return false; - cs->num_params =btf_type_vlen (meta.func_proto); + cs->arg_slot_cnt =btf_type_vlen (meta.func_proto); cs->fastcall = meta.kfunc_flags &KF_FASTCALL ; cs->is_void =btf_type_is_void (btf_type_by_id (meta.btf, meta.func_proto->type));return true;
@@ -18102,7 +18102,7 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,* - includes R1-R5 if corresponding parameter has is described * in the function prototype. */ - clobbered_regs_mask =GENMASK (cs.num_params, cs.is_void ?1 :0 ); + clobbered_regs_mask =GENMASK (cs.arg_slot_cnt, cs.is_void ?1 :0 );/* e.g. if helper call clobbers r{0,1}, expect r{2,3,4,5} in the pattern */ expected_regs_mask = ~clobbered_regs_mask &ALL_CALLER_SAVED_REGS ;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 6/12 bpf: Recognize by-value struct and __int128 kfunc arguments
In reply to Yonghong SongA kfunc taking a struct or union by value is rejected today, and one taking an __int128 is accepted but mis-counted:
Unrecognized R2 type STRUCT
The kfunc arguments walk the same slot as a BPF-to-BPF call:
one argument register per eightbyte, and a 16-byte value takes two. The outgoing stack argument count at the call site follows the slots for the same reason. Similar to BPF-to-BPF aggregate handling, a kfunc aggregate argument is only supported when it is composed of scalars. Everything that maps a kfunc argument to a register has to follow the slots too.
An argument of a single eightbyte lands in the same place under every calling convention, so those are taken. The conventions the JIT has to reconcile do not agree on where a larger one goes, so refuse it for now with
Function f arg#1 type INT cannot be passed at argument slot 1 on this architecture
which the next patch turns into an answer from the JIT. The paths that handle a two-slot argument are therefore unreachable until then.
--- kernel/bpf/verifier.c | 148 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 121 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +121 -27
@@ -12154,12 +12154,30 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta)return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } +static u32 kfunc_arg_slots (const struct btf_type *t) +{ +if (btf_type_is_int (t) ||btf_type_is_struct (t)) +return (t->size +BPF_REG_SIZE -1 ) /BPF_REG_SIZE ; +return 1 ; +} + +static u32 kfunc_proto_slots (const struct btf *btf,const struct btf_type *func_proto) +{ +const struct btf_param *args =btf_params (func_proto); +u32 i, nargs =btf_type_vlen (func_proto), slots_used =0 ; + +for (i =0 ; i < nargs; i++) + slots_used +=kfunc_arg_slots (btf_type_skip_modifiers (btf, args[i].type,NULL )); + +return slots_used; +} +static int get_kfunc_arg_type (struct bpf_verifier_env *env,struct bpf_call_arg_meta *meta, -const struct btf_param *args,int arg,int nargs) +const struct btf_param *args,int arg,int nargs,u32 slot) {const struct btf_type *t, *ref_t =NULL ; -argno_t argno =argno_from_arg (arg +1 ); +argno_t argno =argno_from_arg (slot +1 );const char *ref_tname =NULL ;int arg_type;
@@ -12179,6 +12197,23 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,return KF_ARG_ANYTHING ; } +if (btf_type_is_struct (t)) { +if (!t->size || t->size >2 *BPF_REG_SIZE ) { +verbose (env, +"%s type %s has size %u, only 1 to %d bytes " +"can be passed by value\n" , +reg_arg_name (env, argno),btf_type_str (t), t->size, +2 *BPF_REG_SIZE ); +return -EINVAL ; + } +if (!btf_type_is_scalar_struct (env, meta->btf, t)) { +verbose (env,"%s type %s is not composed of scalars\n" , +reg_arg_name (env, argno),btf_type_str (t)); +return -EINVAL ; + } +return KF_ARG_ANYTHING ; + } +if (!btf_type_is_ptr (t)) {verbose (env,"Unrecognized %s type %s\n" ,reg_arg_name (env, argno),btf_type_str (t));
@@ -12294,7 +12329,7 @@ static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg {const struct btf *btf = meta->btf;const struct btf_param *args; -u32 i, nargs; +u32 i, nargs, slots_used;int arg_type; args = (const struct btf_param *)(meta->func_proto +1 );
@@ -12310,19 +12345,44 @@ static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_argreturn -ENOTSUPP ; } -for (i =0 ; i < nargs; i++) { +for (i =0 , slots_used =0 ; i < nargs; i++) { +const struct btf_type *t; +u32 nslots; + + t =btf_type_skip_modifiers (btf, args[i].type,NULL ); + nslots =kfunc_arg_slots (t); +/* +* The calling conventions the JIT has to reconcile do not +* agree on where an argument of more than one eightbyte goes, +* so refuse one until the JIT can say where this arch puts it. +*/ +if (nslots >1 ) { +verbose (env, +"Function %s arg#%d type %s cannot be passed at " +"argument slot %d on this architecture\n" , + meta->func_name, i,btf_type_str (t), slots_used); +return -EINVAL ; + } + slots_used += nslots; +if (is_kfunc_arg_prog_aux (btf, &args[i]) ||is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))continue ; - arg_type =get_kfunc_arg_type (env, meta, args, i, nargs); + arg_type =get_kfunc_arg_type (env, meta, args, i, nargs, slots_used - nslots);if (arg_type <0 )return arg_type; proto->arg_type[i] = arg_type; } +if (slots_used >MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args ()) { +verbose (env,"JIT does not support kfunc %s() with %d argument slots\n" , + meta->func_name, slots_used); +return -ENOTSUPP ; + } +return 0 ; }
@@ -12897,29 +12957,35 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_meconst struct btf *btf = meta->btf;const struct btf_param *args;struct btf_record *rec; -u32 i, nargs; +u32 i, k, nargs, proto_slots, slots_used, prev_slot =0 , nslots =0 ;int ret; args = (const struct btf_param *)(meta->func_proto +1 ); nargs =btf_type_vlen (meta->func_proto); + proto_slots =kfunc_proto_slots (btf, meta->func_proto); - ret =check_outgoing_stack_args (env, caller, nargs, func_name, btf, args); + ret =check_outgoing_stack_args (env, caller, proto_slots, func_name, btf, + proto_slots == nargs ? args :NULL );if (ret)return ret;/* Check that BTF function arguments match actual types that the * verifier sees. */ -for (i =0 ; i < nargs; i++) { -struct bpf_reg_state *reg =get_func_arg_reg (caller, regs, i); +for (i =0 , slots_used =0 ; i < nargs; + i++, prev_slot = slots_used, slots_used += nslots) { +struct bpf_reg_state *reg =get_func_arg_reg (caller, regs, slots_used);const struct btf_type *t, *ref_t, *resolve_ret;enum bpf_arg_type arg_type =ARG_DONTCARE ; -argno_t argno =argno_from_arg (i +1 ); +argno_t argno =argno_from_arg (slots_used +1 );int regno =reg_from_argno (argno);bool btf_id_fixed_off_ok = true;u32 ref_id = args[i].type, type_size;int kf_arg_type = meta->fn->arg_type[i]; + t =btf_type_skip_modifiers (btf, args[i].type,NULL ); + nslots =kfunc_arg_slots (t); +if (is_kfunc_arg_prog_aux (btf, &args[i])) {/* Reject repeated use bpf_prog_aux */ if (meta->arg_prog) {
@@ -12939,8 +13005,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_meif (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))continue ; - t =btf_type_skip_modifiers (btf, args[i].type,NULL ); -if (btf_type_is_ptr (t)) { ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id); ref_tname =btf_name_by_offset (btf, ref_t->name_off);
@@ -12991,6 +13055,27 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname =btf_name_by_offset (btf, ref_t->name_off); } +/* +* The first register is checked in below KF_ARG_ANYTHING. +* The rest of it has to be a scalar. +*/ +for (k =1 ; k < nslots; k++) { +argno_t hi_argno =argno_from_arg (slots_used + k +1 ); +struct bpf_reg_state *hi =get_func_arg_reg (caller, regs, slots_used + k); + +if (hi->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a scalar\n" ,reg_arg_name (env, hi_argno)); +bpf_diag_call_arg_fmt (env, insn_idx, hi_argno, func_name, +"Pass an integer scalar value for this " +"argument, not a pointer or resource object." , +"the kfunc expects an integer scalar, " +"but %s is %s" , +reg_arg_name (env, hi_argno), +bpf_diag_reg_type_plain (env, hi->type)); +return -EINVAL ; + } + } +switch (base_type (kf_arg_type)) {case KF_ARG_CONST :case KF_ARG_CONST_MEM_SIZE :
@@ -13408,9 +13493,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me fallthrough;case KF_ARG_MEM_SIZE : { -struct bpf_reg_state *buff_reg =get_func_arg_reg (caller, regs, i -1 ); +struct bpf_reg_state *buff_reg =get_func_arg_reg (caller, regs, prev_slot);struct bpf_reg_state *size_reg = reg; -argno_t buff_argno =argno_from_arg (i); +argno_t buff_argno =argno_from_arg (prev_slot +1 );enum bpf_mem_size_failure failure;if (reg->type !=SCALAR_VALUE ) {
@@ -13756,7 +13841,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *const struct btf_param *args;const struct btf_type *t, *ref_t;const struct btf *btf; -u32 nargs, type_size; +u32 i, slot, nargs, type_size;s64 size;if (bpf_fetch_kfunc_arg_meta (env, insn->imm, insn->off, &meta) <0 )
@@ -13765,23 +13850,32 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn * btf = meta.btf; args =btf_params (meta.func_proto); nargs =btf_type_vlen (meta.func_proto); -if (arg >= nargs) + +/* +* @arg is an argument slot and a 16-byte parameter takes two of them, +* so walk the parameters to find the one that starts at this slot. A +* slot holding the upper eightbyte of such a parameter belongs to no +* pointer, and neither does a slot past the last parameter. +*/ +for (i =0 , slot =0 ; i < nargs && slot < arg; i++) + slot +=kfunc_arg_slots (btf_type_skip_modifiers (btf, args[i].type,NULL )); +if (i >= nargs || slot != arg)return 0 ; - t =btf_type_skip_modifiers (btf, args[arg].type,NULL ); + t =btf_type_skip_modifiers (btf, args[i].type,NULL );if (!btf_type_is_ptr (t))return 0 ;/* dynptr: fixed 16-byte on-stack representation */ -if (is_kfunc_arg_dynptr (btf, &args[arg])) { +if (is_kfunc_arg_dynptr (btf, &args[i])) { size =BPF_DYNPTR_SIZE ; goto out; }/* ptr + __sz/__szk pair: size is in the next register */ -if (arg +1 < nargs && - (btf_param_match_suffix (btf, &args[arg +1 ],"__sz" ) || -btf_param_match_suffix (btf, &args[arg +1 ],"__szk" ))) { +if (i +1 < nargs && + (btf_param_match_suffix (btf, &args[i +1 ],"__sz" ) || +btf_param_match_suffix (btf, &args[i +1 ],"__szk" ))) {int size_reg =BPF_REG_1 + arg +1 ;if (aux->const_reg_mask &BIT (size_reg)) {
@@ -13803,7 +13897,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn */* KF_ITER_NEW kfuncs initialize the iterator state at arg 0 */ if (arg ==0 && meta.kfunc_flags &KF_ITER_NEW )return -size; -if (is_kfunc_arg_uninit (btf, &args[arg])) +if (is_kfunc_arg_uninit (btf, &args[i]))return -size;return size; }
@@ -13996,7 +14090,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,struct bpf_insn_aux_data *insn_aux;const char *operation;int err, insn_idx = *insn_idx_p; -u32 i, nargs, ptr_type_id, ret_nregs =1 ; +u32 i, proto_slots, ptr_type_id, ret_nregs =1 ;struct bpf_kfunc_desc *desc;struct btf *desc_btf;int id;
@@ -14422,11 +14516,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (bpf_is_kfunc_pkt_changing (&meta))clear_all_pkt_pointers (env); - nargs =btf_type_vlen (meta.func_proto); -if (nargs >MAX_BPF_FUNC_REG_ARGS ) { + proto_slots =kfunc_proto_slots (desc_btf, meta.func_proto); +if (proto_slots >MAX_BPF_FUNC_REG_ARGS ) {struct bpf_func_state *caller =cur_func (env);struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno]; -u16 out_stack_arg_cnt = nargs -MAX_BPF_FUNC_REG_ARGS ; +u16 out_stack_arg_cnt = proto_slots -MAX_BPF_FUNC_REG_ARGS ;u16 stack_arg_cnt =bpf_in_stack_arg_cnt (caller_info) + out_stack_arg_cnt;if (stack_arg_cnt > caller_info->stack_arg_cnt)
@@ -17993,7 +18087,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,if (err <0 )/* error would be reported later */ return false; - cs->arg_slot_cnt =btf_type_vlen (meta.func_proto); + cs->arg_slot_cnt =kfunc_proto_slots (meta.btf, meta.func_proto); cs->fastcall = meta.kfunc_flags &KF_FASTCALL ; cs->is_void =btf_type_is_void (btf_type_by_id (meta.btf, meta.func_proto->type));return true;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 06/12] bpf: Recognize by-value struct and __int128 kfunc arguments
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] bpf_kfunc_stack_access_bytes incorrectly maps stack slots to BPF_REG_6 through BPF_REG_9 (or higher), causing an out-of-bounds read and bypassing uninitialized memory checks. --
commit 7f4b25d0dad6ba6fed374b3111fcc89d67434d87
Author: Yonghong Song <[email protected]>
bpf: Recognize by-value struct and __int128 kfunc arguments
This commit modifies the BPF verifier to support kfuncs taking structs or int128 by value. It updates argument tracking to walk slots rather than just registers, rejecting multi-slot arguments until JIT support is fully added.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 39 lines
kernel/bpf/verifier.c +15
-6@@ -13765,23 +13850,32 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn * btf = meta.btf; args =btf_params (meta.func_proto); nargs =btf_type_vlen (meta.func_proto); -if (arg >= nargs) + +/* +* @arg is an argument slot and a 16-byte parameter takes two of them, +* so walk the parameters to find the one that starts at this slot. A +* slot holding the upper eightbyte of such a parameter belongs to no +* pointer, and neither does a slot past the last parameter. +*/ +for (i =0 , slot =0 ; i < nargs && slot < arg; i++) + slot +=kfunc_arg_slots (btf_type_skip_modifiers (btf, args[i].type,NULL )); +if (i >= nargs || slot != arg)return 0 ; - t =btf_type_skip_modifiers (btf, args[arg].type,NULL ); + t =btf_type_skip_modifiers (btf, args[i].type,NULL );if (!btf_type_is_ptr (t))return 0 ;/* dynptr: fixed 16-byte on-stack representation */ -if (is_kfunc_arg_dynptr (btf, &args[arg])) { +if (is_kfunc_arg_dynptr (btf, &args[i])) { size =BPF_DYNPTR_SIZE ; goto out; }/* ptr + __sz/__szk pair: size is in the next register */ -if (arg +1 < nargs && - (btf_param_match_suffix (btf, &args[arg +1 ],"__sz" ) || -btf_param_match_suffix (btf, &args[arg +1 ],"__szk" ))) { +if (i +1 < nargs && + (btf_param_match_suffix (btf, &args[i +1 ],"__sz" ) || +btf_param_match_suffix (btf, &args[i +1 ],"__szk" ))) {int size_reg =BPF_REG_1 + arg +1 ;if (aux->const_reg_mask &BIT (size_reg)) {
Quoted 1 line
size = (s64)aux->const_reg_vals[size_reg];
[Severity: High]
This is a pre-existing issue, but does this code lead to an out-of-bounds read or bypass uninitialized memory checks when the pointer argument is passed on the stack?
The size_reg calculation inherently assumes the size argument is always passed in a register. If the pointer argument slot is 4 or higher (which might be more common now that large by-value structs consume multiple slots), the size argument lands on the stack.
In that scenario, size_reg maps to BPF_REG_6 or higher, which are callee-saved registers. A BPF program could place a fake constant in one of these registers to trick the verifier into thinking 0 bytes are read, while the runtime stack argument is non-zero. This could bypass stack initialization checks and potentially leak uninitialized kernel memory.
Furthermore, if the pointer argument slot is 8 or higher, size_reg becomes greater than or equal to 10. Checking BIT(size_reg) and reading from aux->const_reg_vals[size_reg] would result in an out-of-bounds array read, since const_reg_vals is sized for 10 registers. Should this logic be updated to correctly query the BPF stack when the size argument is passed there?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Yonghong Song
reviewerRe: [PATCH bpf-next v2 06/12] bpf: Recognize by-value struct and __int128 kfunc arguments
Quoted 77 lines · On 9/8/26 11:46 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High]bpf_kfunc_stack_access_bytesincorrectly maps stack slots toBPF_REG_6throughBPF_REG_9(or higher), causing an out-of-bounds read and bypassing uninitialized memory checks. --commit
7f4b25d0dad6ba6fed374b3111fcc89d67434d87
Author: Yonghong Song <[email protected]>bpf: Recognize by-value struct and
__int128kfunc argumentsThis commit modifies the BPF verifier to support kfuncs taking structs or int128 by value. It updates argument tracking to walk slots rather than just registers, rejecting multi-slot arguments until JIT support is fully added.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 39 lines
kernel/bpf/verifier.c +15
-6@@ -13765,23 +13850,32 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn * btf = meta.btf; args =btf_params (meta.func_proto); nargs =btf_type_vlen (meta.func_proto); -if (arg >= nargs) + +/* +* @arg is an argument slot and a 16-byte parameter takes two of them, +* so walk the parameters to find the one that starts at this slot. A +* slot holding the upper eightbyte of such a parameter belongs to no +* pointer, and neither does a slot past the last parameter. +*/ +for (i =0 , slot =0 ; i < nargs && slot < arg; i++) + slot +=kfunc_arg_slots (btf_type_skip_modifiers (btf, args[i].type,NULL )); +if (i >= nargs || slot != arg)return 0 ; - t =btf_type_skip_modifiers (btf, args[arg].type,NULL ); + t =btf_type_skip_modifiers (btf, args[i].type,NULL );if (!btf_type_is_ptr (t))return 0 ;/* dynptr: fixed 16-byte on-stack representation */ -if (is_kfunc_arg_dynptr (btf, &args[arg])) { +if (is_kfunc_arg_dynptr (btf, &args[i])) { size =BPF_DYNPTR_SIZE ; goto out; }/* ptr + __sz/__szk pair: size is in the next register */ -if (arg +1 < nargs && - (btf_param_match_suffix (btf, &args[arg +1 ],"__sz" ) || -btf_param_match_suffix (btf, &args[arg +1 ],"__szk" ))) { +if (i +1 < nargs && + (btf_param_match_suffix (btf, &args[i +1 ],"__sz" ) || +btf_param_match_suffix (btf, &args[i +1 ],"__szk" ))) {int size_reg =BPF_REG_1 + arg +1 ;if (aux->const_reg_mask &BIT (size_reg)) {Quoted 1 line
size = (s64)aux->const_reg_vals[size_reg];[Severity: High]
This is a pre-existing issue, but does this code lead to an out-of-bounds read or bypass uninitialized memory checks when the pointer argument is passed on the stack?The
size_regcalculation inherently assumes the size argument is always passed in a register. If the pointer argument slot is 4 or higher (which might be more common now that large by-value structs consume multiple slots), the size argument lands on the stack.In that scenario,
size_regmaps toBPF_REG_6or higher, which are callee-saved registers. A BPF program could place a fake constant in one of these registers to trick the verifier into thinking 0 bytes are read, while the runtime stack argument is non-zero. This could bypass stack initialization checks and potentially leak uninitialized kernel memory.
Quoted 5 lines
Furthermore, if the pointer argument slot is 8 or higher,
size_regbecomes greater than or equal to 10. CheckingBIT(size_reg)and reading from aux->const_reg_vals[size_reg] would result in an out-of-bounds array read, sinceconst_reg_valsis sized for 10 registers. Should this logic be updated to correctly query the BPF stack when the size argument is passed there?
You are right. Will fix.
Quoted 1 line
Yonghong Song
reviewerPATCH v2 7/12 bpf: Prepare kfunc arguments for the JIT from an ABI description
In reply to Yonghong SongThe previous patch refuses a kfunc argument of more than one eightbyte. This patch allows up to 16 byte kfunc arguments.
But different architectures have different ways to map bpf calling convention (no gap, no backfill) to native convention. Rather than have each arch open-code where it wants an argument, describe the following common parameters where each architecture can set their specific items:
struct bpf_jit_arg_abi {
u8 nr_arg_regs;
bool even_reg_align;
bool even_stack_align;
bool split_at_boundary;
bool backfill_after_stack;
};The above four booleans cover x86-64, arm64, RISC-V LP64, PowerPC64 ELFv2 etc. The bpf_jit_place_args() and bpf_jit_plan_arg_moves() utilizes the above information to do proper work to be used in JIT later on.
--- include/linux/bpf.h | 8 +++ include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 32 ++++++++++++ kernel/bpf/btf.c | 3 ++ kernel/bpf/core.c | 94 ++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++++----- 6 files changed, 223 insertions(+), 13 deletions(-)
include/linux/bpf.h +8 -0
@@ -1194,6 +1194,9 @@ struct bpf_prog_offload {u32 jited_len; }; +/* The argument is aligned to 16 bytes. */ +#define BTF_FMODEL_ALIGN16_ARG BIT(0) +/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1)
@@ -1211,6 +1214,11 @@ struct btf_func_model {u8 arg_flags[MAX_BPF_FUNC_ARGS ]; }; +static inline u32 btf_func_model_arg_slots (const struct btf_func_model *m,u32 arg) +{ +return (m->arg_size[arg] +sizeof (u64) -1 ) /sizeof (u64); +} +/* Restore arguments before returning from trampoline to let original function * continue executing. This flag is used for fentry progs when there are no * fexit progs.
include/linux/bpf_verifier.h +1 -0
@@ -1512,6 +1512,7 @@ enum btf_member_kind {bool btf_struct_is_composed_of (struct bpf_verifier_env *env,const struct btf *btf,const struct btf_type *t,u32 member_kinds); +u32 btf_func_arg_align (const struct btf *btf,const struct btf_type *t);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn);
include/linux/filter.h +32 -0
@@ -1248,6 +1248,38 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);bool bpf_jit_supports_private_stack (void );bool bpf_jit_supports_timed_may_goto (void );bool bpf_jit_supports_fsession (void ); + +struct bpf_jit_arg_abi { +/* Argument registers of the kernel convention. */ +u8 nr_arg_regs; +/* Round the register number up to an even one for 16-byte alignment. */ +bool even_reg_align; +/* Round the stack slot up to an even one for 16-byte alignment. */ +bool even_stack_align; +/* An argument may straddle the last register and the stack. */ +bool split_at_boundary; +/* A later argument may reuse a register a stack-passed one skipped. */ +bool backfill_after_stack; +}; + +const struct bpf_jit_arg_abi *bpf_jit_arg_abi (void ); +u32 bpf_jit_place_args (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm,u8 *pos_of_slot); + +/* The JIT's scratch register, in place of an argument slot. */ +#define BPF_JIT_ARG_TMP 0xff + +/* Every argument slot moves at most once, and the scratch goes out and back. */ +#define BPF_JIT_MAX_ARG_MOVES (MAX_BPF_FUNC_ARGS + 2) + +struct bpf_jit_arg_move { +u8 dst; +u8 src; +}; + +u32 bpf_jit_plan_arg_moves (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm, +struct bpf_jit_arg_move *moves);u64 bpf_arch_uaddress_limit (void );void arch_bpf_stack_walk (bool (*consume_fn)(void *cookie,u64 ip,u64 sp,u64 bp),void *cookie);u64 arch_bpf_timed_may_goto (void );
kernel/bpf/btf.c +3 -0
@@ -7579,6 +7579,9 @@ static u8 __get_arg_fmodel_flags(const struct btf *btf, {u8 flags =__get_type_fmodel_flags (t); +if (btf_func_arg_align (btf, t) >sizeof (u64)) + flags |=BTF_FMODEL_ALIGN16_ARG ; +if (btf_param_match_suffix (btf, arg,"__arena__nullable" )) flags |=BTF_FMODEL_ARENA_ARG |BTF_FMODEL_NULLABLE_ARG ;else if (btf_param_match_suffix (btf, arg,"__arena" ))
kernel/bpf/core.c +94 -0
100 lines not on this page · raw message
kernel/bpf/verifier.c +85 -13
139 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 07/12] bpf: Prepare kfunc arguments for the JIT from an ABI description
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Stack buffer overflow in `bpf_jit_plan_arg_moves` and `kfunc_abi_slots` due to undersized arrays…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Stack buffer overflow in bpf_jit_plan_arg_moves and kfunc_abi_slots due to undersized arrays based on maximum arguments rather than maximum slots. --
commit 180655f49c406c35435ba5ebc05f2900a3c56656
Author: Yonghong Song <[email protected]>
bpf: Prepare kfunc arguments for the JIT from an ABI description
The previous patch refuses a kfunc argument of more than one eightbyte. This patch allows up to 16 byte kfunc arguments.
Quoted 4 lines
include/linux/filter.h +0
-0
[ ... ]
include/linux/filter.h bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena); · line 1253
Quoted 7 lines
include/linux/filter.h +6
-0@@ -1248,6 +1248,38 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena); + +/* The JIT's scratch register, in place of an argument slot. */ +#define BPF_JIT_ARG_TMP 0xff + +/* Every argument slot moves at most once, and the scratch goes out and back. */ +#define BPF_JIT_MAX_ARG_MOVES (MAX_BPF_FUNC_ARGS + 2)
[Severity: Medium]
Since arguments can now take up to 2 slots (for 16-byte arguments), could the maximum number of slot moves actually be up to 26 (24 slots + 2) instead of 14?
Quoted 5 lines
include/linux/filter.h +5
-0+ +struct bpf_jit_arg_move { +u8 dst; +u8 src; +};
[ ... ]
Quoted 4 lines
kernel/bpf/core.c +0
-0
[ ... ]
Quoted 13 lines
kernel/bpf/core.c +12
-0@@ -3287,6 +3287,100 @@ bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) + +u32 bpf_jit_place_args (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm,u8 *pos_of_slot) +{ +u32 i, k, nslots, slot =0 , nregs_used =0 , stack_off =0 ; +bool on_stack = false; + +for (i =0 ; i < fm->nr_args; i++) { +bool align16 = fm->arg_flags[i] &BTF_FMODEL_ALIGN16_ARG ; +u32 pos; + + nslots =btf_func_model_arg_slots (fm, i);
[ ... ]
kernel/bpf/core.c bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) · line 3336
Quoted 2 lines
kernel/bpf/core.c +2
-0+for (k =0 ; k < nslots; k++) + pos_of_slot[slot + k] = pos + k;
[Severity: Medium]
Will this overflow the pos_of_slot array if an architecture adopts this API for BPF trampolines?
The callers currently size the array based on MAX_BPF_FUNC_ARGS (12), but slot + k can reach up to 23 because some arguments can require multiple slots.
kernel/bpf/core.c bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) · line 3347
Quoted 11 lines
kernel/bpf/core.c +11
-0+ slot += nslots; + } + +return slot; +} + +u32 bpf_jit_plan_arg_moves (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm, +struct bpf_jit_arg_move *moves) +{ +u8 pos_of_slot[MAX_BPF_FUNC_ARGS ];
[Severity: Medium]
Does this stack array need to be sized for the maximum number of slots (e.g., MAX_BPF_FUNC_ARGS * 2) instead of the maximum number of arguments?
Quoted 3 lines
kernel/bpf/core.c +3
-0+u32 nslots, n =0 , s, back; + + nslots =bpf_jit_place_args (abi, fm, pos_of_slot);
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c static u32 kfunc_proto_slots(const struct btf *btf, const struct btf_type *func_ · line 12178
Quoted 8 lines
kernel/bpf/verifier.c +4
-0@@ -12172,6 +12172,58 @@ static u32 kfunc_proto_slots(const struct btf *btf, const struct btf_type *func_return slots_used; } +static u32 kfunc_abi_slots (const struct btf_func_model *fm) +{ +const struct bpf_jit_arg_abi *abi =bpf_jit_arg_abi (); +u8 pos_of_slot[MAX_BPF_FUNC_ARGS ];
[Severity: Medium]
Similarly, does this stack array also need to be sized to accommodate up to 24 slots rather than 12 arguments to prevent an out-of-bounds write in bpf_jit_place_args()?
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+u32 i, nslots, slots =0 ; + +for (i =0 ; i < fm->nr_args; i++) + slots +=btf_func_model_arg_slots (fm, i); + +if (!abi) +return slots; + + nslots =bpf_jit_place_args (abi, fm, pos_of_slot);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Yonghong Song
reviewerRe: [PATCH bpf-next v2 07/12] bpf: Prepare kfunc arguments for the JIT from an ABI description
Quoted 17 lines · On 9/8/26 11:46 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Stack buffer overflow inbpf_jit_plan_arg_movesandkfunc_abi_slotsdue to undersized arrays based on maximum arguments rather than maximum slots. --commit
180655f49c406c35435ba5ebc05f2900a3c56656
Author: Yonghong Song <[email protected]>bpf: Prepare kfunc arguments for the JIT from an ABI description
The previous patch refuses a kfunc argument of more than one eightbyte. This patch allows up to 16 byte kfunc arguments.
Quoted 4 lines
include/linux/filter.h +0
-0[ ... ]
Quoted 10 lines
include/linux/filter.h bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena); · line 1253
Quoted 7 lines
include/linux/filter.h +6
-0@@ -1248,6 +1248,38 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena); + +/* The JIT's scratch register, in place of an argument slot. */ +#define BPF_JIT_ARG_TMP 0xff + +/* Every argument slot moves at most once, and the scratch goes out and back. */ +#define BPF_JIT_MAX_ARG_MOVES (MAX_BPF_FUNC_ARGS + 2)[Severity: Medium]
Since arguments can now take up to 2 slots (for 16-byte arguments), could the maximum number of slot moves actually be up to 26 (24 slots + 2) instead of 14?
In jit, for x86_64, we have
/* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6
* are passed through regs, the remains are through stack.
*/
if (nr_regs > MAX_BPF_FUNC_ARGS)
return -ENOTSUPP;So the maximum number of regs is 12, so we are okay for x86_64.
But arm64 is different. It has
static int calc_arg_aux(const struct btf_func_model *m,
struct arg_aux *a)
{
int stack_slots, nregs, slots, i;
/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
for (i = 0, nregs = 0; i < m->nr_args; i++) {
slots = (m->arg_size[i] + 7) / 8;
if (nregs + slots <= 8) /* passed through register ? */
nregs += slots;
else
break;
}
...
}So the maximum number of reg arguments can be 24. I think I will add a patch similar to x86_64 above.
Quoted 26 lines
Quoted 6 lines
include/linux/filter.h +5
-0+ +struct bpf_jit_arg_move { +u8 dst; +u8 src; +};[ ... ]
Quoted 4 lines
kernel/bpf/core.c +0
-0[ ... ]
Quoted 13 lines
kernel/bpf/core.c +12
-0@@ -3287,6 +3287,100 @@ bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) + +u32 bpf_jit_place_args (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm,u8 *pos_of_slot) +{ +u32 i, k, nslots, slot =0 , nregs_used =0 , stack_off =0 ; +bool on_stack = false; + +for (i =0 ; i < fm->nr_args; i++) { +bool align16 = fm->arg_flags[i] &BTF_FMODEL_ALIGN16_ARG ; +u32 pos; + + nslots =btf_func_model_arg_slots (fm, i);[ ... ]
Quoted 8 lines
kernel/bpf/core.c bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) · line 3336
Quoted 2 lines
kernel/bpf/core.c +2
-0+for (k =0 ; k < nslots; k++) + pos_of_slot[slot + k] = pos + k;[Severity: Medium]
Will this overflow thepos_of_slotarray if an architecture adopts this API for BPF trampolines?The callers currently size the array based on
MAX_BPF_FUNC_ARGS(12), but slot + k can reach up to 23 because some arguments can require multiple slots.
I will add a check to ensure maximum reg arguments is 12.
Quoted 15 lines
kernel/bpf/core.c bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) · line 3347
Quoted 12 lines
kernel/bpf/core.c +11
-0+ slot += nslots; + } + +return slot; +} + +u32 bpf_jit_plan_arg_moves (const struct bpf_jit_arg_abi *abi, +const struct btf_func_model *fm, +struct bpf_jit_arg_move *moves) +{ +u8 pos_of_slot[MAX_BPF_FUNC_ARGS ];[Severity: Medium]
Does this stack array need to be sized for the maximum number of slots (e.g.,MAX_BPF_FUNC_ARGS* 2) instead of the maximum number of arguments?
This is for kfunc. I will check whether we need to do
MAX_BPF_FUNC_ARGS * 2
or not.
Quoted 14 lines
Quoted 4 lines
kernel/bpf/core.c +3
-0+u32 nslots, n =0 , s, back; + + nslots =bpf_jit_place_args (abi, fm, pos_of_slot);[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -12172,6 +12172,58 @@ static u32 kfunc_proto_slots(const struct btf *btf, const struct btf_type *func_return slots_used; }
Quoted 8 lines
kernel/bpf/verifier.c static u32 kfunc_proto_slots(const struct btf *btf, const struct btf_type *func_ · line 12178
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+static u32 kfunc_abi_slots (const struct btf_func_model *fm) +{ +const struct bpf_jit_arg_abi *abi =bpf_jit_arg_abi (); +u8 pos_of_slot[MAX_BPF_FUNC_ARGS ];[Severity: Medium]
Similarly, does this stack array also need to be sized to accommodate up to 24 slots rather than 12 arguments to prevent an out-of-bounds write inbpf_jit_place_args()?
Yes, I will check whether we need to have 24 slots or not.
Quoted 10 lines
Quoted 10 lines
kernel/bpf/verifier.c +9
-0+u32 i, nslots, slots =0 ; + +for (i =0 ; i < fm->nr_args; i++) + slots +=btf_func_model_arg_slots (fm, i); + +if (!abi) +return slots; + + nslots =bpf_jit_place_args (abi, fm, pos_of_slot);
Yonghong Song
reviewerPATCH v2 8/12 bpf, x86: Move kfunc arguments into the x86-64 calling convention
In reply to Yonghong SongDo proper move from bpf calling convention to x86_64 calling convention to satisfy native requirement.
In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.
--- arch/x86/net/bpf_jit_comp.c | 73 +++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +70 -3
@@ -1839,6 +1839,60 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,return 0 ; } +static const struct bpf_jit_arg_abi x86_arg_abi = { + .nr_arg_regs =6 , + .backfill_after_stack = true, + .even_stack_align = true, +}; + +static const u8 x86_arg_reg[] = { +BPF_REG_1 ,BPF_REG_2 ,BPF_REG_3 ,BPF_REG_4 ,BPF_REG_5 ,X86_REG_R9 , +}; + +/* +* Move the arguments the x86-64 ABI places somewhere other than the argument +* slot the BPF calling convention gave them. @stack_base addresses the +* outgoing stack argument area from RBP. Return the number of emitted bytes. +*/ +static int emit_kfunc_arg_moves (const struct btf_func_model *fm,s32 stack_base,u8 **pprog) +{ +struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES ]; +const u8 nreg = x86_arg_abi.nr_arg_regs; +u8 *prog = *pprog, *start = prog; +u32 i, n; + + n =bpf_jit_plan_arg_moves (&x86_arg_abi, fm, moves); + +for (i =0 ; i < n; i++) { +u8 dst = moves[i].dst, src = moves[i].src, reg; +bool dst_mem = dst !=BPF_JIT_ARG_TMP && dst >= nreg; +bool src_mem = src !=BPF_JIT_ARG_TMP && src >= nreg; + +/* Take the value into a register. */ +if (src ==BPF_JIT_ARG_TMP ) { + reg =AUX_REG ; + }else if (src_mem) { + reg = dst_mem || dst ==BPF_JIT_ARG_TMP ?BPF_REG_AX : x86_arg_reg[dst]; +emit_ldx (&prog,BPF_DW , reg,BPF_REG_FP , + stack_base + (src - nreg) *8 ); + }else { + reg = x86_arg_reg[src]; + } + +/* And leave it where the argument belongs. */ +if (dst ==BPF_JIT_ARG_TMP ) +emit_mov_reg (&prog, true,AUX_REG , reg); +else if (dst_mem) +emit_stx (&prog,BPF_DW ,BPF_REG_FP , reg, + stack_base + (dst - nreg) *8 ); +else if (reg != x86_arg_reg[dst]) +emit_mov_reg (&prog, true, x86_arg_reg[dst], reg); + } + + *pprog = prog; +return prog - start; +} +/* * Rebase the __arena args of a kfunc call to arena kernel addresses, * rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
@@ -1850,11 +1904,17 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog, {u8 *prog = *pprog;u8 *start = prog; -int i; +int i, slot; -for (i =0 ; i <min_t (int, fm->nr_args,MAX_BPF_FUNC_REG_ARGS ); i++) { +for (i =0 , slot =0 ; i < fm->nr_args; i++) { +u32 arg_regs = (fm->arg_size[i] +7 ) /8 ;u8 flags = fm->arg_flags[i]; -u32 reg =BPF_REG_1 + i; +u32 reg; + +if (slot + arg_regs >MAX_BPF_FUNC_REG_ARGS ) +break ; + reg =BPF_REG_1 + slot; + slot += arg_regs;if (!(flags &BTF_FMODEL_ARENA_ARG ))continue ;
@@ -2837,6 +2897,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *if (err <0 )return err; ip += err; + ip +=emit_kfunc_arg_moves (fm, outgoing_arg_base - + outgoing_rsp, &prog); }if (priv_frame_ptr) {push_r9 (&prog);
@@ -4351,6 +4413,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)return true; } +const struct bpf_jit_arg_abi *bpf_jit_arg_abi (void ) +{ +return &x86_arg_abi; +} +bool bpf_jit_supports_stack_args (void ) {return true;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 08/12] bpf, x86: Move kfunc arguments into the x86-64 calling convention
This looks like a bug fix for multi-slot kfunc argument handling. The fix corrects emit_kfunc_arena_args() to count eightbytes (register slots) instead of parameter indices when walking arena…
Quoted 6 lines
commit
7d2a9d78dc93fad535592157230f508d5bc54a7cAuthor: Yonghong Song <[email protected]> Date: Tue Sep 8 23:26:04 2026 -0700bpf, x86: Move kfunc arguments into the x86-64 calling convention
Quoted 7 lines
Do proper move from bpf calling convention to
x86_64calling convention to satisfy native requirement.In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.
Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for multi-slot kfunc argument handling. The fix corrects emit_kfunc_arena_args() to count eightbytes (register slots) instead of parameter indices when walking arena arguments, which would cause incorrect register assignment when kfunc arguments take multiple registers (such as __int128 or large structs).
Should this include a Fixes: tag for the commit that introduced emit_kfunc_arena_args() with the parameter-index bug?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 08/12] bpf, x86: Move kfunc arguments into the x86-64 calling convention
Quoted 19 lines · On 9/9/26 12:29 AM, [email protected] wrote:
Quoted 6 lines
commit
7d2a9d78dc93fad535592157230f508d5bc54a7cAuthor: Yonghong Song <[email protected]> Date: Tue Sep 8 23:26:04 2026 -0700bpf, x86: Move kfunc arguments into the x86-64 calling convention
Quoted 7 lines
Do proper move from bpf calling convention to
x86_64calling convention to satisfy native requirement.In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers.
Signed-off-by Yonghong Song <[email protected]>This looks like a bug fix for multi-slot kfunc argument handling. The fix corrects
emit_kfunc_arena_args()to count eightbytes (register slots) instead of parameter indices when walking arena arguments, which would cause incorrect register assignment when kfunc arguments take multiple registers (such as__int128or large structs).
Quoted 4 lines
Should this include a Fixes: tag for the commit that introduced
emit_kfunc_arena_args()with the parameter-index bug?Fixes 34c3503cb29a ("bpf: Wire up JIT support for 16-byte kfunc returns")
There is no need to have this Fixes tag as it is in the same series.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 9/12 bpf, arm64: Move kfunc arguments into the arm64 calling convention
In reply to Yonghong SongDo proper move from bpf calling convention to arm64 calling convention to satisfy native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.
In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers. The walk takes the func model from the caller now, as the moves need it too, and runs first so that they carry the rebased value.
--- arch/arm64/net/bpf_jit_comp.c | 77 ++++++++++++++++++++++++++++++----- 1 file changed, 67 insertions(+), 10 deletions(-)
arch/arm64/net/bpf_jit_comp.c +67 -10
@@ -1220,6 +1220,12 @@ static int add_exception_handler(const struct bpf_insn *insn,return 0 ; } +static const struct bpf_jit_arg_abi arm64_arg_abi = { + .nr_arg_regs =8 , + .even_reg_align = true, + .even_stack_align = true, +}; +static const u8 stack_arg_reg[] = {A64_R (5 ),A64_R (6 ),A64_R (7 ) };#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct* kern_vm_start. A nullable arg preserves NULL by skipping the add, tested * on the truncated value as arena NULL is offset 0. */ -static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct bpf_insn *insn) +static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct btf_func_model *fm) {const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const struct btf_func_model *fm; -int i; - - fm =bpf_jit_find_kfunc_model (ctx->prog, insn); -if (!fm) -return -EINVAL ; +int i, slot; -for (i =0 ; i <min_t (int, fm->nr_args,MAX_BPF_FUNC_REG_ARGS ); i++) { -const u8 reg = bpf2a64[BPF_REG_1 + i]; +for (i =0 , slot =0 ; i < fm->nr_args; i++) { +u32 arg_regs = (fm->arg_size[i] +7 ) /8 ;u8 flags = fm->arg_flags[i]; +u8 reg; + +if (slot + arg_regs >MAX_BPF_FUNC_REG_ARGS ) +break ; + reg = bpf2a64[BPF_REG_1 + slot]; + slot += arg_regs;if (!(flags &BTF_FMODEL_ARENA_ARG ))continue ;
@@ -1293,6 +1300,45 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insreturn 0 ; } +static bool a64_arg_on_stack (u8 slot) +{ +return slot >= arm64_arg_abi.nr_arg_regs; +} + +static s32 a64_arg_stack_off (u8 slot) +{ +return (slot - arm64_arg_abi.nr_arg_regs) *sizeof (u64); +} + +/* +* AAPCS64 only ever moves an argument to a higher slot, so the planner asks +* for the scratch only to carry an eightbyte from one stack slot to another. +*/ +static void emit_kfunc_arg_moves (struct jit_ctx *ctx,const struct btf_func_model *fm) +{ +struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES ]; +const u8 tmp = bpf2a64[TMP_REG_1 ]; +u32 i, n; + + n =bpf_jit_plan_arg_moves (&arm64_arg_abi, fm, moves); + +for (i =0 ; i < n; i++) { +u8 dst = moves[i].dst, src = moves[i].src, reg; + +if (a64_arg_on_stack (src)) { + reg =a64_arg_on_stack (dst) ? tmp : dst; +emit (A64_LDR64I (reg,A64_SP ,a64_arg_stack_off (src)), ctx); + }else { + reg = src; + } + +if (a64_arg_on_stack (dst)) +emit (A64_STR64I (reg,A64_SP ,a64_arg_stack_off (dst)), ctx); +else if (reg != dst) +emit (A64_MOV (1 , dst, reg), ctx); + } +} +/* JITs an eBPF instruction. * Returns: * 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1716,9 +1762,15 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (ret <0 )return ret;if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) { - ret =emit_kfunc_arena_args (ctx, insn); +const struct btf_func_model *fm; + + fm =bpf_jit_find_kfunc_model (ctx->prog, insn); +if (!fm) +return -EINVAL ; + ret =emit_kfunc_arena_args (ctx, fm);if (ret <0 )return ret; +emit_kfunc_arg_moves (ctx, fm); }emit_call (func_addr, ctx);/*
@@ -2393,6 +2445,11 @@ bool bpf_jit_supports_kfunc_ret_reg_pair(void)return true; } +const struct bpf_jit_arg_abi *bpf_jit_arg_abi (void ) +{ +return &arm64_arg_abi; +} +bool bpf_jit_supports_stack_args (void ) {return true;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 09/12] bpf, arm64: Move kfunc arguments into the arm64 calling convention
This looks like a bug fix for multi-slot kfunc argument support. Should this include:
Quoted 11 lines
commit
7c4f41d1b9c4dfdbada8f6d1b1097849ccb20de8
Author: Yonghong Song <[email protected]>bpf, arm64: Move kfunc arguments into the arm64 calling convention
Do proper move from bpf calling convention to arm64 calling convention to satisfy native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.
Quoted 6 lines
In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers. The walk takes the func model from the caller now, as the moves need it too, and runs first so that they carry the rebased value.
Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for multi-slot kfunc argument support. Should this include:
In v1 of this series, Alexei Starovoitov raised a design concern about the overall approach. He questioned why the BPF calling convention should diverge from x86-64/arm64, noting that both JIT patches doing argument shuffling is a sign that the approach may be wrong on the BPF side. He suggested adjusting the BPF calling convention to match arm64 (the stricter of the two architectures) rather than forcing all JITs to adapt:
"Why cannot we adjust bpf calling convention to match arm64/x86 the best? Since arm64 is stricter, I'd pick that style. There are no kfuncs that use int128 or 16+ byte args, so it's a matter of bpf subprogs calling bpf subprogs. Seems cleaner to adjust what llvm emits instead of forcing all jits to adapt."
The v2 took the opposite approach - keeping the BPF calling convention and implementing JIT argument shuffling in both x86 and arm64. Was this design decision explicitly discussed and agreed upon?
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 30 lines
arch/arm64/net/bpf_jit_comp.c +10
-9@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct* kern_vm_start. A nullable arg preserves NULL by skipping the add, tested * on the truncated value as arena NULL is offset 0. */ -static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct bpf_insn *insn) +static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct btf_func_model *fm) {const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const struct btf_func_model *fm; -int i; - - fm =bpf_jit_find_kfunc_model (ctx->prog, insn); -if (!fm) -return -EINVAL ; +int i, slot; -for (i =0 ; i <min_t (int, fm->nr_args,MAX_BPF_FUNC_REG_ARGS ); i++) { -const u8 reg = bpf2a64[BPF_REG_1 + i]; +for (i =0 , slot =0 ; i < fm->nr_args; i++) { +u32 arg_regs = (fm->arg_size[i] +7 ) /8 ;u8 flags = fm->arg_flags[i]; +u8 reg; + +if (slot + arg_regs >MAX_BPF_FUNC_REG_ARGS ) +break ; + reg = bpf2a64[BPF_REG_1 + slot]; + slot += arg_regs;if (!(flags &BTF_FMODEL_ARENA_ARG ))continue ;
[ ... ]
Quoted 32 lines
arch/arm64/net/bpf_jit_comp.c +28
-0@@ -1293,6 +1300,45 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insreturn 0 ; } +static bool a64_arg_on_stack (u8 slot) +{ +return slot >= arm64_arg_abi.nr_arg_regs; +} + +static s32 a64_arg_stack_off (u8 slot) +{ +return (slot - arm64_arg_abi.nr_arg_regs) *sizeof (u64); +} + +/* +* AAPCS64 only ever moves an argument to a higher slot, so the planner asks +* for the scratch only to carry an eightbyte from one stack slot to another. +*/ +static void emit_kfunc_arg_moves (struct jit_ctx *ctx,const struct btf_func_model *fm) +{ +struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES ]; +const u8 tmp = bpf2a64[TMP_REG_1 ]; +u32 i, n; + + n =bpf_jit_plan_arg_moves (&arm64_arg_abi, fm, moves); + +for (i =0 ; i < n; i++) { +u8 dst = moves[i].dst, src = moves[i].src, reg; + +if (a64_arg_on_stack (src)) { + reg =a64_arg_on_stack (dst) ? tmp : dst; +emit (A64_LDR64I (reg,A64_SP ,a64_arg_stack_off (src)), ctx);
arch/arm64/net/bpf_jit_comp.c static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins · line 1340
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10
-0+ }else { + reg = src; + } + +if (a64_arg_on_stack (dst)) +emit (A64_STR64I (reg,A64_SP ,a64_arg_stack_off (dst)), ctx); +else if (reg != dst) +emit (A64_MOV (1 , dst, reg), ctx); + } +}
This isn't a bug, but would it be clearer to say the plan never needs a scratch on AAPCS64 and that TMP_REG_1 is only this helper's stack-to-stack carrier? As written, the comment reads as though moves[] can contain BPF_JIT_ARG_TMP (which the loop does not handle - unlike the x86 version at arch/x86/net/bpf_jit_comp.c which does), and the ': dst' arm in the conditional looks unreachable for the same reason (src on the stack implies dst on the stack, since dst >= src >= nr_arg_regs).
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 09/12] bpf, arm64: Move kfunc arguments into the arm64 calling convention
Quoted 35 lines · On 9/9/26 12:30 AM, [email protected] wrote:
Quoted 11 lines
commit
7c4f41d1b9c4dfdbada8f6d1b1097849ccb20de8
Author: Yonghong Song <[email protected]>bpf, arm64: Move kfunc arguments into the arm64 calling convention
Do proper move from bpf calling convention to arm64 calling convention to satisfy native requirement. AAPCS64 only ever moves an argument to a higher slot, so the moves need one scratch register to carry an eightbyte from one stack slot to another, and never the one a convention moving an argument down would need.
Quoted 6 lines
In addition, the arena argument walk counts eightbytes rather than parameters, as an argument may take two registers. The walk takes the func model from the caller now, as the moves need it too, and runs first so that they carry the rebased value.
Signed-off-by Yonghong Song <[email protected]>This looks like a bug fix for multi-slot kfunc argument support. Should this include:
Fixes 2092c3247e45 ("bpf: Prepare kfunc arguments for the JIT from an ABI description")In v1 of this series, Alexei Starovoitov raised a design concern about the overall approach. He questioned why the BPF calling convention should diverge from x86-64/arm64, noting that both JIT patches doing argument shuffling is a sign that the approach may be wrong on the BPF side. He suggested adjusting the BPF calling convention to match arm64 (the stricter of the two architectures) rather than forcing all JITs to adapt:
"Why cannot we adjust bpf calling convention to match arm64/x86 the best? Since arm64 is stricter, I'd pick that style. There are no kfuncs that use int128 or 16+ byte args, so it's a matter of bpf subprogs calling bpf subprogs. Seems cleaner to adjust what llvm emits instead of forcing all jits to adapt."
Quoted 3 lines
The v2 took the opposite approach - keeping the BPF calling convention and implementing JIT argument shuffling in both x86 and arm64. Was this design decision explicitly discussed and agreed upon?
Yes, we will keep original bpf calling convention and then jit will do proper adjustment.
Quoted 80 lines
Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 30 lines
arch/arm64/net/bpf_jit_comp.c +10
-9@@ -1262,19 +1268,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct* kern_vm_start. A nullable arg preserves NULL by skipping the add, tested * on the truncated value as arena NULL is offset 0. */ -static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct bpf_insn *insn) +static int emit_kfunc_arena_args (struct jit_ctx *ctx,const struct btf_func_model *fm) {const u8 arena_vm_base = bpf2a64[ARENA_VM_START ]; -const struct btf_func_model *fm; -int i; - - fm =bpf_jit_find_kfunc_model (ctx->prog, insn); -if (!fm) -return -EINVAL ; +int i, slot; -for (i =0 ; i <min_t (int, fm->nr_args,MAX_BPF_FUNC_REG_ARGS ); i++) { -const u8 reg = bpf2a64[BPF_REG_1 + i]; +for (i =0 , slot =0 ; i < fm->nr_args; i++) { +u32 arg_regs = (fm->arg_size[i] +7 ) /8 ;u8 flags = fm->arg_flags[i]; +u8 reg; + +if (slot + arg_regs >MAX_BPF_FUNC_REG_ARGS ) +break ; + reg = bpf2a64[BPF_REG_1 + slot]; + slot += arg_regs;if (!(flags &BTF_FMODEL_ARENA_ARG ))continue ;[ ... ]
Quoted 41 lines
arch/arm64/net/bpf_jit_comp.c +37
-0@@ -1293,6 +1300,45 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insreturn 0 ; } +static bool a64_arg_on_stack (u8 slot) +{ +return slot >= arm64_arg_abi.nr_arg_regs; +} + +static s32 a64_arg_stack_off (u8 slot) +{ +return (slot - arm64_arg_abi.nr_arg_regs) *sizeof (u64); +} + +/* +* AAPCS64 only ever moves an argument to a higher slot, so the planner asks +* for the scratch only to carry an eightbyte from one stack slot to another. +*/ +static void emit_kfunc_arg_moves (struct jit_ctx *ctx,const struct btf_func_model *fm) +{ +struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES ]; +const u8 tmp = bpf2a64[TMP_REG_1 ]; +u32 i, n; + + n =bpf_jit_plan_arg_moves (&arm64_arg_abi, fm, moves); + +for (i =0 ; i < n; i++) { +u8 dst = moves[i].dst, src = moves[i].src, reg; + +if (a64_arg_on_stack (src)) { + reg =a64_arg_on_stack (dst) ? tmp : dst; +emit (A64_LDR64I (reg,A64_SP ,a64_arg_stack_off (src)), ctx); + }else { + reg = src; + } + +if (a64_arg_on_stack (dst)) +emit (A64_STR64I (reg,A64_SP ,a64_arg_stack_off (dst)), ctx); +else if (reg != dst) +emit (A64_MOV (1 , dst, reg), ctx); + }
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins · line 1340
Quoted 1 line
arch/arm64/net/bpf_jit_comp.c +1
-0+}This isn't a bug, but would it be clearer to say the plan never needs a scratch on AAPCS64 and that
TMP_REG_1is only this helper's stack-to-stack carrier? As written, the comment reads as though moves[] can containBPF_JIT_ARG_TMP(which the loop does not handle - unlike the x86 version atarch/x86/net/bpf_jit_comp.cwhich does), and the ': dst' arm in the conditional looks unreachable for the same reason (src on the stack implies dst on the stack, since dst >= src >=nr_arg_regs).
Okay, will fix.
Quoted 9 lines
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 10/12 selftests/bpf: Add C tests for by-value arguments up to 16 bytes
In reply to Yonghong SongExtend the by-value argument test with the aggregate cases, written in C so that they depend on the compiler lowering the argument into a pair of argument registers rather than on a hand-written register layout.
The programs cover a struct and a union that fill two registers, a smaller struct that fills one, and two struct arguments in a row, alongside the __int128 already there. Each has an int argument around it so that a wrong slot count shows up as a wrong value in the parameters beside it; two pairs leave room for only one, which follows them. A global function taking a struct with a pointer member is rejected: the callee would receive the pointer as an opaque scalar.
A struct the argument registers cannot hold reaches the callee partly on the stack, which the interpreter does not implement, so that case is loaded only when the JIT is on.
--- .../selftests/bpf/progs/verifier_int128_arg.c | 165 ++++++++++++++++++ 1 file changed, 165 insertions(+)
tools/testing/selftests/bpf/progs/verifier_int128_arg.c +165 -0
@@ -7,6 +7,131 @@#define MIX_A 0xdeadbeefcafef00dULL#define MIX_B 0x0123456789abcdefULL +struct pair { +__u64 lo; +__u64 hi; +}; + +struct small { +__u32 a; +__u32 b; +}; + +union upair { +__u64 halves[2 ]; +struct { +__u64 lo; +__u64 hi; + } parts; +}; + +struct with_ptr { +void *p; +__u64 x; +}; + +static __noinline __u64 take_pair(int a, struct pair p, int c) +{ +return (__u64 )a + p.lo + p.hi + c; +} + +SEC("tc") +__success __retval(0) +int aggregate_arg_static_struct_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +struct pair p = { .lo = a, .hi = b }; + +if (take_pair (1 , p,2 ) != a + b +3 ) +return 1 ; + +return 0 ; +} + +#if defined(__clang__) + +__noinline __u64 take_pair_global(int a, struct pair p, int c) +{ +return (__u64 )a + p.lo + p.hi + c; +} + +SEC("tc") +__success __retval(0) +int aggregate_arg_global_struct_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +struct pair p = { .lo = a, .hi = b }; + +if (take_pair_global (1 , p,2 ) != a + b +3 ) +return 1 ; + +return 0 ; +} + +__noinline __u64 take_two_pairs_global(struct pair p, struct pair q, int d) +{ +return p.lo + p.hi + q.lo + q.hi + d; +} + +SEC("tc") +__success __retval(0) +int aggregate_arg_two_structs_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +struct pair p = { .lo = a, .hi = b }; +struct pair q = { .lo = a +1 , .hi = b +2 }; + +if (take_two_pairs_global (p, q,3 ) !=2 * a +2 * b +6 ) +return 1 ; + +return 0 ; +} + +__noinline __u64 take_small_global(int a, struct small s, int c) +{ +return (__u64 )a + s.a + s.b + c; +} + +SEC("tc") +__success __retval(0) +int aggregate_arg_small_struct_c_test(struct __sk_buff *skb) +{ +__u32 a = skb->len ^ (__u32 )MIX_A ; +__u32 b = skb->len ^ (__u32 )MIX_B ; +struct small s = { .a = a, .b = b }; + +if (take_small_global (1 , s,2 ) != (__u64 )a + b +3 ) +return 1 ; + +return 0 ; +} + +__noinline __u64 take_upair_global(int a, union upair u, int c) +{ +return (__u64 )a + u.parts.lo + u.parts.hi + c; +} + +SEC("tc") +__success __retval(0) +int aggregate_arg_union_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +union upair u; + + u.halves[0 ] = a; + u.halves[1 ] = b; +if (take_upair_global (1 , u,2 ) != a + b +3 ) +return 1 ; + +return 0 ; +} + +#endif +typedef unsigned __int128 u128 ; __noinline __u64 take_i128_global(int a, u128 v, int c)
@@ -28,4 +153,44 @@ int aggregate_arg_int128_c_test(struct __sk_buff *skb)return 0 ; } +#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +static __noinline __u64 take_spilled_pair(int a, int b, int c, int d, struct pair p) +{ +return (__u64 )a + b + c + d + p.lo + p.hi; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed () +__success__retval (0 ) +int aggregate_arg_spilled_struct_c_test(struct __sk_buff *skb) +{ +__u64 a = skb->len ^MIX_A ; +__u64 b = skb->len ^MIX_B ; +struct pair p = { .lo = a, .hi = b }; +int n = skb->len; + +if (take_spilled_pair (n, n +1 , n +2 , n +3 , p) != a + b +4 * n +6 ) +return 1 ; + +return 0 ; +} + +#endif + +__noinline __u64 take_with_ptr_global(struct with_ptr s) +{ +return s.x; +} + +SEC("tc") +__failure __msg("type STRUCT in take_with_ptr_global() is not composed of scalars") +int aggregate_arg_ptr_member_fail(struct __sk_buff *skb) +{ +struct with_ptr s = { .p = skb, .x = skb->len }; + +return take_with_ptr_global (s); +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 11/12 selftests/bpf: Add inline-asm tests for by-value arguments
In reply to Yonghong SongThe tests cover a struct passed in a register pair, a pointer in one half of it refused at the call site, a struct too large to pass by value, and three placements a global function cannot have: a struct split between the last argument register and the stack, one wholly past the registers, and an __int128 whose two slots push the last parameter out.
GCC passes an aggregate by invisible reference, so a callee it compiles expects a pointer where BTF says the halves of the struct are, and those tests are left to clang.
--- .../selftests/bpf/prog_tests/aggregate_arg.c | 9 + .../selftests/bpf/progs/aggregate_arg_func.c | 155 ++++++++++++++++++ 2 files changed, 164 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_arg.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_func.c
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c new file +9 -0
@@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include "aggregate_arg_func.skel.h" + +void test_aggregate_arg (void ) +{ +RUN_TESTS (aggregate_arg_func); +}
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 11/12] selftests/bpf: Add inline-asm tests for by-value arguments
This isn't a bug, but would it help a later reader to carry the changelog's gcc-invisible-reference rationale into a short comment above the #if defined(__clang__), and to note why the last three…
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c +0
-0
[ ... ]
Quoted 31 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +25
-0@@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128 ; + +struct pair { +__u64 lo; +__u64 hi; +}; + +struct too_big { +__u64 a; +__u64 b; +__u64 c; +}; + +#if defined(__clang__) + +__noinline __u64 global_arg_pair(int a, struct pair p, int c) +{ +return (__u64 )a + p.lo + p.hi + c; +}
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +6
-0+#endif + +__noinline __u64 global_arg_too_big(struct too_big s) +{ +return s.a + s.b + s.c; +}
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +6
-0+#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +__noinline __u64 global_arg_split(int a, int b, int c, int d, struct pair p) +{ +return (__u64 )a + b + c + d + p.lo + p.hi; +}
[ ... ]
Quoted 1 line
+#endif
This isn't a bug, but would it help a later reader to carry the changelog's gcc-invisible-reference rationale into a short comment above the #if defined(__clang__), and to note why the last three cases need __BPF_FEATURE_STACK_ARGUMENT (and maybe label the two #endif closures)?
The sibling file aggregate_ret_func.c documents its equivalent gate:
/*
* gcc returns a by-value struct through a hidden pointer, and emits the
* 'r0 = r1' returning it after the __naked body's exit, leaving the
* subprogram falling through. Build these with clang only. */
#if defined(__clang__)
and other progs files label the closing directive, for example verifier_ldsx.c uses #endif /* __BPF_FEATURE_ADDR_SPACE_CAST */.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 11/12] selftests/bpf: Add inline-asm tests for by-value arguments
Quoted 52 lines · On 9/9/26 12:30 AM, [email protected] wrote:
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c +0
-0[ ... ]
Quoted 31 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +25
-0@@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128 ; + +struct pair { +__u64 lo; +__u64 hi; +}; + +struct too_big { +__u64 a; +__u64 b; +__u64 c; +}; + +#if defined(__clang__) + +__noinline __u64 global_arg_pair(int a, struct pair p, int c) +{ +return (__u64 )a + p.lo + p.hi + c; +}[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +6
-0+#endif + +__noinline __u64 global_arg_too_big(struct too_big s) +{ +return s.a + s.b + s.c; +}[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_func.c new file +6
-0+#if defined(__BPF_FEATURE_STACK_ARGUMENT) + +__noinline __u64 global_arg_split(int a, int b, int c, int d, struct pair p) +{ +return (__u64 )a + b + c + d + p.lo + p.hi; +}[ ... ]
Quoted 5 lines
Quoted 1 line
+#endif
This isn't a bug, but would it help a later reader to carry the changelog's gcc-invisible-reference rationale into a short comment above the #if
defined(__clang__), and to note why the last three cases need__BPF_FEATURE_STACK_ARGUMENT(and maybe label the two #endif closures)?
stack argument is implemented in llvm23, and that is why we want this feature gating.
Quoted 10 lines
The sibling file
aggregate_ret_func.cdocuments its equivalent gate:/*
* gcc returns a by-value struct through a hidden pointer, and emits the
* 'r0 = r1' returning it after the__nakedbody's exit, leaving the
* subprogram falling through. Build these with clang only. */
#ifdefined(__clang__)
Quoted 2 lines
and other progs files label the closing directive, for example
verifier_ldsx.cuses #endif /*__BPF_FEATURE_ADDR_SPACE_CAST*/.
This is fine. We do not want to have this comment in many places. The commit message should explain this.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerPATCH v2 12/12 selftests/bpf: Add tests for by-value kfunc arguments
In reply to Yonghong SongAdd kfuncs taking a 16-byte struct and an __int128 by value, with combinations of <= 8 byte arguments and '> 8 && <= 16' byte arguments, and tests that call them. Each checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly.
Between them the cases cover an argument the two conventions place alike, one arm64 moves between registers, one an ABI moves to or from the stack, and one both ABIs pad the stack for. All of them run on both arches; what differs is how many instructions the JIT emits to get there. Two more cover the rejections that hold everywhere, an aggregate holding a pointer and one too large to pass by value.
test_stack_arg_big() in stack_arg_fail.c passed a 16-byte struct as the sixth argument and asserted the unrecognized stack argument type it used to be reported as. The JIT places that argument now, so the test is removed and the same call is covered by the cases above.
--- .../selftests/bpf/prog_tests/aggregate_arg.c | 2 + .../selftests/bpf/progs/aggregate_arg_kfunc.c | 221 ++++++++++++++++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 16 ++ .../selftests/bpf/progs/stack_arg_fail.c | 10 - .../selftests/bpf/test_kmods/bpf_testmod.c | 77 ++++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 32 +++ 6 files changed, 348 insertions(+), 10 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c
tools/testing/selftests/bpf/prog_tests/aggregate_arg.c +2 -0
@@ -2,8 +2,10 @@/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ #include <test_progs.h> #include "aggregate_arg_func.skel.h" +#include "aggregate_arg_kfunc.skel.h" void test_aggregate_arg (void ) {RUN_TESTS (aggregate_arg_func); +RUN_TESTS (aggregate_arg_kfunc); }
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +221 -0
221 lines not on this page · raw message
tools/testing/selftests/bpf/progs/arena_kfunc.c +16 -0
@@ -228,6 +228,22 @@ int arena_arg_stack(void *ctx)bpf_kfunc_arena_stack_arg_test (1 ,2 ,3 ,4 ,5 , (u64 *)1 );return 0 ; } + +#if defined(__clang__) +/* The struct takes two slots, so the arena pointer is the sixth. */ +SEC("syscall") +__arch_x86_64 +__arch_arm64 +__failure __msg("arena pointer cannot be a stack argument") +int arena_arg_stack_after_pair (void *ctx) +{ +struct prog_test_pair_arg s = { .lo =1 , .hi =2 }; + +bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +bpf_kfunc_call_test_pair_arena_arg (1 ,2 ,3 , s, (u64 *)1 ); +return 0 ; +} +#endif #else SEC("syscall") __arch_x86_64
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0 -10
@@ -3,20 +3,10 @@#include <vmlinux.h> #include <bpf/bpf_helpers.h> -#include "../test_kmods/bpf_testmod_kfunc.h" #include "bpf_misc.h" #if defined(__BPF_FEATURE_STACK_ARGUMENT) -SEC("tc") -__failure __msg("Unrecognized *(R11-8) type STRUCT") -int test_stack_arg_big (struct __sk_buff *skb) -{ -struct prog_test_big_arg s = { .a =1 , .b =2 }; - -return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); -} - SEC("socket") __description("r11 in ALU instruction") __failure __msg("R11 is invalid")
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +77 -0
89 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +32 -0
@@ -61,6 +61,16 @@ struct prog_test_big_arg {__u64 b; }; +struct prog_test_pair_arg {/* 16 bytes: two argument registers */ +__u64 lo; +__u64 hi; +}; + +struct prog_test_ptr_arg {/* 16 bytes, but holds a pointer */ +void *p; +__u64 x; +}; +struct prog_test_ret_pair {/* 16 bytes: R0:R2 */ __u64 lo;__u64 hi;
@@ -221,6 +231,28 @@ __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym; struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym; struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +__u64 bpf_kfunc_call_test_pair_arg(__u64 a, struct prog_test_pair_arg s, __u64 b) __ksym; +#ifdef __SIZEOF_INT128__ +__u64 bpf_kfunc_call_test_i128_arg(__u64 a, __u64 b, __int128 v) __ksym; +__u64 bpf_kfunc_call_test_i128_arg_odd(__u64 a, __int128 v, __u64 b) __ksym; +__u64 bpf_kfunc_call_test_i128_arg_shift(__u64 a, __int128 v, __u64 b, __u64 c, + __u64 d) __ksym; +__u64 bpf_kfunc_call_test_i128_arg_ovf(__u64 a, __int128 v, __u64 b, __u64 c, + __u64 d, __u64 e, __u64 f) __ksym; +__u64 bpf_kfunc_call_test_i128_arg_pad(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + __u64 f, __u64 g, __int128 v) __ksym; +#endif +__u64 bpf_kfunc_call_test_pair_arg_nofit(__u64 a, __u64 b, __u64 c, __u64 d, + struct prog_test_pair_arg s) __ksym; +__u64 bpf_kfunc_call_test_pair_arg_tail(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + struct prog_test_pair_arg s, __u64 f) __ksym; +__u64 bpf_kfunc_call_test_pair_arg_split8(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + __u64 f, __u64 g, + struct prog_test_pair_arg s) __ksym; +__u64 bpf_kfunc_call_test_ptr_arg(struct prog_test_ptr_arg s) __ksym; +__u64 bpf_kfunc_call_test_pair_arena_arg(__u64 a, __u64 b, __u64 c, + struct prog_test_pair_arg s, + __u64 *f__arena) __ksym; struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; struct prog_test_ret_nested bpf_kfunc_call_test_ret_nested(__u64 tag) __ksym; struct prog_test_ret_ptr_arr bpf_kfunc_call_test_ret_ptr_arr(void) __ksym;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 12/12] selftests/bpf: Add tests for by-value kfunc arguments
The changelog describes two new rejection ('__failure') cases, but only one of the two is added by this commit, and a third rejection case that IS added is never mentioned.
Quoted 11 lines
Add kfuncs taking a 16-byte struct and an
__int128by value, with combinations of <= 8 byte arguments and '> 8 && <= 16' byte arguments, and tests that call them. Each checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly.Between them the cases cover an argument the two conventions place alike, one arm64 moves between registers, one an ABI moves to or from the stack, and one both ABIs pad the stack for. All of them run on both arches; what differs is how many instructions the JIT emits to get there. Two more cover the rejections that hold everywhere, an aggregate holding a pointer and one too large to pass by value.
The changelog describes two new rejection ('__failure') cases, but only one of the two is added by this commit, and a third rejection case that IS added is never mentioned.
Looking at the added tests:
1. 'an aggregate holding a pointer' -> present:
progs/aggregate_arg_kfunc.c adds:
SEC("tc") __arch_x86_64 __arch_arm64
__failure __msg("R1 type STRUCT is not composed of scalars") int aggregate_arg_kfunc_ptr_member(struct __sk_buff *skb)
2. 'one too large to pass by value' -> NOT added here:
progs/aggregate_arg_kfunc.c contains exactly one __failure program (the ptr_member one above), and the commit declares no kfunc taking a by-value aggregate larger than 16 bytes in test_kmods/bpf_testmod_kfunc.h, so such a case cannot exist in this commit.
The only test matching that description is aggregate_arg_too_big_fail() in progs/aggregate_arg_func.c with:
__failure __msg("... has size 24, only 1 to 16 bytes can be passed by value")
which was added by the earlier commit b3f328dbfb8a ("selftests/bpf: Add inline-asm tests for by-value arguments") and is untouched here.
3. This commit adds a rejection case the changelog never mentions:
progs/arena_kfunc.c gains 16 lines adding:
/* The struct takes two slots, so the arena pointer is the sixth. */ SEC("syscall") __arch_x86_64 __arch_arm64
__failure __msg("arena pointer cannot be a stack argument") int arena_arg_stack_after_pair(void *ctx)
together with the new bpf_kfunc_call_test_pair_arena_arg() kfunc and its __ksym declaration. arena_kfunc.c is one of the six files in the diffstat but is not described anywhere in the message.
Could the message be updated to say that one new rejection case is added for an aggregate holding a pointer (with the too-large rejection being already covered by aggregate_arg_func.c), and add a sentence for the arena stack-argument rejection in arena_kfunc.c?
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +0
-0
[ ... ]
Quoted 5 lines
+/*
+ * One argument further and the hole pushes the last one off x7 and onto the
+ * arm64 stack, which the JIT does not shift into. The x86-64 ABI packs the
+ * eightbytes, so its last two are on the stack where BPF put them.
+ */
Does the comment accurately reflect what the arm64 JIT does? Looking at bpf_jit_place_args() with arm64_arg_abi:
For bpf_kfunc_call_test_i128_arg_ovf(u64 a, __int128 v, u64 b, u64 c, u64 d, u64 e, u64 f), the arm64 calling convention computes:
slot0(a)->pos0, slot1/2(v)->pos2/3, slot3(b)->4, slot4(c)->5, slot5(d)->6, slot6(e)->7, slot7(f)->pos8
Position 8 is the first arm64 stack slot (a64_arg_on_stack(8) is true), so bpf_jit_plan_arg_moves() emits the move {dst=8, src=7}.
arch/arm64/net/bpf_jit_comp.c:emit_kfunc_arg_moves() handles this with:
if (a64_arg_on_stack(dst))
emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);The space is reserved by kfunc_abi_slots() returning 9 for this kfunc, giving out_stack_arg_cnt = 4, nr_on_stack = 1, and ctx->stack_arg_size becomes 16 bytes.
This test is annotated '__success __retval(0)', which depends on the JIT correctly placing argument f into the arm64 stack. Does the comment need updating to reflect that the JIT does shift into the arm64 stack?
Quoted 4 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-0
[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-1@@ -3,20 +3,10 @@#include <vmlinux.h> #include <bpf/bpf_helpers.h> -#include "../test_kmods/bpf_testmod_kfunc.h" #include "bpf_misc.h" #if defined(__BPF_FEATURE_STACK_ARGUMENT)
tools/testing/selftests/bpf/progs/stack_arg_fail.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-9-SEC("socket") -__failure __msg("Unrecognized *(R11-8) type STRUCT") -int test_stack_arg_big (struct __sk_buff *ctx) -{ -struct prog_test_big_arg s = {1 ,2 }; - -return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); -} -
The changelog says "the test is removed and the same call is covered by the cases above." That holds only on x86_64 and arm64.
The replacement, aggregate_arg_kfunc_straddle() in progs/aggregate_arg_kfunc.c:97-107, makes the byte-identical call bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s) and correctly checks the return value, but it is tagged:
SEC("tc")
__arch_x86_64 __arch_arm64
__load_if_JITed()
__success __retval(0)
test_loader.c:1339 skips a test whose arch_mask excludes the host arch. The deleted test was gated only on __BPF_FEATURE_STACK_ARGUMENT and ran everywhere.
On riscv64, s390x and loongarch this call is no longer exercised at all. Only x86 and arm64 define bpf_jit_arg_abi(); elsewhere gen_kfunc_arg_proto() takes the '!abi && nslots > 1' branch at verifier.c:12428 and rejects the argument with:
"Function %s arg#%d type %s cannot be passed at argument slot %d on this architecture"
That message is new in this series and has no selftest coverage.
So on the non-ABI architectures the series swapped one rejection message for another and this commit removed the only test that exercised the call there. Should a __failure __msg("cannot be passed at argument slot") case be added to restore coverage on those architectures?
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0
-0
[ ... ]
Quoted 14 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14
-0+__bpf_kfunc u64 bpf_kfunc_call_test_pair_arg(u64 a, struct prog_test_pair_arg s, u64 b) +{ +return a + s.lo + s.hi + b; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg(u64 a, u64 b, __int128 v) +{ +return a + b + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_odd(u64 a, __int128 v, u64 b) +{ +return a + b + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}
[ ... ]
Quoted 9 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +9
-0+__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_shift(u64 a, __int128 v, u64 b, u64 c, u64 d) +{ +return a + b + c + d + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_ovf(u64 a, __int128 v, u64 b, u64 c, u64 d, u64 e, u64 f) +{ +return a + b + c + d + e + f + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4
-0+__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_pad(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, __int128 v) +{ +return a + b + c + d + e + f + g + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}
Three of the new kfuncs place the __int128 after a single-register
scalar: bpf_kfunc_call_test_i128_arg_odd(), i128_arg_shift(), and i128_arg_ovf(), all of the shape 'u64 a, __int128 v, ...'.
This is exactly the shape a comment already in this same file warns against. bpf_testmod.c:174-186 says, above bpf_testmod_test_int128_arg():
/*
* The __int128 'a' is the first argument on purpose. On arm64 a 16-byte
* argument must start in an even-numbered register pair, so placing it
* after a single-register scalar would leave a padding register (x1)
* unused. pahole maps parameters to registers positionally and would then
* see the following argument in an "unexpected" register and skip BTF
* encoding of the whole function, making it unattachable. [...] */
On arm64 AAPCS64 rounds NGRN up to an even number for a 16-byte-aligned type, so for (u64 a, __int128 v, u64 b) the actual placement is a=x0, x1 padding, v=x2:x3, b=x4, while a positional mapping expects v=x1:x2 and b=x3.
Per the in-file comment that makes pahole drop the BTF FUNC record for the whole function on arm64. For a kfunc that is worse than for an fentry
target: the name is registered with BTF_ID_FLAGS(), so resolve_btfids would leave the ID at 0 and emit:
'WARN: resolve_btfids: unresolved symbol bpf_kfunc_call_test_i128_arg_odd'
which is fatal when CONFIG_WERROR selects --fatal_warnings. Even when only a warning, libbpf cannot resolve the ksym against the module BTF, so aggregate_arg_kfunc_int128_odd, aggregate_arg_kfunc_int128_shift and aggregate_arg_kfunc_int128_ovf fail to load on arm64 instead of testing the JIT.
bpf_kfunc_call_test_i128_arg_pad() has the same hazard in the milder stack-vs-register form (a..g fill x0..x6, then v is pushed to the stack where x7 would be expected positionally).
The pre-existing bpf_kfunc_call_test_i128() and bpf_testmod_test_int128_arg() both deliberately keep the __int128 in the first position, and only bpf_kfunc_call_test_i128_arg() (u64 a, u64 b, __int128 v) among the new ones happens to land on an even register with no padding.
Is the placement worth confirming against the pahole version used for CI before this lands, given it's the one thing in the change that the in-tree documentation says should not work on arm64?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441
Yonghong Song
reviewerRe: [PATCH bpf-next v2 12/12] selftests/bpf: Add tests for by-value kfunc arguments
Quoted 46 lines · On 9/9/26 12:30 AM, [email protected] wrote:
Quoted 11 lines
Add kfuncs taking a 16-byte struct and an
__int128by value, with combinations of <= 8 byte arguments and '> 8 && <= 16' byte arguments, and tests that call them. Each checks the value the kfunc returns, so a misplaced argument shows up rather than passing quietly.Between them the cases cover an argument the two conventions place alike, one arm64 moves between registers, one an ABI moves to or from the stack, and one both ABIs pad the stack for. All of them run on both arches; what differs is how many instructions the JIT emits to get there. Two more cover the rejections that hold everywhere, an aggregate holding a pointer and one too large to pass by value.
The changelog describes two new rejection ('
__failure') cases, but only one of the two is added by this commit, and a third rejection case that IS added is never mentioned.Looking at the added tests:
1. 'an aggregate holding a pointer' -> present:
progs/aggregate_arg_kfunc.cadds:
SEC("tc")__arch_x86_64__arch_arm64
__failure__msg("R1 type STRUCT is not composed of scalars")intaggregate_arg_kfunc_ptr_member(struct __sk_buff *skb)2. 'one too large to pass by value' -> NOT added here:
progs/aggregate_arg_kfunc.ccontains exactly one__failureprogram (theptr_memberone above), and the commit declares no kfunc taking a by-value aggregate larger than 16 bytes intest_kmods/bpf_testmod_kfunc.h, so such a case cannot exist in this commit.The only test matching that description is
aggregate_arg_too_big_fail()inprogs/aggregate_arg_func.cwith:
__failure__msg("... has size 24, only 1 to 16 bytes can be passed by value")which was added by the earlier commit
b3f328dbfb8a("selftests/bpf: Add inline-asm tests for by-value arguments") and is untouched here.3. This commit adds a rejection case the changelog never mentions:
progs/arena_kfunc.cgains 16 lines adding:
/* The struct takes two slots, so the arena pointer is the sixth. */SEC("syscall")__arch_x86_64__arch_arm64
__failure__msg("arena pointer cannot be a stack argument")intarena_arg_stack_after_pair(void *ctx)
Quoted 8 lines
together with the new
bpf_kfunc_call_test_pair_arena_arg()kfunc and its__ksymdeclaration.arena_kfunc.cis one of the six files in the diffstat but is not described anywhere in the message.Could the message be updated to say that one new rejection case is added for an aggregate holding a pointer (with the too-large rejection being already covered by
aggregate_arg_func.c), and add a sentence for the arena stack-argument rejection inarena_kfunc.c?
Yes, I can do this.
Quoted 29 lines
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c new file +0
-0[ ... ]
Quoted 5 lines
+/*
+ * One argument further and the hole pushes the last one off x7 and onto the
+ * arm64 stack, which the JIT does not shift into. The x86-64 ABI packs the
+ * eightbytes, so its last two are on the stack where BPF put them.
+ */Does the comment accurately reflect what the arm64 JIT does? Looking at
bpf_jit_place_args()witharm64_arg_abi:For
bpf_kfunc_call_test_i128_arg_ovf(u64 a, __int128 v, u64 b, u64 c, u64 d, u64 e, u64 f), the arm64 calling convention computes:slot0(a)->pos0, slot1/2(v)->pos2/3, slot3(b)->4, slot4(c)->5, slot5(d)->6, slot6(e)->7, slot7(f)->pos8Position 8 is the first arm64 stack slot (
a64_arg_on_stack(8)is true), sobpf_jit_plan_arg_moves()emits the move {dst=8, src=7}.arch/arm64/net/bpf_jit_comp.c:emit_kfunc_arg_moves() handles this with:
if (a64_arg_on_stack(dst)) emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);
Quoted 7 lines
The space is reserved by
kfunc_abi_slots()returning 9 for this kfunc, givingout_stack_arg_cnt= 4,nr_on_stack= 1, andctx->stack_arg_sizebecomes 16 bytes.This test is annotated '
__success__retval(0)', which depends on the JIT correctly placing argument f into the arm64 stack. Does the comment need updating to reflect that the JIT does shift into the arm64 stack?
The prog should already mentioned this. Not all of them but a couple of cases to illustrate.
Quoted 47 lines
Quoted 5 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-0[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-1@@ -3,20 +3,10 @@#include <vmlinux.h> #include <bpf/bpf_helpers.h> -#include "../test_kmods/bpf_testmod_kfunc.h" #include "bpf_misc.h" #if defined(__BPF_FEATURE_STACK_ARGUMENT)tools/testing/selftests/bpf/progs/stack_arg_fail.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/progs/stack_arg_fail.c +0
-9-SEC("socket") -__failure __msg("Unrecognized *(R11-8) type STRUCT") -int test_stack_arg_big (struct __sk_buff *ctx) -{ -struct prog_test_big_arg s = {1 ,2 }; - -return bpf_kfunc_call_stack_arg_big (1 ,2 ,3 ,4 ,5 , s); -} -The changelog says "the test is removed and the same call is covered by the cases above." That holds only on
x86_64and arm64.The replacement,
aggregate_arg_kfunc_straddle()in progs/aggregate_arg_kfunc.c:97-107, makes the byte-identical callbpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s)and correctly checks the return value, but it is tagged:SEC("tc") __arch_x86_64 __arch_arm64 __load_if_JITed() __success __retval(0)test_loader.c:1339 skips a test whose
arch_maskexcludes the host arch. The deleted test was gated only on__BPF_FEATURE_STACK_ARGUMENTand ran everywhere.On riscv64, s390x and loongarch this call is no longer exercised at all. Only x86 and arm64 define
bpf_jit_arg_abi(); elsewheregen_kfunc_arg_proto()takes the '!abi && nslots > 1' branch at verifier.c:12428 and rejects the argument with:
Quoted 8 lines
"Function %s arg#%d type %s cannot be passed at argument slot %d on this architecture"
That message is new in this series and has no selftest coverage.
So on the non-ABI architectures the series swapped one rejection message for another and this commit removed the only test that exercised the call there. Should a
__failure__msg("cannot be passed at argument slot")case be added to restore coverage on those architectures?
I didn't test on riscv64, s390x and loongarch. That is why I skipped them.
Quoted 82 lines
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0
-0[ ... ]
Quoted 14 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14
-0+__bpf_kfunc u64 bpf_kfunc_call_test_pair_arg(u64 a, struct prog_test_pair_arg s, u64 b) +{ +return a + s.lo + s.hi + b; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg(u64 a, u64 b, __int128 v) +{ +return a + b + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_odd(u64 a, __int128 v, u64 b) +{ +return a + b + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}[ ... ]
Quoted 9 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +9
-0+__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_shift(u64 a, __int128 v, u64 b, u64 c, u64 d) +{ +return a + b + c + d + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +} + +__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_ovf(u64 a, __int128 v, u64 b, u64 c, u64 d, u64 e, u64 f) +{ +return a + b + c + d + e + f + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4
-0+__bpf_kfunc u64 bpf_kfunc_call_test_i128_arg_pad(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, __int128 v) +{ +return a + b + c + d + e + f + g + (u64)((unsigned __int128 )v >>64 ) + (u64 )v; +}Three of the new kfuncs place the
__int128after a single-register
scalar:bpf_kfunc_call_test_i128_arg_odd(),i128_arg_shift(), andi128_arg_ovf(), all of the shape 'u64 a,__int128v, ...'.This is exactly the shape a comment already in this same file warns against. bpf_testmod.c:174-186 says, above
bpf_testmod_test_int128_arg():/*
* The__int128'a' is the first argument on purpose. On arm64 a 16-byte
* argument must start in an even-numbered register pair, so placing it
* after a single-register scalar would leave a padding register (x1)
* unused. pahole maps parameters to registers positionally and would then
* see the following argument in an "unexpected" register and skip BTF
* encoding of the whole function, making it unattachable. [...] */On arm64 AAPCS64 rounds NGRN up to an even number for a 16-byte-aligned type, so for (u64 a,
__int128v, u64 b) the actual placement is a=x0, x1 padding, v=x2:x3, b=x4, while a positional mapping expects v=x1:x2 and b=x3.Per the in-file comment that makes pahole drop the BTF FUNC record for the whole function on arm64. For a kfunc that is worse than for an fentry
target: the name is registered withBTF_ID_FLAGS(), soresolve_btfidswould leave the ID at 0 and emit:'WARN:
resolve_btfids: unresolved symbolbpf_kfunc_call_test_i128_arg_odd'which is fatal when
CONFIG_WERRORselects --fatal_warnings. Even when only a warning, libbpf cannot resolve the ksym against the module BTF, soaggregate_arg_kfunc_int128_odd,aggregate_arg_kfunc_int128_shiftandaggregate_arg_kfunc_int128_ovffail to load on arm64 instead of testing the JIT.
bpf_kfunc_call_test_i128_arg_pad()has the same hazard in the milder stack-vs-register form (a..g fill x0..x6, then v is pushed to the stack where x7 would be expected positionally).The pre-existing
bpf_kfunc_call_test_i128()andbpf_testmod_test_int128_arg()both deliberately keep the__int128in the first position, and onlybpf_kfunc_call_test_i128_arg()(u64 a, u64 b,__int128v) among the new ones happens to land on an even register with no padding.
Quoted 3 lines
Is the placement worth confirming against the pahole version used for CI before this lands, given it's the one thing in the change that the in-tree documentation says should not work on arm64?
this pahole patch should fix the issue:
https://lore.kernel.org/bpf/[email protected]/
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/34320399441