Add arena argument support to kfuncs and struct_ops
Subsystems (8)
Kumar Kartikeya Dwivedi
maintainerv3 cover Add arena argument support to kfuncs and struct_ops
This is a continuation of patches in [0], with mostly minor changes and reordering. The motivation is covered in that link. A major change is moving to two tags (__arena and __arena_nullable) and moving the changes to JIT to emit more optimized sequences.
Please see commit logs for details.
[0]: https://lore.kernel.org/bpf/[email protected]
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Rebase onto current bpf-next to resolve conflicts.
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Fix documentation to only mention x86 for now. (Sashiko)
* Move arg bitmap from insn_aux_data to kfunc descriptor. (Eduard)
Tejun Heo (9):
bpf: Support __arena and __arena_nullable kfunc argument suffixes
bpf: Support __arena and __arena_nullable on struct_ops arguments bpf, x86: JIT __arena kfunc argument rebasing
bpf, x86: Convert struct_ops arena arguments in the trampoline selftests/bpf: Add kfunc __arena and __arena_nullable argument tests selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments selftests/bpf: Add struct_ops __arena and __arena_nullable argument
tests
bpf, x86: Fix stack-passed arguments for indirect trampolines selftests/bpf: Test stack-passed struct_ops arena arguments
Documentation/bpf/kfuncs.rst | 39 +++ arch/x86/net/bpf_jit_comp.c | 129 ++++++++- include/linux/bpf.h | 25 ++ include/linux/filter.h | 1 + kernel/bpf/bpf_struct_ops.c | 34 ++- kernel/bpf/btf.c | 28 +- kernel/bpf/core.c | 5 + kernel/bpf/trampoline.c | 64 +++++ kernel/bpf/verifier.c | 63 ++++- .../selftests/bpf/prog_tests/arena_kfunc.c | 15 + .../bpf/prog_tests/arena_kfunc_jit.c | 13 + .../bpf/prog_tests/test_struct_ops_arena.c | 74 +++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 260 ++++++++++++++++++ .../selftests/bpf/progs/arena_kfunc_jit.c | 96 +++++++ .../selftests/bpf/progs/struct_ops_arena.c | 115 ++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 ++ .../selftests/bpf/test_kmods/bpf_testmod.c | 95 +++++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 6 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 13 + 19 files changed, 1062 insertions(+), 33 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc_jit.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
base-commit: 60781269e26c786de2bb93fb1e697a5c32ccee48
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH v3 1/9 bpf: Support __arena and __arena_nullable kfunc argument suffixes
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the __arena and __arena_nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.
__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena_nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.
This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data.
JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere.
--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)
Documentation/bpf/kfuncs.rst +29 -0
@@ -278,6 +278,33 @@ An example is given below:: ... } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function
@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------
include/linux/bpf.h +6 -0
@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;
include/linux/filter.h +1 -0
@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );
kernel/bpf/btf.c +17 -1
@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,
@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;
kernel/bpf/core.c +5 -0
@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;
kernel/bpf/verifier.c +39 -1
@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); } +static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)
@@ -11127,6 +11137,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_IRQ_FLAG ,KF_ARG_PTR_TO_RES_SPIN_LOCK ,KF_ARG_PTR_TO_TASK_WORK , + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type {
@@ -11412,7 +11423,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t));return -EINVAL ; } - ref_t =btf_type_skip_modifiers (meta->btf, t->type,NULL ); ref_tname =btf_name_by_offset (meta->btf, ref_t->name_off);
@@ -11461,6 +11471,9 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type =KF_ARG_PTR_TO_RES_SPIN_LOCK ; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type =KF_ARG_PTR_TO_CALLBACK ; + else if (is_kfunc_arg_arena(meta->btf, &args[arg]) || + is_kfunc_arg_arena_nullable(meta->btf, &args[arg])) + arg_type =KF_ARG_PTR_TO_ARENA ; else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL ); +if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + } +if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + } +if (btf_type_is_ptr (t) && (bpf_register_is_null (reg) ||type_may_be_null (reg->type)) && !is_kfunc_arg_nullable (meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Valid allocated arena objects at page offset 0 are incorrectly translated to NULL when passed to…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Valid allocated arena objects at page offset 0 are incorrectly translated to NULL when passed to __arena_nullable kfunc arguments. --
--- Patch [1]: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes --- commit d1b161404c5304b1bf05e11ad5b7dc889d1dcee6
Author: Tejun Heo <[email protected]>
bpf: Support __arena and __arena_nullable kfunc argument suffixes
This commit adds support for __arena and __arena_nullable kfunc argument suffixes, allowing kfuncs to receive directly dereferenceable kernel addresses for arena pointers. It updates the verifier to distill these into argument flags for the JIT to retrieve.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12170
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL ); +if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) {
[Severity: High]
Can a valid allocated arena object at page offset 0 be incorrectly translated to NULL when passed to an __arena_nullable kfunc argument?
Looking at arena_map_alloc() in kernel/bpf/arena.c, the range tree of free pages includes page offset 0:
err = range_tree_set(&arena->rt, 0, attr->max_entries);
If a user process mmaps the arena or sets map_extra at an address where the lower 32 bits are zero (e.g., 0x7f0000000000), a BPF program could allocate an object that gets offset 0. When passed to an __arena_nullable kfunc argument, the JIT would see the lower 32 bits as exactly zero.
Since the commit message states that __arena_nullable skips the rebase when the truncated value (arena offset 0) is zero to preserve NULL, would this cause the kfunc to incorrectly assume a valid optional argument is absent and lead to dropped data or logic errors?
Quoted 24 lines
kernel/bpf/verifier.c +23
-0+if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + } +if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
In reply to Kumar Kartikeya DwivediQuoted 180 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaand__arena_nullableargument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN =kern_vm_start+ (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.
__arenarebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base.__arena_nullablepreserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args()and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state ininsn_aux_data.JITs declare support with
bpf_jit_supports_arena_args()and verification fails with-ENOTSUPPelsewhere.Signed-off-by Tejun Heo <[email protected]>Co-developed-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)Documentation/bpf/kfuncs.rst +29
-0@@ -278,6 +278,33 @@ An example is given below:: } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf.h +6
-0@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;include/linux/filter.h +1
-0@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );kernel/bpf/btf.c +17
-1@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;kernel/bpf/core.c +5
-0@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;kernel/bpf/verifier.c +0
-0@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); }
kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10919
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +
From the verifier's point of view, __arena and __arena_nullable all maps to KF_ARG_PTR_TO_ARENA and require the same check so maybe is_kfunc_arg_arena() alone with two suffix matching is good enough, but it also doesn't hurt to have two separated functions.
Quoted 30 lines
kernel/bpf/verifier.c +4
-1static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)@@ -11127,6 +11137,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_IRQ_FLAG ,KF_ARG_PTR_TO_RES_SPIN_LOCK ,KF_ARG_PTR_TO_TASK_WORK , + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type {@@ -11412,7 +11423,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t));return -EINVAL ; } - ref_t =btf_type_skip_modifiers (meta->btf, t->type,NULL ); ref_tname =btf_name_by_offset (meta->btf, ref_t->name_off);@@ -11461,6 +11471,9 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type =KF_ARG_PTR_TO_RES_SPIN_LOCK ; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type =KF_ARG_PTR_TO_CALLBACK ; + else if (is_kfunc_arg_arena(meta->btf, &args[arg]) || + is_kfunc_arg_arena_nullable(meta->btf, &args[arg])) + arg_type =KF_ARG_PTR_TO_ARENA ; else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12175
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL ); +if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + }
nit: JIT and register-based arg check can be done earlier in get_kfunc_arg_type(). We probably will want a consolidated pass validating bpf_func_proto.
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno));
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12194
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + } +
Can we move arena argument checking to the second switch case below? It only takes adding one condition below.
This help reduce churn in the future when check_kfunc_args() also receieve check_reg_type().
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12197
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (btf_type_is_ptr (t) && (bpf_register_is_null (reg) ||type_may_be_null (reg->type)) && !is_kfunc_arg_nullable (meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",
if (btf_type_is_ptr(t) && base_type(kf_arg_type) != KF_ARG_PTR_TO_ARENA &&
(bpf_register_is_null(reg) || type_may_be_null(reg->type)) &&
!type_may_be_null(kf_arg_type)) {
verbose(env, "Possibly NULL pointer passed to trusted %s\n",Quoted 4 lines
--
2.53.0
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
Quoted 230 lines · Amery Hung
Quoted 180 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaand__arena_nullableargument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN =kern_vm_start+ (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.
__arenarebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base.__arena_nullablepreserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args()and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state ininsn_aux_data.JITs declare support with
bpf_jit_supports_arena_args()and verification fails with-ENOTSUPPelsewhere.Signed-off-by Tejun Heo <[email protected]>Co-developed-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)Documentation/bpf/kfuncs.rst +29
-0@@ -278,6 +278,33 @@ An example is given below:: } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf.h +6
-0@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;include/linux/filter.h +1
-0@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );kernel/bpf/btf.c +17
-1@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;kernel/bpf/core.c +5
-0@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;kernel/bpf/verifier.c +0
-0@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); }kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10919
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +From the verifier's point of view,
__arenaand__arena_nullableall maps toKF_ARG_PTR_TO_ARENAand require the same check so maybeis_kfunc_arg_arena()alone with two suffix matching is good enough, but it also doesn't hurt to have two separated functions.Quoted 33 lines
kernel/bpf/verifier.c +4
-1static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)@@ -11127,6 +11137,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_IRQ_FLAG ,KF_ARG_PTR_TO_RES_SPIN_LOCK ,KF_ARG_PTR_TO_TASK_WORK , + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type {@@ -11412,7 +11423,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t));return -EINVAL ; } - ref_t =btf_type_skip_modifiers (meta->btf, t->type,NULL ); ref_tname =btf_name_by_offset (meta->btf, ref_t->name_off);@@ -11461,6 +11471,9 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type =KF_ARG_PTR_TO_RES_SPIN_LOCK ; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type =KF_ARG_PTR_TO_CALLBACK ; + else if (is_kfunc_arg_arena(meta->btf, &args[arg]) || + is_kfunc_arg_arena_nullable(meta->btf, &args[arg])) + arg_type =KF_ARG_PTR_TO_ARENA ; else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL );
Quoted 11 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12175
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + }nit: JIT and register-based arg check can be done earlier in
get_kfunc_arg_type(). We probably will want a consolidated pass validatingbpf_func_proto.
I moved it up there. We still have prog aux argument doing stack argument rejection later, but I will let you figure out the right placement when you rework these bits.
Only thing left here now is the register type check.
Quoted 26 lines
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno));kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12194
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + } +Can we move arena argument checking to the second switch case below? It only takes adding one condition below.
This help reduce churn in the future when
check_kfunc_args()also receievecheck_reg_type().
Quoted 10 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12197
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (btf_type_is_ptr (t) && (bpf_register_is_null (reg) ||type_may_be_null (reg->type)) && !is_kfunc_arg_nullable (meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",if (btf_type_is_ptr(t) && base_type(kf_arg_type) != KF_ARG_PTR_TO_ARENA && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && !type_may_be_null(kf_arg_type)) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",
To simplify this and avoid special casing, I changed the kf_arg_type to KF_ARG_PTR_TO_ARENA | PTR_MAYBE_NULL, since known NULL is kosher for both cases and we place no static constraints on the precise value.
Re: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
Quoted 244 lines · Kumar Kartikeya Dwivedi
Quoted 230 lines · Amery Hung
Quoted 180 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaand__arena_nullableargument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN =kern_vm_start+ (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.
__arenarebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base.__arena_nullablepreserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args()and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state ininsn_aux_data.JITs declare support with
bpf_jit_supports_arena_args()and verification fails with-ENOTSUPPelsewhere.Signed-off-by Tejun Heo <[email protected]>Co-developed-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)Documentation/bpf/kfuncs.rst +29
-0@@ -278,6 +278,33 @@ An example is given below:: } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf.h +6
-0@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;include/linux/filter.h +1
-0@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );kernel/bpf/btf.c +17
-1@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;kernel/bpf/core.c +5
-0@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;kernel/bpf/verifier.c +0
-0@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); }kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10919
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +From the verifier's point of view,
__arenaand__arena_nullableall maps toKF_ARG_PTR_TO_ARENAand require the same check so maybeis_kfunc_arg_arena()alone with two suffix matching is good enough, but it also doesn't hurt to have two separated functions.Quoted 33 lines
kernel/bpf/verifier.c +4
-1static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)@@ -11127,6 +11137,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_IRQ_FLAG ,KF_ARG_PTR_TO_RES_SPIN_LOCK ,KF_ARG_PTR_TO_TASK_WORK , + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type {@@ -11412,7 +11423,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t));return -EINVAL ; } - ref_t =btf_type_skip_modifiers (meta->btf, t->type,NULL ); ref_tname =btf_name_by_offset (meta->btf, ref_t->name_off);@@ -11461,6 +11471,9 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type =KF_ARG_PTR_TO_RES_SPIN_LOCK ; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type =KF_ARG_PTR_TO_CALLBACK ; + else if (is_kfunc_arg_arena(meta->btf, &args[arg]) || + is_kfunc_arg_arena_nullable(meta->btf, &args[arg])) + arg_type =KF_ARG_PTR_TO_ARENA ; else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL );Quoted 11 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12175
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + }nit: JIT and register-based arg check can be done earlier in
get_kfunc_arg_type(). We probably will want a consolidated pass validatingbpf_func_proto.
Quoted 3 lines
I moved it up there. We still have prog aux argument doing stack argument rejection later, but I will let you figure out the right placement when you rework these bits.
Sounds good to me. The placement is not really a blocker.
Quoted 40 lines
Only thing left here now is the register type check.
Quoted 26 lines
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno));kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12194
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + } +Can we move arena argument checking to the second switch case below? It only takes adding one condition below.
This help reduce churn in the future when
check_kfunc_args()also receievecheck_reg_type().Quoted 10 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12197
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (btf_type_is_ptr (t) && (bpf_register_is_null (reg) ||type_may_be_null (reg->type)) && !is_kfunc_arg_nullable (meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",if (btf_type_is_ptr(t) && base_type(kf_arg_type) != KF_ARG_PTR_TO_ARENA && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && !type_may_be_null(kf_arg_type)) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",
Quoted 3 lines
To simplify this and avoid special casing, I changed the
kf_arg_typetoKF_ARG_PTR_TO_ARENA|PTR_MAYBE_NULL, since known NULL is kosher for both cases and we place no static constraints on the precise value.
Make sense. Thanks for tweaking it!
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
Quoted 291 lines · Amery Hung
Quoted 244 lines · Kumar Kartikeya Dwivedi
Quoted 230 lines · Amery Hung
Quoted 180 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaand__arena_nullableargument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN =kern_vm_start+ (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.
__arenarebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base.__arena_nullablepreserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args()and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state ininsn_aux_data.JITs declare support with
bpf_jit_supports_arena_args()and verification fails with-ENOTSUPPelsewhere.Signed-off-by Tejun Heo <[email protected]>Co-developed-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)Documentation/bpf/kfuncs.rst +29
-0@@ -278,6 +278,33 @@ An example is given below:: } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf.h +6
-0@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;include/linux/filter.h +1
-0@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );kernel/bpf/btf.c +17
-1@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;kernel/bpf/core.c +5
-0@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;kernel/bpf/verifier.c +0
-0@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); }kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10919
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +From the verifier's point of view,
__arenaand__arena_nullableall maps toKF_ARG_PTR_TO_ARENAand require the same check so maybeis_kfunc_arg_arena()alone with two suffix matching is good enough, but it also doesn't hurt to have two separated functions.Quoted 33 lines
kernel/bpf/verifier.c +4
-1static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)@@ -11127,6 +11137,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_IRQ_FLAG ,KF_ARG_PTR_TO_RES_SPIN_LOCK ,KF_ARG_PTR_TO_TASK_WORK , + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type {@@ -11412,7 +11423,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t));return -EINVAL ; } - ref_t =btf_type_skip_modifiers (meta->btf, t->type,NULL ); ref_tname =btf_name_by_offset (meta->btf, ref_t->name_off);@@ -11461,6 +11471,9 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type =KF_ARG_PTR_TO_RES_SPIN_LOCK ; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type =KF_ARG_PTR_TO_CALLBACK ; + else if (is_kfunc_arg_arena(meta->btf, &args[arg]) || + is_kfunc_arg_arena_nullable(meta->btf, &args[arg])) + arg_type =KF_ARG_PTR_TO_ARENA ; else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {@@ -12154,6 +12167,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t =btf_type_skip_modifiers (btf, args[i].type,NULL );Quoted 11 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12175
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+if (base_type (kf_arg_type) ==KF_ARG_PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support kfunc %s() with arena pointer arguments\n" , + func_name); +return -ENOTSUPP ; + }nit: JIT and register-based arg check can be done earlier in
get_kfunc_arg_type(). We probably will want a consolidated pass validatingbpf_func_proto.Quoted 3 lines
I moved it up there. We still have prog aux argument doing stack argument rejection later, but I will let you figure out the right placement when you rework these bits.
Sounds good to me. The placement is not really a blocker.
Quoted 40 lines
Only thing left here now is the register type check.
Quoted 36 lines
Quoted 9 lines
kernel/bpf/verifier.c +9
-0+if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno));kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12194
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +continue ; + } +Can we move arena argument checking to the second switch case below? It only takes adding one condition below.
This help reduce churn in the future when
check_kfunc_args()also receievecheck_reg_type().kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me · line 12197
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (btf_type_is_ptr (t) && (bpf_register_is_null (reg) ||type_may_be_null (reg->type)) && !is_kfunc_arg_nullable (meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",if (btf_type_is_ptr(t) && base_type(kf_arg_type) != KF_ARG_PTR_TO_ARENA && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && !type_may_be_null(kf_arg_type)) { verbose(env, "Possibly NULL pointer passed to trusted %s\n",
Quoted 5 lines
Quoted 3 lines
To simplify this and avoid special casing, I changed the
kf_arg_typetoKF_ARG_PTR_TO_ARENA|PTR_MAYBE_NULL, since known NULL is kosher for both cases and we place no static constraints on the precise value.Make sense. Thanks for tweaking it!
Actually, I realized env->prog->aux->arena is not resolved when get_kfunc_arg_type() is called. In general, this feels like an accidental side effect of the ordering of phases, so I decided to tweak it (move add_kfunc() in add_subprogs_and_kfuncs() out and before check_and_resolve_insns()).
This should make assumptions inside that function much simpler, and avoid such footguns.
It should be a noop functionally, and hopefully more clear ordering of various checks. I prepended those NFC patches to the set, and we can discuss the details in the next version.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 1/9] bpf: Support __arena and __arena_nullable kfunc argument suffixes
In reply to Amery HungQuoted 181 lines · Amery Hung
Quoted 180 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaand__arena_nullableargument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN =kern_vm_start+ (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.
__arenarebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base.__arena_nullablepreserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.This patch adds the verifier side: the suffixes are recognized in
check_kfunc_args()and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state ininsn_aux_data.JITs declare support with
bpf_jit_supports_arena_args()and verification fails with-ENOTSUPPelsewhere.Signed-off-by Tejun Heo <[email protected]>Co-developed-by Kumar Kartikeya Dwivedi <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++++++++ include/linux/bpf.h | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++++- kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 40 +++++++++++++++++++++++++++++++++++- 6 files changed, 97 insertions(+), 2 deletions(-)Documentation/bpf/kfuncs.rst +29
-0@@ -278,6 +278,33 @@ An example is given below:: } +2.3.8 __arena and __arena_nullable Annotations +---------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena_nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf.h +6
-0@@ -1213,6 +1213,12 @@ struct bpf_prog_offload {/* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) +struct btf_func_model {u8 ret_size;u8 ret_flags;include/linux/filter.h +1
-0@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );bool bpf_jit_supports_stack_args (void ); +bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_ptr_xchg (void );kernel/bpf/btf.c +17
-1@@ -7541,6 +7541,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)return flags; } +static u8 __get_arg_fmodel_flags (const struct btf *btf, +const struct btf_param *arg, +const struct btf_type *t) +{ +u8 flags =__get_type_fmodel_flags (t); + +if (btf_param_match_suffix (btf, arg,"__arena" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_ARENA_ARG ; +if (btf_param_match_suffix (btf, arg,"__nullable" ) || +btf_param_match_suffix (btf, arg,"__arena_nullable" )) + flags |=BTF_FMODEL_NULLABLE_ARG ; + +return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func,@@ -7606,7 +7622,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } m->arg_size[i] = ret; - m->arg_flags[i] =__get_type_fmodel_flags (t); + m->arg_flags[i] =__get_arg_fmodel_flags (btf, &args[i], t); } m->nr_args = nargs;return 0 ;kernel/bpf/core.c +5
-0@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)return false; } +bool __weak bpf_jit_supports_arena_args (void ) +{ +return false; +} +bool __weak bpf_jit_supports_far_kfunc_call (void ) {return false;kernel/bpf/verifier.c +0
-0@@ -10907,6 +10907,16 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); }
Quoted 16 lines
kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10919
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} + +static bool is_kfunc_arg_arena_nullable (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena_nullable" ); +} +From the verifier's point of view,
__arenaand__arena_nullableall maps toKF_ARG_PTR_TO_ARENAand require the same check so maybeis_kfunc_arg_arena()alone with two suffix matching is good enough, but it also doesn't hurt to have two separated functions.
Missed in previous reply; I consolidated both into the same is_kfunc_arg_arena() predicate.
Quoted 1 line
[...]
Kumar Kartikeya Dwivedi
maintainerPATCH v3 2/9 bpf: Support __arena and __arena_nullable on struct_ops arguments
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in.
Add the __arena and __arena_nullable stub argument suffixes to make this convenient. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). The trampoline converts the value while saving the arguments into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the program never sees a kernel address and nothing rewrites the ctx after the fact. The converted value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address.
__arena converts unconditionally and the kernel caller must not pass NULL. __arena_nullable preserves NULL, tested on the full 64-bit kernel pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in program's type state already encompasses NULL-ness, so it is not meaningful to force a NULL check for the program.
This patch adds the generic side. bpf_tramp_collect_arena_args() derives the conversion map from the prog's ctx_arg_info, keyed by the flattened ctx byte offset since preceding 16-byte arguments occupy two slots. Only the struct_ops indirect trampoline converts: it dispatches to a single prog whose arena is fixed at generation time. Generic trampolines can mix progs with different arenas and reject arena ctx args defensively, which is unreachable today as only struct_ops progs carry them. Arch trampolines that do not implement the conversion are gated out at verification time with bpf_jit_supports_arena_args().
--- Documentation/bpf/kfuncs.rst | 10 ++++++ include/linux/bpf.h | 19 +++++++++++ kernel/bpf/bpf_struct_ops.c | 34 ++++++++++++++----- kernel/bpf/btf.c | 10 ++++-- kernel/bpf/trampoline.c | 64 ++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 23 ++++++++++--- 6 files changed, 143 insertions(+), 17 deletions(-)
Documentation/bpf/kfuncs.rst +10 -0
@@ -305,6 +305,16 @@ arena argument support (currently x86-64); verification fails otherwise. The program can pass any value without compromising the kernel. A value that does not point into the arena is a program bug. +The suffixes have the same meaning on the arguments of struct_ops stub +functions, with the conversion running in the opposite direction. The +kernel caller passes the kernel arena address and the trampoline converts +it while saving the arguments, so the callback receives an arena pointer +it can dereference directly. With ``__arena`` the kernel caller must not +pass NULL. With ``__arena_nullable`` a NULL kernel pointer arrives as NULL. +However, there is no obligation to prove to the verifier that such a pointer is +non-NULL before use, in-line with existing semantics of arena pointers used in +a program (or obtained from any other source). + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function
include/linux/bpf.h +19 -0
@@ -1292,6 +1292,20 @@ struct bpf_tramp_nodes {int nr_nodes; }; +/* +* Which 8-byte ctx slots of a struct_ops trampoline hold arena kernel +* pointers that save_args() converts to the arena pointer form, +* ctx[slot] = (u32)(kaddr - kern_vm_start). +*/ +struct bpf_tramp_arena_args { +u32 slots; +u32 nullable_slots;/* subset of @slots where NULL is preserved */ +u64 kern_vm_start; +}; + +bool bpf_tramp_collect_arena_args (struct bpf_tramp_nodes *tnodes,u32 flags, +struct bpf_tramp_arena_args *aargs); +struct bpf_tramp_run_ctx ;/* Different use cases for BPF trampoline:
@@ -1693,6 +1707,11 @@ struct bpf_ctx_arg_aux {u32 btf_id;u32 ref_id;bool refcounted; +/* +* We don't encode NULL-ness in the type for the program, but still need +* to distinguish it for the purposes of telling JITs what sequence to emit. +*/ +bool arena_nullable; };struct btf_mod_pair {
kernel/bpf/bpf_struct_ops.c +25 -9
@@ -147,6 +147,8 @@ void bpf_struct_ops_image_free(void *image)#define MAYBE_NULL_SUFFIX "__nullable"#define REFCOUNTED_SUFFIX "__ref" +#define ARENA_SUFFIX "__arena" +#define ARENA_MAYBE_NULL_SUFFIX "__arena_nullable"/* Prepare argument info for every nullable argument of a member of a * struct_ops type.
@@ -159,7 +161,7 @@ void bpf_struct_ops_image_free(void *image)* to provide an array of struct bpf_ctx_arg_aux, which in turn provides * the information that used by the verifier to check the arguments of the * BPF struct_ops program assigned to the member. Here, we only care about -* the arguments that are marked as __nullable. +* the arguments that are marked as __nullable, __ref or __arena. * * The array of struct bpf_ctx_arg_aux is eventually assigned to * prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
@@ -175,7 +177,8 @@ static int prepare_arg_info(struct btf *btf, struct bpf_struct_ops_arg_info *arg_info) {const struct btf_type *stub_func_proto, *pointed_type; -bool is_nullable = false, is_refcounted = false; +bool is_nullable = false, is_refcounted = false, is_arena = false; +bool is_arena_nullable = false;const struct btf_param *stub_args, *args;struct bpf_ctx_arg_aux *info, *info_buf;u32 nargs, arg_no, info_cnt =0 ;
@@ -226,26 +229,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf;for (arg_no =0 ; arg_no < nargs; arg_no++) {/* Skip arguments that is not suffixed with -* "__nullable or __ref". +* "__nullable", "__ref", "__arena" or "__arena_nullable". */ is_nullable =btf_param_match_suffix (btf, &stub_args[arg_no],MAYBE_NULL_SUFFIX ); is_refcounted =btf_param_match_suffix (btf, &stub_args[arg_no],REFCOUNTED_SUFFIX ); + is_arena_nullable =btf_param_match_suffix (btf, &stub_args[arg_no], +ARENA_MAYBE_NULL_SUFFIX ); + is_arena =btf_param_match_suffix (btf, &stub_args[arg_no],ARENA_SUFFIX );if (is_nullable) suffix =MAYBE_NULL_SUFFIX ;else if (is_refcounted) suffix =REFCOUNTED_SUFFIX ; +else if (is_arena_nullable) + suffix =ARENA_MAYBE_NULL_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;else continue ; -/* Should be a pointer to struct */ - pointed_type =btf_type_resolve_ptr (btf, - args[arg_no].type, - &arg_btf_id); -if (!pointed_type || - !btf_type_is_struct (pointed_type)) { +/* Should be a pointer to struct, or any pointer for __arena/__arena_nullable */ + pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id); +if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct (pointed_type))) {pr_warn ("stub function %s has %s tagging to an unsupported type\n" , stub_fname, suffix); goto err_out;
@@ -273,6 +280,15 @@ static int prepare_arg_info(struct btf *btf, }else if (is_refcounted) { info->reg_type =PTR_TRUSTED |PTR_TO_BTF_ID ; info->refcounted = true; + }else if (is_arena || is_arena_nullable) { +/* +* Both types get PTR_TO_ARENA. In verifier state, +* PTR_TO_ARENA encompasses potential NULL values, but +* we do not force the program to check it, or maintain +* precision around it, since it has no safety implication. +*/ + info->reg_type =PTR_TO_ARENA ; + info->arena_nullable = is_arena_nullable; } info++;
kernel/bpf/btf.c +7 -3
@@ -6963,15 +6963,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,return false; } -/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */ +/* +* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or +* PTR_TO_ARENA (both nullable and non-nullable cases). +*/ for (i =0 ; i < prog->aux->ctx_arg_info_size; i++) {const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];u32 type, flag; type =base_type (ctx_arg_info->reg_type); flag =type_flag (ctx_arg_info->reg_type); -if (ctx_arg_info->offset == off && type ==PTR_TO_BUF && - (flag &PTR_MAYBE_NULL )) { +if (ctx_arg_info->offset == off && + (type ==PTR_TO_ARENA || + (type ==PTR_TO_BUF && (flag &PTR_MAYBE_NULL )))) { info->reg_type = ctx_arg_info->reg_type;return true; }
kernel/bpf/trampoline.c +64 -0
82 lines not on this page · raw message
kernel/bpf/verifier.c +18 -5
@@ -18845,6 +18845,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) {const struct btf_type *t, *func_proto;const struct bpf_struct_ops_desc *st_ops_desc; +const struct bpf_struct_ops_arg_info *arg_info;const struct bpf_struct_ops *st_ops;const struct btf_member *member;struct bpf_prog *prog = env->prog;
@@ -18923,10 +18924,23 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)return -EACCES ; } -for (i =0 ; i < st_ops_desc->arg_info[member_idx].cnt; i++) { -if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info = &st_ops_desc->arg_info[member_idx]; +for (i =0 ; i < arg_info->cnt; i++) { +const struct bpf_ctx_arg_aux *info = &arg_info->info[i]; + +if (info->refcounted) has_refcounted_arg = true; -break ; +if (base_type (info->reg_type) ==PTR_TO_ARENA ) { +if (!bpf_jit_supports_arena_args ()) { +verbose (env,"JIT does not support arena arguments\n" ); +return -ENOTSUPP ; + } +if (!prog->aux->arena) { +verbose (env, +"arena argument of %s requires a program with an associated arena\n" , + mname); +return -EINVAL ; + } } }
@@ -18947,8 +18961,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_name = mname; env->ops = st_ops->verifier_ops; -return bpf_prog_ctx_arg_info_init (prog, st_ops_desc->arg_info[member_idx].info, - st_ops_desc->arg_info[member_idx].cnt); +return bpf_prog_ctx_arg_info_init (prog, arg_info->info, arg_info->cnt); }#define SECURITY_PREFIX "security_"
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The `__arena_nullable` BTF parameter suffix is erroneously processed as `__nullable` due to…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The __arena_nullable BTF parameter suffix is erroneously processed as __nullable due to substring matching, breaking PTR_TO_ARENA semantics.
- [High] Attaching tracing programs (fentry/fexit) to struct_ops programs with arena arguments causes them to receive 32-bit user-space offsets typed as valid kernel pointers, leading to SMAP violations and kernel panics.
- [Low] Modified multi-line comments do not conform to the required BPF subsystem comment style.
--
--- Patch [2]: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments ---
Note: The format of this review has been altered due to system restrictions on
reciting original code. Instead of inline diff quotes, this report provides a
free-form summary of the findings.
commit 0582d11e29a7b343a06709e7d3f0cdcc15655dcc
Author: Tejun Heo <[email protected]>
Subject: bpf: Support __arena and __arena_nullable on struct_ops arguments
This patch adds support for the __arena and __arena_nullable suffixes on struct_ops arguments. This allows callbacks to declare parameters as arena pointers, while the kernel caller passes the kernel arena address and the trampoline handles the conversion.
[Severity: High]
In kernel/bpf/bpf_struct_ops.c within prepare_arg_info(), does the if/else chain incorrectly prioritize the shorter suffix? Because btf_param_match_suffix() matches the end of the parameter name, a parameter ending in __arena_nullable also matches the shorter __nullable suffix.
Since the code checks for the nullable condition first, will this cause __arena_nullable parameters to be incorrectly typed as PTR_MAYBE_NULL and PTR_TO_BTF_ID rather than PTR_TO_ARENA? This seems like it could cause the trampoline to skip the arena pointer conversion, passing a raw kernel pointer instead, and potentially leading to verifier errors if used with a non-struct pointer.
[Severity: Low]
This isn't a bug, but in kernel/bpf/bpf_struct_ops.c within prepare_arg_info(), the modified multi-line comments appear to retain the older kernel comment style where text follows the opening delimiter on the same line.
Does the BPF subsystem require multi-line comments to have the opening delimiter on its own line, even when modifying existing comments in a file that previously used the old style?
[Severity: High]
In kernel/bpf/trampoline.c within bpf_trampoline_update(), does the defensive check evaluate the wrong program's context arguments? The code appears to check the attached tracing program (fentry/fexit) for arena arguments rather than the target struct_ops program.
Since tracing programs do not have their ctx_arg_info initialized, could this allow them to successfully attach to struct_ops programs that have arena arguments?
If so, during verification in kernel/bpf/btf.c at btf_ctx_access(), the tracing program's context argument would be typed as a standard PTR_TO_BTF_ID because its ctx_arg_info_size is zero. At runtime, the struct_ops trampoline would convert the kernel pointer to a 32-bit arena offset, which the tracing program would receive and potentially dereference as a standard kernel pointer. Could this sequence lead to SMAP violations and kernel panics?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments
In reply to Kumar Kartikeya DwivediOn Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
Lgtm except for several nits below.
...
Quoted 1 line
kernel/bpf/bpf_struct_ops.c +0
-0
...
Quoted 19 lines
kernel/bpf/bpf_struct_ops.c +4
-1@@ -226,26 +229,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf; for (arg_no = 0; arg_no < nargs; arg_no++) { /* Skip arguments that is not suffixed with - * "__nullable or __ref". + * "__nullable", "__ref", "__arena" or "__arena_nullable". */ is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX); is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no], REFCOUNTED_SUFFIX); + is_arena_nullable =btf_param_match_suffix (btf, &stub_args[arg_no], + ARENA_MAYBE_NULL_SUFFIX); + is_arena =btf_param_match_suffix (btf, &stub_args[arg_no],ARENA_SUFFIX );
if (is_nullable)
suffix =MAYBE_NULL_SUFFIX; else if (is_refcounted) suffix =REFCOUNTED_SUFFIX;
kernel/bpf/bpf_struct_ops.c static int prepare_arg_info(struct btf *btf, · line 249
Quoted 4 lines
kernel/bpf/bpf_struct_ops.c +4
-0+else if (is_arena_nullable) + suffix =ARENA_MAYBE_NULL_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;
As discussed already, that's unfortunate that p__arena and p__nullable are valid, but p__arena__nullable is not, substituted by p__arena_nullable. Same goes for global subprogs.
Quoted 15 lines
else
continue;kernel/bpf/bpf_struct_ops.c +3
-6-/* Should be a pointer to struct */ - pointed_type =btf_type_resolve_ptr (btf, - args[arg_no].type, - &arg_btf_id); -if (!pointed_type || - !btf_type_is_struct(pointed_type)) { +/* Should be a pointer to struct, or any pointer for __arena/__arena_nullable */ + pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id); +if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct (pointed_type))) {
pr_warn("stub function %s has %s tagging to an unsupported type\n", stub_fname, suffix);
gotoerr_out;
...
Quoted 4 lines
kernel/bpf/trampoline.c +0
-0
...
Quoted 9 lines
kernel/bpf/trampoline.c +0
-0@@ -695,6 +743,22 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; }kernel/bpf/trampoline.c +5
-0+/* +* Arena ctx args are converted only by the struct_ops indirect +* trampoline, which dispatches to a single known prog. Generic +* trampolines can mix progs with different arenas, so no conversion +* is possible here. Not reachable today: only struct_ops progs get
kernel/bpf/trampoline.c static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut · line 760
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+* arena ctx args and they never ride generic trampolines. +*/ +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +if (bpf_prog_has_arena_ctx_arg (tnodes[kind].nodes[i]->link->prog)) { + err = -ENOTSUPP ; + goto out; + } + } + }
Nit: should this be done in __bpf_trampoline_link_prog() or
bpf_trampoline_add_prog()? To avoid doing the same check on unlink. Since this is an invariant violation, do we want to add WARN_ONCE?
Quoted 4 lines
+
/* clear all bits exceptSHARE_IPMODIFYandTAIL_CALL_CTX*/tr->flags&= (BPF_TRAMP_F_SHARE_IPMODIFY|BPF_TRAMP_F_TAIL_CALL_CTX);
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments
Quoted 87 lines · Eduard Zingerman
On Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
Lgtm except for several nits below.
...
Quoted 1 line
kernel/bpf/bpf_struct_ops.c +0
-0...
Quoted 19 lines
kernel/bpf/bpf_struct_ops.c +4
-1@@ -226,26 +229,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf; for (arg_no = 0; arg_no < nargs; arg_no++) { /* Skip arguments that is not suffixed with - * "__nullable or __ref". + * "__nullable", "__ref", "__arena" or "__arena_nullable". */ is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX); is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no], REFCOUNTED_SUFFIX); + is_arena_nullable =btf_param_match_suffix (btf, &stub_args[arg_no], + ARENA_MAYBE_NULL_SUFFIX); + is_arena =btf_param_match_suffix (btf, &stub_args[arg_no],ARENA_SUFFIX );
if (is_nullable)
suffix =MAYBE_NULL_SUFFIX; else if (is_refcounted) suffix =REFCOUNTED_SUFFIX;kernel/bpf/bpf_struct_ops.c
Quoted 4 lines
kernel/bpf/bpf_struct_ops.c +4
-0+else if (is_arena_nullable) + suffix =ARENA_MAYBE_NULL_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;As discussed already, that's unfortunate that
p__arenaandp__nullableare valid, butp__arena__nullableis not, substituted byp__arena_nullable. Same goes for global subprogs.Quoted 15 lines
else
continue;kernel/bpf/bpf_struct_ops.c +3
-6-/* Should be a pointer to struct */ - pointed_type =btf_type_resolve_ptr (btf, - args[arg_no].type, - &arg_btf_id); -if (!pointed_type || - !btf_type_is_struct(pointed_type)) { +/* Should be a pointer to struct, or any pointer for __arena/__arena_nullable */ + pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id); +if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct (pointed_type))) {
pr_warn("stub function %s has %s tagging to an unsupported type\n", stub_fname, suffix);
gotoerr_out;...
Quoted 4 lines
kernel/bpf/trampoline.c +0
-0...
Quoted 9 lines
kernel/bpf/trampoline.c +0
-0@@ -695,6 +743,22 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; }kernel/bpf/trampoline.c +5
-0+/* +* Arena ctx args are converted only by the struct_ops indirect +* trampoline, which dispatches to a single known prog. Generic +* trampolines can mix progs with different arenas, so no conversion +* is possible here. Not reachable today: only struct_ops progs get kernel/bpf/trampoline.c
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+* arena ctx args and they never ride generic trampolines. +*/ +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +if (bpf_prog_has_arena_ctx_arg (tnodes[kind].nodes[i]->link->prog)) { + err = -ENOTSUPP ; + goto out; + } + } + }Nit: should this be done in
__bpf_trampoline_link_prog()orbpf_trampoline_add_prog()? To avoid doing the same check on unlink. Since this is an invariant violation, do we want to addWARN_ONCE?
Quoted 6 lines
Quoted 4 lines
+
/* clear all bits exceptSHARE_IPMODIFYandTAIL_CALL_CTX*/tr->flags&= (BPF_TRAMP_F_SHARE_IPMODIFY|BPF_TRAMP_F_TAIL_CALL_CTX);...
Thinking a bit more about this. Since kfuncs already use flags in btf_func_model, I think it would be nice to use them for struct_ops as well. E.g. as in the patch attached (applied on top of this series). Wdyt?
arch/x86/net/bpf_jit_comp.c +14 -19
@@ -3071,12 +3071,12 @@ static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 basestatic void save_args (const struct btf_func_model *m,u8 **prog,int stack_size,bool for_call_origin,u32 flags, -const struct bpf_tramp_arena_args *aargs) +u64 arena_base) {int arg_regs, first_off =0 , nr_regs =0 , nr_stack_slots =0 ;bool use_jmp =bpf_trampoline_use_jmp (flags);int stack_args_off = (use_jmp || (flags &BPF_TRAMP_F_INDIRECT )) ?16 :24 ; -int i, j, slot =0 ; +int i, j;/* Store function arguments to stack. * For a function that accepts two pointers the sequence will be:
@@ -3084,6 +3084,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,* mov QWORD PTR [rbp-0x8],rsi */ for (i =0 ; i <min_t (int, m->nr_args,MAX_BPF_FUNC_ARGS ); i++) { +bool arena_arg = arena_base && (m->arg_flags[i] &BTF_FMODEL_ARENA_ARG ); +bool nullable = m->arg_flags[i] &BTF_FMODEL_NULLABLE_ARG ; + arg_regs = (m->arg_size[i] +7 ) /8 ;/* According to the research of Yonghong, struct members
@@ -3117,10 +3120,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,for (j =0 ; j < arg_regs; j++) {emit_ldx (prog,BPF_DW ,BPF_REG_0 ,BPF_REG_FP , nr_stack_slots *8 + stack_args_off); -if (aargs && (aargs->slots &BIT (slot))) -emit_arena_arg_conv (prog,BPF_REG_0 , - aargs->nullable_slots &BIT (slot), - (u32 )aargs->kern_vm_start); +if (arena_arg) +emit_arena_arg_conv (prog,BPF_REG_0 , nullable, + (u32 )arena_base);emit_stx (prog,BPF_DW ,BPF_REG_FP ,BPF_REG_0 , -stack_size);
@@ -3128,7 +3130,6 @@ static void save_args(const struct btf_func_model *m, u8 **prog, first_off = stack_size; stack_size -=8 ; nr_stack_slots++; - slot++; } }else {/* Only copy the arguments on-stack to current
@@ -3137,7 +3138,6 @@ static void save_args(const struct btf_func_model *m, u8 **prog,*/ if (for_call_origin) { nr_regs += arg_regs; - slot += arg_regs;continue ; }
@@ -3145,16 +3145,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog,for (j =0 ; j < arg_regs; j++) {u32 src = nr_regs ==5 ?X86_REG_R9 :BPF_REG_1 + nr_regs; -if (aargs && (aargs->slots &BIT (slot))) { -emit_arena_arg_conv (prog, src, - aargs->nullable_slots &BIT (slot), - (u32 )aargs->kern_vm_start); +if (arena_arg) { +emit_arena_arg_conv (prog, src, nullable, (u32 )arena_base); src =BPF_REG_0 ; }emit_stx (prog,BPF_DW ,BPF_REG_FP , src, -stack_size); stack_size -=8 ; nr_regs++; - slot++; } } }
@@ -3445,13 +3442,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imstruct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY ];struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT ];struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN ]; -struct bpf_tramp_arena_args aargs;void *orig_call = func_addr;int cookie_off, cookie_cnt;u8 **branches =NULL ; +u64 arena_base;u64 func_meta;u8 *prog; -bool has_aargs;bool save_ret;/*
@@ -3462,7 +3458,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imWARN_ON_ONCE ((flags &BPF_TRAMP_F_INDIRECT ) && (flags & ~(BPF_TRAMP_F_INDIRECT |BPF_TRAMP_F_RET_FENTRY_RET ))); - has_aargs =bpf_tramp_collect_arena_args (tnodes, flags, &aargs); + arena_base =bpf_tramp_arena_base (m, tnodes, flags);/* extra registers for struct arguments */ for (i =0 ; i < m->nr_args; i++) {
@@ -3601,8 +3597,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imemit_store_stack_imm64 (&prog,BPF_REG_0 , -ip_off, (long )func_addr); } -save_args (m, &prog, regs_off, false, flags, - has_aargs ? &aargs :NULL ); +save_args (m, &prog, regs_off, false, flags, arena_base);if (flags &BPF_TRAMP_F_CALL_ORIG ) {/* arg1: mov rdi, im */
@@ -3644,7 +3639,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imif (flags &BPF_TRAMP_F_CALL_ORIG ) {restore_regs (m, &prog, regs_off); -save_args (m, &prog, arg_stack_off, true, flags,NULL ); +save_args (m, &prog, arg_stack_off, true, flags,0 );if (flags &BPF_TRAMP_F_TAIL_CALL_CTX ) {/* Before calling the original function, load the
include/linux/bpf.h +6 -16
@@ -1293,18 +1293,13 @@ struct bpf_tramp_nodes { };/* -* Which 8-byte ctx slots of a struct_ops trampoline hold arena kernel -* pointers that save_args() converts to the arena pointer form, -* ctx[slot] = (u32)(kaddr - kern_vm_start). +* The arena base against which a struct_ops trampoline converts the +* arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF +* ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline +* converts nothing. */ -struct bpf_tramp_arena_args { -u32 slots; -u32 nullable_slots;/* subset of @slots where NULL is preserved */ -u64 kern_vm_start; -}; - -bool bpf_tramp_collect_arena_args (struct bpf_tramp_nodes *tnodes,u32 flags, -struct bpf_tramp_arena_args *aargs); +u64 bpf_tramp_arena_base (const struct btf_func_model *m, +struct bpf_tramp_nodes *tnodes,u32 flags);struct bpf_tramp_run_ctx ;
@@ -1707,11 +1702,6 @@ struct bpf_ctx_arg_aux {u32 btf_id;u32 ref_id;bool refcounted; -/* -* We don't encode NULL-ness in the type for the program, but still need -* to distinguish it for the purposes of telling JITs what sequence to emit. -*/ -bool arena_nullable; };struct btf_mod_pair {
kernel/bpf/bpf_struct_ops.c +5 -1
@@ -174,6 +174,7 @@ static int prepare_arg_info(struct btf *btf,const char *st_ops_name, const char *member_name, const struct btf_type *func_proto, void *stub_func_addr, + struct btf_func_model *model, struct bpf_struct_ops_arg_info *arg_info) { const struct btf_type *stub_func_proto, *pointed_type;
@@ -288,7 +289,9 @@ static int prepare_arg_info(struct btf *btf,* precision around it, since it has no safety implication. */ info->reg_type =PTR_TO_ARENA ; - info->arena_nullable = is_arena_nullable; + model->arg_flags[arg_no] |=BTF_FMODEL_ARENA_ARG ; +if (is_arena_nullable) + model->arg_flags[arg_no] |=BTF_FMODEL_NULLABLE_ARG ; } info++;
@@ -476,6 +479,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc, stub_func_addr = *(void **)(st_ops->cfi_stubs + moff); err =prepare_arg_info (btf, st_ops->name, mname, func_proto, stub_func_addr, + &st_ops->func_models[i], arg_info + i);if (err) goto errout;
kernel/bpf/trampoline.c +17 -24
57 lines not on this page · raw message
Re: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments
Quoted 94 lines · Eduard Zingerman
Quoted 92 lines · Eduard Zingerman
On Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
Lgtm except for several nits below.
...
Quoted 1 line
kernel/bpf/bpf_struct_ops.c +0
-0...
Quoted 13 lines
kernel/bpf/bpf_struct_ops.c +3
-1@@ -226,26 +229,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf;for (arg_no =0 ; arg_no < nargs; arg_no++) {/* Skip arguments that is not suffixed with -* "__nullable or __ref". +* "__nullable", "__ref", "__arena" or "__arena_nullable". */ is_nullable =btf_param_match_suffix (btf, &stub_args[arg_no],MAYBE_NULL_SUFFIX ); is_refcounted =btf_param_match_suffix (btf, &stub_args[arg_no],REFCOUNTED_SUFFIX ); + is_arena_nullable =btf_param_match_suffix (btf, &stub_args[arg_no], +ARENA_MAYBE_NULL_SUFFIX );kernel/bpf/bpf_struct_ops.c static int prepare_arg_info(struct btf *btf, · line 249
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +5
-0+ is_arena =btf_param_match_suffix (btf, &stub_args[arg_no],ARENA_SUFFIX );if (is_nullable) suffix =MAYBE_NULL_SUFFIX ;else if (is_refcounted) suffix =REFCOUNTED_SUFFIX ; +else if (is_arena_nullable) + suffix =ARENA_MAYBE_NULL_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;As discussed already, that's unfortunate that
p__arenaandp__nullableare valid, butp__arena__nullableis not, substituted byp__arena_nullable. Same goes for global subprogs.Quoted 15 lines
kernel/bpf/bpf_struct_ops.c +3
-6else continue ; -/* Should be a pointer to struct */ - pointed_type =btf_type_resolve_ptr (btf, - args[arg_no].type, - &arg_btf_id); -if (!pointed_type || - !btf_type_is_struct (pointed_type)) { +/* Should be a pointer to struct, or any pointer for __arena/__arena_nullable */ + pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id); +if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct (pointed_type))) {pr_warn ("stub function %s has %s tagging to an unsupported type\n" , stub_fname, suffix); goto err_out;...
Quoted 4 lines
kernel/bpf/trampoline.c +0
-0...
Quoted 9 lines
kernel/bpf/trampoline.c +5
-0@@ -695,6 +743,22 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } +/* +* Arena ctx args are converted only by the struct_ops indirect +* trampoline, which dispatches to a single known prog. Generic +* trampolines can mix progs with different arenas, so no conversion +* is possible here. Not reachable today: only struct_ops progs get kernel/bpf/trampoline.c static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut · line 760
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+* arena ctx args and they never ride generic trampolines. +*/ +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +if (bpf_prog_has_arena_ctx_arg (tnodes[kind].nodes[i]->link->prog)) { + err = -ENOTSUPP ; + goto out; + } + } + }Nit: should this be done in
__bpf_trampoline_link_prog()or
bpf_trampoline_add_prog()? To avoid doing the same check on unlink. Since this is an invariant violation, do we want to addWARN_ONCE?Quoted 4 lines
kernel/bpf/trampoline.c +1
-0+/* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY |BPF_TRAMP_F_TAIL_CALL_CTX );
Quoted 6 lines
Quoted 1 line
...
Thinking a bit more about this. Since kfuncs already use flags in
btf_func_model, I think it would be nice to use them forstruct_opsas well. E.g. as in the patch attached (applied on top of this series). Wdyt?
I like that it relies on existing annotations instead of yet another auxiliary field.
The mapping for arena args to different function models can be confusing though.
For the verfication purpose: both __arena and __arena_nullable maps to KF_ARG_PTR_TO_ARENA | PTR_MAYBE_NULL.
For the JIT purpose: __arena maps to BTF_FMODEL_ARENA_ARG; __arena_nullable maps to BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG.
It might deserve a brief inline comment in get_kfunc_arg_type() saying why both tags map to KF_ARG_PTR_TO_ARENA | PTR_MAYBE_NULL.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 2/9] bpf: Support __arena and __arena_nullable on struct_ops arguments
Quoted 107 lines · Amery Hung
Quoted 94 lines · Eduard Zingerman
Quoted 92 lines · Eduard Zingerman
On Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
Lgtm except for several nits below.
...
Quoted 1 line
kernel/bpf/bpf_struct_ops.c +0
-0...
Quoted 13 lines
kernel/bpf/bpf_struct_ops.c +3
-1@@ -226,26 +229,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf;for (arg_no =0 ; arg_no < nargs; arg_no++) {/* Skip arguments that is not suffixed with -* "__nullable or __ref". +* "__nullable", "__ref", "__arena" or "__arena_nullable". */ is_nullable =btf_param_match_suffix (btf, &stub_args[arg_no],MAYBE_NULL_SUFFIX ); is_refcounted =btf_param_match_suffix (btf, &stub_args[arg_no],REFCOUNTED_SUFFIX ); + is_arena_nullable =btf_param_match_suffix (btf, &stub_args[arg_no], +ARENA_MAYBE_NULL_SUFFIX );kernel/bpf/bpf_struct_ops.c static int prepare_arg_info(struct btf *btf, · line 249
Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +5
-0+ is_arena =btf_param_match_suffix (btf, &stub_args[arg_no],ARENA_SUFFIX );if (is_nullable) suffix =MAYBE_NULL_SUFFIX ;else if (is_refcounted) suffix =REFCOUNTED_SUFFIX ; +else if (is_arena_nullable) + suffix =ARENA_MAYBE_NULL_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;As discussed already, that's unfortunate that
p__arenaandp__nullableare valid, butp__arena__nullableis not, substituted byp__arena_nullable. Same goes for global subprogs.Quoted 15 lines
kernel/bpf/bpf_struct_ops.c +3
-6else continue ; -/* Should be a pointer to struct */ - pointed_type =btf_type_resolve_ptr (btf, - args[arg_no].type, - &arg_btf_id); -if (!pointed_type || - !btf_type_is_struct (pointed_type)) { +/* Should be a pointer to struct, or any pointer for __arena/__arena_nullable */ + pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id); +if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct (pointed_type))) {pr_warn ("stub function %s has %s tagging to an unsupported type\n" , stub_fname, suffix); goto err_out;...
Quoted 4 lines
kernel/bpf/trampoline.c +0
-0...
Quoted 9 lines
kernel/bpf/trampoline.c +5
-0@@ -695,6 +743,22 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } +/* +* Arena ctx args are converted only by the struct_ops indirect +* trampoline, which dispatches to a single known prog. Generic +* trampolines can mix progs with different arenas, so no conversion +* is possible here. Not reachable today: only struct_ops progs get kernel/bpf/trampoline.c static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut · line 760
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+* arena ctx args and they never ride generic trampolines. +*/ +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +if (bpf_prog_has_arena_ctx_arg (tnodes[kind].nodes[i]->link->prog)) { + err = -ENOTSUPP ; + goto out; + } + } + }Nit: should this be done in
__bpf_trampoline_link_prog()or
bpf_trampoline_add_prog()? To avoid doing the same check on unlink. Since this is an invariant violation, do we want to addWARN_ONCE?Quoted 4 lines
kernel/bpf/trampoline.c +1
-0+/* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY |BPF_TRAMP_F_TAIL_CALL_CTX );Quoted 6 lines
Quoted 1 line
...
Thinking a bit more about this. Since kfuncs already use flags in
btf_func_model, I think it would be nice to use them forstruct_opsas well. E.g. as in the patch attached (applied on top of this series). Wdyt?I like that it relies on existing annotations instead of yet another auxiliary field.
The mapping for arena args to different function models can be confusing though.
Quoted 8 lines
For the verfication purpose: both
__arenaand__arena_nullablemaps toKF_ARG_PTR_TO_ARENA|PTR_MAYBE_NULL.
For the JIT purpose:__arenamaps toBTF_FMODEL_ARENA_ARG;__arena_nullablemaps toBTF_FMODEL_ARENA_ARG|BTF_FMODEL_NULLABLE_ARG.It might deserve a brief inline comment in
get_kfunc_arg_type()saying why both tags map toKF_ARG_PTR_TO_ARENA|PTR_MAYBE_NULL.
Yeah I will add comments for this. I adopted Eduard's diff (w/ his SoB) on top of the stack.
Kumar Kartikeya Dwivedi
maintainerPATCH v3 3/9 bpf, x86: JIT __arena kfunc argument rebasing
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Implement arena argument rebasing for kfunc calls on x86. R12 already holds kern_vm_start whenever the prog has an arena, so each tagged argument costs two instructions emitted right before the call:
movl %eN, %eN /* truncate, clear the upper 32 bits */
addq %r12, %rN
A nullable argument tests the truncated value and jumps over the add:
movl %eN, %eN
testl %eN, %eN
jz 1f
addq %r12, %rN
1:
addq carries a REX prefix for every argument register and is always three bytes, so the jz displacement is constant. The sequence is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees the rebase and needs no special handling.
bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well.
--- arch/x86/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
arch/x86/net/bpf_jit_comp.c +50 -0
@@ -1678,6 +1678,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,return 0 ; } +/* +* Rebase the __arena args of a kfunc call to arena kernel addresses, +* rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable +* arg preserves NULL by skipping the add, tested on the truncated value as +* arena NULL is offset 0. Return the number of emitted bytes. +*/ +static int emit_kfunc_arena_args (struct bpf_prog *bpf_prog, +const struct bpf_insn *insn,u8 **pprog) +{ +const struct btf_func_model *fm; +u8 *prog = *pprog; +u8 *start = prog; +int i; + + fm =bpf_jit_find_kfunc_model (bpf_prog, insn); +if (!fm) +return -EINVAL ; + +for (i =0 ; i <min_t (int, fm->nr_args,MAX_BPF_FUNC_REG_ARGS ); i++) { +u8 flags = fm->arg_flags[i]; +u32 reg =BPF_REG_1 + i; + +if (!(flags &BTF_FMODEL_ARENA_ARG )) +continue ; +if (WARN_ON_ONCE (!bpf_prog->aux->arena)) +return -EINVAL ; + +/* mov eN, eN: truncate and clear the upper 32 bits */ +emit_mov_reg (&prog, false, reg, reg); +if (flags &BTF_FMODEL_NULLABLE_ARG ) { +/* test eN, eN; jz over the 3-byte add */ +maybe_emit_mod (&prog, reg, reg, false); +EMIT2 (0x85 ,add_2reg (0xC0 , reg, reg)); +EMIT2 (X86_JE ,3 ); + } +/* add rN, r12 */ +maybe_emit_mod (&prog, reg,X86_REG_R12 , true); +EMIT2 (0x01 ,add_2reg (0xC0 , reg,X86_REG_R12 )); + } + + *pprog = prog; +return prog - start; +} +static int do_jit (struct bpf_verifier_env *env,struct bpf_prog *bpf_prog,int *addrs,u8 *image,u8 *rw_image,int oldproglen,struct jit_context *ctx,bool jmp_padding) {
@@ -2583,6 +2627,12 @@ st: insn_off = insn->off; }if (!imm32)return -EINVAL ; +if (src_reg ==BPF_PSEUDO_KFUNC_CALL ) { + err =emit_kfunc_arena_args (bpf_prog, insn, &prog); +if (err <0 ) +return err; + ip += err; + }if (priv_frame_ptr) {push_r9 (&prog); ip +=2 ;
Signature · 1 line
2.53.0
Re: [PATCH bpf-next v3 3/9] bpf, x86: JIT __arena kfunc argument rebasing
Quoted 22 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Implement arena argument rebasing for kfunc calls on x86. R12 already holds
kern_vm_startwhenever the prog has an arena, so each tagged argument costs two instructions emitted right before the call:movl %eN, %eN /* truncate, clear the upper 32 bits */ addq %r12, %rN
A nullable argument tests the truncated value and jumps over the add:
movl %eN, %eN
testl %eN, %eN
jz 1f
addq %r12, %rN
1:addq carries a REX prefix for every argument register and is always three bytes, so the jz displacement is constant. The sequence is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees the rebase and needs no special handling.
Quoted 6 lines
bpf_jit_supports_arena_args()is not flipped yet; that happens when thestruct_opstrampoline side is in place as well.Signed-off-by Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
...
Kumar Kartikeya Dwivedi
maintainerPATCH v3 4/9 bpf, x86: Convert struct_ops arena arguments in the trampoline
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Implement the struct_ops arena argument conversion on x86. save_args() gains the conversion map from bpf_tramp_collect_arena_args() and, as it copies each native argument into the BPF ctx, routes a marked slot through RAX:
movl %esrc, %eax /* truncate and clear the upper 32 bits */
subl $base_lo, %eax
movq %rax, ctx_slot
A nullable slot tests the full 64-bit kernel pointer first:
movq %rsrc, %rax
testq %rax, %rax
jz 1f
subl $base_lo, %eax
1:
movq %rax, ctx_slot
The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through RAX, so only the subtraction (and the NULL test) is inserted there. The marked slots are tracked with a running slot counter shared by the register and stack branches, matching the flattened ctx offsets in ctx_arg_info. The size probe reruns the same emission with the same tnodes, so the image size matches by construction.
With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for x86.
--- arch/x86/net/bpf_jit_comp.c | 64 +++++++++++++++++++++++++++++++++---- 1 file changed, 57 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +57 -7
@@ -3043,12 +3043,39 @@ static int get_nr_used_regs(const struct btf_func_model *m)return nr_used_regs; } +/* +* Convert an arena kernel address into the arena pointer form on its way +* into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg +* preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit +* subtraction both truncates and clears the upper half, so the stored +* value satisfies the JIT invariant for arena pointer registers. +*/ +static void emit_arena_arg_conv (u8 **pprog,u32 src_reg,bool nullable,u32 base_lo) +{ +u8 *prog = *pprog; + +if (nullable) { +if (src_reg !=BPF_REG_0 ) +emit_mov_reg (&prog, true,BPF_REG_0 , src_reg); +/* test rax, rax; jz over the 5-byte sub */ +EMIT3 (0x48 ,0x85 ,0xC0 ); +EMIT2 (X86_JE ,5 ); + }else if (src_reg !=BPF_REG_0 ) { +emit_mov_reg (&prog, false,BPF_REG_0 , src_reg); + } +/* sub eax, base_lo */ +EMIT1_off32 (0x2D , base_lo); + + *pprog = prog; +} +static void save_args (const struct btf_func_model *m,u8 **prog, -int stack_size,bool for_call_origin,u32 flags) +int stack_size,bool for_call_origin,u32 flags, +const struct bpf_tramp_arena_args *aargs) {int arg_regs, first_off =0 , nr_regs =0 , nr_stack_slots =0 ;bool use_jmp =bpf_trampoline_use_jmp (flags); -int i, j; +int i, j, slot =0 ;/* Store function arguments to stack. * For a function that accepts two pointers the sequence will be:
@@ -3089,6 +3116,10 @@ static void save_args(const struct btf_func_model *m, u8 **prog,for (j =0 ; j < arg_regs; j++) {emit_ldx (prog,BPF_DW ,BPF_REG_0 ,BPF_REG_FP , nr_stack_slots *8 +16 + (!use_jmp) *8 ); +if (aargs && (aargs->slots &BIT (slot))) +emit_arena_arg_conv (prog,BPF_REG_0 , + aargs->nullable_slots &BIT (slot), + (u32 )aargs->kern_vm_start);emit_stx (prog,BPF_DW ,BPF_REG_FP ,BPF_REG_0 , -stack_size);
@@ -3096,6 +3127,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog, first_off = stack_size; stack_size -=8 ; nr_stack_slots++; + slot++; } }else {/* Only copy the arguments on-stack to current
@@ -3104,16 +3136,24 @@ static void save_args(const struct btf_func_model *m, u8 **prog,*/ if (for_call_origin) { nr_regs += arg_regs; + slot += arg_regs;continue ; }/* copy the arguments from regs into stack */ for (j =0 ; j < arg_regs; j++) { -emit_stx (prog,BPF_DW ,BPF_REG_FP , - nr_regs ==5 ?X86_REG_R9 :BPF_REG_1 + nr_regs, - -stack_size); +u32 src = nr_regs ==5 ?X86_REG_R9 :BPF_REG_1 + nr_regs; + +if (aargs && (aargs->slots &BIT (slot))) { +emit_arena_arg_conv (prog, src, + aargs->nullable_slots &BIT (slot), + (u32 )aargs->kern_vm_start); + src =BPF_REG_0 ; + } +emit_stx (prog,BPF_DW ,BPF_REG_FP , src, -stack_size); stack_size -=8 ; nr_regs++; + slot++; } } }
@@ -3404,11 +3444,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imstruct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY ];struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT ];struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN ]; +struct bpf_tramp_arena_args aargs;void *orig_call = func_addr;int cookie_off, cookie_cnt;u8 **branches =NULL ;u64 func_meta;u8 *prog; +bool has_aargs;bool save_ret;/*
@@ -3419,6 +3461,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imWARN_ON_ONCE ((flags &BPF_TRAMP_F_INDIRECT ) && (flags & ~(BPF_TRAMP_F_INDIRECT |BPF_TRAMP_F_RET_FENTRY_RET ))); + has_aargs =bpf_tramp_collect_arena_args (tnodes, flags, &aargs); +for (i =0 ; i < m->nr_args; i++) nr_regs += (m->arg_size[i] +7 ) /8 -1 ;
@@ -3553,7 +3597,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imemit_store_stack_imm64 (&prog,BPF_REG_0 , -ip_off, (long )func_addr); } -save_args (m, &prog, regs_off, false, flags); +save_args (m, &prog, regs_off, false, flags, + has_aargs ? &aargs :NULL );if (flags &BPF_TRAMP_F_CALL_ORIG ) {/* arg1: mov rdi, im */
@@ -3595,7 +3640,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imif (flags &BPF_TRAMP_F_CALL_ORIG ) {restore_regs (m, &prog, regs_off); -save_args (m, &prog, arg_stack_off, true, flags); +save_args (m, &prog, arg_stack_off, true, flags,NULL );if (flags &BPF_TRAMP_F_TAIL_CALL_CTX ) {/* Before calling the original function, load the
@@ -4096,6 +4141,11 @@ bool bpf_jit_supports_stack_args(void)return true; } +bool bpf_jit_supports_arena_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (text_poke_copy (dst, src, len) ==NULL )
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH v3 5/9 selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar), the exact rebase semantics via capture kfuncs returning the raw argument (zero low 32 bits arrive as the arena kernel base under __arena and as NULL under __arena_nullable), a NULL round trip through a nullable deref kfunc, five arena arguments in one call, a mixed __arena plus __arena_nullable call exercising both bitmasks on one call site, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type).
The tests run on x86-64 and skip elsewhere, as programs with arena-tagged kfunc args fail verification where the JIT lacks support.
--- .../selftests/bpf/prog_tests/arena_kfunc.c | 15 ++ .../testing/selftests/bpf/progs/arena_kfunc.c | 250 ++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 57 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 10 + 4 files changed, 332 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> + +#include "arena_kfunc.skel.h" + +/* +* The test kfuncs live in bpf_testmod. Resolving kfuncs against module +* BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of +* through the verifier tests' capability-restricted runner. +*/ +void test_arena_kfunc (void ) +{ +RUN_TESTS (arena_kfunc); +}
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +250 -0
250 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +57 -0
69 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +10 -0
@@ -98,6 +98,16 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym; +__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_nullable_arg_test(__u64 *val__arena_nullable) __ksym; +__u64 bpf_kfunc_arena_cap_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_cap_nullable_test(__u64 *val__arena_nullable) __ksym; +__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena, + __u64 *c__arena, __u64 *d__arena, + __u64 *e__arena_nullable) __ksym; +__u64 bpf_kfunc_arena_stack_arg_test(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + __u64 *f__arena) __ksym; +__u64 bpf_kfunc_arena_mixed_test(__u64 *a__arena, __u64 *b__arena_nullable) __ksym;int *bpf_kfunc_call_test_get_rdwr_mem (struct prog_test_ref_kfunc *p,const int rdwr_buf_size) __ksym;int *bpf_kfunc_call_test_get_rdonly_mem (struct prog_test_ref_kfunc *p,const int rdonly_buf_size) __ksym;int *bpf_kfunc_call_test_acq_rdonly_mem (struct prog_test_ref_kfunc *p,const int rdonly_buf_size) __ksym;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `arena_arg_stack` test incorrectly wraps its failure-inducing logic in an `#if…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The arena_arg_stack test incorrectly wraps its failure-inducing logic in an #if defined(__BPF_FEATURE_ADDR_SPACE_CAST) block, which causes the test to falsely fail on compilers without this feature. --
--- Patch [5]: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests --- commit b62ba0a2046a4e44e0da5d972133d046f475c161
Author: Tejun Heo <[email protected]>
selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
This commit adds arena-argument kfuncs to bpf_testmod and a corresponding suite of BPF programs to test various argument forms and rebase semantics. The tests are intended to run on x86-64 where JIT support exists.
Quoted 5 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/arena_kfunc.c line 246
Quoted 9 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +9
-0+SEC("syscall") +__arch_x86_64 +__failure __msg("arena pointer cannot be a stack argument") +int arena_arg_stack (void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) +bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +bpf_kfunc_arena_stack_arg_test (1 ,2 ,3 ,4 ,5 , (u64 *)1 ); +#endif
[Severity: Medium]
Will this test incorrectly fail on toolchains without __BPF_FEATURE_ADDR_SPACE_CAST?
When the feature is missing, the #if block evaluates to false, leaving an empty program that simply returns 0. The BPF verifier will accept this empty program, but because the test is annotated with __failure __msg(...), the test harness expects the verifier to reject it.
Since verification succeeds in this case, does this cause the test harness to flag it as a spurious test failure during CI or user builds?
Quoted 4 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +4
-0+return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
In reply to Kumar Kartikeya DwivediOn Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 6 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +6
-0+SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_forms(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
tools/testing/selftests/bpf/progs/arena_kfunc.c line 43
Quoted 10 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +10
-0+ u64 __arena *val; +u64 ret; + + val =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +if (!val) +return 1 ; + +/* PTR_TO_ARENA argument */ + *val =41 ; + ret =bpf_kfunc_arena_arg_test ((u64 *)val);
Thinking a bit more about the PTR_TO_ARENA vs SCALAR_VALUE in the patch #1. Looking the test cases added, the use case is to pass pointers annotated with __attribute__((address_space(1))) to such functions, e.g. obtained as:
- a return value from bpf_arena_alloc_pages
- or a nested pointer in some arena located structure:
struct foo {
int __arena *ptr; };
...
sturct foo __arena *foo = ...;
bpf_kfunc_arena_arg_test(foo->ptr);
The way these kfuncs are currently generated in vmlinux.h, they lack the address space annotation, and hence you need to explicitly do the casting, as here:
ret = bpf_kfunc_arena_arg_test((u64 *)val);
This is both inconvenient *and* adds a useless address space cast instruction (translation of the cast).
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having __arena annotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).
- In patch #1, it appears that there is no need to allow PTR_TO_ARENA for parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.
...
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Quoted 34 lines · Eduard Zingerman
On Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 6 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +6
-0+SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_forms(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)tools/testing/selftests/bpf/progs/arena_kfunc.c line 43
Quoted 10 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +10
-0+ u64 __arena *val; +u64 ret; + + val =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +if (!val) +return 1 ; + +/* PTR_TO_ARENA argument */ + *val =41 ; + ret =bpf_kfunc_arena_arg_test ((u64 *)val);Thinking a bit more about the
PTR_TO_ARENAvsSCALAR_VALUEin the patch #1. Looking the test cases added, the use case is to pass pointers annotated with __attribute__((address_space(1))) to such functions, e.g. obtained as:
- a return value frombpf_arena_alloc_pages
- or a nested pointer in some arena located structure:struct foo { int __arena *ptr;};
...
sturct foo__arena*foo = ...;
bpf_kfunc_arena_arg_test(foo->ptr);
Quoted 5 lines
The way these kfuncs are currently generated in
vmlinux.h, they lack the address space annotation, and hence you need to explicitly do the casting, as here:ret = bpf_kfunc_arena_arg_test((u64 *)val);
For now they are in bpf_testmod, but your point stands regardless.
Quoted 9 lines
This is both inconvenient *and* adds a useless address space cast instruction (translation of the cast).
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).
Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until resolve_btfids changes are done. That said, I can manually add the annotation for now.
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.
This one I don't understand. Why wouldn't we get PTR_TO_ARENA typed paramters as arguments?
Quoted 2 lines
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
On Tue, 2026-08-04 at 22:14 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
Quoted 5 lines
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until
resolve_btfidschanges are done. That said, I can manually add the annotation for now.
We can fast-track resolve_btfids.
Quoted 6 lines
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.This one I don't understand. Why wouldn't we get
PTR_TO_ARENAtyped paramters as arguments?
(u64 *)val is translated as an address space cast instruction. This instruction yields PTR_TO_ARENA. Except for explicit casts, how else can you pass PTR_TO_ARENA to such a function?
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Quoted 20 lines · Eduard Zingerman
On Tue, 2026-08-04 at 22:14 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
Quoted 5 lines
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until
resolve_btfidschanges are done. That said, I can manually add the annotation for now.We can fast-track
resolve_btfids.Quoted 4 lines
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.
Quoted 6 lines
Quoted 2 lines
This one I don't understand. Why wouldn't we get
PTR_TO_ARENAtyped paramters as arguments?
(u64 *)valis translated as an address space cast instruction. This instruction yieldsPTR_TO_ARENA. Except for explicit casts, how else can you passPTR_TO_ARENAto such a function?
For example, foo->ptr yields a scalar from the verifier point of view.
Quoted 2 lines
...
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Quoted 21 lines · Eduard Zingerman
Quoted 20 lines · Eduard Zingerman
On Tue, 2026-08-04 at 22:14 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
Quoted 5 lines
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until
resolve_btfidschanges are done. That said, I can manually add the annotation for now.We can fast-track
resolve_btfids.Quoted 4 lines
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
This one I don't understand. Why wouldn't we get
PTR_TO_ARENAtyped paramters as arguments?
(u64 *)valis translated as an address space cast instruction. This instruction yieldsPTR_TO_ARENA. Except for explicit casts, how else can you passPTR_TO_ARENAto such a function?For example,
foo->ptryields a scalar from the verifier point of view.
Sure, but I don't think we should reject the case where someone does explicit cast. See e.g. the global function __arena tag, which also accepts both cases.
Quoted 2 lines
Quoted 2 lines
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Quoted 29 lines · Kumar Kartikeya Dwivedi
Quoted 28 lines · Eduard Zingerman
Quoted 26 lines · Eduard Zingerman
On Tue, 2026-08-04 at 22:14 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
Quoted 5 lines
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until
resolve_btfidschanges are done. That said, I can manually add the annotation for now.We can fast-track
resolve_btfids.Quoted 6 lines
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.This one I don't understand. Why wouldn't we get
PTR_TO_ARENAtyped paramters as arguments?
(u64 *)valis translated as an address space cast instruction. This instruction yieldsPTR_TO_ARENA. Except for explicit casts, how else can you passPTR_TO_ARENAto such a function?
Quoted 5 lines
Quoted 2 lines
For example,
foo->ptryields a scalar from the verifier point of view.Sure, but I don't think we should reject the case where someone does explicit cast. See e.g. the global function
__arenatag, which also accepts both cases.
PTR_TO_ARENA is already a valid kernel space pointer, it does not need additional conversion.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 5/9] selftests/bpf: Add kfunc __arena and __arena_nullable argument tests
Quoted 30 lines · Eduard Zingerman
Quoted 29 lines · Kumar Kartikeya Dwivedi
Quoted 28 lines · Eduard Zingerman
Quoted 26 lines · Eduard Zingerman
On Tue, 2026-08-04 at 22:14 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
Quoted 5 lines
Which leads to several conclusions:
- We should reach a consensus regarding KF_ARENA* usage, as these kfuncs would clearly benefit from having__arenaannotations on parameters (on the BPF side, on the host side these should be declared as plain pointers).Yes, but even if we do, for the time begin, we won't have support for automatically emitting it until
resolve_btfidschanges are done. That said, I can manually add the annotation for now.We can fast-track
resolve_btfids.Quoted 6 lines
Quoted 3 lines
- In patch #1, it appears that there is no need to allow
PTR_TO_ARENAfor parameters passed to such a function. At-least I don't see a C code not involving manual casts that would require such support.This one I don't understand. Why wouldn't we get
PTR_TO_ARENAtyped paramters as arguments?
(u64 *)valis translated as an address space cast instruction. This instruction yieldsPTR_TO_ARENA. Except for explicit casts, how else can you passPTR_TO_ARENAto such a function?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
For example,
foo->ptryields a scalar from the verifier point of view.Sure, but I don't think we should reject the case where someone does explicit cast. See e.g. the global function
__arenatag, which also accepts both cases.
PTR_TO_ARENAis already a valid kernel space pointer, it does not need additional conversion.
Nah, that's not true. It is still a 32-bit value.
Kumar Kartikeya Dwivedi
maintainerPATCH v3 6/9 selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Pin the exact rebase sequences the JITs emit for __arena and __arena_nullable kfunc arguments with __jited assertions on x86-64: the unconditional truncate-and-add, the nullable test-and-skip variant, and all five argument registers in one call, which also covers the REX-prefixed encoding of r8 on x86. The capture kfuncs take the argument without dereferencing, so only the emitted code is under test. The tests skip without LLVM disassembler support.
--- .../bpf/prog_tests/arena_kfunc_jit.c | 13 +++ .../selftests/bpf/progs/arena_kfunc_jit.c | 96 +++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc_jit.c
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +13 -0
@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include "arena_kfunc_jit.skel.h" + +/* +* Runs with full capabilities: resolving module kfunc ksyms requires +* CAP_SYS_ADMIN, which rules out the capability-restricted runner. +*/ +void test_arena_kfunc_jit (void ) +{ +RUN_TESTS (arena_kfunc_jit); +}
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c new file +96 -0
96 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 6/9] selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing `env.has_testmod` check can cause test failures when `bpf_testmod.ko` is not loaded. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing env.has_testmod check can cause test failures when bpf_testmod.ko is not loaded. --
--- Patch [6]: [PATCH bpf-next v3 6/9] selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments --- commit 9c67c6873243ff0c5bbb155714a37d780a229111
Author: Tejun Heo <[email protected]>
selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments
This commit pins the exact rebase sequences the JITs emit for __arena and __arena_nullable kfunc arguments with __jited assertions on x86-64.
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +2
-0@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c line 12
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +10
-0+#include <test_progs.h> +#include "arena_kfunc_jit.skel.h" + +/* +* Runs with full capabilities: resolving module kfunc ksyms requires +* CAP_SYS_ADMIN, which rules out the capability-restricted runner. +*/ +void test_arena_kfunc_jit (void ) +{ +RUN_TESTS (arena_kfunc_jit);
[Severity: Medium]
Will this cause test failures in environments where bpf_testmod.ko is not loaded?
Since the loaded BPF programs depend on kfuncs (e.g., bpf_kfunc_arena_cap_test) from bpf_testmod.ko, if the module is absent, kfunc resolution fails and causes program load failures. Because the programs are annotated with __success, test_loader will record these as test failures instead of skipping them.
Should there be an env.has_testmod check before calling RUN_TESTS() to skip the test cleanly?
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +1
-0+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Eduard Zingerman
maintainerRe: [PATCH bpf-next v3 6/9] selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments
In reply to Kumar Kartikeya DwivediOn Mon, 2026-08-03 at 14:51 +0200, Kumar Kartikeya Dwivedi wrote:
...
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +3
-0@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h>
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c line 13
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc_jit.c new file +10
-0+#include "arena_kfunc_jit.skel.h" + +/* +* Runs with full capabilities: resolving module kfunc ksyms requires +* CAP_SYS_ADMIN, which rules out the capability-restricted runner. +*/ +void test_arena_kfunc_jit (void ) +{ +RUN_TESTS (arena_kfunc_jit); +}
Nit: please avoid adding this file, and instead put this
function under prog_tests/verifier.c.
There are multiple tests there that invoke RUN_TESTS instead of RUN for same exact reason.
Same applies to the previous patch.
Quoted 1 line
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c +0
-0
...
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c line 50
Quoted 8 lines
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c +8
-0+SEC("syscall") +__arch_x86_64 +__jited("...") +__jited(" movl %edi, %edi") +__jited(" testl %edi, %edi") +__jited(" je {{.*}}") +__jited(" addq %r12, %rdi") +__success
Note that __jited output generates labels like this:
50: 89 ff movl %edi, %edi 52: 85 ff testl %edi, %edi 54: 74 03 je L0 56: 4c 01 e7 addq %r12, %rdi 59: e8 42 5f 40 00 L0: callq 0x405fa0
I think it would make sense to match L0 in this test.
Quoted 6 lines
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c +6
-0+int arena_arg_jit_nullable (void *ctx) +{ + stash = (u64 )bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +bpf_kfunc_arena_cap_nullable_test ((u64 *)stash); +return 0 ; +}
...
Kumar Kartikeya Dwivedi
maintainerPATCH v3 7/9 selftests/bpf: Add struct_ops __arena and __arena_nullable argument tests
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Add test_arena and test_arena_nullable members to bpf_testmod_ops3 with arena-tagged stub arguments and kfuncs that forward a caller-provided pointer to them. The kfuncs take arena-tagged arguments, so each round trip exercises both conversion directions end to end: the kfunc receives a kernel arena address and the trampoline converts it back to an arena pointer for the callback.
The non-nullable callback dereferences its argument with no NULL branch and captures the raw ctx value, which the trigger program compares against the arena offset of the passed object, pinning the exact (u32)(kaddr - kern_vm_start) conversion. The nullable callback verifies that only a true kernel NULL arrives as NULL. Failure coverage: a program with no arena is rejected when it loads. The tests run on x86-64 and skip elsewhere, as the programs fail verification where the JIT lacks arena argument support.
--- .../bpf/prog_tests/test_struct_ops_arena.c | 74 +++++++++++++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 14 ++- .../selftests/bpf/progs/struct_ops_arena.c | 94 +++++++++++++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 ++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 24 +++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 3 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 2 + 7 files changed, 229 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +74 -0
@@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> + +#include "struct_ops_arena.skel.h" +#include "struct_ops_arena_fail.skel.h" + +#if defined(__x86_64__) +/* +* Attach callbacks with __arena and __arena_nullable arguments and drive +* them through the bpf_testmod_ops3_call_test_arena*() kfuncs. +*/ +static void arena_arg (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +struct struct_ops_arena *skel; +struct bpf_link *link =NULL ; +int err; + + skel =struct_ops_arena__open_and_load (); +if (!ASSERT_OK_PTR (skel,"struct_ops_arena__open_and_load" )) +return ; + + link =bpf_map__attach_struct_ops (skel->maps.testmod_arena); +if (!ASSERT_OK_PTR (link,"attach_struct_ops" )) + goto out; + + err =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.trigger), + &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,0 ,"trigger_retval" ); + +out: +bpf_link__destroy (link); +struct_ops_arena__destroy (skel); +} + +/* +* A program with no arena cannot attach to a member with an __arena +* argument. +*/ +static void arena_arg_fail (void ) +{ +struct struct_ops_arena_fail *skel; + + skel =struct_ops_arena_fail__open_and_load (); +if (ASSERT_ERR_PTR (skel,"struct_ops_arena_fail__open_and_load" )) +return ; + +struct_ops_arena_fail__destroy (skel); +} +#endif + +/* +* Serialized because it attaches the singleton bpf_testmod_ops3, which +* test_struct_ops_private_stack also attaches; registering it twice fails +* with -EEXIST. +*/ +void serial_test_struct_ops_arena (void ) +{ +/* +* Arena struct_ops arguments need JIT support, currently x86-64 only. +* Elsewhere verification fails with "JIT does not support arena +* arguments", so the programs cannot even load. +*/ +#if defined(__x86_64__) +if (test__start_subtest ("arena_arg" )) +arena_arg (); +if (test__start_subtest ("arena_arg_fail" )) +arena_arg_fail (); +#else +test__skip (); +#endif +}
tools/testing/selftests/bpf/progs/arena_kfunc.c +12 -2
@@ -235,16 +235,26 @@ int arena_arg_bad_reg(void *ctx)return 0 ; } +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT) SEC("syscall") __arch_x86_64 __failure __msg("arena pointer cannot be a stack argument")int arena_arg_stack (void *ctx) { -#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 );bpf_kfunc_arena_stack_arg_test (1 ,2 ,3 ,4 ,5 , (u64 *)1 ); -#endif return 0 ; } +#else +SEC("syscall") +__arch_x86_64 +__description("arena_arg_stack: not supported, dummy test") +__success +int arena_arg_stack (void *ctx) +{ +return 0 ; +} +#endif char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/struct_ops_arena.c new file +94 -0
94 lines not on this page · raw message
tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod.h" + +char _license[] SEC("license") = "GPL"; + +/* No arena in the program: attaching to test_arena must be rejected. */ +SEC("struct_ops/test_arena") +int test_arena_no_arena (unsigned long long *ctx) +{ +return 0 ; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena_fail = { + .test_arena = (void *)test_arena_no_arena, +};
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +24 -0
@@ -397,9 +397,21 @@ static int bpf_testmod_test_4(void)return 0 ; } +static int bpf_testmod_ops3__test_arena (u64 *ptr__arena) +{ +return 0 ; +} + +static int bpf_testmod_ops3__test_arena_nullable (u64 *ptr__arena_nullable) +{ +return 0 ; +} +static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, + .test_arena = bpf_testmod_ops3__test_arena, + .test_arena_nullable = bpf_testmod_ops3__test_arena_nullable, };static void bpf_testmod_test_struct_ops3 (void )
@@ -418,6 +430,16 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) st_ops3->test_2 (); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena) +{ +return st_ops3->test_arena (ptr__arena); +} + +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena_nullable) +{ +return st_ops3->test_arena_nullable (ptr__arena_nullable); +} +struct bpf_testmod_btf_type_tag_1 {int a; };
@@ -827,6 +849,8 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable)BTF_ID_FLAGS (func, bpf_kfunc_get_default_trusted_ptr_test);BTF_ID_FLAGS (func, bpf_kfunc_put_default_trusted_ptr_test);BTF_KFUNCS_END (bpf_testmod_common_kfunc_ids)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +3 -0
@@ -106,6 +106,9 @@ struct bpf_testmod_ops2 {struct bpf_testmod_ops3 {int (*test_1)(void );int (*test_2)(void ); +/* Used to test arena pointer arguments. */ +int (*test_arena)(u64 *ptr); +int (*test_arena_nullable)(u64 *ptr); };struct st_ops_args {
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +2 -0
@@ -121,6 +121,8 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;#endif void bpf_testmod_test_mod_kfunc(int i) __ksym; +int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; +int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena_nullable) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 7/9] selftests/bpf: Add struct_ops __arena and __arena_nullable argument tests
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Unconditional dereference of global pointer st_ops3 in new kfuncs can lead to kernel panic if…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unconditional dereference of global pointer st_ops3 in new kfuncs can lead to kernel panic if called before struct_ops attachment. --
--- Patch [7]: [PATCH bpf-next v3 7/9] selftests/bpf: Add struct_ops __arena and __arena_nullable argument tests --- commit f599951dcb563bef61136071f8ea610da5c68910
Author: Tejun Heo <[email protected]>
selftests/bpf: Add struct_ops __arena and __arena_nullable argument tests
This commit adds test_arena and test_arena_nullable members to the bpf_testmod_ops3 struct to test arena-tagged arguments and kfuncs that forward caller-provided pointers. It ensures that the kfuncs correctly receive a kernel arena address and the trampoline converts it back to an arena pointer.
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0
-0
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) · line 435
Quoted 7 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +3
-0@@ -418,6 +430,16 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) st_ops3->test_2 (); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena) +{ +return st_ops3->test_arena (ptr__arena);
[Severity: Medium]
Can this unconditionally dereference st_ops3 without synchronization?
If a BPF program calls bpf_testmod_ops3_call_test_arena() before the bpf_testmod_ops3 struct_ops map is attached, or after it has been detached, st_ops3 would be NULL. Could this lead to a kernel panic?
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) · line 440
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +5
-0+} + +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena_nullable) +{ +return st_ops3->test_arena_nullable (ptr__arena_nullable);
[Severity: Medium]
Similarly, does this have the same risk of a NULL pointer dereference if st_ops3 is NULL when bpf_testmod_ops3_call_test_arena_nullable() is called?
Quoted 2 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +2
-0+} +
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Re: [PATCH bpf-next v3 7/9] selftests/bpf: Add struct_ops __arena and __arena_nullable argument tests
In reply to Kumar Kartikeya DwivediQuoted 18 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Add
test_arenaandtest_arena_nullablemembers tobpf_testmod_ops3with arena-tagged stub arguments and kfuncs that forward a caller-provided pointer to them. The kfuncs take arena-tagged arguments, so each round trip exercises both conversion directions end to end: the kfunc receives a kernel arena address and the trampoline converts it back to an arena pointer for the callback.The non-nullable callback dereferences its argument with no NULL branch and captures the raw ctx value, which the trigger program compares against the arena offset of the passed object, pinning the exact (u32)(kaddr -
kern_vm_start) conversion. The nullable callback verifies that only a true kernel NULL arrives as NULL. Failure coverage: a program with no arena is rejected when it loads. The tests run on x86-64 and skip elsewhere, as the programs fail verification where the JIT lacks arena argument support.
Quoted 3 lines
Signed-off-by Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
...
Quoted 21 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c +0
-0@@ -235,16 +235,26 @@ int arena_arg_bad_reg(void *ctx) return 0; }tools/testing/selftests/bpf/progs/arena_kfunc.c +2
-0+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)SEC("syscall")
__arch_x86_64
__failure__msg("arena pointer cannot be a stack argument")intarena_arg_stack(void *ctx)
{
-#ifdefined(__BPF_FEATURE_ADDR_SPACE_CAST)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);bpf_kfunc_arena_stack_arg_test(1, 2, 3, 4, 5, (u64 *)1); -#endif
return 0;
}
tools/testing/selftests/bpf/progs/arena_kfunc.c int arena_arg_bad_reg(void *ctx) · line 258
Quoted 10 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c +10
-0+#else +SEC("syscall") +__arch_x86_64 +__description("arena_arg_stack: not supported, dummy test") +__success +int arena_arg_stack (void *ctx) +{ +return 0 ; +} +#endif
Nit: should this be a part of the first selftests patch?
...
Kumar Kartikeya Dwivedi
maintainerPATCH v3 8/9 bpf, x86: Fix stack-passed arguments for indirect trampolines
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
save_args() reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of a struct_ops callback with more than six argument slots is read one slot off.
This has gone unnoticed because no in-tree struct_ops member passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treat BPF_TRAMP_F_INDIRECT the same way.
--- arch/x86/net/bpf_jit_comp.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +8 -7
@@ -3075,6 +3075,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog, {int arg_regs, first_off =0 , nr_regs =0 , nr_stack_slots =0 ;bool use_jmp =bpf_trampoline_use_jmp (flags); +int stack_args_off = (use_jmp || (flags &BPF_TRAMP_F_INDIRECT )) ?16 :24 ;int i, j, slot =0 ;/* Store function arguments to stack.
@@ -3106,16 +3107,16 @@ static void save_args(const struct btf_func_model *m, u8 **prog,/* copy function arguments from origin stack frame * into current stack frame. * -* The starting address of the arguments on-stack -* is: -* rbp + 8(push rbp) + -* 8(return addr of origin call) + -* 8(return addr of the caller) -* which means: rbp + 24 +* The arguments on-stack start above the saved rbp +* and the return addresses: two return addresses +* (origin call and caller) when the trampoline is +* entered through the fentry call, so rbp + 24, and +* a single one when it is entered with a jmp or +* called indirectly, so rbp + 16. */ for (j =0 ; j < arg_regs; j++) {emit_ldx (prog,BPF_DW ,BPF_REG_0 ,BPF_REG_FP , - nr_stack_slots *8 +16 + (!use_jmp) *8 ); + nr_stack_slots *8 + stack_args_off);if (aargs && (aargs->slots &BIT (slot))) emit_arena_arg_conv(prog, BPF_REG_0, aargs->nullable_slots & BIT(slot),
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v3 8/9] bpf, x86: Fix stack-passed arguments for indirect trampolines
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Medium] The fix for stack-passed arguments in indirect trampolines was only applied to x86, leaving identical bugs in ARM64, RISC-V, and LoongArch. --
--- Patch [8]: [PATCH bpf-next v3 8/9] bpf, x86: Fix stack-passed arguments for indirect trampolines --- commit 010f92af9c8f5bd1141eaf8fc752e0f9df18e07d
Author: Tejun Heo <[email protected]>
bpf, x86: Fix stack-passed arguments for indirect trampolines
This commit fixes the stack offset for arguments passed on the stack when entering an indirect trampoline on x86. The adjustment accounts for the fact that only the caller's return address is on the stack, not the origin function's return address.
arch/x86/net/bpf_jit_comp.c static void save_args(const struct btf_func_model *m, u8 **prog, · line 3078
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +1
-0@@ -3075,6 +3075,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog, {int arg_regs, first_off =0 , nr_regs =0 , nr_stack_slots =0 ;bool use_jmp =bpf_trampoline_use_jmp (flags); +int stack_args_off = (use_jmp || (flags &BPF_TRAMP_F_INDIRECT )) ?16 :24 ;
[Severity: Medium]
This is a pre-existing issue, but since we are fixing the stack-passed arguments for indirect trampolines on x86 here, should we also address the same issue on other architectures?
For instance, on ARM64, in save_args() in arch/arm64/net/bpf_jit_comp.c, the offset is hardcoded assuming both parent and patched function frames were pushed:
soff = 32; /* on stack arguments start from FP + 32 */
For indirect trampolines, only one frame is pushed, so the arguments would start at FP + 16.
Similarly, in RISC-V's store_args() in arch/riscv/net/bpf_jit_comp64.c:
emit_ld(RV_REG_T1, 16 + (i - RV_MAX_REG_ARGS) * 8, RV_REG_FP, ctx);
And in LoongArch's store_args() in arch/loongarch/net/bpf_jit.c:
emit_insn(ctx, ldd, LOONGARCH_GPR_T1, LOONGARCH_GPR_FP,
16 + (i - LOONGARCH_MAX_REG_ARGS) * 8);
Both of these hardcode the offset relative to FP, ignoring that FP is aligned with SP for indirect trampolines.
If a struct_ops callback is invoked with enough arguments to force stack-passing on these architectures, will they read garbage values instead of the actual arguments?
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +0
-0int i, j, slot =0 ;/* Store function arguments to stack.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [PATCH bpf-next v3 8/9] bpf, x86: Fix stack-passed arguments for indirect trampolines
In reply to Kumar Kartikeya DwivediQuoted 15 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
save_args()reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of astruct_opscallback with more than six argument slots is read one slot off.This has gone unnoticed because no in-tree
struct_opsmember passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treatBPF_TRAMP_F_INDIRECTthe same way.
Quoted 5 lines
Fixes 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING")Cc Jiri Olsa <[email protected]>Signed-off-by Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
...
Kumar Kartikeya Dwivedi
maintainerPATCH v3 9/9 selftests/bpf: Test stack-passed struct_ops arena arguments
In reply to Kumar Kartikeya DwivediFrom: Tejun Heo <[email protected]>
Add a test_arena_stack member with eight leading scalar arguments so the arena pointer is passed on the stack.
The callback validates the first and last scalar ctx slots before dereferencing the pointer in ctx[8]. This exercises the indirect trampoline stack layout and arena conversion together, and prevents a regression where stack arguments are read one slot late.
--- .../selftests/bpf/progs/struct_ops_arena.c | 21 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 14 +++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 3 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 1 + 4 files changed, 39 insertions(+)
tools/testing/selftests/bpf/progs/struct_ops_arena.c +21 -0
@@ -46,10 +46,24 @@ int test_arena_nullable_cb(unsigned long long *ctx)return 0 ; } +SEC("struct_ops/test_arena_stack") +int test_arena_stack_cb (unsigned long long *ctx) +{ + u64 __arena *ptr = (u64 __arena *)ctx[8]; + + arena_touch++; +/* pin the slot layout: the leading args fill ctx[0]..ctx[7] */ +if (ctx[0 ] !=1 || ctx[7 ] !=8 ) +return 0xbad ; + *ptr +=1 ; +return 0 ; +} + SEC(".struct_ops.link")struct bpf_testmod_ops3 testmod_arena = { .test_arena = (void *)test_arena_cb, .test_arena_nullable = (void *)test_arena_nullable_cb, + .test_arena_stack = (void *)test_arena_stack_cb, };SEC ("syscall" )
@@ -88,6 +102,13 @@ int trigger(void *ctx)if (ret !=0xbee )return 7 ; +/* the arena pointer is stack-passed into the trampoline here */ + ret =bpf_testmod_ops3_call_test_arena_stack ((u64 *)val); +if (ret) +return 8 ; +if (*val !=44 ) +return 9 ; + bpf_arena_free_pages(&arena, (void __arena *)val, 1);#endif return 0 ;
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14 -0
@@ -407,11 +407,19 @@ static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena_nullable)return 0 ; } +static int bpf_testmod_ops3__test_arena_stack (u64 a,u64 b,u64 c,u64 d, +u64 e,u64 f,u64 g,u64 h, +u64 *ptr__arena) +{ +return 0 ; +} +static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, .test_arena = bpf_testmod_ops3__test_arena, .test_arena_nullable = bpf_testmod_ops3__test_arena_nullable, + .test_arena_stack = bpf_testmod_ops3__test_arena_stack, };static void bpf_testmod_test_struct_ops3 (void )
@@ -440,6 +448,11 @@ __bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena_nullabreturn st_ops3->test_arena_nullable (ptr__arena_nullable); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena) +{ +return st_ops3->test_arena_stack (1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 , ptr__arena); +} +struct bpf_testmod_btf_type_tag_1 {int a; };
@@ -851,6 +864,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack)BTF_ID_FLAGS (func, bpf_kfunc_get_default_trusted_ptr_test);BTF_ID_FLAGS (func, bpf_kfunc_put_default_trusted_ptr_test);BTF_KFUNCS_END (bpf_testmod_common_kfunc_ids)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +3 -0
@@ -109,6 +109,9 @@ struct bpf_testmod_ops3 {/* Used to test arena pointer arguments. */ int (*test_arena)(u64 *ptr);int (*test_arena_nullable)(u64 *ptr); +/* enough leading args to force @ptr onto the stack on x86 and arm64 */ +int (*test_arena_stack)(u64 a,u64 b,u64 c,u64 d,u64 e,u64 f, +u64 g,u64 h,u64 *ptr); };struct st_ops_args {
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1 -0
@@ -123,6 +123,7 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym; void bpf_testmod_test_mod_kfunc(int i) __ksym; int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena_nullable) __ksym; +int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym;
Signature · 1 line
2.53.0
Re: [PATCH bpf-next v3 9/9] selftests/bpf: Test stack-passed struct_ops arena arguments
Quoted 5 lines · Kumar Kartikeya Dwivedi
From: Tejun Heo <[email protected]>
Add a
test_arena_stackmember with eight leading scalar arguments so the arena pointer is passed on the stack.
Quoted 8 lines
The callback validates the first and last scalar ctx slots before dereferencing the pointer in ctx[8]. This exercises the indirect trampoline stack layout and arena conversion together, and prevents a regression where stack arguments are read one slot late.
Signed-off-by Tejun Heo <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---