bpf: make arena pointers first-class kfunc and struct_ops arguments
Subsystems (8)
Tejun Heo
reviewercover bpf: make arena pointers first-class kfunc and struct_ops arguments
RFC. I am not familiar with the BPF verifier code base and this series is heavily AI-assisted. I reviewed and iterated on all of it but leaned on the AI outright in places.
BPF arena memory is shared between a BPF program and the kernel, but the two sides address it through different bases, so an arena pointer cannot be passed across the kfunc or struct_ops boundary as a pointer. It crosses as a bare scalar, and the receiving side then offsets it to the other address space and casts it by hand. Every kfunc and struct_ops caller that touches arena memory open-codes the same translation.
This series adds an __arena argument suffix that makes the translation automatic. The callee declares the parameter by its real pointer type and dereferences it directly, and the verifier rebases it at the boundary (arena to kernel address into a kfunc, kernel address to arena into a struct_ops callback), with NULL preserved. The pointer arrives ready to use on both sides, with no scalar laundering and no hand-written offset or cast.
The approach comes out of rough discussions with Alexei and Kumar.
Patches 1-4 are the BPF core support plus selftests, the kfunc direction then the struct_ops direction. Patches 5-6 convert two sched_ext consumers to show the result. They are marked NOT_SIGNED_OFF and are not meant to be applied. sched_ext has moved on since, notably the pending cap work, so once the core lands the plan is to pull bpf-next into sched_ext and redo the consumer side there.
Tested: the BPF selftests pass at bpf_jit_harden 0/1/2, the arena, struct_ops and kfunc regression suites are clean, and scx_qmap schedules correctly through the converted paths.
Tejun Heo (6):
bpf: Support __arena suffix for kfunc arguments selftests/bpf: Add kfunc __arena argument tests
bpf: Support __arena suffix on struct_ops stub arguments selftests/bpf: Add struct_ops __arena argument tests
sched_ext: Pass a kernel arena pointer to ops_cid.set_cmask()
sched_ext: Convert scx_bpf_cid_override() to take an arena pointer
Documentation/bpf/kfuncs.rst | 27 ++++ include/linux/bpf_verifier.h | 2 + kernel/bpf/bpf_struct_ops.c | 17 ++- kernel/bpf/btf.c | 10 +- kernel/bpf/fixups.c | 70 +++++++++ kernel/bpf/verifier.c | 103 +++++++++++-- kernel/sched/ext/cid.c | 24 +++- kernel/sched/ext/ext.c | 11 +- kernel/sched/ext/internal.h | 16 +-- tools/sched_ext/include/scx/compat.bpf.h | 8 +- tools/sched_ext/scx_qmap.bpf.c | 13 +- tools/sched_ext/scx_qmap.c | 41 ++++-- tools/sched_ext/scx_qmap.h | 3 + .../selftests/bpf/prog_tests/arena_kfunc.c | 14 ++ .../bpf/prog_tests/test_struct_ops_arena.c | 61 ++++++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 135 ++++++++++++++++++ .../selftests/bpf/progs/struct_ops_arena.c | 67 +++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 +++ .../selftests/bpf/test_kmods/bpf_testmod.c | 45 ++++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 2 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 5 + 21 files changed, 622 insertions(+), 72 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
Signature · 1 line
2.55.0
Tejun Heo
reviewerRFC PATCH 1/6 bpf: Support __arena suffix for kfunc arguments
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the __arena argument suffix and both steps go away. The kfunc declares the parameter by its real pointer type and dereferences it directly. The verifier rebases each one at the call site, rN = kern_vm_start + (u32)rN with NULL preserved. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.
--- Documentation/bpf/kfuncs.rst | 22 +++++++++ include/linux/bpf_verifier.h | 2 + kernel/bpf/verifier.c | 86 +++++++++++++++++++++++++++++++++--- 3 files changed, 105 insertions(+), 5 deletions(-)
Documentation/bpf/kfuncs.rst +22 -0
@@ -250,6 +250,26 @@ This annotation is used to indicate that the argument is a constant string. ... } +2.3.7 __arena Annotation +------------------------ + +This annotation is used to indicate that the pointer argument points into +the program's arena. The verifier translates it to a directly +dereferenceable kernel address before the call, subject to the access +rules described in :ref:`BPF_kfunc_arena_access`. NULL is preserved and the +kfunc is responsible for checking for NULL before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map. The program +can pass any value without compromising the kernel. A value that does not +point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function
@@ -487,6 +507,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------
include/linux/bpf_verifier.h +2 -0
@@ -690,6 +690,8 @@ struct bpf_insn_aux_data {*/ u8 fastcall_spills_num:3; u8 arg_prog:4; +/* bitmask of R1-R5 kfunc args to rebase to arena kernel addresses */ +u8 arg_arena_regs;/* below fields are initialized once */ unsigned int orig_idx;/* original instruction index */
kernel/bpf/verifier.c +81 -5
106 lines not on this page · raw message
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCHSET SLOP RFC 1/6] bpf: Support __arena suffix for kfunc arguments
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The verifier skips size and bounds checks for `__arena` kfunc arguments, relying entirely on a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The verifier skips size and bounds checks for __arena kfunc arguments, relying entirely on a 32KB hardware guard region. This allows a malicious BPF program to overshoot the guard region and cause a kernel oops by passing an offset that results in reading or writing a large struct or buffer. --
--- Patch [1]: [PATCHSET SLOP RFC 1/6] bpf: Support __arena suffix for kfunc arguments --- commit 228de70b9e4499793706318f5047bb9d51313346
Author: Tejun Heo <[email protected]>
bpf: Support __arena suffix for kfunc arguments
This patch adds the __arena suffix for kfunc arguments, allowing BPF programs to pass arena pointers directly to kfuncs. The verifier rebases the register at the call site, avoiding the need for open-coded translation in every kfunc.
Quoted 14 lines
kernel/bpf/verifier.c +6
-0@@ -11975,6 +11980,32 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_continue ; } +if (is_kfunc_arg_arena (btf, &args[i])) { + t =btf_type_skip_modifiers (btf, args[i].type,NULL ); +if (verifier_bug_if (!btf_type_is_ptr (t), env, +"kfunc %s arg#%d has __arena tag on non-pointer" , + func_name, i)) +return -EFAULT ;
[ ... ]
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 12092
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +cur_aux (env)->arg_arena_regs |=BIT (regno -BPF_REG_1 ); +continue ; + }
[Severity: High]
Does skipping the rest of the checks here bypass BTF memory size and buffer bounds tracking for __arena arguments?
Because this relies entirely on the hardware guard region to catch out-of-bounds accesses, could a BPF program pass an offset close to the 4GB arena limit (like 4GB - 1) to a kfunc that takes a large struct or buffer size parameter?
The fault handler strictly limits recovery to the 4GB limit plus a 32KB guard region:
kernel/bpf/arena.c:bpf_arena_handle_page_fault() {
...
if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2)
return false;
...
}If a kfunc access extends beyond this guard region, it seems like it would bypass the fault handler and cause a kernel oops or memory corruption.
Should there be size validation for __arena arguments similar to other memory pointers?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Tejun Heo
reviewerRe: [PATCHSET SLOP RFC 1/6] bpf: Support __arena suffix for kfunc arguments
On __arena kfunc args relying on the guard region:
This is the arena direct-access contract: the callee dereferences arena memory directly and the guard absorbs overshoot, like any arena access. Both kfuncs here read small fixed objects well inside the guard. If we ever pass objects larger than the guard, the fix is to grow the guard, not to bounds-check individual kfuncs.
v2: no code change.
Thanks.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCHSET SLOP RFC 1/6] bpf: Support __arena suffix for kfunc arguments
In reply to Tejun HeoQuoted 28 lines · Tejun Heo
Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.
Add the
__arenaargument suffix and both steps go away. The kfunc declares the parameter by its real pointer type and dereferences it directly. The verifier rebases each one at the call site, rN =kern_vm_start+ (u32)rN with NULL preserved. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts aPTR_TO_ARENAor scalar register, matching global subprog arena arguments.Signed-off-by Tejun Heo <[email protected]>--- Documentation/bpf/kfuncs.rst | 22 +++++++++ include/linux/bpf_verifier.h | 2 + kernel/bpf/verifier.c | 86 +++++++++++++++++++++++++++++++++--- 3 files changed, 105 insertions(+), 5 deletions(-)Documentation/bpf/kfuncs.rst +0
-0@@ -250,6 +250,26 @@ This annotation is used to indicate that the argument is a constant string. }
Documentation/bpf/kfuncs.rst This annotation is used to indicate that the argument is a constant string. · line 261
Quoted 10 lines
Documentation/bpf/kfuncs.rst +9
-0+2.3.7 __arena Annotation +------------------------ + +This annotation is used to indicate that the pointer argument points into +the program's arena. The verifier translates it to a directly +dereferenceable kernel address before the call, subject to the access +rules described in :ref:`BPF_kfunc_arena_access`. NULL is preserved and the +kfunc is responsible for checking for NULL before dereferencing. +
Here we can specify the maximum permitted size (GUARD_SZ/2 = 32kb).
Quoted 139 lines
Documentation/bpf/kfuncs.rst +13
-0+An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map. The program +can pass any value without compromising the kernel. A value that does not +point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function@@ -487,6 +507,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments --------------------------------------------------include/linux/bpf_verifier.h +2
-0@@ -690,6 +690,8 @@ struct bpf_insn_aux_data {*/ u8 fastcall_spills_num:3; u8 arg_prog:4; +/* bitmask of R1-R5 kfunc args to rebase to arena kernel addresses */ +u8 arg_arena_regs;/* below fields are initialized once */ unsigned int orig_idx;/* original instruction index */ kernel/bpf/verifier.c +74
-5@@ -10856,6 +10856,11 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_paramreturn btf_param_match_suffix (btf, arg,"__irq_flag" ); } +static bool is_kfunc_arg_arena (const struct btf *btf,const struct btf_param *arg) +{ +return btf_param_match_suffix (btf, arg,"__arena" ); +} +static bool is_kfunc_arg_scalar_with_name (const struct btf *btf,const struct btf_param *arg,const char *name)@@ -12060,6 +12065,32 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_continue ; } +if (is_kfunc_arg_arena (btf, &args[i])) { + t =btf_type_skip_modifiers (btf, args[i].type,NULL ); +if (verifier_bug_if (!btf_type_is_ptr (t), env, +"kfunc %s arg#%d has __arena tag on non-pointer" , + func_name, i)) +return -EFAULT ; +if (!env->prog->aux->arena) { +verbose (env, +"%s arena pointer requires a program with an associated arena\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (regno <0 ) { +verbose (env,"%s arena pointer cannot be a stack argument\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +if (reg->type !=PTR_TO_ARENA && reg->type !=SCALAR_VALUE ) { +verbose (env,"%s is not a pointer to arena or scalar\n" , +reg_arg_name (env, argno)); +return -EINVAL ; + } +cur_aux (env)->arg_arena_regs |=BIT (regno -BPF_REG_1 ); +continue ; + } +if (is_kfunc_arg_ignore (btf, &args[i]) ||is_kfunc_arg_implicit (meta, i))continue ;@@ -19890,13 +19921,58 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, *cnt =6 ; } -if (env->insn_aux_data[insn_idx].arg_prog) { -u32 regno = env->insn_aux_data[insn_idx].arg_prog; -struct bpf_insn ld_addrs[2 ] = {BPF_LD_IMM64 (regno, (long )env->prog->aux) }; +if (env->insn_aux_data[insn_idx].arg_prog || + env->insn_aux_data[insn_idx].arg_arena_regs) { +unsigned long arena_regs = env->insn_aux_data[insn_idx].arg_arena_regs; +u64 kern_vm_start =bpf_arena_get_kern_vm_start (env->prog->aux->arena); +u32 prog_regno = env->insn_aux_data[insn_idx].arg_prog;int idx = *cnt; +int bit; - insn_buf[idx++] = ld_addrs[0 ]; - insn_buf[idx++] = ld_addrs[1 ]; +if (verifier_bug_if (idx, env,"kfunc call at %d already patched" , insn_idx)) +return -EFAULT ; + +/* +* A kfunc may take both an __arena arg and the __prog aux. Emit +* each present fixup, then append the call once. +*/ +if (arena_regs) { +bool blinding = env->prog->blinding_requested; +int base_reg = blinding ?BPF_REG_0 :BPF_REG_AX ; +struct bpf_insn ld[2 ] = {BPF_LD_IMM64 (base_reg, kern_vm_start) }; + +/* +* rN = kern_vm_start + (u32)rN, NULL left as 0. +* Constant blinding rewrites immediate-carrying insns +* unless they use BPF_REG_AX. With hardening, the +* constant sits in R0 (an AX-destined LD_IMM64 would be +* corrupted) and NULL is tested against AX = 0. Without +* hardening, use the cheaper AX constant and immediate +* test. +*/ + insn_buf[idx++] = ld[0 ]; + insn_buf[idx++] = ld[1 ]; +if (blinding) + insn_buf[idx++] =BPF_MOV64_IMM (BPF_REG_AX ,0 ); + +for_each_set_bit (bit, &arena_regs,MAX_BPF_FUNC_REG_ARGS ) { +int regno =BPF_REG_1 + bit; + + insn_buf[idx++] =BPF_ZEXT_REG (regno); + insn_buf[idx++] = blinding ? +BPF_JMP_REG (BPF_JEQ , regno,BPF_REG_AX ,1 ) : +BPF_JMP_IMM (BPF_JEQ , regno,0 ,1 ); + insn_buf[idx++] =BPF_ALU64_REG (BPF_ADD , regno, base_reg); + } + } +if (prog_regno) {
kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19978
Quoted 10 lines
kernel/bpf/verifier.c +7
-0+struct bpf_insn ld_addrs[2 ] = { +BPF_LD_IMM64 (prog_regno, (long )env->prog->aux) + }; + + insn_buf[idx++] = ld_addrs[0 ]; + insn_buf[idx++] = ld_addrs[1 ]; + } insn_buf[idx++] = *insn; *cnt = idx; }
I think overall this makes sense. One thing we can change is making __arena imply non-NULL arena pointers, which will lead to a more optimized sequence, and __arena_nullable when we want NULL to be preserved in either direction, to represent optional data.
We likely need two bitmaps, one to represent nullness of arena regs and one for arena regs.
Also, instead of handling them in fixup in verifier, we can let JITs handle them because they already hold arena base in some fixed register.
E.g. we can add the following in kernel/bpf/core.c.
struct bpf_jit_arena_args {
u8 regs;
u8 nullable_regs;
};struct bpf_jit_arena_args
bpf_kfunc_arena_args(const struct bpf_verifier_env *env,
const struct bpf_prog *prog, int insn_idx)
{
struct bpf_jit_arena_args args = {};
if (!env)
return args;
insn_idx += prog->aux->subprog_start;
args.regs = env->insn_aux_data[insn_idx].arg_arena_regs;
args.nullable_regs =
env->insn_aux_data[insn_idx].arg_arena_nullable_regs;
return args;
}The accessor expects a zero-based instruction index local to prog. x86’s JIT loop is one-based and must pass i - 1 in such case. You can see bpf_insn_is_indirect_target()'s usage for inspiration.
For now I guess we can add support for x86 and arm64. I'll show pseudocode for both x86, arm64 would be similar in spirit.
We can place a new emit_kfunc_arena_args() right before emit_call().
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL)
emit_kfunc_arena_args(env, bpf_prog, i - 1, &prog);
ip += x86_call_depth_emit_accounting(&prog, func, ip); emit_call(&prog, func, ip);
For emit_kfunc_arena_args(), it would look something like this (pseudocode-ish):
static int
emit_kfunc_arena_args(struct bpf_verifier_env *env,
struct bpf_prog *prog, int insn_idx,
u8 **pprog)
{
struct bpf_jit_arena_args args;
u8 *native_prog = *pprog;
int bit;
args = bpf_kfunc_arena_args(env, prog, insn_idx);
if (WARN_ON_ONCE(args.nullable_regs & ~args.regs))
return -EINVAL;
if (WARN_ON_ONCE(args.regs && !prog->aux->arena))
return -EINVAL;
for (bit = 0; bit < MAX_BPF_FUNC_REG_ARGS; bit++) {
int reg;
if (!(args.regs & BIT(bit)))
continue;
/*
* These logical JIT registers map as:
*
* BPF_REG_1 -> RDI
* BPF_REG_2 -> RSI
* BPF_REG_3 -> RDX
* BPF_REG_4 -> RCX
* BPF_REG_5 -> R8
*/
reg = BPF_REG_1 + bit;
/*
* mov reg32, reg32
*
* Extract the low 32 bits and clear the native register's
* upper 32 bits.
*/
emit_mov_reg(&native_prog, false, reg, reg);
if (args.nullable_regs & BIT(bit)) {
/*
* test reg32, reg32
* jz after_add
*/
emit_test_reg32(&native_prog, reg);
/*
* add reg64, r12 is three bytes for all argument
* registers, so a short jump can skip it directly.
*/
emit_jz_short(&native_prog, /* displacement */ 3);
}
/*
* add reg64, r12
*
* R12 contains kern_vm_start.
*/
emit_add_reg64(&native_prog, reg, X86_REG_R12);
}
*pprog = native_prog;
return 0;
}arm64 will be similar, but I will let AI figure out that one. One opportunity for arm64 case is to use add xN, x28, wN, uxtw single-instruction sequence for non-nullable case.
Lastly, we should add bpf_jit_supports_arena_args(), make it return false from a __weak function in kernel/bpf/core.c, and then override from x86 and arm64 JIT.
Then return an error when arena kfunc args (nullable or not) are used when this predicate is false in check_kfunc_call() logic.
Since we're using the JIT for emitting this, we should not need special blinding related handling.
For tests, I think we cover a lot already, but I'd confirmation on whether nullable/non-nullable cases work correctly, in particular, low32 bits as zero should produce kern_vm_start with __arena, and 0 with __arena_nullable, etc. We can probably throw in tests that test JIT emitted sequence as well, there are several examples that do this in the selftests. Since we'll be limited to x86 and arm64 we can guard the tests to only work on these two arches.
Finally, it would make sense to add documentation for both tags, and we should be done.
--
Next, for the struct_ops side, the main idea would be to do translation before we save args into the context for the program instead of reloading it from context, translating, and then having to store back. Just like program->kernel, we will need some per-JIT handling of kernel->program translation.
Thus, we can avoid injecting a BPF entry prologue or performing a second load/modify/store pass over the saved context. Fold arena conversion into the architecture trampoline’s existing save_args() path while it copies native arguments into the BPF u64 ctx[].
Pseudocode:
/* __arena */ctx_value = (u32)(kaddr - kern_vm_start);
/* __arena_nullable */ctx_value = kaddr ? (u32)(kaddr - kern_vm_start) : 0;
For nullable arguments, test the complete 64-bit native pointer before subtracting or truncating. Only an actual kernel NULL should take the NULL path. Derive arena argument information from prog->aux->ctx_arg_info, keyed by info->offset, which is the offset in the flattened trampoline context. Do not use the logical argument number because preceding 16-byte arguments occupy two u64 slots. This lookup happens while generating the trampoline; no runtime loop or metadata lookup should be emitted.
Only enable this for the single-node struct_ops indirect trampoline. Obtain the program from the struct_ops fentry node and pass optional arena metadata into the initial save_args(..., for_call_origin=false) invocation. Generic trampolines may contain multiple programs with different arenas and must not use this conversion. We can probably make those return -ENOTSUPP if ther eis a p
On x86, translate through RAX and store the translated value directly into the ctx slot:
# __arena, native argument in RDI
movl %edi, %eax subl $kern_vm_start_lo, %eax movq %rax, ctx_slot
# __arena_nullable
movq %rdi, %rax testq %rax, %rax jz 1f subl $kern_vm_start_lo, %eax
1:
movq %rax, ctx_slot
Use the same operation after loading a stack-passed argument into RAX. A 32-bit subtraction is sufficient and also clears the upper 32 bits:
(u32)(kaddr - base) == (u32)kaddr - (u32)base;
On arm64, materialize only lower_32_bits(kern_vm_start) in a temporary such as W11. For register arguments:
// __arena
sub w10, w0, w11
str x10, [sp, #ctx_slot]
// __arena_nullable
mov x10, x0 cbz x10, 1f sub w10, w10, w11
1:
str x10, [sp, #ctx_slot]
For stack-passed arguments, first load the full pointer into X10, perform the same optional CBZ and 32-bit subtraction, and then store X10 into the ctx slot. Writing W10 automatically clears X10’s upper bits, so no separate UXTW is needed.
Non-arena arguments should retain the existing direct save path. Since this is native trampoline code generation, kern_vm_start is not inserted into the BPF instruction stream and requires no constant-blinding special case.
--
Once you respin arm64 experts can help validate arm64 bits, I am no expert there.
Tejun Heo
reviewerRFC PATCH 2/6 selftests/bpf: Add kfunc __arena argument tests
In reply to Tejun HeoAdd arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar, NULL), five arena arguments in one call with NULLs mixed in, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type).
--- .../selftests/bpf/prog_tests/arena_kfunc.c | 14 ++ .../testing/selftests/bpf/progs/arena_kfunc.c | 135 ++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 33 +++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 4 + 4 files changed, 186 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c new file +14 -0
@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo <[email protected]> */ +#include <test_progs.h> + +#include "arena_kfunc.skel.h" + +/* The test kfuncs live in bpf_testmod. Resolving kfuncs against module +* BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of +* through the verifier tests' capability-restricted runner. +*/ +void test_arena_kfunc (void ) +{ +RUN_TESTS (arena_kfunc); +}
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +135 -0
135 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +33 -0
@@ -210,6 +210,37 @@ __bpf_kfunc void bpf_kfunc_common_test(void) { } +__bpf_kfunc u64 bpf_kfunc_arena_arg_test(u64 *val__arena) +{ +u64 old; + +if (!val__arena) +return 0xdeadbeef ; + + old = *val__arena; + *val__arena = old +1 ; +return old; +} + +__bpf_kfunc u64 bpf_kfunc_arena_args5_test(u64 *a__arena, u64 *b__arena, +u64 *c__arena,u64 *d__arena, +u64 *e__arena) +{ +u64 sum =0 ; + +if (a__arena) + sum += *a__arena; +if (b__arena) + sum += *b__arena; +if (c__arena) + sum += *c__arena; +if (d__arena) + sum += *d__arena; +if (e__arena) + sum += *e__arena; +return sum; +} + __bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr,struct bpf_dynptr *ptr__nullable) {
@@ -723,6 +754,8 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_value) BTF_ID_FLAGS(func, bpf_kfunc_common_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_arg_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_args5_test) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test)BTF_ID_FLAGS (func, bpf_kfunc_nested_acquire_nonzero_offset_test,KF_ACQUIRE )
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +4 -0
@@ -98,6 +98,10 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym; +__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena, + __u64 *c__arena, __u64 *d__arena, + __u64 *e__arena) __ksym;int *bpf_kfunc_call_test_get_rdwr_mem (struct prog_test_ref_kfunc *p,const int rdwr_buf_size) __ksym;int *bpf_kfunc_call_test_get_rdonly_mem (struct prog_test_ref_kfunc *p,const int rdonly_buf_size) __ksym;int *bpf_kfunc_call_test_acq_rdonly_mem (struct prog_test_ref_kfunc *p,const int rdonly_buf_size) __ksym;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCHSET SLOP RFC 2/6] selftests/bpf: Add kfunc __arena argument tests
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] The kfunc arena argument fixup logic incorrectly translates a valid pointer to the first page of a…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The kfunc arena argument fixup logic incorrectly translates a valid pointer to the first page of a 4GB-aligned arena as NULL, causing kfuncs to receive a NULL pointer instead of a valid kernel address.
- [Low] Multi-line comment does not follow the mandated BPF subsystem style.
--
--- Patch [2]: [PATCHSET SLOP RFC 2/6] selftests/bpf: Add kfunc __arena argument tests --- commit c3a66495e39c4d671a1065fb44bc6cef2f26d120
Author: Tejun Heo <[email protected]>
selftests/bpf: Add kfunc __arena argument tests
Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar, NULL), five arena arguments in one call with NULLs mixed in, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type).
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c new file +0
-0
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c line 10
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c new file +10
-0@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo <[email protected]> */ +#include <test_progs.h> + +#include "arena_kfunc.skel.h" + +/* The test kfuncs live in bpf_testmod. Resolving kfuncs against module +* BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of +* through the verifier tests' capability-restricted runner. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment follow the BPF subsystem style?
The opening /* should be on its own line:
/*
* The test kfuncs live in bpf_testmod...
*/
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/arena_kfunc.c new file +4
-0+void test_arena_kfunc (void ) +{ +RUN_TESTS (arena_kfunc); +}
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/arena_kfunc.c line 55
Quoted 4 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +4
-0+/* NULL is preserved */ + ret =bpf_kfunc_arena_arg_test (NULL ); +if (ret !=0xdeadbeef ) +return 5 ;
[Severity: High]
Does the kfunc arena argument fixup logic tested here incorrectly translate a valid pointer to the first page of a 4GB-aligned arena as NULL?
In kernel/bpf/verifier.c:bpf_fixup_kfunc_call(), if an arena is created with a 4GB-aligned user_vm_start, and a BPF program passes a pointer to offset 0 within this arena to a kfunc, the fixup logic uses BPF_ZEXT_REG(regno) to clear the upper 32 bits, making the register exactly 0.
This is then explicitly checked by BPF_JMP_IMM(BPF_JEQ, regno, 0, 1), which skips the kern_vm_start base addition.
Would this cause kfuncs to receive a NULL pointer instead of a valid kernel address when passed a pointer to offset 0?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Tejun Heo
reviewerRe: [PATCHSET SLOP RFC 2/6] selftests/bpf: Add kfunc __arena argument tests
On a pointer to offset 0 being translated as NULL:
offset 0 == NULL is the existing arena representation, not something this
introduces: cast_user already collapses any arena pointer with zero low 32 bits to NULL (the x86 JIT emits test/cmove). So no offset-0 pointer distinct from NULL exists across the boundary, and the fixup is consistent with that.
v2: no code change; the flagged comment is reflowed.
Thanks.
Tejun Heo
reviewerRFC PATCH 3/6 bpf: Support __arena suffix on struct_ops stub arguments
In reply to Tejun HeoA struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in.
Add the __arena stub argument suffix and both steps go away. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). An entry prologue injected into the program rebases each such ctx slot once, arena = (u32)(kaddr - kern_vm_start) with NULL preserved, keyed by argument position. The rebased value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address.
--- Documentation/bpf/kfuncs.rst | 5 +++ kernel/bpf/bpf_struct_ops.c | 17 ++++++--- kernel/bpf/btf.c | 10 ++++-- kernel/bpf/fixups.c | 70 ++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 17 ++++++--- 5 files changed, 106 insertions(+), 13 deletions(-)
Documentation/bpf/kfuncs.rst +5 -0
@@ -270,6 +270,11 @@ Calling such a kfunc requires the program to use an arena map. The program can pass any value without compromising the kernel. A value that does not point into the arena is a program bug. +The suffix has the same meaning on the arguments of struct_ops stub +functions. The kernel caller passes the kernel arena address and the +argument arrives in the BPF program as an arena pointer that can be +dereferenced directly, with NULL preserved. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function
kernel/bpf/bpf_struct_ops.c +12 -5
@@ -147,6 +147,7 @@ void bpf_struct_ops_image_free(void *image)#define MAYBE_NULL_SUFFIX "__nullable"#define REFCOUNTED_SUFFIX "__ref" +#define ARENA_SUFFIX "__arena"/* Prepare argument info for every nullable argument of a member of a * struct_ops type.
@@ -159,7 +160,7 @@ void bpf_struct_ops_image_free(void *image)* to provide an array of struct bpf_ctx_arg_aux, which in turn provides * the information that used by the verifier to check the arguments of the * BPF struct_ops program assigned to the member. Here, we only care about -* the arguments that are marked as __nullable. +* the arguments that are marked as __nullable, __ref or __arena. * * The array of struct bpf_ctx_arg_aux is eventually assigned to * prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
@@ -175,7 +176,7 @@ static int prepare_arg_info(struct btf *btf, struct bpf_struct_ops_arg_info *arg_info) {const struct btf_type *stub_func_proto, *pointed_type; -bool is_nullable = false, is_refcounted = false; +bool is_nullable = false, is_refcounted = false, is_arena = false;const struct btf_param *stub_args, *args;struct bpf_ctx_arg_aux *info, *info_buf;u32 nargs, arg_no, info_cnt =0 ;
@@ -226,26 +227,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf;for (arg_no =0 ; arg_no < nargs; arg_no++) {/* Skip arguments that is not suffixed with -* "__nullable or __ref". +* "__nullable", "__ref" or "__arena". */ is_nullable =btf_param_match_suffix (btf, &stub_args[arg_no],MAYBE_NULL_SUFFIX ); is_refcounted =btf_param_match_suffix (btf, &stub_args[arg_no],REFCOUNTED_SUFFIX ); + is_arena =btf_param_match_suffix (btf, &stub_args[arg_no], +ARENA_SUFFIX );if (is_nullable) suffix =MAYBE_NULL_SUFFIX ;else if (is_refcounted) suffix =REFCOUNTED_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;else continue ; -/* Should be a pointer to struct */ +/* Should be a pointer to struct, or any pointer for __arena */ pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id);if (!pointed_type || - !btf_type_is_struct (pointed_type)) { + (!is_arena && !btf_type_is_struct (pointed_type))) {pr_warn ("stub function %s has %s tagging to an unsupported type\n" , stub_fname, suffix); goto err_out;
@@ -273,6 +278,8 @@ static int prepare_arg_info(struct btf *btf, }else if (is_refcounted) { info->reg_type =PTR_TRUSTED |PTR_TO_BTF_ID ; info->refcounted = true; + }else if (is_arena) { + info->reg_type =PTR_TO_ARENA ; } info++;
kernel/bpf/btf.c +7 -3
@@ -6961,15 +6961,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,return false; } -/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */ +/* +* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or +* PTR_TO_ARENA. +*/ for (i =0 ; i < prog->aux->ctx_arg_info_size; i++) {const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];u32 type, flag; type =base_type (ctx_arg_info->reg_type); flag =type_flag (ctx_arg_info->reg_type); -if (ctx_arg_info->offset == off && type ==PTR_TO_BUF && - (flag &PTR_MAYBE_NULL )) { +if (ctx_arg_info->offset == off && + (type ==PTR_TO_ARENA || + (type ==PTR_TO_BUF && (flag &PTR_MAYBE_NULL )))) { info->reg_type = ctx_arg_info->reg_type;return true; }
kernel/bpf/fixups.c +70 -0
82 lines not on this page · raw message
kernel/bpf/verifier.c +12 -5
@@ -18789,6 +18789,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) {const struct btf_type *t, *func_proto;const struct bpf_struct_ops_desc *st_ops_desc; +const struct bpf_struct_ops_arg_info *arg_info;const struct bpf_struct_ops *st_ops;const struct btf_member *member;struct bpf_prog *prog = env->prog;
@@ -18867,10 +18868,17 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)return -EACCES ; } -for (i =0 ; i < st_ops_desc->arg_info[member_idx].cnt; i++) { -if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info = &st_ops_desc->arg_info[member_idx]; +for (i =0 ; i < arg_info->cnt; i++) { +const struct bpf_ctx_arg_aux *info = &arg_info->info[i]; + +if (info->refcounted) has_refcounted_arg = true; -break ; +if (base_type (info->reg_type) ==PTR_TO_ARENA && !prog->aux->arena) { +verbose (env, +"arena argument of %s requires a program with an associated arena\n" , + mname); +return -EINVAL ; } }
@@ -18891,8 +18899,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_name = mname; env->ops = st_ops->verifier_ops; -return bpf_prog_ctx_arg_info_init (prog, st_ops_desc->arg_info[member_idx].info, - st_ops_desc->arg_info[member_idx].cnt); +return bpf_prog_ctx_arg_info_init (prog, arg_info->info, arg_info->cnt); }#define SECURITY_PREFIX "security_"
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCHSET SLOP RFC 3/6] bpf: Support __arena suffix on struct_ops stub arguments
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Tail-calling another struct_ops program causes double-subtraction of the arena base address…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Tail-calling another struct_ops program causes double-subtraction of the arena base address, corrupting arena pointers in the context. --
--- Patch [3]: [PATCHSET SLOP RFC 3/6] bpf: Support __arena suffix on struct_ops stub arguments --- commit 7fff65a383d9791611afe2d53ac2633d4bac8dda
Author: Tejun Heo <[email protected]>
bpf: Support __arena suffix on struct_ops stub arguments
A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in. Add the __arena stub argument suffix and both steps go away.
Quoted 4 lines
kernel/bpf/fixups.c +0
-0@@ -680,6 +680,64 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,
[ ... ]
Quoted 3 lines
kernel/bpf/fixups.c +3
-0+static int gen_arena_arg_prologue (struct bpf_verifier_env *env, +struct bpf_insn *insn_buf) +{
[ ... ]
Quoted 7 lines
kernel/bpf/fixups.c +7
-0+for (i =0 ; i < aux->ctx_arg_info_size; i++) { +int off; + +if (base_type (aux->ctx_arg_info[i].reg_type) !=PTR_TO_ARENA ) +continue ; + off = aux->ctx_arg_info[i].offset; + insn_buf[cnt++] =BPF_LDX_MEM (BPF_DW ,BPF_REG_2 ,BPF_REG_1 , off);
kernel/bpf/fixups.c int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, · line 739
Quoted 10 lines
kernel/bpf/fixups.c +10
-0+ insn_buf[cnt++] = blinding ? +BPF_JMP_REG (BPF_JEQ ,BPF_REG_2 ,BPF_REG_AX ,2 ) : +BPF_JMP_IMM (BPF_JEQ ,BPF_REG_2 ,0 ,2 ); + insn_buf[cnt++] =BPF_ALU64_REG (BPF_SUB ,BPF_REG_2 , base_reg); + insn_buf[cnt++] =BPF_ZEXT_REG (BPF_REG_2 ); + insn_buf[cnt++] =BPF_STX_MEM (BPF_DW ,BPF_REG_1 ,BPF_REG_2 , off); + } + insn_buf[cnt++] = env->prog->insnsi[0 ]; +return cnt; +}
[Severity: High]
Does this in-place modification cause an issue if the program performs a tail call to another struct_ops program?
Since tail calls pass the exact same context array to the callee, and the callee will also have this prologue injected, it seems the already-rebased 32-bit offset will be read and the arena base address will be subtracted a second time. This could severely corrupt the arena pointer offset.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0@@ -18729,10 +18730,17 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
[ ... ]
Quoted 6 lines
kernel/bpf/verifier.c +4
-2-for (i =0 ; i < st_ops_desc->arg_info[member_idx].cnt; i++) { -if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info = &st_ops_desc->arg_info[member_idx]; +for (i =0 ; i < arg_info->cnt; i++) { +const struct bpf_ctx_arg_aux *info = &arg_info->info[i]; +
kernel/bpf/verifier.c static int check_struct_ops_btf_id(struct bpf_verifier_env *env) · line 18883
Quoted 10 lines
kernel/bpf/verifier.c +6
-1+if (info->refcounted) has_refcounted_arg = true; -break ; +if (base_type (info->reg_type) ==PTR_TO_ARENA && !prog->aux->arena) { +verbose (env, +"arena argument of %s requires a program with an associated arena\n" , + mname); +return -EINVAL ; } }
[Severity: High]
Should we also restrict tail calls for programs with __arena arguments here?
Right below this loop, check_struct_ops_btf_id() blocks tail calls if has_refcounted_arg is true:
for (i = 0; i < env->subprog_cnt; i++) {
if (has_refcounted_arg && env->subprog_info[i].has_tail_call) {
verbose(env, "program with __ref argument cannot tail call\n");
return -EINVAL;
}
}Since gen_arena_arg_prologue() modifies the context in-place, blocking tail calls for programs with arena arguments would prevent the double subtraction issue entirely.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Tejun Heo
reviewerPATCH v2 3/6 bpf: Support __arena suffix on struct_ops stub arguments
In reply to Tejun HeoA struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in.
Add the __arena stub argument suffix and both steps go away. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). An entry prologue injected into the program rebases each such ctx slot once, arena = (u32)(kaddr - kern_vm_start) with NULL preserved, keyed by argument position. The rebased value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address.
Because the prologue rewrites the ctx slot in place, tail calling would let the callee's prologue rebase it a second time. A program with an __arena argument is barred from tail calls, matching the existing restriction on __ref arguments.
v2: Reject tail calls for programs with an __arena argument. A tail call
would re-run the entry prologue and rebase the ctx slot twice.
--- Documentation/bpf/kfuncs.rst | 5 +++ kernel/bpf/bpf_struct_ops.c | 17 ++++++--- kernel/bpf/btf.c | 10 ++++-- kernel/bpf/fixups.c | 70 ++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 37 ++++++++++++++----- 5 files changed, 122 insertions(+), 17 deletions(-)
Documentation/bpf/kfuncs.rst +5 -0
@@ -270,6 +270,11 @@ Calling such a kfunc requires the program to use an arena map. The program can pass any value without compromising the kernel. A value that does not point into the arena is a program bug. +The suffix has the same meaning on the arguments of struct_ops stub +functions. The kernel caller passes the kernel arena address and the +argument arrives in the BPF program as an arena pointer that can be +dereferenced directly, with NULL preserved. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function
kernel/bpf/bpf_struct_ops.c +12 -5
@@ -147,6 +147,7 @@ void bpf_struct_ops_image_free(void *image)#define MAYBE_NULL_SUFFIX "__nullable"#define REFCOUNTED_SUFFIX "__ref" +#define ARENA_SUFFIX "__arena"/* Prepare argument info for every nullable argument of a member of a * struct_ops type.
@@ -159,7 +160,7 @@ void bpf_struct_ops_image_free(void *image)* to provide an array of struct bpf_ctx_arg_aux, which in turn provides * the information that used by the verifier to check the arguments of the * BPF struct_ops program assigned to the member. Here, we only care about -* the arguments that are marked as __nullable. +* the arguments that are marked as __nullable, __ref or __arena. * * The array of struct bpf_ctx_arg_aux is eventually assigned to * prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
@@ -175,7 +176,7 @@ static int prepare_arg_info(struct btf *btf, struct bpf_struct_ops_arg_info *arg_info) {const struct btf_type *stub_func_proto, *pointed_type; -bool is_nullable = false, is_refcounted = false; +bool is_nullable = false, is_refcounted = false, is_arena = false;const struct btf_param *stub_args, *args;struct bpf_ctx_arg_aux *info, *info_buf;u32 nargs, arg_no, info_cnt =0 ;
@@ -226,26 +227,30 @@ static int prepare_arg_info(struct btf *btf, info = info_buf;for (arg_no =0 ; arg_no < nargs; arg_no++) {/* Skip arguments that is not suffixed with -* "__nullable or __ref". +* "__nullable", "__ref" or "__arena". */ is_nullable =btf_param_match_suffix (btf, &stub_args[arg_no],MAYBE_NULL_SUFFIX ); is_refcounted =btf_param_match_suffix (btf, &stub_args[arg_no],REFCOUNTED_SUFFIX ); + is_arena =btf_param_match_suffix (btf, &stub_args[arg_no], +ARENA_SUFFIX );if (is_nullable) suffix =MAYBE_NULL_SUFFIX ;else if (is_refcounted) suffix =REFCOUNTED_SUFFIX ; +else if (is_arena) + suffix =ARENA_SUFFIX ;else continue ; -/* Should be a pointer to struct */ +/* Should be a pointer to struct, or any pointer for __arena */ pointed_type =btf_type_resolve_ptr (btf, args[arg_no].type, &arg_btf_id);if (!pointed_type || - !btf_type_is_struct (pointed_type)) { + (!is_arena && !btf_type_is_struct (pointed_type))) {pr_warn ("stub function %s has %s tagging to an unsupported type\n" , stub_fname, suffix); goto err_out;
@@ -273,6 +278,8 @@ static int prepare_arg_info(struct btf *btf, }else if (is_refcounted) { info->reg_type =PTR_TRUSTED |PTR_TO_BTF_ID ; info->refcounted = true; + }else if (is_arena) { + info->reg_type =PTR_TO_ARENA ; } info++;
kernel/bpf/btf.c +7 -3
@@ -6961,15 +6961,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,return false; } -/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */ +/* +* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or +* PTR_TO_ARENA. +*/ for (i =0 ; i < prog->aux->ctx_arg_info_size; i++) {const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];u32 type, flag; type =base_type (ctx_arg_info->reg_type); flag =type_flag (ctx_arg_info->reg_type); -if (ctx_arg_info->offset == off && type ==PTR_TO_BUF && - (flag &PTR_MAYBE_NULL )) { +if (ctx_arg_info->offset == off && + (type ==PTR_TO_ARENA || + (type ==PTR_TO_BUF && (flag &PTR_MAYBE_NULL )))) { info->reg_type = ctx_arg_info->reg_type;return true; }
kernel/bpf/fixups.c +70 -0
82 lines not on this page · raw message
kernel/bpf/verifier.c +28 -9
@@ -18789,10 +18789,11 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) {const struct btf_type *t, *func_proto;const struct bpf_struct_ops_desc *st_ops_desc; +const struct bpf_struct_ops_arg_info *arg_info;const struct bpf_struct_ops *st_ops;const struct btf_member *member;struct bpf_prog *prog = env->prog; -bool has_refcounted_arg = false; +bool has_refcounted_arg = false, has_arena_arg = false;u32 btf_id, member_idx, member_off;struct btf *btf;const char *mname;
@@ -18867,21 +18868,40 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)return -EACCES ; } -for (i =0 ; i < st_ops_desc->arg_info[member_idx].cnt; i++) { -if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info = &st_ops_desc->arg_info[member_idx]; +for (i =0 ; i < arg_info->cnt; i++) { +const struct bpf_ctx_arg_aux *info = &arg_info->info[i]; + +if (info->refcounted) has_refcounted_arg = true; -break ; +if (base_type (info->reg_type) ==PTR_TO_ARENA ) { + has_arena_arg = true; +if (!prog->aux->arena) { +verbose (env, +"arena argument of %s requires a program with an associated arena\n" , + mname); +return -EINVAL ; + } } } -/* Tail call is not allowed for programs with refcounted arguments since we -* cannot guarantee that valid refcounted kptrs will be passed to the callee. +/* +* Tail calls reuse the caller's ctx, so a refcounted arg would reach +* the callee unverified, and an __arena arg, rebased in place by the +* entry prologue, would be rebased a second time by the callee's +* prologue. Neither is safe. */ for (i =0 ; i < env->subprog_cnt; i++) { -if (has_refcounted_arg && env->subprog_info[i].has_tail_call) { +if (!env->subprog_info[i].has_tail_call) +continue ; +if (has_refcounted_arg) {verbose (env,"program with __ref argument cannot tail call\n" );return -EINVAL ; } +if (has_arena_arg) { +verbose (env,"program with __arena argument cannot tail call\n" ); +return -EINVAL ; + } } prog->aux->st_ops = st_ops;
@@ -18891,8 +18911,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_name = mname; env->ops = st_ops->verifier_ops; -return bpf_prog_ctx_arg_info_init (prog, st_ops_desc->arg_info[member_idx].info, - st_ops_desc->arg_info[member_idx].cnt); +return bpf_prog_ctx_arg_info_init (prog, arg_info->info, arg_info->cnt); }#define SECURITY_PREFIX "security_"
Signature · 1 line
2.55.0
Tejun Heo
reviewerRFC PATCH 4/6 selftests/bpf: Add struct_ops __arena argument tests
In reply to Tejun HeoAdd a test_arena member to bpf_testmod_ops3 with an __arena stub argument and a kfunc that forwards a caller-provided pointer to it. The kfunc takes an __arena argument, so the verifier hands it a kernel arena address, which it passes to the callback. The struct_ops entry prologue rebases that address back to an arena pointer for the callback, so the test exercises both conversion directions end to end. The callback dereferences the argument directly and the trigger program verifies the write and NULL passing. A second program with no arena verifies the attach rejection.
--- .../bpf/prog_tests/test_struct_ops_arena.c | 61 +++++++++++++++++ .../selftests/bpf/progs/struct_ops_arena.c | 67 +++++++++++++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 ++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 12 ++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 2 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 1 + 6 files changed, 163 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +61 -0
@@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo <[email protected]> */ +#include <test_progs.h> + +#include "struct_ops_arena.skel.h" +#include "struct_ops_arena_fail.skel.h" + +/* Attach a callback with an __arena argument and drive it through the +* bpf_testmod_ops3_call_test_arena() kfunc. +*/ +static void arena_arg (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, topts); +struct struct_ops_arena *skel; +struct bpf_link *link =NULL ; +int err; + + skel =struct_ops_arena__open_and_load (); +if (!ASSERT_OK_PTR (skel,"struct_ops_arena__open_and_load" )) +return ; + + link =bpf_map__attach_struct_ops (skel->maps.testmod_arena); +if (!ASSERT_OK_PTR (link,"attach_struct_ops" )) + goto out; + + err =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.trigger), + &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,0 ,"trigger_retval" ); + +out: +bpf_link__destroy (link); +struct_ops_arena__destroy (skel); +} + +/* A program with no arena cannot attach to a member with an __arena +* argument. +*/ +static void arena_arg_fail (void ) +{ +struct struct_ops_arena_fail *skel; + + skel =struct_ops_arena_fail__open_and_load (); +if (ASSERT_ERR_PTR (skel,"struct_ops_arena_fail__open_and_load" )) +return ; + +struct_ops_arena_fail__destroy (skel); +} + +/* +* Serialized because it attaches the singleton bpf_testmod_ops3, which +* test_struct_ops_private_stack also attaches; registering it twice fails +* with -EEXIST. +*/ +void serial_test_struct_ops_arena (void ) +{ +if (test__start_subtest ("arena_arg" )) +arena_arg (); +if (test__start_subtest ("arena_arg_fail" )) +arena_arg_fail (); +}
tools/testing/selftests/bpf/progs/struct_ops_arena.c new file +67 -0
67 lines not on this page · raw message
tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo <[email protected]> */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod.h" + +char _license[] SEC("license") = "GPL"; + +/* No arena in the program: attaching to test_arena must be rejected. */ +SEC("struct_ops/test_arena") +int test_arena_no_arena (unsigned long long *ctx) +{ +return 0 ; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena_fail = { + .test_arena = (void *)test_arena_no_arena, +};
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +12 -0
@@ -351,9 +351,15 @@ static int bpf_testmod_test_4(void)return 0 ; } +static int bpf_testmod_ops3__test_arena (u64 *ptr__arena) +{ +return 0 ; +} +static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, + .test_arena = bpf_testmod_ops3__test_arena, };static void bpf_testmod_test_struct_ops3 (void )
@@ -372,6 +378,11 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) st_ops3->test_2 (); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena) +{ +return st_ops3->test_arena (ptr__arena); +} +struct bpf_testmod_btf_type_tag_1 {int a; };
@@ -771,6 +782,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena)BTF_ID_FLAGS (func, bpf_kfunc_get_default_trusted_ptr_test);BTF_ID_FLAGS (func, bpf_kfunc_put_default_trusted_ptr_test);BTF_KFUNCS_END (bpf_testmod_common_kfunc_ids)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +2 -0
@@ -106,6 +106,8 @@ struct bpf_testmod_ops2 {struct bpf_testmod_ops3 {int (*test_1)(void );int (*test_2)(void ); +/* Used to test arena pointer arguments. */ +int (*test_arena)(u64 *ptr); };struct st_ops_args {
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1 -0
@@ -115,6 +115,7 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;#endif void bpf_testmod_test_mod_kfunc(int i) __ksym; +int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCHSET SLOP RFC 4/6] selftests/bpf: Add struct_ops __arena argument tests
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Multi-line comments in BPF-related code must have the opening `/*` on its own line. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Multi-line comments in BPF-related code must have the opening /* on its own line. --
--- Patch [4]: [PATCHSET SLOP RFC 4/6] selftests/bpf: Add struct_ops __arena argument tests --- commit 9e21274b1ecab24a7e9f15aad54b02d04ae38d78
Author: Tejun Heo <[email protected]>
selftests/bpf: Add struct_ops __arena argument tests
This commit adds tests for struct_ops members with an __arena argument. It creates a callback, forwards an arena pointer through a kfunc, and verifies the read/write access as well as attach rejection for programs without an arena.
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +0
-0
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c line 10
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +10
-0@@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo <[email protected]> */ +#include <test_progs.h> + +#include "struct_ops_arena.skel.h" +#include "struct_ops_arena_fail.skel.h" + +/* Attach a callback with an __arena argument and drive it through the +* bpf_testmod_ops3_call_test_arena() kfunc. +*/
[Severity: Low]
This isn't a bug, but the BPF subsystem prefers multi-line comments to have the opening /* on its own line. Should this be updated to match the modern kernel multi-line comment style?
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +2
-0+static void arena_arg (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c line 38
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +7
-0+bpf_link__destroy (link); +struct_ops_arena__destroy (skel); +} + +/* A program with no arena cannot attach to a member with an __arena +* argument. +*/
[Severity: Low]
This isn't a bug, but similarly here, could the opening /* be placed on its own line to align with the subsystem's preferred comment style?
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +2
-0+static void arena_arg_fail (void ) +{
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Tejun Heo
reviewerRe: [PATCHSET SLOP RFC 4/6] selftests/bpf: Add struct_ops __arena argument tests
On the comment style note:
v2 reflows the flagged comments (opening /* on its own line) and adds a reproducer for the patch 3 tail-call rejection: a callback with an __arena argument that tail calls, confirmed rejected.
Thanks.
Tejun Heo
reviewerRFC PATCH 5/6 sched_ext: Pass a kernel arena pointer to ops_cid.set_cmask()
In reply to Tejun HeoThe cid-form set_cmask() callback receives a per-CPU cmask that the kernel builds in the arena. Without an __arena tag the callback would receive the argument as a trusted scx_cmask BTF pointer, and the kernel had to convert the kernel address to the arena pointer form by hand before the call.
Tag the stub argument __arena and pass the kernel arena address directly. The struct_ops entry prologue rebases it to the program's arena pointer, so the manual scx_kaddr_to_arena() conversion and its now-unused helper are removed.
NOT_SIGNED_OFF: to be reworked after bpf-next is pulled into sched_ext
--- kernel/sched/ext/ext.c | 11 ++++++----- kernel/sched/ext/internal.h | 16 +++------------- 2 files changed, 9 insertions(+), 18 deletions(-)
kernel/sched/ext/ext.c +6 -5
@@ -508,12 +508,13 @@ static inline void scx_call_op_set_cpumask(struct scx_sched *sch, struct rq *rq,if (scx_is_cid_type ()) {struct scx_cmask *kern_va = *this_cpu_ptr (sch->set_cmask_scratch);/* -* Build the per-CPU arena cmask and hand BPF its arena address. -* Caller holds the rq lock with IRQs disabled, which makes us -* the sole user of the scratch area. +* Build the per-CPU arena cmask and pass its kernel address. +* The BPF side translates it to an arena pointer. Caller holds +* the rq lock with IRQs disabled, which makes us the sole user +* of the scratch area. */ scx_cpumask_to_cmask (cpumask, kern_va); - sch->ops_cid.set_cmask (task,scx_kaddr_to_arena (sch, kern_va)); + sch->ops_cid.set_cmask (task, kern_va); }else { sch->ops.set_cpumask (task, cpumask); }
@@ -8130,7 +8131,7 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {* set_cmask needs a fresh stub since the second argument type differs. */ static void sched_ext_ops_cid__set_cmask (struct task_struct *p, -const struct scx_cmask *cmask) {} +const struct scx_cmask *cmask__arena) {}static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = { .select_cid = sched_ext_ops__select_cpu,
kernel/sched/ext/internal.h +3 -13
@@ -1125,8 +1125,7 @@ struct scx_sched {* * @arena_pool sub-allocates @arena_map. Each gen_pool chunk is added * at the kernel-side mapping address. @arena_kern_base is the start -* of the arena's kern_vm range. See scx_arena_to_kaddr() and -* scx_kaddr_to_arena(). +* of the arena's kern_vm range. See scx_arena_to_kaddr(). */ struct bpf_map *arena_map;struct gen_pool *arena_pool;
@@ -1135,7 +1134,8 @@ struct scx_sched {/* * Per-CPU arena cmask used by scx_call_op_set_cpumask() to hand a cmask * to ops_cid.set_cmask(). The kernel writes through the stored kern_va -* and hands BPF its arena pointer via scx_kaddr_to_arena(). +* and passes it as the callback argument; the struct_ops entry prologue +* rebases it to the program's arena pointer. */ struct scx_cmask *__percpu *set_cmask_scratch;
@@ -1227,16 +1227,6 @@ static inline void *scx_arena_to_kaddr(struct scx_sched *sch, const void *bpf_ptreturn (void *)(sch->arena_kern_base + (u32 )(uintptr_t )bpf_ptr); } -/** -* scx_kaddr_to_arena - Translate a kernel arena address to its BPF form -* @sch: scheduler whose arena hosts @kaddr -* @kaddr: kernel-side arena address, supplied by trusted kernel code -*/ -static inline void *scx_kaddr_to_arena (struct scx_sched *sch,const void *kaddr) -{ -return (void *)((uintptr_t )kaddr - sch->arena_kern_base); -} -enum scx_wake_flags {/* expose select WF_* flags as enums */ SCX_WAKE_FORK =WF_FORK ,
Signature · 1 line
2.55.0
Tejun Heo
reviewerRFC PATCH 6/6 sched_ext: Convert scx_bpf_cid_override() to take an arena pointer
In reply to Tejun HeoThe mem+size signature requires a verifier-visible buffer, which forces scx_qmap to keep the cpu_to_cid array in writable bss even though nothing writes it after load. Take the array as an __arena argument instead, with the count passed in entries. The verifier hands the kfunc a dereferenceable kernel address and nr_cpu_ids * sizeof(s32) stays within the guard region covered by arena fault recovery, so the kfunc needs no bounds check beyond the entry count validation.
scx_qmap moves the array into struct qmap_arena. As the arena is mmapped at load, the loader now populates it between load and attach instead of before load.
NOT_SIGNED_OFF: to be reworked after bpf-next is pulled into sched_ext
--- kernel/sched/ext/cid.c | 24 ++++++++++---- tools/sched_ext/include/scx/compat.bpf.h | 8 +++-- tools/sched_ext/scx_qmap.bpf.c | 13 ++------ tools/sched_ext/scx_qmap.c | 41 +++++++++++++++--------- tools/sched_ext/scx_qmap.h | 3 ++ 5 files changed, 53 insertions(+), 36 deletions(-)
kernel/sched/ext/cid.c +17 -7
@@ -278,8 +278,8 @@ __bpf_kfunc_start_defs();/** * scx_bpf_cid_override - Install an explicit cpu->cid mapping -* @cpu_to_cid: array of nr_cpu_ids s32 entries (cid for each cpu) -* @cpu_to_cid__sz: must be nr_cpu_ids * sizeof(s32) bytes +* @cpu_to_cid__arena: arena array of nr_cpu_ids s32 entries (cid for each cpu) +* @cnt: number of entries, must be nr_cpu_ids * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * * May only be called from ops.init() of the root scheduler. Replace the
@@ -287,7 +287,7 @@ __bpf_kfunc_start_defs();* must map to a unique cid in [0, num_possible_cpus()). Topo info is cleared. * On invalid input, trigger scx_error() to abort the scheduler. */ -__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, +__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid__arena, u32 cnt,const struct bpf_prog_aux *aux) { cpumask_var_t seen __free(free_cpumask_var) = CPUMASK_VAR_NULL;
@@ -314,14 +314,24 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,return ; } -if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) { -scx_error (sch,"scx_bpf_cid_override: expected %zu bytes, got %u" , - nr_cpu_ids *sizeof (s32), cpu_to_cid__sz); +if (!cpu_to_cid__arena) { +scx_error (sch,"scx_bpf_cid_override: NULL cpu_to_cid" );return ; } +if (cnt != nr_cpu_ids) { +scx_error (sch,"scx_bpf_cid_override: expected %u entries, got %u" , + nr_cpu_ids, cnt); +return ; + } + +/* +* @cpu_to_cid__arena arrives rebased to the arena kernel mapping. +* nr_cpu_ids * sizeof(s32) stays within the guard region covered by +* arena fault recovery, so no explicit bounds check is needed. +*/ for_each_possible_cpu (cpu) { -s32 c = cpu_to_cid[cpu]; +s32 c = cpu_to_cid__arena[cpu];if (!cid_valid (sch, c))return ;
tools/sched_ext/include/scx/compat.bpf.h +5 -3
@@ -7,6 +7,8 @@#ifndef __SCX_COMPAT_BPF_H#define __SCX_COMPAT_BPF_H +#include "bpf_arena_common.bpf.h" +#define __COMPAT_ENUM_OR_ZERO (__type, __ent) \ ({ \ __type __ret = 0; \
@@ -125,12 +127,12 @@ static inline bool scx_bpf_sub_dispatch(u64 cgroup_id)* v7.2: scx_bpf_cid_override() for explicit cpu->cid mapping. Ignore if * missing. */ -void scx_bpf_cid_override___compat(const s32 *cpu_to_cid, u32 cpu_to_cid__sz) __ksym __weak; +void scx_bpf_cid_override___compat(const s32 __arena *cpu_to_cid, u32 cnt) __ksym __weak; -static inline void scx_bpf_cid_override (const s32 *cpu_to_cid,u32 cpu_to_cid__sz) +static inline void scx_bpf_cid_override(const s32 __arena *cpu_to_cid, u32 cnt) {if (bpf_ksym_exists (scx_bpf_cid_override___compat)) -return scx_bpf_cid_override___compat (cpu_to_cid, cpu_to_cid__sz); +return scx_bpf_cid_override___compat (cpu_to_cid, cnt); }/**
tools/sched_ext/scx_qmap.bpf.c +2 -11
@@ -63,13 +63,6 @@ const volatile u32 max_tasks;* 3 = invalid: out-of-range cid */ const volatile u32 cid_override_mode; -/* -* Array lives in bss (writable) because scx_bpf_cid_override()'s BPF -* verifier signature treats its len-paired pointer as read/write - rodata -* fails verification with "write into map forbidden". Userspace populates -* it before SCX_OPS_LOAD, same as rodata, and nothing writes it after. -*/ -s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS ];UEI_DEFINE (uei);
@@ -1094,10 +1087,8 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)* cid space (scx_bpf_nr_cids, cmask_init, etc.). On invalid input, * the kfunc calls scx_error() which aborts the scheduler. */ -if (cid_override_mode) { -scx_bpf_cid_override ((const s32 *)cid_override_cpu_to_cid, - nr_cpu_ids *sizeof (s32)); - } +if (cid_override_mode) +scx_bpf_cid_override (qa.cid_override_cpu_to_cid, nr_cpu_ids);/* * Allocate the task_ctx slab in arena and thread the entire slab onto
tools/sched_ext/scx_qmap.c +26 -15
61 lines not on this page · raw message
tools/sched_ext/scx_qmap.h +3 -0
@@ -62,6 +62,9 @@ struct qmap_arena {struct cpu_ctx cpu_ctxs[SCX_QMAP_MAX_CPUS ]; +/* cid-override test input, populated by the loader before attach */ +__s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS ]; +/* task_ctx slab; allocated and threaded by qmap_init() */ struct task_ctx __arena *task_ctxs; struct task_ctx __arena *task_free_head;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCHSET SLOP RFC 6/6] sched_ext: Convert scx_bpf_cid_override() to take an arena pointer
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing READ_ONCE() when reading from shared arena memory allows a TOCTOU race leading to out-of-bounds writes. --
--- Patch [6]: [PATCHSET SLOP RFC 6/6] sched_ext: Convert scx_bpf_cid_override() to take an arena pointer --- commit 0bba9b04cbf24d912ae66699e643f62ba90a630b
Author: Tejun Heo <[email protected]>
sched_ext: Convert scx_bpf_cid_override() to take an arena pointer
This commit changes scx_bpf_cid_override() to take an arena pointer for the cpu_to_cid array instead of a verifier-visible buffer. It validates the entry count against nr_cpu_ids, relying on the arena's guard region to prevent out-of-bounds memory accesses.
Quoted 4 lines
kernel/sched/ext/cid.c +0
-0
[ ... ]
Quoted 1 line
kernel/sched/ext/cid.c +0
-0@@ -314,14 +314,24 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz,
[ ... ]
kernel/sched/ext/cid.c __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, · line 337
Quoted 11 lines
kernel/sched/ext/cid.c +6
-1+/* +* @cpu_to_cid__arena arrives rebased to the arena kernel mapping. +* nr_cpu_ids * sizeof(s32) stays within the guard region covered by +* arena fault recovery, so no explicit bounds check is needed. +*/ for_each_possible_cpu (cpu) { -s32 c = cpu_to_cid[cpu]; +s32 c = cpu_to_cid__arena[cpu];if (!cid_valid (sch, c))return ;
[Severity: High]
This is a pre-existing issue, but since the array resides in shared userspace-mapped memory (and now an mmaped arena), should READ_ONCE() be used when reading cpu_to_cid__arena[cpu]?
Without READ_ONCE(), the compiler could potentially reload the value of c from memory after the bounds check in cid_valid(sch, c). If userspace concurrently modifies the arena memory between the check and its subsequent use as an array index:
if (cpumask_test_and_set_cpu(c, seen)) {
scx_error(sch, "cid %d assigned to multiple cpus", c);
return;
}
scx_cpu_to_cid_tbl[cpu] = c;
scx_cid_to_cpu_tbl[c] = cpu;Could this Time-of-Check to Time-of-Use (TOCTOU) race result in an out-of-bounds write to scx_cid_to_cpu_tbl[c]?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Tejun Heo
reviewerRe: [PATCHSET SLOP RFC 6/6] sched_ext: Convert scx_bpf_cid_override() to take an arena pointer
On the pre-existing READ_ONCE() note:
Agreed as hardening for concurrently-writable arena memory (in practice c is a register-held local, so the reload doesn't happen, but READ_ONCE() is the more robust idiom). This is a NOT_SIGNED_OFF demo; I'll add it in the sched_ext rework after bpf-next is pulled in, not in this RFC patch.
Thanks.