bpf: Support 16-byte return values in the R0:R2 register pair
Subsystems (12)
Yonghong Song
reviewercover bpf: Support 16-byte return values in the R0:R2 register pair
Until now a BPF subprogram or kfunc could only return a scalar value of up to 8 bytes, passed back in R0. LLVM 23 added the ability to return a value in two registers for the BPF target [1][2]: an __int128, or a struct/union whose size is greater than 8 and up to 16 bytes, is returned in the R0:R2 register pair, with the low 64 bits in R0 and the high 64 bits in R2. (Before LLVM 23 the BPF backend could not return such values at all:
a by-value aggregate return was rejected with "aggregate returns are not supported", and an __int128 return failed in the backend with "unable to allocate function return #1".)
This series teaches the kernel about that convention so that a BPF-to-BPF subprogram call and a kfunc call can return up to 16 bytes. A returned struct or union must be composed only of scalars (recursively); the bytes are handed back as the raw contents of R0:R2. Otherwise allowing pointer members would launder a pointer into a scalar and escape the verifier's provenance and reference tracking. A value larger than 16 bytes remains unsupported, as does a <=8 byte return, which continues to use R0 alone.
Support is built up incrementally so each step is reviewable on its own, and the feature is only turned on end to end by the last enabling patch.
- Patch 1 refactors the global-subprog return check so the per-register scalar validation can be reused for R2.
- Patch 2 to Patch 7 are preparation patches for core verifier handling of R0:R2 returns, JIT support, precision backtracking, liveness analysis, and limitation on trampoline bpf programs.
- Patch 8 relaxes btf_distill_func_proto() and btf_validate_return_type() to accept <=16 byte aggregates, enabling the feature end to end.
- Patches 9-11 add selftests including C and inline asm tests.
- Patch 12 documents the kfunc return-value convention in kfuncs.rst.
Tested on x86 and arm64. The patch set is built on top of the following commit:
Merge branch 'verify-bpf-signed-loader-at-load-time'
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876Yonghong Song (12):
bpf: Factor check_global_ret_scalar_reg() out of the global return
check
bpf: Add verifier support for 16-byte returns in R0:R2
bpf: Wire up JIT support for 16-byte kfunc returns
bpf: Track R2 of register-pair returns in precision backtracking
bpf: Account R2 of register-pair returns in live register analysis
bpf: Force JIT for programs using the R0:R2 register pair
bpf: Reject >8 byte return values on return-reading trampoline paths
bpf: Enable 16-byte aggregate return types
selftests/bpf: Add C tests for 16-byte returns in R0:R2 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns selftests/bpf: Cover tracing on >8 and <=16 byte return targets Documentation/bpf: Document 16-byte kfunc return values in R0:R2
Documentation/bpf/kfuncs.rst | 36 ++ arch/arm64/net/bpf_jit_comp.c | 5 + arch/riscv/net/bpf_jit_comp64.c | 5 + arch/x86/net/bpf_jit_comp.c | 19 + include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 4 + include/linux/filter.h | 1 + kernel/bpf/backtrack.c | 32 +- kernel/bpf/bpf_struct_ops.c | 12 + kernel/bpf/btf.c | 14 +- kernel/bpf/core.c | 7 +- kernel/bpf/liveness.c | 13 +- kernel/bpf/verifier.c | 179 ++++++++- .../selftests/bpf/prog_tests/aggregate_ret.c | 249 ++++++++++++ .../selftests/bpf/progs/aggregate_ret_func.c | 361 ++++++++++++++++++ .../bpf/progs/aggregate_ret_int128_c.c | 48 +++ .../selftests/bpf/progs/aggregate_ret_kfunc.c | 81 ++++ .../selftests/bpf/progs/aggregate_ret_run.c | 160 ++++++++ .../bpf/progs/aggregate_ret_struct_c.c | 98 +++++ .../selftests/bpf/progs/aggregate_ret_trace.c | 44 +++ .../bpf/progs/aggregate_ret_union_c.c | 51 +++ .../selftests/bpf/progs/exceptions_fail.c | 2 +- .../selftests/bpf/test_kmods/bpf_testmod.c | 80 ++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 5 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 38 ++ 25 files changed, 1517 insertions(+), 28 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 1/12 bpf: Factor check_global_ret_scalar_reg() out of the global return check
Function check_global_subprog_return_code() verifies that a global subprogram returns void, an arena pointer, or register R0 which holds a scalar value. Later patches in this series add 16-byte aggregate return support, whose second half is returned in R2 and needs similar validation.
Factor the per-register check into check_global_ret_scalar_reg(env, regno) so it can be reused for R2.
--- kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
kernel/bpf/verifier.c +20 -12
@@ -16679,37 +16679,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const charreturn 0 ; } -static int check_global_subprog_return_code (struct bpf_verifier_env *env) +static int check_global_ret_scalar_reg (struct bpf_verifier_env *env,u32 regno) { -struct bpf_reg_state *reg =reg_state (env,BPF_REG_0 ); -struct bpf_func_state *cur_frame =cur_func (env); +struct bpf_reg_state *reg;int err; -if (subprog_returns_void (env, cur_frame->subprogno)) -return 0 ; - - err =check_reg_arg (env,BPF_REG_0 ,SRC_OP ); + err =check_reg_arg (env, regno,SRC_OP );if (err)return err;/* Pointers to arena are safe to pass between subprograms. */ -if (is_arena_reg (env,BPF_REG_0 )) +if (is_arena_reg (env, regno))return 0 ; -if (is_pointer_value (env,BPF_REG_0 )) { -verbose (env,"R%d leaks addr as return value\n" ,BPF_REG_0 ); +if (is_pointer_value (env, regno)) { +verbose (env,"R%d leaks addr as return value\n" , regno);return -EACCES ; } + reg =reg_state (env, regno);if (reg->type !=SCALAR_VALUE ) { -verbose (env,"At subprogram exit the register R0 is not a scalar value (%s)\n" , -reg_type_str (env, reg->type)); +verbose (env,"At subprogram exit the register R%d is not a scalar value (%s)\n" , + regno,reg_type_str (env, reg->type));return -EINVAL ; }return 0 ; } +static int check_global_subprog_return_code (struct bpf_verifier_env *env) +{ +struct bpf_func_state *cur_frame =cur_func (env); +u32 subprog = cur_frame->subprogno; + +if (subprog_returns_void (env, subprog)) +return 0 ; + +return check_global_ret_scalar_reg (env,BPF_REG_0 ); +} +/* Bitmask with 1s for all caller saved registers */ #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 2/12 bpf: Add verifier support for 16-byte returns in R0:R2
In reply to Yonghong SongLLVM 23 added support for returning a value in two registers for an __int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].
Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:
error: aggregate returns are not supported
and an __int128 return failed later in the backend with:
fatal error: error in backend: unable to allocate function return #1
Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.
This patch is one of several preparation patches that build up <=16 byte return support incrementally:
- the verifier R0:R2 modelling in this patch,
- the x86 JIT move and JIT-capability gate,
- precision-backtracking and live-register tracking of R2,
- forcing the JIT for such programs, and
- guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.
This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
--- kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 88 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +88 -5
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)return btf_type_is_void (type); } +static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +{ +const struct btf_type *type, *func, *func_proto; +const struct btf *btf = env->prog->aux->btf; +u32 btf_id; + +if (!btf || !env->prog->aux->func_info) +return false; + + btf_id = env->prog->aux->func_info[subprog].type_id; + + func =btf_type_by_id (btf, btf_id); +if (!func) +return false; + + func_proto =btf_type_by_id (btf, func->type); +if (!func_proto) +return false; + + type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); +if (!type) +return false; + +if (btf_type_is_struct (type) ||btf_type_is_scalar (type)) +return type->size >8 && type->size <=16 ; + +return false; +} +static const char *subprog_name (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info *info;
@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,clear_caller_saved_regs (env, caller->regs);invalidate_outgoing_stack_args (env,cur_func (env)); -/* All non-void global functions return a 64-bit SCALAR_VALUE. */ +/* +* A non-void global function returns a 64-bit SCALAR_VALUE in +* R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. +*/ if (!subprog_returns_void (env, subprog)) {mark_reg_unknown (env, caller->regs,BPF_REG_0 ); caller->regs[BPF_REG_0 ].subreg_def =DEF_NOT_SUBREG ; +if (bpf_ret_reg_pair (env, subprog)) { +mark_reg_unknown (env, caller->regs,BPF_REG_2 ); + caller->regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + } }if (env->subprog_info[subprog].might_throw) {
@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0; +if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) { +verbose (env,"cannot return stack pointer to the caller\n" ); +return -EINVAL ; + } + caller->regs[BPF_REG_2 ] = callee->regs[BPF_REG_2 ]; + } }/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,return __mark_btf_func_reg_size (env,cur_regs (env), regno, reg_size); } +static void mark_kfunc_ret_reg_size (struct bpf_verifier_env *env, +struct bpf_reg_state *regs,u32 size) +{ +if (size >8 ) { +mark_btf_func_reg_size (env,BPF_REG_0 ,8 ); +mark_reg_unknown (env, regs,BPF_REG_2 ); + regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + }else { +mark_btf_func_reg_size (env,BPF_REG_0 , size); + } +} +static bool is_kfunc_acquire (struct bpf_kfunc_call_arg_meta *meta) {return meta->kfunc_flags &KF_ACQUIRE ;
@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))__mark_reg_const_zero (env, ®s[BPF_REG_0 ]); -mark_btf_func_reg_size (env,BPF_REG_0 , t->size); +mark_kfunc_ret_reg_size (env, regs, t->size); + }else if (btf_type_is_struct (t)) { +/* +* The returned struct comes back as raw register bits modeled +* as an unknown scalar, so it must contain only scalars: +* otherwise a pointer field would be laundered into a scalar +* and escape provenance and reference tracking. +*/ +if (!__btf_type_is_scalar_struct (env, desc_btf, t,0 )) { +verbose (env,"kernel function %s returns %s %s that is not composed of scalars\n" , + func_name,btf_type_str (t), +btf_name_by_offset (desc_btf, t->name_off)); +return -EINVAL ; + } +mark_reg_unknown (env, regs,BPF_REG_0 ); +mark_kfunc_ret_reg_size (env, regs, t->size); }else if (btf_type_is_ptr (t)) { ptr_type =btf_type_skip_modifiers (desc_btf, t->type, &ptr_type_id); err =check_special_kfunc (env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16711,11 +16781,19 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env) {struct bpf_func_state *cur_frame =cur_func (env);u32 subprog = cur_frame->subprogno; +int err;if (subprog_returns_void (env, subprog))return 0 ; -return check_global_ret_scalar_reg (env,BPF_REG_0 ); + err =check_global_ret_scalar_reg (env,BPF_REG_0 ); +if (err) +return err; + +if (!bpf_ret_reg_pair (env, subprog)) +return 0 ; + +return check_global_ret_scalar_reg (env,BPF_REG_2 ); }/* Bitmask with 1s for all caller saved registers */
@@ -17205,10 +17283,15 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,*/ if (cur_frame->subprogno && !cur_frame->in_async_callback_fn && - !cur_frame->in_exception_callback_fn) + !cur_frame->in_exception_callback_fn) { err =check_global_subprog_return_code (env); -else + }else { +if (!cur_frame->subprogno &&bpf_ret_reg_pair (env,0 )) { +verbose (env,"return value larger than 8 bytes is not supported at program exit\n" ); +return -EINVAL ; + } err =check_return_code (env,BPF_REG_0 ,"R0" ); + }if (err)return err;return PROCESS_BPF_EXIT ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2
This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void() share a helper for the func_info -> func -> func_proto -> return type walk?
Quoted 26 lines
kernel/bpf/verifier.c +18
-0@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)return btf_type_is_void (type); } +static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +{ +const struct btf_type *type, *func, *func_proto; +const struct btf *btf = env->prog->aux->btf; +u32 btf_id; + +if (!btf || !env->prog->aux->func_info) +return false; + + btf_id = env->prog->aux->func_info[subprog].type_id; + + func =btf_type_by_id (btf, btf_id); +if (!func) +return false; + + func_proto =btf_type_by_id (btf, func->type); +if (!func_proto) +return false;
kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 454
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ + type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); +if (!type) +return false; + +if (btf_type_is_struct (type) ||btf_type_is_scalar (type)) +return type->size >8 && type->size <=16 ; + +return false; +}
This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void() share a helper for the func_info -> func -> func_proto -> return type walk?
Both functions repeat the same ~8 line sequence:
subprog_returns_void():
btf_id = env->prog->aux->func_info[subprog].type_id; func = btf_type_by_id(btf, btf_id); if (!func) return false; func_proto = btf_type_by_id(btf, func->type); if (!func_proto) return false; type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
They only differ in the final predicate: subprog_returns_void() checks btf_type_is_void(type), while bpf_ret_reg_pair() checks the struct/scalar size.
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Yonghong Song
reviewerRe: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2
Quoted 51 lines · On 7/8/26 2:17 PM, [email protected] wrote:
Quoted 26 lines
kernel/bpf/verifier.c +18
-0@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)return btf_type_is_void (type); } +static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +{ +const struct btf_type *type, *func, *func_proto; +const struct btf *btf = env->prog->aux->btf; +u32 btf_id; + +if (!btf || !env->prog->aux->func_info) +return false; + + btf_id = env->prog->aux->func_info[subprog].type_id; + + func =btf_type_by_id (btf, btf_id); +if (!func) +return false; + + func_proto =btf_type_by_id (btf, func->type); +if (!func_proto) +return false;kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 454
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ + type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); +if (!type) +return false; + +if (btf_type_is_struct (type) ||btf_type_is_scalar (type)) +return type->size >8 && type->size <=16 ; + +return false; +}This isn't a bug, but could
bpf_ret_reg_pair()andsubprog_returns_void()share a helper for thefunc_info-> func ->func_proto-> return type walk?Both functions repeat the same ~8 line sequence:
subprog_returns_void():btf_id = env->prog->aux->func_info[subprog].type_id; func = btf_type_by_id(btf, btf_id); if (!func) return false; func_proto = btf_type_by_id(btf, func->type); if (!func_proto) return false; type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
Quoted 5 lines
They only differ in the final predicate:
subprog_returns_void()checksbtf_type_is_void(type), whilebpf_ret_reg_pair()checks the struct/scalar size.[ ... ]
Indeed, I think this probably worthwhile to factor a new helper.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Eduard Zingerman
maintainerRe: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2
In reply to Yonghong SongQuoted 17 lines · Yonghong Song
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:
error: aggregate returns are not supported
and an
__int128return failed later in the backend with:fatal error: error in backend: unable to allocate function return #1
Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.
Quoted 9 lines
This patch is one of several preparation patches that build up <=16 byte return support incrementally:
- the verifier R0:R2 modelling in this patch,
- the x86 JIT move and JIT-capability gate,
- precision-backtracking and live-register tracking of R2,
- forcing the JIT for such programs, and
- guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.
Nit: I think the above paragraphs belong to a cover letter.
Quoted 24 lines
This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876Signed-off-by Yonghong Song <[email protected]>---
kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---1 file changed, 88 insertions(+), 5 deletions(-)kernel/bpf/verifier.c +0
-0@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) return btf_type_is_void(type); }kernel/bpf/verifier.c +4
-0+static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +{ +const struct btf_type *type, *func, *func_proto; +const struct btf *btf = env->prog->aux->btf;
kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 440
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+u32 btf_id; + +if (!btf || !env->prog->aux->func_info) +return false; + + btf_id = env->prog->aux->func_info[subprog].type_id; + + func =btf_type_by_id (btf, btf_id); +if (!func) +return false;
if (!func) return false checks here and below are not necessary after BTF validation.
Quoted 38 lines
kernel/bpf/verifier.c +15
-0+ + func_proto =btf_type_by_id (btf, func->type); +if (!func_proto) +return false; + + type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); +if (!type) +return false; + +if (btf_type_is_struct (type) ||btf_type_is_scalar (type)) +return type->size >8 && type->size <=16 ; + +return false; +} +static const char *subprog_name(const struct
bpf_verifier_env*env, int subprog) {
structbpf_func_info*info;kernel/bpf/verifier.c +0
-0@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env));kernel/bpf/verifier.c +4
-1-/* All non-void global functions return a 64-bit SCALAR_VALUE. */ +/* +* A non-void global function returns a 64-bit SCALAR_VALUE in +* R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. +*/ if (!subprog_returns_void(env, subprog)) {
mark_reg_unknown(env, caller->regs, BPF_REG_0); caller->regs[BPF_REG_0].subreg_def =DEF_NOT_SUBREG;kernel/bpf/verifier.c +4
-0+if (bpf_ret_reg_pair (env, subprog)) { +mark_reg_unknown (env, caller->regs,BPF_REG_2 ); + caller->regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + }}
if (env->subprog_info[subprog].might_throw) {
kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9865
Quoted 5 lines
kernel/bpf/verifier.c +2
-0@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) } else { /* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0] = *r0; +if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) {
Let's consolidate all such checks in one place and not copy-paste them.
Quoted 13 lines
kernel/bpf/verifier.c +5
-0+verbose (env,"cannot return stack pointer to the caller\n" ); +return -EINVAL ; + } + caller->regs[BPF_REG_2 ] = callee->regs[BPF_REG_2 ]; + } }
/* for callbacks likebpf_looporbpf_for_each_map_elemgo back to callsite,kernel/bpf/verifier.c +0
-0@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size); }kernel/bpf/verifier.c +1
-0+static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,
kernel/bpf/verifier.c static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, · line 10801
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ struct bpf_reg_state *regs, u32 size) +{ +if (size >8 ) { +mark_btf_func_reg_size (env,BPF_REG_0 ,8 ); +mark_reg_unknown (env, regs,BPF_REG_2 ); + regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + }else { +mark_btf_func_reg_size (env,BPF_REG_0 , size); + } +}
This function makes R2 handling very asymmetric compared to R0:
- mark_reg_unknown() for r0 is done in check_kfunc_call()
- DEF_NOT_SUBREG for r0 is done in mark_btf_func_reg_size()
I think the code should be structured to keep R0 and R2 processing uniform.
Quoted 11 lines
+
static boolis_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta){
returnmeta->kfunc_flags&KF_ACQUIRE;kernel/bpf/verifier.c +0
-0@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (meta.btf ==
btf_vmlinux&& (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))__mark_reg_const_zero(env, ®s[BPF_REG_0]);kernel/bpf/verifier.c +2
-1-mark_btf_func_reg_size (env,BPF_REG_0 , t->size); +mark_kfunc_ret_reg_size (env, regs, t->size); + }else if (btf_type_is_struct (t)) {
kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 13277
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* +* The returned struct comes back as raw register bits modeled +* as an unknown scalar, so it must contain only scalars: +* otherwise a pointer field would be laundered into a scalar +* and escape provenance and reference tracking. +*/ +if (!__btf_type_is_scalar_struct (env, desc_btf, t,0 )) { +verbose (env,"kernel function %s returns %s %s that is not composed of scalars\n" , + func_name,btf_type_str (t), +btf_name_by_offset (desc_btf, t->name_off));
Should this also check the size of the struct?
Quoted 7 lines
kernel/bpf/verifier.c +4
-0+return -EINVAL ; + } +mark_reg_unknown (env, regs,BPF_REG_0 ); +mark_kfunc_ret_reg_size (env, regs, t->size);} else if (
btf_type_is_ptr(t)) {
ptr_type=btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err =check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2
Quoted 27 lines · Eduard Zingerman
Quoted 17 lines · Yonghong Song
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:
error: aggregate returns are not supported
and an
__int128return failed later in the backend with:fatal error: error in backend: unable to allocate function return #1
Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.
Quoted 9 lines
This patch is one of several preparation patches that build up <=16 byte return support incrementally:
- the verifier R0:R2 modelling in this patch,
- the x86 JIT move and JIT-capability gate,
- precision-backtracking and live-register tracking of R2,
- forcing the JIT for such programs, and
- guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.
Quoted 1 line
Nit: I think the above paragraphs belong to a cover letter.
Will add this to the cover letter.
Quoted 21 lines
Quoted 21 lines
This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876Signed-off-by Yonghong Song <[email protected]>--- kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 88 insertions(+), 5 deletions(-)kernel/bpf/verifier.c +0
-0@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) return btf_type_is_void(type); }
Quoted 16 lines
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +{ +const struct btf_type *type, *func, *func_proto; +const struct btf *btf = env->prog->aux->btf;kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 440
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+u32 btf_id; + +if (!btf || !env->prog->aux->func_info) +return false; + + btf_id = env->prog->aux->func_info[subprog].type_id; + + func =btf_type_by_id (btf, btf_id); +if (!func) +return false;
if (!func) return falsechecks here and below are not necessary after BTF validation.
Ack. Will remove them.
Quoted 37 lines
Quoted 37 lines
kernel/bpf/verifier.c +23
-1+ + func_proto =btf_type_by_id (btf, func->type); +if (!func_proto) +return false; + + type =btf_type_skip_modifiers (btf, func_proto->type,NULL ); +if (!type) +return false; + +if (btf_type_is_struct (type) ||btf_type_is_scalar (type)) +return type->size >8 && type->size <=16 ; + +return false; +} + static const char *subprog_name(const struct bpf_verifier_env *env, int subprog) { struct bpf_func_info *info;@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); -/* All non-void global functions return a 64-bit SCALAR_VALUE. */ +/* +* A non-void global function returns a 64-bit SCALAR_VALUE in +* R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. +*/ if (!subprog_returns_void(env, subprog)) { mark_reg_unknown(env, caller->regs, BPF_REG_0); caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; +if (bpf_ret_reg_pair (env, subprog)) { +mark_reg_unknown (env, caller->regs,BPF_REG_2 ); + caller->regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + } }
Quoted 8 lines
kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9865
Quoted 7 lines
kernel/bpf/verifier.c +2
-0if (env->subprog_info[subprog].might_throw) {@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) } else { /* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0] = *r0; +if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) {Let's consolidate all such checks in one place and not copy-paste them.
Will try to consolidate them together.
Quoted 28 lines
Quoted 14 lines
kernel/bpf/verifier.c +6
-0+verbose (env,"cannot return stack pointer to the caller\n" ); +return -EINVAL ; + } + caller->regs[BPF_REG_2 ] = callee->regs[BPF_REG_2 ]; + } } /* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size); } +static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,kernel/bpf/verifier.c static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, · line 10801
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ struct bpf_reg_state *regs, u32 size) +{ +if (size >8 ) { +mark_btf_func_reg_size (env,BPF_REG_0 ,8 ); +mark_reg_unknown (env, regs,BPF_REG_2 ); + regs[BPF_REG_2 ].subreg_def =DEF_NOT_SUBREG ; + }else { +mark_btf_func_reg_size (env,BPF_REG_0 , size); + } +}This function makes R2 handling very asymmetric compared to R0:
-mark_reg_unknown()for r0 is done incheck_kfunc_call()
-DEF_NOT_SUBREGfor r0 is done inmark_btf_func_reg_size()
Quoted 2 lines
I think the code should be structured to keep R0 and R2 processing uniform.
Ack
Quoted 15 lines
Quoted 15 lines
kernel/bpf/verifier.c +6
-1+ static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta) { return meta->kfunc_flags & KF_ACQUIRE;@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, ®s[BPF_REG_0]); -mark_btf_func_reg_size (env,BPF_REG_0 , t->size); +mark_kfunc_ret_reg_size (env, regs, t->size); + }else if (btf_type_is_struct (t)) { +/* +* The returned struct comes back as raw register bits modeled +* as an unknown scalar, so it must contain only scalars:
Quoted 8 lines
kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 13277
Quoted 7 lines
kernel/bpf/verifier.c +7
-0+* otherwise a pointer field would be laundered into a scalar +* and escape provenance and reference tracking. +*/ +if (!__btf_type_is_scalar_struct (env, desc_btf, t,0 )) { +verbose (env,"kernel function %s returns %s %s that is not composed of scalars\n" , + func_name,btf_type_str (t), +btf_name_by_offset (desc_btf, t->name_off));Should this also check the size of the struct?
Later on, there is a size condition for struct (<= 16), so we should be okay here.
Quoted 9 lines
Quoted 8 lines
kernel/bpf/verifier.c +4
-0+return -EINVAL ; + } +mark_reg_unknown (env, regs,BPF_REG_0 ); +mark_kfunc_ret_reg_size (env, regs, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);[...]
Yonghong Song
reviewerPATCH 3/12 bpf: Wire up JIT support for 16-byte kfunc returns
In reply to Yonghong SongContinue preparing <=16 byte aggregate return support by wiring up the JIT side of the R0:R2 return convention for kfuncs.
A kfunc returning more than 8 bytes (a small aggregate or an __int128) hands the second half of the result back in RDX, the native x86-64 ABI's second return register. BPF instead expects a 16-byte return in the R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8.
Placing the second return half into R2 is possible on any JIT, but it needs architecture-specific JIT work. Rather than requiring every JIT to implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair() capability, defaulting to false in the generic core; an architecture opts in once its JIT handles the R0:R2 pair, and the remaining ones are left for future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT lacks the capability. Only x86, arm64 and riscv are supported so far.
On arm64 and riscv the native second return register is already BPF R2 (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI). This has been tested on x86 and arm64. The riscv path is expected to work by the same register-mapping reasoning as arm64 but has not been tested.
--- arch/arm64/net/bpf_jit_comp.c | 5 +++++ arch/riscv/net/bpf_jit_comp64.c | 5 +++++ arch/x86/net/bpf_jit_comp.c | 19 +++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ kernel/bpf/verifier.c | 5 +++++ 6 files changed, 40 insertions(+)
arch/arm64/net/bpf_jit_comp.c +5 -0
@@ -2312,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair (void ) +{ +return true; +} +bool bpf_jit_supports_stack_args (void ) {return true;
arch/riscv/net/bpf_jit_comp64.c +5 -0
@@ -2107,6 +2107,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair (void ) +{ +return true; +} +bool bpf_jit_supports_ptr_xchg (void ) {return true;
arch/x86/net/bpf_jit_comp.c +19 -0
@@ -2592,6 +2592,20 @@ st: insn_off = insn->off;return -EINVAL ;if (priv_frame_ptr)pop_r9 (&prog); +if (src_reg ==BPF_PSEUDO_KFUNC_CALL ) { +const struct btf_func_model *fm; + +/* +* A kfunc returning a >8 byte aggregate hands the +* second half back in RDX (the native ABI's second +* return reg), but BPF expects it in R0:R2. BPF R0 +* is RAX (no move needed), while BPF R2 is RSI, so +* copy RDX into RSI. +*/ + fm =bpf_jit_find_kfunc_model (bpf_prog, insn); +if (fm && fm->ret_size >8 ) +emit_mov_reg (&prog, true,BPF_REG_2 ,BPF_REG_3 ); + }break ; }
@@ -4045,6 +4059,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair (void ) +{ +return true; +} +bool bpf_jit_supports_stack_args (void ) {return true;
include/linux/filter.h +1 -0
@@ -1182,6 +1182,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void ); +bool bpf_jit_supports_kfunc_ret_reg_pair (void );bool bpf_jit_supports_stack_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );
kernel/bpf/core.c +5 -0
@@ -3294,6 +3294,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)return false; } +bool __weak bpf_jit_supports_kfunc_ret_reg_pair (void ) +{ +return false; +} +bool __weak bpf_jit_supports_stack_args (void ) {return false;
kernel/bpf/verifier.c +5 -0
@@ -2851,6 +2851,11 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) err =btf_distill_func_proto (&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);if (err)return err; +if (func_model.ret_size >8 && !bpf_jit_supports_kfunc_ret_reg_pair ()) { +verbose (env,"kfunc %s with >8-byte return is not supported by JIT\n" , + kfunc.name); +return -EOPNOTSUPP ; + } desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id;
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 03/12] bpf: Wire up JIT support for 16-byte kfunc returns
Quoted 20 lines · Yonghong Song
Continue preparing <=16 byte aggregate return support by wiring up the JIT side of the R0:R2 return convention for kfuncs.
A kfunc returning more than 8 bytes (a small aggregate or an
__int128) hands the second half of the result back in RDX, the native x86-64 ABI's second return register. BPF instead expects a 16-byte return in the R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2 maps to RSI, so emit aRDX->RSImove after aBPF_PSEUDO_KFUNC_CALLwhose function model reportsret_size> 8.Placing the second return half into R2 is possible on any JIT, but it needs architecture-specific JIT work. Rather than requiring every JIT to implement it at once, add a
bpf_jit_supports_kfunc_ret_reg_pair()capability, defaulting to false in the generic core; an architecture opts in once its JIT handles the R0:R2 pair, and the remaining ones are left for future work. The verifier enforces it inbpf_add_kfunc_call(), rejecting a kfunc whose return is larger than 8 bytes with-EOPNOTSUPPwhen the JIT lacks the capability. Only x86, arm64 and riscv are supported so far.
Quoted 8 lines
On arm64 and riscv the native second return register is already BPF R2 (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2 register pair on return with no extra move, unlike x86 (
RDX->RSI). This has been tested on x86 and arm64. The riscv path is expected to work by the same register-mapping reasoning as arm64 but has not been tested.Signed-off-by Yonghong Song <[email protected]>---
[...]
Yonghong Song
reviewerPATCH 4/12 bpf: Track R2 of register-pair returns in precision backtracking
In reply to Yonghong SongA function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT.
Handle R2 like R0 in the three boundaries where a call defines the return registers:
- static subprog exit (BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0.
--- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/backtrack.c | 32 +++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 13 ++++++++++++- 3 files changed, 45 insertions(+), 2 deletions(-)
include/linux/bpf_verifier.h +2 -0
@@ -1431,6 +1431,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);struct bpf_iarray *bpf_insn_successors (struct bpf_verifier_env *env,u32 idx);void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask);bool bpf_subprog_is_global (const struct bpf_verifier_env *env,int subprog); +bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog); +int bpf_get_kfunc_ret_size (const struct bpf_prog *prog,u32 func_id,u16 offset);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn);
kernel/bpf/backtrack.c +31 -1
@@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,*/ verifier_bug_if (idx +1 != subseq_idx, env,"extra insn from subprog" ); +if (bpf_ret_reg_pair (env, subprog)) +bt_clear_reg (bt,BPF_REG_2 );/* r1-r5 are invalidated after subprog call, * so for global func call it shouldn't be set * anymore
@@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,return -ENOTSUPP ;/* regular helper call sets R0 */ bt_clear_reg (bt,BPF_REG_0 ); +/* On error the return is negative, R2 stays set, and the +* check below catches it as an unexpected register. +*/ +if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +bpf_get_kfunc_ret_size (env->prog, insn->imm, insn->off) >8 ) +bt_clear_reg (bt,BPF_REG_2 );if (bt_reg_mask (bt) &BPF_REGMASK_ARGS ) {/* if backtracking was looking for registers R1-R5 * they should have been found already.
@@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,return -EFAULT ; } }else if (opcode ==BPF_EXIT ) { -bool r0_precise; +bool r0_precise, r2_precise = false; + +/* A subprog returning a value larger than 8 bytes (a +* struct/union or an __int128) uses R2 as a second +* return register alongside R0. When exiting such a +* subprog, R2 is a return register rather than a +* leftover argument, so handle it like R0 below. +*/ +if (subseq_idx -1 >=0 && +bpf_pseudo_call (&env->prog->insnsi[subseq_idx -1 ])) { +int call_idx = subseq_idx -1 ; +int tgt = call_idx + env->prog->insnsi[call_idx].imm +1 ; +int subprog =bpf_find_subprog (env, tgt); + +if (subprog >=0 && +bpf_ret_reg_pair (env, subprog) && +bt_is_reg_set (bt,BPF_REG_2 )) + r2_precise = true; + }/* Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -534,6 +560,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,if (subseq_idx >=0 &&bpf_calls_callback (env, subseq_idx))for (i =BPF_REG_1 ; i <=BPF_REG_5 ; i++)bt_clear_reg (bt, i); +if (r2_precise) +bt_clear_reg (bt,BPF_REG_2 );if (bt_reg_mask (bt) &BPF_REGMASK_ARGS ) {verifier_bug (env,"backtracking exit unexpected regs %x" ,bt_reg_mask (bt));
@@ -558,6 +586,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,if (r0_precise)bt_set_reg (bt,BPF_REG_0 ); +if (r2_precise) +bt_set_reg (bt,BPF_REG_2 );/* r6-r9 and stack slots will stay set in caller frame * bitmasks until we return back from callee(s) */
kernel/bpf/verifier.c +12 -1
@@ -424,7 +424,7 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)return btf_type_is_void (type); } -static bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) +bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog) {const struct btf_type *type, *func, *func_proto;const struct btf *btf = env->prog->aux->btf;
@@ -2521,6 +2521,17 @@ int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,return 0 ; } +int bpf_get_kfunc_ret_size (const struct bpf_prog *prog,u32 func_id,u16 offset) +{ +const struct bpf_kfunc_desc *desc; + + desc =find_kfunc_desc (prog, func_id, offset); +if (!desc) +return -EFAULT ; + +return desc->func_model.ret_size; +} +#define BPF_FD_SLOT_BTF 1ULstatic void fd_slot_set_map (struct bpf_fd_array *slot,struct bpf_map *map)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking
This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where backtrack_insn() was not updated to treat R2 as a return register. Should this carry a…
Quoted 18 lines
A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks inbacktrack_insn()and reject a valid program with-EFAULT.Handle R2 like R0 in the three boundaries where a call defines the return registers:
- static subprog exit (
BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (modelret_size> 8) also sets R2; clear it like R0.
Quoted 1 line
Signed-off-by Yonghong Song <[email protected]>
This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where backtrack_insn() was not updated to treat R2 as a return register. Should this carry a Fixes: tag?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Yonghong Song
reviewerRe: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking
Quoted 18 lines · On 7/8/26 2:10 PM, [email protected] wrote:
Quoted 18 lines
A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks inbacktrack_insn()and reject a valid program with-EFAULT.Handle R2 like R0 in the three boundaries where a call defines the return registers:
- static subprog exit (
BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (modelret_size> 8) also sets R2; clear it like R0.
Quoted 6 lines
Quoted 1 line
Signed-off-by Yonghong Song <[email protected]>This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where
backtrack_insn()was not updated to treat R2 as a return register. Should this carry a Fixes: tag?Fixes ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")
This is still a preparation patch, so Fixes is not necessary. The same for a few later patches.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking
In reply to Yonghong SongQuoted 18 lines · Yonghong Song
A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks inbacktrack_insn()and reject a valid program with-EFAULT.Handle R2 like R0 in the three boundaries where a call defines the return registers:
- static subprog exit (
BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (modelret_size> 8) also sets R2; clear it like R0.
Quoted 2 lines
Signed-off-by Yonghong Song <[email protected]>---
kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 428
Quoted 10 lines
kernel/bpf/backtrack.c +2
-0@@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,*/ verifier_bug_if(idx + 1 != subseq_idx, env, "extra insn from subprog"); +if (bpf_ret_reg_pair (env, subprog)) +bt_clear_reg (bt,BPF_REG_2 );
Nit: please move this a few lines below, so that r0 and r2 are handled nearby.
kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 514
Quoted 10 lines
kernel/bpf/backtrack.c +3
-0/* r1-r5 are invalidated after subprog call,* so for global func call it shouldn't be set * anymore @@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, return -ENOTSUPP; /* regular helper call sets R0 */ bt_clear_reg(bt, BPF_REG_0); +/* On error the return is negative, R2 stays set, and the +* check below catches it as an unexpected register. +*/
Nit: I don't understand the above comment (the code is fine). Maybe drop it altogether?
Quoted 9 lines
kernel/bpf/backtrack.c +3
-0+if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && + bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off) > 8) +bt_clear_reg (bt,BPF_REG_2 ); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { /* if backtracking was looking for registers R1-R5* they should have been found already. @@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, return -EFAULT; }
kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 539
Quoted 10 lines
kernel/bpf/backtrack.c +8
-1} else if (opcode == BPF_EXIT) { -bool r0_precise; +bool r0_precise, r2_precise = false; + +/* A subprog returning a value larger than 8 bytes (a +* struct/union or an __int128) uses R2 as a second +* return register alongside R0. When exiting such a +* subprog, R2 is a return register rather than a +* leftover argument, so handle it like R0 below. +*/
Nit: the comment is too verbose.
Quoted 14 lines
kernel/bpf/backtrack.c +11
-0+if (subseq_idx -1 >=0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1])) { +int call_idx = subseq_idx -1 ; +int tgt = call_idx + env->prog->insnsi[call_idx].imm +1 ; +int subprog =bpf_find_subprog (env, tgt); + +if (subprog >=0 && + bpf_ret_reg_pair(env, subprog) && + bt_is_reg_set(bt, BPF_REG_2)) + r2_precise = true; + }
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
[...]
Yonghong Song
reviewerPATCH 5/12 bpf: Account R2 of register-pair returns in live register analysis
In reply to Yonghong SongA BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.
Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.
For example, for a subprogram returning an __int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):
Live regs before insn:
0: .12345.... (85) call pc+2
1: ..2....... (bf) r0 = r2
2: 0......... (95) exit
3: .......... (b7) r0 = 1
4: 0......... (b7) r2 = 2
5: 0.2....... (95) exit--- kernel/bpf/liveness.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/liveness.c +10 -3
@@ -2062,7 +2062,8 @@ struct insn_live_regs {/* Compute info->{use,def} fields for the instruction */ static void compute_insn_live_regs (struct bpf_verifier_env *env,struct bpf_insn *insn, -struct insn_live_regs *info) +struct insn_live_regs *info, +int subprog) {struct bpf_call_summary cs;u8 class =BPF_CLASS (insn->code);
@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,case BPF_EXIT : def =0 ; use = r0; +if (bpf_ret_reg_pair (env, subprog)) + use |=BIT (BPF_REG_2 );break ;case BPF_CALL : def =ALL_CALLER_SAVED_REGS ;
@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) goto out; } -for (i =0 ; i < insn_cnt; ++i) -compute_insn_live_regs (env, &insns[i], &state[i]); +for (i =0 , j =0 ; i < insn_cnt; ++i) { +/* Advance to the subprog that contains instruction i. */ +while (j +1 < env->subprog_cnt && env->subprog_info[j +1 ].start <= i) + j++; +compute_insn_live_regs (env, &insns[i], &state[i], j); + }/* Forward pass: resolve stack access through FP-derived pointers */ err =bpf_compute_subprog_arg_access (env);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis
This looks like a bug fix for 16-byte return type support. Should this include:
Quoted 28 lines
commit
29d6a87c34e27de90ba01d71870930baf27d4b33Author: Yonghong Song <[email protected]> Date: Wed Jul 8 13:10:05 2026 -0700bpf: Account R2 of register-pair returns in live register analysis
A
BPF_EXITof a subprogram returning a value larger than 8 bytes (a struct/union or an__int128) reads R2 as well as R0, since the second half of the return value is passed back in R2.compute_insn_live_regs()only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.Mark R2 used at a
BPF_EXITwhen the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.For example, for a subprogram returning an
__int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):Live regs before insn:
0: .12345.... (85) call pc+2
1: ..2....... (bf) r0 = r2
2: 0......... (95) exit
3: .......... (b7) r0 = 1
4: 0......... (b7) r2 = 2
5: 0.2....... (95) exit
Quoted 1 line
Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for 16-byte return type support. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Eduard Zingerman
maintainerRe: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis
In reply to Yonghong SongQuoted 54 lines · Yonghong Song
A
BPF_EXITof a subprogram returning a value larger than 8 bytes (a struct/union or an__int128) reads R2 as well as R0, since the second half of the return value is passed back in R2.compute_insn_live_regs()only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.Mark R2 used at a
BPF_EXITwhen the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.For example, for a subprogram returning an
__int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):Live regs before insn:
0: .12345.... (85) call pc+2
1: ..2....... (bf) r0 = r2
2: 0......... (95) exit
3: .......... (b7) r0 = 1
4: 0......... (b7) r2 = 2
5: 0.2....... (95) exitSigned-off-by Yonghong Song <[email protected]>---
kernel/bpf/liveness.c | 13 ++++++++++---1 file changed, 10 insertions(+), 3 deletions(-)kernel/bpf/liveness.c +2
-1@@ -2062,7 +2062,8 @@ struct insn_live_regs { /* Compute info->{use,def} fields for the instruction */ static void compute_insn_live_regs(struct bpf_verifier_env *env, struct bpf_insn *insn, - struct insn_live_regs *info) + struct insn_live_regs *info, + int subprog) { struct bpf_call_summary cs; u8 class = BPF_CLASS(insn->code);@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,case
BPF_EXIT:
def = 0;
use = r0;kernel/bpf/liveness.c +2
-0+if (bpf_ret_reg_pair (env, subprog)) + use |=BIT (BPF_REG_2 );break;
caseBPF_CALL:
def =ALL_CALLER_SAVED_REGS;kernel/bpf/liveness.c +0
-0@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) goto out; }
kernel/bpf/liveness.c int bpf_compute_live_registers(struct bpf_verifier_env *env) · line 2246
Quoted 6 lines
kernel/bpf/liveness.c +4
-2-for (i =0 ; i < insn_cnt; ++i) -compute_insn_live_regs (env, &insns[i], &state[i]); +for (i =0 , j =0 ; i < insn_cnt; ++i) { +/* Advance to the subprog that contains instruction i. */ +while (j +1 < env->subprog_cnt && env->subprog_info[j +1 ].start <= i) + j++;
This loop is very in-elegant. The outer loop can be changed to iterate subprograms, with inner loop iterating instructions. Or just use bpf_find_subprog() when processing EXIT.
Quoted 5 lines
kernel/bpf/liveness.c +2
-0+compute_insn_live_regs (env, &insns[i], &state[i], j); + }
/* Forward pass: resolve stack access through FP-derived pointers */ err =bpf_compute_subprog_arg_access(env);
Yonghong Song
reviewerRe: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis
Quoted 55 lines · Eduard Zingerman
Quoted 54 lines · Yonghong Song
A
BPF_EXITof a subprogram returning a value larger than 8 bytes (a struct/union or an__int128) reads R2 as well as R0, since the second half of the return value is passed back in R2.compute_insn_live_regs()only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.Mark R2 used at a
BPF_EXITwhen the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.For example, for a subprogram returning an
__int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):Live regs before insn:
0: .12345.... (85) call pc+2
1: ..2....... (bf) r0 = r2
2: 0......... (95) exit
3: .......... (b7) r0 = 1
4: 0......... (b7) r2 = 2
5: 0.2....... (95) exitSigned-off-by Yonghong Song <[email protected]>--- kernel/bpf/liveness.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)kernel/bpf/liveness.c +4
-1@@ -2062,7 +2062,8 @@ struct insn_live_regs { /* Compute info->{use,def} fields for the instruction */ static void compute_insn_live_regs(struct bpf_verifier_env *env, struct bpf_insn *insn, - struct insn_live_regs *info) + struct insn_live_regs *info, + int subprog) { struct bpf_call_summary cs; u8 class = BPF_CLASS(insn->code);@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, case BPF_EXIT: def = 0; use = r0; +if (bpf_ret_reg_pair (env, subprog)) + use |=BIT (BPF_REG_2 ); break; case BPF_CALL: def = ALL_CALLER_SAVED_REGS;@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) goto out; }
Quoted 9 lines
kernel/bpf/liveness.c int bpf_compute_live_registers(struct bpf_verifier_env *env) · line 2246
Quoted 6 lines
kernel/bpf/liveness.c +4
-2-for (i =0 ; i < insn_cnt; ++i) -compute_insn_live_regs (env, &insns[i], &state[i]); +for (i =0 , j =0 ; i < insn_cnt; ++i) { +/* Advance to the subprog that contains instruction i. */ +while (j +1 < env->subprog_cnt && env->subprog_info[j +1 ].start <= i) + j++;This loop is very in-elegant. The outer loop can be changed to iterate subprograms, with inner loop iterating instructions. Or just use
bpf_find_subprog()when processing EXIT.
Ack. I prefer to have iterating subprogram in outer loop and iterating insns in the inner loop.
Quoted 6 lines
Quoted 6 lines
kernel/bpf/liveness.c +2
-0+compute_insn_live_regs (env, &insns[i], &state[i], j); + } /* Forward pass: resolve stack access through FP-derived pointers */ err = bpf_compute_subprog_arg_access(env);
Yonghong Song
reviewerPATCH 6/12 bpf: Force JIT for programs using the R0:R2 register pair
In reply to Yonghong SongThe R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is set.
--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 2 +- kernel/bpf/verifier.c | 13 +++++++++++-- 3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/bpf.h +1 -0
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {bool changes_pkt_data;bool might_sleep;bool kprobe_write_ctx; +bool ret_reg_pair;struct {s32 keyring_serial;u8 keyring_type;
kernel/bpf/core.c +1 -1
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || fp->aux->ret_reg_pair) jit_needed = true;if (!bpf_prog_select_interpreter (fp))
kernel/bpf/verifier.c +11 -2
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)return false; } +static bool bpf_ret_reg_pair_mark (struct bpf_verifier_env *env,int subprog) +{ +if (!bpf_ret_reg_pair (env, subprog)) +return false; + + env->prog->aux->ret_reg_pair = true; +return true; +} +static const char *subprog_name (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info *info;
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0; -if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (bpf_ret_reg_pair_mark (env, callee->subprogno)) {if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) {verbose (env,"cannot return stack pointer to the caller\n" );return -EINVAL ;
@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)if (err)return err; -if (!bpf_ret_reg_pair (env, subprog)) +if (!bpf_ret_reg_pair_mark (env, subprog))return 0 ;return check_global_ret_scalar_reg (env,BPF_REG_2 );
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
This looks like a bug fix for the R0:R2 return convention, where the verifier accepted such programs without gating them to JIT-only. The bpf_ret_reg_pair() support that introduced this was added…
Quoted 9 lines
bpf: Force JIT for programs using the R0:R2 register pair
The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when
bpf_prog_aux->ret_reg_pairis set.Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for the R0:R2 return convention, where the verifier accepted such programs without gating them to JIT-only. The bpf_ret_reg_pair() support that introduced this was added earlier in the series. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
In reply to Yonghong SongQuoted 43 lines · Yonghong Song
The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when
bpf_prog_aux->ret_reg_pairis set.Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 2 +- kernel/bpf/verifier.c | 13 +++++++++++-- 3 files changed, 13 insertions(+), 3 deletions(-)include/linux/bpf.h +1
-0@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {bool changes_pkt_data;bool might_sleep;bool kprobe_write_ctx; +bool ret_reg_pair;struct {s32 keyring_serial;u8 keyring_type;kernel/bpf/core.c +1
-1@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || fp->aux->ret_reg_pair) jit_needed = true;if (!bpf_prog_select_interpreter (fp))kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)return false; } +static bool bpf_ret_reg_pair_mark (struct bpf_verifier_env *env,int subprog) +{ +if (!bpf_ret_reg_pair (env, subprog)) +return false; + + env->prog->aux->ret_reg_pair = true;
With 'jit_required' introduction [1], it has no need to introduce 'ret_reg_pair'. 'env->prog->jit_required = true;' is enough.
[1]
https://lore.kernel.org/bpf/[email protected]/
Thanks,
Leon
Quoted 24 lines
kernel/bpf/verifier.c +5
-2+return true; +} +static const char *subprog_name (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info *info;@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0; -if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (bpf_ret_reg_pair_mark (env, callee->subprogno)) {if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) {verbose (env,"cannot return stack pointer to the caller\n" );return -EINVAL ;@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)if (err)return err; -if (!bpf_ret_reg_pair (env, subprog)) +if (!bpf_ret_reg_pair_mark (env, subprog))return 0 ;return check_global_ret_scalar_reg (env,BPF_REG_2 );
Yonghong Song
reviewerRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
Quoted 57 lines · Leon Hwang
Quoted 43 lines · Yonghong Song
The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when
bpf_prog_aux->ret_reg_pairis set.Signed-off-by Yonghong Song <[email protected]>--- include/linux/bpf.h | 1 + kernel/bpf/core.c | 2 +- kernel/bpf/verifier.c | 13 +++++++++++-- 3 files changed, 13 insertions(+), 3 deletions(-)include/linux/bpf.h +1
-0@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {bool changes_pkt_data;bool might_sleep;bool kprobe_write_ctx; +bool ret_reg_pair;struct {s32 keyring_serial;u8 keyring_type;kernel/bpf/core.c +1
-1@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;if (IS_ENABLED (CONFIG_BPF_JIT_ALWAYS_ON ) || -bpf_prog_has_kfunc_call (fp)) +bpf_prog_has_kfunc_call (fp) || fp->aux->ret_reg_pair) jit_needed = true;if (!bpf_prog_select_interpreter (fp))kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)return false; } +static bool bpf_ret_reg_pair_mark (struct bpf_verifier_env *env,int subprog) +{ +if (!bpf_ret_reg_pair (env, subprog)) +return false; + + env->prog->aux->ret_reg_pair = true;With '
jit_required' introduction [1], it has no need to introduce 'ret_reg_pair'. 'env->prog->jit_required= true;' is enough.
Quoted 2 lines
Thanks! I will use jit_required in the next revision. I guess this will be useful for many other cases who do not want to support interpreter.
Quoted 27 lines
Thanks,
LeonQuoted 24 lines
kernel/bpf/verifier.c +5
-2+return true; +} +static const char *subprog_name (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info *info;@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0; -if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (bpf_ret_reg_pair_mark (env, callee->subprogno)) {if (callee->regs[BPF_REG_2 ].type ==PTR_TO_STACK ) {verbose (env,"cannot return stack pointer to the caller\n" );return -EINVAL ;@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)if (err)return err; -if (!bpf_ret_reg_pair (env, subprog)) +if (!bpf_ret_reg_pair_mark (env, subprog))return 0 ;return check_global_ret_scalar_reg (env,BPF_REG_2 );
Eduard Zingerman
maintainerRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
In reply to Yonghong SongOn Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
[...]
kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2679
Quoted 10 lines
kernel/bpf/core.c +0
-0@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;
if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON)||kernel/bpf/core.c +1
-1- bpf_prog_has_kfunc_call(fp)) + bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
This has a conflict with one of the very recent bpf-next commits. While reading locally I fixed the conflict by removing this hunk and adjusting bpf_ret_reg_pair_mark() as below.
Quoted 11 lines
jit_needed= true;
if (!bpf_prog_select_interpreter(fp))kernel/bpf/verifier.c +0
-0@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) return false; }
kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+static bool bpf_ret_reg_pair_mark (struct bpf_verifier_env *env,int subprog) +{ +if (!bpf_ret_reg_pair (env, subprog)) +return false; + + env->prog->aux->ret_reg_pair = true;
Added:
env->prog->jit_required = true;
kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 464
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+return true; +} +
You probably would want to do the same in v2.
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
Quoted 11 lines · Eduard Zingerman
On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
[...]
Quoted 7 lines
kernel/bpf/core.c +0
-0@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct goto finalize;
Quoted 6 lines
kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2679
Quoted 3 lines
kernel/bpf/core.c +1
-1if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) || - bpf_prog_has_kfunc_call(fp)) + bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)This has a conflict with one of the very recent bpf-next commits. While reading locally I fixed the conflict by removing this hunk and adjusting
bpf_ret_reg_pair_mark()as below.
Indeed, once jit_required is available in fp->aux, I can remove ret_reg_pair things.
Quoted 27 lines
Quoted 9 lines
kernel/bpf/core.c +0
-0jit_needed = true; if (!bpf_prog_select_interpreter(fp))kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) return false; } +static bool bpf_ret_reg_pair_mark (struct bpf_verifier_env *env,int subprog) +{ +if (!bpf_ret_reg_pair (env, subprog)) +return false; + + env->prog->aux->ret_reg_pair = true;Added:
env->prog->jit_required = true;kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 464
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+return true; +} +You probably would want to do the same in v2.
[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
In reply to Yonghong SongOn Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
[...]
kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9889
Quoted 6 lines
kernel/bpf/verifier.c +1
-1@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) } else { /* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0] = *r0; -if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (bpf_ret_reg_pair_mark (env, callee->subprogno)) {
Also, doing this while processing exit is a bit strange. I think this should be done in do_check_common().
Quoted 3 lines
kernel/bpf/verifier.c +0
-0if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) { verbose(env, "cannot return stack pointer to the caller\n"); return -EINVAL;
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair
Quoted 4 lines · Eduard Zingerman
On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
[...]
Quoted 8 lines
kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9889
Quoted 6 lines
kernel/bpf/verifier.c +1
-1@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) } else { /* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0] = *r0; -if (bpf_ret_reg_pair (env, callee->subprogno)) { +if (bpf_ret_reg_pair_mark (env, callee->subprogno)) {Also, doing this while processing exit is a bit strange. I think this should be done in
do_check_common().
Before sending this patch, I found two places to do this, prepare_func_exit() and do_check_common(). Since you mentioned, I will do do_check_common() then.
Quoted 5 lines
Quoted 4 lines
kernel/bpf/verifier.c +0
-0if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) { verbose(env, "cannot return stack pointer to the caller\n"); return -EINVAL;[...]
Yonghong Song
reviewerPATCH 7/12 bpf: Reject >8 byte return values on return-reading trampoline paths
In reply to Yonghong Songbtf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.
But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.
This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.
Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.
--- kernel/bpf/bpf_struct_ops.c | 12 ++++++++++++ kernel/bpf/verifier.c | 25 +++++++++++++++++++++++++ 2 files changed, 37 insertions(+)
kernel/bpf/bpf_struct_ops.c +12 -0
@@ -445,6 +445,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc, goto errout; } +/* +* A >8 byte return value is passed back in the R0:R2 register +* pair, which the struct_ops trampoline does not preserve (only +* 8 bytes of the return value are saved and restored). +*/ +if (st_ops->func_models[i].ret_size >8 ) { +pr_warn ("func ptr %s in struct %s has a >8 byte return value, which is not supported\n" , + mname, st_ops->name); + err = -EOPNOTSUPP ; + goto errout; + } + stub_func_addr = *(void **)(st_ops->cfi_stubs + moff); err = prepare_arg_info(btf, st_ops->name, mname, func_proto, stub_func_addr,
kernel/bpf/verifier.c +25 -0
@@ -19112,6 +19112,20 @@ static int btf_id_allow_sleepable(u32 btf_id, unsigned long addr, const struct breturn -EINVAL ; } +static bool attach_uses_trampoline_retval (enum bpf_attach_type type) +{ +switch (type) { +case BPF_MODIFY_RETURN : +case BPF_TRACE_FEXIT : +case BPF_TRACE_FEXIT_MULTI : +case BPF_TRACE_FSESSION : +case BPF_TRACE_FSESSION_MULTI : +return true; +default : +return false; + } +} + int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog,
@@ -19376,6 +19390,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,if (ret <0 )return ret; +if (tgt_info->fmodel.ret_size >8 && +attach_uses_trampoline_retval (prog->expected_attach_type)) { +bpf_log (log, +"Attach to function %s with a >8 byte return value is not supported for this attach type\n" , + tname); +return -EOPNOTSUPP ; + } +/* * *.multi programs don't need an address during program * verification, we just take the module ref if needed.
@@ -19652,6 +19674,9 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt err =btf_distill_func_proto (NULL , btf, t, tname, &tgt_info->fmodel);if (err <0 )return err; +if (tgt_info->fmodel.ret_size >8 && +attach_uses_trampoline_retval (prog->expected_attach_type)) +return -EOPNOTSUPP ;if (btf_is_module (btf)) {/* The bpf program already holds reference to module. */ if (WARN_ON_ONCE (!prog->aux->mod))
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
Quoted 4 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.
Do you have a plan to add >8 byte return value support for them?
Thanks,
Leon
Quoted 1 line
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
Quoted 7 lines · Leon Hwang
Quoted 4 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.Do you have a plan to add >8 byte return value support for them?
I tested with vmlinux BTF built from the latest bpf-next. I got (from vmlinux BTF):
total number of functions: 67500 return type size (> 8 && <= 16): 51 return type size (> 16): 10
So in the initial version, I didn't support >8 byte for trampoline related bpf programs.
But I can certainly do this, we just need to allocate another 8 byte for return value in trampoline.
Quoted 5 lines
Thanks,
LeonQuoted 1 line
[...]
Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
Quoted 20 lines · Yonghong Song
Quoted 9 lines · Leon Hwang
Quoted 6 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following
patch relaxes it to accept a >8 and <=16 byte return value, which for the
BPF target is passed back in the R0:R2 register pair.Do you have a plan to add >8 byte return value support for them?
I tested with vmlinux BTF built from the latest bpf-next. I got (from vmlinux BTF):
total number of functions: 67500
return type size (> 8 && <= 16): 51
return type size (> 16): 10
Quoted 7 lines
So in the initial version, I didn't support >8 byte for trampoline related bpf programs.
But I can certainly do this, we just need to allocate another 8 byte for return value
in trampoline.
The following support should be mentioned in the cover letter.
Thanks,
Leon
Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
In reply to Yonghong SongQuoted 19 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g.
current_time()orns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself.struct_opstrampolines have the same limitation.This only matters for the attach types that actually read the target's return value: fexit,
fmod_retand fsession (plus the_multivariants of fexit and fsession). fentry andfentry_multirun before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.
Quoted 8 lines
Reject a >8 byte return value for the return-reading attach types in
bpf_check_attach_target()andbpf_check_attach_btf_id_multi(), and forstruct_opsinbpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.Signed-off-by Yonghong Song <[email protected]>---
Maybe post this independently from the current patch-set? Seem to be a standalone fix.
Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
Quoted 26 lines · Eduard Zingerman
Quoted 25 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g.
current_time()orns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself.struct_opstrampolines have the same limitation.This only matters for the attach types that actually read the target's return value: fexit,
fmod_retand fsession (plus the_multivariants of fexit and fsession). fentry andfentry_multirun before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.Reject a >8 byte return value for the return-reading attach types in
bpf_check_attach_target()andbpf_check_attach_btf_id_multi(), and forstruct_opsinbpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Yonghong Song <[email protected]>---
Reviewed-by Eduard Zingerman <[email protected]>Maybe post this independently from the current patch-set? Seem to be a standalone fix.
Agreed.
A Fixes tag should be added.
Thanks,
Leon
Yonghong Song
reviewerRe: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
Quoted 31 lines · Leon Hwang
Quoted 30 lines · Eduard Zingerman
Quoted 19 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g.
current_time()orns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself.struct_opstrampolines have the same limitation.This only matters for the attach types that actually read the target's return value: fexit,
fmod_retand fsession (plus the_multivariants of fexit and fsession). fentry andfentry_multirun before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.Quoted 8 lines
Reject a >8 byte return value for the return-reading attach types in
bpf_check_attach_target()andbpf_check_attach_btf_id_multi(), and forstruct_opsinbpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.Signed-off-by Yonghong Song <[email protected]>---
Reviewed-by Eduard Zingerman <[email protected]>
Quoted 7 lines
Quoted 3 lines
Maybe post this independently from the current patch-set? Seem to be a standalone fix.
Agreed.
A Fixes tag should be added.
Ack.
Quoted 4 lines
Thanks,
Leon
Yonghong Song
reviewerRe: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths
In reply to Eduard ZingermanQuoted 26 lines · Eduard Zingerman
Quoted 25 lines · Yonghong Song
btf_distill_func_proto()builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines andstruct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g.
current_time()orns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself.struct_opstrampolines have the same limitation.This only matters for the attach types that actually read the target's return value: fexit,
fmod_retand fsession (plus the_multivariants of fexit and fsession). fentry andfentry_multirun before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.Reject a >8 byte return value for the return-reading attach types in
bpf_check_attach_target()andbpf_check_attach_btf_id_multi(), and forstruct_opsinbpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Yonghong Song <[email protected]>---
Reviewed-by Eduard Zingerman <[email protected]>Maybe post this independently from the current patch-set? Seem to be a standalone fix.
Okay, I will submit this patch separately.
Yonghong Song
reviewerPATCH 8/12 bpf: Enable 16-byte aggregate return types
In reply to Yonghong SongRelax btf_distill_func_proto() to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:
- a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.
A >8 byte scalar (__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that __get_type_size() cannot return in registers at all (e.g. an array), which it already reports as ret < 0.
btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret) and struct_ops function models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.
btf_validate_return_type() is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.
With btf_distill_func_proto() and btf_validate_return_type() relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.
--- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/btf.c | 14 +++++++++++--- kernel/bpf/verifier.c | 6 +++--- .../testing/selftests/bpf/progs/exceptions_fail.c | 2 +- 4 files changed, 17 insertions(+), 7 deletions(-)
include/linux/bpf_verifier.h +2 -0
@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);bool bpf_subprog_is_global (const struct bpf_verifier_env *env,int subprog);bool bpf_ret_reg_pair (struct bpf_verifier_env *env,int subprog);int bpf_get_kfunc_ret_size (const struct bpf_prog *prog,u32 func_id,u16 offset); +bool __btf_type_is_scalar_struct (struct bpf_verifier_env *env,const struct btf *btf, +const struct btf_type *t,int rec);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn);
kernel/bpf/btf.c +11 -3
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } ret =__get_type_size (btf, func->type, &t); -if (ret <0 ||btf_type_is_struct (t)) { +/* +* __get_type_size() already restricts a non-negative ret to void, a +* pointer, an int, an enum or a struct/union, so only the size is checked +* here. +*/ +if (ret <0 || ret >16 ) {bpf_log (log,"The function %s return type %s is unsupported.\n" , tname,btf_type_str (t));
@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btif (btf_type_is_void (t) ||btf_type_is_int (t) ||btf_is_any_enum (t))return 0 ; +if (btf_type_is_struct (t) && t->size <=16 &&__btf_type_is_scalar_struct (env, btf, t,0 )) +return 0 ; +return -EOPNOTSUPP ; }
@@ -8059,8 +8067,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)if (err) {if (is_global) {bpf_log (log, -"Global function %s() return value not void or scalar. " -"Only those are supported.\n" , +"Global function %s() has unsupported return type. " +"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n" , tname); }return err;
kernel/bpf/verifier.c +3 -3
@@ -11082,9 +11082,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_kfunc_call_arg_meta *meta, u3 }/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */ -static bool __btf_type_is_scalar_struct (struct bpf_verifier_env *env, -const struct btf *btf, -const struct btf_type *t,int rec) +bool __btf_type_is_scalar_struct (struct bpf_verifier_env *env, +const struct btf *btf, +const struct btf_type *t,int rec) {const struct btf_type *member_type;const struct btf_member *member;
tools/testing/selftests/bpf/progs/exceptions_fail.c +1 -1
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a) SEC("?tc") __exception_cb(exception_cb_bad_ret_type1) -__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.") +__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")int reject_exception_cb_type_1 (struct __sk_buff *ctx) {bpf_throw (0 );
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. --
--- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types --- commit 5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
Author: Yonghong Song <[email protected]>
Subject: bpf: Enable 16-byte aggregate return types
This patch relaxes btf_distill_func_proto() and btf_validate_return_type() to allow kfuncs to return by-value structs or unions up to 16 bytes. This enables the <=16 byte aggregate return values end to end.
Quoted 5 lines
kernel/bpf/btf.c +0
-0
kernel/bpf/btf.c int btf_distill_func_proto(struct bpf_verifier_log *log, · line 7580
Quoted 10 lines
kernel/bpf/btf.c +6
-1@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } ret =__get_type_size (btf, func->type, &t); -if (ret <0 ||btf_type_is_struct (t)) { +/* +* __get_type_size() already restricts a non-negative ret to void, a +* pointer, an int, an enum or a struct/union, so only the size is checked +* here. +*/ +if (ret <0 || ret >16 ) {
[Severity: High]
Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?
The x86_64 ABI leaves the unused upper bytes of the %rdx register undefined when returning a struct that is not a multiple of 8 bytes.
Looking at mark_kfunc_ret_reg_size() in kernel/bpf/verifier.c, if the struct size is greater than 8:
if (size > 8) {
mark_btf_func_reg_size(env, BPF_REG_0, 8);
mark_reg_unknown(env, regs, BPF_REG_2);
regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
}The verifier explicitly marks BPF_REG_2 as fully initialized, bypassing subreg zero-extension.
And in the x86 JIT do_jit() in arch/x86/net/bpf_jit_comp.c, it moves the full 64-bit register:
fm = bpf_jit_find_kfunc_model(bpf_prog, insn); if (fm && fm->ret_size > 8) emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
Could a BPF program read the undefined bytes from R2, resulting in an information leak?
Quoted 3 lines
kernel/bpf/btf.c +0
-0bpf_log (log,"The function %s return type %s is unsupported.\n" , tname,btf_type_str (t));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Yonghong Song
reviewerRe: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types
Quoted 49 lines · On 7/8/26 1:28 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. ----- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types --- commit
5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
Author: Yonghong Song <[email protected]>
Subject: bpf: Enable 16-byte aggregate return typesThis patch relaxes
btf_distill_func_proto()andbtf_validate_return_type()to allow kfuncs to return by-value structs or unions up to 16 bytes. This enables the <=16 byte aggregate return values end to end.Quoted 5 lines
kernel/bpf/btf.c +0
-0kernel/bpf/btf.c int btf_distill_func_proto(struct bpf_verifier_log *log, · line 7580
Quoted 10 lines
kernel/bpf/btf.c +6
-1@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,return -EINVAL ; } ret =__get_type_size (btf, func->type, &t); -if (ret <0 ||btf_type_is_struct (t)) { +/* +* __get_type_size() already restricts a non-negative ret to void, a +* pointer, an int, an enum or a struct/union, so only the size is checked +* here. +*/ +if (ret <0 || ret >16 ) {[Severity: High]
Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?The
x86_64ABI leaves the unused upper bytes of the %rdx register undefined when returning a struct that is not a multiple of 8 bytes.Looking at
mark_kfunc_ret_reg_size()inkernel/bpf/verifier.c, if the struct size is greater than 8:if (size > 8) { mark_btf_func_reg_size(env, BPF_REG_0, 8); mark_reg_unknown(env, regs, BPF_REG_2); regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG; }The verifier explicitly marks
BPF_REG_2as fully initialized, bypassing subreg zero-extension.And in the x86 JIT
do_jit()inarch/x86/net/bpf_jit_comp.c, it moves the full 64-bit register:
Quoted 6 lines
fm = bpf_jit_find_kfunc_model(bpf_prog, insn); if (fm && fm->ret_size > 8) emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);Could a BPF program read the undefined bytes from R2, resulting in an information leak?
We should be fine here. See llvm patch https://reviews.llvm.org/D131598
which specifically tries to address this issue. The following is an example for R2:
$ cat t.c
struct foo { long a; short b; };
struct foo bar(void);int tar(void) {
struct foo t = bar();
return t.a + t.b;
}
$ clang --target=bpf -O2 -S t.c
$ cat t.s
.file "t.c"
.text
.globl tar # -- Begin function tar
.p2align 3
.type tar,@function
tar: # @tar
# %bb.0:
call bar
w2 <<= 16
w2 s>>= 16
w2 += w0
w0 = w2
exit
.Lfunc_end0:
.size tar, .Lfunc_end0-tar
# -- End function
.addrsigThe caller will do proper narrowing immediately after call return.
Quoted 4 lines
Quoted 4 lines
kernel/bpf/btf.c +0
-0bpf_log (log,"The function %s return type %s is unsupported.\n" , tname,btf_type_str (t));
Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types
In reply to Yonghong SongQuoted 20 lines · Yonghong Song
Relax
btf_distill_func_proto()to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:- a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.A >8 byte scalar (
__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that__get_type_size()cannot return in registers at all (e.g. an array), which it already reports as ret < 0.
btf_distill_func_proto()also builds the trampoline (fentry/fexit/fmod_ret) andstruct_opsfunction models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.
btf_validate_return_type()is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.
Quoted 7 lines
With
btf_distill_func_proto()andbtf_validate_return_type()relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.Signed-off-by Yonghong Song <[email protected]>---
Quoted 11 lines
include/linux/bpf_verifier.h | 2 ++ kernel/bpf/btf.c | 14 +++++++++++--- kernel/bpf/verifier.c | 6 +++--- .../testing/selftests/bpf/progs/exceptions_fail.c | 2 +-
4 files changed, 17 insertions(+), 7 deletions(-)include/linux/bpf_verifier.h +0
-0
include/linux/bpf_verifier.h void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); · line 1437
Quoted 5 lines
include/linux/bpf_verifier.h +2
-0@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog); bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog); int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset); +bool __btf_type_is_scalar_struct (struct bpf_verifier_env *env,const struct btf *btf, +const struct btf_type *t,int rec);
Nit: maybe remove the "__" prefix?
Quoted 25 lines
intbpf_find_subprog(struct bpf_verifier_env *env, int off); boolbpf_is_throw_kfunc(struct bpf_insn *insn);kernel/bpf/btf.c +6
-1@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); -if (ret <0 ||btf_type_is_struct (t)) { +/* +* __get_type_size() already restricts a non-negative ret to void, a +* pointer, an int, an enum or a struct/union, so only the size is checked +* here. +*/ +if (ret <0 || ret >16 ) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t));@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t)) return 0;
kernel/bpf/btf.c static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt · line 7976
Quoted 1 line
kernel/bpf/btf.c +1
-0+if (btf_type_is_struct (t) && t->size <=16 &&__btf_type_is_scalar_struct (env, btf, t,0 ))
Nit: if (__btf_type_is_scalar_struct(env, btf, t, 0) && t->size <= 16)
Quoted 5 lines
kernel/bpf/btf.c +2
-0+return 0 ; +return
-EOPNOTSUPP;
}
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types
Quoted 36 lines · Eduard Zingerman
Quoted 20 lines · Yonghong Song
Relax
btf_distill_func_proto()to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:- a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.A >8 byte scalar (
__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that__get_type_size()cannot return in registers at all (e.g. an array), which it already reports as ret < 0.
btf_distill_func_proto()also builds the trampoline (fentry/fexit/fmod_ret) andstruct_opsfunction models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.
btf_validate_return_type()is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.Quoted 7 lines
With
btf_distill_func_proto()andbtf_validate_return_type()relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.Signed-off-by Yonghong Song <[email protected]>---
Acked-by Eduard Zingerman <[email protected]>Quoted 6 lines
include/linux/bpf_verifier.h | 2 ++ kernel/bpf/btf.c | 14 +++++++++++--- kernel/bpf/verifier.c | 6 +++--- .../testing/selftests/bpf/progs/exceptions_fail.c | 2 +- 4 files changed, 17 insertions(+), 7 deletions(-)
Quoted 11 lines
include/linux/bpf_verifier.h void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); · line 1437
Quoted 10 lines
include/linux/bpf_verifier.h +2
-0@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog); bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog); int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset); +bool __btf_type_is_scalar_struct (struct bpf_verifier_env *env,const struct btf *btf, +const struct btf_type *t,int rec);Nit: maybe remove the "__" prefix?
Good point. Since it becomes a global function, we probably should remove "__" prefix.
Quoted 26 lines
Quoted 26 lines
include/linux/bpf_verifier.h +0
-0int bpf_find_subprog(struct bpf_verifier_env *env, int off); bool bpf_is_throw_kfunc(struct bpf_insn *insn);kernel/bpf/btf.c +6
-1@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); -if (ret <0 ||btf_type_is_struct (t)) { +/* +* __get_type_size() already restricts a non-negative ret to void, a +* pointer, an int, an enum or a struct/union, so only the size is checked +* here. +*/ +if (ret <0 || ret >16 ) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t));@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t)) return 0;
Quoted 2 lines
kernel/bpf/btf.c static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt · line 7976
Quoted 1 line
kernel/bpf/btf.c +1
-0+if (btf_type_is_struct (t) && t->size <=16 &&__btf_type_is_scalar_struct (env, btf, t,0 ))Nit: if (
__btf_type_is_scalar_struct(env, btf, t, 0)&&t->size<= 16)
Yes, this is simpler.
Quoted 7 lines
Quoted 6 lines
kernel/bpf/btf.c +2
-0+return 0 ; + return -EOPNOTSUPP; }[...]
Yonghong Song
reviewerPATCH 9/12 selftests/bpf: Add C tests for 16-byte returns in R0:R2
In reply to Yonghong SongAdd selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an __int128 return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.
The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on __clang_major__ >= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.
--- .../selftests/bpf/prog_tests/aggregate_ret.c | 96 ++++++++++++++++++ .../bpf/progs/aggregate_ret_int128_c.c | 48 +++++++++ .../bpf/progs/aggregate_ret_struct_c.c | 98 +++++++++++++++++++ .../bpf/progs/aggregate_ret_union_c.c | 51 ++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 14 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 11 +++ 6 files changed, 318 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +96 -0
@@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include <network_helpers.h> +#include "aggregate_ret_int128_c.skel.h" +#include "aggregate_ret_struct_c.skel.h" +#include "aggregate_ret_union_c.skel.h" + +/* +* Run one program and check its result. The program returns 0 on success, or +* 1..4 identifying which half of which aggregate (R0:R2) return value came back +* wrong. A return of -1 means the program was built with an LLVM that lacks +* R0:R2 16-byte return support, in which case the subtest is skipped. +*/ +static void run_prog (struct bpf_program *prog,bool may_skip) +{ +char buf[64 ] = {}; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = buf, + .data_size_in = sizeof(buf), + .repeat = 1, + ); +int err; + + err =bpf_prog_test_run_opts (bpf_program__fd (prog), &topts); +if (!ASSERT_OK (err,"test_run" )) +return ; + +if (may_skip && (int )topts.retval ==-1 ) { +test__skip (); +return ; + } + +ASSERT_EQ (topts.retval,0 ,"aggregate_ret_result" ); +} + +void test_aggregate_ret (void ) +{ +struct aggregate_ret_struct_c *skel_struct_c; +struct aggregate_ret_union_c *skel_union_c; +struct aggregate_ret_int128_c *skel_c; + + skel_c =aggregate_ret_int128_c__open_and_load (); +if (!ASSERT_OK_PTR (skel_c,"skel_c_open_load" )) +return ; + +/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */ +if (test__start_subtest ("int128_c" )) +run_prog (skel_c->progs.aggregate_ret_test, true); + +aggregate_ret_int128_c__destroy (skel_c); + + skel_struct_c =aggregate_ret_struct_c__open_and_load (); +if (!ASSERT_OK_PTR (skel_struct_c,"skel_struct_c_open_load" )) +return ; + +/* +* C variant returning a 16-byte struct by value: exercises the struct +* branch of bpf_ret_reg_pair(). Also LLVM 23 only. +*/ +if (test__start_subtest ("struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_struct_c_test, true); + +/* +* Same 16-byte struct return, but from a global (non-static) callee: +* exercises the compiler lowering a global struct-returning function to +* R0:R2 and the verifier's global return path end to end. Also LLVM 23 +* only. +*/ +if (test__start_subtest ("global_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_global_struct_c_test, true); + +/* +* Same 16-byte struct, but returned by value from a kfunc: exercises +* the struct branch of check_kfunc_call() and the JIT RDX->R2 move. +* Also LLVM 23 only. +*/ +if (test__start_subtest ("kfunc_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true); + +aggregate_ret_struct_c__destroy (skel_struct_c); + + skel_union_c =aggregate_ret_union_c__open_and_load (); +if (!ASSERT_OK_PTR (skel_union_c,"skel_union_c_open_load" )) +return ; + +/* +* C variant returning a 16-byte union by value: exercises the union BTF +* kind through the same struct branch of bpf_ret_reg_pair(). +* Also LLVM 23 only. +*/ +if (test__start_subtest ("union_c" )) +run_prog (skel_union_c->progs.aggregate_ret_union_c_test, true); + +aggregate_ret_union_c__destroy (skel_union_c); +}
tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c new file +48 -0
48 lines not on this page · raw message
tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c new file +98 -0
98 lines not on this page · raw message
tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c new file +51 -0
51 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14 -0
@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)return 0 ; } +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ +return ((__int128 )a <<64 ) | b; +} + +__bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b) +{ +struct bpf_kfunc_ret_ll r = { .a = a, .b = b }; + +return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,u64 e,u64 f,u64 g,u64 h,u64 i,u64 j)
@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)BTF_ID_FLAGS (func, bpf_kfunc_call_stack_arg_mix)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +11 -0
@@ -55,6 +55,15 @@ struct prog_test_big_arg {__u64 b; }; +/* +* A 16-byte struct returned by value from a kfunc, returned in the R0:R2 +* register pair. +*/ +struct bpf_kfunc_ret_ll {/* 16 bytes: R0:R2 */ +__u64 a; +__u64 b; +}; +struct prog_test_fail1 {void *p;int x;
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; +struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Unconditional use of `__int128` in `bpf_testmod` breaks compilation on 32-bit host architectures. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional use of __int128 in bpf_testmod breaks compilation on 32-bit host architectures. --
--- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 --- commit 5ed3844739f04c920bb1c37cdf474501b3ca9fef
Author: Yonghong Song <[email protected]>
selftests/bpf: Add C tests for 16-byte returns in R0:R2
This commit adds selftests to exercise 16-byte return values passed in the R0:R2 register pair. The tests include an __int128 return, a 16-byte struct return, and a 16-byte union return.
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0
-0
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) · line 831
Quoted 8 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4
-0@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)return 0 ; } +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ +return ((__int128 )a <<64 ) | b; +}
[Severity: Low]
Will this unconditional use of __int128 break the build on 32-bit host architectures?
The __int128 type is a compiler extension that is typically only available on 64-bit systems.
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c BTF_ID_FLAGS(func, bpf_kfunc_call_test2) · line 1373
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1
-0@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +BTF_ID_FLAGS (func, bpf_kfunc_call_test_i128)
[Severity: Low]
If bpf_kfunc_call_test_i128 needs to be conditionally excluded for 32-bit builds to fix the issue above, should this BTF ID registration also be conditionally excluded to prevent a missing symbol error during the build?
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0
-0
[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym; · line 130
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1
-0@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
[Severity: Low]
Does this declaration also need to be guarded for 64-bit architectures, so that including this header doesn't fail on 32-bit host compilers?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Yonghong Song
reviewerRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
Quoted 55 lines · On 7/8/26 1:24 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional use of__int128inbpf_testmodbreaks compilation on 32-bit host architectures. ----- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 --- commit
5ed3844739f04c920bb1c37cdf474501b3ca9fef
Author: Yonghong Song <[email protected]>selftests/bpf: Add C tests for 16-byte returns in R0:R2
This commit adds selftests to exercise 16-byte return values passed in the R0:R2 register pair. The tests include an
__int128return, a 16-byte struct return, and a 16-byte union return.[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0
-0[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) · line 831
Quoted 8 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4
-0@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)return 0 ; } +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ +return ((__int128 )a <<64 ) | b; +}[Severity: Low]
Will this unconditional use of__int128break the build on 32-bit host architectures?The
__int128type is a compiler extension that is typically only available on 64-bit systems.[ ... ]
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c BTF_ID_FLAGS(func, bpf_kfunc_call_test2) · line 1373
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1
-0@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +BTF_ID_FLAGS (func, bpf_kfunc_call_test_i128)[Severity: Low]
Ifbpf_kfunc_call_test_i128needs to be conditionally excluded for 32-bit builds to fix the issue above, should this BTF ID registration also be conditionally excluded to prevent a missing symbol error during the build?[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0
-0[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym; · line 130
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1
-0@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;struct sock *bpf_kfunc_call_test3 (struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;[Severity: Low]
Does this declaration also need to be guarded for 64-bit architectures, so that including this header doesn't fail on 32-bit host compilers?
We may need to guard with __SIZEOF_INT128__ in selftests. But current CI all supports 64bit, so I am not sure whether we should do it or not.
[email protected]
AI reviewRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
In reply to Yonghong Songthis isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the run_prog() comment above already states that programs built without R0:R2 support…
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +0
-0@@ -0,0 +1,96 @@
[ ... ]
Quoted 6 lines
+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */
[ ... ]
Quoted 23 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +23
-0+/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */ +if (test__start_subtest ("int128_c" )) +run_prog (skel_c->progs.aggregate_ret_test, true); + +aggregate_ret_int128_c__destroy (skel_c); + + skel_struct_c =aggregate_ret_struct_c__open_and_load (); +if (!ASSERT_OK_PTR (skel_struct_c,"skel_struct_c_open_load" )) +return ; + +/* +* C variant returning a 16-byte struct by value: exercises the struct +* branch of bpf_ret_reg_pair(). Also LLVM 23 only. +*/ +if (test__start_subtest ("struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_struct_c_test, true); + +/* +* Same 16-byte struct return, but from a global (non-static) callee: +* exercises the compiler lowering a global struct-returning function to +* R0:R2 and the verifier's global return path end to end. Also LLVM 23 +* only. +*/
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 79
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10
-0+if (test__start_subtest ("global_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_global_struct_c_test, true); + +/* +* Same 16-byte struct, but returned by value from a kfunc: exercises +* the struct branch of check_kfunc_call() and the JIT RDX->R2 move. +* Also LLVM 23 only. +*/ +if (test__start_subtest ("kfunc_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);
this isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the run_prog() comment above already states that programs built without R0:R2 support return -1 and get skipped. Could that note be stated once rather than on every block?
Related, do the per-subtest comments need to name the specific internals (bpf_ret_reg_pair(), check_kfunc_call(), "the JIT RDX->R2 move"), or would a shorter note serve here?
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 62
Quoted 13 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5
-0@@ -55,6 +55,15 @@ struct prog_test_big_arg {__u64 b; }; +/* +* A 16-byte struct returned by value from a kfunc, returned in the R0:R2 +* register pair. +*/ +struct bpf_kfunc_ret_ll {/* 16 bytes: R0:R2 */
this isn't a bug, but the block comment already says the struct is 16 bytes and returned in the R0:R2 register pair, and the trailing /* 16 bytes: R0:R2 */ on the next line repeats that. Could one of the two be dropped?
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 65
Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +3
-0+__u64 a; +__u64 b; +};
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Yonghong Song
reviewerRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
Quoted 45 lines · On 7/8/26 2:10 PM, [email protected] wrote:
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +0
-0@@ -0,0 +1,96 @@[ ... ]
Quoted 6 lines
+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */[ ... ]
Quoted 29 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +29
-0+/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */ +if (test__start_subtest ("int128_c" )) +run_prog (skel_c->progs.aggregate_ret_test, true); + +aggregate_ret_int128_c__destroy (skel_c); + + skel_struct_c =aggregate_ret_struct_c__open_and_load (); +if (!ASSERT_OK_PTR (skel_struct_c,"skel_struct_c_open_load" )) +return ; + +/* +* C variant returning a 16-byte struct by value: exercises the struct +* branch of bpf_ret_reg_pair(). Also LLVM 23 only. +*/ +if (test__start_subtest ("struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_struct_c_test, true); + +/* +* Same 16-byte struct return, but from a global (non-static) callee: +* exercises the compiler lowering a global struct-returning function to +* R0:R2 and the verifier's global return path end to end. Also LLVM 23 +* only. +*/ +if (test__start_subtest ("global_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_global_struct_c_test, true); + +/* +* Same 16-byte struct, but returned by value from a kfunc: exercises +* the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 79
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +4
-0+* Also LLVM 23 only. +*/ +if (test__start_subtest ("kfunc_struct_c" )) +run_prog (skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);this isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the
run_prog()comment above already states that programs built without R0:R2 support return -1 and get skipped. Could that note be stated once rather than on every block?
Ack. Will do.
Quoted 4 lines
Related, do the per-subtest comments need to name the specific internals (
bpf_ret_reg_pair(),check_kfunc_call(), "the JITRDX->R2move"), or would a shorter note serve here?
I guess I can rename these comments, function names themselves already have enough information.
Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0
-0@@ -55,6 +55,15 @@ struct prog_test_big_arg {__u64 b; };
Quoted 9 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 62
Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5
-0+/* +* A 16-byte struct returned by value from a kfunc, returned in the R0:R2 +* register pair. +*/ +struct bpf_kfunc_ret_ll {/* 16 bytes: R0:R2 */ this isn't a bug, but the block comment already says the struct is 16 bytes and returned in the R0:R2 register pair, and the trailing /* 16 bytes: R0:R2 */ on the next line repeats that. Could one of the two be dropped?
Yes, will drop.
Quoted 10 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 65
Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +3
-0+__u64 a; +__u64 b; +};---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219
Eduard Zingerman
maintainerRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
In reply to Yonghong SongQuoted 13 lines · Yonghong Song
Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an
__int128return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on
__clang_major__>= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.Signed-off-by Yonghong Song <[email protected]>---
[...]
Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +14
-0+static void run_prog (struct bpf_program *prog,bool may_skip) +{ +char buf[64 ] = {}; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = buf, + .data_size_in = sizeof(buf), + .repeat = 1, + ); +int err; + + err =bpf_prog_test_run_opts (bpf_program__fd (prog), &topts); +if (!ASSERT_OK (err,"test_run" )) +return ; +
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 38
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10
-0+if (may_skip && (int )topts.retval ==-1 ) { +test__skip (); +return ; + } + +ASSERT_EQ (topts.retval,0 ,"aggregate_ret_result" ); +} + +void test_aggregate_ret (void ) +{
The test_aggregate_ret()/run_prog() duplicate functionality provided by the __retval tag from test_loader.c. To achieve the same skipping mechanics I suggest to:
- extend test_loader.c / bpf_misc.h with a new tag: __skip("reason")
- use this tag as:
#ifdef __clang_major__ >= 23
SEC("tc")
int aggregate_ret_test(struct __sk_buff *skb) { ... }
#else
SEC("tc") __skip("... need clang23 ...")
int aggregate_ret_test(struct __sk_buff *skb) { ... }
#endif
[...]
...
Quoted 3 lines
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c +0
-0... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c +0
-0... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c +0
-0
Nit: all three files can be merged into one.
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2
Quoted 48 lines · Eduard Zingerman
Quoted 13 lines · Yonghong Song
Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an
__int128return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on
__clang_major__>= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.Signed-off-by Yonghong Song <[email protected]>---
[...]
Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +14
-0+static void run_prog (struct bpf_program *prog,bool may_skip) +{ +char buf[64 ] = {}; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = buf, + .data_size_in = sizeof(buf), + .repeat = 1, + ); +int err; + + err =bpf_prog_test_run_opts (bpf_program__fd (prog), &topts); +if (!ASSERT_OK (err,"test_run" )) +return ; +tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 38
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10
-0+if (may_skip && (int )topts.retval ==-1 ) { +test__skip (); +return ; + } + +ASSERT_EQ (topts.retval,0 ,"aggregate_ret_result" ); +} + +void test_aggregate_ret (void ) +{The test_aggregate_ret()/run_prog() duplicate functionality provided by the
__retvaltag fromtest_loader.c. To achieve the same skipping mechanics I suggest to:
- extendtest_loader.c/bpf_misc.hwith a new tag:__skip("reason")
- use this tag as:#ifdef __clang_major__ >= 23
Quoted 9 lines
SEC("tc")
int aggregate_ret_test(struct __sk_buff *skb) { ... }
#else
SEC("tc") __skip("... need clang23 ...")
int aggregate_ret_test(struct __sk_buff *skb) { ... }
#endifAck. We have quite some cases like this (guarded by compiler or feature flag, with __skip makes things simpler.
Quoted 7 lines
[...]
...
Quoted 3 lines
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c +0
-0... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c +0
-0... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c +0
-0Nit: all three files can be merged into one.
Yes, I can merge them.
Quoted 2 lines
[...]
Yonghong Song
reviewerPATCH 10/12 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
In reply to Yonghong SongAdd inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.
--- .../selftests/bpf/prog_tests/aggregate_ret.c | 25 ++ .../selftests/bpf/progs/aggregate_ret_func.c | 361 ++++++++++++++++++ .../selftests/bpf/progs/aggregate_ret_kfunc.c | 81 ++++ .../selftests/bpf/progs/aggregate_ret_run.c | 160 ++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 34 ++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 27 ++ 6 files changed, 688 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +25 -0
@@ -5,6 +5,9 @@#include "aggregate_ret_int128_c.skel.h" #include "aggregate_ret_struct_c.skel.h" #include "aggregate_ret_union_c.skel.h" +#include "aggregate_ret_run.skel.h" +#include "aggregate_ret_func.skel.h" +#include "aggregate_ret_kfunc.skel.h" /* * Run one program and check its result. The program returns 0 on success, or
@@ -39,6 +42,7 @@ void test_aggregate_ret(void)struct aggregate_ret_struct_c *skel_struct_c;struct aggregate_ret_union_c *skel_union_c;struct aggregate_ret_int128_c *skel_c; +struct aggregate_ret_run *skel_run; skel_c =aggregate_ret_int128_c__open_and_load ();if (!ASSERT_OK_PTR (skel_c,"skel_c_open_load" ))
@@ -93,4 +97,25 @@ void test_aggregate_ret(void)run_prog (skel_union_c->progs.aggregate_ret_union_c_test, true);aggregate_ret_union_c__destroy (skel_union_c); + + skel_run =aggregate_ret_run__open_and_load (); +if (!ASSERT_OK_PTR (skel_run,"skel_run_open_load" )) +return ; + +/* Inline-asm variant: compiler-version independent, always runs. */ +if (test__start_subtest ("asm" )) +run_prog (skel_run->progs.aggregate_ret_asm_test, false); + +/* Struct-by-value kfunc returns, also compiler-version independent. */ +if (test__start_subtest ("struct" )) +run_prog (skel_run->progs.aggregate_ret_struct_test, false); + +/* Union-by-value kfunc return, compiler-version independent. */ +if (test__start_subtest ("union" )) +run_prog (skel_run->progs.aggregate_ret_union_test, false); + +aggregate_ret_run__destroy (skel_run); + +RUN_TESTS (aggregate_ret_func); +RUN_TESTS (aggregate_ret_kfunc); }
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +361 -0
361 lines not on this page · raw message
tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file +81 -0
81 lines not on this page · raw message
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +160 -0
160 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +34 -0
@@ -837,6 +837,36 @@ __bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b)return r; } +__bpf_kfunc struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b) +{ +struct bpf_kfunc_ret_li r = { .a = a, .b = b }; + +return r; +} + +__bpf_kfunc struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) +{ +struct bpf_kfunc_ret_ii r = { .a = a, .b = b }; + +return r; +} + +__bpf_kfunc union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b) +{ +union bpf_kfunc_ret_uu r; + + r.halves[0 ] = a; + r.halves[1 ] = b; +return r; +} + +__bpf_kfunc struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag) +{ +struct bpf_kfunc_ret_ptr r = { .p =NULL , .tag = tag }; + +return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,u64 e,u64 f,u64 g,u64 h,u64 i,u64 j)
@@ -1372,6 +1402,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)BTF_ID_FLAGS (func, bpf_kfunc_call_stack_arg_mix)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +27 -0
@@ -64,6 +64,29 @@ struct bpf_kfunc_ret_ll { /* 16 bytes: R0:R2 */__u64 b; }; +struct bpf_kfunc_ret_li {/* 16 bytes: R0:R2 */ +__u64 a; +int b; +}; + +struct bpf_kfunc_ret_ii {/* 8 bytes: R0 only */ +int a; +int b; +}; + +union bpf_kfunc_ret_uu {/* 16 bytes: R0:R2 */ +__u64 halves[2 ]; +struct { +__u64 lo; +__u64 hi; + } parts; +}; + +struct bpf_kfunc_ret_ptr {/* 16 bytes: contains a pointer */ +void *p; +__u64 tag; +}; +struct prog_test_fail1 {void *p;int x;
@@ -129,6 +152,10 @@ long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym; +struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym; +struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym; +struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Hardcoded little-endian assumptions in struct and `__int128` register unpacking will cause…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoded little-endian assumptions in struct and __int128 register unpacking will cause selftest failures on big-endian architectures like s390x. --
--- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns --- commit dcf423ead16608630f40807ad0cde33d495c941d
Author: Yonghong Song <[email protected]>
selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0
-0
[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0
-0@@ -0,0 +1,160 @@
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +6
-0+SEC("tc") +int aggregate_ret_asm_test (struct __sk_buff *skb) +{ +__u64 a = skb->len; +__u64 b = skb->len ^0xdeadbeefULL ; +__u64 lo, hi;
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4
-0+ asmvolatile ( +"r1 = %[a];" +"r2 = %[b];" + "call %[kfunc];"
tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 73
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+"%[lo] = r0;" +"%[hi] = r2;" + : [lo]"=r"(lo), [hi]"=r"(hi) + : [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128) + : "r0", "r1", "r2", "r3", "r4", "r5" + ); +if (hi != a) +return 3 ; +if (lo != b) +return 4 ;
[Severity: Medium]
The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in r0. On a big-endian ABI, the most significant 64 bits (a) will be returned in the first register (r0).
Will this cause the test to fail on big-endian architectures like s390x?
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+ +return 0 ; +} + +SEC("tc") +int aggregate_ret_struct_test (struct __sk_buff *skb) +{ +__u64 a = skb->len; +__u64 b = skb->len ^0xdeadbeefULL ; +__u64 lo, hi;
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +5
-0+/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */ + asmvolatile ( +"r1 = %[a];" +"r2 = %[b];" + "call %[kfunc];"
tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 115
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+"%[lo] = r0;" +"%[hi] = r2;" + : [lo]"=r"(lo), [hi]"=r"(hi) + : [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li) + : "r0", "r1", "r2", "r3", "r4", "r5" + ); +if (lo != a) +return 3 ; +if ((int )hi != (int )b) +return 4 ;
[Severity: Medium]
The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32 bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of the register, causing this assertion to fail?
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +5
-0+ +/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */ + asmvolatile ( +"r1 = %[a];" + "r2 = %[b];"
tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 130
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+"call %[kfunc];" +"%[lo] = r0;" + : [lo]"=r"(lo) + : [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii) + : "r0", "r1", "r2", "r3", "r4", "r5" + ); +if ((int )lo != (int )a) +return 5 ; +if ((int )(lo >>32 ) != (int )b) +return 6 ;
[Severity: Medium]
The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian register layout for the returned 8-byte struct. On big-endian, wouldn't a be in the upper 32 bits and b in the lower 32 bits?
Quoted 3 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +3
-0+ +return 0 ; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Yonghong Song
reviewerRe: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
Quoted 51 lines · On 7/8/26 1:28 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoded little-endian assumptions in struct and__int128register unpacking will cause selftest failures on big-endian architectures like s390x. ----- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns --- commit
dcf423ead16608630f40807ad0cde33d495c941d
Author: Yonghong Song <[email protected]>selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.
Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0
-0[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0
-0@@ -0,0 +1,160 @@[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +6
-0+SEC("tc") +int aggregate_ret_asm_test (struct __sk_buff *skb) +{ +__u64 a = skb->len; +__u64 b = skb->len ^0xdeadbeefULL ; +__u64 lo, hi;[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4
-0+ asmvolatile ( +"r1 = %[a];" +"r2 = %[b];" + "call %[kfunc];"tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 73
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+"%[lo] = r0;" +"%[hi] = r2;" + : [lo]"=r"(lo), [hi]"=r"(hi) + : [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128) + : "r0", "r1", "r2", "r3", "r4", "r5" + ); +if (hi != a) +return 3 ; +if (lo != b) +return 4 ;[Severity: Medium]
The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in r0. On a big-endian ABI, the most significant 64 bits (a) will be returned in the first register (r0).
Quoted 1 line
Will this cause the test to fail on big-endian architectures like s390x?
Good point. I need to test it for s390x.
Quoted 24 lines
Quoted 11 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10
-0+ +return 0 ; +} + +SEC("tc") +int aggregate_ret_struct_test (struct __sk_buff *skb) +{ +__u64 a = skb->len; +__u64 b = skb->len ^0xdeadbeefULL ; +__u64 lo, hi;[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +11
-0+/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */ + asmvolatile ( +"r1 = %[a];" +"r2 = %[b];" +"call %[kfunc];" +"%[lo] = r0;" +"%[hi] = r2;" + : [lo]"=r" (lo), [hi]"=r" (hi) + : [a]"r" (a), [b]"r" (b), [kfunc]"i" (bpf_kfunc_call_test_ret_li) + :"r0" ,"r1" ,"r2" ,"r3" ,"r4" ,"r5" + );
Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 115
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4
-0+if (lo != a) +return 3 ; +if ((int )hi != (int )b) +return 4 ;[Severity: Medium]
The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32 bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of the register, causing this assertion to fail?
The same as above. Need tests with s390x.
Quoted 12 lines
Quoted 12 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +11
-0+ +/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */ + asmvolatile ( +"r1 = %[a];" +"r2 = %[b];" +"call %[kfunc];" +"%[lo] = r0;" + : [lo]"=r" (lo) + : [a]"r" (a), [b]"r" (b), [kfunc]"i" (bpf_kfunc_call_test_ret_ii) + :"r0" ,"r1" ,"r2" ,"r3" ,"r4" ,"r5" + );
Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 130
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4
-0+if ((int )lo != (int )a) +return 5 ; +if ((int )(lo >>32 ) != (int )b) +return 6 ;[Severity: Medium]
The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian register layout for the returned 8-byte struct. On big-endian, wouldn't a be in the upper 32 bits and b in the lower 32 bits?
Will test as the above.
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +3
-0+ +return 0 ; +}
Eduard Zingerman
maintainerRe: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
In reply to Yonghong SongQuoted 6 lines · Yonghong Song
Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.
Signed-off-by Yonghong Song <[email protected]>---
[...]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +0
-0@@ -93,4 +97,25 @@ void test_aggregate_ret(void) run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
aggregate_ret_union_c__destroy(skel_union_c);tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +11
-0+ + skel_run =aggregate_ret_run__open_and_load (); +if (!ASSERT_OK_PTR (skel_run,"skel_run_open_load" )) +return ; + +/* Inline-asm variant: compiler-version independent, always runs. */ +if (test__start_subtest ("asm" )) +run_prog (skel_run->progs.aggregate_ret_asm_test, false); + +/* Struct-by-value kfunc returns, also compiler-version independent. */ +if (test__start_subtest ("struct" ))
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c void test_aggregate_ret(void) · line 120
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +10
-0+run_prog (skel_run->progs.aggregate_ret_struct_test, false); + +/* Union-by-value kfunc return, compiler-version independent. */ +if (test__start_subtest ("union" )) +run_prog (skel_run->progs.aggregate_ret_union_test, false); + +aggregate_ret_run__destroy (skel_run); + +RUN_TESTS (aggregate_ret_func); +RUN_TESTS (aggregate_ret_kfunc);
This can be folded into tools/testing/selftests/bpf/prog_tests/verifier.c.
Quoted 21 lines
}
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +14
-0@@ -0,0 +1,361 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128 ; + +__naked u128 global_agg_good (void ) +{ + asmvolatile ( +"r0 = 0x1234;" /* low 64 bits */ +"r2 = 0x5678;" /* high 64 bits */ + "exit;"
tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 24
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10
-0+ ); +} + +__naked u128 global_agg_bad (void ) +{ + asmvolatile ( +"r0 = 0;" +"exit;" + ); +}
C version of tests from a previous patch already checks global vs. local functions. I suggest dropping that version from C tests and keep in assembly only to reduce the duplication.
[...]
Quoted 9 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +9
-0+SEC("tc") +__success __retval(0) +int aggregate_ret_global_union (void *ctx) +{ +__u64 lo, hi; + + asmvolatile ( +"call %[global_ret_union];" + "%[lo] = r0;"
tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 284
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10
-0+ "%[hi] = r2;" + : [lo]"=r"(lo), [hi]"=r"(hi) + : __imm(global_ret_union) + : "r0", "r1", "r2", "r3", "r4", "r5"); +if (lo !=0x1234 ) +return 1 ; +if (hi !=0x5678 ) +return 2 ; +return 0 ; +}
And here I'd only keep the C version.
Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +8
-0+__naked struct with_ptr global_ret_struct_ptr (void ) +{ + asmvolatile ( +"r0 = 0;" +"r2 = 0;" +"exit;" + ); +}
[...]
Quoted 18 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +18
-0+static __naked u128 agg_callee (void ) +{ + asmvolatile ( +"r0 = 1;" +"r2 = 2;" +"exit;" + ); +} + +SEC("tc") +__log_level(2) +__msg("Live regs before insn:") +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +__msg ("5: 0.2....... (95) exit" )
tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 359
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10
-0+__naked int aggregate_ret_live (void ) +{ + asmvolatile ( +"call %[agg_callee];" +"r0 = r2;" +"exit;" + : + : [agg_callee]"i" (agg_callee) + : __clobber_all); +}
Nit: usually we add such tests to compute_live_registers.c.
[...]
tools/testing/selftests/bpf/progs/aggregate_ret_run.c
Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c +0
-0
I think the test cases in this file are by a large degree redundant. A single test case calling a kfunc returning a small struct and checking returned values would suffice.
[...]
Yonghong Song
reviewerRe: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
Quoted 125 lines · Eduard Zingerman
Quoted 6 lines · Yonghong Song
Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.
Signed-off-by Yonghong Song <[email protected]>---
[...]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +11
-0@@ -93,4 +97,25 @@ void test_aggregate_ret(void) run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true); aggregate_ret_union_c__destroy(skel_union_c); + + skel_run =aggregate_ret_run__open_and_load (); +if (!ASSERT_OK_PTR (skel_run,"skel_run_open_load" )) +return ; + +/* Inline-asm variant: compiler-version independent, always runs. */ +if (test__start_subtest ("asm" )) +run_prog (skel_run->progs.aggregate_ret_asm_test, false); + +/* Struct-by-value kfunc returns, also compiler-version independent. */ +if (test__start_subtest ("struct" ))tools/testing/selftests/bpf/prog_tests/aggregate_ret.c void test_aggregate_ret(void) · line 120
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +10
-0+run_prog (skel_run->progs.aggregate_ret_struct_test, false); + +/* Union-by-value kfunc return, compiler-version independent. */ +if (test__start_subtest ("union" )) +run_prog (skel_run->progs.aggregate_ret_union_test, false); + +aggregate_ret_run__destroy (skel_run); + +RUN_TESTS (aggregate_ret_func); +RUN_TESTS (aggregate_ret_kfunc);This can be folded into
tools/testing/selftests/bpf/prog_tests/verifier.c.Quoted 21 lines
(quoted diff) +0
-0}tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +14
-0@@ -0,0 +1,361 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128 ; + +__naked u128 global_agg_good (void ) +{ + asmvolatile ( +"r0 = 0x1234;" /* low 64 bits */ +"r2 = 0x5678;" /* high 64 bits */ + "exit;"tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 24
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10
-0+ ); +} + +__naked u128 global_agg_bad (void ) +{ + asmvolatile ( +"r0 = 0;" +"exit;" + ); +}C version of tests from a previous patch already checks global vs. local functions. I suggest dropping that version from C tests and keep in assembly only to reduce the duplication.
[...]
Quoted 9 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +9
-0+SEC("tc") +__success __retval(0) +int aggregate_ret_global_union (void *ctx) +{ +__u64 lo, hi; + + asmvolatile ( +"call %[global_ret_union];" + "%[lo] = r0;"tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 284
Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10
-0+ "%[hi] = r2;" + : [lo]"=r"(lo), [hi]"=r"(hi) + : __imm(global_ret_union) + : "r0", "r1", "r2", "r3", "r4", "r5"); +if (lo !=0x1234 ) +return 1 ; +if (hi !=0x5678 ) +return 2 ; +return 0 ; +}And here I'd only keep the C version.
Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +8
-0+__naked struct with_ptr global_ret_struct_ptr (void ) +{ + asmvolatile ( +"r0 = 0;" +"r2 = 0;" +"exit;" + ); +}[...]
Quoted 21 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +21
-0+static __naked u128 agg_callee (void ) +{ + asmvolatile ( +"r0 = 1;" +"r2 = 2;" +"exit;" + ); +} + +SEC("tc") +__log_level(2) +__msg("Live regs before insn:") +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +__msg("5: 0.2....... (95) exit") +__naked int aggregate_ret_live (void ) +{ + asm volatile (
Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 359
Quoted 7 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +7
-0+"call %[agg_callee];" +"r0 = r2;" +"exit;" + : + : [agg_callee]"i"(agg_callee) + : __clobber_all); +}Nit: usually we add such tests to
compute_live_registers.c.
I can move the test to compute_live_registers.c.
Quoted 7 lines
[...]
tools/testing/selftests/bpf/progs/aggregate_ret_run.c
Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c +0
-0I think the test cases in this file are by a large degree redundant. A single test case calling a kfunc returning a small struct and checking returned values would suffice.
Ack. I tried to test for both C and asm test cases. They are indeed some duplicates. I will try to remove the redundant ones in next revision.
Quoted 2 lines
[...]
Yonghong Song
reviewerPATCH 11/12 selftests/bpf: Cover tracing on >8 and <=16 byte return targets
In reply to Yonghong SongTracing attach types differ in whether they need the target's return value preserved across the BPF trampoline, which saves and restores only 8 bytes of it. fentry runs before the target returns and never observes the return value, so it can attach to a function returning a >8 byte value; fexit, fmod_ret and fsession read the return value and must keep rejecting such a target.
Add fentry, fexit, fentry_multi and fexit_multi tests that attach to functions returning a >8 and <=16 byte value (a 16-byte struct and an __int128).
--- .../selftests/bpf/prog_tests/aggregate_ret.c | 128 ++++++++++++++++++ .../selftests/bpf/progs/aggregate_ret_trace.c | 44 ++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 32 +++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 5 + 4 files changed, 209 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +128 -0
@@ -8,6 +8,7 @@#include "aggregate_ret_run.skel.h" #include "aggregate_ret_func.skel.h" #include "aggregate_ret_kfunc.skel.h" +#include "aggregate_ret_trace.skel.h" /* * Run one program and check its result. The program returns 0 on success, or
@@ -37,6 +38,131 @@ static void run_prog(struct bpf_program *prog, bool may_skip)ASSERT_EQ (topts.retval,0 ,"aggregate_ret_result" ); } +/* +* Multi-attach (BPF_TRACE_*_MULTI) requires CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS, +* which today only x86_64 selects. Otherwise bpf_tracing_multi_attach() returns +* -EOPNOTSUPP unconditionally. +*/ +static bool tracing_multi_supported (struct bpf_program *prog) +{ +const char *small_pat ="bpf_testmod:bpf_testmod_trampoline_count_test" ; +struct bpf_link *link; + + link =bpf_program__attach_tracing_multi (prog, small_pat,NULL ); +if (libbpf_get_error (link) == -EOPNOTSUPP ) +return false; +bpf_link__destroy (link); +return true; +} + +static void test_tracing_attach_types (void ) +{ +const char *i128_pat ="bpf_testmod:bpf_testmod_aggregate_ret_i128_fn" ; +int err; + +if (test__start_subtest ("trace_fentry" )) { +struct aggregate_ret_trace *trace =NULL ; + + trace =aggregate_ret_trace__open (); +if (!ASSERT_OK_PTR (trace,"trace_fentry_open" )) +return ; + +bpf_program__set_autoload (trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fentry_multi, false); +bpf_program__set_autoload (trace->progs.fexit_multi, false); + + err =aggregate_ret_trace__load (trace); +if (!ASSERT_OK (err,"trace_fentry_load" )) + goto destroy_trace_fentry; + + err =aggregate_ret_trace__attach (trace); +if (!ASSERT_OK (err,"trace_fentry_attach" )) + goto destroy_trace_fentry; + +ASSERT_OK (trigger_module_test_read (1 ),"trace_fentry_trigger" ); +ASSERT_EQ (trace->bss->fentry_hit,1 ,"trace_fentry_hit" ); + +destroy_trace_fentry: +aggregate_ret_trace__destroy (trace); + } + +if (test__start_subtest ("trace_fexit_reject" )) { +struct aggregate_ret_trace *trace =NULL ; + + trace =aggregate_ret_trace__open (); +if (!ASSERT_OK_PTR (trace,"trace_fexit_open" )) +return ; + +bpf_program__set_autoload (trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fentry_multi, false); +bpf_program__set_autoload (trace->progs.fexit_multi, false); + + err =aggregate_ret_trace__load (trace); +ASSERT_EQ (err, -EOPNOTSUPP ,"trace_fexit_load" ); + +aggregate_ret_trace__destroy (trace); + } + +if (test__start_subtest ("trace_fentry_multi" )) { +struct aggregate_ret_trace *trace =NULL ; +struct bpf_link *link; + + trace =aggregate_ret_trace__open (); +if (!ASSERT_OK_PTR (trace,"trace_fentry_multi_open" )) +return ; + +bpf_program__set_autoload (trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fexit_multi, false); + +if (!ASSERT_OK (aggregate_ret_trace__load (trace),"trace_fentry_multi_load" )) + goto destroy_fentry_multi; + +if (!tracing_multi_supported (trace->progs.fentry_multi)) { +test__skip (); + goto destroy_fentry_multi; + } + + link =bpf_program__attach_tracing_multi (trace->progs.fentry_multi, i128_pat,NULL ); +if (!ASSERT_OK_PTR (link,"trace_fentry_multi_attach" )) + goto destroy_fentry_multi; + +ASSERT_OK (trigger_module_test_read (1 ),"trace_fentry_multi_trigger" ); +ASSERT_EQ (trace->bss->fentry_multi_hit,1 ,"trace_fentry_multi_hit" ); + +bpf_link__destroy (link); +destroy_fentry_multi: +aggregate_ret_trace__destroy (trace); + } + +if (test__start_subtest ("trace_fexit_multi_reject" )) { +struct aggregate_ret_trace *trace =NULL ; +struct bpf_link *link; + + trace =aggregate_ret_trace__open (); +if (!ASSERT_OK_PTR (trace,"trace_fexit_multi_open" )) +return ; + +bpf_program__set_autoload (trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false); +bpf_program__set_autoload (trace->progs.fentry_multi, false); + +if (!ASSERT_OK (aggregate_ret_trace__load (trace),"trace_fexit_multi_load" )) + goto destroy_fexit_multi; + +if (!tracing_multi_supported (trace->progs.fexit_multi)) { +test__skip (); + goto destroy_fexit_multi; + } + + link =bpf_program__attach_tracing_multi (trace->progs.fexit_multi, i128_pat,NULL ); +ASSERT_EQ (libbpf_get_error (link), -EOPNOTSUPP ,"trace_fexit_multi_reject" ); +bpf_link__destroy (link); +destroy_fexit_multi: +aggregate_ret_trace__destroy (trace); + } +} +void test_aggregate_ret (void ) {struct aggregate_ret_struct_c *skel_struct_c;
@@ -118,4 +244,6 @@ void test_aggregate_ret(void)RUN_TESTS (aggregate_ret_func);RUN_TESTS (aggregate_ret_kfunc); + +test_tracing_attach_types (); }
tools/testing/selftests/bpf/progs/aggregate_ret_trace.c new file +44 -0
44 lines not on this page · raw message
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +32 -0
@@ -469,12 +469,34 @@ noinline void bpf_testmod_stacktrace_test_1(void) }int bpf_testmod_fentry_ok; +u64 bpf_testmod_aggregate_ret_seen; noinline int bpf_testmod_trampoline_count_test(void) {return 0 ; } +__noclone noinline struct bpf_testmod_aggregate_ret +bpf_testmod_aggregate_ret_fn (u64 lo,u64 hi) +{ +struct bpf_testmod_aggregate_ret ret = { + .lo = lo, + .hi = hi, + }; + + bpf_testmod_aggregate_ret_seen += lo + hi; +return ret; +} + +__noclone noinline __int128 +bpf_testmod_aggregate_ret_i128_fn (u64 lo,u64 hi) +{ +__int128 ret = ((__int128 )hi <<64 ) | lo; + + bpf_testmod_aggregate_ret_seen += lo + hi; +return ret; +} + noinline ssize_t bpf_testmod_test_read(struct file *file, struct kobject *kobj, const struct bin_attribute *bin_attr,
@@ -492,6 +514,8 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,struct bpf_testmod_struct_arg_5 struct_arg5 = {23 ,24 ,25 ,26 };union bpf_testmod_union_arg_1 union_arg1 = { .arg = {1 } };union bpf_testmod_union_arg_2 union_arg2 = { .arg = {2 ,3 } }; +struct bpf_testmod_aggregate_ret agg_ret; +__int128 agg_i128;int i =1 ;while (bpf_testmod_return_ptr (i))
@@ -534,6 +558,14 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,trace_bpf_testmod_test_nullable_bare_tp (NULL ); + agg_ret =bpf_testmod_aggregate_ret_fn (len, off); +if (agg_ret.lo ==U64_MAX && agg_ret.hi ==U64_MAX ) + goto out; + + agg_i128 =bpf_testmod_aggregate_ret_i128_fn (len, off); +if (agg_i128 == (__int128)-1 ) + goto out; +/* Magic number to enable writable tp */ if (len ==64 ) {struct bpf_testmod_test_writable_ctx writable = {
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +5 -0
@@ -25,6 +25,11 @@ struct bpf_testmod_test_writable_ctx {int val; }; +struct bpf_testmod_aggregate_ret { +__u64 lo; +__u64 hi; +}; +/* BPF iter that returns *value* *n* times in a row */ struct bpf_iter_testmod_seq {s64 value;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 12/12 Documentation/bpf: Document 16-byte kfunc return values in R0:R2
In reply to Yonghong Songkfuncs may now return a value larger than 8 bytes and up to 16 bytes (a 16-byte struct or union, or an __int128), passed back in the R0:R2 register pair. Add a kfunc return-value section documenting this, including that a struct or union up to 8 bytes is returned in R0 alone, that the R0:R2 register pair requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a return value larger than 16 bytes is unsupported.
--- Documentation/bpf/kfuncs.rst | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
Documentation/bpf/kfuncs.rst +36 -0
@@ -501,6 +501,42 @@ is also covered by this recovery. A kfunc handed an arena pointer may therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking against the arena. Larger accesses must verify the range explicitly. +2.9 kfunc Return Values +----------------------- + +A kfunc may return a scalar, a pointer, or a small struct or union by +value. A scalar or pointer of up to 8 bytes is returned in R0, as usual. + +A struct or union returned by value must be composed only of scalars +(recursively). Its bytes are handed back to the program as the raw contents +of R0 (and R2), so a pointer field would otherwise be laundered into a scalar +and escape the verifier's pointer provenance and reference tracking. Such a +struct or union is therefore rejected at load time. + +A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a +16-byte scalar-only struct or union, or an ``__int128``. Such a value is +returned in the register pair R0:R2, matching the convention LLVM uses for +the BPF target: the low 64 bits in R0 and the high 64 bits in R2. A struct +or union of 8 bytes or less is returned in R0 alone. + +:: + + struct bpf_pair { __u64 a, b; }; /* 16 bytes */ + + __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void) + { + struct bpf_pair p = { .a = 1, .b = 2 }; + + return p; /* p.a in R0, p.b in R2 */ + } + +Returning a value in the R0:R2 pair requires the JIT to place the second +half of the return value into R2, which not every architecture supports +right now. A kfunc with a return value larger than 8 bytes is therefore +rejected at load time on a JIT that does not advertise this capability (see +``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run +by the interpreter. A return value larger than 16 bytes is not supported. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations
Signature · 1 line
2.53.0-Meta
Eduard Zingerman
maintainerQuoted 1 line · Yonghong Song
A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.
I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.
I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.
Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT
- forgo the scalar struct requirement for local function return types.
Wdyt?
Yonghong Song
reviewerQuoted 12 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song
A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.
Quoted 6 lines
I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.
Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT
Not sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?
Quoted 2 lines
- forgo the scalar struct requirement for local function return types.
I think this is doable.
Quoted 2 lines
Wdyt?
Eduard Zingerman
maintainerQuoted 18 lines · Yonghong Song
Quoted 15 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song
A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.
I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.
Quoted 8 lines
Quoted 3 lines
Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXITNot sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?
We can detect this statically from compute_live_registers().
If any r2's are alive after a function call the jit_required flag can be set.
Quoted 7 lines
Quoted 2 lines
- forgo the scalar struct requirement for local function return types.
I think this is doable.
Quoted 2 lines
Wdyt?
Yonghong Song
reviewerQuoted 21 lines · Eduard Zingerman
Quoted 20 lines · Yonghong Song
Quoted 18 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song
A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.
I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.
Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT
Quoted 6 lines
Quoted 4 lines
Not sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?
We can detect this statically from
compute_live_registers().
If any r2's are alive after a function call thejit_requiredflag can be set.
Sounds good, I will do some investigation on this.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
- forgo the scalar struct requirement for local function return types.
I think this is doable.
Quoted 1 line
Wdyt?