Skip to contentSkip to conversation

bpf: Support 16-byte return values in the R0:R2 register pair

PATCH Superseded Yonghong Song 5 participants 56 messages Jul 8 – Jul 10

Subsystems (12)
Documentation x86 Kselftest BPF +8 more subsystems

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

cover bpf: Support 16-byte return values in the R0:R2 register pair

Until now a BPF subprogram or kfunc could only return a scalar value of up to 8 bytes, passed back in R0. LLVM 23 added the ability to return a value in two registers for the BPF target [1][2]: an __int128, or a struct/union whose size is greater than 8 and up to 16 bytes, is returned in the R0:R2 register pair, with the low 64 bits in R0 and the high 64 bits in R2. (Before LLVM 23 the BPF backend could not return such values at all:
a by-value aggregate return was rejected with "aggregate returns are not supported", and an __int128 return failed in the backend with "unable to allocate function return #1".)

This series teaches the kernel about that convention so that a BPF-to-BPF subprogram call and a kfunc call can return up to 16 bytes. A returned struct or union must be composed only of scalars (recursively); the bytes are handed back as the raw contents of R0:R2. Otherwise allowing pointer members would launder a pointer into a scalar and escape the verifier's provenance and reference tracking. A value larger than 16 bytes remains unsupported, as does a <=8 byte return, which continues to use R0 alone.

Support is built up incrementally so each step is reviewable on its own, and the feature is only turned on end to end by the last enabling patch.

- Patch 1 refactors the global-subprog return check so the per-register scalar validation can be reused for R2.
- Patch 2 to Patch 7 are preparation patches for core verifier handling of R0:R2 returns, JIT support, precision backtracking, liveness analysis, and limitation on trampoline bpf programs.
- Patch 8 relaxes btf_distill_func_proto() and btf_validate_return_type() to accept <=16 byte aggregates, enabling the feature end to end.
- Patches 9-11 add selftests including C and inline asm tests.
- Patch 12 documents the kfunc return-value convention in kfuncs.rst.

Tested on x86 and arm64. The patch set is built on top of the following commit:

    Merge branch 'verify-bpf-signed-loader-at-load-time'

    [1] https://github.com/llvm/llvm-project/pull/190894
    [2] https://github.com/llvm/llvm-project/pull/206876

Yonghong Song (12):
bpf: Factor check_global_ret_scalar_reg() out of the global return
check
bpf: Add verifier support for 16-byte returns in R0:R2
bpf: Wire up JIT support for 16-byte kfunc returns
bpf: Track R2 of register-pair returns in precision backtracking
bpf: Account R2 of register-pair returns in live register analysis
bpf: Force JIT for programs using the R0:R2 register pair
bpf: Reject >8 byte return values on return-reading trampoline paths
bpf: Enable 16-byte aggregate return types
selftests/bpf: Add C tests for 16-byte returns in R0:R2 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns selftests/bpf: Cover tracing on >8 and <=16 byte return targets Documentation/bpf: Document 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  36 ++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  19 +
 include/linux/bpf.h                           |   1 +
 include/linux/bpf_verifier.h                  |   4 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  32 +-
 kernel/bpf/bpf_struct_ops.c                   |  12 +
 kernel/bpf/btf.c                              |  14 +-
 kernel/bpf/core.c                             |   7 +-
 kernel/bpf/liveness.c                         |  13 +-
 kernel/bpf/verifier.c                         | 179 ++++++++-
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 249 ++++++++++++
 .../selftests/bpf/progs/aggregate_ret_func.c  | 361 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 +++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |  81 ++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 160 ++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  98 +++++
 .../selftests/bpf/progs/aggregate_ret_trace.c |  44 +++
 .../bpf/progs/aggregate_ret_union_c.c         |  51 +++
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  80 ++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   5 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  38 ++
 25 files changed, 1517 insertions(+), 28 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH 1/12 bpf: Factor check_global_ret_scalar_reg() out of the global return check

Function check_global_subprog_return_code() verifies that a global subprogram returns void, an arena pointer, or register R0 which holds a scalar value. Later patches in this series add 16-byte aggregate return support, whose second half is returned in R2 and needs similar validation.

Factor the per-register check into check_global_ret_scalar_reg(env, regno) so it can be reused for R2.

---
 kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)
kernel/bpf/verifier.c +20 -12
index 001ac53825da..3f34ce1a3107 100644
@@ -16679,37 +16679,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	return check_global_ret_scalar_reg(env, BPF_REG_0);
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH 2/12 bpf: Add verifier support for 16-byte returns in R0:R2

In reply to Yonghong Song

LLVM 23 added support for returning a value in two registers for an __int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:

error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.

This patch is one of several preparation patches that build up <=16 byte return support incrementally:
- the verifier R0:R2 modelling in this patch,
- the x86 JIT move and JIT-capability gate,
- precision-backtracking and live-register tracking of R2,
- forcing the JIT for such programs, and
- guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.

This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.

[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876

---
 kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 88 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +88 -5
index 3f34ce1a3107..03ffb5f839fa 100644
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
 	return btf_type_is_void(type);
 }
 
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!subprog_returns_void(env, subprog)) {
 			mark_reg_unknown(env, caller->regs, BPF_REG_0);
 			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
+			if (bpf_ret_reg_pair(env, subprog)) {
+				mark_reg_unknown(env, caller->regs, BPF_REG_2);
+				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+			}
 		}
 
 		if (env->subprog_info[subprog].might_throw) {
@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	} else {
 		/* return to the caller whatever r0 had in the callee */
 		caller->regs[BPF_REG_0] = *r0;
+		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
+				verbose(env, "cannot return stack pointer to the caller\n");
+				return -EINVAL;
+			}
+			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
+		}
 	}
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
 	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
 }
 
+static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,
+				    struct bpf_reg_state *regs, u32 size)
+{
+	if (size > 8) {
+		mark_btf_func_reg_size(env, BPF_REG_0, 8);
+		mark_reg_unknown(env, regs, BPF_REG_2);
+		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+	} else {
+		mark_btf_func_reg_size(env, BPF_REG_0, size);
+	}
+}
+
 static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
-		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16711,11 +16781,19 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 {
 	struct bpf_func_state *cur_frame = cur_func(env);
 	u32 subprog = cur_frame->subprogno;
+	int err;
 
 	if (subprog_returns_void(env, subprog))
 		return 0;
 
-	return check_global_ret_scalar_reg(env, BPF_REG_0);
+	err = check_global_ret_scalar_reg(env, BPF_REG_0);
+	if (err)
+		return err;
+
+	if (!bpf_ret_reg_pair(env, subprog))
+		return 0;
+
+	return check_global_ret_scalar_reg(env, BPF_REG_2);
 }
 
 /* Bitmask with 1s for all caller saved registers */
@@ -17205,10 +17283,15 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	 */
 	if (cur_frame->subprogno &&
 	    !cur_frame->in_async_callback_fn &&
-	    !cur_frame->in_exception_callback_fn)
+	    !cur_frame->in_exception_callback_fn) {
 		err = check_global_subprog_return_code(env);
-	else
+	} else {
+		if (!cur_frame->subprogno && bpf_ret_reg_pair(env, 0)) {
+			verbose(env, "return value larger than 8 bytes is not supported at program exit\n");
+			return -EINVAL;
+		}
 		err = check_return_code(env, BPF_REG_0, "R0");
+	}
 	if (err)
 		return err;
 	return PROCESS_BPF_EXIT;
Signature · 1 line
2.53.0-Meta
#3 lore raw

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void() share a helper for the func_info -> func -> func_proto -> return type walk?
Quoted 26 lines
kernel/bpf/verifier.c +18 -0
index 3f34ce1a3107d..03ffb5f839fa0 100644
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
 	return btf_type_is_void(type);
 }
 
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;

kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 454

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}

This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void() share a helper for the func_info -> func -> func_proto -> return type walk?

Both functions repeat the same ~8 line sequence:

subprog_returns_void():

	btf_id = env->prog->aux->func_info[subprog].type_id;
	func = btf_type_by_id(btf, btf_id);
	if (!func)
		return false;
	func_proto = btf_type_by_id(btf, func->type);
	if (!func_proto)
		return false;
	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);

They only differ in the final predicate: subprog_returns_void() checks btf_type_is_void(type), while bpf_ret_reg_pair() checks the struct/scalar size.

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

author reviewer
#4 lore raw

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

Quoted 51 lines · On 7/8/26 2:17 PM, [email protected] wrote:
Quoted 26 lines
kernel/bpf/verifier.c +18 -0
index 3f34ce1a3107d..03ffb5f839fa0 100644
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
  	return btf_type_is_void(type);
  }
 
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;

kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 454

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}

This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void() share a helper for the func_info -> func -> func_proto -> return type walk?

Both functions repeat the same ~8 line sequence:

subprog_returns_void():

	btf_id = env->prog->aux->func_info[subprog].type_id;
	func = btf_type_by_id(btf, btf_id);
	if (!func)
		return false;
	func_proto = btf_type_by_id(btf, func->type);
	if (!func_proto)
		return false;
	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
Quoted 5 lines

They only differ in the final predicate: subprog_returns_void() checks btf_type_is_void(type), while bpf_ret_reg_pair() checks the struct/scalar size.

[ ... ]

Indeed, I think this probably worthwhile to factor a new helper.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

maintainer
#5 lore raw

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

In reply to Yonghong Song
Quoted 17 lines · Yonghong Song

LLVM 23 added support for returning a value in two registers for an __int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:

  error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

  fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.

Quoted 9 lines

This patch is one of several preparation patches that build up <=16 byte return support incrementally:
  - the verifier R0:R2 modelling in this patch,
  - the x86 JIT move and JIT-capability gate,
  - precision-backtracking and live-register tracking of R2,
  - forcing the JIT for such programs, and
  - guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.

Nit: I think the above paragraphs belong to a cover letter.

Quoted 24 lines

This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

---
 kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---

 1 file changed, 88 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +0 -0
index 3f34ce1a3107..03ffb5f839fa 100644
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
  	return btf_type_is_void(type);
  }

kernel/bpf/verifier.c +4 -0
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;

kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 440

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;

if (!func) return false checks here and below are not necessary after BTF validation.

Quoted 38 lines
kernel/bpf/verifier.c +15 -0
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}
+

 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog) {
  struct bpf_func_info *info;

kernel/bpf/verifier.c +0 -0
@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  		clear_caller_saved_regs(env, caller->regs);
  		invalidate_outgoing_stack_args(env, cur_func(env));

kernel/bpf/verifier.c +4 -1
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */

  if (!subprog_returns_void(env, subprog)) { mark_reg_unknown(env, caller->regs, BPF_REG_0); caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;

kernel/bpf/verifier.c +4 -0
+			if (bpf_ret_reg_pair(env, subprog)) {
+				mark_reg_unknown(env, caller->regs, BPF_REG_2);
+				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+			}

  }

  if (env->subprog_info[subprog].might_throw) {

kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9865

Quoted 5 lines
kernel/bpf/verifier.c +2 -0
@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
  	} else {
  		/* return to the caller whatever r0 had in the callee */
  		caller->regs[BPF_REG_0] = *r0;
+		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {

Let's consolidate all such checks in one place and not copy-paste them.

Quoted 13 lines
kernel/bpf/verifier.c +5 -0
+				verbose(env, "cannot return stack pointer to the caller\n");
+				return -EINVAL;
+			}
+			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
+		}
  	}


  /* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,

kernel/bpf/verifier.c +0 -0
@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
  	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
  }

kernel/bpf/verifier.c +1 -0
+static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,

kernel/bpf/verifier.c static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, · line 10801

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+				    struct bpf_reg_state *regs, u32 size)
+{
+	if (size > 8) {
+		mark_btf_func_reg_size(env, BPF_REG_0, 8);
+		mark_reg_unknown(env, regs, BPF_REG_2);
+		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+	} else {
+		mark_btf_func_reg_size(env, BPF_REG_0, size);
+	}
+}

This function makes R2 handling very asymmetric compared to R0:
- mark_reg_unknown() for r0 is done in check_kfunc_call()
- DEF_NOT_SUBREG for r0 is done in mark_btf_func_reg_size()

I think the code should be structured to keep R0 and R2 processing uniform.

Quoted 11 lines

+
 static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta) {
  return meta->kfunc_flags & KF_ACQUIRE;

kernel/bpf/verifier.c +0 -0
@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,

  if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, &regs[BPF_REG_0]);

kernel/bpf/verifier.c +2 -1
-		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
+	} else if (btf_type_is_struct(t)) {

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 13277

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));

Should this also check the size of the struct?

Quoted 7 lines
kernel/bpf/verifier.c +4 -0
+			return -EINVAL;
+		}
+		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_reg_size(env, regs, t->size);

  } else if (btf_type_is_ptr(t)) {
  ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);

[...]

author reviewer
#6 lore raw

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

Quoted 27 lines · Eduard Zingerman
Quoted 17 lines · Yonghong Song

LLVM 23 added support for returning a value in two registers for an __int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with:

  error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

  fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair.

Quoted 9 lines

This patch is one of several preparation patches that build up <=16 byte return support incrementally:
  - the verifier R0:R2 modelling in this patch,
  - the x86 JIT move and JIT-capability gate,
  - precision-backtracking and live-register tracking of R2,
  - forcing the JIT for such programs, and
  - guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables support for 16-byte returns.

Quoted 1 line

Nit: I think the above paragraphs belong to a cover letter.

Will add this to the cover letter.

Quoted 21 lines
Quoted 21 lines

This patch adds handling for '>8' byte returns in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

---
  kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---
  1 file changed, 88 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +0 -0
index 3f34ce1a3107..03ffb5f839fa 100644
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
  	return btf_type_is_void(type);
  }
 
Quoted 16 lines
Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;

kernel/bpf/verifier.c static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) · line 440

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;

if (!func) return false checks here and below are not necessary after BTF validation.

Ack. Will remove them.

Quoted 37 lines
Quoted 37 lines
kernel/bpf/verifier.c +23 -1
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}
+
  static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
  {
  	struct bpf_func_info *info;
@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  		clear_caller_saved_regs(env, caller->regs);
  		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
  		if (!subprog_returns_void(env, subprog)) {
  			mark_reg_unknown(env, caller->regs, BPF_REG_0);
  			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
+			if (bpf_ret_reg_pair(env, subprog)) {
+				mark_reg_unknown(env, caller->regs, BPF_REG_2);
+				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+			}
  		}
 
Quoted 8 lines

kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9865

Quoted 7 lines
kernel/bpf/verifier.c +2 -0
  		if (env->subprog_info[subprog].might_throw) {
@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
  	} else {
  		/* return to the caller whatever r0 had in the callee */
  		caller->regs[BPF_REG_0] = *r0;
+		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {

Let's consolidate all such checks in one place and not copy-paste them.

Will try to consolidate them together.

Quoted 28 lines
Quoted 14 lines
kernel/bpf/verifier.c +6 -0
+				verbose(env, "cannot return stack pointer to the caller\n");
+				return -EINVAL;
+			}
+			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
+		}
  	}
 
  	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
  	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
  }
 
+static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,

kernel/bpf/verifier.c static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, · line 10801

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+				    struct bpf_reg_state *regs, u32 size)
+{
+	if (size > 8) {
+		mark_btf_func_reg_size(env, BPF_REG_0, 8);
+		mark_reg_unknown(env, regs, BPF_REG_2);
+		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+	} else {
+		mark_btf_func_reg_size(env, BPF_REG_0, size);
+	}
+}

This function makes R2 handling very asymmetric compared to R0:
- mark_reg_unknown() for r0 is done in check_kfunc_call()
- DEF_NOT_SUBREG for r0 is done in mark_btf_func_reg_size()

Quoted 2 lines

I think the code should be structured to keep R0 and R2 processing uniform.

Ack

Quoted 15 lines
Quoted 15 lines
kernel/bpf/verifier.c +6 -1
+
  static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta)
  {
  	return meta->kfunc_flags & KF_ACQUIRE;
@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
  		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
  			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
-		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
Quoted 8 lines

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 13277

Quoted 7 lines
kernel/bpf/verifier.c +7 -0
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));

Should this also check the size of the struct?

Later on, there is a size condition for struct (<= 16), so we should be okay here.

Quoted 9 lines
Quoted 8 lines
kernel/bpf/verifier.c +4 -0
+			return -EINVAL;
+		}
+		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
  	} else if (btf_type_is_ptr(t)) {
  		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
  		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);

[...]

author reviewer
#7 lore raw

PATCH 3/12 bpf: Wire up JIT support for 16-byte kfunc returns

In reply to Yonghong Song

Continue preparing <=16 byte aggregate return support by wiring up the JIT side of the R0:R2 return convention for kfuncs.

A kfunc returning more than 8 bytes (a small aggregate or an __int128) hands the second half of the result back in RDX, the native x86-64 ABI's second return register. BPF instead expects a 16-byte return in the R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8.

Placing the second return half into R2 is possible on any JIT, but it needs architecture-specific JIT work. Rather than requiring every JIT to implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair() capability, defaulting to false in the generic core; an architecture opts in once its JIT handles the R0:R2 pair, and the remaining ones are left for future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT lacks the capability. Only x86, arm64 and riscv are supported so far.

On arm64 and riscv the native second return register is already BPF R2 (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI). This has been tested on x86 and arm64. The riscv path is expected to work by the same register-mapping reasoning as arm64 but has not been tested.

---
 arch/arm64/net/bpf_jit_comp.c   |  5 +++++
 arch/riscv/net/bpf_jit_comp64.c |  5 +++++
 arch/x86/net/bpf_jit_comp.c     | 19 +++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               |  5 +++++
 kernel/bpf/verifier.c           |  5 +++++
 6 files changed, 40 insertions(+)
arch/arm64/net/bpf_jit_comp.c +5 -0
index b0075ece4a6e..a4292e9ec075 100644
@@ -2312,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
arch/riscv/net/bpf_jit_comp64.c +5 -0
index f9d5347ba966..68faa7016d30 100644
@@ -2107,6 +2107,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
arch/x86/net/bpf_jit_comp.c +19 -0
index de7515ea1bea..0c685c93df15 100644
@@ -2592,6 +2592,20 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
+				const struct btf_func_model *fm;
+
+				/*
+				 * A kfunc returning a >8 byte aggregate hands the
+				 * second half back in RDX (the native ABI's second
+				 * return reg), but BPF expects it in R0:R2. BPF R0
+				 * is RAX (no move needed), while BPF R2 is RSI, so
+				 * copy RDX into RSI.
+				 */
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (fm && fm->ret_size > 8)
+					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
+			}
 			break;
 		}
 
@@ -4045,6 +4059,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
include/linux/filter.h +1 -0
index 14acb2455746..308a0e2d5988 100644
@@ -1182,6 +1182,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
kernel/bpf/core.c +5 -0
index 6e19a030da6f..73dc3ee879de 100644
@@ -3294,6 +3294,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
kernel/bpf/verifier.c +5 -0
index 03ffb5f839fa..e3f934cf702d 100644
@@ -2851,6 +2851,11 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8 && !bpf_jit_supports_kfunc_ret_reg_pair()) {
+		verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+			kfunc.name);
+		return -EOPNOTSUPP;
+	}
 
 	desc = &tab->descs[tab->nr_descs++];
 	desc->func_id = func_id;
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v1 3/12
#8 lore raw

Re: [PATCH bpf-next 03/12] bpf: Wire up JIT support for 16-byte kfunc returns

Quoted 20 lines · Yonghong Song

Continue preparing <=16 byte aggregate return support by wiring up the JIT side of the R0:R2 return convention for kfuncs.

A kfunc returning more than 8 bytes (a small aggregate or an __int128) hands the second half of the result back in RDX, the native x86-64 ABI's second return register. BPF instead expects a 16-byte return in the R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8.

Placing the second return half into R2 is possible on any JIT, but it needs architecture-specific JIT work. Rather than requiring every JIT to implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair() capability, defaulting to false in the generic core; an architecture opts in once its JIT handles the R0:R2 pair, and the remaining ones are left for future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT lacks the capability. Only x86, arm64 and riscv are supported so far.

Quoted 8 lines

On arm64 and riscv the native second return register is already BPF R2 (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI). This has been tested on x86 and arm64. The riscv path is expected to work by the same register-mapping reasoning as arm64 but has not been tested.

---

[...]

author reviewer
#9 lore raw

PATCH 4/12 bpf: Track R2 of register-pair returns in precision backtracking

In reply to Yonghong Song

A function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT.

Handle R2 like R0 in the three boundaries where a call defines the return registers:

- static subprog exit (BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0.

---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/backtrack.c       | 32 +++++++++++++++++++++++++++++++-
 kernel/bpf/verifier.c        | 13 ++++++++++++-
 3 files changed, 45 insertions(+), 2 deletions(-)
include/linux/bpf_verifier.h +2 -0
index 317e99b9acc0..fa255f326240 100644
@@ -1431,6 +1431,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
kernel/bpf/backtrack.c +31 -1
index 2e4ae0ef0860..c46c2778bf60 100644
@@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				if (bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* On error the return is negative, R2 stays set, and the
+			 * check below catches it as an unexpected register.
+			 */
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off) > 8)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool r0_precise, r2_precise = false;
+
+			/* A subprog returning a value larger than 8 bytes (a
+			 * struct/union or an __int128) uses R2 as a second
+			 * return register alongside R0. When exiting such a
+			 * subprog, R2 is a return register rather than a
+			 * leftover argument, so handle it like R0 below.
+			 */
+			if (subseq_idx - 1 >= 0 &&
+			    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1])) {
+				int call_idx = subseq_idx - 1;
+				int tgt = call_idx + env->prog->insnsi[call_idx].imm + 1;
+				int subprog = bpf_find_subprog(env, tgt);
+
+				if (subprog >= 0 &&
+				    bpf_ret_reg_pair(env, subprog) &&
+				    bt_is_reg_set(bt, BPF_REG_2))
+					r2_precise = true;
+			}
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -534,6 +560,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+			if (r2_precise)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
@@ -558,6 +586,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
kernel/bpf/verifier.c +12 -1
index e3f934cf702d..fcbf532159a5 100644
@@ -424,7 +424,7 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
 	return btf_type_is_void(type);
 }
 
-static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 {
 	const struct btf_type *type, *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
@@ -2521,6 +2521,17 @@ int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
 	return 0;
 }
 
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset)
+{
+	const struct bpf_kfunc_desc *desc;
+
+	desc = find_kfunc_desc(prog, func_id, offset);
+	if (!desc)
+		return -EFAULT;
+
+	return desc->func_model.ret_size;
+}
+
 #define BPF_FD_SLOT_BTF	1UL
 
 static void fd_slot_set_map(struct bpf_fd_array *slot, struct bpf_map *map)
Signature · 1 line
2.53.0-Meta
#10 lore raw

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where backtrack_insn() was not updated to treat R2 as a return register. Should this carry a…
Quoted 18 lines

A function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT.

Handle R2 like R0 in the three boundaries where a call defines the return registers:

- static subprog exit (BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0.

Quoted 1 line

This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where backtrack_insn() was not updated to treat R2 as a return register. Should this carry a Fixes: tag?

Fixes ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

author reviewer
#11 lore raw

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

Quoted 18 lines · On 7/8/26 2:10 PM, [email protected] wrote:
Quoted 18 lines

A function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT.

Handle R2 like R0 in the three boundaries where a call defines the return registers:

- static subprog exit (BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
- global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
- kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0.

Quoted 6 lines
Quoted 1 line

This looks like a fix for the R0:R2 register-pair return handling that went in earlier in this series, where backtrack_insn() was not updated to treat R2 as a return register. Should this carry a Fixes: tag?

Fixes ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")

This is still a preparation patch, so Fixes is not necessary. The same for a few later patches.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

maintainer Acked-by on v1 4/12
#12 lore raw

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

In reply to Yonghong Song
Quoted 18 lines · Yonghong Song

A function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Previous precision backtracking only treated R0 as a return register at a call/return boundary, so marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT.

Handle R2 like R0 in the three boundaries where a call defines the return registers:

 - static subprog exit (BPF_EXIT): R2 is a return register, not a leftover argument; clear it before the args check and propagate its precision into the callee.
 - global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check.
 - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0.

Quoted 2 lines

---

kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 428

Quoted 10 lines
kernel/bpf/backtrack.c +2 -0
index 2e4ae0ef0860..c46c2778bf60 100644
@@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
  				 */
  				verifier_bug_if(idx + 1 != subseq_idx, env,
  						"extra insn from subprog");
+				if (bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);

Nit: please move this a few lines below, so that r0 and r2 are handled nearby.

kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 514

Quoted 10 lines
kernel/bpf/backtrack.c +3 -0
  				/* r1-r5 are invalidated after subprog call,
  				 * so for global func call it shouldn't be set
  				 * anymore
@@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
  				return -ENOTSUPP;
  			/* regular helper call sets R0 */
  			bt_clear_reg(bt, BPF_REG_0);
+			/* On error the return is negative, R2 stays set, and the
+			 * check below catches it as an unexpected register.
+			 */

Nit: I don't understand the above comment (the code is fine). Maybe drop it altogether?

Quoted 9 lines
kernel/bpf/backtrack.c +3 -0
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off) > 8)
+				bt_clear_reg(bt, BPF_REG_2);
  			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
  				/* if backtracking was looking for registers R1-R5
  				 * they should have been found already.
@@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
  					return -EFAULT;
  			}

kernel/bpf/backtrack.c static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · line 539

Quoted 10 lines
kernel/bpf/backtrack.c +8 -1
  		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool r0_precise, r2_precise = false;
+
+			/* A subprog returning a value larger than 8 bytes (a
+			 * struct/union or an __int128) uses R2 as a second
+			 * return register alongside R0. When exiting such a
+			 * subprog, R2 is a return register rather than a
+			 * leftover argument, so handle it like R0 below.
+			 */

Nit: the comment is too verbose.

Quoted 14 lines
kernel/bpf/backtrack.c +11 -0
+			if (subseq_idx - 1 >= 0 &&
+			    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1])) {
+				int call_idx = subseq_idx - 1;
+				int tgt = call_idx + env->prog->insnsi[call_idx].imm + 1;
+				int subprog = bpf_find_subprog(env, tgt);
+
+				if (subprog >= 0 &&
+				    bpf_ret_reg_pair(env, subprog) &&
+				    bt_is_reg_set(bt, BPF_REG_2))
+					r2_precise = true;
+			}


  /* Backtracking to a nested function call, 'idx' is a part of
  * the inner frame 'subseq_idx' is a part of the outer frame.

[...]

author reviewer
#13 lore raw

PATCH 5/12 bpf: Account R2 of register-pair returns in live register analysis

In reply to Yonghong Song

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.

For example, for a subprogram returning an __int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):

Live regs before insn:

        0: .12345.... (85) call pc+2
        1: ..2....... (bf) r0 = r2
        2: 0......... (95) exit
        3: .......... (b7) r0 = 1
        4: 0......... (b7) r2 = 2
        5: 0.2....... (95) exit
---
 kernel/bpf/liveness.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/liveness.c +10 -3
index 0aadfbae0acc..0977d32fcf01 100644
@@ -2062,7 +2062,8 @@ struct insn_live_regs {
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   int subprog)
 {
 	struct bpf_call_summary cs;
 	u8 class = BPF_CLASS(insn->code);
@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		case BPF_EXIT:
 			def = 0;
 			use = r0;
+			if (bpf_ret_reg_pair(env, subprog))
+				use |= BIT(BPF_REG_2);
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (i = 0, j = 0; i < insn_cnt; ++i) {
+		/* Advance to the subprog that contains instruction i. */
+		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
+			j++;
+		compute_insn_live_regs(env, &insns[i], &state[i], j);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
Signature · 1 line
2.53.0-Meta
#14 lore raw

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

This looks like a bug fix for 16-byte return type support. Should this include:
Quoted 28 lines

commit 29d6a87c34e27de90ba01d71870930baf27d4b33

Author: Yonghong Song <[email protected]>
Date:   Wed Jul 8 13:10:05 2026 -0700

bpf: Account R2 of register-pair returns in live register analysis

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.

For example, for a subprogram returning an __int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):

Live regs before insn:
0: .12345.... (85) call pc+2
1: ..2....... (bf) r0 = r2
2: 0......... (95) exit
3: .......... (b7) r0 = 1
4: 0......... (b7) r2 = 2
5: 0.2....... (95) exit

Quoted 1 line

This looks like a bug fix for 16-byte return type support. Should this include:

Fixes ec219322a9a71 ("bpf: Add verifier support for 16-byte returns in R0:R2")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

maintainer
#15 lore raw

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

In reply to Yonghong Song
Quoted 54 lines · Yonghong Song

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.

For example, for a subprogram returning an __int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):

  Live regs before insn:
        0: .12345.... (85) call pc+2
        1: ..2....... (bf) r0 = r2
        2: 0......... (95) exit
        3: .......... (b7) r0 = 1
        4: 0......... (b7) r2 = 2
        5: 0.2....... (95) exit

---
 kernel/bpf/liveness.c | 13 ++++++++++---

 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/liveness.c +2 -1
index 0aadfbae0acc..0977d32fcf01 100644
@@ -2062,7 +2062,8 @@ struct insn_live_regs {
  /* Compute info->{use,def} fields for the instruction */
  static void compute_insn_live_regs(struct bpf_verifier_env *env,
  				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   int subprog)
  {
  	struct bpf_call_summary cs;
  	u8 class = BPF_CLASS(insn->code);
@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,

  case BPF_EXIT:
  def = 0;
  use = r0;

kernel/bpf/liveness.c +2 -0
+			if (bpf_ret_reg_pair(env, subprog))
+				use |= BIT(BPF_REG_2);

  break;
  case BPF_CALL:
  def = ALL_CALLER_SAVED_REGS;

kernel/bpf/liveness.c +0 -0
@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
  		goto out;
  	}

kernel/bpf/liveness.c int bpf_compute_live_registers(struct bpf_verifier_env *env) · line 2246

Quoted 6 lines
kernel/bpf/liveness.c +4 -2
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (i = 0, j = 0; i < insn_cnt; ++i) {
+		/* Advance to the subprog that contains instruction i. */
+		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
+			j++;

This loop is very in-elegant. The outer loop can be changed to iterate subprograms, with inner loop iterating instructions. Or just use bpf_find_subprog() when processing EXIT.

Quoted 5 lines
kernel/bpf/liveness.c +2 -0
+		compute_insn_live_regs(env, &insns[i], &state[i], j);
+	}


  /* Forward pass: resolve stack access through FP-derived pointers */ err = bpf_compute_subprog_arg_access(env);

author reviewer
#16 lore raw

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

Quoted 55 lines · Eduard Zingerman
Quoted 54 lines · Yonghong Song

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so the callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the R0:R2 register pair. The call site already treats the caller-saved registers R0-R5 as clobbered, so R2 as a result register is covered there.

For example, for a subprogram returning an __int128, the live-registers dump (log level 2) shows both R0 and R2 live before the callee's exit (insn 5):

  Live regs before insn:
        0: .12345.... (85) call pc+2
        1: ..2....... (bf) r0 = r2
        2: 0......... (95) exit
        3: .......... (b7) r0 = 1
        4: 0......... (b7) r2 = 2
        5: 0.2....... (95) exit

---
  kernel/bpf/liveness.c | 13 ++++++++++---
  1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/liveness.c +4 -1
index 0aadfbae0acc..0977d32fcf01 100644
@@ -2062,7 +2062,8 @@ struct insn_live_regs {
  /* Compute info->{use,def} fields for the instruction */
  static void compute_insn_live_regs(struct bpf_verifier_env *env,
  				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   int subprog)
  {
  	struct bpf_call_summary cs;
  	u8 class = BPF_CLASS(insn->code);
@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
  		case BPF_EXIT:
  			def = 0;
  			use = r0;
+			if (bpf_ret_reg_pair(env, subprog))
+				use |= BIT(BPF_REG_2);
  			break;
  		case BPF_CALL:
  			def = ALL_CALLER_SAVED_REGS;
@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
  		goto out;
  	}
 
Quoted 9 lines

kernel/bpf/liveness.c int bpf_compute_live_registers(struct bpf_verifier_env *env) · line 2246

Quoted 6 lines
kernel/bpf/liveness.c +4 -2
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (i = 0, j = 0; i < insn_cnt; ++i) {
+		/* Advance to the subprog that contains instruction i. */
+		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
+			j++;

This loop is very in-elegant. The outer loop can be changed to iterate subprograms, with inner loop iterating instructions. Or just use bpf_find_subprog() when processing EXIT.

Ack. I prefer to have iterating subprogram in outer loop and iterating insns in the inner loop.

Quoted 6 lines
Quoted 6 lines
kernel/bpf/liveness.c +2 -0
+		compute_insn_live_regs(env, &insns[i], &state[i], j);
+	}
 
  	/* Forward pass: resolve stack access through FP-derived pointers */
  	err = bpf_compute_subprog_arg_access(env);
author reviewer
#17 lore raw

PATCH 6/12 bpf: Force JIT for programs using the R0:R2 register pair

In reply to Yonghong Song

The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is set.

---
 include/linux/bpf.h   |  1 +
 kernel/bpf/core.c     |  2 +-
 kernel/bpf/verifier.c | 13 +++++++++++--
 3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/bpf.h +1 -0
index c1a98fa36738..10966aba69a2 100644
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
 	bool changes_pkt_data;
 	bool might_sleep;
 	bool kprobe_write_ctx;
+	bool ret_reg_pair;
 	struct {
 		s32 keyring_serial;
 		u8 keyring_type;
kernel/bpf/core.c +1 -1
index 73dc3ee879de..819a97d25173 100644
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +11 -2
index fcbf532159a5..ca8b2f436c20 100644
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 	return false;
 }
 
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;
+	return true;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	} else {
 		/* return to the caller whatever r0 had in the callee */
 		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
 			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
 				verbose(env, "cannot return stack pointer to the caller\n");
 				return -EINVAL;
@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 	if (err)
 		return err;
 
-	if (!bpf_ret_reg_pair(env, subprog))
+	if (!bpf_ret_reg_pair_mark(env, subprog))
 		return 0;
 
 	return check_global_ret_scalar_reg(env, BPF_REG_2);
Signature · 1 line
2.53.0-Meta
#18 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

This looks like a bug fix for the R0:R2 return convention, where the verifier accepted such programs without gating them to JIT-only. The bpf_ret_reg_pair() support that introduced this was added…
Quoted 9 lines

bpf: Force JIT for programs using the R0:R2 register pair

The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is set.

This looks like a bug fix for the R0:R2 return convention, where the verifier accepted such programs without gating them to JIT-only. The bpf_ret_reg_pair() support that introduced this was added earlier in the series. Should this include:

Fixes ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

#19 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

In reply to Yonghong Song
Quoted 43 lines · Yonghong Song

The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is set.

---
 include/linux/bpf.h   |  1 +
 kernel/bpf/core.c     |  2 +-
 kernel/bpf/verifier.c | 13 +++++++++++--
 3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/bpf.h +1 -0
index c1a98fa36738..10966aba69a2 100644
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
 	bool changes_pkt_data;
 	bool might_sleep;
 	bool kprobe_write_ctx;
+	bool ret_reg_pair;
 	struct {
 		s32 keyring_serial;
 		u8 keyring_type;
kernel/bpf/core.c +1 -1
index 73dc3ee879de..819a97d25173 100644
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +0 -0
index fcbf532159a5..ca8b2f436c20 100644

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461

Quoted 9 lines
kernel/bpf/verifier.c +6 -0
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 	return false;
 }
 
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;

With 'jit_required' introduction [1], it has no need to introduce 'ret_reg_pair'. 'env->prog->jit_required = true;' is enough.

[1]
https://lore.kernel.org/bpf/[email protected]/

Thanks,
Leon

Quoted 24 lines
kernel/bpf/verifier.c +5 -2
+	return true;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	} else {
 		/* return to the caller whatever r0 had in the callee */
 		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
 			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
 				verbose(env, "cannot return stack pointer to the caller\n");
 				return -EINVAL;
@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 	if (err)
 		return err;
 
-	if (!bpf_ret_reg_pair(env, subprog))
+	if (!bpf_ret_reg_pair_mark(env, subprog))
 		return 0;
 
 	return check_global_ret_scalar_reg(env, BPF_REG_2);
author reviewer
#20 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

Quoted 57 lines · Leon Hwang
Quoted 43 lines · Yonghong Song

The R0:R2 return convention introduced in this series is only implemented in the JIT. The BPF interpreter has no notion of a second return register, so a program relying on the R0:R2 convention would compute wrong results under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is set.

---
  include/linux/bpf.h   |  1 +
  kernel/bpf/core.c     |  2 +-
  kernel/bpf/verifier.c | 13 +++++++++++--
  3 files changed, 13 insertions(+), 3 deletions(-)
include/linux/bpf.h +1 -0
index c1a98fa36738..10966aba69a2 100644
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
  	bool changes_pkt_data;
  	bool might_sleep;
  	bool kprobe_write_ctx;
+	bool ret_reg_pair;
  	struct {
  		s32 keyring_serial;
  		u8 keyring_type;
kernel/bpf/core.c +1 -1
index 73dc3ee879de..819a97d25173 100644
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
  		goto finalize;
 
  	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
  		jit_needed = true;
 
  	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +0 -0
index fcbf532159a5..ca8b2f436c20 100644

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461

Quoted 9 lines
kernel/bpf/verifier.c +6 -0
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
  	return false;
  }
 
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;

With 'jit_required' introduction [1], it has no need to introduce 'ret_reg_pair'. 'env->prog->jit_required = true;' is enough.

Quoted 2 lines

[1]
https://lore.kernel.org/bpf/[email protected]/

Thanks! I will use jit_required in the next revision. I guess this will be useful for many other cases who do not want to support interpreter.

Quoted 27 lines

Thanks,
Leon

Quoted 24 lines
kernel/bpf/verifier.c +5 -2
+	return true;
+}
+
  static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
  {
  	struct bpf_func_info *info;
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
  	} else {
  		/* return to the caller whatever r0 had in the callee */
  		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
  			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
  				verbose(env, "cannot return stack pointer to the caller\n");
  				return -EINVAL;
@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
  	if (err)
  		return err;
 
-	if (!bpf_ret_reg_pair(env, subprog))
+	if (!bpf_ret_reg_pair_mark(env, subprog))
  		return 0;
 
  	return check_global_ret_scalar_reg(env, BPF_REG_2);
maintainer
#21 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

In reply to Yonghong Song

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2679

Quoted 10 lines
kernel/bpf/core.c +0 -0
index 73dc3ee879de..819a97d25173 100644
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
  		goto finalize;


  if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||

kernel/bpf/core.c +1 -1
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)

This has a conflict with one of the very recent bpf-next commits. While reading locally I fixed the conflict by removing this hunk and adjusting bpf_ret_reg_pair_mark() as below.

Quoted 11 lines

  jit_needed = true;

  if (!bpf_prog_select_interpreter(fp))

kernel/bpf/verifier.c +0 -0
index fcbf532159a5..ca8b2f436c20 100644
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
  	return false;
  }

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;

Added:

	env->prog->jit_required = true;

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 464

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+	return true;
+}
+

You probably would want to do the same in v2.

[...]

author reviewer
#22 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

Quoted 11 lines · Eduard Zingerman

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

Quoted 7 lines
kernel/bpf/core.c +0 -0
index 73dc3ee879de..819a97d25173 100644
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
  		goto finalize;
 
Quoted 6 lines

kernel/bpf/core.c struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct · line 2679

Quoted 3 lines
kernel/bpf/core.c +1 -1
  	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)

This has a conflict with one of the very recent bpf-next commits. While reading locally I fixed the conflict by removing this hunk and adjusting bpf_ret_reg_pair_mark() as below.

Indeed, once jit_required is available in fp->aux, I can remove ret_reg_pair things.

Quoted 27 lines
Quoted 9 lines
kernel/bpf/core.c +0 -0
  		jit_needed = true;
 
  	if (!bpf_prog_select_interpreter(fp))
kernel/bpf/verifier.c +0 -0
index fcbf532159a5..ca8b2f436c20 100644

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 461

Quoted 9 lines
kernel/bpf/verifier.c +6 -0
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
  	return false;
  }
 
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;

Added:

	env->prog->jit_required = true;

kernel/bpf/verifier.c bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) · line 464

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+	return true;
+}
+

You probably would want to do the same in v2.

[...]

maintainer
#23 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

In reply to Yonghong Song

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9889

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
  	} else {
  		/* return to the caller whatever r0 had in the callee */
  		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {

Also, doing this while processing exit is a bit strange. I think this should be done in do_check_common().

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
  			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
  				verbose(env, "cannot return stack pointer to the caller\n");
  				return -EINVAL;

[...]

author reviewer
#24 lore raw

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

Quoted 4 lines · Eduard Zingerman

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

Quoted 8 lines

kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 9889

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
  	} else {
  		/* return to the caller whatever r0 had in the callee */
  		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {

Also, doing this while processing exit is a bit strange. I think this should be done in do_check_common().

Before sending this patch, I found two places to do this, prepare_func_exit() and do_check_common(). Since you mentioned, I will do do_check_common() then.

Quoted 5 lines
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
  			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
  				verbose(env, "cannot return stack pointer to the caller\n");
  				return -EINVAL;

[...]

author reviewer
#25 lore raw

PATCH 7/12 bpf: Reject >8 byte return values on return-reading trampoline paths

In reply to Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.

Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.

---
 kernel/bpf/bpf_struct_ops.c | 12 ++++++++++++
 kernel/bpf/verifier.c       | 25 +++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
kernel/bpf/bpf_struct_ops.c +12 -0
index 51b16e5f5534..bbb44ef8f87a 100644
@@ -445,6 +445,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
 			goto errout;
 		}
 
+		/*
+		 * A >8 byte return value is passed back in the R0:R2 register
+		 * pair, which the struct_ops trampoline does not preserve (only
+		 * 8 bytes of the return value are saved and restored).
+		 */
+		if (st_ops->func_models[i].ret_size > 8) {
+			pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n",
+				mname, st_ops->name);
+			err = -EOPNOTSUPP;
+			goto errout;
+		}
+
 		stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
 		err = prepare_arg_info(btf, st_ops->name, mname,
 				       func_proto, stub_func_addr,
kernel/bpf/verifier.c +25 -0
index ca8b2f436c20..121f56e3fb16 100644
@@ -19112,6 +19112,20 @@ static int btf_id_allow_sleepable(u32 btf_id, unsigned long addr, const struct b
 	return -EINVAL;
 }
 
+static bool attach_uses_trampoline_retval(enum bpf_attach_type type)
+{
+	switch (type) {
+	case BPF_MODIFY_RETURN:
+	case BPF_TRACE_FEXIT:
+	case BPF_TRACE_FEXIT_MULTI:
+	case BPF_TRACE_FSESSION:
+	case BPF_TRACE_FSESSION_MULTI:
+		return true;
+	default:
+		return false;
+	}
+}
+
 int bpf_check_attach_target(struct bpf_verifier_log *log,
 			    const struct bpf_prog *prog,
 			    const struct bpf_prog *tgt_prog,
@@ -19376,6 +19390,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 		if (ret < 0)
 			return ret;
 
+		if (tgt_info->fmodel.ret_size > 8 &&
+		    attach_uses_trampoline_retval(prog->expected_attach_type)) {
+			bpf_log(log,
+				"Attach to function %s with a >8 byte return value is not supported for this attach type\n",
+				tname);
+			return -EOPNOTSUPP;
+		}
+
 		/*
 		 * *.multi programs don't need an address during program
 		 * verification, we just take the module ref if needed.
@@ -19652,6 +19674,9 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
 	err = btf_distill_func_proto(NULL, btf, t, tname, &tgt_info->fmodel);
 	if (err < 0)
 		return err;
+	if (tgt_info->fmodel.ret_size > 8 &&
+	    attach_uses_trampoline_retval(prog->expected_attach_type))
+		return -EOPNOTSUPP;
 	if (btf_is_module(btf)) {
 		/* The bpf program already holds reference to module. */
 		if (WARN_ON_ONCE(!prog->aux->mod))
Signature · 1 line
2.53.0-Meta
#26 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

Quoted 4 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

Do you have a plan to add >8 byte return value support for them?

Thanks,
Leon

Quoted 1 line

[...]

author reviewer
#27 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

Quoted 7 lines · Leon Hwang
Quoted 4 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

Do you have a plan to add >8 byte return value support for them?

I tested with vmlinux BTF built from the latest bpf-next. I got (from vmlinux BTF):

   total number of functions: 67500
   return type size (> 8 && <= 16): 51
   return type size (> 16): 10

So in the initial version, I didn't support >8 byte for trampoline related bpf programs.

But I can certainly do this, we just need to allocate another 8 byte for return value in trampoline.

Quoted 5 lines

Thanks,
Leon

Quoted 1 line

[...]

#28 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

Quoted 20 lines · Yonghong Song
Quoted 9 lines · Leon Hwang
Quoted 6 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
patch relaxes it to accept a >8 and <=16 byte return value, which for the
BPF target is passed back in the R0:R2 register pair.

Do you have a plan to add >8 byte return value support for them?

I tested with vmlinux BTF built from the latest bpf-next. I got (from vmlinux BTF):

   total number of functions: 67500
   return type size (> 8 && <= 16): 51
   return type size (> 16): 10

Quoted 7 lines

So in the initial version, I didn't support >8 byte for trampoline related bpf programs.

But I can certainly do this, we just need to allocate another 8 byte for return value
in trampoline.

The following support should be mentioned in the cover letter.

Thanks,
Leon

maintainer Reviewed-by on v1 7/12
#29 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

In reply to Yonghong Song
Quoted 19 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.

Quoted 8 lines

Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.

---

Maybe post this independently from the current patch-set? Seem to be a standalone fix.

#30 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

Quoted 26 lines · Eduard Zingerman
Quoted 25 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.

Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.

Quoted 8 lines
Quoted 2 lines

---

Maybe post this independently from the current patch-set? Seem to be a standalone fix.

Agreed.

A Fixes tag should be added.

Thanks,
Leon

author reviewer
#31 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

Quoted 31 lines · Leon Hwang
Quoted 30 lines · Eduard Zingerman
Quoted 19 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.

Quoted 8 lines

Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.

---

Quoted 7 lines
Quoted 3 lines

Maybe post this independently from the current patch-set? Seem to be a standalone fix.

Agreed.

A Fixes tag should be added.

Ack.

Quoted 4 lines

Thanks,
Leon

author reviewer
#32 lore raw

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

In reply to Eduard Zingerman
Quoted 26 lines · Eduard Zingerman
Quoted 25 lines · Yonghong Song

btf_distill_func_proto() builds the function model used for kfunc calls, the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following patch relaxes it to accept a >8 and <=16 byte return value, which for the BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half (RDX / R2) is not preserved across a program that observes the return value. A program attached to a function returning a 16-byte value (e.g. current_time() or ns_to_timespec64(), which return struct timespec64) would corrupt the value seen by the real caller and read a partial return value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession). fentry and fentry_multi run before the target returns and never touch the return value, so they can attach to a >8 byte-returning function safely.

Reject a >8 byte return value for the return-reading attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that would otherwise let such a return through. kfunc and BPF-to-BPF subprogram calls, which the JIT does handle, are unaffected.

Quoted 6 lines
Quoted 2 lines

---

Maybe post this independently from the current patch-set? Seem to be a standalone fix.

Okay, I will submit this patch separately.

author reviewer
#33 lore raw

PATCH 8/12 bpf: Enable 16-byte aggregate return types

In reply to Yonghong Song

Relax btf_distill_func_proto() to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:

- a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that __get_type_size() cannot return in registers at all (e.g. an array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret) and struct_ops function models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.

btf_validate_return_type() is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.

---
 include/linux/bpf_verifier.h                       |  2 ++
 kernel/bpf/btf.c                                   | 14 +++++++++++---
 kernel/bpf/verifier.c                              |  6 +++---
 .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-
 4 files changed, 17 insertions(+), 7 deletions(-)
include/linux/bpf_verifier.h +2 -0
index fa255f326240..ae4dae8e57f9 100644
@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
 int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+				 const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
kernel/bpf/btf.c +11 -3
index dff5c0d91641..bab82d1c8cb9 100644
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))
+		return 0;
+
 	return -EOPNOTSUPP;
 }
 
@@ -8059,8 +8067,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
kernel/bpf/verifier.c +3 -3
index 121f56e3fb16..bf451a2d7095 100644
@@ -11082,9 +11082,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_kfunc_call_arg_meta *meta, u3
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+				 const struct btf *btf,
+				 const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
tools/testing/selftests/bpf/progs/exceptions_fail.c +1 -1
index ac44d60e5066..9708efb93683 100644
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
Signature · 1 line
2.53.0-Meta
#34 lore raw

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. --

--- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types --- commit 5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
Author: Yonghong Song <[email protected]>
Subject: bpf: Enable 16-byte aggregate return types

This patch relaxes btf_distill_func_proto() and btf_validate_return_type() to allow kfuncs to return by-value structs or unions up to 16 bytes. This enables the <=16 byte aggregate return values end to end.

Quoted 5 lines
kernel/bpf/btf.c +0 -0
index dff5c0d916412..bab82d1c8cb98 100644

kernel/bpf/btf.c int btf_distill_func_proto(struct bpf_verifier_log *log, · line 7580

Quoted 10 lines
kernel/bpf/btf.c +6 -1
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {

[Severity: High]
Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?

The x86_64 ABI leaves the unused upper bytes of the %rdx register undefined when returning a struct that is not a multiple of 8 bytes.

Looking at mark_kfunc_ret_reg_size() in kernel/bpf/verifier.c, if the struct size is greater than 8:

	if (size > 8) {
		mark_btf_func_reg_size(env, BPF_REG_0, 8);
		mark_reg_unknown(env, regs, BPF_REG_2);
		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
	}

The verifier explicitly marks BPF_REG_2 as fully initialized, bypassing subreg zero-extension.

And in the x86 JIT do_jit() in arch/x86/net/bpf_jit_comp.c, it moves the full 64-bit register:

				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
				if (fm && fm->ret_size > 8)
					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);

Could a BPF program read the undefined bytes from R2, resulting in an information leak?

Quoted 3 lines
kernel/bpf/btf.c +0 -0
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author reviewer
#35 lore raw

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

Quoted 49 lines · On 7/8/26 1:28 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes. --

--- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types --- commit 5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
Author: Yonghong Song <[email protected]>
Subject: bpf: Enable 16-byte aggregate return types

This patch relaxes btf_distill_func_proto() and btf_validate_return_type() to allow kfuncs to return by-value structs or unions up to 16 bytes. This enables the <=16 byte aggregate return values end to end.

Quoted 5 lines
kernel/bpf/btf.c +0 -0
index dff5c0d916412..bab82d1c8cb98 100644

kernel/bpf/btf.c int btf_distill_func_proto(struct bpf_verifier_log *log, · line 7580

Quoted 10 lines
kernel/bpf/btf.c +6 -1
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
  		return -EINVAL;
  	}
  	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {

[Severity: High]
Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?

The x86_64 ABI leaves the unused upper bytes of the %rdx register undefined when returning a struct that is not a multiple of 8 bytes.

Looking at mark_kfunc_ret_reg_size() in kernel/bpf/verifier.c, if the struct size is greater than 8:

	if (size > 8) {
		mark_btf_func_reg_size(env, BPF_REG_0, 8);
		mark_reg_unknown(env, regs, BPF_REG_2);
		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
	}

The verifier explicitly marks BPF_REG_2 as fully initialized, bypassing subreg zero-extension.

And in the x86 JIT do_jit() in arch/x86/net/bpf_jit_comp.c, it moves the full 64-bit register:

Quoted 6 lines
				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
				if (fm && fm->ret_size > 8)
					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);

Could a BPF program read the undefined bytes from R2, resulting in an information leak?

We should be fine here. See llvm patch https://reviews.llvm.org/D131598
which specifically tries to address this issue. The following is an example for R2:

$ cat t.c
struct foo { long a; short b; };
struct foo bar(void);

int tar(void) {
struct foo t = bar();
return t.a + t.b;
}

$ clang --target=bpf -O2 -S t.c
$ cat t.s
        .file   "t.c"
        .text
        .globl  tar                             # -- Begin function tar
        .p2align        3
        .type   tar,@function
tar:                                    # @tar
# %bb.0:
        call bar
        w2 <<= 16
        w2 s>>= 16
        w2 += w0
        w0 = w2
        exit
.Lfunc_end0:
        .size   tar, .Lfunc_end0-tar
                                        # -- End function
        .addrsig

The caller will do proper narrowing immediately after call return.

Quoted 4 lines
Quoted 4 lines
kernel/bpf/btf.c +0 -0
  		bpf_log(log,
  			"The function %s return type %s is unsupported.\n",
  			tname, btf_type_str(t));
maintainer Acked-by on v1 8/12
#36 lore raw

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

In reply to Yonghong Song
Quoted 20 lines · Yonghong Song

Relax btf_distill_func_proto() to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:

 - a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
 - a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that __get_type_size() cannot return in registers at all (e.g. an array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret) and struct_ops function models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.

btf_validate_return_type() is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.

Quoted 7 lines

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.

---

Quoted 11 lines

 include/linux/bpf_verifier.h                       |  2 ++ kernel/bpf/btf.c                                   | 14 +++++++++++--- kernel/bpf/verifier.c                              |  6 +++--- .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-

 4 files changed, 17 insertions(+), 7 deletions(-)
include/linux/bpf_verifier.h +0 -0
index fa255f326240..ae4dae8e57f9 100644

include/linux/bpf_verifier.h void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); · line 1437

Quoted 5 lines
include/linux/bpf_verifier.h +2 -0
@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
  bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
  bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
  int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+				 const struct btf_type *t, int rec);

Nit: maybe remove the "__" prefix?

Quoted 25 lines


 int bpf_find_subprog(struct bpf_verifier_env *env, int off); bool bpf_is_throw_kfunc(struct bpf_insn *insn);

kernel/bpf/btf.c +6 -1
index dff5c0d91641..bab82d1c8cb9 100644
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
  		return -EINVAL;
  	}
  	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {
  		bpf_log(log,
  			"The function %s return type %s is unsupported.\n",
  			tname, btf_type_str(t));
@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
  	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
  		return 0;

kernel/bpf/btf.c static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt · line 7976

Quoted 1 line
kernel/bpf/btf.c +1 -0
+	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))

Nit: if (__btf_type_is_scalar_struct(env, btf, t, 0) && t->size <= 16)

Quoted 5 lines
kernel/bpf/btf.c +2 -0
+		return 0;
+

  return -EOPNOTSUPP;
 }

[...]

author reviewer
#37 lore raw

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

Quoted 36 lines · Eduard Zingerman
Quoted 20 lines · Yonghong Song

Relax btf_distill_func_proto() to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry:

 - a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target;
 - a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that __get_type_size() cannot return in registers at all (e.g. an array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret) and struct_ops function models. Allowing up to 16 bytes here is safe: the previous patch already rejects >8 byte returns on the return-reading paths, where the trampoline preserves only 8 bytes of the return value.

btf_validate_return_type() is relaxed as it accepts a by-value struct or union up to 16 bytes in addition to void and scalars.

Quoted 7 lines

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the verifier, JIT, precision-backtracking, live-register and trampoline-guard preparation from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end.

---

Quoted 6 lines
  include/linux/bpf_verifier.h                       |  2 ++
  kernel/bpf/btf.c                                   | 14 +++++++++++---
  kernel/bpf/verifier.c                              |  6 +++---
  .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-
  4 files changed, 17 insertions(+), 7 deletions(-)
Quoted 11 lines

include/linux/bpf_verifier.h void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); · line 1437

Quoted 10 lines
include/linux/bpf_verifier.h +2 -0
index fa255f326240..ae4dae8e57f9 100644
@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
  bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
  bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
  int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+				 const struct btf_type *t, int rec);

Nit: maybe remove the "__" prefix?

Good point. Since it becomes a global function, we probably should remove "__" prefix.

Quoted 26 lines
Quoted 26 lines
include/linux/bpf_verifier.h +0 -0
  int bpf_find_subprog(struct bpf_verifier_env *env, int off);
  bool bpf_is_throw_kfunc(struct bpf_insn *insn);
kernel/bpf/btf.c +6 -1
index dff5c0d91641..bab82d1c8cb9 100644
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
  		return -EINVAL;
  	}
  	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {
  		bpf_log(log,
  			"The function %s return type %s is unsupported.\n",
  			tname, btf_type_str(t));
@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
  	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
  		return 0;
 
Quoted 2 lines

kernel/bpf/btf.c static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt · line 7976

Quoted 1 line
kernel/bpf/btf.c +1 -0
+	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))

Nit: if (__btf_type_is_scalar_struct(env, btf, t, 0) && t->size <= 16)

Yes, this is simpler.

Quoted 7 lines
Quoted 6 lines
kernel/bpf/btf.c +2 -0
+		return 0;
+
  	return -EOPNOTSUPP;
  }
 

[...]

author reviewer
#38 lore raw

PATCH 9/12 selftests/bpf: Add C tests for 16-byte returns in R0:R2

In reply to Yonghong Song

Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an __int128 return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.

The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on __clang_major__ >= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.

---
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 96 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        | 48 +++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        | 98 +++++++++++++++++++
 .../bpf/progs/aggregate_ret_union_c.c         | 51 ++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 14 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        | 11 +++
 6 files changed, 318 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +96 -0
new file mode 100644
index 000000000000..d4ecc19684ce
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_union_c.skel.h"
+
+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */
+static void run_prog(struct bpf_program *prog, bool may_skip)
+{
+	char buf[64] = {};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = buf,
+		.data_size_in = sizeof(buf),
+		.repeat = 1,
+	);
+	int err;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+	if (!ASSERT_OK(err, "test_run"))
+		return;
+
+	if (may_skip && (int)topts.retval == -1) {
+		test__skip();
+		return;
+	}
+
+	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
+}
+
+void test_aggregate_ret(void)
+{
+	struct aggregate_ret_struct_c *skel_struct_c;
+	struct aggregate_ret_union_c *skel_union_c;
+	struct aggregate_ret_int128_c *skel_c;
+
+	skel_c = aggregate_ret_int128_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_c, "skel_c_open_load"))
+		return;
+
+	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
+	if (test__start_subtest("int128_c"))
+		run_prog(skel_c->progs.aggregate_ret_test, true);
+
+	aggregate_ret_int128_c__destroy(skel_c);
+
+	skel_struct_c = aggregate_ret_struct_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte struct by value: exercises the struct
+	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
+	 */
+	if (test__start_subtest("struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct return, but from a global (non-static) callee:
+	 * exercises the compiler lowering a global struct-returning function to
+	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
+	 * only.
+	 */
+	if (test__start_subtest("global_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct, but returned by value from a kfunc: exercises
+	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("kfunc_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);
+
+	aggregate_ret_struct_c__destroy(skel_struct_c);
+
+	skel_union_c = aggregate_ret_union_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_union_c, "skel_union_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte union by value: exercises the union BTF
+	 * kind through the same struct branch of bpf_ret_reg_pair().
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("union_c"))
+		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
+
+	aggregate_ret_union_c__destroy(skel_union_c);
+}
tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c new file +48 -0

48 lines not on this page · raw message

tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c new file +98 -0

98 lines not on this page · raw message

tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c new file +51 -0

51 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14 -0
index 30f1cd23093c..3511a4873d6c 100644
@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return ((__int128)a << 64) | b;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b)
+{
+	struct bpf_kfunc_ret_ll r = { .a = a, .b = b };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +11 -0
index c36bb911defa..34d0a352e70d 100644
@@ -55,6 +55,15 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+/*
+ * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
+ * register pair.
+ */
+struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	__u64 b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
Signature · 1 line
2.53.0-Meta
#39 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Unconditional use of `__int128` in `bpf_testmod` breaks compilation on 32-bit host architectures. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional use of __int128 in bpf_testmod breaks compilation on 32-bit host architectures. --

--- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 --- commit 5ed3844739f04c920bb1c37cdf474501b3ca9fef
Author: Yonghong Song <[email protected]>

selftests/bpf: Add C tests for 16-byte returns in R0:R2

This commit adds selftests to exercise 16-byte return values passed in the R0:R2 register pair. The tests include an __int128 return, a 16-byte struct return, and a 16-byte union return.

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0 -0

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) · line 831

Quoted 8 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4 -0
@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return ((__int128)a << 64) | b;
+}

[Severity: Low]
Will this unconditional use of __int128 break the build on 32-bit host architectures?

The __int128 type is a compiler extension that is typically only available on 64-bit systems.

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c BTF_ID_FLAGS(func, bpf_kfunc_call_test2) · line 1373

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 -0
@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)

[Severity: Low]
If bpf_kfunc_call_test_i128 needs to be conditionally excluded for 32-bit builds to fix the issue above, should this BTF ID registration also be conditionally excluded to prevent a missing symbol error during the build?

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0 -0

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym; · line 130

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1 -0
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;

[Severity: Low]
Does this declaration also need to be guarded for 64-bit architectures, so that including this header doesn't fail on 32-bit host compilers?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author reviewer
#40 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

Quoted 55 lines · On 7/8/26 1:24 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional use of __int128 in bpf_testmod breaks compilation on 32-bit host architectures. --

--- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 --- commit 5ed3844739f04c920bb1c37cdf474501b3ca9fef
Author: Yonghong Song <[email protected]>

selftests/bpf: Add C tests for 16-byte returns in R0:R2

This commit adds selftests to exercise 16-byte return values passed in the R0:R2 register pair. The tests include an __int128 return, a 16-byte struct return, and a 16-byte union return.

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0 -0

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) · line 831

Quoted 8 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +4 -0
@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
  	return 0;
  }
 
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return ((__int128)a << 64) | b;
+}

[Severity: Low]
Will this unconditional use of __int128 break the build on 32-bit host architectures?

The __int128 type is a compiler extension that is typically only available on 64-bit systems.

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c BTF_ID_FLAGS(func, bpf_kfunc_call_test2) · line 1373

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 -0
@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
  BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
  BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
  BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)

[Severity: Low]
If bpf_kfunc_call_test_i128 needs to be conditionally excluded for 32-bit builds to fix the issue above, should this BTF ID registration also be conditionally excluded to prevent a missing symbol error during the build?

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0 -0

[ ... ]

Quoted 8 lines

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym; · line 130

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1 -0
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
  struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
  long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
  int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;

[Severity: Low]
Does this declaration also need to be guarded for 64-bit architectures, so that including this header doesn't fail on 32-bit host compilers?

We may need to guard with __SIZEOF_INT128__ in selftests. But current CI all supports 64bit, so I am not sure whether we should do it or not.

#41 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

In reply to Yonghong Song
this isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the run_prog() comment above already states that programs built without R0:R2 support…
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +0 -0
new file mode 100644
index 0000000000000..d4ecc19684cef
@@ -0,0 +1,96 @@

[ ... ]

Quoted 6 lines

+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */

[ ... ]

Quoted 23 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +23 -0
+	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
+	if (test__start_subtest("int128_c"))
+		run_prog(skel_c->progs.aggregate_ret_test, true);
+
+	aggregate_ret_int128_c__destroy(skel_c);
+
+	skel_struct_c = aggregate_ret_struct_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte struct by value: exercises the struct
+	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
+	 */
+	if (test__start_subtest("struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct return, but from a global (non-static) callee:
+	 * exercises the compiler lowering a global struct-returning function to
+	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
+	 * only.
+	 */

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 79

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10 -0
+	if (test__start_subtest("global_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct, but returned by value from a kfunc: exercises
+	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("kfunc_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);

this isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the run_prog() comment above already states that programs built without R0:R2 support return -1 and get skipped. Could that note be stated once rather than on every block?

Related, do the per-subtest comments need to name the specific internals (bpf_ret_reg_pair(), check_kfunc_call(), "the JIT RDX->R2 move"), or would a shorter note serve here?

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 62

Quoted 13 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5 -0
index c36bb911defa7..34d0a352e70dd 100644
@@ -55,6 +55,15 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+/*
+ * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
+ * register pair.
+ */
+struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */

this isn't a bug, but the block comment already says the struct is 16 bytes and returned in the R0:R2 register pair, and the trailing /* 16 bytes: R0:R2 */ on the next line repeats that. Could one of the two be dropped?

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 65

Quoted 3 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +3 -0
+	__u64 a;
+	__u64 b;
+};

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

author reviewer
#42 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

Quoted 45 lines · On 7/8/26 2:10 PM, [email protected] wrote:
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +0 -0
new file mode 100644
index 0000000000000..d4ecc19684cef
@@ -0,0 +1,96 @@

[ ... ]

Quoted 6 lines

+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */

[ ... ]

Quoted 29 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +29 -0
+	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
+	if (test__start_subtest("int128_c"))
+		run_prog(skel_c->progs.aggregate_ret_test, true);
+
+	aggregate_ret_int128_c__destroy(skel_c);
+
+	skel_struct_c = aggregate_ret_struct_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte struct by value: exercises the struct
+	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
+	 */
+	if (test__start_subtest("struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct return, but from a global (non-static) callee:
+	 * exercises the compiler lowering a global struct-returning function to
+	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
+	 * only.
+	 */
+	if (test__start_subtest("global_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct, but returned by value from a kfunc: exercises
+	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
Quoted 8 lines

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 79

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +4 -0
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("kfunc_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);

this isn't a bug, but the "Also LLVM 23 only." note is repeated on each of these per-subtest comments, and the run_prog() comment above already states that programs built without R0:R2 support return -1 and get skipped. Could that note be stated once rather than on every block?

Ack. Will do.

Quoted 4 lines

Related, do the per-subtest comments need to name the specific internals (bpf_ret_reg_pair(), check_kfunc_call(), "the JIT RDX->R2 move"), or would a shorter note serve here?

I guess I can rename these comments, function names themselves already have enough information.

Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +0 -0
index c36bb911defa7..34d0a352e70dd 100644
@@ -55,6 +55,15 @@ struct prog_test_big_arg {
  	__u64 b;
  };
 
Quoted 9 lines

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 62

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +5 -0
+/*
+ * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
+ * register pair.
+ */
+struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */

this isn't a bug, but the block comment already says the struct is 16 bytes and returned in the R0:R2 register pair, and the trailing /* 16 bytes: R0:R2 */ on the next line repeats that. Could one of the two be dropped?

Yes, will drop.

Quoted 10 lines

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h struct prog_test_big_arg { · line 65

Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +3 -0
+	__u64 a;
+	__u64 b;
+};

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

maintainer
#43 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

In reply to Yonghong Song
Quoted 13 lines · Yonghong Song

Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an __int128 return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.

The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on __clang_major__ >= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.

---

[...]

Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +14 -0
+static void run_prog(struct bpf_program *prog, bool may_skip)
+{
+	char buf[64] = {};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = buf,
+		.data_size_in = sizeof(buf),
+		.repeat = 1,
+	);
+	int err;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+	if (!ASSERT_OK(err, "test_run"))
+		return;
+

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 38

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10 -0
+	if (may_skip && (int)topts.retval == -1) {
+		test__skip();
+		return;
+	}
+
+	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
+}
+
+void test_aggregate_ret(void)
+{

The test_aggregate_ret()/run_prog() duplicate functionality provided by the __retval tag from test_loader.c. To achieve the same skipping mechanics I suggest to:
- extend test_loader.c / bpf_misc.h with a new tag: __skip("reason")
- use this tag as:

  #ifdef __clang_major__ >= 23

SEC("tc")
int aggregate_ret_test(struct __sk_buff *skb) { ... }

  #else

SEC("tc") __skip("... need clang23 ...")
int aggregate_ret_test(struct __sk_buff *skb) { ... }

  #endif

[...]

...

Quoted 3 lines
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c +0 -0
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c +0 -0
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c +0 -0

Nit: all three files can be merged into one.

[...]

author reviewer
#44 lore raw

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

Quoted 48 lines · Eduard Zingerman
Quoted 13 lines · Yonghong Song

Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, expressed in C so they rely on the compiler lowering the register-pair return. Tests include an __int128 return (from a subprogram and a kfunc), a 16-byte struct return (from static and global subprograms and a kfunc) and a 16-byte union return.

The R0:R2 return convention is only emitted by LLVM 23 and newer, so the programs are gated on __clang_major__ >= 23; with an older compiler they return a sentinel and the "int128_c", "struct_c", "global_struct_c", "kfunc_struct_c" and "union_c" subtests are skipped.

---

[...]

Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +14 -0
+static void run_prog(struct bpf_program *prog, bool may_skip)
+{
+	char buf[64] = {};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = buf,
+		.data_size_in = sizeof(buf),
+		.repeat = 1,
+	);
+	int err;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+	if (!ASSERT_OK(err, "test_run"))
+		return;
+

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c line 38

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file +10 -0
+	if (may_skip && (int)topts.retval == -1) {
+		test__skip();
+		return;
+	}
+
+	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
+}
+
+void test_aggregate_ret(void)
+{

The test_aggregate_ret()/run_prog() duplicate functionality provided by the __retval tag from test_loader.c. To achieve the same skipping mechanics I suggest to:
- extend test_loader.c / bpf_misc.h with a new tag: __skip("reason")
- use this tag as:

   #ifdef __clang_major__ >= 23
Quoted 9 lines
   SEC("tc")
   int aggregate_ret_test(struct __sk_buff *skb) { ... }

   #else

   SEC("tc") __skip("... need clang23 ...")
   int aggregate_ret_test(struct __sk_buff *skb) { ... }

   #endif

Ack. We have quite some cases like this (guarded by compiler or feature flag, with __skip makes things simpler.

Quoted 7 lines

[...]

...

Quoted 3 lines
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c +0 -0
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c +0 -0
... renamed from tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c +0 -0

Nit: all three files can be merged into one.

Yes, I can merge them.

Quoted 2 lines

[...]

author reviewer
#45 lore raw

PATCH 10/12 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

In reply to Yonghong Song

Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.

---
 .../selftests/bpf/prog_tests/aggregate_ret.c  |  25 ++
 .../selftests/bpf/progs/aggregate_ret_func.c  | 361 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |  81 ++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 160 ++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  34 ++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  27 ++
 6 files changed, 688 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +25 -0
index d4ecc19684ce..2f1e5c704959 100644
@@ -5,6 +5,9 @@
 #include "aggregate_ret_int128_c.skel.h"
 #include "aggregate_ret_struct_c.skel.h"
 #include "aggregate_ret_union_c.skel.h"
+#include "aggregate_ret_run.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
 
 /*
  * Run one program and check its result. The program returns 0 on success, or
@@ -39,6 +42,7 @@ void test_aggregate_ret(void)
 	struct aggregate_ret_struct_c *skel_struct_c;
 	struct aggregate_ret_union_c *skel_union_c;
 	struct aggregate_ret_int128_c *skel_c;
+	struct aggregate_ret_run *skel_run;
 
 	skel_c = aggregate_ret_int128_c__open_and_load();
 	if (!ASSERT_OK_PTR(skel_c, "skel_c_open_load"))
@@ -93,4 +97,25 @@ void test_aggregate_ret(void)
 		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
 
 	aggregate_ret_union_c__destroy(skel_union_c);
+
+	skel_run = aggregate_ret_run__open_and_load();
+	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
+		return;
+
+	/* Inline-asm variant: compiler-version independent, always runs. */
+	if (test__start_subtest("asm"))
+		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
+
+	/* Struct-by-value kfunc returns, also compiler-version independent. */
+	if (test__start_subtest("struct"))
+		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
+
+	/* Union-by-value kfunc return, compiler-version independent. */
+	if (test__start_subtest("union"))
+		run_prog(skel_run->progs.aggregate_ret_union_test, false);
+
+	aggregate_ret_run__destroy(skel_run);
+
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
 }
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +361 -0

361 lines not on this page · raw message

tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file +81 -0

81 lines not on this page · raw message

tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +160 -0

160 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +34 -0
index 3511a4873d6c..c9e211d8a719 100644
@@ -837,6 +837,36 @@ __bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b)
 	return r;
 }
 
+__bpf_kfunc struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+	struct bpf_kfunc_ret_li r = { .a = a, .b = b };
+
+	return r;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct bpf_kfunc_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+
+__bpf_kfunc union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+	union bpf_kfunc_ret_uu r;
+
+	r.halves[0] = a;
+	r.halves[1] = b;
+	return r;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct bpf_kfunc_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1372,6 +1402,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +27 -0
index 34d0a352e70d..9ac31d68e2c8 100644
@@ -64,6 +64,29 @@ struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */
 	__u64 b;
 };
 
+struct bpf_kfunc_ret_li {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	int b;
+};
+
+struct bpf_kfunc_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+union bpf_kfunc_ret_uu {	/* 16 bytes: R0:R2 */
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct bpf_kfunc_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -129,6 +152,10 @@ long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
 __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
 struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
Signature · 1 line
2.53.0-Meta
#46 lore raw

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Hardcoded little-endian assumptions in struct and `__int128` register unpacking will cause…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoded little-endian assumptions in struct and __int128 register unpacking will cause selftest failures on big-endian architectures like s390x. --

--- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns --- commit dcf423ead16608630f40807ad0cde33d495c941d
Author: Yonghong Song <[email protected]>

selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.

Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0 -0
new file mode 100644
index 0000000000000..d6300b31733d8

[ ... ]

Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0 -0
@@ -0,0 +1,160 @@

[ ... ]

Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +6 -0
+SEC("tc")
+int aggregate_ret_asm_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4 -0
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 73

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (hi != a)
+		return 3;
+	if (lo != b)
+		return 4;

[Severity: Medium]
The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in r0. On a big-endian ABI, the most significant 64 bits (a) will be returned in the first register (r0).

Will this cause the test to fail on big-endian architectures like s390x?

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_struct_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +5 -0
+	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 115

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a)
+		return 3;
+	if ((int)hi != (int)b)
+		return 4;

[Severity: Medium]
The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32 bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of the register, causing this assertion to fail?

Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +5 -0
+
+	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 130

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	: [lo]"=r"(lo)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if ((int)lo != (int)a)
+		return 5;
+	if ((int)(lo >> 32) != (int)b)
+		return 6;

[Severity: Medium]
The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian register layout for the returned 8-byte struct. On big-endian, wouldn't a be in the upper 32 bits and b in the lower 32 bits?

Quoted 3 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +3 -0
+
+	return 0;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author reviewer
#47 lore raw

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Quoted 51 lines · On 7/8/26 1:28 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoded little-endian assumptions in struct and __int128 register unpacking will cause selftest failures on big-endian architectures like s390x. --

--- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns --- commit dcf423ead16608630f40807ad0cde33d495c941d
Author: Yonghong Song <[email protected]>

selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.

Quoted 5 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0 -0
new file mode 100644
index 0000000000000..d6300b31733d8

[ ... ]

Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +0 -0
@@ -0,0 +1,160 @@

[ ... ]

Quoted 6 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +6 -0
+SEC("tc")
+int aggregate_ret_asm_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4 -0
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 73

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (hi != a)
+		return 3;
+	if (lo != b)
+		return 4;

[Severity: Medium]
The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in r0. On a big-endian ABI, the most significant 64 bits (a) will be returned in the first register (r0).

Quoted 1 line

Will this cause the test to fail on big-endian architectures like s390x?

Good point. I need to test it for s390x.

Quoted 24 lines
Quoted 11 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +10 -0
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_struct_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;

[ ... ]

Quoted 11 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +11 -0
+	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
Quoted 8 lines

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 115

Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4 -0
+	if (lo != a)
+		return 3;
+	if ((int)hi != (int)b)
+		return 4;

[Severity: Medium]
The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32 bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of the register, causing this assertion to fail?

The same as above. Need tests with s390x.

Quoted 12 lines
Quoted 12 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +11 -0
+
+	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	: [lo]"=r"(lo)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
Quoted 8 lines

tools/testing/selftests/bpf/progs/aggregate_ret_run.c line 130

Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +4 -0
+	if ((int)lo != (int)a)
+		return 5;
+	if ((int)(lo >> 32) != (int)b)
+		return 6;

[Severity: Medium]
The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian register layout for the returned 8-byte struct. On big-endian, wouldn't a be in the upper 32 bits and b in the lower 32 bits?

Will test as the above.

Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/bpf/progs/aggregate_ret_run.c new file +3 -0
+
+	return 0;
+}
maintainer
#48 lore raw

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

In reply to Yonghong Song
Quoted 6 lines · Yonghong Song

Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.

---

[...]

Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +0 -0
@@ -93,4 +97,25 @@ void test_aggregate_ret(void)
  		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);

aggregate_ret_union_c__destroy(skel_union_c);

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +11 -0
+
+	skel_run = aggregate_ret_run__open_and_load();
+	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
+		return;
+
+	/* Inline-asm variant: compiler-version independent, always runs. */
+	if (test__start_subtest("asm"))
+		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
+
+	/* Struct-by-value kfunc returns, also compiler-version independent. */
+	if (test__start_subtest("struct"))

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c void test_aggregate_ret(void) · line 120

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +10 -0
+		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
+
+	/* Union-by-value kfunc return, compiler-version independent. */
+	if (test__start_subtest("union"))
+		run_prog(skel_run->progs.aggregate_ret_union_test, false);
+
+	aggregate_ret_run__destroy(skel_run);
+
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);

This can be folded into tools/testing/selftests/bpf/prog_tests/verifier.c.

Quoted 21 lines

 }

tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +14 -0
new file mode 100644
index 000000000000..31e176103374
@@ -0,0 +1,361 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 24

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10 -0
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}

C version of tests from a previous patch already checks global vs. local functions. I suggest dropping that version from C tests and keep in assembly only to reduce the duplication.

[...]

Quoted 9 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +9 -0
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_union(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_union];"
+	"%[lo] = r0;"

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 284

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10 -0
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_union)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}

And here I'd only keep the C version.

Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +8 -0
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}

[...]

Quoted 18 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +18 -0
+static __naked u128 agg_callee(void)
+{
+	asm volatile (
+	"r0 = 1;"
+	"r2 = 2;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__log_level(2)
+__msg("Live regs before insn:")
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 359

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10 -0
+__naked int aggregate_ret_live(void)
+{
+	asm volatile (
+	"call %[agg_callee];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: [agg_callee]"i"(agg_callee)
+	: __clobber_all);
+}

Nit: usually we add such tests to compute_live_registers.c.

[...]

tools/testing/selftests/bpf/progs/aggregate_ret_run.c

Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c +0 -0

I think the test cases in this file are by a large degree redundant. A single test case calling a kfunc returning a small struct and checking returned values would suffice.

[...]

author reviewer
#49 lore raw

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Quoted 125 lines · Eduard Zingerman
Quoted 6 lines · Yonghong Song

Add inline-asm and subprogram tests for more coverage, including BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive and negative tests are added.

---

[...]

Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +11 -0
@@ -93,4 +97,25 @@ void test_aggregate_ret(void)
  		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
 
  	aggregate_ret_union_c__destroy(skel_union_c);
+
+	skel_run = aggregate_ret_run__open_and_load();
+	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
+		return;
+
+	/* Inline-asm variant: compiler-version independent, always runs. */
+	if (test__start_subtest("asm"))
+		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
+
+	/* Struct-by-value kfunc returns, also compiler-version independent. */
+	if (test__start_subtest("struct"))

tools/testing/selftests/bpf/prog_tests/aggregate_ret.c void test_aggregate_ret(void) · line 120

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +10 -0
+		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
+
+	/* Union-by-value kfunc return, compiler-version independent. */
+	if (test__start_subtest("union"))
+		run_prog(skel_run->progs.aggregate_ret_union_test, false);
+
+	aggregate_ret_run__destroy(skel_run);
+
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);

This can be folded into tools/testing/selftests/bpf/prog_tests/verifier.c.

Quoted 21 lines
(quoted diff) +0 -0
  }
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +14 -0
new file mode 100644
index 000000000000..31e176103374
@@ -0,0 +1,361 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 24

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10 -0
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}

C version of tests from a previous patch already checks global vs. local functions. I suggest dropping that version from C tests and keep in assembly only to reduce the duplication.

[...]

Quoted 9 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +9 -0
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_union(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_union];"
+	"%[lo] = r0;"

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 284

Quoted 10 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +10 -0
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_union)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}

And here I'd only keep the C version.

Quoted 8 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +8 -0
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}

[...]

Quoted 21 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +21 -0
+static __naked u128 agg_callee(void)
+{
+	asm volatile (
+	"r0 = 1;"
+	"r2 = 2;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__log_level(2)
+__msg("Live regs before insn:")
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")
+__naked int aggregate_ret_live(void)
+{
+	asm volatile (
Quoted 8 lines

tools/testing/selftests/bpf/progs/aggregate_ret_func.c line 359

Quoted 7 lines
tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file +7 -0
+	"call %[agg_callee];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: [agg_callee]"i"(agg_callee)
+	: __clobber_all);
+}

Nit: usually we add such tests to compute_live_registers.c.

I can move the test to compute_live_registers.c.

Quoted 7 lines

[...]

tools/testing/selftests/bpf/progs/aggregate_ret_run.c

Quoted 1 line
tools/testing/selftests/bpf/progs/aggregate_ret_run.c +0 -0

I think the test cases in this file are by a large degree redundant. A single test case calling a kfunc returning a small struct and checking returned values would suffice.

Ack. I tried to test for both C and asm test cases. They are indeed some duplicates. I will try to remove the redundant ones in next revision.

Quoted 2 lines

[...]

author reviewer
#50 lore raw

PATCH 11/12 selftests/bpf: Cover tracing on >8 and <=16 byte return targets

In reply to Yonghong Song

Tracing attach types differ in whether they need the target's return value preserved across the BPF trampoline, which saves and restores only 8 bytes of it. fentry runs before the target returns and never observes the return value, so it can attach to a function returning a >8 byte value; fexit, fmod_ret and fsession read the return value and must keep rejecting such a target.

Add fentry, fexit, fentry_multi and fexit_multi tests that attach to functions returning a >8 and <=16 byte value (a 16-byte struct and an __int128).

---
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 128 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_trace.c |  44 ++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  32 +++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   5 +
 4 files changed, 209 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
tools/testing/selftests/bpf/prog_tests/aggregate_ret.c +128 -0
index 2f1e5c704959..7caf815a2c06 100644
@@ -8,6 +8,7 @@
 #include "aggregate_ret_run.skel.h"
 #include "aggregate_ret_func.skel.h"
 #include "aggregate_ret_kfunc.skel.h"
+#include "aggregate_ret_trace.skel.h"
 
 /*
  * Run one program and check its result. The program returns 0 on success, or
@@ -37,6 +38,131 @@ static void run_prog(struct bpf_program *prog, bool may_skip)
 	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
 }
 
+/*
+ * Multi-attach (BPF_TRACE_*_MULTI) requires CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS,
+ * which today only x86_64 selects. Otherwise bpf_tracing_multi_attach() returns
+ * -EOPNOTSUPP unconditionally.
+ */
+static bool tracing_multi_supported(struct bpf_program *prog)
+{
+	const char *small_pat = "bpf_testmod:bpf_testmod_trampoline_count_test";
+	struct bpf_link *link;
+
+	link = bpf_program__attach_tracing_multi(prog, small_pat, NULL);
+	if (libbpf_get_error(link) == -EOPNOTSUPP)
+		return false;
+	bpf_link__destroy(link);
+	return true;
+}
+
+static void test_tracing_attach_types(void)
+{
+	const char *i128_pat = "bpf_testmod:bpf_testmod_aggregate_ret_i128_fn";
+	int err;
+
+	if (test__start_subtest("trace_fentry")) {
+		struct aggregate_ret_trace *trace = NULL;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fentry_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		err = aggregate_ret_trace__load(trace);
+		if (!ASSERT_OK(err, "trace_fentry_load"))
+			goto destroy_trace_fentry;
+
+		err = aggregate_ret_trace__attach(trace);
+		if (!ASSERT_OK(err, "trace_fentry_attach"))
+			goto destroy_trace_fentry;
+
+		ASSERT_OK(trigger_module_test_read(1), "trace_fentry_trigger");
+		ASSERT_EQ(trace->bss->fentry_hit, 1, "trace_fentry_hit");
+
+destroy_trace_fentry:
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fexit_reject")) {
+		struct aggregate_ret_trace *trace = NULL;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fexit_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		err = aggregate_ret_trace__load(trace);
+		ASSERT_EQ(err, -EOPNOTSUPP, "trace_fexit_load");
+
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fentry_multi")) {
+		struct aggregate_ret_trace *trace = NULL;
+		struct bpf_link *link;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fentry_multi_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		if (!ASSERT_OK(aggregate_ret_trace__load(trace), "trace_fentry_multi_load"))
+			goto destroy_fentry_multi;
+
+		if (!tracing_multi_supported(trace->progs.fentry_multi)) {
+			test__skip();
+			goto destroy_fentry_multi;
+		}
+
+		link = bpf_program__attach_tracing_multi(trace->progs.fentry_multi, i128_pat, NULL);
+		if (!ASSERT_OK_PTR(link, "trace_fentry_multi_attach"))
+			goto destroy_fentry_multi;
+
+		ASSERT_OK(trigger_module_test_read(1), "trace_fentry_multi_trigger");
+		ASSERT_EQ(trace->bss->fentry_multi_hit, 1, "trace_fentry_multi_hit");
+
+		bpf_link__destroy(link);
+destroy_fentry_multi:
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fexit_multi_reject")) {
+		struct aggregate_ret_trace *trace = NULL;
+		struct bpf_link *link;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fexit_multi_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+
+		if (!ASSERT_OK(aggregate_ret_trace__load(trace), "trace_fexit_multi_load"))
+			goto destroy_fexit_multi;
+
+		if (!tracing_multi_supported(trace->progs.fexit_multi)) {
+			test__skip();
+			goto destroy_fexit_multi;
+		}
+
+		link = bpf_program__attach_tracing_multi(trace->progs.fexit_multi, i128_pat, NULL);
+		ASSERT_EQ(libbpf_get_error(link), -EOPNOTSUPP, "trace_fexit_multi_reject");
+		bpf_link__destroy(link);
+destroy_fexit_multi:
+		aggregate_ret_trace__destroy(trace);
+	}
+}
+
 void test_aggregate_ret(void)
 {
 	struct aggregate_ret_struct_c *skel_struct_c;
@@ -118,4 +244,6 @@ void test_aggregate_ret(void)
 
 	RUN_TESTS(aggregate_ret_func);
 	RUN_TESTS(aggregate_ret_kfunc);
+
+	test_tracing_attach_types();
 }
tools/testing/selftests/bpf/progs/aggregate_ret_trace.c new file +44 -0

44 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +32 -0
index c9e211d8a719..b558c538400e 100644
@@ -469,12 +469,34 @@ noinline void bpf_testmod_stacktrace_test_1(void)
 }
 
 int bpf_testmod_fentry_ok;
+u64 bpf_testmod_aggregate_ret_seen;
 
 noinline int bpf_testmod_trampoline_count_test(void)
 {
 	return 0;
 }
 
+__noclone noinline struct bpf_testmod_aggregate_ret
+bpf_testmod_aggregate_ret_fn(u64 lo, u64 hi)
+{
+	struct bpf_testmod_aggregate_ret ret = {
+		.lo = lo,
+		.hi = hi,
+	};
+
+	bpf_testmod_aggregate_ret_seen += lo + hi;
+	return ret;
+}
+
+__noclone noinline __int128
+bpf_testmod_aggregate_ret_i128_fn(u64 lo, u64 hi)
+{
+	__int128 ret = ((__int128)hi << 64) | lo;
+
+	bpf_testmod_aggregate_ret_seen += lo + hi;
+	return ret;
+}
+
 noinline ssize_t
 bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 		      const struct bin_attribute *bin_attr,
@@ -492,6 +514,8 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 	struct bpf_testmod_struct_arg_5 struct_arg5 = {23, 24, 25, 26};
 	union bpf_testmod_union_arg_1 union_arg1 = { .arg = {1} };
 	union bpf_testmod_union_arg_2 union_arg2 = { .arg = {2, 3} };
+	struct bpf_testmod_aggregate_ret agg_ret;
+	__int128 agg_i128;
 	int i = 1;
 
 	while (bpf_testmod_return_ptr(i))
@@ -534,6 +558,14 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 
 	trace_bpf_testmod_test_nullable_bare_tp(NULL);
 
+	agg_ret = bpf_testmod_aggregate_ret_fn(len, off);
+	if (agg_ret.lo == U64_MAX && agg_ret.hi == U64_MAX)
+		goto out;
+
+	agg_i128 = bpf_testmod_aggregate_ret_i128_fn(len, off);
+	if (agg_i128 == (__int128)-1)
+		goto out;
+
 	/* Magic number to enable writable tp */
 	if (len == 64) {
 		struct bpf_testmod_test_writable_ctx writable = {
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +5 -0
index 863fd10f1619..4dcbfd64de62 100644
@@ -25,6 +25,11 @@ struct bpf_testmod_test_writable_ctx {
 	int val;
 };
 
+struct bpf_testmod_aggregate_ret {
+	__u64 lo;
+	__u64 hi;
+};
+
 /* BPF iter that returns *value* *n* times in a row */
 struct bpf_iter_testmod_seq {
 	s64 value;
Signature · 1 line
2.53.0-Meta
author reviewer
#51 lore raw

PATCH 12/12 Documentation/bpf: Document 16-byte kfunc return values in R0:R2

In reply to Yonghong Song

kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a 16-byte struct or union, or an __int128), passed back in the R0:R2 register pair. Add a kfunc return-value section documenting this, including that a struct or union up to 8 bytes is returned in R0 alone, that the R0:R2 register pair requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a return value larger than 16 bytes is unsupported.

---
 Documentation/bpf/kfuncs.rst | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
Documentation/bpf/kfuncs.rst +36 -0
index c801a330aece..be4f489a9f16 100644
@@ -501,6 +501,42 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively). Its bytes are handed back to the program as the raw contents
+of R0 (and R2), so a pointer field would otherwise be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. Such a
+struct or union is therefore rejected at load time.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+16-byte scalar-only struct or union, or an ``__int128``. Such a value is
+returned in the register pair R0:R2, matching the convention LLVM uses for
+the BPF target: the low 64 bits in R0 and the high 64 bits in R2. A struct
+or union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
Signature · 1 line
2.53.0-Meta
maintainer
#52 lore raw
In reply to Yonghong Song
Quoted 1 line · Yonghong Song

A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.

I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.

I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.

Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT
- forgo the scalar struct requirement for local function return types.

Wdyt?

author reviewer
#53 lore raw
Quoted 12 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song

A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.

I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.

Quoted 6 lines

I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.

Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT

Not sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?

Quoted 2 lines

- forgo the scalar struct requirement for local function return types.

I think this is doable.

Quoted 2 lines

Wdyt?

maintainer
#54 lore raw
Quoted 18 lines · Yonghong Song
Quoted 15 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song

A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.

I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.

I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.

Quoted 8 lines
Quoted 3 lines

Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT

Not sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?

We can detect this statically from compute_live_registers().
If any r2's are alive after a function call the jit_required flag can be set.

Quoted 7 lines
Quoted 2 lines

- forgo the scalar struct requirement for local function return types.

I think this is doable.

Quoted 2 lines

Wdyt?

author reviewer
#55 lore raw
Quoted 21 lines · Eduard Zingerman
Quoted 20 lines · Yonghong Song
Quoted 18 lines · Eduard Zingerman
Quoted 1 line · Yonghong Song

A design question regarding to how this series handles local functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part of an aggregate return.

I think this unnecessarily adds an additional discrepancy between how the return values are handled. The practical experience shows that existing differences between local and global functions are already a constant source of pain and confusion.

I'd also argue that lifting the above limitations for local functions would make the implementation somewhat simpler.

Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT

Quoted 6 lines
Quoted 4 lines

Not sure about this. If r2 is always copied from callee to caller, and interpreter still only supports return r0, then this will imply interpreter will never be used? I know interpreter will be eventually removed, but at this point, people still use interpreter?

We can detect this statically from compute_live_registers().
If any r2's are alive after a function call the jit_required flag can be set.

Sounds good, I will do some investigation on this.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

- forgo the scalar struct requirement for local function return types.

I think this is doable.

Quoted 1 line

Wdyt?