Skip to contentSkip to conversation

Add arena argument support to kfuncs and struct_ops

PATCH Superseded Kumar Kartikeya Dwivedi 5 participants 32 messages Aug 5 – Aug 7

Subsystems (8)
Documentation x86 Kselftest BPF +4 more subsystems

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v4 cover Add arena argument support to kfuncs and struct_ops

This is a continuation of patches in [0], with mostly minor changes and reordering. The motivation is covered in that link. A major change is moving to two tags (__arena and __arena__nullable) and moving the changes to JIT to emit more optimized sequences.

Please see commit logs for details.

[0]: https://lore.kernel.org/bpf/[email protected]

Changelog:
----------
v3 -> v4
v3: https://lore.kernel.org/bpf/[email protected]

* Rename __arena_nullable to __arena__nullable and prioritize the composite suffix over __nullable during argument classification. (Sashiko, Eduard)
* Resolve instructions before collecting subprograms and kfuncs so kfunc prototype validation can use associated arena state.
* Move the arena kfunc and JIT-sequence test entry points into prog_tests/verifier.c. (Eduard)
* Match the generated L0 target and call in nullable JIT assertions. (Eduard)
* Route arena kfunc validation through the common argument-checking path. (Amery)
* Reuse btf_func_model argument flags for struct_ops arena arguments instead of maintaining separate trampoline slot metadata. (Eduard)
* Check the generic-trampoline arena argument invariant at link time and warn once on violations. (Eduard)
* Reject tracing attachments to struct_ops programs with arena context arguments whose indirect trampolines convert the pointers. (Sashiko)

v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]

* Rebase onto current bpf-next to resolve conflicts.

v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Fix documentation to only mention x86 for now. (Sashiko)
* Move arg bitmap from insn_aux_data to kfunc descriptor. (Eduard)

Kumar Kartikeya Dwivedi (4):
bpf: Rename 'early' BTF checking as a preparation phase
bpf: Split subprogram and kfunc collection
bpf: Collect kfuncs after resolving program resources
bpf: Reject tracing progs for struct_ops with arena args

Tejun Heo (9):
bpf: Support __arena and __arena__nullable kfunc argument suffixes
bpf: Support __arena and __arena__nullable on struct_ops arguments bpf, x86: JIT __arena kfunc argument rebasing
bpf, x86: Convert struct_ops arena arguments in the trampoline selftests/bpf: Add kfunc __arena and __arena__nullable argument tests selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments selftests/bpf: Add struct_ops __arena and __arena__nullable argument
tests
bpf, x86: Fix stack-passed arguments for indirect trampolines selftests/bpf: Test stack-passed struct_ops arena arguments

 Documentation/bpf/kfuncs.rst                  |  39 +++
 arch/x86/net/bpf_jit_comp.c                   | 122 +++++++-
 include/linux/bpf.h                           |  16 ++
 include/linux/bpf_verifier.h                  |   4 +-
 include/linux/filter.h                        |   1 +
 kernel/bpf/bpf_struct_ops.c                   |  52 +++-
 kernel/bpf/btf.c                              |  28 +-
 kernel/bpf/check_btf.c                        |  14 +-
 kernel/bpf/core.c                             |   5 +
 kernel/bpf/trampoline.c                       |  37 +++
 kernel/bpf/verifier.c                         | 141 ++++++++--
 .../bpf/prog_tests/test_struct_ops_arena.c    |  74 +++++
 .../selftests/bpf/prog_tests/verifier.c       |  11 +
 .../testing/selftests/bpf/progs/arena_kfunc.c | 260 ++++++++++++++++++
 .../selftests/bpf/progs/arena_kfunc_jit.c     |  98 +++++++
 .../selftests/bpf/progs/struct_ops_arena.c    | 115 ++++++++
 .../bpf/progs/struct_ops_arena_fail.c         |  20 ++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  95 +++++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   6 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  13 +
 20 files changed, 1092 insertions(+), 59 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc_jit.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c

base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e

Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH v4 1/13 bpf: Rename 'early' BTF checking as a preparation phase

BTF processing is split around subprogram discovery. The first phase gets program BTF and imports func_info because a BTF-tagged exception callback may not be referenced by any instruction. Subprogram discovery needs this metadata to find it.

The later phase validates func_info and line_info against the complete subprogram table and applies CO-RE relocations. This split breaks a real dependency cycle rather than merely running the same checks early.

Rename bpf_check_btf_info_early() and check_btf_func_early() to preparation names that reflect this role. Add short call-site comments to make the two phases and their responsibilities clear.

No functional change is intended.

---
 include/linux/bpf_verifier.h |  4 ++--
 kernel/bpf/check_btf.c       | 14 +++++++-------
 kernel/bpf/verifier.c        |  4 +++-
 3 files changed, 12 insertions(+), 10 deletions(-)
include/linux/bpf_verifier.h +2 -2
index a2a40caca0a0..a9555d17fd8e 100644
@@ -1177,8 +1177,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
 		*btf_id = key & 0x7FFFFFFF;
 }
 
-int bpf_check_btf_info_early(struct bpf_verifier_env *env,
-			     const union bpf_attr *attr, bpfptr_t uattr);
+int bpf_prepare_btf_info(struct bpf_verifier_env *env,
+			 const union bpf_attr *attr, bpfptr_t uattr);
 int bpf_check_btf_info(struct bpf_verifier_env *env,
 		       const union bpf_attr *attr, bpfptr_t uattr);
 
kernel/bpf/check_btf.c +7 -7
index 93bebe6fe12e..0e8b3ccc7a5b 100644
@@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
 #define MIN_BPF_FUNCINFO_SIZE	8
 #define MAX_FUNCINFO_REC_SIZE	252
 
-static int check_btf_func_early(struct bpf_verifier_env *env,
-				const union bpf_attr *attr,
-				bpfptr_t uattr)
+static int prepare_btf_func(struct bpf_verifier_env *env,
+			    const union bpf_attr *attr,
+			    bpfptr_t uattr)
 {
 	u32 krec_size = sizeof(struct bpf_func_info);
 	const struct btf_type *type, *func_proto;
@@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,
 	return err;
 }
 
-int bpf_check_btf_info_early(struct bpf_verifier_env *env,
-			     const union bpf_attr *attr,
-			     bpfptr_t uattr)
+int bpf_prepare_btf_info(struct bpf_verifier_env *env,
+			 const union bpf_attr *attr,
+			 bpfptr_t uattr)
 {
 	struct btf *btf;
 	int err;
@@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env,
 	}
 	env->prog->aux->btf = btf;
 
-	err = check_btf_func_early(env, attr, uattr);
+	err = prepare_btf_func(env, attr, uattr);
 	if (err)
 		return err;
 	return 0;
kernel/bpf/verifier.c +3 -1
index d925197c2e5f..129e50888b90 100644
@@ -20317,7 +20317,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		INIT_LIST_HEAD(&env->explored_states[i]);
 	INIT_LIST_HEAD(&env->free_list);
 
-	ret = bpf_check_btf_info_early(env, attr, uattr);
+	/* Prepare BTF and func_info needed to discover all subprograms. */
+	ret = bpf_prepare_btf_info(env, attr, uattr);
 	if (ret < 0)
 		goto skip_full_check;
 
@@ -20329,6 +20330,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* Validate BTF against the complete subprogram layout and apply CO-RE. */
 	ret = bpf_check_btf_info(env, attr, uattr);
 	if (ret < 0)
 		goto skip_full_check;
Signature · 1 line
2.53.0
Reviewed-by on v4 1/13
#2 lore raw

Re: [PATCH bpf-next v4 01/13] bpf: Rename 'early' BTF checking as a preparation phase

Quoted 10 lines · Kumar Kartikeya Dwivedi

BTF processing is split around subprogram discovery. The first phase gets program BTF and imports func_info because a BTF-tagged exception callback may not be referenced by any instruction. Subprogram discovery needs this metadata to find it.

The later phase validates func_info and line_info against the complete subprogram table and applies CO-RE relocations. This split breaks a real dependency cycle rather than merely running the same checks early.

Quoted 7 lines

Rename bpf_check_btf_info_early() and check_btf_func_early() to preparation names that reflect this role. Add short call-site comments to make the two phases and their responsibilities clear.

No functional change is intended.

Quoted 86 lines
---
 include/linux/bpf_verifier.h |  4 ++--
 kernel/bpf/check_btf.c       | 14 +++++++-------
 kernel/bpf/verifier.c        |  4 +++-
 3 files changed, 12 insertions(+), 10 deletions(-)
include/linux/bpf_verifier.h +2 -2
index a2a40caca0a0..a9555d17fd8e 100644
@@ -1177,8 +1177,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
                *btf_id = key & 0x7FFFFFFF;
 }
 
-int bpf_check_btf_info_early(struct bpf_verifier_env *env,
-                            const union bpf_attr *attr, bpfptr_t uattr);
+int bpf_prepare_btf_info(struct bpf_verifier_env *env,
+                        const union bpf_attr *attr, bpfptr_t uattr);
 int bpf_check_btf_info(struct bpf_verifier_env *env,
                       const union bpf_attr *attr, bpfptr_t uattr);
 
kernel/bpf/check_btf.c +7 -7
index 93bebe6fe12e..0e8b3ccc7a5b 100644
@@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
 #define MIN_BPF_FUNCINFO_SIZE  8
 #define MAX_FUNCINFO_REC_SIZE  252
 
-static int check_btf_func_early(struct bpf_verifier_env *env,
-                               const union bpf_attr *attr,
-                               bpfptr_t uattr)
+static int prepare_btf_func(struct bpf_verifier_env *env,
+                           const union bpf_attr *attr,
+                           bpfptr_t uattr)
 {
        u32 krec_size = sizeof(struct bpf_func_info);
        const struct btf_type *type, *func_proto;
@@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,
        return err;
 }
 
-int bpf_check_btf_info_early(struct bpf_verifier_env *env,
-                            const union bpf_attr *attr,
-                            bpfptr_t uattr)
+int bpf_prepare_btf_info(struct bpf_verifier_env *env,
+                        const union bpf_attr *attr,
+                        bpfptr_t uattr)
 {
        struct btf *btf;
        int err;
@@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env,
        }
        env->prog->aux->btf = btf;
 
-       err = check_btf_func_early(env, attr, uattr);
+       err = prepare_btf_func(env, attr, uattr);
        if (err)
                return err;
        return 0;
kernel/bpf/verifier.c +3 -2
index d925197c2e5f..129e50888b90 100644
@@ -20317,7 +20317,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
                INIT_LIST_HEAD(&env->explored_states[i]);
        INIT_LIST_HEAD(&env->free_list);
 
-       ret = bpf_check_btf_info_early(env, attr, uattr);
+       /* Prepare BTF and func_info needed to discover all subprograms. */
+       ret = bpf_prepare_btf_info(env, attr, uattr);
        if (ret < 0)
                goto skip_full_check;
 
@@ -20329,6 +20330,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
        if (ret < 0)
                goto skip_full_check;
 
+       /* Validate BTF against the complete subprogram layout and apply CO-RE. */
        ret = bpf_check_btf_info(env, attr, uattr);
        if (ret < 0)
                goto skip_full_check;
--

2.53.0

author maintainer
#3 lore raw

PATCH v4 2/13 bpf: Split subprogram and kfunc collection

In reply to Kumar Kartikeya Dwivedi

add_subprog_and_kfunc() combines two operations with different ordering requirements. Subprogram discovery must precede validation of func_info and line_info, while kfunc descriptors are only needed by the verifier after its initial program setup is complete.

Split the helper into add_subprogs() and add_kfuncs() so each operation can be placed according to its actual dependencies. Keep both calls adjacent and in their existing phase for now, and add short comments describing their roles.

No functional change is intended for valid programs.

---
 kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++++++++++---------
 1 file changed, 32 insertions(+), 9 deletions(-)
kernel/bpf/verifier.c +32 -9
index 129e50888b90..24b163c2bd63 100644
@@ -2842,7 +2842,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	return 0;
 }
 
-static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
+static int add_subprogs(struct bpf_verifier_env *env)
 {
 	struct bpf_subprog_info *subprog = env->subprog_info;
 	int i, ret, insn_cnt = env->prog->len, ex_cb_insn;
@@ -2854,8 +2854,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 		return ret;
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
-		if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn) &&
-		    !bpf_pseudo_kfunc_call(insn))
+		if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
 			continue;
 
 		if (!env->bpf_capable) {
@@ -2863,11 +2862,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 			return -EPERM;
 		}
 
-		if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn))
-			ret = add_subprog(env, i + insn->imm + 1);
-		else
-			ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
-
+		ret = add_subprog(env, i + insn->imm + 1);
 		if (ret < 0)
 			return ret;
 	}
@@ -2905,6 +2900,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 	return 0;
 }
 
+static int add_kfuncs(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	int i, ret, insn_cnt = env->prog->len;
+
+	for (i = 0; i < insn_cnt; i++, insn++) {
+		if (!bpf_pseudo_kfunc_call(insn))
+			continue;
+
+		if (!env->bpf_capable) {
+			verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
+			return -EPERM;
+		}
+
+		ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+
 static int check_subprogs(struct bpf_verifier_env *env)
 {
 	int i, subprog_start, subprog_end, off, cur_subprog = 0;
@@ -20322,7 +20339,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
-	ret = add_subprog_and_kfunc(env);
+	/* Discover all subprograms before validating their layout and BTF. */
+	ret = add_subprogs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	/* Collect the kfunc descriptors used during verification. */
+	ret = add_kfuncs(env);
 	if (ret < 0)
 		goto skip_full_check;
 
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH bpf-next v4 02/13] bpf: Split subprogram and kfunc collection

The rename of add_subprog_and_kfunc() leaves a dangling reference in kernel/sched/ext/ext.c. The comment in scx_kfunc_filter() still says:

kernel/bpf/verifier.c int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) · line 2845

Quoted 10 lines
kernel/bpf/verifier.c +1 -1
index 129e50888b90..24b163c2bd63 100644
@@ -2842,7 +2842,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	return 0;
 }
 
-static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
+static int add_subprogs(struct bpf_verifier_env *env)

The rename of add_subprog_and_kfunc() leaves a dangling reference in kernel/sched/ext/ext.c. The comment in scx_kfunc_filter() still says:

/*
* add_subprog_and_kfunc() collects all kfunc calls, including dead code
* guarded by bpf_ksym_exists(), before check_attach_btf_id() sets
* prog->aux->st_ops. Allow all kfuncs when st_ops is not yet set;
* do_check_main() re-runs the filter with st_ops set and enforces the
* actual restrictions.
*/

Should this be updated to reference add_kfuncs() instead?

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
 {
 	struct bpf_subprog_info *subprog = env->subprog_info;
 	int i, ret, insn_cnt = env->prog->len, ex_cb_insn;

[ ... ]

Quoted 26 lines
kernel/bpf/verifier.c +22 -0
@@ -2905,6 +2900,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 	return 0;
 }
 
+static int add_kfuncs(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insn = env->prog->insnsi;
+	int i, ret, insn_cnt = env->prog->len;
+
+	for (i = 0; i < insn_cnt; i++, insn++) {
+		if (!bpf_pseudo_kfunc_call(insn))
+			continue;
+
+		if (!env->bpf_capable) {
+			verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
+			return -EPERM;
+		}
+
+		ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
+		if (ret < 0)
+			return ret;
+	}
+
+	return 0;
+}
+

[ ... ]

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
@@ -20322,7 +20339,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 

kernel/bpf/verifier.c int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, · line 20350

Quoted 10 lines
kernel/bpf/verifier.c +7 -1
-	ret = add_subprog_and_kfunc(env);
+	/* Discover all subprograms before validating their layout and BTF. */
+	ret = add_subprogs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	/* Collect the kfunc descriptors used during verification. */
+	ret = add_kfuncs(env);
 	if (ret < 0)
 		goto skip_full_check;

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31047716344

Reviewed-by on v4 2/13
#5 lore raw

Re: [PATCH bpf-next v4 02/13] bpf: Split subprogram and kfunc collection

In reply to Kumar Kartikeya Dwivedi
Quoted 6 lines · Kumar Kartikeya Dwivedi

add_subprog_and_kfunc() combines two operations with different ordering requirements. Subprogram discovery must precede validation of func_info and line_info, while kfunc descriptors are only needed by the verifier after its initial program setup is complete.

Quoted 7 lines

Split the helper into add_subprogs() and add_kfuncs() so each operation can be placed according to its actual dependencies. Keep both calls adjacent and in their existing phase for now, and add short comments describing their roles.

No functional change is intended for valid programs.

Quoted 45 lines
---
 kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++++++++++---------
 1 file changed, 32 insertions(+), 9 deletions(-)
kernel/bpf/verifier.c +4 -8
index 129e50888b90..24b163c2bd63 100644
@@ -2842,7 +2842,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
        return 0;
 }
 
-static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
+static int add_subprogs(struct bpf_verifier_env *env)
 {
        struct bpf_subprog_info *subprog = env->subprog_info;
        int i, ret, insn_cnt = env->prog->len, ex_cb_insn;
@@ -2854,8 +2854,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
                return ret;
 
        for (i = 0; i < insn_cnt; i++, insn++) {
-               if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn) &&
-                   !bpf_pseudo_kfunc_call(insn))
+               if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
                        continue;
 
                if (!env->bpf_capable) {
@@ -2863,11 +2862,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
                        return -EPERM;
                }
 
-               if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn))
-                       ret = add_subprog(env, i + insn->imm + 1);
-               else
-                       ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
-
+               ret = add_subprog(env, i + insn->imm + 1);
                if (ret < 0)
                        return ret;
        }
@@ -2905,6 +2900,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
        return 0;
 }
 
+static int add_kfuncs(struct bpf_verifier_env *env)

kernel/bpf/verifier.c static int add_subprog_and_kfunc(struct bpf_verifier_env *env) · line 2913

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+{
+       struct bpf_insn *insn = env->prog->insnsi;
+       int i, ret, insn_cnt = env->prog->len;
+
+       for (i = 0; i < insn_cnt; i++, insn++) {
+               if (!bpf_pseudo_kfunc_call(insn))
+                       continue;
+
+               if (!env->bpf_capable) {
+                       verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");

nit: Should this and the error message in add_subprogs() be updated to be more specific?

Quoted 33 lines
kernel/bpf/verifier.c +18 -2
+                       return -EPERM;
+               }
+
+               ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
+               if (ret < 0)
+                       return ret;
+       }
+
+       return 0;
+}
+
 static int check_subprogs(struct bpf_verifier_env *env)
 {
        int i, subprog_start, subprog_end, off, cur_subprog = 0;
@@ -20322,7 +20339,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
        if (ret < 0)
                goto skip_full_check;
 
-       ret = add_subprog_and_kfunc(env);
+       /* Discover all subprograms before validating their layout and BTF. */
+       ret = add_subprogs(env);
+       if (ret < 0)
+               goto skip_full_check;
+
+       /* Collect the kfunc descriptors used during verification. */
+       ret = add_kfuncs(env);
        if (ret < 0)
                goto skip_full_check;
 
--

2.53.0

author maintainer
#6 lore raw

PATCH v4 3/13 bpf: Collect kfuncs after resolving program resources

In reply to Kumar Kartikeya Dwivedi

The kfunc descriptors include argument prototypes generated while calls are collected. Some argument classifications need program auxiliary state derived from referenced maps, such as the arena associated with the program.

This avoids a footgun in get_kfunc_arg_type() checks where we do validation on whether program has prog->aux->arena and it hasn't been resolved yet.

check_and_resolve_insns() records used maps and populates that state. It must remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations, so that instruction validation and the program tag observe the relocated instruction stream.

Move only add_kfuncs() after instruction and resource resolution. Subprogram discovery and validation remain before the full BTF phase because that phase needs the complete subprogram layout. Add a short comment describing the resource resolution phase at the call site.

---
 kernel/bpf/verifier.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +6 -5
index 24b163c2bd63..b62e77949542 100644
@@ -20344,11 +20344,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
-	/* Collect the kfunc descriptors used during verification. */
-	ret = add_kfuncs(env);
-	if (ret < 0)
-		goto skip_full_check;
-
 	ret = check_subprogs(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -20358,10 +20353,16 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* Build kfunc prototypes after resolving program resources. */
+	ret = add_kfuncs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	if (bpf_prog_is_offloaded(env->prog->aux)) {
 		ret = bpf_prog_offload_verifier_prep(env->prog);
 		if (ret)
Signature · 1 line
2.53.0
Reviewed-by on v4 3/13
#7 lore raw

Re: [PATCH bpf-next v4 03/13] bpf: Collect kfuncs after resolving program resources

Quoted 14 lines · Kumar Kartikeya Dwivedi

The kfunc descriptors include argument prototypes generated while calls are collected. Some argument classifications need program auxiliary state derived from referenced maps, such as the arena associated with the program.

This avoids a footgun in get_kfunc_arg_type() checks where we do validation on whether program has prog->aux->arena and it hasn't been resolved yet.

check_and_resolve_insns() records used maps and populates that state. It must remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations, so that instruction validation and the program tag observe the relocated instruction stream.

Quoted 6 lines

Move only add_kfuncs() after instruction and resource resolution. Subprogram discovery and validation remain before the full BTF phase because that phase needs the complete subprogram layout. Add a short comment describing the resource resolution phase at the call site.

Quoted 41 lines
---
 kernel/bpf/verifier.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +6 -6
index 24b163c2bd63..b62e77949542 100644
@@ -20344,11 +20344,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
        if (ret < 0)
                goto skip_full_check;
 
-       /* Collect the kfunc descriptors used during verification. */
-       ret = add_kfuncs(env);
-       if (ret < 0)
-               goto skip_full_check;
-
        ret = check_subprogs(env);
        if (ret < 0)
                goto skip_full_check;
@@ -20358,10 +20353,16 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
        if (ret < 0)
                goto skip_full_check;
 
+       /* Validate instructions and resolve the program's referenced resources. */
        ret = check_and_resolve_insns(env);
        if (ret < 0)
                goto skip_full_check;
 
+       /* Build kfunc prototypes after resolving program resources. */
+       ret = add_kfuncs(env);
+       if (ret < 0)
+               goto skip_full_check;
+
        if (bpf_prog_is_offloaded(env->prog->aux)) {
                ret = bpf_prog_offload_verifier_prep(env->prog);
                if (ret)
--

2.53.0

author maintainer
#8 lore raw

PATCH v4 4/13 bpf: Support __arena and __arena__nullable kfunc argument suffixes

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data.

JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere.

---
 Documentation/bpf/kfuncs.rst | 29 ++++++++++++++++++++
 include/linux/bpf.h          |  6 +++++
 include/linux/filter.h       |  1 +
 kernel/bpf/btf.c             | 18 ++++++++++++-
 kernel/bpf/core.c            |  5 ++++
 kernel/bpf/verifier.c        | 51 +++++++++++++++++++++++++++++++++---
 6 files changed, 105 insertions(+), 5 deletions(-)
Documentation/bpf/kfuncs.rst +29 -0
index cbde86d082cc..cd9b90d072cf 100644
@@ -278,6 +278,33 @@ An example is given below::
                 ...
         }
 
+2.3.8 __arena and __arena__nullable Annotations
+-----------------------------------------------
+
+Both annotations indicate that the pointer argument points into the
+calling program's arena. The JIT rebases the value at the call site so
+the kfunc receives a directly dereferenceable kernel address, subject to
+the access rules described in :ref:`BPF_kfunc_arena_access` (at most
+``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access).
+
+With ``__arena`` the rebase is unconditional and the argument is never
+NULL: a value whose lower 32 bits are zero arrives as the arena base
+address (arena offset 0). The kfunc must not check the argument for NULL.
+With ``__arena__nullable`` such a value arrives as NULL instead and the
+kfunc must check before dereferencing.
+
+An example is given below::
+
+        __bpf_kfunc int bpf_process_item(struct item *item__arena)
+        {
+        ...
+        }
+
+Calling such a kfunc requires the program to use an arena map and a JIT with
+arena argument support (currently x86-64); verification fails otherwise. The
+program can pass any value without compromising the kernel. A value that does
+not point into the arena is a program bug.
+
 .. _BPF_kfunc_nodef:
 
 2.4 Using an existing kernel function
@@ -515,6 +542,8 @@ In order to accommodate such requirements, the verifier will enforce strict
 PTR_TO_BTF_ID type matching if two types have the exact same name, with one
 being suffixed with ``___init``.
 
+.. _BPF_kfunc_arena_access:
+
 2.8 Accessing arena memory through kfunc arguments
 --------------------------------------------------
 
include/linux/bpf.h +6 -0
index 73bacfc6444d..f47556b56a48 100644
@@ -1210,6 +1210,12 @@ struct bpf_prog_offload {
 /* The argument is signed. */
 #define BTF_FMODEL_SIGNED_ARG		BIT(1)
 
+/* The argument is an arena pointer. */
+#define BTF_FMODEL_ARENA_ARG		BIT(2)
+
+/* The argument is nullable. */
+#define BTF_FMODEL_NULLABLE_ARG		BIT(3)
+
 struct btf_func_model {
 	u8 ret_size;
 	u8 ret_flags;
include/linux/filter.h +1 -0
index 32d5297c557e..36ce3403fe59 100644
@@ -1183,6 +1183,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
 bool bpf_jit_supports_stack_args(void);
+bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_ptr_xchg(void);
kernel/bpf/btf.c +17 -1
index 42414633cf26..4ff6148ae8e8 100644
@@ -7539,6 +7539,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)
 	return flags;
 }
 
+static u8 __get_arg_fmodel_flags(const struct btf *btf,
+				 const struct btf_param *arg,
+				 const struct btf_type *t)
+{
+	u8 flags = __get_type_fmodel_flags(t);
+
+	if (btf_param_match_suffix(btf, arg, "__arena__nullable"))
+		flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG;
+	else if (btf_param_match_suffix(btf, arg, "__arena"))
+		flags |= BTF_FMODEL_ARENA_ARG;
+	else if (btf_param_match_suffix(btf, arg, "__nullable"))
+		flags |= BTF_FMODEL_NULLABLE_ARG;
+
+	return flags;
+}
+
 int btf_distill_func_proto(struct bpf_verifier_log *log,
 			   struct btf *btf,
 			   const struct btf_type *func,
@@ -7604,7 +7620,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 			return -EINVAL;
 		}
 		m->arg_size[i] = ret;
-		m->arg_flags[i] = __get_type_fmodel_flags(t);
+		m->arg_flags[i] = __get_arg_fmodel_flags(btf, &args[i], t);
 	}
 	m->nr_args = nargs;
 	return 0;
kernel/bpf/core.c +5 -0
index e2076667b245..a3e1fae32eac 100644
@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_arena_args(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_far_kfunc_call(void)
 {
 	return false;
kernel/bpf/verifier.c +47 -4

101 lines not on this page · raw message

Signature · 1 line
2.53.0
#9 lore raw

Re: [PATCH bpf-next v4 04/13] bpf: Support __arena and __arena__nullable kfunc argument suffixes

Quoted 24 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

Quoted 4 lines

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

Since __arena__nullable will match is_kfunc_arg_arena() case and go through JIT + regno check, and get its PTR_MAYBE_NULL anyway. How about just keep it as __arena_nullable to simplify the patch?

1. No need to introudce is_kfunc_arg_arena_nullable() and changes in is_kfunc_arg_nullable()
2. is_kfunc_arg_arena() returns btf_param_match_suffix(btf, arg, "__arena_nullable") || btf_param_match_suffix(btf, arg, "__arena");

[...]

Quoted 10 lines
kernel/bpf/verifier.c +2 -0
index b62e77949542..2b7f6f6bbe76 100644
@@ -10909,9 +10909,16 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
        return btf_param_match_suffix(btf, arg, "__refcounted_kptr");
 }
 
+static bool is_kfunc_arg_arena_nullable(const struct btf *btf,
+                                       const struct btf_param *arg)

kernel/bpf/verifier.c static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf · line 10922

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
+{
+       return btf_param_match_suffix(btf, arg, "__arena__nullable");
+}
+
 static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
 {
-       return btf_param_match_suffix(btf, arg, "__nullable");
+       return !is_kfunc_arg_arena_nullable(btf, arg) &&
+              btf_param_match_suffix(btf, arg, "__nullable");
 }

No need for the changes above.

kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10942

Quoted 10 lines
kernel/bpf/verifier.c +4 -0
 static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
@@ -10929,6 +10936,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
        return btf_param_match_suffix(btf, arg, "__irq_flag");
 }
 
+static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
+{
+       return is_kfunc_arg_arena_nullable(btf, arg) ||
+              btf_param_match_suffix(btf, arg, "__arena");

return btf_param_match_suffix(btf, arg, "__arena_nullable") ||

        btf_param_match_suffix(btf, arg, "__arena");

[...]

author maintainer
#10 lore raw

Re: [PATCH bpf-next v4 04/13] bpf: Support __arena and __arena__nullable kfunc argument suffixes

Quoted 30 lines · Amery Hung
Quoted 24 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

Quoted 4 lines

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

Quoted 6 lines

Since __arena__nullable will match is_kfunc_arg_arena() case and go through JIT + regno check, and get its PTR_MAYBE_NULL anyway. How about just keep it as __arena_nullable to simplify the patch?

1. No need to introudce is_kfunc_arg_arena_nullable() and changes in is_kfunc_arg_nullable()

For consistency, would you prefer that I don't manually set | PTR_MAYBE_NULL and let is_kfunc_arg_nullable() handle that? That would be another way to address this.

In some sense, __arena includes __nullable for the purposes of type checking, so it might make sense to add it to the predicate that determines NULL-ness, then it will acquire PTR_MAYBE_NULL automatically.

We will still drop is_kfunc_arg_arena_nullable() though.

Anyhow, I don't have any strong preference one way or the other, but thought I'd float this as an alternative since it appears to fit better, and details are hidden the predicates.

Quoted 3 lines

2. is_kfunc_arg_arena() returns btf_param_match_suffix(btf, arg, "__arena_nullable") || btf_param_match_suffix(btf, arg, "__arena");

Yeah, makes sense.

Quoted 31 lines

[...]

Quoted 10 lines
kernel/bpf/verifier.c +2 -0
index b62e77949542..2b7f6f6bbe76 100644
@@ -10909,9 +10909,16 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
        return btf_param_match_suffix(btf, arg, "__refcounted_kptr");
 }
 
+static bool is_kfunc_arg_arena_nullable(const struct btf *btf,
+                                       const struct btf_param *arg)

kernel/bpf/verifier.c static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf · line 10922

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
+{
+       return btf_param_match_suffix(btf, arg, "__arena__nullable");
+}
+
 static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
 {
-       return btf_param_match_suffix(btf, arg, "__nullable");
+       return !is_kfunc_arg_arena_nullable(btf, arg) &&
+              btf_param_match_suffix(btf, arg, "__nullable");
 }

No need for the changes above.

Quoted 6 lines
kernel/bpf/verifier.c +0 -0
 static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
@@ -10929,6 +10936,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
        return btf_param_match_suffix(btf, arg, "__irq_flag");
 }
 
Quoted 8 lines

kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10942

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
+{
+       return is_kfunc_arg_arena_nullable(btf, arg) ||
+              btf_param_match_suffix(btf, arg, "__arena");

return btf_param_match_suffix(btf, arg, "__arena_nullable") ||

        btf_param_match_suffix(btf, arg, "__arena");

Ack, I'll adjust this bit.

Quoted 1 line

[...]

author maintainer
#11 lore raw

Re: [PATCH bpf-next v4 04/13] bpf: Support __arena and __arena__nullable kfunc argument suffixes

Quoted 46 lines · Kumar Kartikeya Dwivedi
Quoted 30 lines · Amery Hung
Quoted 24 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

Quoted 4 lines

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

Quoted 6 lines

Since __arena__nullable will match is_kfunc_arg_arena() case and go through JIT + regno check, and get its PTR_MAYBE_NULL anyway. How about just keep it as __arena_nullable to simplify the patch?

1. No need to introudce is_kfunc_arg_arena_nullable() and changes in is_kfunc_arg_nullable()

For consistency, would you prefer that I don't manually set | PTR_MAYBE_NULL and let is_kfunc_arg_nullable() handle that? That would be another way to address this.

In some sense, __arena includes __nullable for the purposes of type checking, so it might make sense to add it to the predicate that determines NULL-ness, then it will acquire PTR_MAYBE_NULL automatically.

Quoted 5 lines

We will still drop is_kfunc_arg_arena_nullable() though.

Anyhow, I don't have any strong preference one way or the other, but thought I'd float this as an alternative since it appears to fit better, and details are hidden the predicates.

This will amount to adding extra OR for __arena prefix match inside is_kfunc_arg_nullable(), since __nullable should already match on __arena__nullable.

Quoted 50 lines
Quoted 4 lines

2. is_kfunc_arg_arena() returns btf_param_match_suffix(btf, arg, "__arena_nullable") || btf_param_match_suffix(btf, arg, "__arena");

Yeah, makes sense.

Quoted 31 lines

[...]

Quoted 10 lines
kernel/bpf/verifier.c +2 -0
index b62e77949542..2b7f6f6bbe76 100644
@@ -10909,9 +10909,16 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
        return btf_param_match_suffix(btf, arg, "__refcounted_kptr");
 }
 
+static bool is_kfunc_arg_arena_nullable(const struct btf *btf,
+                                       const struct btf_param *arg)

kernel/bpf/verifier.c static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf · line 10922

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
+{
+       return btf_param_match_suffix(btf, arg, "__arena__nullable");
+}
+
 static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
 {
-       return btf_param_match_suffix(btf, arg, "__nullable");
+       return !is_kfunc_arg_arena_nullable(btf, arg) &&
+              btf_param_match_suffix(btf, arg, "__nullable");
 }

No need for the changes above.

Quoted 6 lines
kernel/bpf/verifier.c +0 -0
 static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
@@ -10929,6 +10936,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
        return btf_param_match_suffix(btf, arg, "__irq_flag");
 }
 
Quoted 8 lines

kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10942

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
+{
+       return is_kfunc_arg_arena_nullable(btf, arg) ||
+              btf_param_match_suffix(btf, arg, "__arena");

return btf_param_match_suffix(btf, arg, "__arena_nullable") ||

        btf_param_match_suffix(btf, arg, "__arena");

Ack, I'll adjust this bit.

Quoted 1 line

[...]

#12 lore raw

Re: [PATCH bpf-next v4 04/13] bpf: Support __arena and __arena__nullable kfunc argument suffixes

Quoted 37 lines · Kumar Kartikeya Dwivedi
Quoted 35 lines · Kumar Kartikeya Dwivedi
Quoted 34 lines · Amery Hung
Quoted 24 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

Quoted 4 lines

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

Since __arena__nullable will match is_kfunc_arg_arena() case and go through JIT + regno check, and get its PTR_MAYBE_NULL anyway. How about just keep it as __arena_nullable to simplify the patch?

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

1. No need to introudce is_kfunc_arg_arena_nullable() and changes in is_kfunc_arg_nullable()

For consistency, would you prefer that I don't manually set | PTR_MAYBE_NULL and let is_kfunc_arg_nullable() handle that? That would be another way to address this.

Yeah. This also makes sense. No strong preference here.

Quoted 66 lines
Quoted 10 lines

In some sense, __arena includes __nullable for the purposes of type checking, so it might make sense to add it to the predicate that determines NULL-ness, then it will acquire PTR_MAYBE_NULL automatically.

We will still drop is_kfunc_arg_arena_nullable() though.

Anyhow, I don't have any strong preference one way or the other, but thought I'd float this as an alternative since it appears to fit better, and details are hidden the predicates.

This will amount to adding extra OR for __arena prefix match inside is_kfunc_arg_nullable(), since __nullable should already match on __arena__nullable.

Quoted 50 lines
Quoted 4 lines

2. is_kfunc_arg_arena() returns btf_param_match_suffix(btf, arg, "__arena_nullable") || btf_param_match_suffix(btf, arg, "__arena");

Yeah, makes sense.

Quoted 31 lines

[...]

Quoted 10 lines
kernel/bpf/verifier.c +2 -0
index b62e77949542..2b7f6f6bbe76 100644
@@ -10909,9 +10909,16 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
        return btf_param_match_suffix(btf, arg, "__refcounted_kptr");
 }
 
+static bool is_kfunc_arg_arena_nullable(const struct btf *btf,
+                                       const struct btf_param *arg)

kernel/bpf/verifier.c static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf · line 10922

Quoted 10 lines
kernel/bpf/verifier.c +6 -1
+{
+       return btf_param_match_suffix(btf, arg, "__arena__nullable");
+}
+
 static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
 {
-       return btf_param_match_suffix(btf, arg, "__nullable");
+       return !is_kfunc_arg_arena_nullable(btf, arg) &&
+              btf_param_match_suffix(btf, arg, "__nullable");
 }

No need for the changes above.

Quoted 6 lines
kernel/bpf/verifier.c +0 -0
 static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
@@ -10929,6 +10936,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
        return btf_param_match_suffix(btf, arg, "__irq_flag");
 }
 
Quoted 8 lines

kernel/bpf/verifier.c static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param · line 10942

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
+{
+       return is_kfunc_arg_arena_nullable(btf, arg) ||
+              btf_param_match_suffix(btf, arg, "__arena");

return btf_param_match_suffix(btf, arg, "__arena_nullable") ||

        btf_param_match_suffix(btf, arg, "__arena");

Ack, I'll adjust this bit.

Quoted 1 line

[...]

maintainer Acked-by on v4 4/13
#13 lore raw

Re: [PATCH bpf-next v4 04/13] bpf: Support __arena and __arena__nullable kfunc argument suffixes

In reply to Kumar Kartikeya Dwivedi
Quoted 33 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation.

Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments.

__arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call.

The double separator makes the annotations composable:
__arena__nullable also ends in __nullable. Match the composite suffix first when classifying kfunc arguments and function-model flags so it retains arena semantics while carrying the nullable flag.

This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data.

Quoted 7 lines

JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere.

---

author maintainer
#14 lore raw

PATCH v4 5/13 bpf: Support __arena and __arena__nullable on struct_ops arguments

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in.

Add the __arena and __arena__nullable stub argument suffixes to make this convenient. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). The trampoline converts the value while saving the arguments into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the program never sees a kernel address and nothing rewrites the ctx after the fact. The converted value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address.

__arena converts unconditionally and the kernel caller must not pass NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the program's type state already encompasses NULL-ness, so it is not meaningful to force a NULL check for the program.

The composite suffix intentionally ends in __nullable. Classify __arena__nullable before the generic suffix so scalar arena pointees do not take the generic nullable BTF pointer path.

This patch adds the generic side. bpf_tramp_collect_arena_args() derives the conversion map from the prog's ctx_arg_info, keyed by the flattened ctx byte offset since preceding 16-byte arguments occupy two slots. prepare_arg_info() also records arena and nullable argument flags in the struct_ops function model for arch trampolines. Only the struct_ops indirect trampoline converts: it dispatches to a single prog whose arena is fixed at generation time. Generic trampolines can mix progs with different arenas and reject arena ctx args defensively, which is unreachable today as only struct_ops progs carry them. Arch trampolines that do not implement the conversion are gated out at verification time with bpf_jit_supports_arena_args().

---
 Documentation/bpf/kfuncs.rst | 10 +++++++
 include/linux/bpf.h          | 19 +++++++++++++
 kernel/bpf/bpf_struct_ops.c  | 53 +++++++++++++++++++++++++----------
 kernel/bpf/btf.c             | 10 +++++--
 kernel/bpf/trampoline.c      | 54 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        | 23 +++++++++++----
 6 files changed, 147 insertions(+), 22 deletions(-)
Documentation/bpf/kfuncs.rst +10 -0
index cd9b90d072cf..c044659b7689 100644
@@ -305,6 +305,16 @@ arena argument support (currently x86-64); verification fails otherwise. The
 program can pass any value without compromising the kernel. A value that does
 not point into the arena is a program bug.
 
+The suffixes have the same meaning on the arguments of struct_ops stub
+functions, with the conversion running in the opposite direction. The
+kernel caller passes the kernel arena address and the trampoline converts
+it while saving the arguments, so the callback receives an arena pointer
+it can dereference directly. With ``__arena`` the kernel caller must not
+pass NULL. With ``__arena__nullable`` a NULL kernel pointer arrives as NULL.
+However, there is no obligation to prove to the verifier that such a pointer is
+non-NULL before use, in-line with existing semantics of arena pointers used in
+a program (or obtained from any other source).
+
 .. _BPF_kfunc_nodef:
 
 2.4 Using an existing kernel function
include/linux/bpf.h +19 -0
index f47556b56a48..0d1773af8373 100644
@@ -1289,6 +1289,20 @@ struct bpf_tramp_nodes {
 	int nr_nodes;
 };
 
+/*
+ * Which 8-byte ctx slots of a struct_ops trampoline hold arena kernel
+ * pointers that save_args() converts to the arena pointer form,
+ * ctx[slot] = (u32)(kaddr - kern_vm_start).
+ */
+struct bpf_tramp_arena_args {
+	u32 slots;
+	u32 nullable_slots;	/* subset of @slots where NULL is preserved */
+	u64 kern_vm_start;
+};
+
+bool bpf_tramp_collect_arena_args(struct bpf_tramp_nodes *tnodes, u32 flags,
+				  struct bpf_tramp_arena_args *aargs);
+
 struct bpf_tramp_run_ctx;
 
 /* Different use cases for BPF trampoline:
@@ -1690,6 +1704,11 @@ struct bpf_ctx_arg_aux {
 	u32 btf_id;
 	u32 ref_id;
 	bool refcounted;
+	/*
+	 * We don't encode NULL-ness in the type for the program, but still need
+	 * to distinguish it for the purposes of telling JITs what sequence to emit.
+	 */
+	bool arena_nullable;
 };
 
 struct btf_mod_pair {
kernel/bpf/bpf_struct_ops.c +39 -14

102 lines not on this page · raw message

kernel/bpf/btf.c +7 -3
index 4ff6148ae8e8..6606187ed4f4 100644
@@ -6963,15 +6963,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
 		return false;
 	}
 
-	/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */
+	/*
+	 * Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or
+	 * PTR_TO_ARENA (both nullable and non-nullable cases).
+	 */
 	for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
 		const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
 		u32 type, flag;
 
 		type = base_type(ctx_arg_info->reg_type);
 		flag = type_flag(ctx_arg_info->reg_type);
-		if (ctx_arg_info->offset == off && type == PTR_TO_BUF &&
-		    (flag & PTR_MAYBE_NULL)) {
+		if (ctx_arg_info->offset == off &&
+		    (type == PTR_TO_ARENA ||
+		     (type == PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) {
 			info->reg_type = ctx_arg_info->reg_type;
 			return true;
 		}
kernel/bpf/trampoline.c +54 -0

66 lines not on this page · raw message

kernel/bpf/verifier.c +18 -5
index 2b7f6f6bbe76..6897b08dd010 100644
@@ -18872,6 +18872,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
 {
 	const struct btf_type *t, *func_proto;
 	const struct bpf_struct_ops_desc *st_ops_desc;
+	const struct bpf_struct_ops_arg_info *arg_info;
 	const struct bpf_struct_ops *st_ops;
 	const struct btf_member *member;
 	struct bpf_prog *prog = env->prog;
@@ -18950,10 +18951,23 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
 		return -EACCES;
 	}
 
-	for (i = 0; i < st_ops_desc->arg_info[member_idx].cnt; i++) {
-		if (st_ops_desc->arg_info[member_idx].info[i].refcounted) {
+	arg_info = &st_ops_desc->arg_info[member_idx];
+	for (i = 0; i < arg_info->cnt; i++) {
+		const struct bpf_ctx_arg_aux *info = &arg_info->info[i];
+
+		if (info->refcounted)
 			has_refcounted_arg = true;
-			break;
+		if (base_type(info->reg_type) == PTR_TO_ARENA) {
+			if (!bpf_jit_supports_arena_args()) {
+				verbose(env, "JIT does not support arena arguments\n");
+				return -ENOTSUPP;
+			}
+			if (!prog->aux->arena) {
+				verbose(env,
+					"arena argument of %s requires a program with an associated arena\n",
+					mname);
+				return -EINVAL;
+			}
 		}
 	}
 
@@ -18974,8 +18988,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
 	prog->aux->attach_func_name = mname;
 	env->ops = st_ops->verifier_ops;
 
-	return bpf_prog_ctx_arg_info_init(prog, st_ops_desc->arg_info[member_idx].info,
-					  st_ops_desc->arg_info[member_idx].cnt);
+	return bpf_prog_ctx_arg_info_init(prog, arg_info->info, arg_info->cnt);
 }
 #define SECURITY_PREFIX "security_"
 
Signature · 1 line
2.53.0
#15 lore raw

Re: [PATCH bpf-next v4 05/13] bpf: Support __arena and __arena__nullable on struct_ops arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Extension (`freplace`) and tracing (`fentry`/`fexit`) programs attached to `struct_ops`…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Extension (freplace) and tracing (fentry/fexit) programs attached to struct_ops programs with __arena arguments bypass type safety, treating user-controlled 32-bit arena offsets as trusted kernel pointers (PTR_TO_BTF_ID). --

--- Patch [5]: [PATCH bpf-next v4 05/13] bpf: Support __arena and __arena__nullable on struct_ops arguments ---
(Note: The formatting of this report is altered due to system recitation restrictions. I am providing a free-form summary of the findings rather than quoting the original code inline.)

commit fe44e46765df675218e6d1c19ad3b275b47f1966
Author: Tejun Heo <[email protected]>
Subject: bpf: Support __arena and __arena__nullable on struct_ops arguments

This patch adds support for __arena and __arena__nullable suffixes on BPF struct_ops arguments. It allows passing arena pointers directly to struct_ops callbacks by converting kernel addresses to arena offsets in the trampoline context array.

[Severity: Critical]
Does this allow extension (freplace) and tracing (fentry/fexit) programs to bypass type safety when attaching to struct_ops programs that use __arena arguments?

In kernel/bpf/btf.c:btf_ctx_access(), the verifier iterates over prog->aux->ctx_arg_info_size to enforce PTR_TO_ARENA. However, for extension and tracing programs, ctx_arg_info_size is 0. This causes the verifier to skip the arena type check and fall through, incorrectly typing the argument as a trusted kernel pointer (PTR_TO_BTF_ID) via the target program translation.

During attachment in kernel/bpf/trampoline.c:__bpf_trampoline_link_prog(), the safeguard calls bpf_prog_has_arena_ctx_arg(node->link->prog) on the attaching program instead of the target program. Because the attaching program lacks ctx_arg_info, this check returns false and allows the unsafe attachment to proceed.

At runtime, the struct_ops trampoline converts the kernel pointer to a 32-bit arena offset in the context array. Since the extension or tracing program was verified believing this argument is a PTR_TO_BTF_ID, could this allow it to dereference a user-controlled 32-bit arena offset as a trusted kernel pointer?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
maintainer
#16 lore raw

Re: [PATCH bpf-next v4 05/13] bpf: Support __arena and __arena__nullable on struct_ops arguments

In reply to Kumar Kartikeya Dwivedi

On Wed, 2026-08-05 at 23:04 +0200, Kumar Kartikeya Dwivedi wrote:

Lgtm, but we botched communication regarding the squash.

Quoted 12 lines
include/linux/bpf.h +0 -0
index f47556b56a48..0d1773af8373 100644
@@ -1289,6 +1289,20 @@ struct bpf_tramp_nodes {
  	int nr_nodes;
  };

include/linux/bpf.h +4 -0
+/*
+ * Which 8-byte ctx slots of a struct_ops trampoline hold arena kernel
+ * pointers that save_args() converts to the arena pointer form,
+ * ctx[slot] = (u32)(kaddr - kern_vm_start).

include/linux/bpf.h struct bpf_tramp_nodes { · line 1305

Quoted 10 lines
include/linux/bpf.h +10 -0
+ */
+struct bpf_tramp_arena_args {
+	u32 slots;
+	u32 nullable_slots;	/* subset of @slots where NULL is preserved */
+	u64 kern_vm_start;
+};
+
+bool bpf_tramp_collect_arena_args(struct bpf_tramp_nodes *tnodes, u32 flags,
+				  struct bpf_tramp_arena_args *aargs);
+

"bpf, x86: Convert struct_ops arena arguments in the trampoline" removes the type and rewrites bpf_tramp_collect_arena_args() as bpf_tramp_arena_base(). Please move these definitions here to avoid unnecessary churn. Same for 'arena_nullable' field below.

Quoted 16 lines

 struct bpf_tramp_run_ctx;

 /* Different use cases for BPF trampoline:

include/linux/bpf.h +5 -0
@@ -1690,6 +1704,11 @@ struct bpf_ctx_arg_aux {
  	u32 btf_id;
  	u32 ref_id;
  	bool refcounted;
+	/*
+	 * We don't encode NULL-ness in the type for the program, but still need
+	 * to distinguish it for the purposes of telling JITs what sequence to emit.
+	 */
+	bool arena_nullable;
  };


 struct btf_mod_pair {

kernel/bpf/bpf_struct_ops.c +0 -0

...

Quoted 34 lines
kernel/bpf/bpf_struct_ops.c +8 -4
@@ -225,27 +229,35 @@ static int prepare_arg_info(struct btf *btf,
  	/* Prepare info for every nullable argument */
  	info = info_buf;
  	for (arg_no = 0; arg_no < nargs; arg_no++) {
-		/* Skip arguments that is not suffixed with
-		 * "__nullable or __ref".
+		/*
+		 * Skip arguments that are not suffixed with "__arena__nullable",
+		 * "__arena", "__nullable", or "__ref".
  		 */
-		is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
-						     MAYBE_NULL_SUFFIX);
+		is_arena_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
+							   ARENA_MAYBE_NULL_SUFFIX);
+		is_arena = btf_param_match_suffix(btf, &stub_args[arg_no], ARENA_SUFFIX);
+		is_nullable = !is_arena_nullable &&
+			      btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX);
  		is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no],
  						       REFCOUNTED_SUFFIX);

kernel/bpf/bpf_struct_ops.c +5 -1
-		if (is_nullable)
+		if (is_arena_nullable)
+			suffix = ARENA_MAYBE_NULL_SUFFIX;
+		else if (is_arena)
+			suffix = ARENA_SUFFIX;
+		else if (is_nullable)

  suffix = MAYBE_NULL_SUFFIX; else if (is_refcounted) suffix = REFCOUNTED_SUFFIX; else
  continue;

kernel/bpf/bpf_struct_ops.c +0 -2
-		/* Should be a pointer to struct */
-		pointed_type = btf_type_resolve_ptr(btf,

kernel/bpf/bpf_struct_ops.c static int prepare_arg_info(struct btf *btf, · line 260

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +6 -4
-						    args[arg_no].type,
-						    &arg_btf_id);
-		if (!pointed_type ||
-		    !btf_type_is_struct(pointed_type)) {
+		/*
+		 * Should be a pointer to struct, or any pointer for __arena or
+		 * __arena__nullable.
+		 */
+		pointed_type = btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id);
+		if (!pointed_type || (!is_arena && !is_arena_nullable && !btf_type_is_struct(pointed_type))) {

Nit: it is so painful to read conditions like this, adding two variables 'ptr_to_struct'
and 'ptr_to_arena' would make it much nicer. (since respin is needed anyway).

...

author maintainer
#17 lore raw

PATCH v4 6/13 bpf, x86: JIT __arena kfunc argument rebasing

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Implement arena argument rebasing for kfunc calls on x86. R12 already holds kern_vm_start whenever the prog has an arena, so each tagged argument costs two instructions emitted right before the call:

movl %eN, %eN /* truncate, clear the upper 32 bits */
addq %r12, %rN

A nullable argument tests the truncated value and jumps over the add:

movl %eN, %eN
testl %eN, %eN
jz 1f
addq %r12, %rN
1:

addq carries a REX prefix for every argument register and is always three bytes, so the jz displacement is constant. The sequence is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees the rebase and needs no special handling.

bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well.

---
 arch/x86/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
arch/x86/net/bpf_jit_comp.c +50 -0
index 01e7ce569c1e..817977797e59 100644
@@ -1678,6 +1678,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 	return 0;
 }
 
+/*
+ * Rebase the __arena args of a kfunc call to arena kernel addresses,
+ * rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
+ * arg preserves NULL by skipping the add, tested on the truncated value as
+ * arena NULL is offset 0. Return the number of emitted bytes.
+ */
+static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
+				 const struct bpf_insn *insn, u8 **pprog)
+{
+	const struct btf_func_model *fm;
+	u8 *prog = *pprog;
+	u8 *start = prog;
+	int i;
+
+	fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+	if (!fm)
+		return -EINVAL;
+
+	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
+		u8 flags = fm->arg_flags[i];
+		u32 reg = BPF_REG_1 + i;
+
+		if (!(flags & BTF_FMODEL_ARENA_ARG))
+			continue;
+		if (WARN_ON_ONCE(!bpf_prog->aux->arena))
+			return -EINVAL;
+
+		/* mov eN, eN: truncate and clear the upper 32 bits */
+		emit_mov_reg(&prog, false, reg, reg);
+		if (flags & BTF_FMODEL_NULLABLE_ARG) {
+			/* test eN, eN; jz over the 3-byte add */
+			maybe_emit_mod(&prog, reg, reg, false);
+			EMIT2(0x85, add_2reg(0xC0, reg, reg));
+			EMIT2(X86_JE, 3);
+		}
+		/* add rN, r12 */
+		maybe_emit_mod(&prog, reg, X86_REG_R12, true);
+		EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
+	}
+
+	*pprog = prog;
+	return prog - start;
+}
+
 static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
 		  u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
 {
@@ -2583,6 +2627,12 @@ st:			insn_off = insn->off;
 			}
 			if (!imm32)
 				return -EINVAL;
+			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
+				err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+				if (err < 0)
+					return err;
+				ip += err;
+			}
 			if (priv_frame_ptr) {
 				push_r9(&prog);
 				ip += 2;
Signature · 1 line
2.53.0
author maintainer
#18 lore raw

PATCH v4 7/13 bpf, x86: Convert struct_ops arena arguments in the trampoline

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Implement the struct_ops arena argument conversion on x86. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through RAX:

movl %esrc, %eax /* truncate and clear the upper 32 bits */
subl $base_lo, %eax
movq %rax, ctx_slot

A nullable argument tests the full 64-bit kernel pointer first:

movq %rsrc, %rax
testq %rax, %rax
jz 1f
subl $base_lo, %eax
1:
movq %rax, ctx_slot

The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through RAX, so only the subtraction (and the NULL test) is inserted there.

Keeping arena and nullable classification in btf_func_model avoids a parallel trampoline slot bitmap. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline; other trampolines pass zero and perform no conversion. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction.

With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for x86.

---
 arch/x86/net/bpf_jit_comp.c | 57 +++++++++++++++++++++++++++++++++----
 include/linux/bpf.h         | 22 ++++----------
 kernel/bpf/bpf_struct_ops.c |  1 -
 kernel/bpf/trampoline.c     | 41 +++++++++++---------------
 4 files changed, 74 insertions(+), 47 deletions(-)
arch/x86/net/bpf_jit_comp.c +51 -6
index 817977797e59..cb8ee0f3a642 100644
@@ -3043,8 +3043,35 @@ static int get_nr_used_regs(const struct btf_func_model *m)
 	return nr_used_regs;
 }
 
+/*
+ * Convert an arena kernel address into the arena pointer form on its way
+ * into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg
+ * preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit
+ * subtraction both truncates and clears the upper half, so the stored
+ * value satisfies the JIT invariant for arena pointer registers.
+ */
+static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
+{
+	u8 *prog = *pprog;
+
+	if (nullable) {
+		if (src_reg != BPF_REG_0)
+			emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
+		/* test rax, rax; jz over the 5-byte sub */
+		EMIT3(0x48, 0x85, 0xC0);
+		EMIT2(X86_JE, 5);
+	} else if (src_reg != BPF_REG_0) {
+		emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
+	}
+	/* sub eax, base_lo */
+	EMIT1_off32(0x2D, base_lo);
+
+	*pprog = prog;
+}
+
 static void save_args(const struct btf_func_model *m, u8 **prog,
-		      int stack_size, bool for_call_origin, u32 flags)
+		      int stack_size, bool for_call_origin, u32 flags,
+		      u64 arena_base)
 {
 	int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
 	bool use_jmp = bpf_trampoline_use_jmp(flags);
@@ -3056,6 +3083,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 	 * mov QWORD PTR [rbp-0x8],rsi
 	 */
 	for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) {
+		bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
+		bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
+
 		arg_regs = (m->arg_size[i] + 7) / 8;
 
 		/* According to the research of Yonghong, struct members
@@ -3089,6 +3119,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 			for (j = 0; j < arg_regs; j++) {
 				emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
 					 nr_stack_slots * 8 + 16 + (!use_jmp) * 8);
+				if (arena_arg)
+					emit_arena_arg_conv(prog, BPF_REG_0, nullable,
+							    (u32)arena_base);
 				emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
 					 -stack_size);
 
@@ -3109,9 +3142,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 
 			/* copy the arguments from regs into stack */
 			for (j = 0; j < arg_regs; j++) {
-				emit_stx(prog, BPF_DW, BPF_REG_FP,
-					 nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
-					 -stack_size);
+				u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
+
+				if (arena_arg) {
+					emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
+					src = BPF_REG_0;
+				}
+				emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
 				stack_size -= 8;
 				nr_regs++;
 			}
@@ -3407,6 +3444,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	void *orig_call = func_addr;
 	int cookie_off, cookie_cnt;
 	u8 **branches = NULL;
+	u64 arena_base;
 	u64 func_meta;
 	u8 *prog;
 	bool save_ret;
@@ -3419,6 +3457,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) &&
 		     (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET)));
 
+	arena_base = bpf_tramp_arena_base(m, tnodes, flags);
+
 	for (i = 0; i < m->nr_args; i++)
 		nr_regs += (m->arg_size[i] + 7) / 8 - 1;
 
@@ -3553,7 +3593,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
 	}
 
-	save_args(m, &prog, regs_off, false, flags);
+	save_args(m, &prog, regs_off, false, flags, arena_base);
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
 		/* arg1: mov rdi, im */
@@ -3595,7 +3635,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
 		restore_regs(m, &prog, regs_off);
-		save_args(m, &prog, arg_stack_off, true, flags);
+		save_args(m, &prog, arg_stack_off, true, flags, 0);
 
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
 			/* Before calling the original function, load the
@@ -4096,6 +4136,11 @@ bool bpf_jit_supports_stack_args(void)
 	return true;
 }
 
+bool bpf_jit_supports_arena_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (text_poke_copy(dst, src, len) == NULL)
include/linux/bpf.h +6 -16
index 0d1773af8373..00aaa23b4f7e 100644
@@ -1290,18 +1290,13 @@ struct bpf_tramp_nodes {
 };
 
 /*
- * Which 8-byte ctx slots of a struct_ops trampoline hold arena kernel
- * pointers that save_args() converts to the arena pointer form,
- * ctx[slot] = (u32)(kaddr - kern_vm_start).
+ * The arena base against which a struct_ops trampoline converts the
+ * arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF
+ * ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline
+ * converts nothing.
  */
-struct bpf_tramp_arena_args {
-	u32 slots;
-	u32 nullable_slots;	/* subset of @slots where NULL is preserved */
-	u64 kern_vm_start;
-};
-
-bool bpf_tramp_collect_arena_args(struct bpf_tramp_nodes *tnodes, u32 flags,
-				  struct bpf_tramp_arena_args *aargs);
+u64 bpf_tramp_arena_base(const struct btf_func_model *m,
+			 struct bpf_tramp_nodes *tnodes, u32 flags);
 
 struct bpf_tramp_run_ctx;
 
@@ -1704,11 +1699,6 @@ struct bpf_ctx_arg_aux {
 	u32 btf_id;
 	u32 ref_id;
 	bool refcounted;
-	/*
-	 * We don't encode NULL-ness in the type for the program, but still need
-	 * to distinguish it for the purposes of telling JITs what sequence to emit.
-	 */
-	bool arena_nullable;
 };
 
 struct btf_mod_pair {
kernel/bpf/bpf_struct_ops.c +0 -1
index ebfb7fff453e..ced7b18246a9 100644
@@ -288,7 +288,6 @@ static int prepare_arg_info(struct btf *btf,
 			 * precision around it, since it has no safety implication.
 			 */
 			info->reg_type = PTR_TO_ARENA;
-			info->arena_nullable = is_arena_nullable;
 			model->arg_flags[arg_no] |= BTF_FMODEL_ARENA_ARG;
 			if (is_arena_nullable)
 				model->arg_flags[arg_no] |= BTF_FMODEL_NULLABLE_ARG;
kernel/bpf/trampoline.c +17 -24

57 lines not on this page · raw message

Signature · 1 line
2.53.0
maintainer Acked-by on v4 7/13
#19 lore raw

Re: [PATCH bpf-next v4 07/13] bpf, x86: Convert struct_ops arena arguments in the trampoline

Quoted 33 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Implement the struct_ops arena argument conversion on x86. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through RAX:

  movl %esrc, %eax      /* truncate and clear the upper 32 bits */ subl $base_lo, %eax
  movq %rax, ctx_slot

A nullable argument tests the full 64-bit kernel pointer first:

  movq  %rsrc, %rax
  testq %rax, %rax
  jz    1f
  subl  $base_lo, %eax
1:
  movq  %rax, ctx_slot

The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through RAX, so only the subtraction (and the NULL test) is inserted there.

Keeping arena and nullable classification in btf_func_model avoids a parallel trampoline slot bitmap. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline; other trampolines pass zero and perform no conversion. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction.

Quoted 6 lines

With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for x86.

---

[But please fix the botched squash discussed earlier]

...

author maintainer
#20 lore raw

PATCH v4 8/13 selftests/bpf: Add kfunc __arena and __arena__nullable argument tests

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar), the exact rebase semantics via capture kfuncs returning the raw argument (zero low 32 bits arrive as the arena kernel base under __arena and as NULL under __arena__nullable), a NULL round trip through a nullable deref kfunc, five arena arguments in one call, a mixed __arena plus __arena__nullable call exercising both bitmasks on one call site, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type).

The tests run on x86-64 and skip elsewhere, as programs with arena-tagged kfunc args fail verification where the JIT lacks support.

---
 .../selftests/bpf/prog_tests/verifier.c       |   8 +
 .../testing/selftests/bpf/progs/arena_kfunc.c | 260 ++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  57 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  10 +
 4 files changed, 335 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c
tools/testing/selftests/bpf/prog_tests/verifier.c +8 -0
index b79bafca68f7..7eaee71783b6 100644
@@ -2,6 +2,7 @@
 
 #include <test_progs.h>
 
+#include "arena_kfunc.skel.h"
 #include "cap_helpers.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
@@ -161,6 +162,13 @@ static void run_tests_aux(const char *skel_name,
 
 #define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL)
 
+/*
+ * The test kfuncs live in bpf_testmod. Resolving kfuncs against module
+ * BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of
+ * through the verifier tests' capability-restricted runner.
+ */
+void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
+
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
 void test_verifier_arena(void)                { RUN(verifier_arena); }
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +260 -0

260 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +57 -0

69 lines not on this page · raw message

tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +10 -0
index c36bb911defa..3317436b03a3 100644
@@ -98,6 +98,16 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym;
 void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym;
 
 void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym;
+__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym;
+__u64 bpf_kfunc_arena_nullable_arg_test(__u64 *val__arena__nullable) __ksym;
+__u64 bpf_kfunc_arena_cap_test(__u64 *val__arena) __ksym;
+__u64 bpf_kfunc_arena_cap_nullable_test(__u64 *val__arena__nullable) __ksym;
+__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena,
+				 __u64 *c__arena, __u64 *d__arena,
+				 __u64 *e__arena__nullable) __ksym;
+__u64 bpf_kfunc_arena_stack_arg_test(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				     __u64 *f__arena) __ksym;
+__u64 bpf_kfunc_arena_mixed_test(__u64 *a__arena, __u64 *b__arena__nullable) __ksym;
 int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) __ksym;
 int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
 int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
Signature · 1 line
2.53.0
maintainer Acked-by on v4 8/13
#21 lore raw

Re: [PATCH bpf-next v4 08/13] selftests/bpf: Add kfunc __arena and __arena__nullable argument tests

Quoted 14 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar), the exact rebase semantics via capture kfuncs returning the raw argument (zero low 32 bits arrive as the arena kernel base under __arena and as NULL under __arena__nullable), a NULL round trip through a nullable deref kfunc, five arena arguments in one call, a mixed __arena plus __arena__nullable call exercising both bitmasks on one call site, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type).

Quoted 6 lines

The tests run on x86-64 and skip elsewhere, as programs with arena-tagged kfunc args fail verification where the JIT lacks support.

---

...

Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +0 -0
index b79bafca68f7..7eaee71783b6 100644
@@ -2,6 +2,7 @@


 #include <test_progs.h>

+#include "arena_kfunc.skel.h"
 #include "cap_helpers.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"

tools/testing/selftests/bpf/prog_tests/verifier.c +0 -0
@@ -161,6 +162,13 @@ static void run_tests_aux(const char *skel_name,
Quoted 8 lines


 #define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL)
+/*
+ * The test kfuncs live in bpf_testmod. Resolving kfuncs against module
+ * BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of
+ * through the verifier tests' capability-restricted runner.
+ */

Nit: useless comment.

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
+void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
+

 void test_verifier_align(void)                { RUN(verifier_align); } void test_verifier_and(void)                  { RUN(verifier_and); } void test_verifier_arena(void)                { RUN(verifier_arena); }

tools/testing/selftests/bpf/progs/arena_kfunc.c new file +0 -0
new file mode 100644
index 000000000000..d6c382ce81af

...

Quoted 35 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +35 -0
+SEC("syscall")
+__arch_x86_64
+__success __retval(0)
+int arena_arg_rebase(void *ctx)
+{
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+	u64 __arena *val;
+	u64 base, off;
+
+	val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
+	if (!val)
+		return 1;
+
+	base = bpf_kfunc_arena_cap_test(NULL);
+	if (!base)
+		return 2;
+
+	/* only the low 32 bits contribute */
+	stash = 0xbadc0ffe00000000;
+	if (bpf_kfunc_arena_cap_test((u64 *)stash) != base)
+		return 3;
+
+	off = (u32)(u64)val;
+	if (bpf_kfunc_arena_cap_test((u64 *)val) != base + off)
+		return 4;
+
+	if (bpf_kfunc_arena_cap_nullable_test(NULL) != 0)
+		return 5;
+
+	stash = 0xbadc0ffe00000000;
+	if (bpf_kfunc_arena_cap_nullable_test((u64 *)stash) != 0)
+		return 6;
+
+	if (bpf_kfunc_arena_cap_nullable_test((u64 *)val) != base + off)
+		return 7;

tools/testing/selftests/bpf/progs/arena_kfunc.c line 115

Quoted 10 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +10 -0
+
+	bpf_arena_free_pages(&arena, (void __arena *)val, 1);
+#endif
+	return 0;
+}
+
+SEC("syscall")
+__arch_x86_64
+__success __retval(0)
+int arena_arg_nullable(void *ctx)

Nit: This test seem to test the same thing as the group of
bpf_kfunc_arena_cap_nullable_test() in the previous test. I'd drop the one or the other.

Quoted 21 lines
tools/testing/selftests/bpf/progs/arena_kfunc.c new file +21 -0
+{
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+	u64 __arena *val;
+	u64 ret;
+
+	val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
+	if (!val)
+		return 1;
+
+	*val = 41;
+	ret = bpf_kfunc_arena_nullable_arg_test((u64 *)val);
+	if (ret != 41 || *val != 42)
+		return 2;
+
+	if (bpf_kfunc_arena_nullable_arg_test(NULL) != 0xdeadbeef)
+		return 3;
+
+	bpf_arena_free_pages(&arena, (void __arena *)val, 1);
+#endif
+	return 0;
+}

...

author maintainer
#22 lore raw

PATCH v4 9/13 selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Pin the exact rebase sequences the JITs emit for __arena and __arena__nullable kfunc arguments with __jited assertions on x86-64: the unconditional truncate-and-add, the nullable test-and-skip variant, and all five argument registers in one call, which also covers the REX-prefixed encoding of r8 on x86. The capture kfuncs take the argument without dereferencing, so only the emitted code is under test. The tests skip without LLVM disassembler support.

---
 .../selftests/bpf/prog_tests/verifier.c       |  3 +
 .../selftests/bpf/progs/arena_kfunc_jit.c     | 98 +++++++++++++++++++
 2 files changed, 101 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc_jit.c
tools/testing/selftests/bpf/prog_tests/verifier.c +3 -0
index 7eaee71783b6..110a33b6e1c2 100644
@@ -3,6 +3,7 @@
 #include <test_progs.h>
 
 #include "arena_kfunc.skel.h"
+#include "arena_kfunc_jit.skel.h"
 #include "cap_helpers.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
@@ -169,6 +170,8 @@ static void run_tests_aux(const char *skel_name,
  */
 void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
+void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
+
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
 void test_verifier_arena(void)                { RUN(verifier_arena); }
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c new file +98 -0

98 lines not on this page · raw message

Signature · 1 line
2.53.0
maintainer Acked-by on v4 9/13
#23 lore raw

Re: [PATCH bpf-next v4 09/13] selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments

Quoted 10 lines · Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Pin the exact rebase sequences the JITs emit for __arena and __arena__nullable kfunc arguments with __jited assertions on x86-64: the unconditional truncate-and-add, the nullable test-and-skip variant, and all five argument registers in one call, which also covers the REX-prefixed encoding of r8 on x86. The capture kfuncs take the argument without dereferencing, so only the emitted code is under test. The tests skip without LLVM disassembler support.

Quoted 3 lines

---

...

author maintainer
#24 lore raw

PATCH v4 10/13 selftests/bpf: Add struct_ops __arena and __arena__nullable argument tests

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Add test_arena and test_arena_nullable members to bpf_testmod_ops3 with arena-tagged stub arguments and kfuncs that forward a caller-provided pointer to them. The kfuncs take arena-tagged arguments, so each round trip exercises both conversion directions end to end: the kfunc receives a kernel arena address and the trampoline converts it back to an arena pointer for the callback.

The non-nullable callback dereferences its argument with no NULL branch and captures the raw ctx value, which the trigger program compares against the arena offset of the passed object, pinning the exact (u32)(kaddr - kern_vm_start) conversion. The nullable callback verifies that only a true kernel NULL arrives as NULL. Failure coverage: a program with no arena is rejected when it loads. The tests run on x86-64 and skip elsewhere, as the programs fail verification where the JIT lacks arena argument support.

---
 .../bpf/prog_tests/test_struct_ops_arena.c    | 74 +++++++++++++++
 .../selftests/bpf/progs/struct_ops_arena.c    | 94 +++++++++++++++++++
 .../bpf/progs/struct_ops_arena_fail.c         | 20 ++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 24 +++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |  3 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  2 +
 6 files changed, 217 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c
 create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file +74 -0
new file mode 100644
index 000000000000..323d707c543f
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+
+#include "struct_ops_arena.skel.h"
+#include "struct_ops_arena_fail.skel.h"
+
+#if defined(__x86_64__)
+/*
+ * Attach callbacks with __arena and __arena__nullable arguments and drive
+ * them through the bpf_testmod_ops3_call_test_arena*() kfuncs.
+ */
+static void arena_arg(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, topts);
+	struct struct_ops_arena *skel;
+	struct bpf_link *link = NULL;
+	int err;
+
+	skel = struct_ops_arena__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load"))
+		return;
+
+	link = bpf_map__attach_struct_ops(skel->maps.testmod_arena);
+	if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
+		goto out;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger),
+				     &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, 0, "trigger_retval");
+
+out:
+	bpf_link__destroy(link);
+	struct_ops_arena__destroy(skel);
+}
+
+/*
+ * A program with no arena cannot attach to a member with an __arena
+ * argument.
+ */
+static void arena_arg_fail(void)
+{
+	struct struct_ops_arena_fail *skel;
+
+	skel = struct_ops_arena_fail__open_and_load();
+	if (ASSERT_ERR_PTR(skel, "struct_ops_arena_fail__open_and_load"))
+		return;
+
+	struct_ops_arena_fail__destroy(skel);
+}
+#endif
+
+/*
+ * Serialized because it attaches the singleton bpf_testmod_ops3, which
+ * test_struct_ops_private_stack also attaches; registering it twice fails
+ * with -EEXIST.
+ */
+void serial_test_struct_ops_arena(void)
+{
+	/*
+	 * Arena struct_ops arguments need JIT support, currently x86-64 only.
+	 * Elsewhere verification fails with "JIT does not support arena
+	 * arguments", so the programs cannot even load.
+	 */
+#if defined(__x86_64__)
+	if (test__start_subtest("arena_arg"))
+		arena_arg();
+	if (test__start_subtest("arena_arg_fail"))
+		arena_arg_fail();
+#else
+	test__skip();
+#endif
+}
tools/testing/selftests/bpf/progs/struct_ops_arena.c new file +94 -0

94 lines not on this page · raw message

tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c new file +20 -0
new file mode 100644
index 000000000000..1c0ec727d637
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod.h"
+
+char _license[] SEC("license") = "GPL";
+
+/* No arena in the program: attaching to test_arena must be rejected. */
+SEC("struct_ops/test_arena")
+int test_arena_no_arena(unsigned long long *ctx)
+{
+	return 0;
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_ops3 testmod_arena_fail = {
+	.test_arena = (void *)test_arena_no_arena,
+};
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +24 -0
index b92ac1e5df1d..64ca43744c59 100644
@@ -397,9 +397,21 @@ static int bpf_testmod_test_4(void)
 	return 0;
 }
 
+static int bpf_testmod_ops3__test_arena(u64 *ptr__arena)
+{
+	return 0;
+}
+
+static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable)
+{
+	return 0;
+}
+
 static struct bpf_testmod_ops3 __bpf_testmod_ops3 = {
 	.test_1 = bpf_testmod_test_3,
 	.test_2 = bpf_testmod_test_4,
+	.test_arena = bpf_testmod_ops3__test_arena,
+	.test_arena_nullable = bpf_testmod_ops3__test_arena_nullable,
 };
 
 static void bpf_testmod_test_struct_ops3(void)
@@ -418,6 +430,16 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void)
 	st_ops3->test_2();
 }
 
+__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena)
+{
+	return st_ops3->test_arena(ptr__arena);
+}
+
+__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nullable)
+{
+	return st_ops3->test_arena_nullable(ptr__arena__nullable);
+}
+
 struct bpf_testmod_btf_type_tag_1 {
 	int a;
 };
@@ -827,6 +849,8 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE)
 BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1)
 BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2)
+BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena)
+BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable)
 BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test);
 BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test);
 BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +3 -0
index 863fd10f1619..c367ec856776 100644
@@ -106,6 +106,9 @@ struct bpf_testmod_ops2 {
 struct bpf_testmod_ops3 {
 	int (*test_1)(void);
 	int (*test_2)(void);
+	/* Used to test arena pointer arguments. */
+	int (*test_arena)(u64 *ptr);
+	int (*test_arena_nullable)(u64 *ptr);
 };
 
 struct st_ops_args {
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +2 -0
index 3317436b03a3..a5b6b5db0dd5 100644
@@ -121,6 +121,8 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;
 #endif
 
 void bpf_testmod_test_mod_kfunc(int i) __ksym;
+int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym;
+int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym;
 
 __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
 				__u32 c, __u64 d) __ksym;
Signature · 1 line
2.53.0
#25 lore raw

Re: [PATCH bpf-next v4 10/13] selftests/bpf: Add struct_ops __arena and __arena__nullable argument tests

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing NULL check for the global pointer `st_ops3` and its function pointers in…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing NULL check for the global pointer st_ops3 and its function pointers in bpf_testmod_ops3_call_test_arena and bpf_testmod_ops3_call_test_arena_nullable kfuncs. --

--- Patch [10]: [PATCH bpf-next v4 10/13] selftests/bpf: Add struct_ops __arena and __arena__nullable argument tests --- commit c92a05fea3ca93b4550591e23b6734b40facb5bc
Author: Tejun Heo <[email protected]>

selftests/bpf: Add struct_ops __arena and __arena__nullable argument tests

Add test_arena and test_arena_nullable members to bpf_testmod_ops3 with arena-tagged stub arguments and kfuncs that forward a caller-provided pointer to them.

Quoted 4 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +0 -0
index b92ac1e5df1d2..64ca43744c593 100644

[ ... ]

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) · line 435

Quoted 7 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +3 -0
@@ -418,6 +430,16 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void)
 	st_ops3->test_2();
 }
 
+__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena)
+{
+	return st_ops3->test_arena(ptr__arena);

[Severity: Medium]
Will this cause a kernel panic if a BPF program calls this kfunc before the struct_ops is attached, leaving st_ops3 as NULL?

Additionally, if a user provides a struct_ops map that omits the test_arena callback, could st_ops3->test_arena be NULL, leading to a NULL function pointer dereference?

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) · line 440

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +5 -0
+}
+
+__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nullable)
+{
+	return st_ops3->test_arena_nullable(ptr__arena__nullable);

[Severity: Medium]
Similarly, can st_ops3 or st_ops3->test_arena_nullable be NULL here under the same conditions, causing a panic?

Quoted 5 lines
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +2 -0
+}
+
 struct bpf_testmod_btf_type_tag_1 {
 	int a;
 };
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author maintainer
#26 lore raw

PATCH v4 11/13 bpf, x86: Fix stack-passed arguments for indirect trampolines

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

save_args() reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of a struct_ops callback with more than six argument slots is read one slot off.

This has gone unnoticed because no in-tree struct_ops member passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treat BPF_TRAMP_F_INDIRECT the same way.

Fixes 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING")
Cc Jiri Olsa <[email protected]>
---
 arch/x86/net/bpf_jit_comp.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
arch/x86/net/bpf_jit_comp.c +8 -7
index cb8ee0f3a642..5c5c32aba239 100644
@@ -3075,6 +3075,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 {
 	int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
 	bool use_jmp = bpf_trampoline_use_jmp(flags);
+	int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24;
 	int i, j;
 
 	/* Store function arguments to stack.
@@ -3109,16 +3110,16 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 			/* copy function arguments from origin stack frame
 			 * into current stack frame.
 			 *
-			 * The starting address of the arguments on-stack
-			 * is:
-			 *   rbp + 8(push rbp) +
-			 *   8(return addr of origin call) +
-			 *   8(return addr of the caller)
-			 * which means: rbp + 24
+			 * The arguments on-stack start above the saved rbp
+			 * and the return addresses: two return addresses
+			 * (origin call and caller) when the trampoline is
+			 * entered through the fentry call, so rbp + 24, and
+			 * a single one when it is entered with a jmp or
+			 * called indirectly, so rbp + 16.
 			 */
 			for (j = 0; j < arg_regs; j++) {
 				emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
-					 nr_stack_slots * 8 + 16 + (!use_jmp) * 8);
+					 nr_stack_slots * 8 + stack_args_off);
 				if (arena_arg)
 					emit_arena_arg_conv(prog, BPF_REG_0, nullable,
 							    (u32)arena_base);
Signature · 1 line
2.53.0
#27 lore raw

Re: [PATCH bpf-next v4 11/13] bpf, x86: Fix stack-passed arguments for indirect trampolines

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Incomplete fix for stack-passed arguments in struct_ops trampolines leaves identical bugs unfixed on ARM64, RISC-V, and LoongArch --

--- Patch [11]: [PATCH bpf-next v4 11/13] bpf, x86: Fix stack-passed arguments for indirect trampolines --- commit 08733054ec575a092fdc6b844fb5b2af4d47355a
Author: Tejun Heo <[email protected]>

bpf, x86: Fix stack-passed arguments for indirect trampolines

This commit corrects the stack offset for reading stack-passed arguments in indirect trampolines on x86. It adjusts the offset from rbp + 24 to rbp + 16 to account for the single return address on the stack when the trampoline is called through a function pointer.

arch/x86/net/bpf_jit_comp.c static void save_args(const struct btf_func_model *m, u8 **prog, · line 3078

Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +1 -0
index cb8ee0f3a6423..5c5c32aba2396 100644
@@ -3075,6 +3075,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 {
 	int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
 	bool use_jmp = bpf_trampoline_use_jmp(flags);
+	int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24;

[Severity: High]
This is a pre-existing issue, but while fixing this for x86, are arm64, riscv, and loongarch also affected by the same struct_ops stack-passed arguments bug?

Looking at arm64 in arch/arm64/net/bpf_jit_comp.c:save_args(), the offset seems unconditionally hardcoded to 32, assuming an fentry stack frame layout:

soff = 32; /* on stack arguments start from FP + 32 */

Similarly for riscv in arch/riscv/net/bpf_jit_comp64.c:store_args():

emit_ld(RV_REG_T1, 16 + (i - RV_MAX_REG_ARGS) * 8, RV_REG_FP, ctx);

And loongarch in arch/loongarch/net/bpf_jit.c:store_args():

emit_insn(ctx, ldd, LOONGARCH_GPR_T1, LOONGARCH_GPR_FP,

          16 + (i - LOONGARCH_MAX_REG_ARGS) * 8);

Do these architectures also need a similar adjustment for BPF_TRAMP_F_INDIRECT to avoid reading out-of-bounds or garbage values from the stack for struct_ops callbacks with more than 8 arguments?

Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +0 -0
 	int i, j;
 
 	/* Store function arguments to stack.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author maintainer
#28 lore raw

PATCH v4 12/13 selftests/bpf: Test stack-passed struct_ops arena arguments

In reply to Kumar Kartikeya Dwivedi

From: Tejun Heo <[email protected]>

Add a test_arena_stack member with eight leading scalar arguments so the arena pointer is passed on the stack.

The callback validates the first and last scalar ctx slots before dereferencing the pointer in ctx[8]. This exercises the indirect trampoline stack layout and arena conversion together, and prevents a regression where stack arguments are read one slot late.

---
 .../selftests/bpf/progs/struct_ops_arena.c    | 21 +++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 14 +++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |  3 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  1 +
 4 files changed, 39 insertions(+)
tools/testing/selftests/bpf/progs/struct_ops_arena.c +21 -0
index 40c856a748d2..ba04c73d8d96 100644
@@ -46,10 +46,24 @@ int test_arena_nullable_cb(unsigned long long *ctx)
 	return 0;
 }
 
+SEC("struct_ops/test_arena_stack")
+int test_arena_stack_cb(unsigned long long *ctx)
+{
+	u64 __arena *ptr = (u64 __arena *)ctx[8];
+
+	arena_touch++;
+	/* pin the slot layout: the leading args fill ctx[0]..ctx[7] */
+	if (ctx[0] != 1 || ctx[7] != 8)
+		return 0xbad;
+	*ptr += 1;
+	return 0;
+}
+
 SEC(".struct_ops.link")
 struct bpf_testmod_ops3 testmod_arena = {
 	.test_arena = (void *)test_arena_cb,
 	.test_arena_nullable = (void *)test_arena_nullable_cb,
+	.test_arena_stack = (void *)test_arena_stack_cb,
 };
 
 SEC("syscall")
@@ -88,6 +102,13 @@ int trigger(void *ctx)
 	if (ret != 0xbee)
 		return 7;
 
+	/* the arena pointer is stack-passed into the trampoline here */
+	ret = bpf_testmod_ops3_call_test_arena_stack((u64 *)val);
+	if (ret)
+		return 8;
+	if (*val != 44)
+		return 9;
+
 	bpf_arena_free_pages(&arena, (void __arena *)val, 1);
 #endif
 	return 0;
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +14 -0
index 64ca43744c59..2963c29f96ba 100644
@@ -407,11 +407,19 @@ static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable)
 	return 0;
 }
 
+static int bpf_testmod_ops3__test_arena_stack(u64 a, u64 b, u64 c, u64 d,
+					      u64 e, u64 f, u64 g, u64 h,
+					      u64 *ptr__arena)
+{
+	return 0;
+}
+
 static struct bpf_testmod_ops3 __bpf_testmod_ops3 = {
 	.test_1 = bpf_testmod_test_3,
 	.test_2 = bpf_testmod_test_4,
 	.test_arena = bpf_testmod_ops3__test_arena,
 	.test_arena_nullable = bpf_testmod_ops3__test_arena_nullable,
+	.test_arena_stack = bpf_testmod_ops3__test_arena_stack,
 };
 
 static void bpf_testmod_test_struct_ops3(void)
@@ -440,6 +448,11 @@ __bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nulla
 	return st_ops3->test_arena_nullable(ptr__arena__nullable);
 }
 
+__bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena)
+{
+	return st_ops3->test_arena_stack(1, 2, 3, 4, 5, 6, 7, 8, ptr__arena);
+}
+
 struct bpf_testmod_btf_type_tag_1 {
 	int a;
 };
@@ -851,6 +864,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1)
 BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2)
 BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena)
 BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable)
+BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack)
 BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test);
 BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test);
 BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +3 -0
index c367ec856776..33f2af5b7085 100644
@@ -109,6 +109,9 @@ struct bpf_testmod_ops3 {
 	/* Used to test arena pointer arguments. */
 	int (*test_arena)(u64 *ptr);
 	int (*test_arena_nullable)(u64 *ptr);
+	/* enough leading args to force @ptr onto the stack on x86 and arm64 */
+	int (*test_arena_stack)(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f,
+				u64 g, u64 h, u64 *ptr);
 };
 
 struct st_ops_args {
tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +1 -0
index a5b6b5db0dd5..dc7c40bd1299 100644
@@ -123,6 +123,7 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;
 void bpf_testmod_test_mod_kfunc(int i) __ksym;
 int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym;
 int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym;
+int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym;
 
 __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
 				__u32 c, __u64 d) __ksym;
Signature · 1 line
2.53.0
author maintainer
#29 lore raw

PATCH v4 13/13 bpf: Reject tracing progs for struct_ops with arena args

In reply to Kumar Kartikeya Dwivedi

Reject tracing attachments to a target program with arena context arguments. The struct_ops indirect trampoline converts those arguments before entering the target, so a generic tracing trampoline would otherwise expose arena offsets using the target BTF pointer type.

---
 include/linux/bpf.h     |  1 +
 kernel/bpf/trampoline.c | 10 ----------
 kernel/bpf/verifier.c   | 21 +++++++++++++++++++++
 3 files changed, 22 insertions(+), 10 deletions(-)
include/linux/bpf.h +1 -0
index 00aaa23b4f7e..68ff66787ae1 100644
@@ -2367,6 +2367,7 @@ static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
 
 int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
 			       const struct bpf_ctx_arg_aux *info, u32 cnt);
+bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog);
 
 #if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM)
 int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
kernel/bpf/trampoline.c +0 -10
index b349e0817184..e07af35ed040 100644
@@ -529,16 +529,6 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
 	return tnodes;
 }
 
-static bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
-		if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
-			return true;
-	return false;
-}
-
 /*
  * The arena base against which save_args() converts the arguments marked
  * with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline
kernel/bpf/verifier.c +21 -0
index 6897b08dd010..60d1ea9a094a 100644
@@ -18868,6 +18868,16 @@ int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
 	return prog->aux->ctx_arg_info ? 0 : -ENOMEM;
 }
 
+bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
+{
+	int i;
+
+	for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
+		if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
+			return true;
+	return false;
+}
+
 static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
 {
 	const struct btf_type *t, *func_proto;
@@ -19255,6 +19265,17 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			bpf_log(log, "Subprog %s doesn't exist\n", tname);
 			return -EINVAL;
 		}
+		/*
+		 * A struct_ops indirect trampoline converts arena arguments
+		 * before invoking its program. A tracing program attached to the
+		 * main program would see the converted offset as a regular BTF
+		 * pointer.
+		 */
+		if (prog_tracing && subprog == 0 &&
+		    bpf_prog_has_arena_ctx_arg(tgt_prog)) {
+			bpf_log(log, "Cannot trace a target with arena context arguments\n");
+			return -EOPNOTSUPP;
+		}
 		if (aux->func && aux->func[subprog]->aux->exception_cb) {
 			bpf_log(log,
 				"%s programs cannot attach to exception callback\n",
Signature · 1 line
2.53.0
#30 lore raw

Re: [PATCH bpf-next v4 13/13] bpf: Reject tracing progs for struct_ops with arena args

Moving bpf_prog_has_arena_ctx_arg() from kernel/bpf/trampoline.c (where it was static) to kernel/bpf/verifier.c creates a link-time undefined symbol with CONFIG_BPF_JIT=y and CONFIG_BPF_SYSCALL=n?
Quoted 20 lines
include/linux/bpf.h +1 -0
index 00aaa23b4f7e..68ff66787ae1 100644
@@ -2367,6 +2367,7 @@ static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
 
 int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
 			       const struct bpf_ctx_arg_aux *info, u32 cnt);
+bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog);
 
 #if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM)
 int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
kernel/bpf/trampoline.c +0 -0
index b349e0817184..e07af35ed040 100644
@@ -529,16 +529,6 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
 	return tnodes;
 }
 

kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 541

Quoted 10 lines
kernel/bpf/trampoline.c +0 -10
-static bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
-		if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
-			return true;
-	return false;
-}
-

Moving bpf_prog_has_arena_ctx_arg() from kernel/bpf/trampoline.c (where it was static) to kernel/bpf/verifier.c creates a link-time undefined symbol with CONFIG_BPF_JIT=y and CONFIG_BPF_SYSCALL=n?

The two objects have different Kconfig gates in kernel/bpf/Makefile:

  obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o ...
  obj-$(CONFIG_BPF_JIT) += trampoline.o

So trampoline.o can be built without verifier.o. But the trampoline still calls bpf_prog_has_arena_ctx_arg() here:

kernel/bpf/trampoline.c:__bpf_trampoline_link_prog() {
    kind = bpf_attach_type_to_tramp(node->link->prog);
    /*
     * Arena ctx args are converted only by struct_ops indirect
     * trampolines. They must never be attached to a generic trampoline.
     */
    if (WARN_ON_ONCE(bpf_prog_has_arena_ctx_arg(node->link->prog)))
        return -ENOTSUPP;

The call is not guarded by any conditional. The function that contains it is also non-static and exported (bpf_trampoline_link_prog() calls it from outside any Kconfig guard).

The new prototype in include/linux/bpf.h sits outside every Kconfig block as well (just under the _LINUX_BPF_H include guard), and there is no stub for the !CONFIG_BPF_SYSCALL case.

arch/powerpc/configs/mpc885_ads_defconfig sets CONFIG_BPF_JIT=y and CONFIG_NET=y (which selects BPF) but never sets CONFIG_BPF_SYSCALL, and nothing it enables selects BPF_SYSCALL either. CONFIG_BPF_JIT only depends on BPF and HAVE_CBPF_JIT || HAVE_EBPF_JIT; PPC selects HAVE_EBPF_JIT unconditionally so that config is reachable.

Other cross-object calls from trampoline.c follow one of two patterns. The other verifier.c symbol, bpf_check_attach_btf_id_multi(), is called only inside a block that explicitly includes defined(CONFIG_BPF_SYSCALL). And bpf_prog_inc_misses_counter(), a syscall.c symbol, is declared inside ifdef CONFIG_BPF_SYSCALL with a static inline stub in the else.

Would it work to either add the same stub pattern for bpf_prog_has_arena_ctx_arg() in the header, or keep it static in trampoline.c and give verifier.c its own copy?

Quoted 3 lines
kernel/bpf/trampoline.c +0 -0
 /*
  * The arena base against which save_args() converts the arguments marked
  * with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline

[ ... ]

Quoted 8 lines
kernel/bpf/verifier.c +0 -0
index 6897b08dd010..60d1ea9a094a 100644
@@ -18868,6 +18868,16 @@ int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
 	return prog->aux->ctx_arg_info ? 0 : -ENOMEM;
 }
 

kernel/bpf/verifier.c int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog, · line 18880

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
+{
+	int i;
+
+	for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
+		if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
+			return true;
+	return false;
+}
+

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31047716344

maintainer
#31 lore raw

Re: [PATCH bpf-next v4 13/13] bpf: Reject tracing progs for struct_ops with arena args

In reply to Kumar Kartikeya Dwivedi

On Wed, 2026-08-05 at 23:04 +0200, Kumar Kartikeya Dwivedi wrote:

...

Quoted 8 lines
kernel/bpf/trampoline.c +0 -0
index b349e0817184..e07af35ed040 100644
@@ -529,16 +529,6 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
  	return tnodes;
  }

kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 540

Quoted 9 lines
kernel/bpf/trampoline.c +0 -9
-static bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
-{
-	int i;
-
-	for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
-		if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
-			return true;
-	return false;
-}

Let's pick one place for this function throughout the series and don't move it.

Also, a selftest?

...