Skip to contentSkip to conversation

bpf/verifier: Expand the usage scenarios of bpf_kptr_xchg

PATCH Superseded Chengkaitao 2 participants 6 messages Feb 3

Subsystems (4)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover bpf/verifier: Expand the usage scenarios of bpf_kptr_xchg

From: Chengkaitao Com <[email protected]>

When using bpf_kptr_xchg, we triggered the following error:
31: (85) call bpf_kptr_xchg#194
function calls are not allowed while holding a lock
bpf_kptr_xchg can now be used in lock-held contexts, so we extended its usage scope in [patch 1/2].

When writing test cases using bpf_kptr_xchg and bpf_rbtree_*, the following approach must be followed:

	bpf_spin_lock(&lock);
	rb_n = bpf_rbtree_root(&root);
	while (rb_n && can_loop) {
		rb_n = bpf_rbtree_remove(&root, rb_n);
		if (!rb_n)
			goto fail;

		tnode = container_of(rb_n, struct tree_node, node);
		node_data = bpf_kptr_xchg(&tnode->node_data, NULL);
		if (!node_data)
			goto fail;

		data = node_data->data;
		/* use data to do something */

		node_data = bpf_kptr_xchg(&tnode->node_data, node_data);
		if (node_data)
			goto fail;

		bpf_rbtree_add(&root, rb_n, less);

		if (lookup_key < tnode->key)
			rb_n = bpf_rbtree_left(&root, rb_n);
		else
			rb_n = bpf_rbtree_right(&root, rb_n);
	}
	bpf_spin_unlock(&lock);

The above illustrates a lock-remove-read-add-unlock workflow, which exhibits lower performance. To address this, we introduced support for a streamlined lock-read-unlock operation in [patch 2/2].

Changes in v5:
- add lastname
Changes in v4:
- Fix the dead logic issue in the test case
Changes in v3:
- Fix compilation errors
Changes in v2:
- Allow using bpf_kptr_xchg even if the NON_OWN_REF flag is set
- Add test case

Link to v4:
https://lore.kernel.org/all/[email protected]/
Link to V3:
https://lore.kernel.org/all/[email protected]/
Link to V2:
https://lore.kernel.org/all/[email protected]/
Link to V1:
https://lore.kernel.org/all/[email protected]/

Chengkaitao Com (3):
bpf/verifier: allow calling bpf_kptr_xchg while holding a lock bpf/verifier: allow using bpf_kptr_xchg even if the NON_OWN_REF flag
is set
selftests/bpf: Add supplementary tests for bpf_kptr_xchg

 kernel/bpf/verifier.c                         |   7 +-
 .../testing/selftests/bpf/prog_tests/rbtree.c |   6 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   4 +
 .../selftests/bpf/progs/rbtree_search_kptr.c  | 167 ++++++++++++++++++
 4 files changed, 182 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/rbtree_search_kptr.c
Signature · 1 line
2.50.1 (Apple Git-155)
#1 lore raw

PATCH v5 1/3 bpf/verifier: allow calling bpf_kptr_xchg while holding a lock

From: Chengkaitao Com <[email protected]>

For the following scenario:

struct tree_node {
    struct bpf_rb_node node;
    struct request __kptr *req;
    u64 key;
};

struct bpf_rb_root tree_root __contains(tree_node, node); struct bpf_spin_lock tree_lock;

If we need to traverse all nodes in the rbtree, retrieve the __kptr pointer from each node, and read kernel data from the referenced object, using bpf_kptr_xchg appears unavoidable.

This patch skips the BPF verifier checks for bpf_kptr_xchg when called while holding a lock.

---
 kernel/bpf/verifier.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +2 -1
index 3135643d5695..05a6a6606b6c 100644
@@ -20387,7 +20387,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 
 			if (env->cur_state->active_locks) {
 				if ((insn->src_reg == BPF_REG_0 &&
-				     insn->imm != BPF_FUNC_spin_unlock) ||
+				     insn->imm != BPF_FUNC_spin_unlock &&
+				     insn->imm != BPF_FUNC_kptr_xchg) ||
 				    (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
 				     (insn->off != 0 || !kfunc_spin_allowed(insn->imm)))) {
 					verbose(env,
Signature · 1 line
2.50.1 (Apple Git-155)
#2 lore raw

PATCH v5 2/3 bpf/verifier: allow using bpf_kptr_xchg even if the NON_OWN_REF flag is set

In reply to Chengkaitao

From: Chengkaitao Com <[email protected]>

When traversing an rbtree using bpf_rbtree_left/right, if bpf_kptr_xchg is used to access the __kptr pointer contained in a node, it currently requires first removing the node with bpf_rbtree_remove and clearing the NON_OWN_REF flag, then re-adding the node to the original rbtree with bpf_rbtree_add after usage. This process significantly degrades rbtree traversal performance. The patch enables accessing __kptr pointers with the NON_OWN_REF flag set while holding the lock, eliminating the need for this remove-read-readd sequence.

---
 kernel/bpf/verifier.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +3 -1
index 05a6a6606b6c..bb3ff4bbb3a2 100644
@@ -9260,7 +9260,8 @@ static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE }
 static const struct bpf_reg_types kptr_xchg_dest_types = {
 	.types = {
 		PTR_TO_MAP_VALUE,
-		PTR_TO_BTF_ID | MEM_ALLOC
+		PTR_TO_BTF_ID | MEM_ALLOC,
+		PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF
 	}
 };
 static const struct bpf_reg_types dynptr_types = {
@@ -9420,6 +9421,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
 	}
 	case PTR_TO_BTF_ID | MEM_ALLOC:
 	case PTR_TO_BTF_ID | MEM_PERCPU | MEM_ALLOC:
+	case PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF:
 		if (meta->func_id != BPF_FUNC_spin_lock && meta->func_id != BPF_FUNC_spin_unlock &&
 		    meta->func_id != BPF_FUNC_kptr_xchg) {
 			verifier_bug(env, "unimplemented handling of MEM_ALLOC");
Signature · 1 line
2.50.1 (Apple Git-155)
#3 lore raw

Re: [PATCH v5 2/3] bpf/verifier: allow using bpf_kptr_xchg even if the NON_OWN_REF flag is set

Is the MEM_RCU variant also needed here?

kernel/bpf/verifier.c static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE } · line 9306

Quoted 13 lines
kernel/bpf/verifier.c +2 -1
index 2dd860ae12..bb45fa5612 100644
@@ -9300,7 +9300,8 @@ static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE }
 static const struct bpf_reg_types kptr_xchg_dest_types = {
 	.types = {
 		PTR_TO_MAP_VALUE,
-		PTR_TO_BTF_ID | MEM_ALLOC
+		PTR_TO_BTF_ID | MEM_ALLOC,
+		PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF
 	}
 };

Is the MEM_RCU variant also needed here?

When ref_set_non_owning() marks a register as NON_OWN_REF, it also adds MEM_RCU if the node has a bpf_refcount field (refcount_off >= 0):

    reg->type |= NON_OWN_REF;
    if (rec->refcount_off >= 0)
        reg->type |= MEM_RCU;

This means rbtree nodes with both a __kptr field and a bpf_refcount field would have type PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF | MEM_RCU, which won't match any entry in kptr_xchg_dest_types.

Should this also add:

PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF | MEM_RCU

to handle that case?

kernel/bpf/verifier.c static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · line 9466

Quoted 7 lines
kernel/bpf/verifier.c +1 -0
@@ -9460,6 +9461,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
 	}
 	case PTR_TO_BTF_ID | MEM_ALLOC:
 	case PTR_TO_BTF_ID | MEM_PERCPU | MEM_ALLOC:
+	case PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF:
 		if (meta->func_id != BPF_FUNC_spin_lock && meta->func_id != BPF_FUNC_spin_unlock &&
 		    meta->func_id != BPF_FUNC_kptr_xchg) {

Similarly, does this switch need a case for the MEM_RCU variant?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21614799102

AI-authorship-score: low
AI-authorship-explanation: The commit addresses a specific practical problem with minimal targeted changes, showing domain expertise typical of human kernel developers rather than AI-generated comprehensive solutions.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: Missing MEM_RCU variant may prevent bpf_kptr_xchg on refcounted rbtree nodes, but this affects only a subset of use cases and causes verifier rejection rather than runtime issues.

#4 lore raw

Re: [PATCH v5 2/3] bpf/verifier: allow using bpf_kptr_xchg even if the NON_OWN_REF flag is set

Quoted 49 lines · On Tue, Feb 3, 2026 at 11:04 AM <[email protected]> wrote:
Quoted 6 lines
kernel/bpf/verifier.c +0 -0
index 2dd860ae12..bb45fa5612 100644

kernel/bpf/verifier.c static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE } · line 9306

Quoted 8 lines
kernel/bpf/verifier.c +2 -1
@@ -9300,7 +9300,8 @@ static const struct bpf_reg_types timer_types = { .types = { PTR_TO_MAP_VALUE }
 static const struct bpf_reg_types kptr_xchg_dest_types = {
      .types = {
              PTR_TO_MAP_VALUE,
-             PTR_TO_BTF_ID | MEM_ALLOC
+             PTR_TO_BTF_ID | MEM_ALLOC,
+             PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF
      }
 };

Is the MEM_RCU variant also needed here?

When ref_set_non_owning() marks a register as NON_OWN_REF, it also adds MEM_RCU if the node has a bpf_refcount field (refcount_off >= 0):

    reg->type |= NON_OWN_REF;
    if (rec->refcount_off >= 0)
        reg->type |= MEM_RCU;

This means rbtree nodes with both a __kptr field and a bpf_refcount field would have type PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF | MEM_RCU, which won't match any entry in kptr_xchg_dest_types.

Should this also add:

PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF | MEM_RCU

to handle that case?

kernel/bpf/verifier.c static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · line 9466

Quoted 7 lines
kernel/bpf/verifier.c +1 -0
@@ -9460,6 +9461,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
      }
      case PTR_TO_BTF_ID | MEM_ALLOC:
      case PTR_TO_BTF_ID | MEM_PERCPU | MEM_ALLOC:
+     case PTR_TO_BTF_ID | MEM_ALLOC | NON_OWN_REF:
              if (meta->func_id != BPF_FUNC_spin_lock && meta->func_id != BPF_FUNC_spin_unlock &&
                  meta->func_id != BPF_FUNC_kptr_xchg) {

Similarly, does this switch need a case for the MEM_RCU variant?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

Quoted 7 lines

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21614799102

AI-authorship-score: low
AI-authorship-explanation: The commit addresses a specific practical problem with minimal targeted changes, showing domain expertise typical of human kernel developers rather than AI-generated comprehensive solutions.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: Missing MEM_RCU variant may prevent bpf_kptr_xchg on refcounted rbtree nodes, but this affects only a subset of use cases and causes verifier rejection rather than runtime issues.

I will try to add this case. thanks

Signature · 2 lines
Yours,
Chengkaitao
#5 lore raw

PATCH v5 3/3 selftests/bpf: Add supplementary tests for bpf_kptr_xchg

In reply to Chengkaitao

From: Chengkaitao Com <[email protected]>

1. Allow using bpf_kptr_xchg while holding a lock.
2. When the rb_node contains a __kptr pointer, we do not need to perform a remove-read-add operation.

This patch implements the following workflow:
1. Construct a rbtree with 16 elements.
2. Traverse the rbtree, locate the kptr pointer in the target node, and read the content pointed to by the pointer.
3. Remove all nodes from the rbtree.

---
 .../testing/selftests/bpf/prog_tests/rbtree.c |   6 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   4 +
 .../selftests/bpf/progs/rbtree_search_kptr.c  | 167 ++++++++++++++++++
 3 files changed, 177 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/rbtree_search_kptr.c
tools/testing/selftests/bpf/prog_tests/rbtree.c +6 -0
index d8f3d7a45fe9..a854fb38e418 100644
@@ -9,6 +9,7 @@
 #include "rbtree_btf_fail__wrong_node_type.skel.h"
 #include "rbtree_btf_fail__add_wrong_type.skel.h"
 #include "rbtree_search.skel.h"
+#include "rbtree_search_kptr.skel.h"
 
 static void test_rbtree_add_nodes(void)
 {
@@ -193,3 +194,8 @@ void test_rbtree_search(void)
 {
 	RUN_TESTS(rbtree_search);
 }
+
+void test_rbtree_search_kptr(void)
+{
+	RUN_TESTS(rbtree_search_kptr);
+}
tools/testing/selftests/bpf/progs/bpf_misc.h +4 -0
index c9bfbe1bafc1..0904fe14ad1d 100644
@@ -188,6 +188,10 @@
 #define POINTER_VALUE	0xbadcafe
 #define TEST_DATA_LEN	64
 
+#ifndef __aligned
+#define __aligned(x) __attribute__((aligned(x)))
+#endif
+
 #ifndef __used
 #define __used __attribute__((used))
 #endif
tools/testing/selftests/bpf/progs/rbtree_search_kptr.c new file +167 -0

167 lines not on this page · raw message

Signature · 1 line
2.50.1 (Apple Git-155)