Skip to contentSkip to conversation

Introduce KF_FORBID_SLEEP modifier for acquire/release kfuncs

PATCH Superseded Puranjay Mohan 5 participants 14 messages Feb 18 – Feb 20

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Introduce KF_FORBID_SLEEP modifier for acquire/release kfuncs

Some BPF kfuncs acquire resources that prevent sleeping - a lock, a reference to an object under a lock, a preempt-disable section. Today there is no way for the verifier to know that holding a particular acquired reference means sleeping is forbidden. Programs either run entirely in sleepable or non-sleepable context, with no way to express "sleeping is forbidden right now, but will be allowed once I release this reference."

This series adds KF_FORBID_SLEEP, a new kfunc flag that can be combined with KF_ACQUIRE. When a kfunc annotated with KF_ACQUIRE | KF_FORBID_SLEEP is called, the verifier tags the acquired reference with forbid_sleep and increments a per-state forbid_sleep_count counter. When the reference is released (through corresponding KF_RELEASE kfunc), the counter is decremented. The verifier checks this counter everywhere it decides whether sleeping is allowed — sleepable helpers, sleepable kfuncs, global function calls, and resource leak checks.

This is fully generic. Any pair of KF_ACQUIRE/KF_RELEASE kfuncs can opt into sleep prohibition by adding KF_FORBID_SLEEP to the acquire side.

To make this useful for iterators specifically, the series first extends the verifier's iterator support to allow KF_ACQUIRE on _next and KF_RELEASE on a separate kfunc taking an __iter argument. The verifier tracks the acquired reference on the iterator's stack slot (st->id) and auto-releases it on the next _next call and on the DRAINED (NULL) path, so the acquire/release is transparent to programs that don't need mid-loop release.

Iterator KF_ACQUIRE support is not useful on its own right now, but it becomes the foundation for KF_FORBID_SLEEP: an iterator whose _next is annotated with KF_ACQUIRE | KF_FORBID_SLEEP can now express "holding this pointer forbids sleeping; calling _release invalidates the pointer and re-enables sleeping."

The task_vma iterator is the first user. It holds mmap_lock during iteration, preventing sleepable helpers like bpf_copy_from_user(). Currently, this can lead to a deadlock as the fault path also takes mmap_lock.

With this series, a BPF program can release the lock mid-iteration:

    bpf_for_each(task_vma, vma, task, 0) {
        u64 start = vma->vm_start;

        /* sleeping forbidden, but vma pointer access allowed */
        bpf_iter_task_vma_release(&___it);
        /* mmap_lock released, vma pointer invalidated */
        /* sleeping is fine here */

        bpf_copy_from_user(&buf, sizeof(buf), (void *)start);
    }

The series is organized as:

Patch 1: KF_ACQUIRE/KF_RELEASE plumbing for iterators in the
verifier. Pure infrastructure, no behavioral change to existing iterators.

Patch 2: KF_FORBID_SLEEP flag and forbid_sleep_count machinery.
Generic, works for any KF_ACQUIRE kfunc - iterator or not.

Patch 3: Move mmap_lock acquisition from _new to _next in the
task_vma iterator, preparing for re-acquisition after release.

Patch 4: Wire up task_vma as the first user: annotate _next with
KF_ACQUIRE | KF_FORBID_SLEEP, add bpf_iter_task_vma_release().

Patch 5: Selftests covering the runtime path (release + copy_from_user)
and verifier rejection of invalid patterns (sleeping without release, VMA access after release, double release, release without acquire, nested iterator interaction).

Puranjay Mohan (5):
bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators
bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs
bpf: Move locking to bpf_iter_task_vma_next()
bpf: Add split iteration support to task_vma iterator selftests/bpf: Add tests for split task_vma iterator

 include/linux/bpf_verifier.h                  |   2 +
 include/linux/btf.h                           |   1 +
 kernel/bpf/helpers.c                          |   3 +-
 kernel/bpf/task_iter.c                        |  44 ++++--
 kernel/bpf/verifier.c                         | 121 +++++++++++++++--
 .../testing/selftests/bpf/bpf_experimental.h  |   1 +
 .../testing/selftests/bpf/prog_tests/iters.c  |  13 ++
 .../selftests/bpf/progs/iters_task_vma.c      |  39 ++++++
 .../bpf/progs/iters_task_vma_nosleep.c        | 125 ++++++++++++++++++
 9 files changed, 325 insertions(+), 24 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma_nosleep.c

base-commit: 4c51f90d45dca71e7974ed5a7c40b9c04a6c6762

Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/5 bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators

Some iterators hold resources (like mmap_lock in task_vma) that prevent sleeping. To allow BPF programs to release such resources mid-iteration and call sleepable helpers, the verifier needs to track acquire/release semantics on iterator _next pointers.

Repurpose the st->id field on STACK_ITER slots to track the ref_obj_id of the pointer returned by _next when the kfunc is annotated with KF_ACQUIRE. This is safe because st->id is initialized to 0 by __mark_reg_known_zero() in mark_stack_slots_iter() and is not compared in stacksafe() for STACK_ITER slots.

The lifecycle is:

_next (KF_ACQUIRE):
- auto-release old ref if st->id != 0
- acquire new ref, store ref_obj_id in st->id
- DRAINED branch: release via st->id, set st->id = 0
- ACTIVE branch: keeps ref, st->id tracks it

_release (KF_RELEASE + __iter arg):
- read st->id, release_reference(), set st->id = 0

_destroy:
- release st->id if non-zero before releasing iterator's own ref

---
 kernel/bpf/verifier.c | 67 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 63 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +63 -4
index 0162f946032f..aa48180b6073 100644
@@ -355,6 +355,11 @@ struct bpf_kfunc_call_arg_meta {
 		u8 spi;
 		u8 frameno;
 	} iter;
+	/* Set when a kfunc takes an __iter arg. Used by the KF_RELEASE
+	 * path to release the reference tracked on the iterator slot
+	 * (st->id) instead of requiring a refcounted PTR_TO_BTF_ID arg.
+	 */
+	bool has_iter_arg;
 	struct bpf_map_desc map;
 	u64 mem_size;
 };
@@ -1083,6 +1088,22 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 	return 0;
 }
 
+/* Release the acquired reference tracked by iter_st->id, if any.
+ * Used during auto-release in _next, DRAINED handling, and _destroy.
+ */
+static int iter_release_acquired_ref(struct bpf_verifier_env *env,
+				     struct bpf_reg_state *iter_st)
+{
+	int err;
+
+	if (!iter_st->id)
+		return 0;
+	err = release_reference(env, iter_st->id);
+	if (!err)
+		iter_st->id = 0;
+	return err;
+}
+
 static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *reg, int nr_slots)
 {
@@ -1097,8 +1118,13 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
 		struct bpf_stack_state *slot = &state->stack[spi - i];
 		struct bpf_reg_state *st = &slot->spilled_ptr;
 
-		if (i == 0)
+		if (i == 0) {
+			/* Release any outstanding acquired ref tracked by
+			 * st->id before releasing the iterator's own ref.
+			 */
+			WARN_ON_ONCE(iter_release_acquired_ref(env, st));
 			WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
+		}
 
 		__mark_reg_not_init(env, st);
 
@@ -8943,6 +8969,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 		/* remember meta->iter info for process_iter_next_call() */
 		meta->iter.spi = spi;
 		meta->iter.frameno = reg->frameno;
+		meta->has_iter_arg = true;
 		meta->ref_obj_id = iter_ref_obj_id(env, reg, spi);
 
 		if (is_iter_destroy_kfunc(meta)) {
@@ -9178,8 +9205,10 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 	/* mark current iter state as drained and assume returned NULL */
 	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
 	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
-
-	return 0;
+	/* If _next acquired a ref (KF_ACQUIRE), release it in the DRAINED
+	 * branch since NULL was returned.
+	 */
+	return iter_release_acquired_ref(env, cur_iter);
 }
 
 static bool arg_type_is_mem_size(enum bpf_arg_type type)
@@ -13797,7 +13826,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		}
 	}
 
-	if (is_kfunc_release(meta) && !meta->release_regno) {
+	if (is_kfunc_release(meta) && !meta->release_regno && !meta->has_iter_arg) {
 		verbose(env, "release kernel function %s expects refcounted PTR_TO_BTF_ID\n",
 			func_name);
 		return -EINVAL;
@@ -14205,6 +14234,21 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		}
 		if (err)
 			return err;
+	} else if (meta.has_iter_arg && is_kfunc_release(&meta)) {
+		/* For KF_RELEASE kfuncs taking an __iter arg, release the
+		 * reference tracked by st->id on the iterator slot.
+		 */
+		struct bpf_reg_state *iter_st;
+
+		iter_st = get_iter_from_state(env->cur_state, &meta);
+		if (!iter_st->id) {
+			verbose(env, "no acquired reference to release\n");
+			return -EINVAL;
+		}
+		err = release_reference(env, iter_st->id);
+		if (err)
+			return err;
+		iter_st->id = 0;
 	}
 
 	if (meta.func_id == special_kfunc_list[KF_bpf_list_push_front_impl] ||
@@ -14356,6 +14400,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			regs[BPF_REG_0].id = ++env->id_gen;
 		}
 		mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *));
+		/* For iterators with KF_ACQUIRE, auto-release the previous
+		 * iteration's ref before acquiring a new one, and after
+		 * acquisition track the new ref on the iter slot.
+		 */
+		struct bpf_reg_state *iter_acquire_st = NULL;
+
+		if (is_iter_next_kfunc(&meta) && is_kfunc_acquire(&meta)) {
+			iter_acquire_st = get_iter_from_state(env->cur_state, &meta);
+			err = iter_release_acquired_ref(env, iter_acquire_st);
+			if (err)
+				return err;
+		}
 		if (is_kfunc_acquire(&meta)) {
 			int id = acquire_reference(env, insn_idx);
 
@@ -14368,6 +14424,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			ref_set_non_owning(env, &regs[BPF_REG_0]);
 		}
 
+		if (iter_acquire_st)
+			iter_acquire_st->id = regs[BPF_REG_0].ref_obj_id;
+
 		if (reg_may_point_to_spin_lock(&regs[BPF_REG_0]) && !regs[BPF_REG_0].id)
 			regs[BPF_REG_0].id = ++env->id_gen;
 	} else if (btf_type_is_void(t)) {
Signature · 1 line
2.47.3
#2 lore raw

Re: [PATCH bpf-next 1/5] bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators

Quoted 35 lines · Puranjay Mohan

Some iterators hold resources (like mmap_lock in task_vma) that prevent sleeping. To allow BPF programs to release such resources mid-iteration and call sleepable helpers, the verifier needs to track acquire/release semantics on iterator _next pointers.

Repurpose the st->id field on STACK_ITER slots to track the ref_obj_id of the pointer returned by _next when the kfunc is annotated with KF_ACQUIRE. This is safe because st->id is initialized to 0 by __mark_reg_known_zero() in mark_stack_slots_iter() and is not compared in stacksafe() for STACK_ITER slots.

The lifecycle is:

_next (KF_ACQUIRE):
- auto-release old ref if st->id != 0
- acquire new ref, store ref_obj_id in st->id
- DRAINED branch: release via st->id, set st->id = 0
- ACTIVE branch: keeps ref, st->id tracks it

_release (KF_RELEASE + __iter arg):
- read st->id, release_reference(), set st->id = 0

_destroy:
- release st->id if non-zero before releasing iterator's own ref

---
  kernel/bpf/verifier.c | 67 ++++++++++++++++++++++++++++++++++++++++---
  1 file changed, 63 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +0 -0
index 0162f946032f..aa48180b6073 100644

kernel/bpf/verifier.c struct bpf_kfunc_call_arg_meta { · line 358

Quoted 4 lines
kernel/bpf/verifier.c +1 -0
@@ -355,6 +355,11 @@ struct bpf_kfunc_call_arg_meta {
  		u8 spi;
  		u8 frameno;
  	} iter;
+	/* Set when a kfunc takes an __iter arg. Used by the KF_RELEASE

For new comments we use kernel style with /* on the first line and then your text goes to the next line.

Quoted 98 lines
kernel/bpf/verifier.c +47 -4
+	 * path to release the reference tracked on the iterator slot
+	 * (st->id) instead of requiring a refcounted PTR_TO_BTF_ID arg.
+	 */
+	bool has_iter_arg;
  	struct bpf_map_desc map;
  	u64 mem_size;
  };
@@ -1083,6 +1088,22 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
  	return 0;
  }
 
+/* Release the acquired reference tracked by iter_st->id, if any.
+ * Used during auto-release in _next, DRAINED handling, and _destroy.
+ */
+static int iter_release_acquired_ref(struct bpf_verifier_env *env,
+				     struct bpf_reg_state *iter_st)
+{
+	int err;
+
+	if (!iter_st->id)
+		return 0;
+	err = release_reference(env, iter_st->id);
+	if (!err)
+		iter_st->id = 0;
+	return err;
+}
+
  static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
  				   struct bpf_reg_state *reg, int nr_slots)
  {
@@ -1097,8 +1118,13 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
  		struct bpf_stack_state *slot = &state->stack[spi - i];
  		struct bpf_reg_state *st = &slot->spilled_ptr;
 
-		if (i == 0)
+		if (i == 0) {
+			/* Release any outstanding acquired ref tracked by
+			 * st->id before releasing the iterator's own ref.
+			 */
+			WARN_ON_ONCE(iter_release_acquired_ref(env, st));
  			WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
+		}
 
  		__mark_reg_not_init(env, st);
 
@@ -8943,6 +8969,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
  		/* remember meta->iter info for process_iter_next_call() */
  		meta->iter.spi = spi;
  		meta->iter.frameno = reg->frameno;
+		meta->has_iter_arg = true;
  		meta->ref_obj_id = iter_ref_obj_id(env, reg, spi);
 
  		if (is_iter_destroy_kfunc(meta)) {
@@ -9178,8 +9205,10 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
  	/* mark current iter state as drained and assume returned NULL */
  	cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
  	__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
-
-	return 0;
+	/* If _next acquired a ref (KF_ACQUIRE), release it in the DRAINED
+	 * branch since NULL was returned.
+	 */
+	return iter_release_acquired_ref(env, cur_iter);
  }
 
  static bool arg_type_is_mem_size(enum bpf_arg_type type)
@@ -13797,7 +13826,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
  		}
  	}
 
-	if (is_kfunc_release(meta) && !meta->release_regno) {
+	if (is_kfunc_release(meta) && !meta->release_regno && !meta->has_iter_arg) {
  		verbose(env, "release kernel function %s expects refcounted PTR_TO_BTF_ID\n",
  			func_name);
  		return -EINVAL;
@@ -14205,6 +14234,21 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  		}
  		if (err)
  			return err;
+	} else if (meta.has_iter_arg && is_kfunc_release(&meta)) {
+		/* For KF_RELEASE kfuncs taking an __iter arg, release the
+		 * reference tracked by st->id on the iterator slot.
+		 */
+		struct bpf_reg_state *iter_st;
+
+		iter_st = get_iter_from_state(env->cur_state, &meta);
+		if (!iter_st->id) {
+			verbose(env, "no acquired reference to release\n");
+			return -EINVAL;
+		}
+		err = release_reference(env, iter_st->id);
+		if (err)
+			return err;
+		iter_st->id = 0;
  	}
 
  	if (meta.func_id == special_kfunc_list[KF_bpf_list_push_front_impl] ||

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14407

Quoted 8 lines
kernel/bpf/verifier.c +5 -0
@@ -14356,6 +14400,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  			regs[BPF_REG_0].id = ++env->id_gen;
  		}
  		mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *));
+		/* For iterators with KF_ACQUIRE, auto-release the previous
+		 * iteration's ref before acquiring a new one, and after
+		 * acquisition track the new ref on the iter slot.
+		 */
+		struct bpf_reg_state *iter_acquire_st = NULL;

This variable declaration shouldn't probably be here.

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14411

Quoted 4 lines
kernel/bpf/verifier.c +4 -0
+
+		if (is_iter_next_kfunc(&meta) && is_kfunc_acquire(&meta)) {
+			iter_acquire_st = get_iter_from_state(env->cur_state, &meta);
+			err = iter_release_acquired_ref(env, iter_acquire_st);

The acquire and release paths for iterator references in check_kfunc_call() are far apart, which makes the lifecycle hard to follow. The guard conditions are also asymmetric:
meta.has_iter_arg for release vs is_iter_next_kfunc() for acquire.

More broadly, could iterator release be unified into the release_regno dispatch, similar to dynptrs? E.g., set meta->release_regno in process_iter_arg(), then add an iterator sub-case inside the if (meta.release_regno) block, keeping all release logic in one place.

Quoted 7 lines
kernel/bpf/verifier.c +3 -0
+			if (err)
+				return err;
+		}
  		if (is_kfunc_acquire(&meta)) {
  			int id = acquire_reference(env, insn_idx);
 

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14432

Quoted 9 lines
kernel/bpf/verifier.c +3 -0
@@ -14368,6 +14424,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
  			ref_set_non_owning(env, &regs[BPF_REG_0]);
  		}
 
+		if (iter_acquire_st)
+			iter_acquire_st->id = regs[BPF_REG_0].ref_obj_id;
+
  		if (reg_may_point_to_spin_lock(&regs[BPF_REG_0]) && !regs[BPF_REG_0].id)
  			regs[BPF_REG_0].id = ++env->id_gen;
  	} else if (btf_type_is_void(t)) {

Have you explored marking iter_next as both KF_ACQUIRE and KF_RELEASE, so the auto-release of the previous iteration's ref hits the existing release path naturally?
I understand it doesn't work as-is - the release branch errors out on the first _next call, also last call does not do acquire. But maybe there is a clean solution.

maintainer
#3 lore raw

Re: [PATCH bpf-next 1/5] bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators

In reply to Puranjay Mohan

On Wed, 2026-02-18 at 10:25 -0800, Puranjay Mohan wrote:

[...]

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13829

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -13797,7 +13826,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		}
 	}
 
-	if (is_kfunc_release(meta) && !meta->release_regno) {
+	if (is_kfunc_release(meta) && !meta->release_regno && !meta->has_iter_arg) {

This makes the check too strict, right?
E.g. for most of the release kfuncs there would be no iter arg, hence the check won't ever be true for those kfuncs. Do we have tests for this check?
Either we don't have the tests or the check itself is redundant.

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
 		verbose(env, "release kernel function %s expects refcounted PTR_TO_BTF_ID\n",
 			func_name);
 		return -EINVAL;

[...]

Quoted 6 lines
kernel/bpf/verifier.c +2 -0
@@ -14356,6 +14400,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			regs[BPF_REG_0].id = ++env->id_gen;
 		}
 		mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *));
+		/* For iterators with KF_ACQUIRE, auto-release the previous
+		 * iteration's ref before acquiring a new one, and after

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14414

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		 * acquisition track the new ref on the iter slot.
+		 */
+		struct bpf_reg_state *iter_acquire_st = NULL;
+
+		if (is_iter_next_kfunc(&meta) && is_kfunc_acquire(&meta)) {
+			iter_acquire_st = get_iter_from_state(env->cur_state, &meta);
+			err = iter_release_acquired_ref(env, iter_acquire_st);
+			if (err)
+				return err;
+		}

I second Mykyta's points here.

[...]

#4 lore raw

Re: [PATCH bpf-next 1/5] bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators

Quoted 9 lines · Eduard Zingerman

On Wed, 2026-02-18 at 10:25 -0800, Puranjay Mohan wrote:

[...]

Quoted 4 lines
(quoted diff) +0 -0
@@ -13797,7 +13826,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
              }
      }
 
Quoted 8 lines

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13829

Quoted 2 lines
kernel/bpf/verifier.c +1 -1
-     if (is_kfunc_release(meta) && !meta->release_regno) {
+     if (is_kfunc_release(meta) && !meta->release_regno && !meta->has_iter_arg) {

This makes the check too strict, right?
E.g. for most of the release kfuncs there would be no iter arg, hence the check won't ever be true for those kfuncs. Do we have tests for this check?
Either we don't have the tests or the check itself is redundant.

A release kfunc should either have meta->release_regno set or it should take an __iter arg, that is what this is checking. If we leave it like it was then every release function that take __iter would fail her. But anyway I will be doing it the better way how Mykyta Suggested, so this will be gone.

Quoted 27 lines
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
              verbose(env, "release kernel function %s expects refcounted PTR_TO_BTF_ID\n",
                      func_name);
              return -EINVAL;

[...]

Quoted 6 lines
kernel/bpf/verifier.c +2 -0
@@ -14356,6 +14400,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                      regs[BPF_REG_0].id = ++env->id_gen;
              }
              mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *));
+             /* For iterators with KF_ACQUIRE, auto-release the previous
+              * iteration's ref before acquiring a new one, and after

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14414

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+              * acquisition track the new ref on the iter slot.
+              */
+             struct bpf_reg_state *iter_acquire_st = NULL;
+
+             if (is_iter_next_kfunc(&meta) && is_kfunc_acquire(&meta)) {
+                     iter_acquire_st = get_iter_from_state(env->cur_state, &meta);
+                     err = iter_release_acquired_ref(env, iter_acquire_st);
+                     if (err)
+                             return err;
+             }

I second Mykyta's points here.

[...]

maintainer
#5 lore raw

Re: [PATCH bpf-next 1/5] bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators

Quoted 20 lines · Puranjay Mohan
Quoted 10 lines · Eduard Zingerman

On Wed, 2026-02-18 at 10:25 -0800, Puranjay Mohan wrote:

[...]

Quoted 5 lines
(quoted diff) +0 -0
@@ -13797,7 +13826,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct

bpf_kfunc_call_
              }
      }

Quoted 8 lines
Quoted 2 lines

-     if (is_kfunc_release(meta) && !meta->release_regno) {
+     if (is_kfunc_release(meta) && !meta->release_regno && !meta->has_iter_arg) {

This makes the check too strict, right?
E.g. for most of the release kfuncs there would be no iter arg, hence the check won't ever be true for those kfuncs. Do we have tests for this check?
Either we don't have the tests or the check itself is redundant.

Quoted 4 lines

A release kfunc should either have meta->release_regno set or it should take an __iter arg, that is what this is checking. If we leave it like it was then every release function that take __iter would fail her.

Oh, right, I missed the point here.

Quoted 2 lines

But anyway I will be doing it the better way how Mykyta Suggested, so this will be gone.

#6 lore raw

PATCH 2/5 bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs

In reply to Puranjay Mohan

With KF_ACQUIRE support for iterators in place, we need a way to tell the verifier that holding a particular acquired reference forbids sleeping. For example, task_vma's _next holds mmap_lock, so sleeping between _next and _release must be rejected.

Add a KF_FORBID_SLEEP flag (1 << 17) that can be combined with KF_ACQUIRE. When acquire_reference() is called for such a kfunc, the reference is tagged with forbid_sleep=true and a per-state forbid_sleep_count counter is incremented. When the reference is released through release_reference_nomark(), the counter is decremented in the same loop that already scans the refs array.

The counter is checked wherever the verifier decides if sleeping is allowed.

This is generic and works for both iterator and non-iterator kfuncs. For iterators, the auto-release and explicit _release from the previous commit handle the counter decrement automatically via release_reference().

---
 include/linux/bpf_verifier.h |  2 ++
 include/linux/btf.h          |  1 +
 kernel/bpf/verifier.c        | 54 ++++++++++++++++++++++++++++++------
 3 files changed, 49 insertions(+), 8 deletions(-)
include/linux/bpf_verifier.h +2 -0
index c1e30096ea7b..39904401df3d 100644
@@ -260,6 +260,7 @@ struct bpf_reference_state {
 	 * it matches on unlock.
 	 */
 	void *ptr;
+	bool forbid_sleep;	/* ref prevents sleeping while held */
 };
 
 struct bpf_retval_range {
@@ -420,6 +421,7 @@ struct bpf_verifier_state {
 	u32 active_lock_id;
 	void *active_lock_ptr;
 	u32 active_rcu_locks;
+	u32 forbid_sleep_count;
 
 	bool speculative;
 	bool in_sleepable;
include/linux/btf.h +1 -0
index 48108471c5b1..c326c5ba49cb 100644
@@ -79,6 +79,7 @@
 #define KF_ARENA_ARG1   (1 << 14) /* kfunc takes an arena pointer as its first argument */
 #define KF_ARENA_ARG2   (1 << 15) /* kfunc takes an arena pointer as its second argument */
 #define KF_IMPLICIT_ARGS (1 << 16) /* kfunc has implicit arguments supplied by the verifier */
+#define KF_FORBID_SLEEP  (1 << 17) /* acquired reference forbids sleeping while held */
 
 /*
  * Tag marking a kernel function as a kfunc. This is meant to minimize the
kernel/bpf/verifier.c +46 -8

162 lines not on this page · raw message

Signature · 1 line
2.47.3
#7 lore raw

Re: [PATCH bpf-next 2/5] bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs

Quoted 11 lines · Puranjay Mohan
kernel/bpf/verifier.c +3 -0
@@ -20054,6 +20086,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
        if (old->active_rcu_locks != cur->active_rcu_locks)
                return false;
 
+       if (old->forbid_sleep_count != cur->forbid_sleep_count)
+               return false;
+
        if (!check_ids(old->active_irq_id, cur->active_irq_id, idmap))
                return false;
 

kernel/bpf/verifier.c static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · line 20107

Quoted 6 lines
kernel/bpf/verifier.c +3 -0
@@ -20067,6 +20102,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
                        return false;
                switch (old->refs[i].type) {
                case REF_TYPE_PTR:
+                       if (old->refs[i].forbid_sleep != cur->refs[i].forbid_sleep)
+                               return false;
+                       break;

I don't like yet another special case.
We already have active_locks, active_rcu_locks, etc.
All that is way too specific. Let's figure out a generic way
or piggy back on existing active*, since all of them already prevent sleeping:
static inline bool in_sleepable_context(struct bpf_verifier_env *env)

{
        return !env->cur_state->active_rcu_locks &&
               !env->cur_state->active_preempt_locks &&
               !env->cur_state->active_locks &&
               !env->cur_state->active_irq_id &&
               in_sleepable(env);
}
#8 lore raw

Re: [PATCH bpf-next 2/5] bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs

Quoted 25 lines · Alexei Starovoitov
Quoted 11 lines · Puranjay Mohan
kernel/bpf/verifier.c +3 -0
@@ -20054,6 +20086,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
        if (old->active_rcu_locks != cur->active_rcu_locks)
                return false;
 
+       if (old->forbid_sleep_count != cur->forbid_sleep_count)
+               return false;
+
        if (!check_ids(old->active_irq_id, cur->active_irq_id, idmap))
                return false;
 

kernel/bpf/verifier.c static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · line 20107

Quoted 6 lines
kernel/bpf/verifier.c +3 -0
@@ -20067,6 +20102,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
                        return false;
                switch (old->refs[i].type) {
                case REF_TYPE_PTR:
+                       if (old->refs[i].forbid_sleep != cur->refs[i].forbid_sleep)
+                               return false;
+                       break;

I don't like yet another special case.
We already have active_locks, active_rcu_locks, etc.
All that is way too specific. Let's figure out a generic way
or piggy back on existing active*, since all of them already prevent sleeping:
static inline bool in_sleepable_context(struct bpf_verifier_env *env)

Quoted 7 lines
{
        return !env->cur_state->active_rcu_locks &&
               !env->cur_state->active_preempt_locks &&
               !env->cur_state->active_locks &&
               !env->cur_state->active_irq_id &&
               in_sleepable(env);
}

I thought about this and looked at how each of these are used. Each of these carry semantics beyond sleep prevention, so they can't simply be consolidated or reused. But forbid_sleep_count is not strictly needed as we can just loop over every reference in the refs array and see if any reference has the nosleep flag set, but this would mean iterating the list every time. I tried using one of these already existing active* but it would add more complexity.

maintainer
#9 lore raw

Re: [PATCH bpf-next 2/5] bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs

In reply to Puranjay Mohan

On Wed, 2026-02-18 at 10:25 -0800, Puranjay Mohan wrote:

[...]

kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10518

Quoted 6 lines
kernel/bpf/verifier.c +2 -0
@@ -10510,6 +10514,8 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
 		if (state->refs[i].type != REF_TYPE_PTR)
 			continue;
 		if (state->refs[i].id == ref_obj_id) {
+			if (state->refs[i].forbid_sleep)
+				state->forbid_sleep_count--;

Nit: I'd do it inside release_reference_state() just to have a
fool-proof point for decrementing this counter.
As it stands release_reference_state() is called from two more places and one need to think if .forbid_sleep_count needs to be handled there as well (it is not, at the moment).

kernel/bpf/verifier.c static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 10857

Quoted 10 lines
kernel/bpf/verifier.c +2 -1
 			release_reference_state(state, i);
 			return 0;
 		}
@@ -10847,7 +10853,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 		if (env->subprog_info[subprog].might_sleep &&
 		    (env->cur_state->active_rcu_locks || env->cur_state->active_preempt_locks ||
-		     env->cur_state->active_irq_id || !in_sleepable(env))) {
+		     env->cur_state->active_irq_id || env->cur_state->forbid_sleep_count ||
+		     !in_sleepable(env))) {

The whole expression is the same as '!in_sleepable_context(env)', right?

kernel/bpf/verifier.c static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit · line 11449

Quoted 12 lines
kernel/bpf/verifier.c +5 -0
 			verbose(env, "global functions that may sleep are not allowed in non-sleepable context,\n"
 				     "i.e., in a RCU/IRQ/preempt-disabled section, or in\n"
 				     "a non-sleepable BPF program context\n");
@@ -11435,6 +11442,11 @@ static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit
 		return -EINVAL;
 	}
 
+	if (check_lock && env->cur_state->forbid_sleep_count) {
+		verbose(env, "%s cannot be used inside nosleep region\n", prefix);
+		return -EINVAL;
+	}
+

Is this check really necessary?
Some resource is forbidding sleep here, so there would be an error report already.

[...]

kernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 11681

Quoted 12 lines
kernel/bpf/verifier.c +8 -0
@@ -11658,6 +11671,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 		}
 	}
 
+	if (env->cur_state->forbid_sleep_count) {
+		if (fn->might_sleep) {
+			verbose(env, "sleepable helper %s#%d in nosleep region\n",
+				func_id_name(func_id), func_id);
+			return -EINVAL;
+		}
+	}
+

Wdyt about consolidating this code as something like:

  if (fn->might_sleep && !in_sleepable_context(env)) {
	verbose(env, "sleepable helper %s#%d in %s region\n",
		func_id_name(func_id), func_id, non_sleepable_context_description(env));
	return -EINVAL;

}

?

[...]

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14146

Quoted 9 lines
kernel/bpf/verifier.c +5 -0
@@ -14113,6 +14139,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return -EACCES;
 	}
 
+	if (sleepable && env->cur_state->forbid_sleep_count) {
+		verbose(env, "sleepable kfunc %s in nosleep region\n", func_name);
+		return -EACCES;
+	}
+

Above this check there is:

        if (sleepable && !in_sleepable(env)) {
                verbose(env, "program must be sleepable to call sleepable kfunc %s\n", func_name);
                return -EACCES;
        }

Should it be sleepable && !in_sleepable_context(env) instead? If it should, then this hunk would be redundant.

[...]

#10 lore raw

Re: [PATCH bpf-next 2/5] bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs

Quoted 5 lines · Eduard Zingerman

On Wed, 2026-02-18 at 10:25 -0800, Puranjay Mohan wrote:

[...]

Quoted 12 lines

kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10518

Quoted 6 lines
kernel/bpf/verifier.c +2 -0
@@ -10510,6 +10514,8 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
              if (state->refs[i].type != REF_TYPE_PTR)
                      continue;
              if (state->refs[i].id == ref_obj_id) {
+                     if (state->refs[i].forbid_sleep)
+                             state->forbid_sleep_count--;

Nit: I'd do it inside release_reference_state() just to have a
fool-proof point for decrementing this counter.
As it stands release_reference_state() is called from two more places and one need to think if .forbid_sleep_count needs to be handled there as well (it is not, at the moment).

Yes, I will move it there.

Quoted 6 lines
Quoted 6 lines
kernel/bpf/verifier.c +0 -0
                      release_reference_state(state, i);
                      return 0;
              }
@@ -10847,7 +10853,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
Quoted 7 lines

kernel/bpf/verifier.c static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 10857

Quoted 5 lines
kernel/bpf/verifier.c +2 -1
              if (env->subprog_info[subprog].might_sleep &&
                  (env->cur_state->active_rcu_locks || env->cur_state->active_preempt_locks ||
-                  env->cur_state->active_irq_id || !in_sleepable(env))) {
+                  env->cur_state->active_irq_id || env->cur_state->forbid_sleep_count ||
+                  !in_sleepable(env))) {

The whole expression is the same as '!in_sleepable_context(env)', right?

Yes, in this case because if (env->cur_state->active_locks) { already prints for this specific case, and if active_locks==0 then all this hunk is equivalent to !in_sleepable_context(env), will change it in next version.

Quoted 8 lines
Quoted 8 lines
kernel/bpf/verifier.c +0 -0
                      verbose(env, "global functions that may sleep are not allowed in non-sleepable context,\n"
                                   "i.e., in a RCU/IRQ/preempt-disabled section, or in\n"
                                   "a non-sleepable BPF program context\n");
@@ -11435,6 +11442,11 @@ static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit
              return -EINVAL;
      }
 
Quoted 8 lines

kernel/bpf/verifier.c static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit · line 11449

Quoted 5 lines
kernel/bpf/verifier.c +5 -0
+     if (check_lock && env->cur_state->forbid_sleep_count) {
+             verbose(env, "%s cannot be used inside nosleep region\n", prefix);
+             return -EINVAL;
+     }
+

Is this check really necessary?
Some resource is forbidding sleep here, so there would be an error report already.

Yeah, maybe we will support KF_FORBID_SLEEP in future without reference tracking, but for now I will remove it in the next version.

Quoted 7 lines

[...]

Quoted 4 lines
(quoted diff) +0 -0
@@ -11658,6 +11671,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
              }
      }
 
Quoted 10 lines

kernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 11681

Quoted 8 lines
kernel/bpf/verifier.c +8 -0
+     if (env->cur_state->forbid_sleep_count) {
+             if (fn->might_sleep) {
+                     verbose(env, "sleepable helper %s#%d in nosleep region\n",
+                             func_id_name(func_id), func_id);
+                     return -EINVAL;
+             }
+     }
+

Wdyt about consolidating this code as something like:

I agree, will add a preparatory patch that does this and will use it in this one

Quoted 23 lines
  if (fn->might_sleep && !in_sleepable_context(env)) {
        verbose(env, "sleepable helper %s#%d in %s region\n",
                func_id_name(func_id), func_id, non_sleepable_context_description(env));
        return -EINVAL;
  }

?

[...]

kernel/bpf/verifier.c static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 14146

Quoted 9 lines
kernel/bpf/verifier.c +5 -0
@@ -14113,6 +14139,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
              return -EACCES;
      }
 
+     if (sleepable && env->cur_state->forbid_sleep_count) {
+             verbose(env, "sleepable kfunc %s in nosleep region\n", func_name);
+             return -EACCES;
+     }
+

Above this check there is:

Quoted 7 lines
        if (sleepable && !in_sleepable(env)) {
                verbose(env, "program must be sleepable to call sleepable kfunc %s\n", func_name);
                return -EACCES;
        }

Should it be sleepable && !in_sleepable_context(env) instead? If it should, then this hunk would be redundant.

This is just for printing the correct kind of message. Actually this should go below after

» if (env->cur_state->active_irq_id && sleepable) {
» » verbose(env, "kernel func %s is sleepable within
IRQ-disabled region\n", func_name);
» » return -EACCES;
» }

» if (is_kfunc_rcu_protected(&meta) && !in_rcu_cs(env)) {
» » verbose(env, "kernel func %s requires RCU critical
section protection\n", func_name);
» » return -EACCES;
» }

sleepable && !in_sleepable_context(env) will catch all of them together, so maybe I could consolidate all of then with a helper
like: non_sleepable_context_description()
but some of these checks and intermingled and would make reviewing difficult so I will send it as a follow-up.

#11 lore raw

PATCH 3/5 bpf: Move locking to bpf_iter_task_vma_next()

In reply to Puranjay Mohan

The current implementation of task_vma iterator takes the mmap_lock in the _new() function and holds it for the entire duration of the iterator. The next commits will allow releasing the lock in the middle of the iteration and it would mean that the _next() call should re-take the mmap_lock.

Move the mmap_lock setup to bpf_iter_task_vma_next()

---
 kernel/bpf/task_iter.c | 34 +++++++++++++++++++++++-----------
 1 file changed, 23 insertions(+), 11 deletions(-)
kernel/bpf/task_iter.c +23 -11
index 98d9b4c0daff..a85115c191e4 100644
@@ -799,6 +799,8 @@ struct bpf_iter_task_vma_kern_data {
 	struct mm_struct *mm;
 	struct mmap_unlock_irq_work *work;
 	struct vma_iterator vmi;
+	u64 last_addr;
+	bool locked;
 };
 
 struct bpf_iter_task_vma {
@@ -819,7 +821,6 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
 				      struct task_struct *task, u64 addr)
 {
 	struct bpf_iter_task_vma_kern *kit = (void *)it;
-	bool irq_work_busy = false;
 	int err;
 
 	BUILD_BUG_ON(sizeof(struct bpf_iter_task_vma_kern) != sizeof(struct bpf_iter_task_vma));
@@ -840,14 +841,8 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
 		goto err_cleanup_iter;
 	}
 
-	/* kit->data->work == NULL is valid after bpf_mmap_unlock_get_irq_work */
-	irq_work_busy = bpf_mmap_unlock_get_irq_work(&kit->data->work);
-	if (irq_work_busy || !mmap_read_trylock(kit->data->mm)) {
-		err = -EBUSY;
-		goto err_cleanup_iter;
-	}
-
-	vma_iter_init(&kit->data->vmi, kit->data->mm, addr);
+	kit->data->locked = false;
+	kit->data->last_addr = addr;
 	return 0;
 
 err_cleanup_iter:
@@ -862,10 +857,26 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
 __bpf_kfunc struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it)
 {
 	struct bpf_iter_task_vma_kern *kit = (void *)it;
+	struct vm_area_struct *vma;
 
 	if (!kit->data) /* bpf_iter_task_vma_new failed */
 		return NULL;
-	return vma_next(&kit->data->vmi);
+
+	if (!kit->data->locked) {
+		bool irq_work_busy;
+
+		irq_work_busy = bpf_mmap_unlock_get_irq_work(&kit->data->work);
+		if (irq_work_busy || !mmap_read_trylock(kit->data->mm))
+			return NULL;
+
+		kit->data->locked = true;
+		vma_iter_init(&kit->data->vmi, kit->data->mm, kit->data->last_addr);
+	}
+
+	vma = vma_next(&kit->data->vmi);
+	if (vma)
+		kit->data->last_addr = vma->vm_end;
+	return vma;
 }
 
 __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
@@ -873,7 +884,8 @@ __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
 	struct bpf_iter_task_vma_kern *kit = (void *)it;
 
 	if (kit->data) {
-		bpf_mmap_unlock_mm(kit->data->work, kit->data->mm);
+		if (kit->data->locked)
+			bpf_mmap_unlock_mm(kit->data->work, kit->data->mm);
 		put_task_struct(kit->data->task);
 		bpf_mem_free(&bpf_global_ma, kit->data);
 	}
Signature · 1 line
2.47.3
#12 lore raw

PATCH 4/5 bpf: Add split iteration support to task_vma iterator

In reply to Puranjay Mohan

Wire up the task_vma iterator to use the KF_ACQUIRE/KF_RELEASE and KF_FORBID_SLEEP infrastructure from the previous commits.

Annotate bpf_iter_task_vma_next with KF_ACQUIRE | KF_FORBID_SLEEP so each returned VMA pointer becomes a tracked reference that prevents sleeping. Add bpf_iter_task_vma_release(it__iter) with KF_RELEASE that releases the mmap read lock, drops the acquired reference, and re-enables sleeping. The next call to _next will re-acquire the lock via mmap_read_trylock().

This enables a split iteration pattern:

  bpf_for_each(task_vma, vma, task, 0) {
      u64 start = vma->vm_start;
      /* vma access allowed, sleeping forbidden */
      bpf_iter_task_vma_release(&___it);
      /* vma access forbidden, sleeping allowed */
      bpf_copy_from_user(&buf, sizeof(buf), (void *)start);
  }
---
 kernel/bpf/helpers.c   |  3 ++-
 kernel/bpf/task_iter.c | 10 ++++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)
kernel/bpf/helpers.c +2 -1
index 7ac32798eb04..b3921c22810e 100644
@@ -4605,8 +4605,9 @@ BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
 BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
 BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
-BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL | KF_ACQUIRE | KF_FORBID_SLEEP)
 BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
+BTF_ID_FLAGS(func, bpf_iter_task_vma_release, KF_RELEASE)
 #ifdef CONFIG_CGROUPS
 BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
 BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
kernel/bpf/task_iter.c +10 -0
index a85115c191e4..55b264d634db 100644
@@ -879,6 +879,16 @@ __bpf_kfunc struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_v
 	return vma;
 }
 
+__bpf_kfunc void bpf_iter_task_vma_release(struct bpf_iter_task_vma *it__iter)
+{
+	struct bpf_iter_task_vma_kern *kit = (void *)it__iter;
+
+	if (kit->data && kit->data->locked) {
+		bpf_mmap_unlock_mm(kit->data->work, kit->data->mm);
+		kit->data->locked = false;
+	}
+}
+
 __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
 {
 	struct bpf_iter_task_vma_kern *kit = (void *)it;
Signature · 1 line
2.47.3
#13 lore raw

PATCH 5/5 selftests/bpf: Add tests for split task_vma iterator

In reply to Puranjay Mohan

Add runtime and verifier tests for the split task_vma iteration model.

The runtime test iter_task_vma_release_and_copy iterates VMAs, saves vm_start, releases mmap_lock via bpf_iter_task_vma_release(), then calls bpf_copy_from_user() on the saved address.

The verifier tests cover:
- nosleep_iter_release_then_sleep: release then sleep is accepted
- nosleep_iter_sleep_without_release: sleep without release is rejected with "in nosleep region"
- nosleep_iter_vma_access_after_release: VMA access after release is rejected with "invalid mem access 'scalar'"

---
 .../testing/selftests/bpf/bpf_experimental.h  |   1 +
 .../testing/selftests/bpf/prog_tests/iters.c  |  13 ++
 .../selftests/bpf/progs/iters_task_vma.c      |  39 ++++++
 .../bpf/progs/iters_task_vma_nosleep.c        | 125 ++++++++++++++++++
 4 files changed, 178 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma_nosleep.c
tools/testing/selftests/bpf/bpf_experimental.h +1 -0
index 9df77e59d4f5..531d6c6aab45 100644
@@ -165,6 +165,7 @@ extern int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
 				 struct task_struct *task,
 				 __u64 addr) __ksym;
 extern struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it) __ksym;
+extern void bpf_iter_task_vma_release(struct bpf_iter_task_vma *it) __ksym;
 extern void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it) __ksym;
 
 /* Convenience macro to wrap over bpf_obj_drop_impl */
tools/testing/selftests/bpf/prog_tests/iters.c +13 -0
index a539980a2fbe..977114f0e88f 100644
@@ -21,6 +21,7 @@
 #include "iters_css_task.skel.h"
 #include "iters_css.skel.h"
 #include "iters_task_failure.skel.h"
+#include "iters_task_vma_nosleep.skel.h"
 
 static void subtest_num_iters(void)
 {
@@ -152,6 +153,17 @@ static void subtest_task_vma_iters(void)
 	if (!ASSERT_EQ(skel->bss->vmas_seen, seen, "vmas_seen_eq"))
 		goto cleanup;
 
+	/* Test release+sleepable: trigger the release_and_copy program */
+	skel->bss->release_vmas_seen = 0;
+	err = iters_task_vma__attach(skel);
+	if (!ASSERT_OK(err, "skel_reattach"))
+		goto cleanup;
+
+	getpgid(skel->bss->target_pid);
+	iters_task_vma__detach(skel);
+
+	ASSERT_GT(skel->bss->release_vmas_seen, 0, "release_vmas_seen_gt_zero");
+
 cleanup:
 	if (f)
 		fclose(f);
@@ -322,4 +334,5 @@ void test_iters(void)
 	if (test__start_subtest("css"))
 		subtest_css_iters();
 	RUN_TESTS(iters_task_failure);
+	RUN_TESTS(iters_task_vma_nosleep);
 }
tools/testing/selftests/bpf/progs/iters_task_vma.c +39 -0
index dc0c3691dcc2..59065f9da4f8 100644
@@ -40,4 +40,43 @@ int iter_task_vma_for_each(const void *ctx)
 	return 0;
 }
 
+unsigned int release_vmas_seen = 0;
+
+SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
+int iter_task_vma_release_and_copy(const void *ctx)
+{
+	struct task_struct *task = bpf_get_current_task_btf();
+	struct vm_area_struct *vma;
+	unsigned int seen = 0;
+
+	if (task->pid != target_pid)
+		return 0;
+
+	if (release_vmas_seen)
+		return 0;
+
+	bpf_for_each(task_vma, vma, task, 0) {
+		__u64 start;
+		char buf[8];
+
+		if (bpf_cmp_unlikely(seen, >=, 1000))
+			break;
+
+		/* Phase 1: mmap_lock held, read VMA data */
+		start = vma->vm_start;
+
+		/* Transition: release mmap_lock */
+		bpf_iter_task_vma_release(&___it);
+		/* VMA pointer is now invalid; sleepable helpers allowed */
+
+		/* Phase 2: mmap_lock released, sleepable call */
+		bpf_copy_from_user(&buf, sizeof(buf), (void *)start);
+
+		seen++;
+	}
+
+	release_vmas_seen = seen;
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/iters_task_vma_nosleep.c new file +125 -0

125 lines not on this page · raw message

Signature · 1 line
2.47.3