Introduce KF_FORBID_SLEEP modifier for acquire/release kfuncs
v2 cover Introduce KF_FORBID_SLEEP modifier for acquire/release kfuncs
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
Changes in v1 -> v2:
- Add a patch to consolidate sleepable context error message printing in check_helper_call(), has no functional changes (Eduard)
- In check_kfunc_call() for KF_RELEASE and __iter arg, use release_regno like dynptr rather than custom handling (Mykyta)
- Fix some comments to follow correct style (Mykyta)
- Move state->forbid_sleep_count-- to release_reference_state() (Eduard)
- Remove error message in check_resource_leak() for forbid_sleep_count because it is redundant and will never trigger (Eduard)
- Consolidated some checks and prints (Eduard)
Some BPF kfuncs acquire resources that prevent sleeping - a lock, a reference to an object under a lock, a preempt-disable section. Today there is no way for the verifier to know that holding a particular acquired reference means sleeping is forbidden. Programs either run entirely in sleepable or non-sleepable context, with no way to express "sleeping is forbidden right now, but will be allowed once I release this reference."
This series adds KF_FORBID_SLEEP, a new kfunc flag that can be combined with KF_ACQUIRE. When a kfunc annotated with KF_ACQUIRE | KF_FORBID_SLEEP is called, the verifier tags the acquired reference with forbid_sleep and increments a per-state forbid_sleep_count counter. When the reference is released (through corresponding KF_RELEASE kfunc), the counter is decremented. The verifier checks this counter everywhere it decides whether sleeping is allowed — sleepable helpers, sleepable kfuncs, global function calls, and resource leak checks.
This is fully generic. Any pair of KF_ACQUIRE/KF_RELEASE kfuncs can opt into sleep prohibition by adding KF_FORBID_SLEEP to the acquire side.
To make this useful for iterators specifically, the series first extends the verifier's iterator support to allow KF_ACQUIRE on _next and KF_RELEASE on a separate kfunc taking an __iter argument. The verifier tracks the acquired reference on the iterator's stack slot (st->id) and auto-releases it on the next _next call and on the DRAINED (NULL) path, so the acquire/release is transparent to programs that don't need mid-loop release.
Iterator KF_ACQUIRE support is not useful on its own right now, but it becomes the foundation for KF_FORBID_SLEEP: an iterator whose _next is annotated with KF_ACQUIRE | KF_FORBID_SLEEP can now express "holding this pointer forbids sleeping; calling _release invalidates the pointer and re-enables sleeping."
The task_vma iterator is the first user. It holds mmap_lock during iteration, preventing sleepable helpers like bpf_copy_from_user(). Currently, this can lead to a deadlock as the fault path also takes mmap_lock.
With this series, a BPF program can release the lock mid-iteration:
bpf_for_each(task_vma, vma, task, 0) {
u64 start = vma->vm_start;
/* sleeping forbidden, but vma pointer access allowed */
bpf_iter_task_vma_release(&___it);
/* mmap_lock released, vma pointer invalidated */
/* sleeping is fine here */
bpf_copy_from_user(&buf, sizeof(buf), (void *)start);
}The series is organized as:
Patch 1: KF_ACQUIRE/KF_RELEASE plumbing for iterators in the
verifier. Pure infrastructure, no behavioral change to existing iterators.
Patch 2: Consolidate sleepable context error message printing
in check_helper_call(), no functional changes
Patch 3: KF_FORBID_SLEEP flag and forbid_sleep_count machinery.
Generic, works for any KF_ACQUIRE kfunc - iterator or not.
Patch 4: Move mmap_lock acquisition from _new to _next in the
task_vma iterator, preparing for re-acquisition after release.
Patch 5: Wire up task_vma as the first user: annotate _next with
KF_ACQUIRE | KF_FORBID_SLEEP, add bpf_iter_task_vma_release().
Patch 6: Selftests covering the runtime path (release + copy_from_user)
and verifier rejection of invalid patterns (sleeping without release, VMA access after release, double release, release without acquire, nested iterator interaction).
Puranjay Mohan (6):
bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators
bpf: consolidate sleepable context error message printing
bpf: Add KF_FORBID_SLEEP modifier for KF_ACQUIRE kfuncs
bpf: Move locking to bpf_iter_task_vma_next()
bpf: Add split iteration support to task_vma iterator selftests/bpf: Add tests for split task_vma iterator
include/linux/bpf_verifier.h | 2 + include/linux/btf.h | 1 + kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 3 +- kernel/bpf/task_iter.c | 44 ++++-- kernel/bpf/verifier.c | 147 +++++++++++++----- .../testing/selftests/bpf/bpf_experimental.h | 1 + .../testing/selftests/bpf/prog_tests/iters.c | 13 ++ .../selftests/bpf/progs/iters_task_vma.c | 39 +++++ .../bpf/progs/iters_task_vma_nosleep.c | 125 +++++++++++++++ 10 files changed, 331 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma_nosleep.c
base-commit: 9cd168a2720bc614647dd20bc5d1c308c56a44ec
Signature · 1 line
2.47.3
PATCH v2 1/6 bpf: Add KF_ACQUIRE and KF_RELEASE support for iterators
Some iterators hold resources (like mmap_lock in task_vma) that prevent sleeping. To allow BPF programs to release such resources mid-iteration and call sleepable helpers, the verifier needs to track acquire/release semantics on iterator _next pointers.
Repurpose the st->id field on STACK_ITER slots to track the ref_obj_id of the pointer returned by _next when the kfunc is annotated with KF_ACQUIRE. This is safe because st->id is initialized to 0 by __mark_reg_known_zero() in mark_stack_slots_iter() and is not compared in stacksafe() for STACK_ITER slots.
The lifecycle is:
_next (KF_ACQUIRE):
- auto-release old ref if st->id != 0
- acquire new ref, store ref_obj_id in st->id
- DRAINED branch: release via st->id, set st->id = 0
- ACTIVE branch: keeps ref, st->id tracks it
_release (KF_RELEASE + __iter arg):
- read st->id, release_reference(), set st->id = 0
_destroy:
- release st->id if non-zero before releasing iterator's own ref
--- kernel/bpf/verifier.c | 61 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 58 insertions(+), 3 deletions(-)
kernel/bpf/verifier.c +58 -3
@@ -1038,6 +1038,7 @@ static void __mark_reg_known_zero(struct bpf_reg_state *reg);static bool in_rcu_cs (struct bpf_verifier_env *env);static bool is_kfunc_rcu_protected (struct bpf_kfunc_call_arg_meta *meta); +static bool is_kfunc_release (struct bpf_kfunc_call_arg_meta *meta); static int mark_stack_slots_iter(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta,
@@ -1083,6 +1084,23 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,return 0 ; } +/* +* Release the acquired reference tracked by iter_st->id, if any. +* Used during auto-release in _next, DRAINED handling, and _destroy. +*/ +static int iter_release_acquired_ref (struct bpf_verifier_env *env, +struct bpf_reg_state *iter_st) +{ +int err; + +if (!iter_st->id) +return 0 ; + err =release_reference (env, iter_st->id); +if (!err) + iter_st->id =0 ; +return err; +} +static int unmark_stack_slots_iter (struct bpf_verifier_env *env,struct bpf_reg_state *reg,int nr_slots) {
@@ -1097,8 +1115,14 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,struct bpf_stack_state *slot = &state->stack[spi - i];struct bpf_reg_state *st = &slot->spilled_ptr; -if (i ==0 ) +if (i ==0 ) { +/* +* Release any outstanding acquired ref tracked by st->id +* before releasing the iterator's own ref. +*/ +WARN_ON_ONCE (iter_release_acquired_ref (env, st));WARN_ON_ONCE (release_reference (env, st->ref_obj_id)); + }__mark_reg_not_init (env, st);
@@ -8943,6 +8967,8 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id/* remember meta->iter info for process_iter_next_call() */ meta->iter.spi = spi; meta->iter.frameno = reg->frameno; +if (is_kfunc_release (meta)) + meta->release_regno = regno; meta->ref_obj_id =iter_ref_obj_id (env, reg, spi);if (is_iter_destroy_kfunc (meta)) {
@@ -9178,8 +9204,11 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,/* mark current iter state as drained and assume returned NULL */ cur_iter->iter.state =BPF_ITER_STATE_DRAINED ;__mark_reg_const_zero (env, &cur_fr->regs[BPF_REG_0 ]); - -return 0 ; +/* +* If _next acquired a ref (KF_ACQUIRE), release it in the DRAINED branch since NULL +* was returned. +*/ +return iter_release_acquired_ref (env, cur_iter); }static bool arg_type_is_mem_size (enum bpf_arg_type type)
@@ -14197,6 +14226,17 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,if (meta.initialized_dynptr.ref_obj_id) { err =unmark_stack_slots_dynptr (env, reg); + }else if (base_type (reg->type) ==PTR_TO_STACK ) { +struct bpf_reg_state *iter_st; + + iter_st =get_iter_from_state (env->cur_state, &meta); +if (!iter_st->id) { +verbose (env,"no acquired reference to release\n" ); +return -EINVAL ; + } + err =release_reference (env, iter_st->id); +if (!err) + iter_st->id =0 ; }else { err =release_reference (env, reg->ref_obj_id);if (err)
@@ -14274,6 +14314,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,__mark_reg_const_zero (env, ®s[BPF_REG_0 ]);mark_btf_func_reg_size (env,BPF_REG_0 , t->size); }else if (btf_type_is_ptr (t)) { +struct bpf_reg_state *iter_acquire_st =NULL ; + ptr_type =btf_type_skip_modifiers (desc_btf, t->type, &ptr_type_id); err =check_special_kfunc (env, &meta, regs, insn_aux, ptr_type, desc_btf);if (err) {
@@ -14356,6 +14398,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, regs[BPF_REG_0 ].id = ++env->id_gen; }mark_btf_func_reg_size (env,BPF_REG_0 ,sizeof (void *)); +/* +* For iterators with KF_ACQUIRE, auto-release the previous iteration's ref before +* acquiring a new one, and after acquisition track the new ref on the iter slot. +*/ +if (is_iter_next_kfunc (&meta) &&is_kfunc_acquire (&meta)) { + iter_acquire_st =get_iter_from_state (env->cur_state, &meta); + err =iter_release_acquired_ref (env, iter_acquire_st); +if (err) +return err; + }if (is_kfunc_acquire (&meta)) {int id =acquire_reference (env, insn_idx);
@@ -14368,6 +14420,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,ref_set_non_owning (env, ®s[BPF_REG_0 ]); } +if (iter_acquire_st) + iter_acquire_st->id = regs[BPF_REG_0 ].ref_obj_id; +if (reg_may_point_to_spin_lock (®s[BPF_REG_0 ]) && !regs[BPF_REG_0 ].id) regs[BPF_REG_0 ].id = ++env->id_gen; }else if (btf_type_is_void (t)) {
Signature · 1 line
2.47.3
PATCH v2 2/6 bpf: consolidate sleepable context error message printing
In reply to Puranjay Mohancheck_helper_call() prints the error message for every env->cur_state->active* element when calling a sleepable helper. Consolidate all of them into a single print statement.
--- kernel/bpf/verifier.c | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-)
kernel/bpf/verifier.c +18 -22
@@ -11574,6 +11574,19 @@ static inline bool in_sleepable_context(struct bpf_verifier_env *env)in_sleepable (env); } +static const char *non_sleepable_context_description (struct bpf_verifier_env *env) +{ +if (env->cur_state->active_rcu_locks) +return "rcu_read_lock region" ; +if (env->cur_state->active_preempt_locks) +return "non-preemptible region" ; +if (env->cur_state->active_irq_id) +return "IRQ-disabled region" ; +if (env->cur_state->active_locks) +return "lock region" ; +return "non-sleepable context" ; +} +static int check_helper_call (struct bpf_verifier_env *env,struct bpf_insn *insn,int *insn_idx_p) {
@@ -11634,28 +11647,11 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnreturn err; } -if (env->cur_state->active_rcu_locks) { -if (fn->might_sleep) { -verbose (env,"sleepable helper %s#%d in rcu_read_lock region\n" , -func_id_name (func_id), func_id); -return -EINVAL ; - } - } - -if (env->cur_state->active_preempt_locks) { -if (fn->might_sleep) { -verbose (env,"sleepable helper %s#%d in non-preemptible region\n" , -func_id_name (func_id), func_id); -return -EINVAL ; - } - } - -if (env->cur_state->active_irq_id) { -if (fn->might_sleep) { -verbose (env,"sleepable helper %s#%d in IRQ-disabled region\n" , -func_id_name (func_id), func_id); -return -EINVAL ; - } +if (fn->might_sleep && !in_sleepable_context (env)) { +verbose (env,"sleepable helper %s#%d in %s\n" , +func_id_name (func_id), func_id, +non_sleepable_context_description (env)); +return -EINVAL ; }/* Track non-sleepable context for helpers. */
Signature · 1 line
2.47.3
Quoted 111 lines · Puranjay Mohan
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
Changes in v1 -> v2:
- Add a patch to consolidate sleepable context error message printing incheck_helper_call(), has no functional changes (Eduard)
- Incheck_kfunc_call()forKF_RELEASEand__iterarg, userelease_regnolike dynptr rather than custom handling (Mykyta)
- Fix some comments to follow correct style (Mykyta)
- Move state->forbid_sleep_count-- torelease_reference_state()(Eduard)
- Remove error message incheck_resource_leak()forforbid_sleep_countbecause it is redundant and will never trigger (Eduard)
- Consolidated some checks and prints (Eduard)Some BPF kfuncs acquire resources that prevent sleeping - a lock, a reference to an object under a lock, a preempt-disable section. Today there is no way for the verifier to know that holding a particular acquired reference means sleeping is forbidden. Programs either run entirely in sleepable or non-sleepable context, with no way to express "sleeping is forbidden right now, but will be allowed once I release this reference."
This series adds
KF_FORBID_SLEEP, a new kfunc flag that can be combined withKF_ACQUIRE. When a kfunc annotated withKF_ACQUIRE|KF_FORBID_SLEEPis called, the verifier tags the acquired reference withforbid_sleepand increments a per-stateforbid_sleep_countcounter. When the reference is released (through correspondingKF_RELEASEkfunc), the counter is decremented. The verifier checks this counter everywhere it decides whether sleeping is allowed — sleepable helpers, sleepable kfuncs, global function calls, and resource leak checks.This is fully generic. Any pair of KF_ACQUIRE/KF_RELEASE kfuncs can opt into sleep prohibition by adding
KF_FORBID_SLEEPto the acquire side.To make this useful for iterators specifically, the series first extends the verifier's iterator support to allow
KF_ACQUIREon_nextandKF_RELEASEon a separate kfunc taking an__iterargument. The verifier tracks the acquired reference on the iterator's stack slot (st->id) and auto-releases it on the next_nextcall and on the DRAINED (NULL) path, so the acquire/release is transparent to programs that don't need mid-loop release.Iterator
KF_ACQUIREsupport is not useful on its own right now, but it becomes the foundation forKF_FORBID_SLEEP: an iterator whose_nextis annotated withKF_ACQUIRE|KF_FORBID_SLEEPcan now express "holding this pointer forbids sleeping; calling_releaseinvalidates the pointer and re-enables sleeping."The
task_vmaiterator is the first user. It holdsmmap_lockduring iteration, preventing sleepable helpers likebpf_copy_from_user(). Currently, this can lead to a deadlock as the fault path also takesmmap_lock.With this series, a BPF program can release the lock mid-iteration:
bpf_for_each(task_vma, vma, task, 0) { u64 start = vma->vm_start; /* sleeping forbidden, but vma pointer access allowed */ bpf_iter_task_vma_release(&___it); /* mmap_lock released, vma pointer invalidated */ /* sleeping is fine here */ bpf_copy_from_user(&buf, sizeof(buf), (void *)start); }The series is organized as:
Patch 1: KF_ACQUIRE/KF_RELEASE plumbing for iterators in the
verifier. Pure infrastructure, no behavioral change to existing iterators.Patch 2: Consolidate sleepable context error message printing
incheck_helper_call(), no functional changesPatch 3:
KF_FORBID_SLEEPflag andforbid_sleep_countmachinery.
Generic, works for anyKF_ACQUIREkfunc - iterator or not.Patch 4: Move
mmap_lockacquisition from_newto_nextin the
task_vmaiterator, preparing for re-acquisition after release.Patch 5: Wire up
task_vmaas the first user: annotate_nextwith
KF_ACQUIRE|KF_FORBID_SLEEP, addbpf_iter_task_vma_release().Patch 6: Selftests covering the runtime path (release +
copy_from_user)
and verifier rejection of invalid patterns (sleeping without release, VMA access after release, double release, release without acquire, nested iterator interaction).Puranjay Mohan (6):
bpf: AddKF_ACQUIREandKF_RELEASEsupport for iterators
bpf: consolidate sleepable context error message printing
bpf: AddKF_FORBID_SLEEPmodifier forKF_ACQUIREkfuncs
bpf: Move locking tobpf_iter_task_vma_next()
bpf: Add split iteration support totask_vmaiterator selftests/bpf: Add tests for splittask_vmaiteratorinclude/linux/bpf_verifier.h | 2 + include/linux/btf.h | 1 + kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 3 +- kernel/bpf/task_iter.c | 44 ++++-- kernel/bpf/verifier.c | 147 +++++++++++++----- .../testing/selftests/bpf/bpf_experimental.h | 1 + .../testing/selftests/bpf/prog_tests/iters.c | 13 ++ .../selftests/bpf/progs/iters_task_vma.c | 39 +++++ .../bpf/progs/iters_task_vma_nosleep.c | 125 +++++++++++++++ 10 files changed, 331 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/iters_task_vma_nosleep.c
Quoted 4 lines
base-commit:
9cd168a2720bc614647dd20bc5d1c308c56a44ec--
2.47.3
Some patches were dropped from this, will resend again.