bpf: Invalidate RCU pointers after final spin unlock
Subsystems (4)
v2 cover bpf: Invalidate RCU pointers after final spin unlock
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed task_struct use-after-free.
Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.
Testing used fresh QEMU/KVM guests with KASAN enabled. The patched focused test passed all three expected outcomes. The full task_kfunc test passed all 39 subtests, and the selected RCU, refcount, and spin-lock group had no failures.
---
v2:
- Rebase onto bpf-next commit 60781269e26c.
- Target bpf-next and split the fix from its selftests, as requested.
- Add positive controls for RCU contexts that remain valid after unlock.
v1: https://lore.kernel.org/r/[email protected]
Ning Ding (2):
bpf: Invalidate RCU pointers after final spin unlock selftests/bpf: Test RCU pointer invalidation after spin unlock
kernel/bpf/verifier.c | 5 ++ .../selftests/bpf/prog_tests/task_kfunc.c | 2 + .../selftests/bpf/progs/task_kfunc_common.h | 12 +++++ .../selftests/bpf/progs/task_kfunc_failure.c | 24 ++++++++++ .../selftests/bpf/progs/task_kfunc_success.c | 48 +++++++++++++++++++ 5 files changed, 91 insertions(+)
base-commit: 60781269e26c786de2bb93fb1e697a5c32ccee48
Signature · 1 line
2.43.0
PATCH v2 1/2 bpf: Invalidate RCU pointers after final spin unlock
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final bpf_spin_unlock() ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a task_struct use-after-free in __bpf_get_task_stack().
Record whether the program is in an RCU-protected context before releasing the lock. Invalidate RCU-protected pointers only when the unlock leaves the final such context. This preserves valid pointers in non-sleepable programs and inside an explicit RCU read-side section.
--- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
@@ -206,6 +206,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parstatic int release_reference_nomark (struct bpf_verifier_state *state,int id);static int release_reference (struct bpf_verifier_env *env,int id);static void invalidate_non_owning_refs (struct bpf_verifier_env *env); +static void invalidate_rcu_protected_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env);static bool is_tracing_prog_type (enum bpf_prog_type type); static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -7165,6 +7166,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_statereturn err; } }else { +bool was_in_rcu_cs;void *ptr;int type;
@@ -7192,10 +7194,13 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_stateverbose (env,"%s_unlock cannot be out of order\n" , lock_str);return -EINVAL ; } + was_in_rcu_cs =in_rcu_cs (env);if (release_lock_state (cur, type, reg->id, ptr)) {verbose (env,"%s_unlock of different lock\n" , lock_str);return -EINVAL ; } +if (was_in_rcu_cs && !in_rcu_cs (env)) +invalidate_rcu_protected_refs (env);invalidate_non_owning_refs (env); }
Signature · 1 line
2.43.0
Kumar Kartikeya Dwivedi
maintainerQuoted 11 lines · Ning Ding
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final
bpf_spin_unlock()ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered atask_structuse-after-free in__bpf_get_task_stack().Record whether the program is in an RCU-protected context before releasing the lock. Invalidate RCU-protected pointers only when the unlock leaves the final such context. This preserves valid pointers in non-sleepable programs and inside an explicit RCU read-side section.
Quoted 5 lines
Fixes 5861d1e8dbc4 ("bpf: Allow bpf_spin_{lock,unlock} in sleepable progs")Assisted-by Codex:gpt-5.6-solAssisted-by ChatGPT:GPT-5.6-ProSigned-off-by Ning Ding <[email protected]>---
This patch set makes sense. I think while we are at it, we should improve the behavior here.
For RCU unlock, we currently unconditionally invalidate refs when it hits zero, even though we may have other sources of RCU protection. The only reason it doesn't matter right now is probably because we cannot do a RCU read unlock within a spin lock critical section.
We also do not currently consider bpf_preempt_disable() as providing RCU protection, even though we should.
Could you extend the series with these changes? And corresponding tests where possible? It might not be possible for all combinations, but we can test those that are permitted.
Unlike the correctness fix in this change, these are mostly usability paper cuts, so do not need a Fixes: tag.
Thanks!
Quoted 1 line
[...]
PATCH v2 2/2 selftests/bpf: Test RCU pointer invalidation after spin unlock
In reply to Ning DingThe verifier previously accepted a task kptr after the final spin unlock ended its RCU protection in a sleepable program. The pointer could then be used after the task was freed.
Add a negative test for that case. Add positive controls showing that the pointer remains valid in a non-sleepable program and while an explicit RCU read-side section is still active.
--- .../selftests/bpf/prog_tests/task_kfunc.c | 2 + .../selftests/bpf/progs/task_kfunc_common.h | 12 +++++ .../selftests/bpf/progs/task_kfunc_failure.c | 24 ++++++++++ .../selftests/bpf/progs/task_kfunc_success.c | 48 +++++++++++++++++++ 4 files changed, 86 insertions(+)
tools/testing/selftests/bpf/prog_tests/task_kfunc.c +2 -0
@@ -176,6 +176,8 @@ static const char * const success_tests[] = {"test_task_from_pid_current" ,"test_task_from_pid_invalid" ,"task_kfunc_acquire_trusted_walked" , +"task_kfunc_acquire_after_spin_unlock_non_sleepable" , +"task_kfunc_acquire_after_spin_unlock_explicit_rcu" ,"test_task_kfunc_flavor_relo" , "test_task_kfunc_flavor_relo_not_found", };
tools/testing/selftests/bpf/progs/task_kfunc_common.h +12 -0
@@ -20,6 +20,18 @@ struct {__uint (max_entries,1 ); } __tasks_kfunc_map SEC(".maps"); +struct task_kptr_lock_value { +struct bpf_spin_lock lock; + struct task_struct __kptr * task; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct task_kptr_lock_value); + __uint(max_entries, 1); +} task_kptr_lock_map SEC(".maps"); +struct task_struct *bpf_task_acquire (struct task_struct *p) __ksym; void bpf_task_release(struct task_struct *p) __ksym;struct task_struct *bpf_task_from_pid (s32 pid) __ksym;
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +24 -0
@@ -378,3 +378,27 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flagreturn 0 ; } + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("R1 must be a rcu pointer") +int BPF_PROG(task_kfunc_acquire_after_final_spin_unlock) +{ +struct task_kptr_lock_value *v; +struct task_struct *task, *acquired; +int key =0 ; + + v =bpf_map_lookup_elem (&task_kptr_lock_map, &key); +if (!v) +return 0 ; + +bpf_spin_lock (&v->lock); + task = v->task; +bpf_spin_unlock (&v->lock); +if (!task) +return 0 ; + + acquired =bpf_task_acquire (task); +if (acquired) +bpf_task_release (acquired); +return 0 ; +}
tools/testing/selftests/bpf/progs/task_kfunc_success.c +48 -0
60 lines not on this page · raw message
Signature · 1 line
2.43.0
This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:
Quoted 11 lines · On Mon, 3 Aug 2026 04:26:07 -0700 you wrote:
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed
task_structuse-after-free.Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.
[...]
Here is the summary with links:
- [bpf-next,v2,1/2] bpf: Invalidate RCU pointers after final spin unlock
https://git.kernel.org/bpf/bpf-next/c/180c7000712d - [bpf-next,v2,2/2] selftests/bpf: Test RCU pointer invalidation after spin unlock https://git.kernel.org/bpf/bpf-next/c/bb2df6fd891d
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 9 lines · Ning Ding
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed
task_structuse-after-free.Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.
I think if we are doing this we should do it for all things that act as an entry to a RCU read side critical section [1]:
1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock
As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.
Paul, can you ack this? Let us know if there are more things that can make a BPF program enter a read side critical section.
Thanks,
Puranjay
[1] https://docs.kernel.org/RCU/whatisRCU.html#:~:text=Note%20that%20anything,side%20critical%20sections.
Quoted 20 lines · Puranjay Mohan
Quoted 9 lines · Ning Ding
In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed
task_structuse-after-free.Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.
I think if we are doing this we should do it for all things that act as an entry to a RCU read side critical section [1]:
1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock
Quoted 5 lines
As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.
Paul, can you ack this? Let us know if there are more things that can make a BPF program enter a read side critical section.
This is quite true!
There is a range of complexity/capability here. A set of partially overlapping RCU read-side critical sections are implicitly "flattened" into one long critical section covering the full set. For example:
spin_lock(&mylock); // starts RCU read-side critical section... local_irq_disable(); spin_unlock(&mylock); preempt_disable(); local_irq_enable(); local_bh_disable(); preempt_enable(); local_bh_enable(); // ... and continues to here.
And rcutorture tests randomly chosen combinations up to eight segments long.
So it is an interesting choice as to how much of this should be accounted for by BPF. Choose wisely!!! ;-)
Thanx, Paul
Quoted 4 lines
Thanks,
Puranjay[1] https://docs.kernel.org/RCU/whatisRCU.html#:~:text=Note%20that%20anything,side%20critical%20sections.
Quoted 7 lines
Quoted 7 lines
1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lockAs you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.
Hi, I have written the follow-up patch to account for preemption-disabled and irq-disabled regions. But still working on the selftests.
While checking rcu gaps, I also found what looks like a separate issue for state pruning also regarding the MEM_RCU and PTR_UNTRUSTED. A sample example is:
bpf_rcu_read_lock();
bpf_iter_task_new(&it, NULL, BPF_TASK_ITER_ALL_PROCS);
if (condition) {
/* Safe: the outer RCU lock remains active. */
bpf_rcu_read_lock();
bpf_rcu_read_unlock();
} else {
/* Unsafe: protection reaches zero, creating an unguarded gap. */
bpf_rcu_read_unlock();
bpf_rcu_read_lock();
}
/* Both branches have one active RCU lock here. */
bpf_iter_task_next(&it);
bpf_rcu_read_unlock();But STACK_ITER comparison in stacksafe() does not compare spilled_ptr.type. So the verifier considers the trusted and untrusted iterator states are equivalent. It can possibly prune the unsafe path before is_iter_reg_valid_init() rejects it.
Kumar Kartikeya Dwivedi
maintainerQuoted 32 lines · Ning Ding
Quoted 7 lines
Quoted 7 lines
1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lockAs you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.
Hi, I have written the follow-up patch to account for preemption-disabled and irq-disabled regions. But still working on the selftests.
While checking rcu gaps, I also found what looks like a separate issue for state pruning also regarding theMEM_RCUandPTR_UNTRUSTED. A sample example is:bpf_rcu_read_lock(); bpf_iter_task_new(&it, NULL, BPF_TASK_ITER_ALL_PROCS); if (condition) { /* Safe: the outer RCU lock remains active. */ bpf_rcu_read_lock(); bpf_rcu_read_unlock(); } else { /* Unsafe: protection reaches zero, creating an unguarded gap. */ bpf_rcu_read_unlock(); bpf_rcu_read_lock(); } /* Both branches have one active RCU lock here. */ bpf_iter_task_next(&it); bpf_rcu_read_unlock();
Quoted 5 lines
But
STACK_ITERcomparison instacksafe()does not compare spilled_ptr.type. So the verifier considers the trusted and untrusted iterator states are equivalent. It can possibly prune the unsafe path beforeis_iter_reg_valid_init()rejects it.
Sounds plausible. Once you have a fix and test that triggers bad pruning behavior reliably, please send it as a patch set.