Skip to contentSkip to conversation

bpf: Invalidate RCU pointers after final spin unlock

PATCH Applied Ning Ding 5 participants 9 messages Aug 3 – Aug 6

Subsystems (4)
lore
Share & tools
lore

Ning Ding

author
#0 lore raw

v2 cover bpf: Invalidate RCU pointers after final spin unlock

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed task_struct use-after-free.

Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.

Testing used fresh QEMU/KVM guests with KASAN enabled. The patched focused test passed all three expected outcomes. The full task_kfunc test passed all 39 subtests, and the selected RCU, refcount, and spin-lock group had no failures.

---
v2:
- Rebase onto bpf-next commit 60781269e26c.
- Target bpf-next and split the fix from its selftests, as requested.
- Add positive controls for RCU contexts that remain valid after unlock.

v1: https://lore.kernel.org/r/[email protected]

Ning Ding (2):
bpf: Invalidate RCU pointers after final spin unlock selftests/bpf: Test RCU pointer invalidation after spin unlock

 kernel/bpf/verifier.c                         |  5 ++
 .../selftests/bpf/prog_tests/task_kfunc.c     |  2 +
 .../selftests/bpf/progs/task_kfunc_common.h   | 12 +++++
 .../selftests/bpf/progs/task_kfunc_failure.c  | 24 ++++++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 48 +++++++++++++++++++
 5 files changed, 91 insertions(+)

base-commit: 60781269e26c786de2bb93fb1e697a5c32ccee48

Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v2 1/2 bpf: Invalidate RCU pointers after final spin unlock

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final bpf_spin_unlock() ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a task_struct use-after-free in __bpf_get_task_stack().

Record whether the program is in an RCU-protected context before releasing the lock. Invalidate RCU-protected pointers only when the unlock leaves the final such context. This preserves valid pointers in non-sleepable programs and inside an explicit RCU read-side section.

Fixes 5861d1e8dbc4 ("bpf: Allow bpf_spin_{lock,unlock} in sleepable progs")
Assisted-by Codex:gpt-5.6-sol
Assisted-by ChatGPT:GPT-5.6-Pro
---
 kernel/bpf/verifier.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
index b274004fccfd9..7439afdc851a7 100644
@@ -206,6 +206,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int par
 static int release_reference_nomark(struct bpf_verifier_state *state, int id);
 static int release_reference(struct bpf_verifier_env *env, int id);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
+static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static bool is_tracing_prog_type(enum bpf_prog_type type);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -7165,6 +7166,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
 			return err;
 		}
 	} else {
+		bool was_in_rcu_cs;
 		void *ptr;
 		int type;
 
@@ -7192,10 +7194,13 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state
 			verbose(env, "%s_unlock cannot be out of order\n", lock_str);
 			return -EINVAL;
 		}
+		was_in_rcu_cs = in_rcu_cs(env);
 		if (release_lock_state(cur, type, reg->id, ptr)) {
 			verbose(env, "%s_unlock of different lock\n", lock_str);
 			return -EINVAL;
 		}
+		if (was_in_rcu_cs && !in_rcu_cs(env))
+			invalidate_rcu_protected_refs(env);
 
 		invalidate_non_owning_refs(env);
 	}
Signature · 1 line
2.43.0
#2 lore raw
Quoted 11 lines · Ning Ding

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final bpf_spin_unlock() ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a task_struct use-after-free in __bpf_get_task_stack().

Record whether the program is in an RCU-protected context before releasing the lock. Invalidate RCU-protected pointers only when the unlock leaves the final such context. This preserves valid pointers in non-sleepable programs and inside an explicit RCU read-side section.

Quoted 5 lines
Fixes 5861d1e8dbc4 ("bpf: Allow bpf_spin_{lock,unlock} in sleepable progs")
Assisted-by Codex:gpt-5.6-sol
Assisted-by ChatGPT:GPT-5.6-Pro

---

This patch set makes sense. I think while we are at it, we should improve the behavior here.

For RCU unlock, we currently unconditionally invalidate refs when it hits zero, even though we may have other sources of RCU protection. The only reason it doesn't matter right now is probably because we cannot do a RCU read unlock within a spin lock critical section.

We also do not currently consider bpf_preempt_disable() as providing RCU protection, even though we should.

Could you extend the series with these changes? And corresponding tests where possible? It might not be possible for all combinations, but we can test those that are permitted.

Unlike the correctness fix in this change, these are mostly usability paper cuts, so do not need a Fixes: tag.

Thanks!

Quoted 1 line

[...]

author
#3 lore raw

PATCH v2 2/2 selftests/bpf: Test RCU pointer invalidation after spin unlock

In reply to Ning Ding

The verifier previously accepted a task kptr after the final spin unlock ended its RCU protection in a sleepable program. The pointer could then be used after the task was freed.

Add a negative test for that case. Add positive controls showing that the pointer remains valid in a non-sleepable program and while an explicit RCU read-side section is still active.

Assisted-by Codex:gpt-5.6-sol
Assisted-by ChatGPT:GPT-5.6-Pro
---
 .../selftests/bpf/prog_tests/task_kfunc.c     |  2 +
 .../selftests/bpf/progs/task_kfunc_common.h   | 12 +++++
 .../selftests/bpf/progs/task_kfunc_failure.c  | 24 ++++++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 48 +++++++++++++++++++
 4 files changed, 86 insertions(+)
tools/testing/selftests/bpf/prog_tests/task_kfunc.c +2 -0
index e6e95c1416e65..fbd7855712c1a 100644
@@ -176,6 +176,8 @@ static const char * const success_tests[] = {
 	"test_task_from_pid_current",
 	"test_task_from_pid_invalid",
 	"task_kfunc_acquire_trusted_walked",
+	"task_kfunc_acquire_after_spin_unlock_non_sleepable",
+	"task_kfunc_acquire_after_spin_unlock_explicit_rcu",
 	"test_task_kfunc_flavor_relo",
 	"test_task_kfunc_flavor_relo_not_found",
 };
tools/testing/selftests/bpf/progs/task_kfunc_common.h +12 -0
index e9c4fea7a4bba..052c9d0e3e2a8 100644
@@ -20,6 +20,18 @@ struct {
 	__uint(max_entries, 1);
 } __tasks_kfunc_map SEC(".maps");
 
+struct task_kptr_lock_value {
+	struct bpf_spin_lock lock;
+	struct task_struct __kptr * task;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__type(key, int);
+	__type(value, struct task_kptr_lock_value);
+	__uint(max_entries, 1);
+} task_kptr_lock_map SEC(".maps");
+
 struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
 void bpf_task_release(struct task_struct *p) __ksym;
 struct task_struct *bpf_task_from_pid(s32 pid) __ksym;
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +24 -0
index 5c99b1e6532bf..c0e7216b34193 100644
@@ -378,3 +378,27 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flag
 
 	return 0;
 }
+
+SEC("?fentry.s/" SYS_PREFIX "sys_getpgid")
+__failure __msg("R1 must be a rcu pointer")
+int BPF_PROG(task_kfunc_acquire_after_final_spin_unlock)
+{
+	struct task_kptr_lock_value *v;
+	struct task_struct *task, *acquired;
+	int key = 0;
+
+	v = bpf_map_lookup_elem(&task_kptr_lock_map, &key);
+	if (!v)
+		return 0;
+
+	bpf_spin_lock(&v->lock);
+	task = v->task;
+	bpf_spin_unlock(&v->lock);
+	if (!task)
+		return 0;
+
+	acquired = bpf_task_acquire(task);
+	if (acquired)
+		bpf_task_release(acquired);
+	return 0;
+}
tools/testing/selftests/bpf/progs/task_kfunc_success.c +48 -0

60 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw
In reply to Ning Ding
This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Kumar Kartikeya Dwivedi <[email protected]>:

Quoted 11 lines · On Mon, 3 Aug 2026 04:26:07 -0700 you wrote:

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed task_struct use-after-free.

Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.

[...]

Here is the summary with links:
- [bpf-next,v2,1/2] bpf: Invalidate RCU pointers after final spin unlock

    https://git.kernel.org/bpf/bpf-next/c/180c7000712d
  - [bpf-next,v2,2/2] selftests/bpf: Test RCU pointer invalidation after spin unlock
    https://git.kernel.org/bpf/bpf-next/c/bb2df6fd891d

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#5 lore raw
In reply to Ning Ding
Quoted 9 lines · Ning Ding

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed task_struct use-after-free.

Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.

I think if we are doing this we should do it for all things that act as an entry to a RCU read side critical section [1]:

1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock

As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.

Paul, can you ack this? Let us know if there are more things that can make a BPF program enter a read side critical section.

Thanks,
Puranjay

[1] https://docs.kernel.org/RCU/whatisRCU.html#:~:text=Note%20that%20anything,side%20critical%20sections.

#6 lore raw
Quoted 20 lines · Puranjay Mohan
Quoted 9 lines · Ning Ding

In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final spin unlock ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a KASAN-confirmed task_struct use-after-free.

Patch 1 invalidates RCU-protected pointers only when an unlock leaves the final RCU-protected context. Patch 2 adds a negative sleepable test and positive controls for non-sleepable and explicit-RCU contexts.

I think if we are doing this we should do it for all things that act as an entry to a RCU read side critical section [1]:

1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock

Quoted 5 lines

As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.

Paul, can you ack this? Let us know if there are more things that can make a BPF program enter a read side critical section.

This is quite true!

There is a range of complexity/capability here. A set of partially overlapping RCU read-side critical sections are implicitly "flattened" into one long critical section covering the full set. For example:

	spin_lock(&mylock); // starts RCU read-side critical section...
	local_irq_disable();
	spin_unlock(&mylock);
	preempt_disable();
	local_irq_enable();
	local_bh_disable();
	preempt_enable();
	local_bh_enable();  // ... and continues to here.

And rcutorture tests randomly chosen combinations up to eight segments long.

So it is an interesting choice as to how much of this should be accounted for by BPF. Choose wisely!!! ;-)

Thanx, Paul

Quoted 4 lines

Thanks,
Puranjay

[1] https://docs.kernel.org/RCU/whatisRCU.html#:~:text=Note%20that%20anything,side%20critical%20sections.

author
#7 lore raw
Quoted 7 lines
Quoted 7 lines

1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock

As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.

Hi, I have written the follow-up patch to account for preemption-disabled and irq-disabled regions. But still working on the selftests.
While checking rcu gaps, I also found what looks like a separate issue for state pruning also regarding the MEM_RCU and PTR_UNTRUSTED. A sample example is:

  bpf_rcu_read_lock();
  bpf_iter_task_new(&it, NULL, BPF_TASK_ITER_ALL_PROCS);

  if (condition) {
  /* Safe: the outer RCU lock remains active. */
  bpf_rcu_read_lock();
  bpf_rcu_read_unlock();
  } else {
  /* Unsafe: protection reaches zero, creating an unguarded gap. */
  bpf_rcu_read_unlock();
  bpf_rcu_read_lock();
  }

  /* Both branches have one active RCU lock here. */
  bpf_iter_task_next(&it);
  bpf_rcu_read_unlock();

But STACK_ITER comparison in stacksafe() does not compare spilled_ptr.type. So the verifier considers the trusted and untrusted iterator states are equivalent. It can possibly prune the unsafe path before is_iter_reg_valid_init() rejects it.

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/kernel/bpf/states.c?id=6655c409707ec8ce9ce0850ffe4fe02331fd4d9c#n806

#8 lore raw
Quoted 32 lines · Ning Ding
Quoted 7 lines
Quoted 7 lines

1. Disabling preemption
2. Disabling bottom halves
3. Disabling interrupts
4. Acquiring a spin lock

As you are doing it for 4, can you also do it for 1 and 3? I don't think BPF can disable bottom halves right now.

Hi, I have written the follow-up patch to account for preemption-disabled and irq-disabled regions. But still working on the selftests.
While checking rcu gaps, I also found what looks like a separate issue for state pruning also regarding the MEM_RCU and PTR_UNTRUSTED. A sample example is:

  bpf_rcu_read_lock();
  bpf_iter_task_new(&it, NULL, BPF_TASK_ITER_ALL_PROCS);

  if (condition) {
  /* Safe: the outer RCU lock remains active. */
  bpf_rcu_read_lock();
  bpf_rcu_read_unlock();
  } else {
  /* Unsafe: protection reaches zero, creating an unguarded gap. */
  bpf_rcu_read_unlock();
  bpf_rcu_read_lock();
  }

  /* Both branches have one active RCU lock here. */
  bpf_iter_task_next(&it);
  bpf_rcu_read_unlock();
Quoted 5 lines

But STACK_ITER comparison in stacksafe() does not compare spilled_ptr.type. So the verifier considers the trusted and untrusted iterator states are equivalent. It can possibly prune the unsafe path before is_iter_reg_valid_init() rejects it.

Sounds plausible. Once you have a fix and test that triggers bad pruning behavior reliably, please send it as a patch set.

Quoted 1 line

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/kernel/bpf/states.c?id=6655c409707ec8ce9ce0850ffe4fe02331fd4d9c#n806