Skip to contentSkip to conversation

bpf: Track overlapping RCU protection

PATCH In review Ning Ding 3 participants 9 messages Aug 5 – Aug 10

Subsystems (4)
lore
Share & tools
lore

Ning Ding

author
#0 lore raw

cover bpf: Track overlapping RCU protection

Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.

Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.

This follows review of the applied spin-unlock fix series [1].

Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq

[1] https://lore.kernel.org/r/[email protected]

Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protection

 kernel/bpf/verifier.c                         |  13 +-
 .../selftests/bpf/prog_tests/task_kfunc.c     |   6 +
 .../selftests/bpf/progs/task_kfunc_common.h   |   2 +
 .../selftests/bpf/progs/task_kfunc_failure.c  |  49 ++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 147 ++++++++++++++++++
 5 files changed, 214 insertions(+), 3 deletions(-)

base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e

Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/2 bpf: Account for preempt and IRQ state in RCU protection

Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.

Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.

---
 kernel/bpf/verifier.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/verifier.c +10 -3
index d925197c2e5f7..a1005783db300 100644
@@ -4452,8 +4452,9 @@ static bool in_sleepable(struct bpf_verifier_env *env)
 static bool in_rcu_cs(struct bpf_verifier_env *env)
 {
 	return env->cur_state->active_rcu_locks ||
-	       env->cur_state->active_locks ||
-	       !in_sleepable(env);
+	       env->cur_state->active_irq_id ||
+	       env->cur_state->active_preempt_locks ||
+	       env->cur_state->active_locks || !in_sleepable(env);
 }
 
 /* Once GCC supports btf_type_tag the following mechanism will be replaced with tag check */
@@ -11663,6 +11664,9 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 		err = unmark_stack_slot_irq_flag(env, reg, kfunc_class);
 		if (err)
 			return err;
+
+		if (!in_rcu_cs(env))
+			invalidate_rcu_protected_refs(env);
 	}
 	return 0;
 }
@@ -13181,7 +13185,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			verbose(env, "unmatched rcu read unlock (kernel function %s)\n", func_name);
 			return -EINVAL;
 		}
-		if (--env->cur_state->active_rcu_locks == 0)
+		env->cur_state->active_rcu_locks--;
+		if (!in_rcu_cs(env))
 			invalidate_rcu_protected_refs(env);
 	} else if (preempt_disable) {
 		env->cur_state->active_preempt_locks++;
@@ -13191,6 +13196,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return -EINVAL;
 		}
 		env->cur_state->active_preempt_locks--;
+		if (!in_rcu_cs(env))
+			invalidate_rcu_protected_refs(env);
 	}
 
 	if (sleepable && !in_sleepable_context(env)) {
Signature · 1 line
2.43.0
maintainer Acked-by on v1 1/2
#2 lore raw

Re: [PATCH bpf-next 1/2] bpf: Account for preempt and IRQ state in RCU protection

Quoted 11 lines · Ning Ding

Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.

Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.

---

Hi Ning,

Thank you for identifying and fixing this issue!

...

maintainer
#3 lore raw

Re: [PATCH bpf-next 1/2] bpf: Account for preempt and IRQ state in RCU protection

Quoted 13 lines · Eduard Zingerman
Quoted 11 lines · Ning Ding

Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.

Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.

---

Quoted 7 lines

Hi Ning,

Thank you for identifying and fixing this issue!

...

Please disregard the above email, it was meant for the "bpf: Compare iterator types during state pruning" series.

author
#4 lore raw

PATCH 2/2 selftests/bpf: Test overlapping RCU protection

In reply to Ning Ding

Add task kptr tests that keep RCU protection active after a spin or RCU unlock when preemption or IRQs remain disabled.

Also test the reverse order with explicit RCU. Verify that task kptrs are rejected after leaving the final preemption-disabled or IRQ-disabled region.

---
 .../selftests/bpf/prog_tests/task_kfunc.c     |   6 +
 .../selftests/bpf/progs/task_kfunc_common.h   |   2 +
 .../selftests/bpf/progs/task_kfunc_failure.c  |  49 ++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 147 ++++++++++++++++++
 4 files changed, 204 insertions(+)
tools/testing/selftests/bpf/prog_tests/task_kfunc.c +6 -0
index fbd7855712c1a..30d403028f984 100644
@@ -178,6 +178,12 @@ static const char * const success_tests[] = {
 	"task_kfunc_acquire_trusted_walked",
 	"task_kfunc_acquire_after_spin_unlock_non_sleepable",
 	"task_kfunc_acquire_after_spin_unlock_explicit_rcu",
+	"task_kfunc_acquire_after_spin_unlock_preempt_disabled",
+	"task_kfunc_acquire_after_spin_unlock_irq_disabled",
+	"task_kfunc_acquire_after_rcu_unlock_preempt_disabled",
+	"task_kfunc_acquire_after_rcu_unlock_irq_disabled",
+	"task_kfunc_acquire_after_preempt_enable_explicit_rcu",
+	"task_kfunc_acquire_after_irq_restore_explicit_rcu",
 	"test_task_kfunc_flavor_relo",
 	"test_task_kfunc_flavor_relo_not_found",
 };
tools/testing/selftests/bpf/progs/task_kfunc_common.h +2 -0
index 052c9d0e3e2a8..a0c599b58c290 100644
@@ -38,6 +38,8 @@ struct task_struct *bpf_task_from_pid(s32 pid) __ksym;
 struct task_struct *bpf_task_from_vpid(s32 vpid) __ksym;
 void bpf_rcu_read_lock(void) __ksym;
 void bpf_rcu_read_unlock(void) __ksym;
+void bpf_local_irq_save(unsigned long *flags) __weak __ksym;
+void bpf_local_irq_restore(unsigned long *flags) __weak __ksym;
 
 static inline struct __tasks_kfunc_map_value *tasks_kfunc_map_value_lookup(struct task_struct *p)
 {
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +49 -0

52 lines not on this page · raw message

tools/testing/selftests/bpf/progs/task_kfunc_success.c +147 -0

153 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw
In reply to Ning Ding
Quoted 9 lines · Ning Ding

Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.

Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.

Quoted 7 lines

This follows review of the applied spin-unlock fix series [1].

Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq

Overall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.

Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.

Also, do we need that was_in_rcu_cs logic for bpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.

https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]

pw-bot: cr

Quoted 15 lines

[1] https://lore.kernel.org/r/[email protected]

Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protection

 kernel/bpf/verifier.c                         |  13 +-
 .../selftests/bpf/prog_tests/task_kfunc.c     |   6 +
 .../selftests/bpf/progs/task_kfunc_common.h   |   2 +
 .../selftests/bpf/progs/task_kfunc_failure.c  |  49 ++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 147 ++++++++++++++++++
 5 files changed, 214 insertions(+), 3 deletions(-)

base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e

#6 lore raw
Quoted 25 lines · Kumar Kartikeya Dwivedi
Quoted 9 lines · Ning Ding

Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.

Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.

Quoted 7 lines

This follows review of the applied spin-unlock fix series [1].

Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq

Overall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.

Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.

Quoted 8 lines

Also, do we need that was_in_rcu_cs logic for bpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.

https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]

pw-bot: cr

I think from cursory look, the reason is that we forced any MEM_RCU pointer to be invalidated before, even when program wasn't sleepable, which does not happen now. The failing test used a non-sleepable program. So making it sleepable should preserve the failure case. In some ways, this is probably an improvement over current behavior.

Quoted 15 lines
Quoted 15 lines

[1] https://lore.kernel.org/r/[email protected]

Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protection

 kernel/bpf/verifier.c                         |  13 +-
 .../selftests/bpf/prog_tests/task_kfunc.c     |   6 +
 .../selftests/bpf/progs/task_kfunc_common.h   |   2 +
 .../selftests/bpf/progs/task_kfunc_failure.c  |  49 ++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 147 ++++++++++++++++++
 5 files changed, 214 insertions(+), 3 deletions(-)

base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e

#7 lore raw
Quoted 32 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Kumar Kartikeya Dwivedi
Quoted 9 lines · Ning Ding

Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.

Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.

Quoted 7 lines

This follows review of the applied spin-unlock fix series [1].

Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq

Overall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.

Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.

Also, do we need that was_in_rcu_cs logic for bpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.

https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]

Quoted 9 lines
Quoted 2 lines

pw-bot: cr

I think from cursory look, the reason is that we forced any MEM_RCU pointer to be invalidated before, even when program wasn't sleepable, which does not happen now. The failing test used a non-sleepable program. So making it sleepable should preserve the failure case. In some ways, this is probably an improvement over current behavior.

I fixed this up since it was minor and applied the series. Thanks.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

[1] https://lore.kernel.org/r/[email protected]

Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protection

 kernel/bpf/verifier.c                         |  13 +-
 .../selftests/bpf/prog_tests/task_kfunc.c     |   6 +
 .../selftests/bpf/progs/task_kfunc_common.h   |   2 +
 .../selftests/bpf/progs/task_kfunc_failure.c  |  49 ++++++
 .../selftests/bpf/progs/task_kfunc_success.c  | 147 ++++++++++++++++++
 5 files changed, 214 insertions(+), 3 deletions(-)

base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e

author
#8 lore raw
Quoted 1 line

I fixed this up since it was minor and applied the series. Thanks.

Sounds good, thanks!