bpf: Track overlapping RCU protection
Subsystems (4)
cover bpf: Track overlapping RCU protection
Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.
Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.
This follows review of the applied spin-unlock fix series [1].
Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq
[1] https://lore.kernel.org/r/[email protected]
Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protection
kernel/bpf/verifier.c | 13 +- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 5 files changed, 214 insertions(+), 3 deletions(-)
base-commit: 11c1e836710dcba03e50454a4eedfdbaf8d3050e
Signature · 1 line
2.43.0
PATCH 1/2 bpf: Account for preempt and IRQ state in RCU protection
Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.
Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.
--- kernel/bpf/verifier.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/bpf/verifier.c +10 -3
@@ -4452,8 +4452,9 @@ static bool in_sleepable(struct bpf_verifier_env *env)static bool in_rcu_cs (struct bpf_verifier_env *env) {return env->cur_state->active_rcu_locks || - env->cur_state->active_locks || - !in_sleepable (env); + env->cur_state->active_irq_id || + env->cur_state->active_preempt_locks || + env->cur_state->active_locks || !in_sleepable (env); }/* Once GCC supports btf_type_tag the following mechanism will be replaced with tag check */
@@ -11663,6 +11664,9 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state * err =unmark_stack_slot_irq_flag (env, reg, kfunc_class);if (err)return err; + +if (!in_rcu_cs (env)) +invalidate_rcu_protected_refs (env); }return 0 ; }
@@ -13181,7 +13185,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,verbose (env,"unmatched rcu read unlock (kernel function %s)\n" , func_name);return -EINVAL ; } -if (--env->cur_state->active_rcu_locks ==0 ) + env->cur_state->active_rcu_locks--; +if (!in_rcu_cs (env))invalidate_rcu_protected_refs (env); }else if (preempt_disable) { env->cur_state->active_preempt_locks++;
@@ -13191,6 +13196,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EINVAL ; } env->cur_state->active_preempt_locks--; +if (!in_rcu_cs (env)) +invalidate_rcu_protected_refs (env); }if (sleepable && !in_sleepable_context (env)) {
Signature · 1 line
2.43.0
Re: [PATCH bpf-next 1/2] bpf: Account for preempt and IRQ state in RCU protection
Quoted 11 lines · Ning Ding
Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but
in_rcu_cs()does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.Include preemption-disabled and IRQ-disabled state in
in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.Signed-off-by Ning Ding <[email protected]>---
Hi Ning,
Thank you for identifying and fixing this issue!
...
Eduard Zingerman
maintainerRe: [PATCH bpf-next 1/2] bpf: Account for preempt and IRQ state in RCU protection
Quoted 13 lines · Eduard Zingerman
Quoted 11 lines · Ning Ding
Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but
in_rcu_cs()does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends.Include preemption-disabled and IRQ-disabled state in
in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends.Signed-off-by Ning Ding <[email protected]>---
Quoted 7 lines
Hi Ning,
Thank you for identifying and fixing this issue!
Acked-by Eduard Zingerman <[email protected]>...
Please disregard the above email, it was meant for the "bpf: Compare iterator types during state pruning" series.
PATCH 2/2 selftests/bpf: Test overlapping RCU protection
In reply to Ning DingAdd task kptr tests that keep RCU protection active after a spin or RCU unlock when preemption or IRQs remain disabled.
Also test the reverse order with explicit RCU. Verify that task kptrs are rejected after leaving the final preemption-disabled or IRQ-disabled region.
--- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 4 files changed, 204 insertions(+)
tools/testing/selftests/bpf/prog_tests/task_kfunc.c +6 -0
@@ -178,6 +178,12 @@ static const char * const success_tests[] = {"task_kfunc_acquire_trusted_walked" ,"task_kfunc_acquire_after_spin_unlock_non_sleepable" ,"task_kfunc_acquire_after_spin_unlock_explicit_rcu" , +"task_kfunc_acquire_after_spin_unlock_preempt_disabled" , +"task_kfunc_acquire_after_spin_unlock_irq_disabled" , +"task_kfunc_acquire_after_rcu_unlock_preempt_disabled" , +"task_kfunc_acquire_after_rcu_unlock_irq_disabled" , +"task_kfunc_acquire_after_preempt_enable_explicit_rcu" , +"task_kfunc_acquire_after_irq_restore_explicit_rcu" ,"test_task_kfunc_flavor_relo" , "test_task_kfunc_flavor_relo_not_found", };
tools/testing/selftests/bpf/progs/task_kfunc_common.h +2 -0
@@ -38,6 +38,8 @@ struct task_struct *bpf_task_from_pid(s32 pid) __ksym;struct task_struct *bpf_task_from_vpid (s32 vpid) __ksym; void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym; +void bpf_local_irq_save(unsigned long *flags) __weak __ksym; +void bpf_local_irq_restore(unsigned long *flags) __weak __ksym;static inline struct __tasks_kfunc_map_value *tasks_kfunc_map_value_lookup (struct task_struct *p) {
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +49 -0
52 lines not on this page · raw message
tools/testing/selftests/bpf/progs/task_kfunc_success.c +147 -0
153 lines not on this page · raw message
Signature · 1 line
2.43.0
Kumar Kartikeya Dwivedi
maintainerQuoted 9 lines · Ning Ding
Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.
Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.
Quoted 7 lines
This follows review of the applied spin-unlock fix series [1].
Tested in QEMU/KVM:
./test_progs -ttask_kfunc
./test_progs -tpreempt_lock
./test_progs -t irq
Overall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.
Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.
Also, do we need that was_in_rcu_cs logic for bpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]
pw-bot: cr
Quoted 15 lines
[1] https://lore.kernel.org/r/[email protected]
Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protectionkernel/bpf/verifier.c | 13 +- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 5 files changed, 214 insertions(+), 3 deletions(-)base-commit:
11c1e836710dcba03e50454a4eedfdbaf8d3050e
Kumar Kartikeya Dwivedi
maintainerQuoted 25 lines · Kumar Kartikeya Dwivedi
Quoted 9 lines · Ning Ding
Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.
Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.
Quoted 7 lines
This follows review of the applied spin-unlock fix series [1].
Tested in QEMU/KVM:
./test_progs -ttask_kfunc
./test_progs -tpreempt_lock
./test_progs -t irqOverall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.
Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.
Quoted 8 lines
Also, do we need that
was_in_rcu_cslogic forbpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]
pw-bot: cr
I think from cursory look, the reason is that we forced any MEM_RCU pointer to be invalidated before, even when program wasn't sleepable, which does not happen now. The failing test used a non-sleepable program. So making it sleepable should preserve the failure case. In some ways, this is probably an improvement over current behavior.
Quoted 15 lines
Quoted 15 lines
[1] https://lore.kernel.org/r/[email protected]
Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protectionkernel/bpf/verifier.c | 13 +- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 5 files changed, 214 insertions(+), 3 deletions(-)base-commit:
11c1e836710dcba03e50454a4eedfdbaf8d3050e
Kumar Kartikeya Dwivedi
maintainerQuoted 32 lines · Kumar Kartikeya Dwivedi
Quoted 31 lines · Kumar Kartikeya Dwivedi
Quoted 9 lines · Ning Ding
Preemption-disabled and IRQ-disabled regions provide RCU protection, but the verifier does not account for them. Current implementation can invalidate a task kptr while another RCU source remains active, or keep it valid after the final source ends.
Track these regions and invalidate RCU-protected pointers only after the last protection ends. Add task kptr tests for overlapping protection and final-exit rejection.
Quoted 7 lines
This follows review of the applied spin-unlock fix series [1].
Tested in QEMU/KVM:
./test_progs -ttask_kfunc
./test_progs -tpreempt_lock
./test_progs -t irqOverall, looks good. There are various failures in CI, so you might need to update or adjust existing selftests.
Make sure to include selftest changes for _existing_ tests within the kernel commit making changes, so as to preserve bisection. Your extra tests should still go in a separate commit.
Also, do we need that
was_in_rcu_cslogic forbpf_spin_lock(). Would it be possible to drop it and do it the same way you've done it for other cases? If that makes sense, I would fold that cleanup in the first patch as well.https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]
Quoted 9 lines
Quoted 2 lines
pw-bot: cr
I think from cursory look, the reason is that we forced any
MEM_RCUpointer to be invalidated before, even when program wasn't sleepable, which does not happen now. The failing test used a non-sleepable program. So making it sleepable should preserve the failure case. In some ways, this is probably an improvement over current behavior.
I fixed this up since it was minor and applied the series. Thanks.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
[1] https://lore.kernel.org/r/[email protected]
Ning Ding (2):
bpf: Account for preempt and IRQ state in RCU protection selftests/bpf: Test overlapping RCU protectionkernel/bpf/verifier.c | 13 +- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 5 files changed, 214 insertions(+), 3 deletions(-)base-commit:
11c1e836710dcba03e50454a4eedfdbaf8d3050e
Quoted 1 line
I fixed this up since it was minor and applied the series. Thanks.
Sounds good, thanks!