Fix global subprog verification context
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainerv3 cover Fix global subprog verification context
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Clarify that globals are checked only in contexts from which they are reached, with two passes only when both contexts occur. (Alexei)
* Implement in_sleepable_context() as !in_rcu_cs(). (Alexei, Eduard)
* Keep the original subprogram traversal order and repeat until all called contexts have been verified. (Eduard)
* Explain why instruction accounting needs the saved total and why the exception callback uses the main program's context. (Eduard)
* Rename the do_check_common() parameter to is_sleepable. (BPF CI)
* Preserve the harmless global calls with __weak __noinline and check their instruction totals across both contexts. Drop the redundant task-work global RCU test and retain the review acknowledgment. (Eduard)
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Preserve the harmless global subprog calls in the positive test with barrier() so LLVM cannot eliminate the intended coverage.
* Return zero explicitly from workqueue callbacks after global subprog calls.
* Document cumulative instruction accounting across verification contexts.
* Fix BPF multi-line comment formatting.
Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contexts
include/linux/bpf.h | 5 +- kernel/bpf/verifier.c | 71 ++++++------ .../bpf/progs/verifier_async_cb_context.c | 106 ++++++++++++++++++ 3 files changed, 146 insertions(+), 36 deletions(-)
base-commit: 29fcf9ce4bbdf4cc339def776860e1c0095bca98
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH v3 1/2 bpf: Verify global subprogs in each sleepability context
Global subprograms are verified independently with a fresh verifier root. do_check_common() currently seeds that root's in_sleepable state from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point.
In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making in_rcu_cs() true and allowing loads of RCU-protected kptrs to produce trusted MEM_RCU pointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free.
The verifier's execution-context predicates are complementary. A state is sleepable only when in_sleepable is set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, while in_rcu_cs() treats a non-sleepable state as implicitly protected.
Use this relationship to represent a global subprogram caller with only the result of in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makes in_rcu_cs() true without copying caller-owned lock state.
Track only the contexts in which each global subprogram is actually reached. Verify it once if all reachable calls use the same context, and twice only if both sleepable and non-sleepable calls reach it. Calls found while verifying globals or asynchronous callbacks mark further contexts for checking. Repeat the existing subprogram walk until all called contexts have been verified; unreachable global calls remain unchecked.
Accumulate instruction counts over those verification passes. Preserve the total recorded before each pass, since path accounting has already added this pass's synchronous instructions and its root total must also include asynchronous subprograms.
This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid.
--- include/linux/bpf.h | 5 +-- kernel/bpf/verifier.c | 71 ++++++++++++++++++++++--------------------- 2 files changed, 40 insertions(+), 36 deletions(-)
include/linux/bpf.h +3 -2
@@ -1682,8 +1682,9 @@ static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flagsstruct bpf_func_info_aux {u16 linkage;bool unreliable; -bool called :1 ; -bool verified :1 ; +/* Indexed by in_sleepable. */ +bool called[2 ]; +bool verified[2 ]; };enum bpf_jit_poke_reason {
kernel/bpf/verifier.c +37 -34
143 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH bpf v3 1/2] bpf: Verify global subprogs in each sleepability context
Quoted 42 lines · Kumar Kartikeya Dwivedi
Global subprograms are verified independently with a fresh verifier root.
do_check_common()currently seeds that root'sin_sleepablestate from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point.In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making
in_rcu_cs()true and allowing loads of RCU-protected kptrs to produce trustedMEM_RCUpointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free.The verifier's execution-context predicates are complementary. A state is sleepable only when
in_sleepableis set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, whilein_rcu_cs()treats a non-sleepable state as implicitly protected.Use this relationship to represent a global subprogram caller with only the result of
in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makesin_rcu_cs()true without copying caller-owned lock state.Track only the contexts in which each global subprogram is actually reached. Verify it once if all reachable calls use the same context, and twice only if both sleepable and non-sleepable calls reach it. Calls found while verifying globals or asynchronous callbacks mark further contexts for checking. Repeat the existing subprogram walk until all called contexts have been verified; unreachable global calls remain unchecked.
Accumulate instruction counts over those verification passes. Preserve the total recorded before each pass, since path accounting has already added this pass's synchronous instructions and its root total must also include asynchronous subprograms.
This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid.
Quoted 6 lines
Fixes 81f1d7a583fa ("bpf: wq: add bpf_wq_set_callback_impl")Fixes 38aa7003e369 ("bpf: task work scheduling kfuncs")Reported-by Nicholas Carlini <[email protected]>Suggested-by Nicholas Carlini <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
...
Kumar Kartikeya Dwivedi
maintainerPATCH v3 2/2 selftests/bpf: Test global subprog callback contexts
In reply to Kumar Kartikeya DwivediExercise global subprogram verification from workqueue callbacks, which can run in a sleepable context even when the containing program is not sleepable. An unprotected callback must not let the global subprogram use implicit RCU protection inherited from the program.
Add a negative case which loads an RCU-protected task kptr in a global subprogram reached from a workqueue callback. It fails on an unfixed kernel because the program is incorrectly accepted. Also cover a workqueue callback protected by an explicit RCU read-side critical section, where the same global subprogram remains valid.
Call the same harmless global subprogram directly from the main program and from an unprotected callback. Mark it __weak __noinline so both calls survive optimization, and check that its instruction statistics account for both verification contexts. This also verifies that global calls from callbacks are not rejected wholesale. Workqueue callbacks return zero explicitly after the global call, as required by their contract.
--- .../bpf/progs/verifier_async_cb_context.c | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+)
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +106 -0
@@ -9,6 +9,11 @@ char _license[] SEC("license") = "GPL"; +struct task_struct *bpf_task_acquire (struct task_struct *p) __ksym; +void bpf_task_release(struct task_struct *p) __ksym; +void bpf_rcu_read_lock(void) __ksym; +void bpf_rcu_read_unlock(void) __ksym; +/* Timer tests */ struct timer_elem {
@@ -164,6 +169,7 @@ int syscall_btf_find_prog(void *ctx)struct wq_elem {struct bpf_wq w; + struct task_struct __kptr *task; };struct {
@@ -217,6 +223,106 @@ int wq_sleepable_prog(void *ctx)return 0 ; } +__noinline int wq_global_acquire(void) +{ +struct task_struct *task, *acquired; +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + + task = val->task; +if (!task) +return 0 ; + + acquired =bpf_task_acquire (task); +if (acquired) +bpf_task_release (acquired); +return 0 ; +} + +static int wq_global_rcu_cb (void *map,int *key,void *value) +{ +wq_global_acquire (); +return 0 ; +} + +SEC("fentry/bpf_fentry_test1") +__failure __msg("R1 must be a rcu pointer") +int wq_global_rcu_prog(void *ctx) +{ +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_cb,0 ); +return 0 ; +} + +static int wq_global_rcu_lock_cb (void *map,int *key,void *value) +{ +bpf_rcu_read_lock (); +wq_global_acquire (); +bpf_rcu_read_unlock (); +return 0 ; +} + +SEC("fentry/bpf_fentry_test1") +__success +int wq_global_rcu_lock_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global subprog in non-sleepable and protected contexts. */ +wq_global_acquire (); + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_lock_cb,0 ); +return 0 ; +} + +__weak __noinline int wq_global_no_rcu(void) +{ +return 0 ; +} + +static int wq_global_no_rcu_cb (void *map,int *key,void *value) +{ +wq_global_no_rcu (); +return 0 ; +} + +SEC("fentry/bpf_fentry_test1") +__success __log_level(4) +__msg("subprog {{[0-9]+}} (wq_global_no_rcu) global insns_self 4 insns_total 4 stack 0") +int wq_global_no_rcu_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global in non-sleepable and unprotected contexts. */ +wq_global_no_rcu (); + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_no_rcu_cb,0 ); +return 0 ; +} +/* Task work tests */ struct task_work_elem {
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerQuoted 7 lines · Kumar Kartikeya Dwivedi
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.
By mistake, I tagged bpf tree for this one, sorry. I can respin again after a round of feedback or let someone else decide to apply it to either tree, if this version is landable.
Quoted 37 lines
Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]* Clarify that globals are checked only in contexts from which they are reached, with two passes only when both contexts occur. (Alexei)
* Implementin_sleepable_context()as !in_rcu_cs(). (Alexei, Eduard)
* Keep the original subprogram traversal order and repeat until all called contexts have been verified. (Eduard)
* Explain why instruction accounting needs the saved total and why the exception callback uses the main program's context. (Eduard)
* Rename thedo_check_common()parameter tois_sleepable. (BPF CI)
* Preserve the harmless global calls with__weak__noinlineand check their instruction totals across both contexts. Drop the redundant task-work global RCU test and retain the review acknowledgment. (Eduard)v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]* Preserve the harmless global subprog calls in the positive test with
barrier()so LLVM cannot eliminate the intended coverage.
* Return zero explicitly from workqueue callbacks after global subprog calls.
* Document cumulative instruction accounting across verification contexts.
* Fix BPF multi-line comment formatting.Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contextsinclude/linux/bpf.h | 5 +- kernel/bpf/verifier.c | 71 ++++++------ .../bpf/progs/verifier_async_cb_context.c | 106 ++++++++++++++++++ 3 files changed, 146 insertions(+), 36 deletions(-)base-commit:
29fcf9ce4bbdf4cc339def776860e1c0095bca98
Eduard Zingerman
maintainerQuoted 9 lines · Kumar Kartikeya Dwivedi
Quoted 7 lines · Kumar Kartikeya Dwivedi
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.
Quoted 3 lines
By mistake, I tagged bpf tree for this one, sorry. I can respin again after a round of feedback or let someone else decide to apply it to either tree, if this version is landable.
It applies and builds cleanly, I'd just land it in bpf.
This series was applied to bpf/bpf.git (master) by Eduard Zingerman <[email protected]>:
Hello:
This series was applied to bpf/bpf.git (master)
by Eduard Zingerman <[email protected]>:
Quoted 8 lines · On Mon, 14 Sep 2026 15:19:19 +0200 you wrote:
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.
[...]
Here is the summary with links:
- [bpf,v3,1/2] bpf: Verify global subprogs in each sleepability context
https://git.kernel.org/bpf/bpf/c/3506b9a2f108 - [bpf,v3,2/2] selftests/bpf: Test global subprog callback contexts https://git.kernel.org/bpf/bpf/c/4499609f99fd
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html