Fix global subprog verification context
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainercover Fix global subprog verification context
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can cause verifier's safety assumptions to be violated. Implement a fix by performing verification twice but only if different contexts are exercised, and add selftests to ensure coverage.
Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contexts
include/linux/bpf.h | 5 +- kernel/bpf/verifier.c | 76 ++++++---- .../bpf/progs/verifier_async_cb_context.c | 131 ++++++++++++++++++ 3 files changed, 180 insertions(+), 32 deletions(-)
base-commit: 3ccdb07813829ba9487273e75d1cb238cfa774c1
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH 1/2 bpf: Verify global subprogs in each sleepability context
Global subprograms are verified independently with a fresh verifier root. do_check_common() currently seeds that root's in_sleepable state from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point.
In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making in_rcu_cs() true and allowing loads of RCU-protected kptrs to produce trusted MEM_RCU pointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free.
The verifier's execution-context predicates are complementary. A state is sleepable only when in_sleepable is set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, while in_rcu_cs() treats a non-sleepable state as implicitly protected.
Use this relationship to represent a global subprogram caller with only the result of in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makes in_rcu_cs() true without copying caller-owned lock state.
Record whether each global subprogram is called with either in_sleepable value and verify it once for every observed value. Walk global subprograms in caller-before-callee order so the values propagate through global call chains, and repeat until every discovered context has been verified to cover asynchronous callback cycles.
This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid.
--- include/linux/bpf.h | 5 +-- kernel/bpf/verifier.c | 76 ++++++++++++++++++++++++++----------------- 2 files changed, 49 insertions(+), 32 deletions(-)
include/linux/bpf.h +3 -2
@@ -1650,8 +1650,9 @@ static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flagsstruct bpf_func_info_aux {u16 linkage;bool unreliable; -bool called :1 ; -bool verified :1 ; +/* Indexed by in_sleepable. */ +bool called[2 ]; +bool verified[2 ]; };enum bpf_jit_poke_reason {
kernel/bpf/verifier.c +46 -30
149 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v1 1/2] bpf: Verify global subprogs in each sleepability context
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] BPF subsystem multi-line comment style violation. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] BPF subsystem multi-line comment style violation.
--
commit aea430d1f0450a56e5505b581236a015c8f1396b
Author: Kumar Kartikeya Dwivedi <[email protected]>
bpf: Verify global subprogs in each sleepability context
This commit updates the BPF verifier to independently verify global subprograms across different sleepability contexts, rather than relying on the sleepable state of the main program. It modifies do_check_subprogs to track and iteratively verify contexts as they propagate through asynchronous callbacks.
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
Quoted 1 line
kernel/bpf/verifier.c +0
-0@@ -19749,45 +19756,54 @@ static int do_check_subprogs(struct bpf_verifier_env *env)
[ ... ]
kernel/bpf/verifier.c static int do_check_subprogs(struct bpf_verifier_env *env) · line 19809
Quoted 7 lines
kernel/bpf/verifier.c +2
-2-/* We can't loop forever as we verify at least one global subprog on -* each pass. +/* We can't loop forever as each pass verifies at least one new context, +* and there are only two contexts per global subprog. */ if (new_cnt) goto again;
[Severity: Low]
Does this new comment block violate the preferred multi-line comment style for the BPF subsystem? The official style mandates that multi-line comments should place the opening /* on its own line before the text begins.
Could it be reformatted to look like this instead?
/*
* We can't loop forever as each pass verifies at least one new context,
* and there are only two contexts per global subprog.
*/
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH bpf-next v1 1/2] bpf: Verify global subprogs in each sleepability context
In reply to Kumar Kartikeya DwivediThis isn't a bug, but would it be worth naming this parameter is_sleepable instead? There's a file-scope helper in_sleepable() at kernel/bpf/verifier.c and the parameter now shadows it within this…
Quoted 14 lines
include/linux/bpf.h +3
-2@@ -1650,8 +1650,9 @@ static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flagsstruct bpf_func_info_aux {u16 linkage;bool unreliable; -bool called :1 ; -bool verified :1 ; +/* Indexed by in_sleepable. */ +bool called[2 ]; +bool verified[2 ]; };
[ ... ]
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c static void free_states(struct bpf_verifier_env *env) · line 19569
Quoted 7 lines
kernel/bpf/verifier.c +1
-1@@ -19560,13 +19565,14 @@ static void free_states(struct bpf_verifier_env *env) } } -static int do_check_common (struct bpf_verifier_env *env,int subprog) +static int do_check_common (struct bpf_verifier_env *env,int subprog,bool in_sleepable) {
This isn't a bug, but would it be worth naming this parameter is_sleepable instead? There's a file-scope helper in_sleepable() at kernel/bpf/verifier.c and the parameter now shadows it within this function, so any future code added here cannot call the helper without renaming. The sibling function that seeds the same field for async callbacks, push_async_cb(), uses the name is_sleepable for its equivalent parameter.
kernel/bpf/verifier.c static void free_states(struct bpf_verifier_env *env) · line 19576
Quoted 7 lines
kernel/bpf/verifier.c +1
-0bool pop_log = !(env->log.level &BPF_LOG_LEVEL2 );struct bpf_subprog_info *sub =subprog_info (env, subprog);struct bpf_prog_aux *aux = env->prog->aux;struct bpf_verifier_state *state;struct bpf_reg_state *regs; +u32 old_insns_total = sub->insns_total;u32 insn_processed = env->insn_processed;
This isn't a bug, but could the commit message add a sentence about the insns_total accounting change here? A global subprog verified in both contexts now reports the sum of both passes in subprog_info[].insns_total, which affects BPF_LOG_STATS output and veristat per-subprog instruction counts. The code is correct but the behavioral change isn't mentioned in the changelog.
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/33943332260
Kumar Kartikeya Dwivedi
maintainerPATCH 2/2 selftests/bpf: Test global subprog callback contexts
In reply to Kumar Kartikeya DwivediExercise global subprogram verification from workqueue and task-work callbacks. Both callback types can run in a sleepable context even when the containing program is not sleepable, so an unprotected callback must not let the global subprogram use implicit RCU protection inherited from the program.
Add negative cases which load an RCU-protected task kptr in a global subprogram reached from each callback type. The tests fail on an unfixed kernel because the programs are incorrectly accepted.
Also cover a workqueue callback protected by an explicit RCU read-side critical section. Finally, call the same harmless global subprogram directly from the main program and from an unprotected callback. This requires both non-sleepable and sleepable verification roots and proves that global calls from callbacks are not rejected wholesale.
--- .../bpf/progs/verifier_async_cb_context.c | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+)
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +131 -0
@@ -9,6 +9,11 @@ char _license[] SEC("license") = "GPL"; +struct task_struct *bpf_task_acquire (struct task_struct *p) __ksym; +void bpf_task_release(struct task_struct *p) __ksym; +void bpf_rcu_read_lock(void) __ksym; +void bpf_rcu_read_unlock(void) __ksym; +/* Timer tests */ struct timer_elem {
@@ -66,6 +71,7 @@ int timer_sleepable_prog(void *ctx)struct wq_elem {struct bpf_wq w; + struct task_struct __kptr *task; };struct {
@@ -119,6 +125,105 @@ int wq_sleepable_prog(void *ctx)return 0 ; } +__noinline int wq_global_acquire(void) +{ +struct task_struct *task, *acquired; +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + + task = val->task; +if (!task) +return 0 ; + + acquired =bpf_task_acquire (task); +if (acquired) +bpf_task_release (acquired); +return 0 ; +} + +static int wq_global_rcu_cb (void *map,int *key,void *value) +{ +return wq_global_acquire (); +} + +SEC("fentry/bpf_fentry_test1") +__failure __msg("R1 must be a rcu pointer") +int wq_global_rcu_prog(void *ctx) +{ +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_cb,0 ); +return 0 ; +} + +static int wq_global_rcu_lock_cb (void *map,int *key,void *value) +{ +int ret; + +bpf_rcu_read_lock (); + ret =wq_global_acquire (); +bpf_rcu_read_unlock (); +return ret; +} + +SEC("fentry/bpf_fentry_test1") +__success +int wq_global_rcu_lock_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global subprog in non-sleepable and protected contexts. */ +wq_global_acquire (); + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_lock_cb,0 ); +return 0 ; +} + +__noinline int wq_global_no_rcu(void) +{ +return 0 ; +} + +static int wq_global_no_rcu_cb (void *map,int *key,void *value) +{ +return wq_global_no_rcu (); +} + +SEC("fentry/bpf_fentry_test1") +__success +int wq_global_no_rcu_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global in non-sleepable and unprotected contexts. */ +wq_global_no_rcu (); + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_no_rcu_cb,0 ); +return 0 ; +} +/* Task work tests */ struct task_work_elem {
@@ -179,3 +284,29 @@ int task_work_sleepable_prog(void *ctx)bpf_task_work_schedule_resume (task, &val->tw, &task_work_map, task_work_cb);return 0 ; } + +static int task_work_global_rcu_cb (struct bpf_map *map,void *key,void *value) +{ +return wq_global_acquire (); +} + +SEC("fentry/bpf_fentry_test1") +__failure __msg("R1 must be a rcu pointer") +int task_work_global_rcu_prog(void *ctx) +{ +struct task_work_elem *val; +struct task_struct *task; +int key =0 ; + + val =bpf_map_lookup_elem (&task_work_map, &key); +if (!val) +return 0 ; + + task =bpf_get_current_task_btf (); +if (!task) +return 0 ; + +bpf_task_work_schedule_resume (task, &val->tw, &task_work_map, + task_work_global_rcu_cb); +return 0 ; +}
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v1 2/2] selftests/bpf: Test global subprog callback contexts
Does wq_global_rcu_lock_cb() verify with this return value?
Quoted 16 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +5
-0@@ -9,6 +9,11 @@ char _license[] SEC("license") = "GPL"; +struct task_struct *bpf_task_acquire (struct task_struct *p) __ksym; +void bpf_task_release(struct task_struct *p) __ksym; +void bpf_rcu_read_lock(void) __ksym; +void bpf_rcu_read_unlock(void) __ksym; +/* Timer tests */ struct timer_elem {
[ ... ]
Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +1
-0@@ -66,6 +71,7 @@ int timer_sleepable_prog(void *ctx)struct wq_elem {struct bpf_wq w; + struct task_struct __kptr *task; };struct {
[ ... ]
Quoted 44 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +40
-0@@ -119,6 +125,105 @@ int wq_sleepable_prog(void *ctx)return 0 ; } +__noinline int wq_global_acquire(void) +{ +struct task_struct *task, *acquired; +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + + task = val->task; +if (!task) +return 0 ; + + acquired =bpf_task_acquire (task); +if (acquired) +bpf_task_release (acquired); +return 0 ; +} + +static int wq_global_rcu_cb (void *map,int *key,void *value) +{ +return wq_global_acquire (); +} + +SEC("fentry/bpf_fentry_test1") +__failure __msg("R1 must be a rcu pointer") +int wq_global_rcu_prog(void *ctx) +{ +struct wq_elem *val; +int key =0 ; + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_cb,0 ); +return 0 ; +}
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c int wq_sleepable_prog(void *ctx) · line 177
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +10
-0+ +static int wq_global_rcu_lock_cb (void *map,int *key,void *value) +{ +int ret; + +bpf_rcu_read_lock (); + ret =wq_global_acquire (); +bpf_rcu_read_unlock (); +return ret; +}
Does wq_global_rcu_lock_cb() verify with this return value?
Workqueue callbacks must return exactly 0. When bpf_wq_set_callback is processed, the callback's return range gets constrained to [0, 0] in set_timer_callback_state() via:
kernel/bpf/verifier.c:set_timer_callback_state() {
...
callee->callback_ret_range = retval_range(0, 0);
callee->in_async_callback_fn = true;
...
}Later when the callback returns, check_return_code() enforces this:
kernel/bpf/verifier.c:check_return_code() {
...
if (frame->in_async_callback_fn) {
range = frame->callback_ret_range;
goto enforce_retval;
}
...
}A call to a global subprogram leaves R0 as an unbounded scalar. In check_func_call(), the bpf_subprog_is_global() path does:
kernel/bpf/verifier.c:check_func_call() {
...
if (bpf_subprog_is_global(env, subprog)) {
...
mark_reg_unknown(env, caller->regs, ret_regs[0]);
...
}
...
}So returning ret here only verifies because the compiler constant-folds wq_global_acquire()'s return to 0 (every path returns 0). The same issue appears in wq_global_rcu_cb() above, which also returns the result of wq_global_acquire().
If wq_global_acquire() ever gets a non-zero return path, or if the optimizer doesn't fold it, wq_global_rcu_lock_prog's __success annotation fails with "At async callback return the register R0 has ... should have been in [0, 0]", and wq_global_rcu_prog's __msg("R1 must be a rcu pointer") never appears because bpf_check() runs do_check_main() before do_check_subprogs() and the callback's return-code error aborts verification early.
The existing callbacks in progs/wq.c all return literal constants (wq.c:wq_cb_sleepable() and wq.c:wq_cb()). Would wq_global_acquire(); return 0; in both wq_global_rcu_cb and wq_global_rcu_lock_cb remove the dependency on optimizer behavior?
Quoted 29 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +29
-0+ +SEC("fentry/bpf_fentry_test1") +__success +int wq_global_rcu_lock_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global subprog in non-sleepable and protected contexts. */ +wq_global_acquire (); + + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_rcu_lock_cb,0 ); +return 0 ; +} + +__noinline int wq_global_no_rcu(void) +{ +return 0 ; +} + +static int wq_global_no_rcu_cb (void *map,int *key,void *value) +{ +return wq_global_no_rcu (); +}
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c int wq_sleepable_prog(void *ctx) · line 216
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +10
-0+ +SEC("fentry/bpf_fentry_test1") +__success +int wq_global_no_rcu_prog (void *ctx) +{ +struct wq_elem *val; +int key =0 ; + +/* Verify the same global in non-sleepable and unprotected contexts. */ +wq_global_no_rcu ();
Can the calls to wq_global_no_rcu() get optimized away?
The function has an empty body (return 0), takes no arguments, is not __weak, and its address is never taken. At -O2 (which selftests always use via tools/testing/selftests/bpf/Makefile), LLVM can infer memory(none)/willreturn/nounwind and constant-propagate the return value. Both call sites then become dead: the direct call here (result discarded) and the call in wq_global_no_rcu_cb() (whose result IPSCCP replaces with 0). The __noinline attribute only blocks inlining, not interprocedural attribute inference or DCE.
If the calls are elided, there is no relocation to the subprog, libbpf never appends it to the program, check_func_call() never runs the bpf_subprog_is_global() path:
kernel/bpf/verifier.c:check_func_call() {
...
if (bpf_subprog_is_global(env, subprog)) {
...
}
...
}and do_check_subprogs() never calls do_check_common(env, subprog, 0) or do_check_common(env, subprog, 1) for it:
kernel/bpf/verifier.c:do_check_subprogs() {
for (i = 1; i < env->subprog_cnt; i++) {
...
if (sub_aux->called[0])
ret = do_check_common(env, i, 0);
if (sub_aux->called[1])
ret = do_check_common(env, i, 1);
...
}
}The test still passes __success because wq_global_no_rcu_prog loads successfully, but it covers nothing. This is the only test that exercises successful sleepable-root verification of a global subprog and the caller-before-callee/repeat loop that patch 1/2 adds. The changelog claims "call the same harmless global subprogram directly from the main program and from an unprotected callback. This requires both non-sleepable and sleepable verification roots and proves that global calls from callbacks are not rejected wholesale", but this coverage would silently not happen.
Searching progs/ for a zero-argument global subprog returning a constant matches only the two functions this patch adds. Every other plain-__noinline global subprog either takes pointer/scalar arguments or performs memory accesses (so memory(none) cannot be inferred), and files that need an opaque global subprog use __weak __noinline for this reason:
progs/verifier_global_subprogs.c:__weak __noinline int noop_global_func() progs/summarization.c:__weak __noinline int noop_global_func()
wq_global_acquire() survives only because it calls bpf_map_lookup_elem() through the constant function pointer from bpf_helper_defs.h, whose memory effects LLVM cannot infer.
Would declaring wq_global_no_rcu() as __weak __noinline, or giving it an observable side effect, prevent the calls from being removed? This could be confirmed by checking llvm-objdump -d on the compiled .bpf.o to see if any call -1 relocation to wq_global_no_rcu remains in wq_global_no_rcu_cb.
Quoted 9 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +9
-0+ + val =bpf_map_lookup_elem (&wq_map, &key); +if (!val) +return 0 ; + +bpf_wq_init (&val->w, &wq_map,0 ); +bpf_wq_set_callback (&val->w, wq_global_no_rcu_cb,0 ); +return 0 ; +}
[ ... ]
Quoted 20 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +16
-0@@ -179,3 +284,29 @@ int task_work_sleepable_prog(void *ctx)bpf_task_work_schedule_resume (task, &val->tw, &task_work_map, task_work_cb);return 0 ; } + +static int task_work_global_rcu_cb (struct bpf_map *map,void *key,void *value) +{ +return wq_global_acquire (); +} + +SEC("fentry/bpf_fentry_test1") +__failure __msg("R1 must be a rcu pointer") +int task_work_global_rcu_prog(void *ctx) +{ +struct task_work_elem *val; +struct task_struct *task; +int key =0 ; + + val =bpf_map_lookup_elem (&task_work_map, &key); +if (!val)
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c int task_work_sleepable_prog(void *ctx) · line 312
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +10
-0+return 0 ; + + task =bpf_get_current_task_btf (); +if (!task) +return 0 ; + +bpf_task_work_schedule_resume (task, &val->tw, &task_work_map, + task_work_global_rcu_cb); +return 0 ; +}
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/33943332260
Kumar Kartikeya Dwivedi
maintainerQuoted 11 lines · Kumar Kartikeya Dwivedi
Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can cause verifier's safety assumptions to be violated. Implement a fix by performing verification twice but only if different contexts are exercised, and add selftests to ensure coverage.
Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contexts
CI failure in GCC-BPF run looks unrelated.