Skip to contentSkip to conversation

Fix global subprog verification context

PATCH Applied Kumar Kartikeya Dwivedi 3 participants 7 messages Sep 14 – Sep 16

Subsystems (4)
lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v3 cover Fix global subprog verification context

Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.

Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]

* Clarify that globals are checked only in contexts from which they are reached, with two passes only when both contexts occur. (Alexei)
* Implement in_sleepable_context() as !in_rcu_cs(). (Alexei, Eduard)
* Keep the original subprogram traversal order and repeat until all called contexts have been verified. (Eduard)
* Explain why instruction accounting needs the saved total and why the exception callback uses the main program's context. (Eduard)
* Rename the do_check_common() parameter to is_sleepable. (BPF CI)
* Preserve the harmless global calls with __weak __noinline and check their instruction totals across both contexts. Drop the redundant task-work global RCU test and retain the review acknowledgment. (Eduard)

v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Preserve the harmless global subprog calls in the positive test with barrier() so LLVM cannot eliminate the intended coverage.
* Return zero explicitly from workqueue callbacks after global subprog calls.
* Document cumulative instruction accounting across verification contexts.
* Fix BPF multi-line comment formatting.

Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contexts

 include/linux/bpf.h                           |   5 +-
 kernel/bpf/verifier.c                         |  71 ++++++------
 .../bpf/progs/verifier_async_cb_context.c     | 106 ++++++++++++++++++
 3 files changed, 146 insertions(+), 36 deletions(-)

base-commit: 29fcf9ce4bbdf4cc339def776860e1c0095bca98

Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH v3 1/2 bpf: Verify global subprogs in each sleepability context

Global subprograms are verified independently with a fresh verifier root. do_check_common() currently seeds that root's in_sleepable state from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point.

In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making in_rcu_cs() true and allowing loads of RCU-protected kptrs to produce trusted MEM_RCU pointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free.

The verifier's execution-context predicates are complementary. A state is sleepable only when in_sleepable is set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, while in_rcu_cs() treats a non-sleepable state as implicitly protected.

Use this relationship to represent a global subprogram caller with only the result of in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makes in_rcu_cs() true without copying caller-owned lock state.

Track only the contexts in which each global subprogram is actually reached. Verify it once if all reachable calls use the same context, and twice only if both sleepable and non-sleepable calls reach it. Calls found while verifying globals or asynchronous callbacks mark further contexts for checking. Repeat the existing subprogram walk until all called contexts have been verified; unreachable global calls remain unchecked.

Accumulate instruction counts over those verification passes. Preserve the total recorded before each pass, since path accounting has already added this pass's synchronous instructions and its root total must also include asynchronous subprograms.

This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid.

Fixes 81f1d7a583fa ("bpf: wq: add bpf_wq_set_callback_impl")
Fixes 38aa7003e369 ("bpf: task work scheduling kfuncs")
---
 include/linux/bpf.h   |  5 +--
 kernel/bpf/verifier.c | 71 ++++++++++++++++++++++---------------------
 2 files changed, 40 insertions(+), 36 deletions(-)
include/linux/bpf.h +3 -2
index 2a5fa346aada..7217e8ddf64b 100644
@@ -1682,8 +1682,9 @@ static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags
 struct bpf_func_info_aux {
 	u16 linkage;
 	bool unreliable;
-	bool called : 1;
-	bool verified : 1;
+	/* Indexed by in_sleepable. */
+	bool called[2];
+	bool verified[2];
 };
 
 enum bpf_jit_poke_reason {
kernel/bpf/verifier.c +37 -34

143 lines not on this page · raw message

Signature · 1 line
2.53.0
maintainer Acked-by on v3 1/2
#2 lore raw

Re: [PATCH bpf v3 1/2] bpf: Verify global subprogs in each sleepability context

Quoted 42 lines · Kumar Kartikeya Dwivedi

Global subprograms are verified independently with a fresh verifier root. do_check_common() currently seeds that root's in_sleepable state from the program, even though a global subprogram can also run from callbacks whose execution context differs from the program's main entry point.

In particular, workqueue and task-work callbacks are sleepable even when the containing program is not. A global subprogram of that program is therefore verified as non-sleepable, making in_rcu_cs() true and allowing loads of RCU-protected kptrs to produce trusted MEM_RCU pointers. The same subprogram can then be called from a sleepable callback without a classic RCU reader. It can retain such a pointer while the object is freed and use it after free.

The verifier's execution-context predicates are complementary. A state is sleepable only when in_sleepable is set and no RCU, preemption, IRQ, or lock region is active. Each condition which prevents sleeping also provides RCU protection, while in_rcu_cs() treats a non-sleepable state as implicitly protected.

Use this relationship to represent a global subprogram caller with only the result of in_sleepable_context(). A protected sleepable caller is normalized to in_sleepable=false at the independent verification root. This both prevents sleepable operations and makes in_rcu_cs() true without copying caller-owned lock state.

Track only the contexts in which each global subprogram is actually reached. Verify it once if all reachable calls use the same context, and twice only if both sleepable and non-sleepable calls reach it. Calls found while verifying globals or asynchronous callbacks mark further contexts for checking. Repeat the existing subprogram walk until all called contexts have been verified; unreachable global calls remain unchecked.

Accumulate instruction counts over those verification passes. Preserve the total recorded before each pass, since path accounting has already added this pass's synchronous instructions and its root total must also include asynchronous subprograms.

This makes an unprotected callback verify the global subprogram as sleepable, turning its RCU-protected kptr load into an untrusted pointer. Protected callers and global subprograms which do not depend on implicit RCU protection remain valid.

Quoted 6 lines
Fixes 81f1d7a583fa ("bpf: wq: add bpf_wq_set_callback_impl")
Fixes 38aa7003e369 ("bpf: task work scheduling kfuncs")

---

...

author maintainer
#3 lore raw

PATCH v3 2/2 selftests/bpf: Test global subprog callback contexts

In reply to Kumar Kartikeya Dwivedi

Exercise global subprogram verification from workqueue callbacks, which can run in a sleepable context even when the containing program is not sleepable. An unprotected callback must not let the global subprogram use implicit RCU protection inherited from the program.

Add a negative case which loads an RCU-protected task kptr in a global subprogram reached from a workqueue callback. It fails on an unfixed kernel because the program is incorrectly accepted. Also cover a workqueue callback protected by an explicit RCU read-side critical section, where the same global subprogram remains valid.

Call the same harmless global subprogram directly from the main program and from an unprotected callback. Mark it __weak __noinline so both calls survive optimization, and check that its instruction statistics account for both verification contexts. This also verifies that global calls from callbacks are not rejected wholesale. Workqueue callbacks return zero explicitly after the global call, as required by their contract.

---
 .../bpf/progs/verifier_async_cb_context.c     | 106 ++++++++++++++++++
 1 file changed, 106 insertions(+)
tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +106 -0
index e0926767bbd3..1b653bfb63eb 100644
@@ -9,6 +9,11 @@
 
 char _license[] SEC("license") = "GPL";
 
+struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+void bpf_rcu_read_lock(void) __ksym;
+void bpf_rcu_read_unlock(void) __ksym;
+
 /* Timer tests */
 
 struct timer_elem {
@@ -164,6 +169,7 @@ int syscall_btf_find_prog(void *ctx)
 
 struct wq_elem {
 	struct bpf_wq w;
+	struct task_struct __kptr *task;
 };
 
 struct {
@@ -217,6 +223,106 @@ int wq_sleepable_prog(void *ctx)
 	return 0;
 }
 
+__noinline int wq_global_acquire(void)
+{
+	struct task_struct *task, *acquired;
+	struct wq_elem *val;
+	int key = 0;
+
+	val = bpf_map_lookup_elem(&wq_map, &key);
+	if (!val)
+		return 0;
+
+	task = val->task;
+	if (!task)
+		return 0;
+
+	acquired = bpf_task_acquire(task);
+	if (acquired)
+		bpf_task_release(acquired);
+	return 0;
+}
+
+static int wq_global_rcu_cb(void *map, int *key, void *value)
+{
+	wq_global_acquire();
+	return 0;
+}
+
+SEC("fentry/bpf_fentry_test1")
+__failure __msg("R1 must be a rcu pointer")
+int wq_global_rcu_prog(void *ctx)
+{
+	struct wq_elem *val;
+	int key = 0;
+
+	val = bpf_map_lookup_elem(&wq_map, &key);
+	if (!val)
+		return 0;
+
+	bpf_wq_init(&val->w, &wq_map, 0);
+	bpf_wq_set_callback(&val->w, wq_global_rcu_cb, 0);
+	return 0;
+}
+
+static int wq_global_rcu_lock_cb(void *map, int *key, void *value)
+{
+	bpf_rcu_read_lock();
+	wq_global_acquire();
+	bpf_rcu_read_unlock();
+	return 0;
+}
+
+SEC("fentry/bpf_fentry_test1")
+__success
+int wq_global_rcu_lock_prog(void *ctx)
+{
+	struct wq_elem *val;
+	int key = 0;
+
+	/* Verify the same global subprog in non-sleepable and protected contexts. */
+	wq_global_acquire();
+
+	val = bpf_map_lookup_elem(&wq_map, &key);
+	if (!val)
+		return 0;
+
+	bpf_wq_init(&val->w, &wq_map, 0);
+	bpf_wq_set_callback(&val->w, wq_global_rcu_lock_cb, 0);
+	return 0;
+}
+
+__weak __noinline int wq_global_no_rcu(void)
+{
+	return 0;
+}
+
+static int wq_global_no_rcu_cb(void *map, int *key, void *value)
+{
+	wq_global_no_rcu();
+	return 0;
+}
+
+SEC("fentry/bpf_fentry_test1")
+__success __log_level(4)
+__msg("subprog {{[0-9]+}} (wq_global_no_rcu) global insns_self 4 insns_total 4 stack 0")
+int wq_global_no_rcu_prog(void *ctx)
+{
+	struct wq_elem *val;
+	int key = 0;
+
+	/* Verify the same global in non-sleepable and unprotected contexts. */
+	wq_global_no_rcu();
+
+	val = bpf_map_lookup_elem(&wq_map, &key);
+	if (!val)
+		return 0;
+
+	bpf_wq_init(&val->w, &wq_map, 0);
+	bpf_wq_set_callback(&val->w, wq_global_no_rcu_cb, 0);
+	return 0;
+}
+
 /* Task work tests */
 
 struct task_work_elem {
Signature · 1 line
2.53.0
author maintainer
#4 lore raw
In reply to Kumar Kartikeya Dwivedi
Quoted 7 lines · Kumar Kartikeya Dwivedi

Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.

By mistake, I tagged bpf tree for this one, sorry. I can respin again after a round of feedback or let someone else decide to apply it to either tree, if this version is landable.

Quoted 37 lines

Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]

* Clarify that globals are checked only in contexts from which they are reached, with two passes only when both contexts occur. (Alexei)
* Implement in_sleepable_context() as !in_rcu_cs(). (Alexei, Eduard)
* Keep the original subprogram traversal order and repeat until all called contexts have been verified. (Eduard)
* Explain why instruction accounting needs the saved total and why the exception callback uses the main program's context. (Eduard)
* Rename the do_check_common() parameter to is_sleepable. (BPF CI)
* Preserve the harmless global calls with __weak __noinline and check their instruction totals across both contexts. Drop the redundant task-work global RCU test and retain the review acknowledgment. (Eduard)

v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Preserve the harmless global subprog calls in the positive test with barrier() so LLVM cannot eliminate the intended coverage.
* Return zero explicitly from workqueue callbacks after global subprog calls.
* Document cumulative instruction accounting across verification contexts.
* Fix BPF multi-line comment formatting.

Kumar Kartikeya Dwivedi (2):
bpf: Verify global subprogs in each sleepability context selftests/bpf: Test global subprog callback contexts

 include/linux/bpf.h                           |   5 +-
 kernel/bpf/verifier.c                         |  71 ++++++------
 .../bpf/progs/verifier_async_cb_context.c     | 106 ++++++++++++++++++
 3 files changed, 146 insertions(+), 36 deletions(-)

base-commit: 29fcf9ce4bbdf4cc339def776860e1c0095bca98

maintainer
#5 lore raw
Quoted 9 lines · Kumar Kartikeya Dwivedi
Quoted 7 lines · Kumar Kartikeya Dwivedi

Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.

Quoted 3 lines

By mistake, I tagged bpf tree for this one, sorry. I can respin again after a round of feedback or let someone else decide to apply it to either tree, if this version is landable.

It applies and builds cleanly, I'd just land it in bpf.

#6 lore raw
In reply to Kumar Kartikeya Dwivedi
This series was applied to bpf/bpf.git (master) by Eduard Zingerman <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Eduard Zingerman <[email protected]>:

Quoted 8 lines · On Mon, 14 Sep 2026 15:19:19 +0200 you wrote:

Global subprogs can be invoked in both sleepable and non-sleepable contexts, but are verified based on context assumptions for the program type as a whole and not the precise context in which they are invoked. Nicholas identified that this can violate verifier safety assumptions. Verify global subprogs once for each observed sleepability context and add workqueue callback regression tests.

[...]

Here is the summary with links:
- [bpf,v3,1/2] bpf: Verify global subprogs in each sleepability context

    https://git.kernel.org/bpf/bpf/c/3506b9a2f108
  - [bpf,v3,2/2] selftests/bpf: Test global subprog callback contexts
    https://git.kernel.org/bpf/bpf/c/4499609f99fd

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html