Skip to contentSkip to conversation

Fix for bpf_wq retry loop during free

PATCH Applied Kumar Kartikeya Dwivedi 2 participants 4 messages Feb 5 BPF

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

cover Fix for bpf_wq retry loop during free

Small fix and improvement to ensure cancel_work() can handle the case where wq callback is running, and doesn't lead to call_rcu_tasks_trace() repeatedly after failing cancel_work, if wq callback is not pending.

Kumar Kartikeya Dwivedi (2):
bpf: Check for running wq callback when freeing bpf_async_cb
bpf: Reset prog callback in bpf_async_cancel_and_free()

 kernel/bpf/helpers.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

base-commit: 6e951a9b1af5dc063dfe9e17ca433be099527c43

Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH 1/2 bpf: Check for running wq callback when freeing bpf_async_cb

When freeing a bpf_async_cb in bpf_async_cb_rcu_tasks_trace_free(), in case the wq callback is not scheduled, doing cancel_work() currently returns false and leads to retry of RCU tasks trace grace period. If the callback is never scheduled, we keep retrying indefinitely and don't put the prog reference.

Since the only race we care about here is against a potentially running wq callback in the first grace period, it should finish by the second grace period, hence check work_busy() result to detect presence of running wq callback if it's not pending, otherwise free the object immediately without retrying.

Reasoning behind the check and its correctness with racing wq callback
invocation: cancel_work is supposed to be synchronized, hence calling it first and getting false would mean that work is definitely not pending, at this point, either the work is not scheduled at all or already running, or we race and it already finished by the time we checked for it using work_busy(). In case it is running, we synchronize using pool->lock to check the current work running there, if we match, it means we extend the wait by another grace period using retry = true, otherwise either the work already finished running or was never scheduled, so we can free the bpf_async_cb right away.

Fixes 1bfbc267ec91 ("bpf: Enable bpf_timer and bpf_wq in any context")
---
 kernel/bpf/helpers.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/helpers.c +1 -1
index 01052f8664eb..b7aec34540c2 100644
@@ -1257,7 +1257,7 @@ static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
 			retry = true;
 		break;
 	case BPF_ASYNC_TYPE_WQ:
-		if (!cancel_work(&w->work))
+		if (!cancel_work(&w->work) && work_busy(&w->work))
 			retry = true;
 		break;
 	}
Signature · 1 line
2.47.3
author maintainer
#2 lore raw

PATCH 2/2 bpf: Reset prog callback in bpf_async_cancel_and_free()

In reply to Kumar Kartikeya Dwivedi

Replace prog and callback in bpf_async_cb after removing visibility of bpf_async_cb in bpf_async_cancel_and_free() to increase the chances the scheduled async callbacks short-circuit execution and exit early, and not starting a RCU tasks trace section. This improves the overall time spent in running the wq selftest.

---
 kernel/bpf/helpers.c | 1 +
 1 file changed, 1 insertion(+)
kernel/bpf/helpers.c +1 -0
index b7aec34540c2..a4f039cee88b 100644
@@ -1664,6 +1664,7 @@ static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
 	if (!cb)
 		return;
 
+	bpf_async_update_prog_callback(cb, NULL, NULL);
 	/*
 	 * No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
 	 * refcnt. Either synchronously or asynchronously in irq_work.
Signature · 1 line
2.47.3
#3 lore raw
In reply to Kumar Kartikeya Dwivedi
Quoted 4 lines · Kumar Kartikeya Dwivedi

Small fix and improvement to ensure cancel_work() can handle the case where wq callback is running, and doesn't lead to call_rcu_tasks_trace() repeatedly after failing cancel_work, if wq callback is not pending.

Applied. Thanks!