Fix for bpf_wq retry loop during free
Kumar Kartikeya Dwivedi
maintainercover Fix for bpf_wq retry loop during free
Small fix and improvement to ensure cancel_work() can handle the case where wq callback is running, and doesn't lead to call_rcu_tasks_trace() repeatedly after failing cancel_work, if wq callback is not pending.
Kumar Kartikeya Dwivedi (2):
bpf: Check for running wq callback when freeing bpf_async_cb
bpf: Reset prog callback in bpf_async_cancel_and_free()
kernel/bpf/helpers.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
base-commit: 6e951a9b1af5dc063dfe9e17ca433be099527c43
Signature · 1 line
2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH 1/2 bpf: Check for running wq callback when freeing bpf_async_cb
When freeing a bpf_async_cb in bpf_async_cb_rcu_tasks_trace_free(), in case the wq callback is not scheduled, doing cancel_work() currently returns false and leads to retry of RCU tasks trace grace period. If the callback is never scheduled, we keep retrying indefinitely and don't put the prog reference.
Since the only race we care about here is against a potentially running wq callback in the first grace period, it should finish by the second grace period, hence check work_busy() result to detect presence of running wq callback if it's not pending, otherwise free the object immediately without retrying.
Reasoning behind the check and its correctness with racing wq callback
invocation: cancel_work is supposed to be synchronized, hence calling it first and getting false would mean that work is definitely not pending, at this point, either the work is not scheduled at all or already running, or we race and it already finished by the time we checked for it using work_busy(). In case it is running, we synchronize using pool->lock to check the current work running there, if we match, it means we extend the wait by another grace period using retry = true, otherwise either the work already finished running or was never scheduled, so we can free the bpf_async_cb right away.
--- kernel/bpf/helpers.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/helpers.c +1 -1
@@ -1257,7 +1257,7 @@ static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu) retry = true;break ;case BPF_ASYNC_TYPE_WQ : -if (!cancel_work (&w->work)) +if (!cancel_work (&w->work) &&work_busy (&w->work)) retry = true;break ; }
Signature · 1 line
2.47.3
Kumar Kartikeya Dwivedi
maintainerPATCH 2/2 bpf: Reset prog callback in bpf_async_cancel_and_free()
In reply to Kumar Kartikeya DwivediReplace prog and callback in bpf_async_cb after removing visibility of bpf_async_cb in bpf_async_cancel_and_free() to increase the chances the scheduled async callbacks short-circuit execution and exit early, and not starting a RCU tasks trace section. This improves the overall time spent in running the wq selftest.
--- kernel/bpf/helpers.c | 1 + 1 file changed, 1 insertion(+)
kernel/bpf/helpers.c +1 -0
@@ -1664,6 +1664,7 @@ static void bpf_async_cancel_and_free(struct bpf_async_kern *async)if (!cb)return ; +bpf_async_update_prog_callback (cb,NULL ,NULL );/* * No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last * refcnt. Either synchronously or asynchronously in irq_work.
Signature · 1 line
2.47.3
Alexei Starovoitov
AppliedQuoted 4 lines · Kumar Kartikeya Dwivedi
Small fix and improvement to ensure
cancel_work()can handle the case where wq callback is running, and doesn't lead tocall_rcu_tasks_trace()repeatedly after failingcancel_work, if wq callback is not pending.
Applied. Thanks!