io_uring: restore RCU read section in io_req_local_work_add()
PATCH 1/1 io_uring: restore RCU read section in io_req_local_work_add()
The task-work refactor that moved io_req_local_work_add() out of io_uring.c into the new io_uring/tw.c dropped the whole-body guard(rcu)() that used to cover the function body.
For DEFER_TASKRUN rings the ring teardown still relies on that RCU read section pairing with its grace period:
/* pairs with RCU read section in io_req_local_work_add() */ if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) synchronize_rcu(); io_ring_ctx_free(ctx);
io_req_local_work_add() keeps dereferencing ctx after mpscq_push() has published the request to the work list (ctx->cq_wait_nr, and ctx->submitter_task in the final wake_up_state()), without holding a ctx reference across that window. The RCU read section was the only thing guaranteeing an in-flight adder had finished touching ctx before io_ring_ctx_free() ran; synchronize_rcu() only waits for readers that are actually inside an RCU read-side critical section. With the guard gone the grace period no longer pairs with anything on the add side, so ctx can be freed and reused while io_req_local_work_add() is still using it.
Restore the guard(rcu)() over the function body, matching the teardown pairing and the pre-refactor code (the guard was present when the function still lived in io_uring.c in v6.12).
--- io_uring/tw.c | 3 +++ 1 file changed, 3 insertions(+)
io_uring/tw.c +3 -0
@@ -153,6 +153,9 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags)struct io_ring_ctx *ctx = req->ctx;int nr_wait; +/* pairs with synchronize_rcu() in io_ring_exit_work() */ +guard (rcu)(); +/* * We don't know how many requests there are in the link and whether * they can even be queued lazily, fall back to non-lazy.
Signature · 1 line
2.43.0
Quoted 8 lines · Woraphat Khiaodaeng
The task-work refactor that moved
io_req_local_work_add()out ofio_uring.cinto the newio_uring/tw.cdropped the whole-body guard(rcu)() that used to cover the function body.For
DEFER_TASKRUNrings the ring teardown still relies on that RCU read section pairing with its grace period:[...]
Applied, thanks!
[1/1] io_uring: restore RCU read section in io_req_local_work_add()
commit: 648790e0952789527ec68548edbedbc0fcff43b5
Best regards,
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 9 lines · Jens Axboe
Quoted 7 lines · Woraphat Khiaodaeng
The task-work refactor that moved
io_req_local_work_add()out ofio_uring.cinto the newio_uring/tw.cdropped the whole-body guard(rcu)() that used to cover the function body.For
DEFER_TASKRUNrings the ring teardown still relies on that RCU read section pairing with its grace period:
Quoted 6 lines
Quoted 1 line
[...]
Applied, thanks!
[1/1]
io_uring: restore RCU read section inio_req_local_work_add()commit: 648790e0952789527ec68548edbedbc0fcff43b5
Side note: with your patch applied, we can also do:
as io_ctx_mark_taskrun() is no longer called outside the RCU read side lock anymore. So I did that on top.
Signature · 1 line
Jens Axboe