Skip to contentSkip to conversation

io_uring: restore RCU read section in io_req_local_work_add()

PATCH Applied Woraphat Khiaodaeng 2 participants 3 messages Jul 9 io_uring

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring: restore RCU read section in io_req_local_work_add()

The task-work refactor that moved io_req_local_work_add() out of io_uring.c into the new io_uring/tw.c dropped the whole-body guard(rcu)() that used to cover the function body.

For DEFER_TASKRUN rings the ring teardown still relies on that RCU read section pairing with its grace period:

	/* pairs with RCU read section in io_req_local_work_add() */
	if (ctx->flags & IORING_SETUP_DEFER_TASKRUN)
		synchronize_rcu();
	io_ring_ctx_free(ctx);

io_req_local_work_add() keeps dereferencing ctx after mpscq_push() has published the request to the work list (ctx->cq_wait_nr, and ctx->submitter_task in the final wake_up_state()), without holding a ctx reference across that window. The RCU read section was the only thing guaranteeing an in-flight adder had finished touching ctx before io_ring_ctx_free() ran; synchronize_rcu() only waits for readers that are actually inside an RCU read-side critical section. With the guard gone the grace period no longer pairs with anything on the add side, so ctx can be freed and reused while io_req_local_work_add() is still using it.

Restore the guard(rcu)() over the function body, matching the teardown pairing and the pre-refactor code (the guard was present when the function still lived in io_uring.c in v6.12).

Fixes d46ab2c98aba ("io_uring: switch local task_work to a mpscq")
---
 io_uring/tw.c | 3 +++
 1 file changed, 3 insertions(+)
io_uring/tw.c +3 -0
index a4c872870..4f7a0d107 100644
@@ -153,6 +153,9 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags)
 	struct io_ring_ctx *ctx = req->ctx;
 	int nr_wait;
 
+	/* pairs with synchronize_rcu() in io_ring_exit_work() */
+	guard(rcu)();
+
 	/*
 	 * We don't know how many requests there are in the link and whether
 	 * they can even be queued lazily, fall back to non-lazy.
Signature · 1 line
2.43.0
maintainer Applied
#1 lore raw
Quoted 8 lines · Woraphat Khiaodaeng

The task-work refactor that moved io_req_local_work_add() out of io_uring.c into the new io_uring/tw.c dropped the whole-body guard(rcu)() that used to cover the function body.

For DEFER_TASKRUN rings the ring teardown still relies on that RCU read section pairing with its grace period:

[...]

Applied, thanks!

[1/1] io_uring: restore RCU read section in io_req_local_work_add()

      commit: 648790e0952789527ec68548edbedbc0fcff43b5

Best regards,

Signature · 1 line
Jens Axboe
maintainer
#2 lore raw
Quoted 9 lines · Jens Axboe
Quoted 7 lines · Woraphat Khiaodaeng

The task-work refactor that moved io_req_local_work_add() out of io_uring.c into the new io_uring/tw.c dropped the whole-body guard(rcu)() that used to cover the function body.

For DEFER_TASKRUN rings the ring teardown still relies on that RCU read section pairing with its grace period:

Quoted 6 lines
Quoted 1 line

[...]

Applied, thanks!

[1/1] io_uring: restore RCU read section in io_req_local_work_add()

      commit: 648790e0952789527ec68548edbedbc0fcff43b5

Side note: with your patch applied, we can also do:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/commit/?h=io_uring-7.2&id=f3176c8ac4217c88fe1147ab084c47092921ffc4

as io_ctx_mark_taskrun() is no longer called outside the RCU read side lock anymore. So I did that on top.

Signature · 1 line
Jens Axboe