Linked request fix
Jens Axboe
maintainercover Linked request fix
Hi,
A small series closing some gaps on linked requests, where iterating a chain must hold either ->uring_lock OR ->timeout_lock, and modifying any existing change must hold both. Most cases already do, just a few gaps that should be buttoned up.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/3 io_uring: hold uring_lock when walking link chain in io_wq_free_work()
io_wq_free_work() calls io_req_find_next() from io-wq worker context, which reads and clears req->link without holding any lock. This can potentially race with other paths that mutate the same chain under ctx->uring_lock.
Take ctx->uring_lock around the io_req_find_next() call. Only requests with IO_REQ_LINK_FLAGS reach this path, which is not the hot path.
--- io_uring/io_uring.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
io_uring/io_uring.c +6 -1
@@ -1452,8 +1452,13 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work)struct io_kiocb *nxt =NULL ;if (req_ref_put_and_test_atomic (req)) { -if (req->flags &IO_REQ_LINK_FLAGS ) +if (req->flags &IO_REQ_LINK_FLAGS ) { +struct io_ring_ctx *ctx = req->ctx; + +mutex_lock (&ctx->uring_lock); nxt =io_req_find_next (req); +mutex_unlock (&ctx->uring_lock); + }io_free_req (req); }return nxt ? &nxt->work :NULL ;
Signature · 1 line
2.53.0
Jens Axboe
maintainerPATCH 2/3 io_uring: defer linked-timeout chain splice out of hrtimer context
In reply to Jens Axboeio_link_timeout_fn() is the hrtimer callback that fires when a linked timeout expires. It currently calls io_remove_next_linked(prev) under ctx->timeout_lock to splice the timeout request out of the link chain. This is the only chain-mutation site that runs without ctx->uring_lock, because hrtimer callbacks cannot take a mutex. Defer the splicing until the task_work callback.
--- io_uring/timeout.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
io_uring/timeout.c +14 -2
@@ -284,6 +284,10 @@ static struct io_kiocb *__io_disarm_linked_timeout(struct io_kiocb *req, struct io_timeout *timeout = io_kiocb_to_cmd(link, struct io_timeout);io_remove_next_linked (req); + +/* If this is NULL, then timer already claimed it and will complete it */ +if (!timeout->head) +return NULL ; timeout->head =NULL ;if (hrtimer_try_to_cancel (&io->timer) !=-1 ) {list_del (&timeout->list);
@@ -367,6 +371,14 @@ static void io_req_task_link_timeout(struct io_tw_req tw_req, io_tw_token_t tw)int ret;if (prev) { +/* +* splice the linked timeout out of prev's chain if the regular +* completion path didn't already do it. +*/ +if (prev->link == req) + prev->link = req->link; + req->link =NULL ; +if (!tw.cancel) {struct io_cancel_data cd = { .ctx = req->ctx,
@@ -401,10 +413,10 @@ static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer)/* * We don't expect the list to be empty, that will only happen if we -* race with the completion of the linked work. +* race with the completion of the linked work. Splice of prev is +* done in io_req_task_link_timeout(), if needed. */ if (prev) { -io_remove_next_linked (prev);if (!req_ref_inc_not_zero (prev)) prev =NULL ; }
Signature · 1 line
2.53.0
Jens Axboe
maintainerPATCH 3/3 io_uring: hold uring_lock across io_kill_timeouts() in cancel path
In reply to Jens Axboeio_uring_try_cancel_requests() dropped ctx->uring_lock before calling io_kill_timeouts(), which walks each timeout's link chain via io_match_task() to test REQ_F_INFLIGHT. With chain mutation now serialized by ctx->uring_lock, that walk needs the lock too.
--- io_uring/cancel.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/cancel.c +1 -1
@@ -561,8 +561,8 @@ __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, ret |=io_waitid_remove_all (ctx, tctx, cancel_all); ret |=io_futex_remove_all (ctx, tctx, cancel_all); ret |=io_uring_try_cancel_uring_cmd (ctx, tctx, cancel_all); -mutex_unlock (&ctx->uring_lock); ret |=io_kill_timeouts (ctx, tctx, cancel_all); +mutex_unlock (&ctx->uring_lock);if (tctx) ret |=io_run_task_work () >0 ;else
Signature · 1 line
2.53.0