io_uring/tw: serialize ctx->retry_llist with ->uring_lock
Jens Axboe
maintainerPATCH 1/1 io_uring/tw: serialize ctx->retry_llist with ->uring_lock
The DEFER_TASKRUN local task work paths all run under ctx->uring_lock, which serializes them with each other and with the rest of the ring's hot paths. io_move_task_work_from_local() is the exception - it's called from io_ring_exit_work() on a kworker without holding the lock and from the iopoll cancelation side right after dropping it.
->work_llist is fine with this, as it's only ever updated via the expected paths. But the ->retry_llist is updated while runing, and hence it could potentially race between normal task_work running and the task-has-exited shutdown path.
Simply grab ->uring_lock while moving the local work to the fallback list for exit purposes, which nicely serializes it across both the normal additions and the exit prune path.
---
io_uring/tw.c +11 -1
@@ -273,8 +273,18 @@ void io_req_task_work_add_remote(struct io_kiocb *req, unsigned flags)void __cold io_move_task_work_from_local (struct io_ring_ctx *ctx) { -struct llist_node *node =llist_del_all (&ctx->work_llist); +struct llist_node *node; +/* +* Running the work items may utilize ->retry_llist as a means +* for capping the number of task_work entries run at the same +* time. But that list can potentially race with moving the work +* from here, if the task is exiting. As any normal task_work +* running holds ->uring_lock already, just guard this slow path +* with ->uring_lock to avoid racing on ->retry_llist. +*/ +guard (mutex)(&ctx->uring_lock); + node =llist_del_all (&ctx->work_llist);__io_fallback_tw (node, false); node =llist_del_all (&ctx->retry_llist);__io_fallback_tw (node, false);
Signature · 1 line
Jens Axboe