defer tw fixes and other cleanups
cover defer tw fixes and other cleanups
1-4 fix some defer tw problems, 5-6 are just dropped into the pile.
note: we intentionally break defer-taskrun test here.
Pavel Begunkov (6):
io_uring: further limit non-owner defer-tw cq waiting
io_uring: disallow defer-tw run w/ no submitters io_uring/iopoll: fix unexpected returns io_uring/iopoll: unify tw breaking logic
io_uring: add fast path for io_run_local_work()
io_uring: remove unused return from io_disarm_next
io_uring/io_uring.c | 35 ++++++++++++++++++----------------- io_uring/io_uring.h | 6 ++++++ io_uring/timeout.c | 13 +++---------- io_uring/timeout.h | 2 +- 4 files changed, 28 insertions(+), 28 deletions(-)
Signature · 1 line
2.37.2
PATCH 1/6 io_uring: further limit non-owner defer-tw cq waiting
In case of DEFER_TASK_WORK we try to restrict waiters to only one task, which is also the only submitter; however, we don't do it reliably, which might be very confusing and backfire in the future. E.g. we currently allow multiple tasks in io_iopoll_check().
--- io_uring/io_uring.c | 6 ++++++ io_uring/io_uring.h | 11 +++++++++++ 2 files changed, 17 insertions(+)
io_uring/io_uring.c +6 -0
@@ -1398,6 +1398,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)int ret =0 ;unsigned long check_cq; +if (!io_allowed_run_tw (ctx)) +return -EEXIST ; + check_cq =READ_ONCE (ctx->check_cq);if (unlikely (check_cq)) {if (check_cq &BIT (IO_CHECK_CQ_OVERFLOW_BIT ))
@@ -2386,6 +2389,9 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,ktime_t timeout =KTIME_MAX ;int ret; +if (!io_allowed_run_tw (ctx)) +return -EEXIST ; + do { /* always run at least 1 task work to process local work */ ret = io_run_task_work_ctx(ctx);
io_uring/io_uring.h +11 -0
@@ -329,4 +329,15 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return container_of(node, struct io_kiocb, comp_list); } +static inline bool io_allowed_run_tw (struct io_ring_ctx *ctx) +{ +if (!(ctx->flags &IORING_SETUP_DEFER_TASKRUN )) +return true; +if (unlikely (ctx->submitter_task != current)) { +/* maybe this is before any submissions */ +return !ctx->submitter_task; + } +return true; +} +#endif
Signature · 1 line
2.37.2
PATCH 2/6 io_uring: disallow defer-tw run w/ no submitters
In reply to Pavel BegunkovWe try to restrict CQ waiters when IORING_SETUP_DEFER_TASKRUN is set, but if nothing has been submitted yet it'll allow any waiter, which violates the contract.
--- io_uring/io_uring.c | 7 +------ io_uring/io_uring.h | 9 ++------- 2 files changed, 3 insertions(+), 13 deletions(-)
io_uring/io_uring.c +1 -6
@@ -1169,13 +1169,8 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool locked)int ret;unsigned int loops =1 ; -if (unlikely (ctx->submitter_task != current)) { -/* maybe this is before any submissions */ -if (!ctx->submitter_task) -return 0 ; - +if (unlikely (ctx->submitter_task != current))return -EEXIST ; - } node =io_llist_xchg (&ctx->work_llist, &fake); ret =0 ;
io_uring/io_uring.h +2 -7
@@ -331,13 +331,8 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx)static inline bool io_allowed_run_tw (struct io_ring_ctx *ctx) { -if (!(ctx->flags &IORING_SETUP_DEFER_TASKRUN )) -return true; -if (unlikely (ctx->submitter_task != current)) { -/* maybe this is before any submissions */ -return !ctx->submitter_task; - } -return true; +return likely (!(ctx->flags &IORING_SETUP_DEFER_TASKRUN ) || + ctx->submitter_task == current); }#endif
Signature · 1 line
2.37.2
PATCH 3/6 io_uring/iopoll: fix unexpected returns
In reply to Pavel BegunkovWe may propagate a positive return value of io_run_task_work() out of io_iopoll_check(), which breaks our tests. io_run_task_work() doesn't return anything useful for us, ignore the return value.
--- io_uring/io_uring.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
io_uring/io_uring.c +1 -4
@@ -1435,12 +1435,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)u32 tail = ctx->cached_cq_tail;mutex_unlock (&ctx->uring_lock); - ret =io_run_task_work (); +io_run_task_work ();mutex_lock (&ctx->uring_lock); -if (ret <0 ) -break ; -/* some requests don't go through iopoll_list */ if (tail != ctx->cached_cq_tail ||wq_list_empty (&ctx->iopoll_list))
Signature · 1 line
2.37.2
PATCH 4/6 io_uring/iopoll: unify tw breaking logic
In reply to Pavel BegunkovLet's keep checks for whether to break the iopoll loop or not same for normal and defer tw, this includes ->cached_cq_tail checks guarding against polling more than asked for.
--- io_uring/io_uring.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
io_uring/io_uring.c +7 -7
@@ -1428,21 +1428,21 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)*/ if (wq_list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { +u32 tail = ctx->cached_cq_tail; +if (!llist_empty (&ctx->work_llist))__io_run_local_work (ctx, true); +if (task_work_pending (current) ||wq_list_empty (&ctx->iopoll_list)) { -u32 tail = ctx->cached_cq_tail; -mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); - -/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail || -wq_list_empty (&ctx->iopoll_list)) -break ; } +/* some requests don't go through iopoll_list */ +if (tail != ctx->cached_cq_tail || +wq_list_empty (&ctx->iopoll_list)) +break ; } ret =io_do_iopoll (ctx, !min);if (ret <0 )
Signature · 1 line
2.37.2
PATCH 6/6 io_uring: remove unused return from io_disarm_next
In reply to Pavel BegunkovWe removed conditional io_commit_cqring_flush() guarding against spurious eventfd and the io_disarm_next()'s return value is not used anymore, just void it.
--- io_uring/timeout.c | 13 +++---------- io_uring/timeout.h | 2 +- 2 files changed, 4 insertions(+), 11 deletions(-)
io_uring/timeout.c +3 -10
@@ -149,11 +149,10 @@ static inline void io_remove_next_linked(struct io_kiocb *req) nxt->link =NULL ; } -bool io_disarm_next (struct io_kiocb *req) +void io_disarm_next (struct io_kiocb *req) __must_hold(&req->ctx->completion_lock) {struct io_kiocb *link =NULL ; -bool posted = false;if (req->flags &REQ_F_ARM_LTIMEOUT ) { link = req->link;
@@ -161,7 +160,6 @@ bool io_disarm_next(struct io_kiocb *req)if (link && link->opcode ==IORING_OP_LINK_TIMEOUT ) {io_remove_next_linked (req);io_req_tw_post_queue (link, -ECANCELED ,0 ); - posted = true; } }else if (req->flags &REQ_F_LINK_TIMEOUT ) {struct io_ring_ctx *ctx = req->ctx;
@@ -169,17 +167,12 @@ bool io_disarm_next(struct io_kiocb *req)spin_lock_irq (&ctx->timeout_lock); link =io_disarm_linked_timeout (req);spin_unlock_irq (&ctx->timeout_lock); -if (link) { - posted = true; +if (link)io_req_tw_post_queue (link, -ECANCELED ,0 ); - } }if (unlikely ((req->flags &REQ_F_FAIL ) && - !(req->flags &REQ_F_HARDLINK ))) { - posted |= (req->link !=NULL ); + !(req->flags &REQ_F_HARDLINK )))io_fail_links (req); - } -return posted; } struct io_kiocb *__io_disarm_linked_timeout(struct io_kiocb *req,
io_uring/timeout.h +1 -1
@@ -27,7 +27,7 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd);__cold bool io_kill_timeouts (struct io_ring_ctx *ctx,struct task_struct *tsk,bool cancel_all);void io_queue_linked_timeout (struct io_kiocb *req); -bool io_disarm_next (struct io_kiocb *req); +void io_disarm_next (struct io_kiocb *req);int io_timeout_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_link_timeout_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);
Signature · 1 line
2.37.2
PATCH 5/6 io_uring: add fast path for io_run_local_work()
In reply to Pavel BegunkovWe'll grab uring_lock and call __io_run_local_work() with several atomics inside even if there are no task works. Skip it if ->work_llist is empty.
--- io_uring/io_uring.c | 3 +++ 1 file changed, 3 insertions(+)
io_uring/io_uring.c +3 -0
@@ -1208,6 +1208,9 @@ int io_run_local_work(struct io_ring_ctx *ctx)bool locked;int ret; +if (llist_empty (&ctx->work_llist)) +return 0 ; + locked =mutex_trylock (&ctx->uring_lock); ret =__io_run_local_work (ctx, locked);if (locked)
Signature · 1 line
2.37.2
Re: [PATCH 1/6] io_uring: further limit non-owner defer-tw cq waiting
In reply to Pavel BegunkovQuoted 60 lines · Pavel Begunkov
In case of
DEFER_TASK_WORKwe try to restrict waiters to only one task,
which is also the only submitter; however, we don't do it reliably, which might be very confusing and backfire in the future. E.g. we currently allow multiple tasks inio_iopoll_check().Fixes dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN")Signed-off-by Pavel Begunkov <[email protected]>---
io_uring/io_uring.c | 6 ++++++
io_uring/io_uring.h | 11 +++++++++++2 files changed, 17 insertions(+)io_uring/io_uring.c +0
-0@@ -1398,6 +1398,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) int ret = 0; unsigned long check_cq;
+ if (!io_allowed_run_tw(ctx))
+ return-EEXIST;
+
check_cq =READ_ONCE(ctx->check_cq);
if (unlikely(check_cq)) {
if (check_cq&BIT(IO_CHECK_CQ_OVERFLOW_BIT))io_uring/io_uring.c +0
-0@@ -2386,6 +2389,9 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ktime_t timeout = KTIME_MAX; int ret;
+ if (!io_allowed_run_tw(ctx))
+ return-EEXIST;
+
do {
/* always run at least 1 task work to process local work */
ret =io_run_task_work_ctx(ctx);io_uring/io_uring.h +0
-0@@ -329,4 +329,15 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return container_of(node, struct io_kiocb, comp_list); }io_uring/io_uring.h +11
-0+static inline bool io_allowed_run_tw (struct io_ring_ctx *ctx) +{ + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) + return true; + if (unlikely(ctx->submitter_task != current)) { + /* maybe this is before any submissions */ + return !ctx->submitter_task; + } + return true; +} +
Quoted 1 line
#endif
Re: [PATCH 2/6] io_uring: disallow defer-tw run w/ no submitters
In reply to Pavel BegunkovQuoted 11 lines · Pavel Begunkov
We try to restrict CQ waiters when
IORING_SETUP_DEFER_TASKRUNis set, but if nothing has been submitted yet it'll allow any waiter, which violates the contract.Fixes dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN")Signed-off-by Pavel Begunkov <[email protected]>---
io_uring/io_uring.c | 7 +------
io_uring/io_uring.h | 9 ++-------2 files changed, 3 insertions(+), 13 deletions(-)
Re: [PATCH 3/6] io_uring/iopoll: fix unexpected returns
In reply to Pavel BegunkovQuoted 10 lines · Pavel Begunkov
We may propagate a positive return value of
io_run_task_work()out ofio_iopoll_check(), which breaks our tests.io_run_task_work()doesn't return anything useful for us, ignore the return value.Fixes dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN")Signed-off-by Pavel Begunkov <[email protected]>---
io_uring/io_uring.c | 5 +----1 file changed, 1 insertion(+), 4 deletions(-)
Re: [PATCH 5/6] io_uring: add fast path for io_run_local_work()
In reply to Pavel BegunkovQuoted 9 lines · Pavel Begunkov
We'll grab
uring_lockand call__io_run_local_work()with several atomics inside even if there are no task works. Skip it if -Quoted 1 line
work_llistis empty.
Signed-off-by Pavel Begunkov <[email protected]>---
io_uring/io_uring.c | 3 +++1 file changed, 3 insertions(+)
Quoted 12 lines · Pavel Begunkov
1-4 fix some defer tw problems, 5-6 are just dropped into the pile.
note: we intentionally break defer-taskrun test here.
Pavel Begunkov (6):
io_uring: further limit non-owner defer-tw cq waiting
io_uring: disallow defer-tw run w/ no submittersio_uring/iopoll: fix unexpected returnsio_uring/iopoll: unify tw breaking logic
io_uring: add fast path forio_run_local_work()
io_uring: remove unused return fromio_disarm_next
Quoted 1 line
[...]
Applied, thanks!
[1/6] io_uring: further limit non-owner defer-tw cq waiting
commit: 3ecccd114f6030f882e0dbc80d4958a80e0acb1c
[2/6] io_uring: disallow defer-tw run w/ no submitters
commit: 598ee40dc2602dea491b72d003c505e4e5ce4e36
[3/6] io_uring/iopoll: fix unexpected returns
commit: 6ccbce854d5d30d328cf4735869c40a13113c89a
[4/6] io_uring/iopoll: unify tw breaking logic
commit: c701863860cb6bc822ab0c96a351f8cd21c82b26
[5/6] io_uring: add fast path for io_run_local_work()
commit: 2592425110ebfdc1f8b5d89bb4460d5b028840a6
[6/6] io_uring: remove unused return from io_disarm_next
commit: 62073e8ffd01ca543eab391e0af9a3194a6e9df3Best regards,
Signature · 1 line
Jens Axboe