io_uring: maintain drain requests' logic
PATCH 1/1 io_uring: maintain drain requests' logic
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)
sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.
--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
fs/io_uring.c +7 -2
@@ -455,6 +455,7 @@ struct io_ring_ctx {struct callback_head *exit_task_work;struct wait_queue_head hash_wait; +unsigned multishot_cqes;/* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;
@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; -return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->multishot_cqes != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;
@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) {struct io_ring_ctx *ctx = req->ctx;unsigned flags =IORING_CQE_F_MORE ; +bool multishot_poll = !(req->poll.events &EPOLLONESHOT );if (!error && req->poll.canceled) { error = -ECANCELED ;
@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true; flags =0 ; } +if (multishot_poll) + ctx->multishot_cqes++; +io_commit_cqring (ctx);return !(flags &IORING_CQE_F_MORE ); }
Signature · 1 line
1.8.3.1
Jens Axboe
maintainerQuoted 9 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.
Good point, we need to do something here... Looks simple enough to me, though I'd probably prefer if we rename 'multishot_cqes' to 'persistent_sqes' or something like that. It's likely not the last user of having 1:M mappings between sqe and cqe, so might as well try and name it a bit more appropriately.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.
Quoted 6 lines
Good point, we need to do something here... Looks simple enough to me, though I'd probably prefer if we rename '
multishot_cqes' to 'persistent_sqes' or something like that. It's likely not the last user of having 1:M mappings between sqe and cqe, so might as well try and name it a bit more appropriately.
persistent_sqes makes sense to me.
Quoted 47 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +4
-2@@ -455,6 +455,7 @@ struct io_ring_ctx {struct callback_head *exit_task_work;struct wait_queue_head hash_wait; +unsigned multishot_cqes;/* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; -return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->multishot_cqes != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) {struct io_ring_ctx *ctx = req->ctx;unsigned flags =IORING_CQE_F_MORE ; +bool multishot_poll = !(req->poll.events &EPOLLONESHOT );if (!error && req->poll.canceled) { error = -ECANCELED ;
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 6 lines
fs/io_uring.c +3
-0@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true; flags =0 ; } +if (multishot_poll) + ctx->multishot_cqes++; +
We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one. E.g. is failed __io_cqring_fill_event() in io_poll_complete() fine? Other places?
Btw, we can use some tests :)
Quoted 4 lines
fs/io_uring.c +0
-0io_commit_cqring (ctx);return !(flags &IORING_CQE_F_MORE ); }
Signature · 1 line
Pavel Begunkov
Quoted 47 lines · Pavel Begunkov
Quoted 44 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +4
-2@@ -455,6 +455,7 @@ struct io_ring_ctx {struct callback_head *exit_task_work;struct wait_queue_head hash_wait; +unsigned multishot_cqes;/* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; -return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->multishot_cqes != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) {struct io_ring_ctx *ctx = req->ctx;unsigned flags =IORING_CQE_F_MORE ; +bool multishot_poll = !(req->poll.events &EPOLLONESHOT );
Quoted 12 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 9 lines
fs/io_uring.c +3
-0if (!error && req->poll.canceled) { error = -ECANCELED ;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true; flags =0 ; } +if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Hi Pavel, I saw a killing request like iopoll_remove or async_cancel call io_cqring_fill_event() to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):
(1). add poll --> cancel poll, an ECANCELED cqe.
1sqe:1cqe all good
(2). add poll --> trigger event(queued to task_work) --> cancel poll,
an ECANCELED cqe --> task_work runs, another ECANCELED cqe.
1sqe:2cqesI suggest we shall only emit one ECANCELED cqe.
Currently I only account cqe through io_poll_complete(), so ECANCELED cqe from io_poll_remove or async_cancel etc are not counted in.
Quoted 2 lines
E.g. is failed
__io_cqring_fill_event()inio_poll_complete()fine? Other places?
a failed __io_cqring_fill_event() doesn't produce a cqe but increment ctx->cached_cq_overflow, as long as a cqe is produced or cached_cq_overflow is +=1, it is ok.
Quoted 2 lines
Btw, we can use some tests :)
I'll do more tests.
Quoted 7 lines
Quoted 6 lines
fs/io_uring.c +0
-0io_commit_cqring (ctx);return !(flags &IORING_CQE_F_MORE ); }
Quoted 55 lines · Hao Xu
Quoted 46 lines · Pavel Begunkov
Quoted 43 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)
sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>---
fs/io_uring.c| 9 +++++++--1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +1
-0@@ -455,6 +455,7 @@ struct io_ring_ctx { struct callback_head *exit_task_work; struct wait_queue_head hash_wait; + unsigned multishot_cqes; /* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) if (unlikely(req->flags & REQ_F_IO_DRAIN)) { struct io_ring_ctx *ctx = req->ctx;- return seq !=
ctx->cached_cq_tail
- +READ_ONCE(ctx->cached_cq_overflow); + return seq +ctx->multishot_cqes!=ctx->cached_cq_tail+ +READ_ONCE(ctx->cached_cq_overflow);
}
return false;fs/io_uring.c +1
-0@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) { struct io_ring_ctx *ctx = req->ctx; unsigned flags = IORING_CQE_F_MORE; + bool multishot_poll = !(req->poll.events & EPOLLONESHOT); if (!error && req->poll.canceled) { error = -ECANCELED;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true;Quoted 8 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 5 lines
fs/io_uring.c +3
-0flags = 0; } + if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Quoted 5 lines
Hi Pavel, I saw a killing request like
iopoll_removeorasync_cancelcallio_cqring_fill_event()to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):
(1). add poll --> cancel poll, an ECANCELED cqe.
1sqe:1cqe all good
(2). add poll --> triggerevent(queued to task_work)--> cancel poll, an ECANCELED cqe -->task_workruns, another ECANCELED cqe.
1sqe:2cqes
Those should emit a CQE on behalf of the request they're cancelling only when it's definitely cancelled and not going to fill it itself. E.g. if io_poll_cancel() found it and removed from all the list and core's poll infra.
At least before multi-cqe it should have been working fine.
Quoted 5 lines
I suggest we shall only emit one ECANCELED cqe.
Currently I only account cqe throughio_poll_complete(), so ECANCELED cqe fromio_poll_removeorasync_canceletc are not counted in.Quoted 2 lines
E.g. is failed
__io_cqring_fill_event()inio_poll_complete()fine? Other places?a failed
__io_cqring_fill_event()doesn't produce a cqe but incrementctx->cached_cq_overflow, as long as a cqe is produced orcached_cq_overflowis +=1, it is ok.
Not claiming that the case is broken, but cached_cq_overflow is considered in req_need_defer() as well, so from its perspective there is no much difference between succeed fill_event() or not.
Quoted 3 lines
Quoted 2 lines
Btw, we can use some tests :)
I'll do more tests.
Perfect!
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
fs/io_uring.c +0
-0io_commit_cqring(ctx); return !(flags & IORING_CQE_F_MORE); }
Signature · 1 line
Pavel Begunkov
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 78 lines · Hao Xu
Quoted 62 lines · Pavel Begunkov
Quoted 55 lines · Hao Xu
Quoted 46 lines · Pavel Begunkov
Quoted 43 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)
sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>---
fs/io_uring.c| 9 +++++++--1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +1
-0@@ -455,6 +455,7 @@ struct io_ring_ctx { struct callback_head *exit_task_work; struct wait_queue_head hash_wait; + unsigned multishot_cqes; /* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) if (unlikely(req->flags & REQ_F_IO_DRAIN)) { struct io_ring_ctx *ctx = req->ctx;- return seq !=
ctx->cached_cq_tail
- +READ_ONCE(ctx->cached_cq_overflow); + return seq +ctx->multishot_cqes!=ctx->cached_cq_tail+ +READ_ONCE(ctx->cached_cq_overflow);
}
return false;fs/io_uring.c +1
-0@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) { struct io_ring_ctx *ctx = req->ctx; unsigned flags = IORING_CQE_F_MORE; + bool multishot_poll = !(req->poll.events & EPOLLONESHOT); if (!error && req->poll.canceled) { error = -ECANCELED;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true;Quoted 8 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 5 lines
fs/io_uring.c +3
-0flags = 0; } + if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Quoted 5 lines
Hi Pavel, I saw a killing request like
iopoll_removeorasync_cancelcallio_cqring_fill_event()to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):
(1). add poll --> cancel poll, an ECANCELED cqe.
1sqe:1cqe all good
(2). add poll --> triggerevent(queued to task_work)--> cancel poll, an ECANCELED cqe -->task_workruns, another ECANCELED cqe.
1sqe:2cqesQuoted 7 lines
Those should emit a CQE on behalf of the request they're cancelling only when it's definitely cancelled and not going to fill it itself. E.g. if
io_poll_cancel()found it and removed from all the list and core's poll infra.At least before multi-cqe it should have been working fine.
I haven't done a test for this, but from the code logic, there could be case below:
io_poll_add() |
io_poll_remove(event happen)io_poll_wake() |io_poll_remove_one| io_poll_remove_waitqs | io_cqring_fill_event(-ECANCELED) |
Quoted 6 lines
task_workrun(io_poll_task_func) |
io_poll_complete() |req->poll.canceledis true, \ |
__io_cqring_fill_event(-ECANCELED) |two ECANCELED cqes, is there anything I missed?
Definitely may be be, but need to take a closer look
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
I suggest we shall only emit one ECANCELED cqe.
Currently I only account cqe throughio_poll_complete(), so ECANCELED cqe fromio_poll_removeorasync_canceletc are not counted in.Quoted 2 lines
E.g. is failed
__io_cqring_fill_event()inio_poll_complete()fine? Other places?a failed
__io_cqring_fill_event()doesn't produce a cqe but incrementctx->cached_cq_overflow, as long as a cqe is produced orcached_cq_overflowis +=1, it is ok.Not claiming that the case is broken, but
cached_cq_overflowis considered inreq_need_defer()as well, so from its perspective there is no much difference between succeedfill_event()or not.Quoted 3 lines
Quoted 2 lines
Btw, we can use some tests :)
I'll do more tests.
Perfect!
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/io_uring.c +0
-0io_commit_cqring(ctx); return !(flags & IORING_CQE_F_MORE); }
Signature · 1 line
Pavel Begunkov
Quoted 84 lines · Pavel Begunkov
Quoted 83 lines · Hao Xu
Quoted 62 lines · Pavel Begunkov
Quoted 55 lines · Hao Xu
Quoted 46 lines · Pavel Begunkov
Quoted 43 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)
sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +4
-1@@ -455,6 +455,7 @@ struct io_ring_ctx { struct callback_head *exit_task_work; struct wait_queue_head hash_wait; + unsigned multishot_cqes; /* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) if (unlikely(req->flags & REQ_F_IO_DRAIN)) { struct io_ring_ctx *ctx = req->ctx; - return seq != ctx->cached_cq_tail - + READ_ONCE(ctx->cached_cq_overflow); + return seq + ctx->multishot_cqes != ctx->cached_cq_tail + + READ_ONCE(ctx->cached_cq_overflow); } return false;@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) { struct io_ring_ctx *ctx = req->ctx; unsigned flags = IORING_CQE_F_MORE; + bool multishot_poll = !(req->poll.events & EPOLLONESHOT); if (!error && req->poll.canceled) { error = -ECANCELED;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true;Quoted 8 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 5 lines
fs/io_uring.c +3
-0flags = 0; } + if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Quoted 5 lines
Hi Pavel, I saw a killing request like
iopoll_removeorasync_cancelcallio_cqring_fill_event()to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):
(1). add poll --> cancel poll, an ECANCELED cqe.
1sqe:1cqe all good
(2). add poll --> triggerevent(queued to task_work)--> cancel poll, an ECANCELED cqe -->task_workruns, another ECANCELED cqe.
1sqe:2cqesQuoted 7 lines
Those should emit a CQE on behalf of the request they're cancelling only when it's definitely cancelled and not going to fill it itself. E.g. if
io_poll_cancel()found it and removed from all the list and core's poll infra.At least before multi-cqe it should have been working fine.
I haven't done a test for this, but from the code logic, there could be case below:
io_poll_add() |
io_poll_remove(event happen)io_poll_wake() |io_poll_remove_one| io_poll_remove_waitqs | io_cqring_fill_event(-ECANCELED) |
task_workrun(io_poll_task_func) |
io_poll_complete() |req->poll.canceledis true, \ |
__io_cqring_fill_event(-ECANCELED) |
Quoted 4 lines
Quoted 1 line
two ECANCELED cqes, is there anything I missed?
Definitely may be be, but need to take a closer look
I'll do some test to test if this issue exists, and make some change if it does.
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
I suggest we shall only emit one ECANCELED cqe.
Currently I only account cqe throughio_poll_complete(), so ECANCELED cqe fromio_poll_removeorasync_canceletc are not counted in.Quoted 2 lines
E.g. is failed
__io_cqring_fill_event()inio_poll_complete()fine? Other places?a failed
__io_cqring_fill_event()doesn't produce a cqe but incrementctx->cached_cq_overflow, as long as a cqe is produced orcached_cq_overflowis +=1, it is ok.Not claiming that the case is broken, but
cached_cq_overflowis considered inreq_need_defer()as well, so from its perspective there is no much difference between succeedfill_event()or not.Quoted 3 lines
Quoted 2 lines
Btw, we can use some tests :)
I'll do more tests.
Perfect!
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/io_uring.c +0
-0io_commit_cqring(ctx); return !(flags & IORING_CQE_F_MORE); }
Jens Axboe
maintainerQuoted 85 lines · Hao Xu
Quoted 84 lines · Pavel Begunkov
Quoted 83 lines · Hao Xu
Quoted 62 lines · Pavel Begunkov
Quoted 55 lines · Hao Xu
Quoted 46 lines · Pavel Begunkov
Quoted 43 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +4
-1@@ -455,6 +455,7 @@ struct io_ring_ctx {struct callback_head *exit_task_work;struct wait_queue_head hash_wait; +unsigned multishot_cqes;/* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; - return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->multishot_cqes != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) {struct io_ring_ctx *ctx = req->ctx;unsigned flags =IORING_CQE_F_MORE ; +bool multishot_poll = !(req->poll.events &EPOLLONESHOT );if (!error && req->poll.canceled) { error = -ECANCELED ;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true;Quoted 8 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4918
Quoted 5 lines
fs/io_uring.c +3
-0flags =0 ; } +if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Quoted 5 lines
Hi Pavel, I saw a killing request like
iopoll_removeorasync_cancelcallio_cqring_fill_event()to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):(1). add poll --> cancel poll, an ECANCELED cqe. 1sqe:1cqe all good (2). add poll --> trigger event(queued to task_work) --> cancel poll, an ECANCELED cqe --> task_work runs, another ECANCELED cqe. 1sqe:2cqesQuoted 7 lines
Those should emit a CQE on behalf of the request they're cancelling only when it's definitely cancelled and not going to fill it itself. E.g. if
io_poll_cancel()found it and removed from all the list and core's poll infra.At least before multi-cqe it should have been working fine.
I haven't done a test for this, but from the code logic, there could be case below:
io_poll_add()|io_poll_remove
(event happen)io_poll_wake() |io_poll_remove_one| io_poll_remove_waitqs | io_cqring_fill_event(-ECANCELED) | task_work run(io_poll_task_func) | io_poll_complete() | req->poll.canceled is true, \ | __io_cqring_fill_event(-ECANCELED) |
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
two ECANCELED cqes, is there anything I missed?
Definitely may be be, but need to take a closer look
I'll do some test to test if this issue exists, and make some change if it does.
How about something like this? Seems pointless to have an extra variable for this, when we already track if we're going to do more completions for this event or not. Also places the variable where it makes the most sense, and plenty of pad space there too.
Warning: totally untested. Would be great if you could, and hoping you're going to send out a v2.
fs/io_uring.c +6 -2
@@ -423,6 +423,7 @@ struct io_ring_ctx {unsigned cq_mask;atomic_t cq_timeouts;unsigned cq_last_tm_flush; +unsigned cq_extra;unsigned long cq_check_overflow;struct wait_queue_head cq_wait;struct fasync_struct *cq_fasync;
@@ -1183,8 +1184,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; -return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->cq_extra != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;
@@ -4894,6 +4895,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true; flags =0 ; } +if (flags &IORING_CQE_F_MORE ) + ctx->cq_extra++; +io_commit_cqring (ctx);return !(flags &IORING_CQE_F_MORE ); }
Signature · 1 line
Jens Axboe
Quoted 93 lines · Jens Axboe
Quoted 85 lines · Hao Xu
Quoted 84 lines · Pavel Begunkov
Quoted 83 lines · Hao Xu
Quoted 62 lines · Pavel Begunkov
Quoted 55 lines · Hao Xu
Quoted 46 lines · Pavel Begunkov
Quoted 43 lines · Hao Xu
Now that we have multishot poll requests, one sqe can emit multiple cqes. given below example:
sqe0(multishot poll)-->sqe1-->sqe2(drain req)sqe2 is designed to issue after sqe0 and sqe1 completed, but since sqe0 is a multishot poll request, sqe2 may be issued after sqe0's event triggered twice before sqe1 completed. This isn't what users leverage drain requests for.
Here a simple solution is to ignore all multishot poll cqes, which means drain requests won't wait those request to be done.Signed-off-by Hao Xu <[email protected]>--- fs/io_uring.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)fs/io_uring.c +4
-1@@ -455,6 +455,7 @@ struct io_ring_ctx {struct callback_head *exit_task_work;struct wait_queue_head hash_wait; +unsigned multishot_cqes;/* Keep this last, we don't need it for the fast path */ struct work_struct exit_work;@@ -1181,8 +1182,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; - return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->multishot_cqes != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;@@ -4897,6 +4898,7 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) {struct io_ring_ctx *ctx = req->ctx;unsigned flags =IORING_CQE_F_MORE ; +bool multishot_poll = !(req->poll.events &EPOLLONESHOT );if (!error && req->poll.canceled) { error = -ECANCELED ;@@ -4911,6 +4913,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true;Quoted 8 lines
fs/io_uring.c static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) · line 4900
Quoted 5 lines
fs/io_uring.c +3
-0flags =0 ; } +if (multishot_poll) + ctx->multishot_cqes++; +We need to make sure we do that only for a non-final complete, i.e. not killing request, otherwise it'll double account the last one.
Quoted 5 lines
Hi Pavel, I saw a killing request like
iopoll_removeorasync_cancelcallio_cqring_fill_event()to create an ECANCELED cqe for the original poll request. So there could be cases like(even for single poll request):(1). add poll --> cancel poll, an ECANCELED cqe. 1sqe:1cqe all good (2). add poll --> trigger event(queued to task_work) --> cancel poll, an ECANCELED cqe --> task_work runs, another ECANCELED cqe. 1sqe:2cqesQuoted 7 lines
Those should emit a CQE on behalf of the request they're cancelling only when it's definitely cancelled and not going to fill it itself. E.g. if
io_poll_cancel()found it and removed from all the list and core's poll infra.At least before multi-cqe it should have been working fine.
I haven't done a test for this, but from the code logic, there could be case below:
io_poll_add()|io_poll_remove
(event happen)io_poll_wake() |io_poll_remove_one| io_poll_remove_waitqs | io_cqring_fill_event(-ECANCELED) | task_work run(io_poll_task_func) | io_poll_complete() | req->poll.canceled is true, \ | __io_cqring_fill_event(-ECANCELED) |Quoted 6 lines
Quoted 4 lines
Quoted 1 line
two ECANCELED cqes, is there anything I missed?
Definitely may be be, but need to take a closer look
I'll do some test to test if this issue exists, and make some change if it does.
Quoted 8 lines
How about something like this? Seems pointless to have an extra variable for this, when we already track if we're going to do more completions for this event or not. Also places the variable where it makes the most sense, and plenty of pad space there too.
Warning: totally untested. Would be great if you could, and hoping you're going to send out a v2.
I'm writting a test for it, will send them together soon.
Quoted 35 lines
fs/io_uring.c +6
-2@@ -423,6 +423,7 @@ struct io_ring_ctx {unsigned cq_mask;atomic_t cq_timeouts;unsigned cq_last_tm_flush; +unsigned cq_extra;unsigned long cq_check_overflow;struct wait_queue_head cq_wait;struct fasync_struct *cq_fasync;@@ -1183,8 +1184,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq)if (unlikely (req->flags &REQ_F_IO_DRAIN )) {struct io_ring_ctx *ctx = req->ctx; -return seq != ctx->cached_cq_tail - +READ_ONCE (ctx->cached_cq_overflow); +return seq + ctx->cq_extra != ctx->cached_cq_tail + +READ_ONCE (ctx->cached_cq_overflow); }return false;@@ -4894,6 +4895,9 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error) req->poll.done = true; flags =0 ; } +if (flags &IORING_CQE_F_MORE ) + ctx->cq_extra++; +io_commit_cqring (ctx);return !(flags &IORING_CQE_F_MORE ); }