io_uring: pick up link work on submit reference drop
Jens Axboe
maintainerPATCH 1/1 io_uring: pick up link work on submit reference drop
If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.
---
fs/io_uring.c +9 -2
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 ); } -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt) +io_put_req (req); +else +io_put_req_find_next (req, &nxt);if (ret) {req_set_fail_links (req);
Signature · 1 line
Jens Axboe
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );
fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)
It can't even get here, because of the submission ref, isn't it? would the following do?
- io_put_req(req);
+ io_put_req_find_next(req, &nxt);
BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.
Quoted 7 lines
fs/io_uring.c +3
-0+io_put_req (req); + else + io_put_req_find_next(req, &nxt);if (ret) {req_set_fail_links (req);
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542
Quoted 2 lines
fs/io_uring.c +1
-1-io_put_req (req); +io_put_req_find_next (req, &nxt);
I don't think it can, let me make that change. And test.
Quoted 3 lines
BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.
We absolutely do need two references, unfortunately. Otherwise we could complete the io_kiocb deep down the stack through the callback.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 30 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- io_put_req(req); + io_put_req_find_next(req, &nxt);I don't think it can, let me make that change. And test.
Because I'm a clown, the patch applied with offset. I meant to modify the __io_queue_sqe() path, as mentioned in the commit message. Here's the right one, dropped the check
The other one would not be correct without the nxt check, as the work queue handler could pass it back. For the __io_queue_sqe() path, we should do a 5.7 cleanup of the 'nxt passing, though. I don't want to do that for 5.6.
commit 7df2fa5c9f6e92b2f80f4699425b463973d5242b
Author: Jens Axboe <[email protected]> Date: Tue Feb 25 13:25:41 2020 -0700
io_uring: pick up link work on submit reference drop
If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.
fs/io_uring.c +1 -1
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe) err:/* drop submission reference */ -io_put_req (req); +io_put_req_find_next (req, &nxt);if (linked_timeout) {if (!ret)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 36 lines · Jens Axboe
Quoted 30 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- io_put_req(req); + io_put_req_find_next(req, &nxt);I don't think it can, let me make that change. And test.
Quoted 8 lines
Because I'm a clown, the patch applied with offset. I meant to modify the
__io_queue_sqe()path, as mentioned in the commit message. Here's the right one, dropped the checkThe other one would not be correct without the nxt check, as the work queue handler could pass it back. For the
__io_queue_sqe()path, we should do a 5.7 cleanup of the 'nxt passing, though. I don't want to do that for 5.6.
So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And io_put_req_find_next() also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:
commit eff5fe974f332c1b86c9bb274627e88b4ecbbc85
Author: Jens Axboe <[email protected]> Date: Tue Feb 25 13:25:41 2020 -0700
io_uring: pick up link work on submit reference drop
If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.
This also fixes an issue with io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.
fs/io_uring.c +4 -4
@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req) __attribute__((nonnull))static void io_put_req_find_next (struct io_kiocb *req,struct io_kiocb **nxtptr) { -io_req_find_next (req, nxtptr); - -if (refcount_dec_and_test (&req->refs)) +if (refcount_dec_and_test (&req->refs)) { +io_req_find_next (req, nxtptr);__io_free_req (req); + } }static void io_put_req (struct io_kiocb *req)
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe) err:/* drop submission reference */ -io_put_req (req); +io_put_req_find_next (req, &nxt);if (linked_timeout) {if (!ret)
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And
io_put_req_find_next()also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:
It was weird indeed, it looks good. And now it's safe to do the same in io_wq_submit_work().
Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.
Quoted 46 lines
commit
eff5fe974f332c1b86c9bb274627e88b4ecbbc85Author: Jens Axboe <[email protected]> Date: Tue Feb 25 13:25:41 2020 -0700
io_uring: pick up link work on submit reference dropIf work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.This also fixes an issue with
io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.Signed-off-by Jens Axboe <[email protected]>fs/io_uring.c +4
-4@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req) __attribute__((nonnull))static void io_put_req_find_next (struct io_kiocb *req,struct io_kiocb **nxtptr) { -io_req_find_next (req, nxtptr); - -if (refcount_dec_and_test (&req->refs)) +if (refcount_dec_and_test (&req->refs)) { +io_req_find_next (req, nxtptr);__io_free_req (req); + } }static void io_put_req (struct io_kiocb *req)@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe) err:/* drop submission reference */ -io_put_req (req); +io_put_req_find_next (req, &nxt);if (linked_timeout) {if (!ret)
Signature · 1 line
Pavel Begunkov
Pavel Begunkov
Reviewed-byQuoted 7 lines · Pavel Begunkov
Quoted 5 lines · Jens Axboe
So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And
io_put_req_find_next()also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:
Quoted 5 lines
It was weird indeed, it looks good. And now it's safe to do the same in
io_wq_submit_work().Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.
... and it will return @nxt==NULL as well, as there is a ref hold by io_{put/get}_work().
Let's have this patch as is, and I'll cook up something on top
Quoted 48 lines
Quoted 47 lines
commit
eff5fe974f332c1b86c9bb274627e88b4ecbbc85Author: Jens Axboe <[email protected]> Date: Tue Feb 25 13:25:41 2020 -0700
io_uring: pick up link work on submit reference dropIf work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.This also fixes an issue with
io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.Signed-off-by Jens Axboe <[email protected]>fs/io_uring.c +4
-4@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req) __attribute__((nonnull))static void io_put_req_find_next (struct io_kiocb *req,struct io_kiocb **nxtptr) { -io_req_find_next (req, nxtptr); - -if (refcount_dec_and_test (&req->refs)) +if (refcount_dec_and_test (&req->refs)) { +io_req_find_next (req, nxtptr);__io_free_req (req); + } }static void io_put_req (struct io_kiocb *req)@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe) err:/* drop submission reference */ -io_put_req (req); +io_put_req_find_next (req, &nxt);if (linked_timeout) {if (!ret)
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 9 lines · Pavel Begunkov
Quoted 5 lines · Jens Axboe
So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And
io_put_req_find_next()also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:It was weird indeed, it looks good. And now it's safe to do the same in
io_wq_submit_work().
It is.
Quoted 2 lines
Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.
Indeed! That means we can get rid of that. We should do that for 5.7, though.
Signature · 1 line
Jens Axboe
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542
Quoted 2 lines
fs/io_uring.c +1
-1-io_put_req (req); +io_put_req_find_next (req, &nxt);I don't think it can, let me make that change. And test.
Quoted 6 lines
Quoted 3 lines
BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.
We absolutely do need two references, unfortunately. Otherwise we could complete the
io_kiocbdeep down the stack through the callback.
And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?
E.g. io_read(), as I see all its work is bound by lifetime of io_read() call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned from io_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.
Do I miss something?
Signature · 1 line
Pavel Begunkov
Quoted 48 lines · Pavel Begunkov
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542
Quoted 2 lines
fs/io_uring.c +1
-1-io_put_req (req); +io_put_req_find_next (req, &nxt);I don't think it can, let me make that change. And test.
Quoted 6 lines
Quoted 3 lines
BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.
We absolutely do need two references, unfortunately. Otherwise we could complete the
io_kiocbdeep down the stack through the callback.And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?
Quoted 6 lines
E.g.
io_read(), as I see all its work is bound by lifetime ofio_read()call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned fromio_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.Do I miss something?
Hmm, just started to question myself, whether handlers can be not as synchronous as described...
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 54 lines · Pavel Begunkov
Quoted 53 lines · Pavel Begunkov
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe
If work completes inline, then we should pick up a dependent link item in
__io_queue_sqe()as well. If we don't do so, we're forced to go async with that item, which is suboptimal.Signed-off-by Jens Axboe <[email protected]>---
fs/io_uring.c +0
-0@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr) }while (1 );fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539
Quoted 10 lines
fs/io_uring.c +6
-2} -/* drop submission reference */ -io_put_req (req); +/* +* Drop submission reference. In case the handler already dropped the +* completion reference, then it didn't pick up any potential link +* work. If 'nxt' isn't set, try and do that here. +*/ +if (nxt)It can't even get here, because of the submission ref, isn't it? would the following do?
fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542
Quoted 2 lines
fs/io_uring.c +1
-1-io_put_req (req); +io_put_req_find_next (req, &nxt);I don't think it can, let me make that change. And test.
Quoted 6 lines
Quoted 3 lines
BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.
We absolutely do need two references, unfortunately. Otherwise we could complete the
io_kiocbdeep down the stack through the callback.And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?E.g.
io_read(), as I see all its work is bound by lifetime ofio_read()call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned fromio_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.
Quoted 4 lines
Quoted 1 line
Do I miss something?
Hmm, just started to question myself, whether handlers can be not as synchronous as described...
It very much can complete the req after io_read() returns, that's what happens for any async disk request! By the time io_read() returns, the request could be completed, or it could just be in-flight. This is different from lots of the other opcodes, where the actual call returns completion sync (either success, or EAGAIN).
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
It very much can complete the req after
io_read()returns, that's what happens for any async disk request! By the timeio_read()returns, the request could be completed, or it could just be in-flight. This is different from lots of the other opcodes, where the actual call returns completion sync (either success, or EAGAIN).
For some reason, I've got the idea that it do the same things as __vfs_read/write. At least I don't see the difference between io_read_prep()+io_read() and new_sync_read().
Thanks for the explanation, I should drop these futile attempts.
Signature · 1 line
Pavel Begunkov