post F_COMP_LOCKED optimisations
cover post F_COMP_LOCKED optimisations
A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].
Pavel Begunkov (2):
io_uring: optimise COMP_LOCK-less flush_completion
io_uring: optimise io_fail_links()
fs/io_uring.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
Signature · 1 line
2.24.0
PATCH 1/2 io_uring: optimise COMP_LOCK-less flush_completion
After removing REQ_F_COMP_LOCKED io_submit_flush_completions() was left deferring completions via task_work_add() in the hot path, that might end up in a performance regression.
io_put_req() takes ->completion_lock only when at least one of REQ_F_FAIL_LINK or REQ_F_LINK_TIMEOUT is set. There is also REQ_F_WORK_INITIALIZED, freeing which while holding the lock have to be avoided because it may deadlock with work.fs->lock. And if none of them is set we can put under ->completion_lock and save an extra unlock/lock.
That actually works even better because it also works for most linked requests unlike it was before.
--- fs/io_uring.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
fs/io_uring.c +10 -3
@@ -1601,12 +1601,19 @@ static void io_submit_flush_completions(struct io_comp_state *cs) req = list_first_entry(&cs->list, struct io_kiocb, compl.list);list_del (&req->compl.list);__io_cqring_fill_event (req, req->result, req->compl.cflags); -if (!(req->flags &REQ_F_LINK_HEAD )) { -io_put_req_deferred (req,1 ); - }else { + +/* +* io_free_req() doesn't care about completion_lock unless one +* of these flags is set. REQ_F_WORK_INITIALIZED is in the list +* because of a potential deadlock with req->work.fs->lock +*/ +if (req->flags & (REQ_F_FAIL_LINK |REQ_F_LINK_TIMEOUT + |REQ_F_WORK_INITIALIZED )) {spin_unlock_irq (&ctx->completion_lock);io_put_req (req);spin_lock_irq (&ctx->completion_lock); + }else { +io_put_req (req); } }io_commit_cqring (ctx);
Signature · 1 line
2.24.0
PATCH 2/2 io_uring: optimise io_fail_links()
In reply to Pavel BegunkovOptimise put handling in __io_fail_links() after removing REQ_F_COMP_LOCKED.
--- fs/io_uring.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
fs/io_uring.c +10 -1
@@ -1816,7 +1816,16 @@ static void __io_fail_links(struct io_kiocb *req)trace_io_uring_fail_link (req, link);io_cqring_fill_event (link, -ECANCELED ); -io_put_req_deferred (link,2 ); + +/* +* It's ok to free under spinlock as they're not linked anymore, +* but avoid REQ_F_WORK_INITIALIZED because it may deadlock on +* work.fs->lock. +*/ +if (link->flags |REQ_F_WORK_INITIALIZED ) +io_put_req_deferred (link,2 ); +else +io_double_put_req (link); }io_commit_cqring (ctx);
Signature · 1 line
2.24.0
Re: [PATCH 2/2] io_uring: optimise io_fail_links()
Quoted 33 lines · Pavel Begunkov
Optimise put handling in
__io_fail_links()after removingREQ_F_COMP_LOCKED.Signed-off-by Pavel Begunkov <[email protected]>--- fs/io_uring.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)fs/io_uring.c +10
-1@@ -1816,7 +1816,16 @@ static void __io_fail_links(struct io_kiocb *req)trace_io_uring_fail_link (req, link);io_cqring_fill_event (link, -ECANCELED ); -io_put_req_deferred (link,2 ); + +/* +* It's ok to free under spinlock as they're not linked anymore, +* but avoid REQ_F_WORK_INITIALIZED because it may deadlock on +* work.fs->lock. +*/ +if (link->flags |REQ_F_WORK_INITIALIZED ) +io_put_req_deferred (link,2 ); +else +io_double_put_req (link); }io_commit_cqring (ctx);Signature · 1 line
2.24.0
This part of commit 9c357fed168a ("io_uring: fix REQ_F_COMP_LOCKED by killing it") causes a clang warning:
fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]
if (link->flags | REQ_F_WORK_INITIALIZED)
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~1 warning generated.
According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add
fs/io_uring.c +3 -3
@@ -1813,10 +1813,10 @@ static void __io_fail_links(struct io_kiocb *req)* but avoid REQ_F_WORK_INITIALIZED because it may deadlock on * work.fs->lock. */ -if (link->flags |REQ_F_WORK_INITIALIZED ) -io_put_req_deferred (link,2 ); -else +if (link->flags &REQ_F_WORK_INITIALIZED )io_double_put_req (link); +else +io_put_req_deferred (link,2 ); }io_commit_cqring (ctx);
Re: [PATCH 2/2] io_uring: optimise io_fail_links()
Quoted 17 lines · Nathan Chancellor
Review
Quoted 11 lines · Pavel Begunkov
(quoted diff) +10
-1- io_put_req_deferred(link, 2); + + /* + * It's ok to free under spinlock as they're not linked anymore, + * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on + * work.fs->lock. + */ + if (link->flags | REQ_F_WORK_INITIALIZED) + io_put_req_deferred(link, 2); + else + io_double_put_req(link);
fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]if (link->flags | REQ_F_WORK_INITIALIZED) ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
Quoted 7 lines
1 warning generated.
According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add
I have no idea what have happened, but yeah, there should be "&", though without any additional negation. That's because deferred version is safer.
Nathan, thanks for letting know!
Jens, could you please fold in the change below.
fs/io_uring.c +1 -1
@@ -1813,7 +1813,7 @@ static void __io_fail_links(struct io_kiocb *req)* but avoid REQ_F_WORK_INITIALIZED because it may deadlock on * work.fs->lock. */ -if (link->flags |REQ_F_WORK_INITIALIZED ) +if (link->flags &REQ_F_WORK_INITIALIZED )io_put_req_deferred (link,2 );else io_double_put_req (link);
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 2/2] io_uring: optimise io_fail_links()
Quoted 26 lines · Pavel Begunkov
Quoted 17 lines · Nathan Chancellor
Review
Quoted 11 lines · Pavel Begunkov
(quoted diff) +10
-1- io_put_req_deferred(link, 2); + + /* + * It's ok to free under spinlock as they're not linked anymore, + * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on + * work.fs->lock. + */ + if (link->flags | REQ_F_WORK_INITIALIZED) + io_put_req_deferred(link, 2); + else + io_double_put_req(link);
fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]if (link->flags | REQ_F_WORK_INITIALIZED) ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~Quoted 7 lines
1 warning generated.
According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add
Quoted 6 lines
I have no idea what have happened, but yeah, there should be "&", though without any additional negation. That's because deferred version is safer.
Nathan, thanks for letting know!
Jens, could you please fold in the change below.
Done.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 3 lines · Pavel Begunkov
A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].
Part of this is undoubtedly that we only use the task_work for things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 for task_work.
That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].
Quoted 7 lines
Part of this is undoubtedly that we only use the
task_workfor things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 fortask_work.That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?
Yep, was thinking the same. It's better to be in 5.10, and that would be easier to do by squashing.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].
Part of this is undoubtedly that we only use the
task_workfor things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 fortask_work.
Quoted 6 lines
Quoted 3 lines
That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?
Yep, was thinking the same. It's better to be in 5.10, and that would be easier to do by squashing.
OK, done.
Signature · 1 line
Jens Axboe