Skip to contentSkip to conversation

post F_COMP_LOCKED optimisations

PATCH In review Pavel Begunkov 3 participants 9 messages Oct 14, 2020 – Oct 15, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover post F_COMP_LOCKED optimisations

A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].

Pavel Begunkov (2):
io_uring: optimise COMP_LOCK-less flush_completion
io_uring: optimise io_fail_links()

 fs/io_uring.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
Signature · 1 line
2.24.0
#1 lore raw

PATCH 1/2 io_uring: optimise COMP_LOCK-less flush_completion

After removing REQ_F_COMP_LOCKED io_submit_flush_completions() was left deferring completions via task_work_add() in the hot path, that might end up in a performance regression.

io_put_req() takes ->completion_lock only when at least one of REQ_F_FAIL_LINK or REQ_F_LINK_TIMEOUT is set. There is also REQ_F_WORK_INITIALIZED, freeing which while holding the lock have to be avoided because it may deadlock with work.fs->lock. And if none of them is set we can put under ->completion_lock and save an extra unlock/lock.

That actually works even better because it also works for most linked requests unlike it was before.

---
 fs/io_uring.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
fs/io_uring.c +10 -3
index cb2640f6fdb2..f61af4d487fd 100644
@@ -1601,12 +1601,19 @@ static void io_submit_flush_completions(struct io_comp_state *cs)
 		req = list_first_entry(&cs->list, struct io_kiocb, compl.list);
 		list_del(&req->compl.list);
 		__io_cqring_fill_event(req, req->result, req->compl.cflags);
-		if (!(req->flags & REQ_F_LINK_HEAD)) {
-			io_put_req_deferred(req, 1);
-		} else {
+
+		/*
+		 * io_free_req() doesn't care about completion_lock unless one
+		 * of these flags is set. REQ_F_WORK_INITIALIZED is in the list
+		 * because of a potential deadlock with req->work.fs->lock
+		 */
+		if (req->flags & (REQ_F_FAIL_LINK|REQ_F_LINK_TIMEOUT
+				 |REQ_F_WORK_INITIALIZED)) {
 			spin_unlock_irq(&ctx->completion_lock);
 			io_put_req(req);
 			spin_lock_irq(&ctx->completion_lock);
+		} else {
+			io_put_req(req);
 		}
 	}
 	io_commit_cqring(ctx);
Signature · 1 line
2.24.0
#2 lore raw

PATCH 2/2 io_uring: optimise io_fail_links()

In reply to Pavel Begunkov

Optimise put handling in __io_fail_links() after removing REQ_F_COMP_LOCKED.

---
 fs/io_uring.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/io_uring.c +10 -1
index f61af4d487fd..271a016e8507 100644
@@ -1816,7 +1816,16 @@ static void __io_fail_links(struct io_kiocb *req)
 		trace_io_uring_fail_link(req, link);
 
 		io_cqring_fill_event(link, -ECANCELED);
-		io_put_req_deferred(link, 2);
+
+		/*
+		 * It's ok to free under spinlock as they're not linked anymore,
+		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
+		 * work.fs->lock.
+		 */
+		if (link->flags | REQ_F_WORK_INITIALIZED)
+			io_put_req_deferred(link, 2);
+		else
+			io_double_put_req(link);
 	}
 
 	io_commit_cqring(ctx);
Signature · 1 line
2.24.0
maintainer
#6 lore raw
In reply to Pavel Begunkov
Quoted 3 lines · Pavel Begunkov

A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].

Part of this is undoubtedly that we only use the task_work for things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 for task_work.

That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?

Signature · 1 line
Jens Axboe
#7 lore raw
Quoted 5 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov

A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].

Quoted 7 lines

Part of this is undoubtedly that we only use the task_work for things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 for task_work.

That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?

Yep, was thinking the same. It's better to be in 5.10, and that would be easier to do by squashing.

Signature · 1 line
Pavel Begunkov
maintainer
#8 lore raw
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov

A naive test with io_uring-bench with [nops, submit/complete batch=32] shows 5500 vs 8500 KIOPS. Not really representative but might hurt if left without [1/2].

Part of this is undoubtedly that we only use the task_work for things that need signaling, for the deferred put case we should be able to get by with using notfy == 0 for task_work.

Quoted 6 lines
Quoted 3 lines

That said, it's nice to avoid when possible. At this point, I'd like to fold these into the last patch of the original series. What do you think?

Yep, was thinking the same. It's better to be in 5.10, and that would be easier to do by squashing.

OK, done.

Signature · 1 line
Jens Axboe
#3 lore raw

Re: [PATCH 2/2] io_uring: optimise io_fail_links()

In reply to Pavel Begunkov
Quoted 33 lines · Pavel Begunkov

Optimise put handling in __io_fail_links() after removing REQ_F_COMP_LOCKED.

---
 fs/io_uring.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/io_uring.c +10 -1
index f61af4d487fd..271a016e8507 100644
@@ -1816,7 +1816,16 @@ static void __io_fail_links(struct io_kiocb *req)
 		trace_io_uring_fail_link(req, link);
 
 		io_cqring_fill_event(link, -ECANCELED);
-		io_put_req_deferred(link, 2);
+
+		/*
+		 * It's ok to free under spinlock as they're not linked anymore,
+		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
+		 * work.fs->lock.
+		 */
+		if (link->flags | REQ_F_WORK_INITIALIZED)
+			io_put_req_deferred(link, 2);
+		else
+			io_double_put_req(link);
 	}
 
 	io_commit_cqring(ctx);
Signature · 1 line
2.24.0

This part of commit 9c357fed168a ("io_uring: fix REQ_F_COMP_LOCKED by killing it") causes a clang warning:

fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]

                if (link->flags | REQ_F_WORK_INITIALIZED)
                    ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~

1 warning generated.

According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add

fs/io_uring.c +3 -3
index 66c41d53a9d3..28b1a0fede3e 100644
@@ -1813,10 +1813,10 @@ static void __io_fail_links(struct io_kiocb *req)
 		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
 		 * work.fs->lock.
 		 */
-		if (link->flags | REQ_F_WORK_INITIALIZED)
-			io_put_req_deferred(link, 2);
-		else
+		if (link->flags & REQ_F_WORK_INITIALIZED)
 			io_double_put_req(link);
+		else
+			io_put_req_deferred(link, 2);
 	}
 
 	io_commit_cqring(ctx);
#4 lore raw

Re: [PATCH 2/2] io_uring: optimise io_fail_links()

Quoted 17 lines · Nathan Chancellor

Review

Quoted 11 lines · Pavel Begunkov
(quoted diff) +10 -1
-		io_put_req_deferred(link, 2);
+
+		/*
+		 * It's ok to free under spinlock as they're not linked anymore,
+		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
+		 * work.fs->lock.
+		 */
+		if (link->flags | REQ_F_WORK_INITIALIZED)
+			io_put_req_deferred(link, 2);
+		else
+			io_double_put_req(link);

fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]

                if (link->flags | REQ_F_WORK_INITIALIZED)
                    ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
Quoted 7 lines

1 warning generated.

According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add

I have no idea what have happened, but yeah, there should be "&", though without any additional negation. That's because deferred version is safer.

Nathan, thanks for letting know!
Jens, could you please fold in the change below.

fs/io_uring.c +1 -1
index 66c41d53a9d3..2c83c2688ec5 100644
@@ -1813,7 +1813,7 @@ static void __io_fail_links(struct io_kiocb *req)
 		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
 		 * work.fs->lock.
 		 */
-		if (link->flags | REQ_F_WORK_INITIALIZED)
+		if (link->flags & REQ_F_WORK_INITIALIZED)
 			io_put_req_deferred(link, 2);
 		else
 			io_double_put_req(link);
Signature · 1 line
Pavel Begunkov
maintainer
#5 lore raw

Re: [PATCH 2/2] io_uring: optimise io_fail_links()

Quoted 26 lines · Pavel Begunkov
Quoted 17 lines · Nathan Chancellor

Review

Quoted 11 lines · Pavel Begunkov
(quoted diff) +10 -1
-		io_put_req_deferred(link, 2);
+
+		/*
+		 * It's ok to free under spinlock as they're not linked anymore,
+		 * but avoid REQ_F_WORK_INITIALIZED because it may deadlock on
+		 * work.fs->lock.
+		 */
+		if (link->flags | REQ_F_WORK_INITIALIZED)
+			io_put_req_deferred(link, 2);
+		else
+			io_double_put_req(link);

fs/io_uring.c:1816:19: warning: bitwise or with non-zero value always evaluates to true [-Wtautological-bitwise-compare]

                if (link->flags | REQ_F_WORK_INITIALIZED)
                    ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
Quoted 7 lines

1 warning generated.

According to the comment, was it intended for that to be a bitwise AND then negated to check for the absence of it? If so, wouldn't it be clearer to flip the condition so that a negation is not necessary like below? I can send a formal patch if my analysis is correct but if not, feel free to fix it yourself and add

Quoted 6 lines

I have no idea what have happened, but yeah, there should be "&", though without any additional negation. That's because deferred version is safer.

Nathan, thanks for letting know!
Jens, could you please fold in the change below.

Done.

Signature · 1 line
Jens Axboe