Skip to contentSkip to conversation

io_uring: pick up link work on submit reference drop

PATCH In review Jens Axboe 2 participants 12 messages Feb 25, 2020 – Feb 26, 2020

Subsystems (2)
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

PATCH 1/1 io_uring: pick up link work on submit reference drop

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +9 -2
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)
+		io_put_req(req);
+	else
+		io_put_req_find_next(req, &nxt);
 
 	if (ret) {
 		req_set_fail_links(req);
Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

- io_put_req(req);
+ io_put_req_find_next(req, &nxt);

BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.

Quoted 7 lines
fs/io_uring.c +3 -0
+		io_put_req(req);
+	else
+		io_put_req_find_next(req, &nxt);
 
 	if (ret) {
 		req_set_fail_links(req);
Signature · 1 line
Pavel Begunkov
author maintainer
#2 lore raw
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542

Quoted 2 lines
fs/io_uring.c +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Quoted 3 lines

BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.

We absolutely do need two references, unfortunately. Otherwise we could complete the io_kiocb deep down the stack through the callback.

Signature · 1 line
Jens Axboe
author maintainer
#3 lore raw
Quoted 30 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Because I'm a clown, the patch applied with offset. I meant to modify the __io_queue_sqe() path, as mentioned in the commit message. Here's the right one, dropped the check

The other one would not be correct without the nxt check, as the work queue handler could pass it back. For the __io_queue_sqe() path, we should do a 5.7 cleanup of the 'nxt passing, though. I don't want to do that for 5.6.

commit 7df2fa5c9f6e92b2f80f4699425b463973d5242b

Author: Jens Axboe <[email protected]>
Date:   Tue Feb 25 13:25:41 2020 -0700

io_uring: pick up link work on submit reference drop

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

fs/io_uring.c +1 -1
index ffd9bfa84d86..c4ed8601e225 100644
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 
 err:
 	/* drop submission reference */
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);
 
 	if (linked_timeout) {
 		if (!ret)
Signature · 1 line
Jens Axboe
author maintainer
#4 lore raw
Quoted 36 lines · Jens Axboe
Quoted 30 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Quoted 8 lines

Because I'm a clown, the patch applied with offset. I meant to modify the __io_queue_sqe() path, as mentioned in the commit message. Here's the right one, dropped the check

The other one would not be correct without the nxt check, as the work queue handler could pass it back. For the __io_queue_sqe() path, we should do a 5.7 cleanup of the 'nxt passing, though. I don't want to do that for 5.6.

So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And io_put_req_find_next() also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:

commit eff5fe974f332c1b86c9bb274627e88b4ecbbc85

Author: Jens Axboe <[email protected]>
Date:   Tue Feb 25 13:25:41 2020 -0700

io_uring: pick up link work on submit reference drop

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

This also fixes an issue with io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.

fs/io_uring.c +4 -4
index ffd9bfa84d86..f79ca494bb56 100644
@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req)
 __attribute__((nonnull))
 static void io_put_req_find_next(struct io_kiocb *req, struct io_kiocb **nxtptr)
 {
-	io_req_find_next(req, nxtptr);
-
-	if (refcount_dec_and_test(&req->refs))
+	if (refcount_dec_and_test(&req->refs)) {
+		io_req_find_next(req, nxtptr);
 		__io_free_req(req);
+	}
 }
 
 static void io_put_req(struct io_kiocb *req)
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 
 err:
 	/* drop submission reference */
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);
 
 	if (linked_timeout) {
 		if (!ret)
Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 5 lines · Jens Axboe

So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And io_put_req_find_next() also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:

It was weird indeed, it looks good. And now it's safe to do the same in io_wq_submit_work().

Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.

Quoted 46 lines

commit eff5fe974f332c1b86c9bb274627e88b4ecbbc85

Author: Jens Axboe <[email protected]>
Date:   Tue Feb 25 13:25:41 2020 -0700

io_uring: pick up link work on submit reference drop

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

This also fixes an issue with io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.

fs/io_uring.c +4 -4
index ffd9bfa84d86..f79ca494bb56 100644
@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req)
 __attribute__((nonnull))
 static void io_put_req_find_next(struct io_kiocb *req, struct io_kiocb **nxtptr)
 {
-	io_req_find_next(req, nxtptr);
-
-	if (refcount_dec_and_test(&req->refs))
+	if (refcount_dec_and_test(&req->refs)) {
+		io_req_find_next(req, nxtptr);
 		__io_free_req(req);
+	}
 }
 
 static void io_put_req(struct io_kiocb *req)
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 
 err:
 	/* drop submission reference */
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);
 
 	if (linked_timeout) {
 		if (!ret)
Signature · 1 line
Pavel Begunkov
Reviewed-by
#6 lore raw
Quoted 7 lines · Pavel Begunkov
Quoted 5 lines · Jens Axboe

So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And io_put_req_find_next() also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:

Quoted 5 lines

It was weird indeed, it looks good. And now it's safe to do the same in io_wq_submit_work().

Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.

... and it will return @nxt==NULL as well, as there is a ref hold by io_{put/get}_work().

Let's have this patch as is, and I'll cook up something on top

Quoted 48 lines
Quoted 47 lines

commit eff5fe974f332c1b86c9bb274627e88b4ecbbc85

Author: Jens Axboe <[email protected]>
Date:   Tue Feb 25 13:25:41 2020 -0700

io_uring: pick up link work on submit reference drop

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

This also fixes an issue with io_put_req_find_next(), which always looks up the next work item. That should only be done if we're dropping the last reference to the request, to prevent multiple lookups of the same work item.

fs/io_uring.c +4 -4
index ffd9bfa84d86..f79ca494bb56 100644
@@ -1483,10 +1483,10 @@ static void io_free_req(struct io_kiocb *req)
 __attribute__((nonnull))
 static void io_put_req_find_next(struct io_kiocb *req, struct io_kiocb **nxtptr)
 {
-	io_req_find_next(req, nxtptr);
-
-	if (refcount_dec_and_test(&req->refs))
+	if (refcount_dec_and_test(&req->refs)) {
+		io_req_find_next(req, nxtptr);
 		__io_free_req(req);
+	}
 }
 
 static void io_put_req(struct io_kiocb *req)
@@ -4749,7 +4749,7 @@ static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 
 err:
 	/* drop submission reference */
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);
 
 	if (linked_timeout) {
 		if (!ret)
Signature · 1 line
Pavel Begunkov
author maintainer
#7 lore raw
In reply to Pavel Begunkov
Quoted 9 lines · Pavel Begunkov
Quoted 5 lines · Jens Axboe

So this found something funky, we really should only be picking up the next request if we're dropping the final reference to the request. And io_put_req_find_next() also says that in the comment, but it always looks it up. That doesn't seem safe at all, I think this is what it should be:

It was weird indeed, it looks good. And now it's safe to do the same in io_wq_submit_work().

It is.

Quoted 2 lines

Interestingly, this means that passing @nxt into the handlers is useless, as they won't ever return !=NULL, isn't it? I'll prepare the cleanup.

Indeed! That means we can get rid of that. We should do that for 5.7, though.

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to Jens Axboe
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542

Quoted 2 lines
fs/io_uring.c +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Quoted 6 lines
Quoted 3 lines

BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.

We absolutely do need two references, unfortunately. Otherwise we could complete the io_kiocb deep down the stack through the callback.

And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?

E.g. io_read(), as I see all its work is bound by lifetime of io_read() call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned from io_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.

Do I miss something?

Signature · 1 line
Pavel Begunkov
#9 lore raw
Quoted 48 lines · Pavel Begunkov
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542

Quoted 2 lines
fs/io_uring.c +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Quoted 6 lines
Quoted 3 lines

BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.

We absolutely do need two references, unfortunately. Otherwise we could complete the io_kiocb deep down the stack through the callback.

And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?

Quoted 6 lines

E.g. io_read(), as I see all its work is bound by lifetime of io_read() call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned from io_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.

Do I miss something?

Hmm, just started to question myself, whether handlers can be not as synchronous as described...

Signature · 1 line
Pavel Begunkov
author maintainer
#10 lore raw
Quoted 54 lines · Pavel Begunkov
Quoted 53 lines · Pavel Begunkov
Quoted 35 lines · Jens Axboe
Quoted 29 lines · Pavel Begunkov
Quoted 14 lines · Jens Axboe

If work completes inline, then we should pick up a dependent link item in __io_queue_sqe() as well. If we don't do so, we're forced to go async with that item, which is suboptimal.

---

fs/io_uring.c +0 -0
index ffd9bfa84d86..160cf1b0f478 100644
@@ -4531,8 +4531,15 @@ static void io_wq_submit_work(struct io_wq_work **workptr)
 		} while (1);

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4539

Quoted 10 lines
fs/io_uring.c +6 -2
 	}
 
-	/* drop submission reference */
-	io_put_req(req);
+	/*
+	 * Drop submission reference. In case the handler already dropped the
+	 * completion reference, then it didn't pick up any potential link
+	 * work. If 'nxt' isn't set, try and do that here.
+	 */
+	if (nxt)

It can't even get here, because of the submission ref, isn't it? would the following do?

fs/io_uring.c static void io_wq_submit_work(struct io_wq_work **workptr) · line 4542

Quoted 2 lines
fs/io_uring.c +1 -1
-	io_put_req(req);
+	io_put_req_find_next(req, &nxt);

I don't think it can, let me make that change. And test.

Quoted 6 lines
Quoted 3 lines

BTW, as I mentioned before, it appears to me, we don't even need completion ref as it always pinned by the submission ref. I'll resurrect the patches doing that, but after your poll work will land.

We absolutely do need two references, unfortunately. Otherwise we could complete the io_kiocb deep down the stack through the callback.

And I need your knowledge here to not make mistakes :)
I remember the conversation about the necessity of submission ref, that's to make sure it won't be killed in the middle of block layer, etc. But what about removing the completion ref then?

E.g. io_read(), as I see all its work is bound by lifetime of io_read() call, so it's basically synchronous from the caller perspective. In other words, it can't complete req after it returned from io_read(). And that would mean it's save to have only submission ref after dealing with poll and other edge cases.

Quoted 4 lines
Quoted 1 line

Do I miss something?

Hmm, just started to question myself, whether handlers can be not as synchronous as described...

It very much can complete the req after io_read() returns, that's what happens for any async disk request! By the time io_read() returns, the request could be completed, or it could just be in-flight. This is different from lots of the other opcodes, where the actual call returns completion sync (either success, or EAGAIN).

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 5 lines · Jens Axboe

It very much can complete the req after io_read() returns, that's what happens for any async disk request! By the time io_read() returns, the request could be completed, or it could just be in-flight. This is different from lots of the other opcodes, where the actual call returns completion sync (either success, or EAGAIN).

For some reason, I've got the idea that it do the same things as __vfs_read/write. At least I don't see the difference between io_read_prep()+io_read() and new_sync_read().
Thanks for the explanation, I should drop these futile attempts.

Signature · 1 line
Pavel Begunkov