Skip to contentSkip to conversation

Send and receive bundles

PATCH In review Jens Axboe 2 participants 13 messages Apr 20, 2024 – May 1, 2024 io_uring

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

v2 cover Send and receive bundles

Hi,

I went back to the drawing board a bit on the send multishot, and this is what came out.

First support was added for provided buffers for send. This works like provided buffers for recv/recvmsg, and the intent here to use the buffer ring queue as an outgoing sequence for sending.

But the real meat is adding support for picking multiple buffers at the time, what I dubbed "bundles" here. Rather than just pick a single buffer for send, it can pick a bunch of them and send them in one go. The idea here is that the expensive part of a request is not the sqe issue, it's the fact that we have to do each buffer separately. That entails calling all the way down into the networking stack, locking the socket, checking what needs doing afterwards (like flushing the backlog), unlocking the socket, etc. If we have an outgoing send queue, then pick what buffers we have (up to a certain cap), and pass them to the networking stack in one go.

Bundles must be used with provided buffers, obviously. At completion time, they pass the starting buffer ID in cqe->flags, like any other provided buffer completion. cqe->res is the TOTAL number of bytes sent, so it's up to the application to iterate buffers to figure out how many completed. This part is trivial. I'll push the proxy changes out soon, just need to cleanup them up as I did the sendmsg bundling too and would love to compare.

With that in place, I added support for recv for bundles as well. Exactly the same as the send side - if we have a known amount of data pending, pick enough buffers to satisfy the receive and post a single completion for that round. Buffer ID in cqe->flags, cqe->res is the total number of buffers sent. Receive can be used with multishot as well - fire off one multishot recv, and keep getting big completions. Unfortunately, recvmsg multishot is just not as efficient as recv, as it carries additional data that needs copying. recv multishot with bundles provide a good alternative to recvmsg, if all you need is more than one range of data. I'll compare these too soon as well.

This is obviously a bigger win for smaller packets than for large ones, as the overall cost of entering sys_sendmsg/sys_recvmsg() in terms of throughput decreases as the packet size increases. For the extreme end, using 32b packets, performance increases substantially. Runtime for proxying 32b packets between three machines on a 10G link for the test:

Send ring: 3462 msec 1183Mbit
Send ring + bundles 844 msec 4853Mbit

and bundles reach 100% bandwidth at 80b of packet size, compared to send ring alone needing 320b to reach 95% of bandwidth (I didn't redo that test so don't have the 100% number).

Patches are on top of my for-6.9/io_uring branch and can also be found here:

https://git.kernel.dk/cgit/linux/log/?h=io_uring-recvsend-bundle

Changes since v1:
- Shuffle some hunks around
- Fix various bugs
- Rebase on current 6.10 branch

 include/linux/io_uring_types.h |   3 +
 include/uapi/linux/io_uring.h  |  10 ++
 io_uring/io_uring.c            |   3 +-
 io_uring/kbuf.c                | 157 +++++++++++++++++-
 io_uring/kbuf.h                |  53 ++++--
 io_uring/net.c                 | 284 ++++++++++++++++++++++++++++-----
 io_uring/opdef.c               |   1 +
 7 files changed, 456 insertions(+), 55 deletions(-)
Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH v2 1/5 io_uring/net: add generic multishot retry helper

This is just moving io_recv_prep_retry() higher up so it can get used for sends as well, and rename it to be generically useful for both sends and receives.

---
 io_uring/net.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
io_uring/net.c +12 -12
index a1da8a2ebf15..dc310f0bfe4c 100644
@@ -185,6 +185,17 @@ static int io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg,
 	return 0;
 }
 
+static inline void io_mshot_prep_retry(struct io_kiocb *req,
+				       struct io_async_msghdr *kmsg)
+{
+	struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);
+
+	req->flags &= ~REQ_F_BL_EMPTY;
+	sr->done_io = 0;
+	sr->len = 0; /* get from the provided buffer */
+	req->buf_index = sr->buf_group;
+}
+
 #ifdef CONFIG_COMPAT
 static int io_compat_msg_copy_hdr(struct io_kiocb *req,
 				  struct io_async_msghdr *iomsg,
@@ -658,17 +669,6 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	return io_recvmsg_prep_setup(req);
 }
 
-static inline void io_recv_prep_retry(struct io_kiocb *req,
-				      struct io_async_msghdr *kmsg)
-{
-	struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);
-
-	req->flags &= ~REQ_F_BL_EMPTY;
-	sr->done_io = 0;
-	sr->len = 0; /* get from the provided buffer */
-	req->buf_index = sr->buf_group;
-}
-
 /*
  * Finishes io_recv and io_recvmsg.
  *
@@ -694,7 +694,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret,
 		struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);
 		int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE;
 
-		io_recv_prep_retry(req, kmsg);
+		io_mshot_prep_retry(req, kmsg);
 		/* Known not-empty or unknown state, retry */
 		if (cflags & IORING_CQE_F_SOCK_NONEMPTY || kmsg->msg.msg_inq < 0) {
 			if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY)
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH v2 2/5 io_uring/net: add provided buffer support for IORING_OP_SEND

In reply to Jens Axboe

It's pretty trivial to wire up provided buffer support for the send side, just like how it's done the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring.

One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, completes successfully, posts CQE
5) sendB is issued, completes successfully, posts CQE

All is fine. Requests are always issued in-order, and both complete inline as most sends do.

However, if we're flooding socket1 with sends, the following could also result from the same sequence:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, completes successfully, posts CQE
7) sendA is retried, completes successfully, posts CQE

Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead:

1) App queues dataA for sendA, queues sendA for socket1
2) App queues dataB for sendB queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA")
7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB")

Now we've sent the data in order, and everybody is happy.

It's worth noting that this also opens the door for supporting multishot sends, as provided buffers would be a prerequisite for that. Those can trigger either when new buffers are added to the outgoing ring, or (if stalled due to lack of space) when space frees up in the socket.

---
 io_uring/net.c   | 25 ++++++++++++++++++++-----
 io_uring/opdef.c |  1 +
 2 files changed, 21 insertions(+), 5 deletions(-)
io_uring/net.c +20 -5
index dc310f0bfe4c..13685d133582 100644
@@ -364,10 +364,12 @@ static int io_send_setup(struct io_kiocb *req)
 		kmsg->msg.msg_name = &kmsg->addr;
 		kmsg->msg.msg_namelen = sr->addr_len;
 	}
-	ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter);
-	if (unlikely(ret < 0))
-		return ret;
-
+	if (!io_do_buffer_select(req)) {
+		ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len,
+				  &kmsg->msg.msg_iter);
+		if (unlikely(ret < 0))
+			return ret;
+	}
 	return 0;
 }
 
@@ -480,6 +482,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);
 	struct io_async_msghdr *kmsg = req->async_data;
 	struct socket *sock;
+	unsigned int cflags;
 	unsigned flags;
 	int min_ret = 0;
 	int ret;
@@ -492,6 +495,17 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	    (sr->flags & IORING_RECVSEND_POLL_FIRST))
 		return -EAGAIN;
 
+	if (io_do_buffer_select(req)) {
+		size_t len = sr->len;
+		void __user *buf;
+
+		buf = io_buffer_select(req, &len, issue_flags);
+		if (unlikely(!buf))
+			return -ENOBUFS;
+		sr->buf = buf;
+		sr->len = len;
+	}
+
 	flags = sr->msg_flags;
 	if (issue_flags & IO_URING_F_NONBLOCK)
 		flags |= MSG_DONTWAIT;
@@ -521,7 +535,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	else if (sr->done_io)
 		ret = sr->done_io;
 	io_req_msg_cleanup(req, issue_flags);
-	io_req_set_res(req, ret, 0);
+	cflags = io_put_kbuf(req, issue_flags);
+	io_req_set_res(req, ret, cflags);
 	return IOU_OK;
 }
 
io_uring/opdef.c +1 -0
index a16f73938ebb..2de5cca9504e 100644
@@ -281,6 +281,7 @@ const struct io_issue_def io_issue_defs[] = {
 		.pollout		= 1,
 		.audit_skip		= 1,
 		.ioprio			= 1,
+		.buffer_select		= 1,
 #if defined(CONFIG_NET)
 		.async_size		= sizeof(struct io_async_msghdr),
 		.prep			= io_sendmsg_prep,
Signature · 1 line
2.43.0
author maintainer
#10 lore raw

PATCH v2 3/5 io_uring/kbuf: add helpers for getting/peeking multiple buffers

In reply to Jens Axboe

Our provided buffer interface only allows selection of a single buffer. Add an API that allows getting/peeking multiple buffers at the same time.

This is only implemented for the ring provided buffers. It could be added for the legacy provided buffers as well, but since it's strongly encouraged to use the new interface, let's keep it simpler and just provide it for the new API. The legacy interface will always just select a single buffer.

There are two new main functions:

io_buffers_select(), which selects up as many buffers as it can. The caller supplies the iovec array, and io_buffers_select() may allocate a bigger array if the 'out_len' being passed in is non-zero and bigger than what fits in the provided iovec. Buffers grabbed with this helper are permanently assigned.

io_buffers_peek(), which works like io_buffers_select(), except they can be recycled, if needed. Callers using either of these functions should call io_put_kbufs() rather than io_put_kbuf() at completion time. The peek interface must be called with the ctx locked from peek to completion.

This add a bit state for the request:

- REQ_F_BUFFERS_COMMIT, which means that the the buffers have been peeked and should be committed to the buffer ring head when they are put as part of completion. Prior to this, req->buf_list was cleared to NULL when committed.

---
 include/linux/io_uring_types.h |   3 +
 io_uring/kbuf.c                | 157 ++++++++++++++++++++++++++++++++-
 io_uring/kbuf.h                |  53 +++++++++--
 3 files changed, 201 insertions(+), 12 deletions(-)
include/linux/io_uring_types.h +3 -0
index c47f412cf18e..7a6b190c7da7 100644
@@ -472,6 +472,7 @@ enum {
 	REQ_F_CAN_POLL_BIT,
 	REQ_F_BL_EMPTY_BIT,
 	REQ_F_BL_NO_RECYCLE_BIT,
+	REQ_F_BUFFERS_COMMIT_BIT,
 
 	/* not a real bit, just to check we're not overflowing the space */
 	__REQ_F_LAST_BIT,
@@ -550,6 +551,8 @@ enum {
 	REQ_F_BL_EMPTY		= IO_REQ_FLAG(REQ_F_BL_EMPTY_BIT),
 	/* don't recycle provided buffers for this request */
 	REQ_F_BL_NO_RECYCLE	= IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT),
+	/* buffer ring head needs incrementing on put */
+	REQ_F_BUFFERS_COMMIT	= IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT),
 };
 
 typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts);
io_uring/kbuf.c +154 -3

183 lines not on this page · raw message

io_uring/kbuf.h +44 -9

94 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#11 lore raw

PATCH v2 4/5 io_uring/net: support bundles for send

In reply to Jens Axboe

If IORING_OP_SEND is used with provided buffers, the caller may also set IORING_RECVSEND_BUNDLE to turn it into a multi-buffer send. The idea is that an application can fill outgoing buffers in a provided buffer group, and then arm a single send that will service them all. Once there are no more buffers to send, or if the requested length has been sent, the request posts a single completion for all the buffers.

This only enables it for IORING_OP_SEND, IORING_OP_SENDMSG is coming in a separate patch. However, this patch does do a lot of the prep work that makes wiring up the sendmsg variant pretty trivial. They share the prep side.

---
 include/uapi/linux/io_uring.h |   9 +++
 io_uring/net.c                | 145 ++++++++++++++++++++++++++++++----
 2 files changed, 137 insertions(+), 17 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index a7f847543a7f..7f583927c908 100644
@@ -351,11 +351,20 @@ enum io_uring_op {
  *				0 is reported if zerocopy was actually possible.
  *				IORING_NOTIF_USAGE_ZC_COPIED if data was copied
  *				(at least partially).
+ *
+ * IORING_RECVSEND_BUNDLE	Used with IOSQE_BUFFER_SELECT. If set, send will
+ *				grab as many buffers from the buffer group ID
+ *				given and send them all. The completion result
+ *				will be the number of buffers send, with the
+ *				starting buffer ID in cqe->flags as per usual
+ *				for provided buffer usage. The buffers will be
+ *				contigious from the starting buffer ID.
  */
 #define IORING_RECVSEND_POLL_FIRST	(1U << 0)
 #define IORING_RECV_MULTISHOT		(1U << 1)
 #define IORING_RECVSEND_FIXED_BUF	(1U << 2)
 #define IORING_SEND_ZC_REPORT_USAGE	(1U << 3)
+#define IORING_RECVSEND_BUNDLE		(1U << 4)
 
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
io_uring/net.c +128 -17

196 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#12 lore raw

PATCH v2 5/5 io_uring/net: support bundles for recv

In reply to Jens Axboe

If IORING_OP_RECV is used with provided buffers, the caller may also set IORING_RECVSEND_BUNDLE to turn it into a multi-buffer recv. This grabs buffers available and receives into them, posting a single completion for all of it.

This can be used with multishot receive as well, or without it.

Now that both send and receive support bundles, add a feature flag for it as well. If IORING_FEAT_RECVSEND_BUNDLE is set after registering the ring, then the kernel supports bundles for recv and send.

---
 include/uapi/linux/io_uring.h |  15 +++--
 io_uring/io_uring.c           |   3 +-
 io_uring/net.c                | 116 ++++++++++++++++++++++++++++------
 3 files changed, 105 insertions(+), 29 deletions(-)
include/uapi/linux/io_uring.h +8 -7
index 7f583927c908..f093cb2300d9 100644
@@ -352,13 +352,13 @@ enum io_uring_op {
  *				IORING_NOTIF_USAGE_ZC_COPIED if data was copied
  *				(at least partially).
  *
- * IORING_RECVSEND_BUNDLE	Used with IOSQE_BUFFER_SELECT. If set, send will
- *				grab as many buffers from the buffer group ID
- *				given and send them all. The completion result
- *				will be the number of buffers send, with the
- *				starting buffer ID in cqe->flags as per usual
- *				for provided buffer usage. The buffers will be
- *				contigious from the starting buffer ID.
+ * IORING_RECVSEND_BUNDLE	Used with IOSQE_BUFFER_SELECT. If set, send or
+ *				recv will grab as many buffers from the buffer
+ *				group ID given and send them all. The completion
+ *				result 	will be the number of buffers send, with
+ *				the starting buffer ID in cqe->flags as per
+ *				usual for provided buffer usage. The buffers
+ *				will be	contigious from the starting buffer ID.
  */
 #define IORING_RECVSEND_POLL_FIRST	(1U << 0)
 #define IORING_RECV_MULTISHOT		(1U << 1)
@@ -529,6 +529,7 @@ struct io_uring_params {
 #define IORING_FEAT_CQE_SKIP		(1U << 11)
 #define IORING_FEAT_LINKED_FILE		(1U << 12)
 #define IORING_FEAT_REG_REG_RING	(1U << 13)
+#define IORING_FEAT_RECVSEND_BUNDLE	(1U << 14)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/io_uring.c +2 -1
index 3c9087f37c43..2675cffbd9a4 100644
@@ -3570,7 +3570,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,
 			IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED |
 			IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS |
 			IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP |
-			IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING;
+			IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING |
+			IORING_FEAT_RECVSEND_BUNDLE;
 
 	if (copy_to_user(params, p, sizeof(*p))) {
 		ret = -EFAULT;
io_uring/net.c +95 -21

177 lines not on this page · raw message

Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH 2/5] io_uring/net: add provided buffer support for IORING_OP_SEND

In reply to Jens Axboe
Quoted 40 lines · Jens Axboe

It's pretty trivial to wire up provided buffer support for the send side, just like how it's done the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring.

One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, completes successfully, posts CQE
5) sendB is issued, completes successfully, posts CQE

All is fine. Requests are always issued in-order, and both complete inline as most sends do.

However, if we're flooding socket1 with sends, the following could also result from the same sequence:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, completes successfully, posts CQE
7) sendA is retried, completes successfully, posts CQE

Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead:

Quoted 9 lines

1) App queues dataA for sendA, queues sendA for socket1
2) App queues dataB for sendB queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA")
7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB")

Hi Jens,

If I understand correctly, when sending a buffer, we set sr->len to be the smallest between the buffer size and what was requested in sqe->len. But, when we disconnect the buffer from the request, we can get in a situation where the buffers and requests mismatch, and only one buffer gets sent.

Say we are sending two buffers through non-bundle sends with different sizes to the same socket in this order:

buff[1]->len = 128
buff[2]->len = 256

And SQEs like this:

sqe[1]->len = 128
sqe[2]->len = 256

If sqe1 picks buff1 it is all good. But, if sqe[2] runs first, then sqe[1] picks buff2, and it will only send the first 128, won't it? Looking at the patch I don't see how you avoid this condition, but perhaps I'm missing something?

One suggestion would be requiring sqe->len to be 0 when using send with provided buffers, so we simply use the entire buffer in
the ring. wdyt?

Thanks,

Signature · 1 line
Gabriel Krisman Bertazi
#4 lore raw

Re: [PATCH 2/5] io_uring/net: add provided buffer support for IORING_OP_SEND

Gabriel Krisman Bertazi <[email protected]> writes:

...

Quoted 2 lines

situation where the buffers and requests mismatch, and only one buffer gets sent.

Sorry, here I meant that *only part of a buffer* might get sent because we truncate to sqe->len. As in the example I gave.

Signature · 1 line
Gabriel Krisman Bertazi
author maintainer
#5 lore raw

Re: [PATCH 2/5] io_uring/net: add provided buffer support for IORING_OP_SEND

In reply to Gabriel Krisman Bertazi
Quoted 71 lines · Gabriel Krisman Bertazi
Quoted 40 lines · Jens Axboe

It's pretty trivial to wire up provided buffer support for the send side, just like how it's done the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring.

One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, completes successfully, posts CQE
5) sendB is issued, completes successfully, posts CQE

All is fine. Requests are always issued in-order, and both complete inline as most sends do.

However, if we're flooding socket1 with sends, the following could also result from the same sequence:

1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, completes successfully, posts CQE
7) sendA is retried, completes successfully, posts CQE

Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead:

Quoted 9 lines

1) App queues dataA for sendA, queues sendA for socket1
2) App queues dataB for sendB queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA")
7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB")

Hi Jens,

If I understand correctly, when sending a buffer, we set sr->len to be the smallest between the buffer size and what was requested in sqe->len. But, when we disconnect the buffer from the request, we can get in a situation where the buffers and requests mismatch, and only one buffer gets sent.

Say we are sending two buffers through non-bundle sends with different sizes to the same socket in this order:

buff[1]->len = 128
buff[2]->len = 256

And SQEs like this:

sqe[1]->len = 128
sqe[2]->len = 256

Quoted 8 lines

If sqe1 picks buff1 it is all good. But, if sqe[2] runs first, then sqe[1] picks buff2, and it will only send the first 128, won't it? Looking at the patch I don't see how you avoid this condition, but perhaps I'm missing something?

One suggestion would be requiring sqe->len to be 0 when using send with provided buffers, so we simply use the entire buffer in
the ring. wdyt?

It might not hurt to just enforce it to be 0, in fact I think any sane use case would do that and I don't think the above use case is a very valid one. It's a bit of "you get to keep both pieces when it breaks".

Do you want to send a patch that just enforces it to be 0? We do have that requirement in other spots for provided buffers and multishot, so I think it'll make sense to do here too regardless of the sanity of the use case.

Signature · 1 line
Jens Axboe
#6 lore raw

PATCH 1/1 io_uring: Require zeroed sqe->len on provided-buffers send

When sending from a provided buffer, we set sr->len to be the smallest between the actual buffer size and sqe->len. But, now that we disconnect the buffer from the submission request, we can get in a situation where the buffers and requests mismatch, and only part of a buffer gets sent. Assume:

* buf[1]->len = 128; buf[2]->len = 256
* sqe[1]->len = 128; sqe[2]->len = 256

If sqe1 runs first, it picks buff[1] and it's all good. But, if sqe[2] runs first, sqe[1] picks buff[2], and the last half of buff[2] is never sent.

While arguably the use-case of different-length sends is questionable, it has already raised confusion with potential users of this feature. Let's make the interface less tricky by forcing the length to only come from the buffer ring entry itself.

Fixes ac5f71a3d9d7 ("io_uring/net: add provided buffer support for IORING_OP_SEND")
---
 io_uring/net.c | 2 ++
 1 file changed, 2 insertions(+)
io_uring/net.c +2 -0
index 51c41d771c50..ffe37dd77a74 100644
@@ -423,6 +423,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (req->flags & REQ_F_BUFFER_SELECT && sr->len)
+		return -EINVAL;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)
Signature · 1 line
2.44.0
author maintainer
#7 lore raw

Re: [PATCH] io_uring: Require zeroed sqe->len on provided-buffers send

Quoted 29 lines · Gabriel Krisman Bertazi

When sending from a provided buffer, we set sr->len to be the smallest between the actual buffer size and sqe->len. But, now that we disconnect the buffer from the submission request, we can get in a situation where the buffers and requests mismatch, and only part of a buffer gets sent. Assume:

* buf[1]->len = 128; buf[2]->len = 256
* sqe[1]->len = 128; sqe[2]->len = 256

If sqe1 runs first, it picks buff[1] and it's all good. But, if sqe[2] runs first, sqe[1] picks buff[2], and the last half of buff[2] is never sent.

While arguably the use-case of different-length sends is questionable, it has already raised confusion with potential users of this feature. Let's make the interface less tricky by forcing the length to only come from the buffer ring entry itself.

Fixes ac5f71a3d9d7 ("io_uring/net: add provided buffer support for IORING_OP_SEND")
---
 io_uring/net.c | 2 ++
 1 file changed, 2 insertions(+)
io_uring/net.c +0 -0
index 51c41d771c50..ffe37dd77a74 100644

io_uring/net.c int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) · line 430

Quoted 8 lines
io_uring/net.c +2 -0
@@ -423,6 +423,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (req->flags & REQ_F_BUFFER_SELECT && sr->len)
+		return -EINVAL;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)

Why not put it in io_send(), under io_do_buffer_select()? Then you can get rid of the:

.max_len = min_not_zero(sr->len, INT_MAX),

and just do

.max_len = INT_MAX,

instead as well.

Signature · 1 line
Jens Axboe
#8 lore raw

Re: [PATCH] io_uring: Require zeroed sqe->len on provided-buffers send

Quoted 42 lines · Jens Axboe
Quoted 29 lines · Gabriel Krisman Bertazi

When sending from a provided buffer, we set sr->len to be the smallest between the actual buffer size and sqe->len. But, now that we disconnect the buffer from the submission request, we can get in a situation where the buffers and requests mismatch, and only part of a buffer gets sent. Assume:

* buf[1]->len = 128; buf[2]->len = 256
* sqe[1]->len = 128; sqe[2]->len = 256

If sqe1 runs first, it picks buff[1] and it's all good. But, if sqe[2] runs first, sqe[1] picks buff[2], and the last half of buff[2] is never sent.

While arguably the use-case of different-length sends is questionable, it has already raised confusion with potential users of this feature. Let's make the interface less tricky by forcing the length to only come from the buffer ring entry itself.

Fixes ac5f71a3d9d7 ("io_uring/net: add provided buffer support for IORING_OP_SEND")
---
 io_uring/net.c | 2 ++
 1 file changed, 2 insertions(+)
io_uring/net.c +0 -0
index 51c41d771c50..ffe37dd77a74 100644

io_uring/net.c int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) · line 430

Quoted 8 lines
io_uring/net.c +2 -0
@@ -423,6 +423,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (req->flags & REQ_F_BUFFER_SELECT && sr->len)
+		return -EINVAL;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)

Why not put it in io_send(), under io_do_buffer_select()? Then you can get rid of the:

Quoted 6 lines

.max_len = min_not_zero(sr->len, INT_MAX),

and just do

.max_len = INT_MAX,

Mostly because I'd expect this kind of validation of userspace data to be done early in ->prep, when we are consuming the sqe. But more importantly, if I read the code correctly, doing it under io_do_buffer_select() in io_send() is more convoluted because we have that backward jump in case we don't send the full set of buffers in the bundle case, and we dirty sr->len with the actual returned buffer length.

since we already checked in prep, we can safely ignore it in the io_do_buffer_select, anyway. What do you think of the below?

-- >8 --
Subject: [PATCH] io_uring: Require zeroed sqe->len on provided-buffers send

When sending from a provided buffer, we set sr->len to be the smallest between the actual buffer size and sqe->len. But, now that we disconnect the buffer from the submission request, we can get in a situation where the buffers and requests mismatch, and only part of a buffer gets sent. Assume:

* buf[1]->len = 128; buf[2]->len = 256
* sqe[1]->len = 128; sqe[2]->len = 256

If sqe1 runs first, it picks buff[1] and it's all good. But, if sqe[2] runs first, sqe[1] picks buff[2], and the last half of buff[2] is never sent.

While arguably the use-case of different-length sends is questionable, it has already raised confusion with potential users of this feature. Let's make the interface less tricky by forcing the length to only come from the buffer ring entry itself.

Fixes ac5f71a3d9d7 ("io_uring/net: add provided buffer support for IORING_OP_SEND")

---
v2:
- Disregard sr->len when selecting buffer in io_send()

---
 io_uring/net.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
io_uring/net.c +3 -1
index 51c41d771c50..cf43053a25b7 100644
@@ -423,6 +423,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (req->flags & REQ_F_BUFFER_SELECT && sr->len)
+		return -EINVAL;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)
@@ -586,7 +588,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	if (io_do_buffer_select(req)) {
 		struct buf_sel_arg arg = {
 			.iovs = &kmsg->fast_iov,
-			.max_len = min_not_zero(sr->len, INT_MAX),
+			.max_len = INT_MAX,
 			.nr_iovs = 1,
 			.mode = KBUF_MODE_EXPAND,
 		};
Signature · 1 line
2.44.0
author maintainer
#9 lore raw

Re: [PATCH] io_uring: Require zeroed sqe->len on provided-buffers send

Quoted 58 lines · Gabriel Krisman Bertazi
Quoted 42 lines · Jens Axboe
Quoted 29 lines · Gabriel Krisman Bertazi

When sending from a provided buffer, we set sr->len to be the smallest between the actual buffer size and sqe->len. But, now that we disconnect the buffer from the submission request, we can get in a situation where the buffers and requests mismatch, and only part of a buffer gets sent. Assume:

* buf[1]->len = 128; buf[2]->len = 256
* sqe[1]->len = 128; sqe[2]->len = 256

If sqe1 runs first, it picks buff[1] and it's all good. But, if sqe[2] runs first, sqe[1] picks buff[2], and the last half of buff[2] is never sent.

While arguably the use-case of different-length sends is questionable, it has already raised confusion with potential users of this feature. Let's make the interface less tricky by forcing the length to only come from the buffer ring entry itself.

Fixes ac5f71a3d9d7 ("io_uring/net: add provided buffer support for IORING_OP_SEND")
---
 io_uring/net.c | 2 ++
 1 file changed, 2 insertions(+)
io_uring/net.c +0 -0
index 51c41d771c50..ffe37dd77a74 100644

io_uring/net.c int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) · line 430

Quoted 8 lines
io_uring/net.c +2 -0
@@ -423,6 +423,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (req->flags & REQ_F_BUFFER_SELECT && sr->len)
+		return -EINVAL;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)

Why not put it in io_send(), under io_do_buffer_select()? Then you can get rid of the:

Quoted 6 lines

.max_len = min_not_zero(sr->len, INT_MAX),

and just do

.max_len = INT_MAX,

Mostly because I'd expect this kind of validation of userspace data to be done early in ->prep, when we are consuming the sqe. But more importantly, if I read the code correctly, doing it under io_do_buffer_select() in io_send() is more convoluted because we have that backward jump in case we don't send the full set of buffers in the bundle case, and we dirty sr->len with the actual returned buffer length.

Quoted 2 lines

since we already checked in prep, we can safely ignore it in the io_do_buffer_select, anyway. What do you think of the below?

Yep, I think that looks very reasonable. I'll queue it up, thanks!

Signature · 1 line
Jens Axboe