Support for provided buffers for send
Jens Axboe
maintainercover Support for provided buffers for send
Hi,
We never supported provided buffers for sends, because it didn't seem to make a lot of sense. But it actually does make a lot of sense! If an app is receiving data, doing something with it, and then sending either the same or another buffer out based on that, then if we use provided buffers for sends we can guarantee that the sends are serialized. This is because provided buffer rings are FIFO ordered, as it's a ring buffer, and hence it doesn't really matter if you have more than one send inflight.
This provides a nice efficiency win, but more importantly, it reduces the complexity in the application as it no longer needs to track a potential backlog of sends. The app just sets up a send based buffer ring, exactly like it does for incoming data. And that's it, no more dealing with serialized sends.
In some testing with proxy [1], in basic shuffling of packets I see a 36% improvement with this over manually dealing with sends. That's a pretty big win on top of making the app simpler.
This also opens the door for multishot send requests, which is an interesting future venue to pursue!
You can also find the patches here:
https://git.kernel.dk/cgit/linux/log/?h=io_uring-send-queue
[1] https://git.kernel.dk/cgit/liburing/tree/examples/proxy.c
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/2 io_uring/net: add provided buffer support for IORING_OP_SEND
It's pretty trivial to wire up provided buffer support for the send side, just like we do on the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring.
One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine:
1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, completes successfully, posts CQE
5) sendB is issued, completes successfully, posts CQE
All is fine. Requests are always issued in-order, and both complete inline as most sends do.
However, if we're flooding socket1 with sends, the following could also result from the same sequence:
1) App queues sendA for socket1
2) App queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, completes successfully, posts CQE
7) sendA is retried, completes successfully, posts CQE
Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead:
1) App queues dataA for sendA, queues sendA for socket1
2) App queues dataB for sendB queues sendB for socket1
3) App does io_uring_submit()
4) sendA is issued, socket1 is full, poll is armed for retry
5) Space frees up in socket1, this triggers sendA retry via task_work
6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA")
7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB")
Now we've sent the data in order, and everybody is happy.
It's worth noting that this also opens the door for supporting multishot sends, as provided buffers would be a prerequisite for that. Those can trigger either when new buffers are added to the outgoing ring, or (if stalled due to lack of space) when space frees up in the socket.
--- include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 3 ++- io_uring/net.c | 19 ++++++++++++++++--- io_uring/opdef.c | 1 + 4 files changed, 20 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +1 -0
@@ -522,6 +522,7 @@ struct io_uring_params {#define IORING_FEAT_CQE_SKIP (1U << 11)#define IORING_FEAT_LINKED_FILE (1U << 12)#define IORING_FEAT_REG_REG_RING (1U << 13) +#define IORING_FEAT_SEND_BUFS (1U << 14)/* * io_uring_register(2) opcodes and arguments
io_uring/io_uring.c +2 -1
@@ -3962,7 +3962,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,IORING_FEAT_POLL_32BITS |IORING_FEAT_SQPOLL_NONFIXED |IORING_FEAT_EXT_ARG |IORING_FEAT_NATIVE_WORKERS |IORING_FEAT_RSRC_TAGS |IORING_FEAT_CQE_SKIP | -IORING_FEAT_LINKED_FILE |IORING_FEAT_REG_REG_RING ; +IORING_FEAT_LINKED_FILE |IORING_FEAT_REG_REG_RING | +IORING_FEAT_SEND_BUFS ;if (copy_to_user (params, p,sizeof (*p))) { ret = -EFAULT ;
io_uring/net.c +16 -3
@@ -365,8 +365,10 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) {struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); -struct msghdr msg; +size_t len = sr->len; +unsigned int cflags;struct socket *sock; +struct msghdr msg;unsigned flags;int min_ret =0 ;int ret;
@@ -399,7 +401,17 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)if (unlikely (!sock))return -ENOTSOCK ; - ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, &msg.msg_iter); +if (io_do_buffer_select (req)) { +void __user *buf; + + buf =io_buffer_select (req, &len, issue_flags); +if (!buf) +return -ENOBUFS ; + sr->buf = buf; + sr->len = len; + } + + ret =import_ubuf (ITER_SOURCE , sr->buf, len, &msg.msg_iter);if (unlikely (ret))return ret;
@@ -431,7 +443,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; -io_req_set_res (req, ret,0 ); + cflags =io_put_kbuf (req, issue_flags); +io_req_set_res (req, ret, cflags);return IOU_OK ; }
io_uring/opdef.c +1 -0
@@ -273,6 +273,7 @@ const struct io_issue_def io_issue_defs[] = { .audit_skip = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1,#if defined(CONFIG_NET ) .prep = io_sendmsg_prep, .issue = io_send,
Signature · 1 line
2.43.0
Jens Axboe
maintainerPATCH 2/2 io_uring/net: add provided buffer support for IORING_OP_SENDMSG
In reply to Jens AxboeAdds provided buffer support for sendmsg as well, see the previous commit that added it to IORING_OP_SEND for a longer explanation of why this makes sense.
--- io_uring/net.c | 15 ++++++++++++++- io_uring/opdef.c | 1 + 2 files changed, 15 insertions(+), 1 deletion(-)
io_uring/net.c +14 -1
@@ -304,6 +304,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);struct io_async_msghdr iomsg, *kmsg;struct socket *sock; +unsigned int cflags;unsigned flags;int min_ret =0 ;int ret;
@@ -326,6 +327,17 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST))return io_setup_async_msg (req, kmsg, issue_flags); +if (io_do_buffer_select (req)) { +void __user *buf; +size_t len =0 ; + + buf =io_buffer_select (req, &len, issue_flags); +if (!buf) +return -ENOBUFS ; + +iov_iter_ubuf (&kmsg->msg.msg_iter,ITER_SOURCE , buf, len); + } + flags = sr->msg_flags;if (issue_flags &IO_URING_F_NONBLOCK ) flags |=MSG_DONTWAIT ;
@@ -357,7 +369,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; -io_req_set_res (req, ret,0 ); + cflags =io_put_kbuf (req, issue_flags); +io_req_set_res (req, ret, cflags);return IOU_OK ; }
io_uring/opdef.c +1 -0
@@ -139,6 +139,7 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1,#if defined(CONFIG_NET ) .prep = io_sendmsg_prep, .issue = io_sendmsg,
Signature · 1 line
2.43.0