Add support for provided registered buffers
Jens Axboe
maintainercover Add support for provided registered buffers
Hi,
Normally a request can take a provided buffer, which means "pick a buffer from group X and do IO to/from it", or it can use a registered buffer, which means "use the buffer at index Y and do IO to/from it". For things like O_DIRECT and network zero copy, registered buffers can be used to speedup the operation, as they avoid repeated get_user_pages() and page referencing calls for each IO operation.
Normal (non zero copy) send supports bundles, which is a way to pick multiple provided buffers at once and send them. send zero copy only supports registered buffers, and hence can only send a single buffer at the time.
This patchset adds support for using a mix of provided and registered buffers, where the provided buffers merely provide an index into which registered buffers to use. This enables using provided buffers for send zc in general, but also bundles where multiple buffers are picked. This is done by changing how the provided buffers are intepreted. Normally a provided buffer has an address, length, and buffer ID associated with it. The address tells the kernel where the IO should occur. If both fixed and provided buffers are asked for, the provided buffer address field is instead an encoding of the registered buffer index and the offset within that buffer. With that in place, using a combination of the two can work.
Patches 1-4 are just cleanup and prep, patch 5 adds the basic definition of what a fixed provided buffer looks like, patch 6 adds support for kbuf to map into a bvec directly, and finally patch 7 adds support for send zero copy to use this mechanism.
More details available in the actual patches. Tested with kperf using zero copy RX and TX, easily reaching 100G speeds with a single thread doing 4k sends and receives.
Kernel tree here:
https://git.kernel.dk/cgit/linux/log/?h=io_uring-sendzc-provided
include/uapi/linux/io_uring.h | 8 ++ io_uring/kbuf.c | 180 +++++++++++++++++++++++++++---- io_uring/kbuf.h | 9 +- io_uring/net.c | 192 ++++++++++++++++++++++------------ io_uring/net.h | 10 +- io_uring/opdef.c | 1 + 6 files changed, 309 insertions(+), 91 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/7 io_uring/kbuf: mark buf_sel_arg mode as KBUF_MODE_FREE once allocated
If kbuf expands the iovec array, then it doesn't matter if the caller didn't set KBUF_MODE_FREE or not, as once allocated it should be marked as such. Ensure that this is the case.
--- io_uring/kbuf.c | 1 + 1 file changed, 1 insertion(+)
io_uring/kbuf.c +1 -0
@@ -245,6 +245,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,kfree (arg->iovs); arg->iovs = iov; nr_iovs = nr_avail; + arg->mode |=KBUF_MODE_FREE ; }else if (nr_avail < nr_iovs) { nr_iovs = nr_avail; }
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 2/7 io_uring/kbuf: change io_provided_buffers_select() calling convention
In reply to Jens AxboeMake it more like the ring provided buffers in what arguments it takes, and use similar ordering as well. This makes the code more consistent
--- io_uring/kbuf.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
io_uring/kbuf.c +6 -5
@@ -117,10 +117,11 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len,return NULL ; } -static int io_provided_buffers_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -struct iovec *iov) +static int io_provided_buffers_select (struct io_kiocb *req, +struct buf_sel_arg *arg, +struct io_buffer_list *bl,size_t *len) { +struct iovec *iov = arg->iovs;void __user *buf; buf =io_provided_buffer_select (req, len, bl);
@@ -311,7 +312,7 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg,io_kbuf_commit (req, bl, arg->out_len, ret); } }else { - ret =io_provided_buffers_select (req, &arg->out_len, bl, arg->iovs); + ret =io_provided_buffers_select (req, arg, bl, &arg->out_len); } out_unlock:io_ring_submit_unlock (ctx, issue_flags);
@@ -338,7 +339,7 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) }/* don't support multiple buffer selections for legacy */ -return io_provided_buffers_select (req, &arg->max_len, bl, arg->iovs); +return io_provided_buffers_select (req, arg, bl, &arg->max_len); } static int __io_remove_buffers(struct io_ring_ctx *ctx,
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 3/7 io_uring/net: abstract out io_send_import() helper
In reply to Jens AxboeIf provided buffers are used, this helper can be used to import the necessary data from a provided buffer group. Only one user so far, but add it in preparation of adding another one. While doing so, also split the actual import into an iov_iter out into a separate helper.
In preparation for needing to know the number of mapped segments, return that instead. It still returns < 0 on error.
--- io_uring/net.c | 75 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 25 deletions(-)
io_uring/net.c +50 -25
@@ -578,28 +578,33 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } -int io_send (struct io_kiocb *req,unsigned int issue_flags) +static int __io_send_import (struct io_kiocb *req,struct buf_sel_arg *arg, +int nsegs,unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);struct io_async_msghdr *kmsg = req->async_data; -struct socket *sock; -unsigned flags; -int min_ret =0 ; -int ret; +int ret = nsegs; - sock =sock_from_file (req->file); -if (unlikely (!sock)) -return -ENOTSOCK ; +if (nsegs ==1 ) { + sr->buf = arg->iovs[0 ].iov_base; + ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, + &kmsg->msg.msg_iter); +if (unlikely (ret <0 )) +return ret; + }else { +iov_iter_init (&kmsg->msg.msg_iter,ITER_SOURCE , arg->iovs, + nsegs, arg->out_len); + } -if (!(req->flags &REQ_F_POLLED ) && - (sr->flags &IORING_RECVSEND_POLL_FIRST )) -return -EAGAIN ; +return nsegs; +} - flags = sr->msg_flags; -if (issue_flags &IO_URING_F_NONBLOCK ) - flags |=MSG_DONTWAIT ; +static int io_send_import (struct io_kiocb *req,unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); +struct io_async_msghdr *kmsg = req->async_data; +int ret =1 ; -retry_bundle:if (io_do_buffer_select (req)) {struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov,
@@ -629,18 +634,38 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) } sr->len = arg.out_len; -if (ret ==1 ) { - sr->buf = arg.iovs[0 ].iov_base; - ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, - &kmsg->msg.msg_iter); -if (unlikely (ret)) -return ret; - }else { -iov_iter_init (&kmsg->msg.msg_iter,ITER_SOURCE , - arg.iovs, ret, arg.out_len); - } +return __io_send_import (req, &arg, ret, issue_flags); } +return ret; +} + +int io_send (struct io_kiocb *req,unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); +struct io_async_msghdr *kmsg = req->async_data; +struct socket *sock; +unsigned flags; +int min_ret =0 ; +int ret; + + sock =sock_from_file (req->file); +if (unlikely (!sock)) +return -ENOTSOCK ; + +if (!(req->flags &REQ_F_POLLED ) && + (sr->flags &IORING_RECVSEND_POLL_FIRST )) +return -EAGAIN ; + + flags = sr->msg_flags; +if (issue_flags &IO_URING_F_NONBLOCK ) + flags |=MSG_DONTWAIT ; + +retry_bundle: + ret =io_send_import (req, issue_flags); +if (unlikely (ret <0 )) +return ret; +/* * If MSG_WAITALL is set, or this is a bundle send, then we need * the full amount. If just bundle is set, if we do a short send
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 4/7 io_uring/net: move send zc fixed buffer import into helper
In reply to Jens AxboeIn preparation to making the fixed buffer importing a bit more elaborate in terms of what it supports.
--- io_uring/net.c | 77 ++++++++++++++++++++++++++++---------------------- 1 file changed, 44 insertions(+), 33 deletions(-)
io_uring/net.c +44 -33
@@ -81,6 +81,9 @@ struct io_sr_msg {struct io_kiocb *notif; }; +static int io_sg_from_iter (struct sk_buff *skb,struct iov_iter *from, +size_t length); +/* * Number of times we'll try and do receives if there's more data. If we * exceed this limit, then add us to the back of the queue and retry from
@@ -578,6 +581,37 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } +static int io_send_zc_import_single (struct io_kiocb *req, +unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); +struct io_async_msghdr *kmsg = req->async_data; +struct io_ring_ctx *ctx = req->ctx; +struct io_mapped_ubuf *imu; +int ret; +u16 idx; + + ret = -EFAULT ; +io_ring_submit_lock (ctx, issue_flags); +if (sr->buf_index < ctx->nr_user_bufs) { + idx =array_index_nospec (sr->buf_index, ctx->nr_user_bufs); + imu =READ_ONCE (ctx->user_bufs[idx]); +io_req_set_rsrc_node (sr->notif, ctx); + ret =0 ; + } +io_ring_submit_unlock (ctx, issue_flags); + +if (unlikely (ret)) +return ret; + + ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, imu, + (u64 )(uintptr_t )sr->buf, sr->len); +if (unlikely (ret)) +return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; +return 0 ; +} +static int __io_send_import (struct io_kiocb *req,struct buf_sel_arg *arg,int nsegs,unsigned int issue_flags) {
@@ -1365,40 +1399,17 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags)struct io_async_msghdr *kmsg = req->async_data;int ret; -if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { -struct io_ring_ctx *ctx = req->ctx; -struct io_mapped_ubuf *imu; -int idx; - - ret = -EFAULT ; -io_ring_submit_lock (ctx, issue_flags); -if (sr->buf_index < ctx->nr_user_bufs) { - idx =array_index_nospec (sr->buf_index, ctx->nr_user_bufs); - imu =READ_ONCE (ctx->user_bufs[idx]); -io_req_set_rsrc_node (sr->notif, ctx); - ret =0 ; - } -io_ring_submit_unlock (ctx, issue_flags); +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) +return io_send_zc_import_single (req, issue_flags); -if (unlikely (ret)) -return ret; - - ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, imu, - (u64 )(uintptr_t )sr->buf, sr->len); -if (unlikely (ret)) -return ret; - kmsg->msg.sg_from_iter = io_sg_from_iter; - }else { - ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, &kmsg->msg.msg_iter); -if (unlikely (ret)) -return ret; - ret =io_notif_account_mem (sr->notif, sr->len); -if (unlikely (ret)) -return ret; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; - } - -return ret; + ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, &kmsg->msg.msg_iter); +if (unlikely (ret)) +return ret; + ret =io_notif_account_mem (sr->notif, sr->len); +if (unlikely (ret)) +return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; +return 0 ; }int io_send_zc (struct io_kiocb *req,unsigned int issue_flags)
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 5/7 io_uring: add ability for provided buffer to index registered buffers
In reply to Jens AxboeThis just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf {
__u64 addr;
__u32 len;
__u16 bid;
__u16 resv;
};where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a get_user_pages() and page(s) inc+dec, and are used for things like O_DIRECT on storage and zero copy send.
Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of buf->addr changes. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr = ((__u64) regbuf_offset << IOU_BUF_OFFSET_BITS) | regbuf_index; buf->len = len;
and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)
include/uapi/linux/io_uring.h +8 -0
@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL) +/* * Flags for IORING_REGISTER_PBUF_RING. *
Signature · 1 line
2.45.2
Re: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 50 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +0
-0
include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 742
Quoted 10 lines
include/uapi/linux/io_uring.h +7
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)
32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 55 lines · Pavel Begunkov
Quoted 54 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +1
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the
Quoted 8 lines
include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 742
Quoted 5 lines
include/uapi/linux/io_uring.h +5
-0+* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
For linux, the max read/write value has always been INT_MAX & PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.
Signature · 1 line
Jens Axboe
Re: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 65 lines · Jens Axboe
Quoted 55 lines · Pavel Begunkov
Quoted 54 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +1
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the Quoted 8 lines
include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 742
Quoted 5 lines
include/uapi/linux/io_uring.h +5
-0+* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Quoted 4 lines
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
Right, and I don't think it's unreasonable considering with how much memory systems have nowadays, and we think that one large registered buffer is a good thing.
Quoted 6 lines
For linux, the max read/write value has always been
INT_MAX&PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.
That's why I mentioned IO size, you can register a very large buffer and do IO with a small subchunk of it, even if that "small" is 4G, but it still needs to be addressed. I think we need at least an order of magnitude or two more space for it to last for a bit.
Can it steal bits from IOU_BUF_REGBUF_BITS?
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 66 lines · Pavel Begunkov
Quoted 65 lines · Jens Axboe
Quoted 63 lines · Pavel Begunkov
Quoted 50 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +0
-0include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 741
Quoted 9 lines
include/uapi/linux/io_uring.h +6
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Quoted 8 lines
Quoted 4 lines
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
Right, and I don't think it's unreasonable considering with how much memory systems have nowadays, and we think that one large registered buffer is a good thing.
Agree - but at the same time, not a big hardship to chunk up the region into 8G chunks rather than allow, eg, a 64G region. Would be nice if it wasn't a requirement, but unsure how to make that work otherwise.
And not a lot of complaints on having 1G be the size, even from the varnish side where they register hundreds of gigs of memory.
Quoted 7 lines
Quoted 6 lines
For linux, the max read/write value has always been
INT_MAX&PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.
Quoted 6 lines
That's why I mentioned IO size, you can register a very large buffer and do IO with a small subchunk of it, even if that "small" is 4G, but it still needs to be addressed. I think we need at least an order of magnitude or two more space for it to last for a bit.
Can it steal bits from
IOU_BUF_REGBUF_BITS?
As mentooned, we can definitely move the one bit, which would bring is to 31/33, and ~2GB IO size (max in linux) and ~8G of offset. That can be done without having any tradeoffs. Beyond that, and we're starting to limit the transfer size, eg it's tradeoff of allowing more offset (and hence bigger registered regions) vs transfer size. You could probably argue that 1G would be fine, and hence make it 30/34, which would allow 16GB registered buffers. Just unsure if it's worth it, as neither of those would allow really huge registered buffers - and does it matter if your buffer registrations are chunked at 8G vs 16G? Probably not.
Signature · 1 line
Jens Axboe
Re: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 97 lines · Jens Axboe
Quoted 66 lines · Pavel Begunkov
Quoted 65 lines · Jens Axboe
Quoted 63 lines · Pavel Begunkov
Quoted 50 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +0
-0include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 741
Quoted 9 lines
include/uapi/linux/io_uring.h +6
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Quoted 8 lines
Quoted 4 lines
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
Right, and I don't think it's unreasonable considering with how much memory systems have nowadays, and we think that one large registered buffer is a good thing.
Agree - but at the same time, not a big hardship to chunk up the region into 8G chunks rather than allow, eg, a 64G region. Would be nice if it wasn't a requirement, but unsure how to make that work otherwise.
And not a lot of complaints on having 1G be the size, even from the varnish side where they register hundreds of gigs of memory.
Quoted 7 lines
Quoted 6 lines
For linux, the max read/write value has always been
INT_MAX&PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.Quoted 6 lines
That's why I mentioned IO size, you can register a very large buffer and do IO with a small subchunk of it, even if that "small" is 4G, but it still needs to be addressed. I think we need at least an order of magnitude or two more space for it to last for a bit.
Can it steal bits from
IOU_BUF_REGBUF_BITS?
Quoted 9 lines
As mentooned, we can definitely move the one bit, which would bring is to 31/33, and ~2GB IO size (max in linux) and ~8G of offset. That can be done without having any tradeoffs. Beyond that, and we're starting to limit the transfer size, eg it's tradeoff of allowing more offset (and hence bigger registered regions) vs transfer size. You could probably argue that 1G would be fine, and hence make it 30/34, which would allow 16GB registered buffers. Just unsure if it's worth it, as neither of those would allow really huge registered buffers - and does it matter if your buffer registrations are chunked at 8G vs 16G? Probably not.
6/7 packs offset and the reg buffer index into the same u64, not the len. I'm don't see how it affects the len
idx = addr & ((1ULL << IOU_BUF_REGBUF_BITS) - 1); addr >>= IOU_BUF_REGBUF_BITS;
*offset = addr & ((1ULL << IOU_BUF_OFFSET_BITS) - 1);
So the tradeoff is with the max size of the registered buffer table. I doubt you need 2^32, if each is at least 4KB, it's at least 16TB.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 111 lines · Pavel Begunkov
Quoted 97 lines · Jens Axboe
Quoted 66 lines · Pavel Begunkov
Quoted 65 lines · Jens Axboe
Quoted 63 lines · Pavel Begunkov
Quoted 50 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +0
-0include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 741
Quoted 9 lines
include/uapi/linux/io_uring.h +6
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Quoted 8 lines
Quoted 4 lines
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
Right, and I don't think it's unreasonable considering with how much memory systems have nowadays, and we think that one large registered buffer is a good thing.
Agree - but at the same time, not a big hardship to chunk up the region into 8G chunks rather than allow, eg, a 64G region. Would be nice if it wasn't a requirement, but unsure how to make that work otherwise.
And not a lot of complaints on having 1G be the size, even from the varnish side where they register hundreds of gigs of memory.
Quoted 7 lines
Quoted 6 lines
For linux, the max read/write value has always been
INT_MAX&PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.Quoted 6 lines
That's why I mentioned IO size, you can register a very large buffer and do IO with a small subchunk of it, even if that "small" is 4G, but it still needs to be addressed. I think we need at least an order of magnitude or two more space for it to last for a bit.
Can it steal bits from
IOU_BUF_REGBUF_BITS?Quoted 9 lines
As mentooned, we can definitely move the one bit, which would bring is to 31/33, and ~2GB IO size (max in linux) and ~8G of offset. That can be done without having any tradeoffs. Beyond that, and we're starting to limit the transfer size, eg it's tradeoff of allowing more offset (and hence bigger registered regions) vs transfer size. You could probably argue that 1G would be fine, and hence make it 30/34, which would allow 16GB registered buffers. Just unsure if it's worth it, as neither of those would allow really huge registered buffers - and does it matter if your buffer registrations are chunked at 8G vs 16G? Probably not.
6/7 packs offset and the reg buffer index into the same u64, not the len. I'm don't see how it affects the len
Quoted 7 lines
idx = addr & ((1ULL <<
IOU_BUF_REGBUF_BITS) - 1); addr >>=IOU_BUF_REGBUF_BITS;
*offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS) - 1);So the tradeoff is with the max size of the registered buffer table. I doubt you need 2^32, if each is at least 4KB, it's at least 16TB.
Ah yes, nevermind, the len is of course separate. Yes indeed that falls out nicely then, we can just reserve eg 16 bits for registered buffers. And that leaves 48 bits for the offset, which would hold more than enough. Even at that and 8G max buffer size, that'd be half a TB of buffer space. And there's no reason we can't just increase the registered buffer size on 64-bit to much beyond that. Maybe play it safe and set aside 20 bits for the buffer index, and that leaves 44 bits for the registered buffer space? Or 16TB of registered buffer space.
What do you think?
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH 5/7] io_uring: add ability for provided buffer to index registered buffers
Quoted 129 lines · Jens Axboe
Quoted 111 lines · Pavel Begunkov
Quoted 97 lines · Jens Axboe
Quoted 66 lines · Pavel Begunkov
Quoted 65 lines · Jens Axboe
Quoted 63 lines · Pavel Begunkov
Quoted 50 lines · Jens Axboe
This just adds the necessary shifts that define what a provided buffer that is merely an index into a registered buffer looks like. A provided buffer looks like the following:
struct io_uring_buf { __u64 addr; __u32 len; __u16 bid; __u16 resv; };where 'addr' holds a userspace address, 'len' is the length of the buffer, and 'bid' is the buffer ID identifying the buffer. This works fine for a virtual address, but it cannot be used efficiently denote a registered buffer. Registered buffers are pre-mapped into the kernel for more efficient IO, avoiding a
get_user_pages()and page(s) inc+dec, and are used for things likeO_DIRECTon storage and zero copy send.Particularly for the send case, it'd be useful to support a mix of provided and registered buffers. This enables the use of using a provided ring buffer to serialize sends, and also enables the use of send bundles, where a send can pick multiple buffers and send them all at once.
If provided buffers are used as an index into registered buffers, the meaning of
buf->addrchanges. If registered buffer index 'regbuf_index' is desired, with a length of 'len' and the offset 'regbuf_offset' from the start of the buffer, then the application would fill out the entry as follows:
buf->addr= ((__u64)regbuf_offset<<IOU_BUF_OFFSET_BITS) |regbuf_index;buf->len= len;and otherwise add it to the buffer ring as usual. The kernel will then first pick a buffer from the desired buffer group ID, and then decode which registered buffer to use for the transfer.
This provides a way to use both registered and provided buffers at the same time.
Signed-off-by Jens Axboe <[email protected]>--- include/uapi/linux/io_uring.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/uapi/linux/io_uring.h +0
-0include/uapi/linux/io_uring.h struct io_uring_buf_ring { · line 741
Quoted 9 lines
include/uapi/linux/io_uring.h +6
-0@@ -733,6 +733,14 @@ struct io_uring_buf_ring { }; }; +/* +* When provided buffers are used as indices into registered buffers, the +* lower IOU_BUF_REGBUF_BITS indicate the index into the registered buffers, +* and the upper IOU_BUF_OFFSET_BITS indicate the offset into that buffer. +*/ +#define IOU_BUF_REGBUF_BITS (32ULL) +#define IOU_BUF_OFFSET_BITS (32ULL)32 bit is fine for IO size but not enough to store offsets, it can only address under 4GB registered buffers.
Quoted 8 lines
Quoted 4 lines
I did think about that - at least as it stands, registered buffers are limited to 1GB in size. That's how it's been since that got added. Now, for the future, we may obviously lift that limitation, and yeah then 32-bits would not necessarily be enough for the offset.
Right, and I don't think it's unreasonable considering with how much memory systems have nowadays, and we think that one large registered buffer is a good thing.
Agree - but at the same time, not a big hardship to chunk up the region into 8G chunks rather than allow, eg, a 64G region. Would be nice if it wasn't a requirement, but unsure how to make that work otherwise.
And not a lot of complaints on having 1G be the size, even from the varnish side where they register hundreds of gigs of memory.
Quoted 7 lines
Quoted 6 lines
For linux, the max read/write value has always been
INT_MAX&PAGE_MASK, so we could make do with 31 bits for the size, which would bump the offset to 33-bits, or 8G. That'd leave enough room for, at least, 8G buffers, or 8x what we support now. Which is probably fine, you'd just split your buffer registrations into 8G chunks, if you want to register more than 8G of memory.Quoted 6 lines
That's why I mentioned IO size, you can register a very large buffer and do IO with a small subchunk of it, even if that "small" is 4G, but it still needs to be addressed. I think we need at least an order of magnitude or two more space for it to last for a bit.
Can it steal bits from
IOU_BUF_REGBUF_BITS?Quoted 9 lines
As mentooned, we can definitely move the one bit, which would bring is to 31/33, and ~2GB IO size (max in linux) and ~8G of offset. That can be done without having any tradeoffs. Beyond that, and we're starting to limit the transfer size, eg it's tradeoff of allowing more offset (and hence bigger registered regions) vs transfer size. You could probably argue that 1G would be fine, and hence make it 30/34, which would allow 16GB registered buffers. Just unsure if it's worth it, as neither of those would allow really huge registered buffers - and does it matter if your buffer registrations are chunked at 8G vs 16G? Probably not.
6/7 packs offset and the reg buffer index into the same u64, not the len. I'm don't see how it affects the len
Quoted 7 lines
idx = addr & ((1ULL <<
IOU_BUF_REGBUF_BITS) - 1); addr >>=IOU_BUF_REGBUF_BITS;
*offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS) - 1);So the tradeoff is with the max size of the registered buffer table. I doubt you need 2^32, if each is at least 4KB, it's at least 16TB.
Ah yes, nevermind, the len is of course separate. Yes indeed that falls out nicely then, we can just reserve eg 16 bits for registered buffers. And that leaves 48 bits for the offset, which would hold more than enough. Even at that and 8G max buffer size, that'd be half a TB of buffer space. And there's no reason we can't just increase the registered buffer size on 64-bit to much beyond that. Maybe play it safe and set aside 20 bits for the buffer index, and that leaves 44 bits for the registered buffer space? Or 16TB of registered buffer space.
Quoted 1 line
What do you think?
I changed it to 20/44 as that seems to be the best split. It allows much larger registered buffers, and 1M of them. That should be plenty.
Might even make sense to just up the registered buffer size at this point too as a separate change, there's no reason for it to be limited to 1G. We can keep 1G on 32-bit as it could only go to to 4G-1 on that anyway. But for 64-bit, we can bump it higher.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 6/7 io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC
In reply to Jens AxboeThe provided buffer helpers always map to iovecs. Add a new mode, KBUF_MODE_BVEC, which instead maps it to a bio_vec array instead. For use with zero-copy scenarios, where the caller would want to turn it into a bio_vec anyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into a bio_vec array.
Since it's now managing both iovecs and bvecs, change the naming of buf_sel_arg->nr_iovs member to nr_vecs instead.
--- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-)
io_uring/kbuf.c +153 -17
@@ -16,6 +16,7 @@#include "opdef.h" #include "kbuf.h" #include "memmap.h" +#include "rsrc.h" /* BIDs are addressed by a 16-bit field in a CQE */ #define MAX_BIDS_PER_BGID (1 << 16)
@@ -117,20 +118,135 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len,return NULL ; } +static struct io_mapped_ubuf *io_ubuf_from_buf (struct io_ring_ctx *ctx, +u64 addr,unsigned int *offset) +{ +struct io_mapped_ubuf *imu; +u16 idx; + +/* +* Get registered buffer index and offset, encoded into the +* addr base value. +*/ + idx = addr & ((1ULL <<IOU_BUF_REGBUF_BITS ) -1 ); + addr >>=IOU_BUF_REGBUF_BITS ; + *offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS ) -1 ); + +if (unlikely (idx >= ctx->nr_user_bufs)) +return ERR_PTR (-EFAULT ); + + idx =array_index_nospec (idx, ctx->nr_user_bufs); + imu =READ_ONCE (ctx->user_bufs[idx]); +if (unlikely (*offset >= imu->len)) +return ERR_PTR (-EFAULT ); + +return imu; +} + +static bool io_expand_bvecs (struct buf_sel_arg *arg) +{ +int nvecs = arg->nr_vecs +8 ; +struct bio_vec *bv; + +if (!(arg->mode &KBUF_MODE_EXPAND )) +return false; + + bv =kmalloc_array (nvecs,sizeof (struct bio_vec ),GFP_KERNEL ); +if (unlikely (!bv)) +return false; +memcpy (bv, arg->bvecs, arg->nr_vecs *sizeof (*bv)); +if (arg->mode &KBUF_MODE_FREE ) +kfree (arg->bvecs); + arg->bvecs = bv; + arg->nr_vecs = nvecs; + arg->mode |=KBUF_MODE_FREE ; +return true; +} + +static int io_fill_bvecs (struct io_ring_ctx *ctx,u64 addr, +struct buf_sel_arg *arg,unsigned int len, +int *vec_off) +{ +struct bio_vec *src, *src_prv =NULL ; +struct io_mapped_ubuf *imu; +unsigned int llen = len; +unsigned int offset; + + imu =io_ubuf_from_buf (ctx, addr, &offset); +if (unlikely (IS_ERR (imu))) +return PTR_ERR (imu); + +if (unlikely (offset >= imu->len || len > imu->len)) +return -EOVERFLOW ; +if (unlikely (offset > imu->len - len)) +return -EOVERFLOW ; + + src = imu->bvec; +if (offset > src->bv_len) { +unsigned long seg_skip; + + offset -= src->bv_len; + seg_skip =1 + (offset >> imu->folio_shift); + offset &= ((1UL << imu->folio_shift) -1 ); + src += seg_skip; + } + +do { +unsigned int this_len = len; + +if (this_len + offset > src->bv_len) + this_len = src->bv_len - offset; + +/* +* If contig with previous bio_vec, merge it to minimize the +* number of segments needed. If not, then add a new segment, +* expanding the number of available slots, if needed. +*/ +if (src_prv && +page_folio (src_prv->bv_page) ==page_folio (src->bv_page) && + src_prv->bv_page +1 == src->bv_page) { + arg->bvecs[*vec_off -1 ].bv_len += this_len; + }else { +struct bio_vec *dst; + +if (*vec_off == arg->nr_vecs && !io_expand_bvecs (arg)) +break ; + + dst = &arg->bvecs[*vec_off]; + dst->bv_page = src->bv_page; + dst->bv_len = this_len; + dst->bv_offset = offset; + (*vec_off)++; + } + offset =0 ; + len -= this_len; + src_prv = src++; + }while (len); + +return llen - len; +} +static int io_provided_buffers_select (struct io_kiocb *req,struct buf_sel_arg *arg,struct io_buffer_list *bl,size_t *len) { -struct iovec *iov = arg->iovs;void __user *buf; +int ret; buf =io_provided_buffer_select (req, len, bl);if (unlikely (!buf))return -ENOBUFS ; - iov[0 ].iov_base = buf; - iov[0 ].iov_len = *len; -return 1 ; +if (arg->mode &KBUF_MODE_BVEC ) { +u64 addr = (unsigned long )(uintptr_t ) buf; + + *len =io_fill_bvecs (req->ctx, addr, arg, *len, &ret); + }else { + arg->iovs[0 ].iov_base = buf; + arg->iovs[0 ].iov_len = *len; + ret =1 ; + } +return ret; } static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len,
@@ -196,13 +312,16 @@ void __user *io_buffer_select(struct io_kiocb *req, size_t *len,#define PEEK_MAX_IMPORT 256static int io_ring_buffers_peek (struct io_kiocb *req,struct buf_sel_arg *arg, -struct io_buffer_list *bl) +struct io_buffer_list *bl,int *nbufs) {struct io_uring_buf_ring *br = bl->buf_ring;struct iovec *iov = arg->iovs; -int nr_iovs = arg->nr_iovs; +int nr_iovs = arg->nr_vecs;__u16 nr_avail, tail, head;struct io_uring_buf *buf; +int vec_off; + +BUILD_BUG_ON (sizeof (struct iovec ) >sizeof (struct bio_vec )); tail =smp_load_acquire (&br->tail); head = bl->head;
@@ -236,10 +355,12 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,/* * only alloc a bigger array if we know we have data to map, eg not -* a speculative peek operation. +* a speculative peek operation. Note that struct bio_vec and +* struct iovec are the same size, so we can use them interchangably +* here as it's just for sizing purposes. */ if (arg->mode &KBUF_MODE_EXPAND && nr_avail > nr_iovs && arg->max_len) { - iov =kmalloc_array (nr_avail,sizeof (struct iovec ),GFP_KERNEL ); + iov =kmalloc_array (nr_avail,sizeof (struct bio_vec ),GFP_KERNEL );if (unlikely (!iov))return -ENOMEM ;if (arg->mode &KBUF_MODE_FREE )
@@ -255,6 +376,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,if (!arg->max_len) arg->max_len =INT_MAX ; + vec_off =0 ; req->buf_index = buf->bid; do { u32 len = buf->len;
@@ -266,15 +388,25 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, buf->len = len; } - iov->iov_base =u64_to_user_ptr (buf->addr); - iov->iov_len = len; - iov++; +if (arg->mode &KBUF_MODE_BVEC ) { +int ret; + + ret =io_fill_bvecs (req->ctx, buf->addr, arg, len, &vec_off); +if (unlikely (ret <0 )) +return ret; + len = ret; + }else { + iov->iov_base =u64_to_user_ptr (buf->addr); + iov->iov_len = len; + iov++; + vec_off++; + } arg->out_len += len; arg->max_len -= len; + (*nbufs)++;if (!arg->max_len)break ; - buf =io_ring_head_to_buf (br, ++head, bl->mask); }while (--nr_iovs);
@@ -283,7 +415,7 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, req->flags |=REQ_F_BUFFER_RING ; req->buf_list = bl; -return iov - arg->iovs; +return vec_off; } int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg,
@@ -299,7 +431,9 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, goto out_unlock;if (bl->flags &IOBL_BUF_RING ) { - ret =io_ring_buffers_peek (req, arg, bl); +int nbufs =0 ; + + ret =io_ring_buffers_peek (req, arg, bl, &nbufs);/* * Don't recycle these buffers if we need to go through poll. * Nobody else can use them anyway, and holding on to provided
@@ -307,9 +441,9 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg,* side anyway with normal buffers. Besides, we already * committed them, they cannot be put back in the queue. */ -if (ret >0 ) { +if (nbufs) { req->flags |=REQ_F_BUFFERS_COMMIT |REQ_F_BL_NO_RECYCLE ; -io_kbuf_commit (req, bl, arg->out_len, ret); +io_kbuf_commit (req, bl, arg->out_len, nbufs); } }else { ret =io_provided_buffers_select (req, arg, bl, &arg->out_len);
@@ -332,7 +466,9 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg)return -ENOENT ;if (bl->flags &IOBL_BUF_RING ) { - ret =io_ring_buffers_peek (req, arg, bl); +int nbufs =0 ; + + ret =io_ring_buffers_peek (req, arg, bl, &nbufs);if (ret >0 ) req->flags |=REQ_F_BUFFERS_COMMIT ;return ret;
io_uring/kbuf.h +7 -2
@@ -53,13 +53,18 @@ enum {KBUF_MODE_EXPAND =1 ,/* if bigger vec allocated, free old one */ KBUF_MODE_FREE =2 , +/* turn into bio_vecs, not iovecs */ + KBUF_MODE_BVEC = 4, };struct buf_sel_arg { -struct iovec *iovs; +union { +struct iovec *iovs; +struct bio_vec *bvecs; + };size_t out_len;size_t max_len; -unsigned short nr_iovs; +unsigned short nr_vecs;unsigned short mode; };
io_uring/net.c +5 -5
@@ -643,17 +643,17 @@ static int io_send_import(struct io_kiocb *req, unsigned int issue_flags)struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov, .max_len =min_not_zero (sr->len,INT_MAX ), - .nr_iovs =1 , + .nr_vecs =1 , };if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; + arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; arg.mode =KBUF_MODE_FREE ; }if (!(sr->flags &IORING_RECVSEND_BUNDLE )) - arg.nr_iovs =1 ; + arg.nr_vecs =1 ;else arg.mode |=KBUF_MODE_EXPAND ;
@@ -1140,12 +1140,12 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg sr->flags & IORING_RECVSEND_BUNDLE) {struct buf_sel_arg arg = { .iovs = &kmsg->fast_iov, - .nr_iovs =1 , + .nr_vecs =1 , .mode =KBUF_MODE_EXPAND , };if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; + arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; arg.mode |=KBUF_MODE_FREE ; }
Signature · 1 line
2.45.2
Re: [PATCH 6/7] io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC
Quoted 21 lines · Jens Axboe
The provided buffer helpers always map to iovecs. Add a new mode,
KBUF_MODE_BVEC, which instead maps it to abio_vecarray instead. For use with zero-copy scenarios, where the caller would want to turn it into abio_vecanyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into abio_vecarray.Since it's now managing both iovecs and bvecs, change the naming of
buf_sel_arg->nr_iovsmember tonr_vecsinstead.Signed-off-by Jens Axboe <[email protected]>--- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-)io_uring/kbuf.c +0
-0
...
io_uring/kbuf.c static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, · line 133
Quoted 13 lines
io_uring/kbuf.c +13
-0+static struct io_mapped_ubuf *io_ubuf_from_buf (struct io_ring_ctx *ctx, +u64 addr,unsigned int *offset) +{ +struct io_mapped_ubuf *imu; +u16 idx; + +/* +* Get registered buffer index and offset, encoded into the +* addr base value. +*/ + idx = addr & ((1ULL <<IOU_BUF_REGBUF_BITS ) -1 ); + addr >>=IOU_BUF_REGBUF_BITS ; + *offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS ) -1 );
There are two ABI questions with that. First why not use just user addresses instead of offsets? It's more consistent with how everything else works. Surely it could've been offsets for all registered buffers ops from the beggining, but it's not.
And the second, we need to start getting rid of the global node queue, if we do, this will need to allocate an array of nodes, store an imu list or something similar, which will be just as terrible as it sounds, and then it'll need another cache, sprinking more checks and handling into the hot path and so on. That's the reason the vectored registered buffer patch supports juts one registered buffer to index per request, and I believe this one should do that as well.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 6/7] io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC
Quoted 33 lines · Pavel Begunkov
Quoted 21 lines · Jens Axboe
The provided buffer helpers always map to iovecs. Add a new mode,
KBUF_MODE_BVEC, which instead maps it to abio_vecarray instead. For use with zero-copy scenarios, where the caller would want to turn it into abio_vecanyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into abio_vecarray.Since it's now managing both iovecs and bvecs, change the naming of
buf_sel_arg->nr_iovsmember tonr_vecsinstead.Signed-off-by Jens Axboe <[email protected]>--- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-)io_uring/kbuf.c +0
-0...
Quoted 10 lines
io_uring/kbuf.c +10
-0+static struct io_mapped_ubuf *io_ubuf_from_buf (struct io_ring_ctx *ctx, +u64 addr,unsigned int *offset) +{ +struct io_mapped_ubuf *imu; +u16 idx; + +/* +* Get registered buffer index and offset, encoded into the +* addr base value. +*/
Quoted 8 lines
io_uring/kbuf.c static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, · line 133
Quoted 3 lines
io_uring/kbuf.c +3
-0+ idx = addr & ((1ULL <<IOU_BUF_REGBUF_BITS ) -1 ); + addr >>=IOU_BUF_REGBUF_BITS ; + *offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS ) -1 );There are two ABI questions with that. First why not use just user addresses instead of offsets? It's more consistent with how everything else works. Surely it could've been offsets for all registered buffers ops from the beggining, but it's not.
How would that work? You need to pass in addr + buffer index for that. The usual approach is doing that, and then 'addr' tells you the offset within the buffer, eg you can just do a subtraction to get your offset. But you can't pass in both addr + index in a provided buffer, which is why it's using buf->addr to encode index + offset for that, rather than rely on the addr for the offset too.
The alternative obviously is to just do the 'addr' and have that be both index and offset, in which case you'd need to lookup the buffer. And that's certainly a no-go.
Quoted 8 lines
And the second, we need to start getting rid of the global node queue, if we do, this will need to allocate an array of nodes, store an imu list or something similar, which will be just as terrible as it sounds, and then it'll need another cache, sprinking more checks and handling into the hot path and so on. That's the reason the vectored registered buffer patch supports juts one registered buffer to index per request, and I believe this one should do that as well.
Yeah agree, the global node queue is getting in the way of more important things too, like applications using registered files and not seeing reclaim in due time. I think that's the main issue here with the ring wide queue, and certainly something that needs sorting sooner rather than later.
Limiting this patch to just dealing with a single imu would be perfectly fine I think, the intended use case here is really large registered buffers, not little ones. And having a bundle be limited to a single buffer would be perfectly fine. I can make that change for sure.
Signature · 1 line
Jens Axboe
Re: [PATCH 6/7] io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC
Quoted 43 lines · Jens Axboe
Quoted 33 lines · Pavel Begunkov
Quoted 21 lines · Jens Axboe
The provided buffer helpers always map to iovecs. Add a new mode,
KBUF_MODE_BVEC, which instead maps it to abio_vecarray instead. For use with zero-copy scenarios, where the caller would want to turn it into abio_vecanyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into abio_vecarray.Since it's now managing both iovecs and bvecs, change the naming of
buf_sel_arg->nr_iovsmember tonr_vecsinstead.Signed-off-by Jens Axboe <[email protected]>--- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-)io_uring/kbuf.c +0
-0...
Quoted 10 lines
io_uring/kbuf.c +10
-0+static struct io_mapped_ubuf *io_ubuf_from_buf (struct io_ring_ctx *ctx, +u64 addr,unsigned int *offset) +{ +struct io_mapped_ubuf *imu; +u16 idx; + +/* +* Get registered buffer index and offset, encoded into the +* addr base value. +*/ Quoted 8 lines
io_uring/kbuf.c static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, · line 133
Quoted 3 lines
io_uring/kbuf.c +3
-0+ idx = addr & ((1ULL <<IOU_BUF_REGBUF_BITS ) -1 ); + addr >>=IOU_BUF_REGBUF_BITS ; + *offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS ) -1 );There are two ABI questions with that. First why not use just user addresses instead of offsets? It's more consistent with how everything else works. Surely it could've been offsets for all registered buffers ops from the beggining, but it's not.
Quoted 1 line
How would that work? You need to pass in addr + buffer index for that.
I guess it depends on the second part then, that is if you want to preserve the layout, in which case you can just use sqe->buf_index
Quoted 9 lines
The usual approach is doing that, and then 'addr' tells you the offset within the buffer, eg you can just do a subtraction to get your offset. But you can't pass in both addr + index in a provided buffer, which is why it's using
buf->addrto encode index + offset for that, rather than rely on the addr for the offset too.The alternative obviously is to just do the 'addr' and have that be both index and offset, in which case you'd need to lookup the buffer. And that's certainly a no-go.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 6/7] io_uring/kbuf: add support for mapping type KBUF_MODE_BVEC
Quoted 44 lines · Pavel Begunkov
Quoted 43 lines · Jens Axboe
Quoted 41 lines · Pavel Begunkov
Quoted 21 lines · Jens Axboe
The provided buffer helpers always map to iovecs. Add a new mode,
KBUF_MODE_BVEC, which instead maps it to abio_vecarray instead. For use with zero-copy scenarios, where the caller would want to turn it into abio_vecanyway, and this avoids first iterating and filling out and iovec array, only for the caller to then iterate it again and turn it into abio_vecarray.Since it's now managing both iovecs and bvecs, change the naming of
buf_sel_arg->nr_iovsmember tonr_vecsinstead.Signed-off-by Jens Axboe <[email protected]>--- io_uring/kbuf.c | 170 +++++++++++++++++++++++++++++++++++++++++++----- io_uring/kbuf.h | 9 ++- io_uring/net.c | 10 +-- 3 files changed, 165 insertions(+), 24 deletions(-)io_uring/kbuf.c +0
-0...
io_uring/kbuf.c static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, · line 133
Quoted 13 lines
io_uring/kbuf.c +13
-0+static struct io_mapped_ubuf *io_ubuf_from_buf (struct io_ring_ctx *ctx, +u64 addr,unsigned int *offset) +{ +struct io_mapped_ubuf *imu; +u16 idx; + +/* +* Get registered buffer index and offset, encoded into the +* addr base value. +*/ + idx = addr & ((1ULL <<IOU_BUF_REGBUF_BITS ) -1 ); + addr >>=IOU_BUF_REGBUF_BITS ; + *offset = addr & ((1ULL <<IOU_BUF_OFFSET_BITS ) -1 );There are two ABI questions with that. First why not use just user addresses instead of offsets? It's more consistent with how everything else works. Surely it could've been offsets for all registered buffers ops from the beggining, but it's not.
Quoted 5 lines
Quoted 1 line
How would that work? You need to pass in addr + buffer index for that.
I guess it depends on the second part then, that is if you want to preserve the layout, in which case you can just use
sqe->buf_index
The whole point is to make provided AND registered buffers work together. And you can't pass in a buffer group ID _and_ a registered buffer index in the SQE.
And for provided buffers, furthermore the point is that the buffer itself holds information about where to transfer to/from. Once you've added your buffer, you don't need to further track it, when it gets picked it has all the information on where the transfer occurs.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 7/7 io_uring/net: add provided buffer and bundle support to send zc
In reply to Jens AxboeProvided buffers inform the kernel which buffer group ID to pick a buffer from for transfer. Normally that buffer contains the usual addr + length information, as well as a buffer ID that is passed back at completion time to inform the application of which buffer was used for the transfer.
However, if registered and provided buffers are combined, then the provided buffer must instead tell the kernel which registered buffer index should be used, and the length/offset within that buffer. Rather than store the addr + length, the application must instead store this information instead.
If provided buffers are used with send zc, then those buffers must be an index into a registered buffer. Change the mapping type to use KBUF_MODE_BVEC, which tells the kbuf handlers to turn the mappings into bio_vecs rather than iovecs. Then all that is needed is to setup our iov_iterator to use iov_iter_bvec().
--- io_uring/net.c | 64 +++++++++++++++++++++++++++++++++--------------- io_uring/net.h | 10 ++++++-- io_uring/opdef.c | 1 + 3 files changed, 53 insertions(+), 22 deletions(-)
io_uring/net.c +44 -20
@@ -83,6 +83,8 @@ struct io_sr_msg {static int io_sg_from_iter (struct sk_buff *skb,struct iov_iter *from,size_t length); +static int io_sg_from_iter_iovec (struct sk_buff *skb,struct iov_iter *from, +size_t length);/* * Number of times we'll try and do receives if there's more data. If we
@@ -581,33 +583,34 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)return IOU_OK ; } -static int io_send_zc_import_single (struct io_kiocb *req, -unsigned int issue_flags) +static int __io_send_zc_import (struct io_kiocb *req, +struct io_async_msghdr *kmsg,int nsegs) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); -struct io_async_msghdr *kmsg = req->async_data;struct io_ring_ctx *ctx = req->ctx;struct io_mapped_ubuf *imu;int ret;u16 idx; - ret = -EFAULT ; -io_ring_submit_lock (ctx, issue_flags); -if (sr->buf_index < ctx->nr_user_bufs) { +if (req->flags &REQ_F_BUFFER_SELECT ) { +struct bio_vec *bv = kmsg->free_bvec ?: &kmsg->fast_bvec; + +WARN_ON_ONCE (bv == &kmsg->fast_bvec && nsegs >1 ); +iov_iter_bvec (&kmsg->msg.msg_iter,ITER_SOURCE , bv, nsegs, sr->len); + }else { +if (WARN_ON_ONCE (nsegs !=1 )) +return -EFAULT ; +if (unlikely (sr->buf_index >= ctx->nr_user_bufs)) +return -EFAULT ; idx =array_index_nospec (sr->buf_index, ctx->nr_user_bufs); imu =READ_ONCE (ctx->user_bufs[idx]); -io_req_set_rsrc_node (sr->notif, ctx); - ret =0 ; - } -io_ring_submit_unlock (ctx, issue_flags); -if (unlikely (ret)) -return ret; + ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, imu, + (u64 )(uintptr_t )sr->buf, sr->len); +if (unlikely (ret)) +return ret; + } - ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, imu, - (u64 )(uintptr_t )sr->buf, sr->len); -if (unlikely (ret)) -return ret; kmsg->msg.sg_from_iter = io_sg_from_iter;return 0 ; }
@@ -619,6 +622,16 @@ static int __io_send_import(struct io_kiocb *req, struct buf_sel_arg *arg,struct io_async_msghdr *kmsg = req->async_data;int ret = nsegs; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) { +io_ring_submit_lock (req->ctx, issue_flags); +io_req_set_rsrc_node (sr->notif, req->ctx); + ret =__io_send_zc_import (req, kmsg, nsegs); +io_ring_submit_unlock (req->ctx, issue_flags); +if (unlikely (ret <0 )) +return ret; +return nsegs; + } +if (nsegs ==1 ) { sr->buf = arg->iovs[0 ].iov_base; ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len,
@@ -646,10 +659,13 @@ static int io_send_import(struct io_kiocb *req, unsigned int issue_flags) .nr_vecs = 1, }; +if (sr->flags &IORING_RECVSEND_FIXED_BUF ) + arg.mode |=KBUF_MODE_BVEC ; +if (kmsg->free_iov) { arg.nr_vecs = kmsg->free_iov_nr; arg.iovs = kmsg->free_iov; - arg.mode =KBUF_MODE_FREE ; + arg.mode |=KBUF_MODE_FREE ; }if (!(sr->flags &IORING_RECVSEND_BUNDLE ))
@@ -1280,7 +1296,8 @@ void io_send_zc_cleanup(struct io_kiocb *req) } } -#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) +#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | \ + IORING_RECVSEND_FIXED_BUF | IORING_RECVSEND_BUNDLE)#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE)int io_send_zc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe)
@@ -1399,8 +1416,13 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags)struct io_async_msghdr *kmsg = req->async_data;int ret; + ret =io_send_import (req, issue_flags); +if (unlikely (ret <0 )) +return ret; +if (req->flags &REQ_F_BUFFER_SELECT ) +return 0 ;if (sr->flags &IORING_RECVSEND_FIXED_BUF ) -return io_send_zc_import_single (req, issue_flags); +return __io_send_zc_import (req, kmsg,1 ); ret =import_ubuf (ITER_SOURCE , sr->buf, sr->len, &kmsg->msg.msg_iter);if (unlikely (ret))
@@ -1416,6 +1438,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg);struct io_async_msghdr *kmsg = req->async_data; +unsigned int cflags;struct socket *sock;unsigned msg_flags;int ret, min_ret =0 ;
@@ -1476,7 +1499,8 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags)io_notif_flush (zc->notif);io_req_msg_cleanup (req,0 ); } -io_req_set_res (req, ret,IORING_CQE_F_MORE ); + cflags =io_put_kbuf (req, ret, issue_flags); +io_req_set_res (req, ret, cflags |IORING_CQE_F_MORE );return IOU_OK ; }
io_uring/net.h +8 -2
@@ -5,9 +5,15 @@struct io_async_msghdr {#if defined(CONFIG_NET ) -struct iovec fast_iov; +union { +struct iovec fast_iov; +struct bio_vec fast_bvec; + };/* points to an allocated iov, if NULL we use fast_iov instead */ -struct iovec *free_iov; +union { +struct iovec *free_iov; +struct bio_vec *free_bvec; + };int free_iov_nr;int namelen;__kernel_size_t controllen;
io_uring/opdef.c +1 -0
@@ -422,6 +422,7 @@ const struct io_issue_def io_issue_defs[] = { .needs_file = 1, .unbound_nonreg_file = 1, .pollout = 1, + .buffer_select = 1, .audit_skip = 1, .ioprio = 1,#if defined(CONFIG_NET )
Signature · 1 line
2.45.2
Re: [PATCH 7/7] io_uring/net: add provided buffer and bundle support to send zc
Quoted 20 lines · Jens Axboe
Provided buffers inform the kernel which buffer group ID to pick a buffer from for transfer. Normally that buffer contains the usual addr + length information, as well as a buffer ID that is passed back at completion time to inform the application of which buffer was used for the transfer.
However, if registered and provided buffers are combined, then the provided buffer must instead tell the kernel which registered buffer index should be used, and the length/offset within that buffer. Rather than store the addr + length, the application must instead store this information instead.
If provided buffers are used with send zc, then those buffers must be an index into a registered buffer. Change the mapping type to use
KBUF_MODE_BVEC, which tells the kbuf handlers to turn the mappings intobio_vecsrather than iovecs. Then all that is needed is to setup ouriov_iteratorto useiov_iter_bvec().Signed-off-by Jens Axboe <[email protected]>---
...
Quoted 8 lines
io_uring/net.h +0
-0@@ -5,9 +5,15 @@struct io_async_msghdr {#if defined(CONFIG_NET )
io_uring/net.h line 15
Quoted 10 lines
io_uring/net.h +7
-2-struct iovec fast_iov; +union { +struct iovec fast_iov; +struct bio_vec fast_bvec; + };/* points to an allocated iov, if NULL we use fast_iov instead */ -struct iovec *free_iov; +union { +struct iovec *free_iov; +struct bio_vec *free_bvec;
I'd rather not do it like that, aliasing with reusing memory and counting the number is a recipe for disaster when scattered across code. E.g. seems you change all(?) iovec allocations to allocate based on the size of the larger structure.
Counting bytes as in my series is less fragile, otherwise it needs a new structure and a set of helpers that can be kept together.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH 7/7] io_uring/net: add provided buffer and bundle support to send zc
Quoted 45 lines · Pavel Begunkov
Quoted 20 lines · Jens Axboe
Provided buffers inform the kernel which buffer group ID to pick a buffer from for transfer. Normally that buffer contains the usual addr + length information, as well as a buffer ID that is passed back at completion time to inform the application of which buffer was used for the transfer.
However, if registered and provided buffers are combined, then the provided buffer must instead tell the kernel which registered buffer index should be used, and the length/offset within that buffer. Rather than store the addr + length, the application must instead store this information instead.
If provided buffers are used with send zc, then those buffers must be an index into a registered buffer. Change the mapping type to use
KBUF_MODE_BVEC, which tells the kbuf handlers to turn the mappings intobio_vecsrather than iovecs. Then all that is needed is to setup ouriov_iteratorto useiov_iter_bvec().Signed-off-by Jens Axboe <[email protected]>---
...
Quoted 7 lines
io_uring/net.h +0
-0@@ -5,9 +5,15 @@struct io_async_msghdr {#if defined(CONFIG_NET )io_uring/net.h line 14
Quoted 10 lines
io_uring/net.h +7
-2-struct iovec fast_iov; +union { +struct iovec fast_iov; +struct bio_vec fast_bvec; + };/* points to an allocated iov, if NULL we use fast_iov instead */ -struct iovec *free_iov; +union { +struct iovec *free_iov; +struct bio_vec *free_bvec;I'd rather not do it like that, aliasing with reusing memory and counting the number is a recipe for disaster when scattered across code. E.g. seems you change all(?) iovec allocations to allocate based on the size of the larger structure.
Quoted 2 lines
Counting bytes as in my series is less fragile, otherwise it needs a new structure and a set of helpers that can be kept together.
I have been pondering this, because I'm not a huge fan either. But outside of the space side, it does come out pretty nicely/clean. This series is really just a WIP posting as per the RFC, mostly just so we can come up with something that's clean enough and works for both cases, as it does have the caching that your series does not. And to facilitate some more efficient TX/RX zero copy testing.
I'd love a separate struct for these two, but that kind of gets in the way of the usual iovec imports. I'll get back to this soonish, it's not a 6.13 thing by any stretch.
Signature · 1 line
Jens Axboe
Re: [PATCH 7/7] io_uring/net: add provided buffer and bundle support to send zc
Quoted 49 lines · Jens Axboe
Quoted 45 lines · Pavel Begunkov
Quoted 20 lines · Jens Axboe
Provided buffers inform the kernel which buffer group ID to pick a buffer from for transfer. Normally that buffer contains the usual addr + length information, as well as a buffer ID that is passed back at completion time to inform the application of which buffer was used for the transfer.
However, if registered and provided buffers are combined, then the provided buffer must instead tell the kernel which registered buffer index should be used, and the length/offset within that buffer. Rather than store the addr + length, the application must instead store this information instead.
If provided buffers are used with send zc, then those buffers must be an index into a registered buffer. Change the mapping type to use
KBUF_MODE_BVEC, which tells the kbuf handlers to turn the mappings intobio_vecsrather than iovecs. Then all that is needed is to setup ouriov_iteratorto useiov_iter_bvec().Signed-off-by Jens Axboe <[email protected]>---
...
Quoted 7 lines
io_uring/net.h +0
-0@@ -5,9 +5,15 @@struct io_async_msghdr {#if defined(CONFIG_NET )io_uring/net.h line 14
Quoted 10 lines
io_uring/net.h +7
-2-struct iovec fast_iov; +union { +struct iovec fast_iov; +struct bio_vec fast_bvec; + };/* points to an allocated iov, if NULL we use fast_iov instead */ -struct iovec *free_iov; +union { +struct iovec *free_iov; +struct bio_vec *free_bvec;I'd rather not do it like that, aliasing with reusing memory and counting the number is a recipe for disaster when scattered across code. E.g. seems you change all(?) iovec allocations to allocate based on the size of the larger structure.
Quoted 2 lines
Counting bytes as in my series is less fragile, otherwise it needs a new structure and a set of helpers that can be kept together.
Quoted 6 lines
I have been pondering this, because I'm not a huge fan either. But outside of the space side, it does come out pretty nicely/clean. This series is really just a WIP posting as per the RFC, mostly just so we can come up with something that's clean enough and works for both cases, as it does have the caching that your series does not. And to facilitate some more efficient TX/RX zero copy testing.
The thing is, I know how to implement caching on top of my series, but as I commented in the other thread, I don't think it can reuse the incremental helper well. At least it can try to unify the imu checking / parsing, bvec copy-setup, but that's minor.
Signature · 1 line
Pavel Begunkov
Quoted 9 lines · Jens Axboe
Hi,
Normally a request can take a provided buffer, which means "pick a buffer from group X and do IO to/from it", or it can use a registered buffer, which means "use the buffer at index Y and do IO to/from it". For things like
O_DIRECTand network zero copy, registered buffers can be used to speedup the operation, as they avoid repeatedget_user_pages()and page referencing calls for each IO operation.
Quoted 3 lines
Normal (non zero copy) send supports bundles, which is a way to pick multiple provided buffers at once and send them. send zero copy only supports registered buffers, and hence can only send a single buffer
That's not true, has never been, send[msg] zc work just fine with normal (non-registered) buffers.
Quoted 1 line
at the time.
And that's covered by the posted series for vectored registered buffers support.
Quoted 11 lines
This patchset adds support for using a mix of provided and registered buffers, where the provided buffers merely provide an index into which registered buffers to use. This enables using provided buffers for send zc in general, but also bundles where multiple buffers are picked. This is done by changing how the provided buffers are intepreted. Normally a provided buffer has an address, length, and buffer ID associated with it. The address tells the kernel where the IO should occur. If both fixed and provided buffers are asked for, the provided buffer address field is instead an encoding of the registered buffer index and the offset within that buffer. With that in place, using a combination of the two can work.
What the series doesn't say is how it works with notifications and what is the proposed user API in regard to it, it's the main if not the only fundamental distinctive part of the SENDZC API.
Quoted 21 lines
Patches 1-4 are just cleanup and prep, patch 5 adds the basic definition of what a fixed provided buffer looks like, patch 6 adds support for kbuf to map into a bvec directly, and finally patch 7 adds support for send zero copy to use this mechanism.
More details available in the actual patches. Tested with kperf using zero copy RX and TX, easily reaching 100G speeds with a single thread doing 4k sends and receives.
Kernel tree here:
https://git.kernel.dk/cgit/linux/log/?h=io_uring-sendzc-provided
include/uapi/linux/io_uring.h | 8 ++ io_uring/kbuf.c | 180 +++++++++++++++++++++++++++---- io_uring/kbuf.h | 9 +- io_uring/net.c | 192 ++++++++++++++++++++++------------ io_uring/net.h | 10 +- io_uring/opdef.c | 1 + 6 files changed, 309 insertions(+), 91 deletions(-)
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe
Hi,
Normally a request can take a provided buffer, which means "pick a buffer from group X and do IO to/from it", or it can use a registered buffer, which means "use the buffer at index Y and do IO to/from it". For things like
O_DIRECTand network zero copy, registered buffers can be used to speedup the operation, as they avoid repeatedget_user_pages()and page referencing calls for each IO operation.
Quoted 6 lines
Quoted 3 lines
Normal (non zero copy) send supports bundles, which is a way to pick multiple provided buffers at once and send them. send zero copy only supports registered buffers, and hence can only send a single buffer
That's not true, has never been, send[msg] zc work just fine with normal (non-registered) buffers.
That's not what I'm saying, perhaps it isn't clear. What I'm trying to say is that it only supports registered buffers, it does not support provided buffers. It obviously does support regular user provided buffers that aren't registered or provided, I figured that goes without saying explicitly.
Quoted 4 lines
Quoted 1 line
at the time.
And that's covered by the posted series for vectored registered buffers support.
Right, for sendmsg.
Quoted 12 lines
Quoted 11 lines
This patchset adds support for using a mix of provided and registered buffers, where the provided buffers merely provide an index into which registered buffers to use. This enables using provided buffers for send zc in general, but also bundles where multiple buffers are picked. This is done by changing how the provided buffers are intepreted. Normally a provided buffer has an address, length, and buffer ID associated with it. The address tells the kernel where the IO should occur. If both fixed and provided buffers are asked for, the provided buffer address field is instead an encoding of the registered buffer index and the offset within that buffer. With that in place, using a combination of the two can work.
Quoted 3 lines
What the series doesn't say is how it works with notifications and what is the proposed user API in regard to it, it's the main if not the only fundamental distinctive part of the SENDZC API.
Should not change that? You'll should get the usual two notifications on send complete, and reuse safe.
Signature · 1 line
Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Pavel Begunkov
Quoted 12 lines · Jens Axboe
Hi,
Normally a request can take a provided buffer, which means "pick a buffer from group X and do IO to/from it", or it can use a registered buffer, which means "use the buffer at index Y and do IO to/from it". For things like
O_DIRECTand network zero copy, registered buffers can be used to speedup the operation, as they avoid repeatedget_user_pages()and page referencing calls for each IO operation.Normal (non zero copy) send supports bundles, which is a way to pick multiple provided buffers at once and send them. send zero copy only supports registered buffers, and hence can only send a single buffer
Quoted 8 lines
Quoted 2 lines
That's not true, has never been, send[msg] zc work just fine with normal (non-registered) buffers.
That's not what I'm saying, perhaps it isn't clear. What I'm trying to say is that it only supports registered buffers, it does not support provided buffers. It obviously does support regular user provided buffers that aren't registered or provided, I figured that goes without saying explicitly.
Normally goes without saying yes, but the confusion here is because of a more or less explicit implication (or at least I read it so) "it only supports registered buffers => selected buffer support should support registered buffers, which it adds"
Does the series allows provided buffers with normal user memory?
Quoted 19 lines
Quoted 4 lines
Quoted 1 line
at the time.
And that's covered by the posted series for vectored registered buffers support.
Right, for sendmsg.
Quoted 12 lines
Quoted 11 lines
This patchset adds support for using a mix of provided and registered buffers, where the provided buffers merely provide an index into which registered buffers to use. This enables using provided buffers for send zc in general, but also bundles where multiple buffers are picked. This is done by changing how the provided buffers are intepreted. Normally a provided buffer has an address, length, and buffer ID associated with it. The address tells the kernel where the IO should occur. If both fixed and provided buffers are asked for, the provided buffer address field is instead an encoding of the registered buffer index and the offset within that buffer. With that in place, using a combination of the two can work.
Quoted 6 lines
Quoted 3 lines
What the series doesn't say is how it works with notifications and what is the proposed user API in regard to it, it's the main if not the only fundamental distinctive part of the SENDZC API.
Should not change that? You'll should get the usual two notifications on send complete, and reuse safe.
Right you get a notification, but what is it supposed to mean to the user? Like "the notification indicates that all buffers that are consumed by this request can be reused". Multishot is not a thing, but how the user has to track what buffers are consumed by this request? I assume it posts a CQE per buffer completion, right?
And let's say you have send heavy workload where the user pushes more than the socket can take, i.e. it has to wait to send more and there is always something to send. Does it poll-retry as it's usually done for multishots? How notifications are paced? i.e. it'll continue hooking more and more buffers onto the same notification locking all the previously used buffers.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 25 lines · Pavel Begunkov
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Pavel Begunkov
Quoted 12 lines · Jens Axboe
Hi,
Normally a request can take a provided buffer, which means "pick a buffer from group X and do IO to/from it", or it can use a registered buffer, which means "use the buffer at index Y and do IO to/from it". For things like
O_DIRECTand network zero copy, registered buffers can be used to speedup the operation, as they avoid repeatedget_user_pages()and page referencing calls for each IO operation.Normal (non zero copy) send supports bundles, which is a way to pick multiple provided buffers at once and send them. send zero copy only supports registered buffers, and hence can only send a single buffer
Quoted 8 lines
Quoted 2 lines
That's not true, has never been, send[msg] zc work just fine with normal (non-registered) buffers.
That's not what I'm saying, perhaps it isn't clear. What I'm trying to say is that it only supports registered buffers, it does not support provided buffers. It obviously does support regular user provided buffers that aren't registered or provided, I figured that goes without saying explicitly.
Quoted 4 lines
Normally goes without saying yes, but the confusion here is because of a more or less explicit implication (or at least I read it so) "it only supports registered buffers => selected buffer support should support registered buffers, which it adds"
I'll expand it to be more clear.
Quoted 1 line
Does the series allows provided buffers with normal user memory?
Yep, it should allow either picking one (or more, for bundles) provided buffers, and the provided buffer is either normal user memory, or it's indices into registered buffers.
Quoted 19 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
This patchset adds support for using a mix of provided and registered buffers, where the provided buffers merely provide an index into which registered buffers to use. This enables using provided buffers for send zc in general, but also bundles where multiple buffers are picked. This is done by changing how the provided buffers are intepreted. Normally a provided buffer has an address, length, and buffer ID associated with it. The address tells the kernel where the IO should occur. If both fixed and provided buffers are asked for, the provided buffer address field is instead an encoding of the registered buffer index and the offset within that buffer. With that in place, using a combination of the two can work.
Quoted 6 lines
Quoted 3 lines
What the series doesn't say is how it works with notifications and what is the proposed user API in regard to it, it's the main if not the only fundamental distinctive part of the SENDZC API.
Should not change that? You'll should get the usual two notifications on send complete, and reuse safe.
Quoted 6 lines
Right you get a notification, but what is it supposed to mean to the user? Like "the notification indicates that all buffers that are consumed by this request can be reused". Multishot is not a thing, but how the user has to track what buffers are consumed by this request? I assume it posts a CQE per buffer completion, right?
Depends on if it's bundles or not. For a non-bundle, a single buffer is picked, and that buffer is either user memory or it's an index into a registered buffer. For that, completions work just like they do now - a single one is posted for the expected inline completion with buffer ID, and one is posted for reuse laster.
If it's a bundle, it works the same way, two completions are posted. The first expected inline one will have a buffer ID, and the length will tell you how many consecutive buffers were sent/consumed. Then the reuse notification goes with that previous completion.
Quoted 6 lines
And let's say you have send heavy workload where the user pushes more than the socket can take, i.e. it has to wait to send more and there is always something to send. Does it poll-retry as it's usually done for multishots? How notifications are paced? i.e. it'll continue hooking more and more buffers onto the same notification locking all the previously used buffers.
If it sends nothing, nothing is consumed. If it's a partial send, then buffers are kept (as nobody else should send them anyway), and it's retried based on the poll trigger. For the latter case, completion is postd at the end, when the picked buffers are done. For pacing, you can limit the amount of data sent by just setting ->len to your desired bundle/batch size.
Signature · 1 line
Jens Axboe