io_uring: add kernel-managed buffer rings
v3 cover io_uring: add kernel-managed buffer rings
Currently, io_uring buffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through the IOU_PBUF_RING_KERNEL_MANAGED flag.
There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
This is split out from the fuse-over-io_uring series in [2], which needs the kernel to own and manage buffers shared between the fuse server and the kernel. The link to the fuse tree that uses the commits in this series is in [3].
This series is on top of the for-7.1/io_uring branch in Jens' io-uring tree (commit ee1d7dc33990). The corresponding liburing changes are in [4] and will be submitted after the changes in this patchset have landed.
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t
[2] https://lore.kernel.org/linux-fsdevel/[email protected]/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy_for_v3/
[4] https://github.com/joannekoong/liburing/commits/pbuf_kernel_managed/
Changelog
---------
Changes from v1 -> v3:
* Incorporate Jens' feedback, including fixing wraparound int promotion bug
* uapi: merge kmbuf into pbuf interface/apis as IOU_PBUF_RING_KERNEL_MANAGED flag (Pavel)
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t
Changes since [1]:
* add "if (bl)" check for recycling API (Bernd)
* check mul overflow, use GFP_USER, use PTR as return type (Christoph)
* fix bl->ring leak (me)
Joanne Koong (8):
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: return buffer id in buffer selection io_uring/kbuf: add recycling for kernel managed buffer rings io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: export io_ring_buffer_select() io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
include/linux/io_uring/cmd.h | 53 ++++++- include/linux/io_uring_types.h | 10 +- include/uapi/linux/io_uring.h | 16 ++- io_uring/kbuf.c | 250 +++++++++++++++++++++++++++++---- io_uring/kbuf.h | 11 +- io_uring/memmap.c | 111 +++++++++++++++ io_uring/memmap.h | 4 + io_uring/uring_cmd.c | 6 +- 8 files changed, 427 insertions(+), 34 deletions(-)
Signature · 1 line
2.47.3
PATCH v3 1/8 io_uring/kbuf: add support for kernel-managed buffer rings
Add support for kernel-managed buffer rings (kmbuf rings), which allow the kernel to allocate and manage the backing buffers for a buffer ring, rather than requiring the application to provide and manage them.
Internally, the IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.
At the uapi level, kernel-managed buffer rings are created through the pbuf interface with the IOU_PBUF_RING_KERNEL_MANAGED flag set. The io_uring_buf_reg struct is modified to allow taking in a buf_size instead of a ring_addr. To create a kernel-managed buffer ring, the caller must set the IOU_PBUF_RING_MMAP flag as well to indicate that the kernel will allocate the memory for the ring. When the caller mmaps the ring, they will get back a virtual mapping to the buffer memory.
--- include/uapi/linux/io_uring.h | 16 ++++- io_uring/kbuf.c | 97 ++++++++++++++++++++++++----- io_uring/kbuf.h | 6 +- io_uring/memmap.c | 111 ++++++++++++++++++++++++++++++++++ io_uring/memmap.h | 4 ++ 5 files changed, 215 insertions(+), 19 deletions(-)
include/uapi/linux/io_uring.h +15 -1
@@ -892,15 +892,29 @@ struct io_uring_buf_ring {* use of it will consume only as much as it needs. This * requires that both the kernel and application keep * track of where the current read/recv index is at. +* IOU_PBUF_RING_KERNEL_MANAGED: If set, kernel allocates and manages the memory +* for the ring and its buffers. The application must set +* the buffer size through reg->buf_size and the size must +* be page-aligned. When the application subsequently calls +* mmap(2) with +* IORING_OFF_PBUF_RING | (bgid << IORING_OFF_PBUF_SHIFT), +* the virtual mapping returned is a contiguous mapping of +* the buffers. If set, IOU_PBUF_RING_MMAP must be set as +* well. */ enum io_uring_register_pbuf_ring_flags {IOU_PBUF_RING_MMAP =1 ,IOU_PBUF_RING_INC =2 , +IOU_PBUF_RING_KERNEL_MANAGED =4 , };/* argument for IORING_(UN)REGISTER_PBUF_RING */ struct io_uring_buf_reg { -__u64 ring_addr; +union { +__u64 ring_addr; +/* used if reg->flags & IOU_PBUF_RING_KERNEL_MANAGED */ +__u32 buf_size; + };__u32 ring_entries;__u16 bgid;__u16 flags;
io_uring/kbuf.c +81 -16
150 lines not on this page · raw message
io_uring/kbuf.h +4 -2
@@ -7,9 +7,11 @@enum {/* ring mapped provided buffers */ -IOBL_BUF_RING =1 , +IOBL_BUF_RING =1 ,/* buffers are consumed incrementally rather than always fully */ -IOBL_INC =2 , +IOBL_INC =2 , +/* buffers are kernel managed */ +IOBL_KERNEL_MANAGED =4 , };struct io_buffer_list {
io_uring/memmap.c +111 -0
141 lines not on this page · raw message
io_uring/memmap.h +4 -0
@@ -22,6 +22,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr,struct io_uring_region_desc *reg, unsigned long mmap_offset); +int io_create_region_multi_buf (struct io_ring_ctx *ctx, +struct io_mapped_region *mr, +unsigned int nr_bufs,unsigned int buf_size); +static inline void *io_region_get_ptr (struct io_mapped_region *mr) {return mr->ptr;
Signature · 1 line
2.47.3
PATCH v3 2/8 io_uring/kbuf: support kernel-managed buffer rings in buffer selection
In reply to Joanne KoongAllow kernel-managed buffers to be selected. This requires modifying the io_br_sel struct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.
Auto-commit any selected kernel-managed buffer.
--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 16 ++++++++++++---- 2 files changed, 16 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +4 -4
@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };
io_uring/kbuf.c +12 -4
@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the
@@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true; -/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && !io_is_uring_cmd (req))return true;return false;
@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; - sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr); +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; }
Signature · 1 line
2.47.3
Re: [PATCH v3 2/8] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
Quoted 4 lines · Joanne Koong
Allow kernel-managed buffers to be selected. This requires modifying the
io_br_selstruct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.
[...]
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 211
Quoted 9 lines
io_uring/kbuf.c +4
-1@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; - sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr); +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr));
Nit: For IOBL_KERNEL_MANAGED buffers, "struct io_uring_buf" is stored in a normal kernel allocation that isn't concurrently accessed from anywhere else, right? So no READ_ONCE() is needed in the IOBL_KERNEL_MANAGED case?
Re: [PATCH v3 2/8] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
Quoted 7 lines · Jann Horn
Quoted 4 lines · Joanne Koong
Allow kernel-managed buffers to be selected. This requires modifying the
io_br_selstruct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.[...]
Quoted 14 lines
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 211
Quoted 9 lines
io_uring/kbuf.c +4
-1@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; - sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr); +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr));Nit: For
IOBL_KERNEL_MANAGEDbuffers, "structio_uring_buf" is stored in a normal kernel allocation that isn't concurrently accessed from anywhere else, right? So noREAD_ONCE()is needed in theIOBL_KERNEL_MANAGEDcase?
Yes you're right, I had copied it over from the userptr case but it's unnecessary here. I'll submit a follow-up to take this out or maybe Jens can remove it locally.
Thanks,
Joanne
PATCH v3 3/8 io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Joanne KoongAdd kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.
This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
--- include/linux/io_uring/cmd.h | 17 +++++++++++ io_uring/kbuf.c | 55 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 5 ++++ 3 files changed, 77 insertions(+)
include/linux/io_uring/cmd.h +17 -0
@@ -91,6 +91,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,bool io_uring_mshot_cmd_post_cqe (struct io_uring_cmd *ioucmd,struct io_br_sel *sel,unsigned int issue_flags); +int io_uring_buf_ring_pin (struct io_uring_cmd *cmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **out_bl); +int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd,unsigned buf_group, +unsigned issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -133,6 +137,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, {return true; } +static inline int io_uring_buf_ring_pin (struct io_uring_cmd *cmd, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +55 -0
73 lines not on this page · raw message
io_uring/kbuf.h +5 -0
@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {
Signature · 1 line
2.47.3
PATCH v3 4/8 io_uring/kbuf: return buffer id in buffer selection
In reply to Joanne KoongReturn the id of the selected buffer in io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.
--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +1 -1
@@ -78,7 +78,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);/* * Select a buffer from the provided buffer group for multishot uring_cmd. -* Returns the selected buffer address and size. +* Returns the selected buffer address, size, and id. */ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, unsigned buf_group, size_t *len,
include/linux/io_uring_types.h +2 -0
@@ -100,6 +100,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id; };
io_uring/kbuf.c +5 -2
@@ -206,6 +206,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; + sel.buf_id = req->buf_index;if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr);else
@@ -229,10 +230,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index; + } }io_ring_submit_unlock (req->ctx, issue_flags);return sel;
Signature · 1 line
2.47.3
PATCH v3 5/8 io_uring/kbuf: add recycling for kernel managed buffer rings
In reply to Joanne KoongAdd an interface for buffers to be recycled back into a kernel-managed buffer ring.
--- include/linux/io_uring/cmd.h | 11 +++++++++ io_uring/kbuf.c | 48 ++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
include/linux/io_uring/cmd.h +11 -0
@@ -95,6 +95,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **out_bl);int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd,unsigned int buf_group, +u64 addr,unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -150,6 +154,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +48 -0
54 lines not on this page · raw message
Signature · 1 line
2.47.3
PATCH v3 6/8 io_uring/kbuf: add io_uring_is_kmbuf_ring()
In reply to Joanne Koongio_uring_is_kmbuf_ring() returns true if there is a kernel-managed buffer ring at the specified buffer group.
This is a preparatory patch for upcoming fuse kernel-managed buffer support, which needs to ensure the buffer ring registered by the server is a kernel-managed buffer ring.
--- include/linux/io_uring/cmd.h | 9 +++++++++ io_uring/kbuf.c | 20 ++++++++++++++++++++ 2 files changed, 29 insertions(+)
include/linux/io_uring/cmd.h +9 -0
@@ -99,6 +99,9 @@ int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd,unsigned int buf_group,u64 addr,unsigned int len,unsigned int bid,unsigned int issue_flags); + +bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -161,6 +164,12 @@ static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd, +unsigned int buf_group, +unsigned int issue_flags) +{ +return false; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +20 -0
@@ -917,3 +917,23 @@ struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx,return NULL ;return &bl->region; } + +bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct io_buffer_list *bl; +bool is_kmbuf_ring = false; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) && (bl->flags &IOBL_KERNEL_MANAGED )) { +WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )); + is_kmbuf_ring = true; + } + +io_ring_submit_unlock (ctx, issue_flags); +return is_kmbuf_ring; +} +EXPORT_SYMBOL_GPL (io_uring_is_kmbuf_ring);
Signature · 1 line
2.47.3
PATCH v3 7/8 io_uring/kbuf: export io_ring_buffer_select()
In reply to Joanne KoongExport io_ring_buffer_select() so that it may be used by callers who pass in a pinned bufring without needing to grab the io_uring mutex.
This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
--- include/linux/io_uring/cmd.h | 14 ++++++++++++++ io_uring/kbuf.c | 7 ++++--- 2 files changed, 18 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +14 -0
@@ -102,6 +102,10 @@ int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group,unsigned int issue_flags); + +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -170,6 +174,16 @@ static inline bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, {return false; } +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; +return sel; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +4 -3
@@ -230,9 +230,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,return false; } -static struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -unsigned int issue_flags) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) {struct io_uring_buf_ring *br = bl->buf_ring;__u16 tail, head = bl->head;
@@ -266,6 +266,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }return sel; } +EXPORT_SYMBOL_GPL (io_ring_buffer_select);struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags)
Signature · 1 line
2.47.3
PATCH v3 8/8 io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
In reply to Joanne KoongWhen uring_cmd operations select a buffer, the completion queue entry should indicate which buffer was selected.
Set IORING_CQE_F_BUFFER on the completed entry and encode the buffer index if a buffer was selected.
This change is needed in order to relay to userspace which selected buffer contains the data.
--- io_uring/uring_cmd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/uring_cmd.c +5 -1
@@ -151,6 +151,7 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); +u32 cflags =0 ;if (WARN_ON_ONCE (req->flags &REQ_F_APOLL_MULTISHOT ))return ;
@@ -160,7 +161,10 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (ret <0 )req_set_fail (req); -io_req_set_res (req, ret,0 ); +if (req->flags & (REQ_F_BUFFER_SELECTED |REQ_F_BUFFER_RING )) + cflags |=IORING_CQE_F_BUFFER | + (req->buf_index <<IORING_CQE_BUFFER_SHIFT ); +io_req_set_res (req, ret, cflags);if (is_cqe32) {if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;
Signature · 1 line
2.47.3
Quoted 14 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
Quoted 1 line
[...]
Applied, thanks!
[1/8] io_uring/kbuf: add support for kernel-managed buffer rings
commit: ff168843d80cd1855b646c5e8be2c6aa5bfb8adb
[2/8] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
commit: bf4a6eb27f8135f36a5286bd5ba87bf6468c1283
[3/8] io_uring/kbuf: add buffer ring pinning/unpinning
commit: b6ffe5399bc04abab4da9afb2ba092e6269d7077
[4/8] io_uring/kbuf: return buffer id in buffer selection
commit: 9aa9e55e8bad1b65c9bcc2892101b06ae00c96d9
[5/8] io_uring/kbuf: add recycling for kernel managed buffer rings
commit: de48bc0efc58d1ebf7e35c36dde6eb0e6b246b8c
[6/8] io_uring/kbuf: add io_uring_is_kmbuf_ring()
commit: 08b57b67602a98abf4b454a363263cba9fcfc91f
[7/8] io_uring/kbuf: export io_ring_buffer_select()
commit: 4706d1f235ff94f19b5c818bc0a3abd735903695
[8/8] io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
commit: 3515a2aedcdf459fc851df40a3712fc4a9a060f7Best regards,
Signature · 1 line
Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 14 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
Quoted 1 line
[...]
Applied, thanks!
[1/8]
io_uring/kbuf: add support for kernel-managed buffer rings
Quoted 15 lines
commit: ff168843d80cd1855b646c5e8be2c6aa5bfb8adb
[2/8] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
commit: bf4a6eb27f8135f36a5286bd5ba87bf6468c1283
[3/8] io_uring/kbuf: add buffer ring pinning/unpinning
commit: b6ffe5399bc04abab4da9afb2ba092e6269d7077
[4/8] io_uring/kbuf: return buffer id in buffer selection
commit: 9aa9e55e8bad1b65c9bcc2892101b06ae00c96d9
[5/8] io_uring/kbuf: add recycling for kernel managed buffer rings
commit: de48bc0efc58d1ebf7e35c36dde6eb0e6b246b8c
[6/8] io_uring/kbuf: add io_uring_is_kmbuf_ring()
commit: 08b57b67602a98abf4b454a363263cba9fcfc91f
[7/8] io_uring/kbuf: export io_ring_buffer_select()
commit: 4706d1f235ff94f19b5c818bc0a3abd735903695
[8/8] io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
commit: 3515a2aedcdf459fc851df40a3712fc4a9a060f7After reading the io_uring/net.c code and considering how it'll work for selected kernel buffers, I don't see a non-hacky way for fuse to take advantage of selected buffers for socket receives. I'm realizing that it requires too much special casing to make it compatible with fuse (eg relying on the bufring being permanently pinned, different locking requirements for the bufring for fuse vs. traditional io-uring paths since fuse shouldn't contend for the io_uring mutex in the request dispatching path). Kernel-managed buffers doesn't tie in as nicely as I'd thought with other io-uring requests either. For IORING_OP_READ / IORING_OP_WRITE requests, the kernel buffer would still need to be placed in a fixed buffer since the sqe unions both buf_index and buf_group and I don't see a clean way to route both of those fields through without making the uapi worse. One of the principal reasons for having kernel-managed ring buffers be part of the io-uring infrastructure was so it could be easily integrated with other io-uring requests, but I'm realizing this doesn't hold weight. I think Pavel was right that it'd be cleaner for io-uring if this logic lived inside fuse instead of being part of the io-uring uapi.
Jens, could you drop this patchset from the for-7.1 tree? I'm going to move this logic into fuse instead.
Thanks,
Joanne
Quoted 7 lines
Best regards,
--
Jens Axboe
Quoted 23 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
This is split out from the fuse-over-io_uring series in [2], which needs the kernel to own and manage buffers shared between the fuse server and the kernel. The link to the fuse tree that uses the commits in this series is in [3].
This series is on top of the for-7.1/io_uring branch in Jens' io-uring tree (commit
ee1d7dc33990). The corresponding liburing changes are in [4] and will be submitted after the changes in this patchset have landed.
Quoted 7 lines
Thanks,
Joanne[1] https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t
[2] https://lore.kernel.org/linux-fsdevel/[email protected]/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy_for_v3/
[4] https://github.com/joannekoong/liburing/commits/pbuf_kernel_managed/
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has to implement things two times - not ideal. Couldn't we have pbuf for non-privileged users and basically depcrecate the existing fuse io-uring buffer API? In the sense that it needs to be further supported for some time, but won't get any new feature. Different buffer sizes would then only be supported through kbuf/pbuf?
Thanks,
Bernd
Quoted 25 lines · Bernd Schubert
Quoted 23 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
This is split out from the fuse-over-io_uring series in [2], which needs the kernel to own and manage buffers shared between the fuse server and the kernel. The link to the fuse tree that uses the commits in this series is in [3].
This series is on top of the for-7.1/io_uring branch in Jens' io-uring tree (commit
ee1d7dc33990). The corresponding liburing changes are in [4] and will be submitted after the changes in this patchset have landed.
Quoted 8 lines
Hi Bernd,
Quoted 5 lines
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just
Is your motivation for wanting different request sizes for the ring buffers so that it can optimize the memory costs of the buffers? I agree that trying to reduce the memory footprint of the buffers is very important. The main reason I ended up going with the buffer ring design was for that purpose. When kbuf incremental buffer consumption is added in the future (I plan to submit it separately once all the io-uring pieces of the fuse-zero-copy patchset land), this will allow non-overlapping regions of the individual buffer to be used across multiple different-sized requests concurrently.
From my point of view, this is better than allocating variable-sized buffers upfront because:
a) entries are fully maximized. With variable-sized buffers, the big buffers would be reserved specifically for payload requests while the small buffers would be reserved specifically for metadata requests. We could allocate '# entries' amount of small buffers, but for big buffers there would be less than '# entries'. If the server needs to service a lot of concurrent I/O requests, then the ring gets throttled on the limited number of big buffers available.
b) it best maximizes buffer memory. A request could need a buffer of any size so with variable-sized buffers, there's extra space in the buffer that is still being wasted. For example, for large payload requests, the big buffers would need to be the size of the max payload size (eg default 1 MB) but a lot of requests will fall under that. With incremental buffer consumption, only however many bytes used by the request are reserved in the buffer.
c) there's no overhead with having to (as you pointed out) keep the buffers tracked and sorted into per-sized lists. If we wanted to use variable-sized buffers with kbufs instead of using incremental buffer consumption, the best way to do that would be to allocate a separate kbufring to support payload requests vs metadata requests.
Quoted 4 lines
register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has
Non-privileged fuse servers use kbufs as well. It's only zero-copying that is not possible for non-privileged servers.
Quoted 2 lines
to implement things two times - not ideal. Couldn't we have pbuf for non-privileged users and basically depcrecate the existing fuse io-uring
I don't think this is necessary because kbufs works for both non-privileged and privileged servers. For how the buffer gets used by the server/kernel, pbufs are not an option here because the kernel has to be the one to recycle back the buffer (since it needs to read / copy data the server returns back in the buffer).
Quoted 3 lines
buffer API? In the sense that it needs to be further supported for some time, but won't get any new feature. Different buffer sizes would then only be supported through kbuf/pbuf?
I hope I understood your questions correctly, but if I misread anything, please let me know. I am going to be updating and submitting the fuse patches next week - the main update will be changing the headers to go through a registered memory region (which I only realized existed after the discussion with Pavel in v1) instead of as a registered buffer, as that will allow us to avoid the per I/O lookup overhead and drop the patch for the "io_uring_fixed_index_get()/io_uring_fixed_index_put()" refcount dance altogether.
Thanks,
Joanne
Quoted 4 lines
Thanks,
Bernd
Quoted 43 lines · Joanne Koong
Quoted 25 lines · Bernd Schubert
Quoted 23 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
This is split out from the fuse-over-io_uring series in [2], which needs the kernel to own and manage buffers shared between the fuse server and the kernel. The link to the fuse tree that uses the commits in this series is in [3].
This series is on top of the for-7.1/io_uring branch in Jens' io-uring tree (commit
ee1d7dc33990). The corresponding liburing changes are in [4] and will be submitted after the changes in this patchset have landed.Quoted 8 lines
Hi Bernd,
Quoted 5 lines
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just
Quoted 9 lines
Is your motivation for wanting different request sizes for the ring buffers so that it can optimize the memory costs of the buffers? I agree that trying to reduce the memory footprint of the buffers is very important. The main reason I ended up going with the buffer ring design was for that purpose. When kbuf incremental buffer consumption is added in the future (I plan to submit it separately once all the io-uring pieces of the fuse-zero-copy patchset land), this will allow non-overlapping regions of the individual buffer to be used across multiple different-sized requests concurrently.
That is also fine.
Quoted 10 lines
From my point of view, this is better than allocating variable-sized buffers upfront because:
a) entries are fully maximized. With variable-sized buffers, the big buffers would be reserved specifically for payload requests while the small buffers would be reserved specifically for metadata requests. We could allocate '# entries' amount of small buffers, but for big buffers there would be less than '# entries'. If the server needs to service a lot of concurrent I/O requests, then the ring gets throttled on the limited number of big buffers available.
I would like to see something like 8K, 16K, 32K, 128K.
Quoted 8 lines
b) it best maximizes buffer memory. A request could need a buffer of any size so with variable-sized buffers, there's extra space in the buffer that is still being wasted. For example, for large payload requests, the big buffers would need to be the size of the max payload size (eg default 1 MB) but a lot of requests will fall under that. With incremental buffer consumption, only however many bytes used by the request are reserved in the buffer.
Doesn't that cause fragmentation?
Quoted 6 lines
c) there's no overhead with having to (as you pointed out) keep the buffers tracked and sorted into per-sized lists. If we wanted to use variable-sized buffers with kbufs instead of using incremental buffer consumption, the best way to do that would be to allocate a separate kbufring to support payload requests vs metadata requests.
Yeah, I had thought of multiple kbuf rings, with different sizes.
Quoted 8 lines
Quoted 5 lines
register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has
Non-privileged fuse servers use kbufs as well. It's only zero-copying that is not possible for non-privileged servers.
Non-privileged cannot pin, at least by default mlock size is 8MB. I was under the impression that kbuf would be always pinned, but I need to read over it again.
Quoted 9 lines
Quoted 3 lines
to implement things two times - not ideal. Couldn't we have pbuf for non-privileged users and basically depcrecate the existing fuse io-uring
I don't think this is necessary because kbufs works for both non-privileged and privileged servers. For how the buffer gets used by the server/kernel, pbufs are not an option here because the kernel has to be the one to recycle back the buffer (since it needs to read / copy data the server returns back in the buffer).
I was thinking to set a flag or take ref count and to disallow pbuf destruction.
Quoted 5 lines
Quoted 4 lines
buffer API? In the sense that it needs to be further supported for some time, but won't get any new feature. Different buffer sizes would then only be supported through kbuf/pbuf?
Quoted 9 lines
I hope I understood your questions correctly, but if I misread anything, please let me know. I am going to be updating and submitting the fuse patches next week - the main update will be changing the headers to go through a registered memory region (which I only realized existed after the discussion with Pavel in v1) instead of as a registered buffer, as that will allow us to avoid the per I/O lookup overhead and drop the patch for the "io_uring_fixed_index_get()/io_uring_fixed_index_put()" refcount dance altogether.
I will try to review ASAP when you submit.
Thanks,
Bernd
Quoted 36 lines · Bernd Schubert
Quoted 11 lines · Joanne Koong
Quoted 2 lines · Bernd Schubert
On 3/6/26 01:32, Joanne Koong wrote:
Hi Bernd,
Quoted 5 lines
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just
Quoted 9 lines
Is your motivation for wanting different request sizes for the ring buffers so that it can optimize the memory costs of the buffers? I agree that trying to reduce the memory footprint of the buffers is very important. The main reason I ended up going with the buffer ring design was for that purpose. When kbuf incremental buffer consumption is added in the future (I plan to submit it separately once all the io-uring pieces of the fuse-zero-copy patchset land), this will allow non-overlapping regions of the individual buffer to be used across multiple different-sized requests concurrently.
That is also fine.
Quoted 10 lines
From my point of view, this is better than allocating variable-sized buffers upfront because:
a) entries are fully maximized. With variable-sized buffers, the big buffers would be reserved specifically for payload requests while the small buffers would be reserved specifically for metadata requests. We could allocate '# entries' amount of small buffers, but for big buffers there would be less than '# entries'. If the server needs to service a lot of concurrent I/O requests, then the ring gets throttled on the limited number of big buffers available.
Quoted 1 line
I would like to see something like 8K, 16K, 32K, 128K.
My worry is that for I/O heavy workloads with large read/write payloads (eg client access patterns reading/writing MBs at a time), the limited number of big enough buffers becomes the throttling bottleneck.
Quoted 11 lines
Quoted 9 lines
b) it best maximizes buffer memory. A request could need a buffer of any size so with variable-sized buffers, there's extra space in the buffer that is still being wasted. For example, for large payload requests, the big buffers would need to be the size of the max payload size (eg default 1 MB) but a lot of requests will fall under that. With incremental buffer consumption, only however many bytes used by the request are reserved in the buffer.
Doesn't that cause fragmentation?
With incremetnal buffer consumption, there's not fragmentation in the classical sense (eg scattered unusable holes). The buffer gets recycled back into the ring as a whole once all the requests in it have completed (tracked by refcounting).
I think the concern is that if the server is very slow to fulfill requests and the workload pattern has it so that slow requests are packed into the same buffer as fast requests across all the buffers in the queue and that queue has all its buffers saturated, then the next buffer is available only once the slow request has completed. We can mitigate this by assigning the request to a queue on the nearest numa node as a fallback if we detect that case. We could also do the same thing to mitigate the variable-sized buffer scenario where there's not enough big buffers for the queue, but I think that logic ends up a bit more complex.
I think overall we're able to support both incremental buffer consumption + variable-sized buffers if there's a need for it in the future where the server would like to choose.
Quoted 16 lines
Quoted 7 lines
c) there's no overhead with having to (as you pointed out) keep the buffers tracked and sorted into per-sized lists. If we wanted to use variable-sized buffers with kbufs instead of using incremental buffer consumption, the best way to do that would be to allocate a separate kbufring to support payload requests vs metadata requests.
Yeah, I had thought of multiple kbuf rings, with different sizes.
Quoted 6 lines
Quoted 5 lines
register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has
Quoted 6 lines
Quoted 2 lines
Non-privileged fuse servers use kbufs as well. It's only zero-copying that is not possible for non-privileged servers.
Non-privileged cannot pin, at least by default mlock size is 8MB. I was under the impression that kbuf would be always pinned, but I need to read over it again.
The kbufs get accounted to the user's mlock usage (this happens in __io_account_mem()). If the user running the unprivileged server doesn't belong to a group that has high enough mlock limits, they'll have to use regular fuse over-io-uring buffers instead of kbufs for most of their queues.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
to implement things two times - not ideal. Couldn't we have pbuf for non-privileged users and basically depcrecate the existing fuse io-uring
Quoted 8 lines
Quoted 5 lines
I don't think this is necessary because kbufs works for both non-privileged and privileged servers. For how the buffer gets used by the server/kernel, pbufs are not an option here because the kernel has to be the one to recycle back the buffer (since it needs to read / copy data the server returns back in the buffer).
I was thinking to set a flag or take ref count and to disallow pbuf destruction.
I don't think we can prevent the buffers from being freed by userspace except by pinning them but then that would need to attribute the buffers towards the mlock count.
Quoted 16 lines
Quoted 6 lines
Quoted 5 lines
buffer API? In the sense that it needs to be further supported for some time, but won't get any new feature. Different buffer sizes would then only be supported through kbuf/pbuf?
Quoted 9 lines
I hope I understood your questions correctly, but if I misread anything, please let me know. I am going to be updating and submitting the fuse patches next week - the main update will be changing the headers to go through a registered memory region (which I only realized existed after the discussion with Pavel in v1) instead of as a registered buffer, as that will allow us to avoid the per I/O lookup overhead and drop the patch for the "io_uring_fixed_index_get()/io_uring_fixed_index_put()" refcount dance altogether.
Quoted 1 line
I will try to review ASAP when you submit.
Thank you!
Thanks,
Joanne
Quoted 6 lines
Thanks,
Bernd
Quoted 99 lines · Joanne Koong
Quoted 36 lines · Bernd Schubert
Quoted 11 lines · Joanne Koong
Quoted 2 lines · Bernd Schubert
On 3/6/26 01:32, Joanne Koong wrote:
Hi Bernd,
Quoted 5 lines
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just
Quoted 9 lines
Is your motivation for wanting different request sizes for the ring buffers so that it can optimize the memory costs of the buffers? I agree that trying to reduce the memory footprint of the buffers is very important. The main reason I ended up going with the buffer ring design was for that purpose. When kbuf incremental buffer consumption is added in the future (I plan to submit it separately once all the io-uring pieces of the fuse-zero-copy patchset land), this will allow non-overlapping regions of the individual buffer to be used across multiple different-sized requests concurrently.
That is also fine.
Quoted 10 lines
From my point of view, this is better than allocating variable-sized buffers upfront because:
a) entries are fully maximized. With variable-sized buffers, the big buffers would be reserved specifically for payload requests while the small buffers would be reserved specifically for metadata requests. We could allocate '# entries' amount of small buffers, but for big buffers there would be less than '# entries'. If the server needs to service a lot of concurrent I/O requests, then the ring gets throttled on the limited number of big buffers available.Quoted 1 line
I would like to see something like 8K, 16K, 32K, 128K.
My worry is that for I/O heavy workloads with large read/write payloads (eg client access patterns reading/writing MBs at a time), the limited number of big enough buffers becomes the throttling bottleneck.
Quoted 11 lines
Quoted 9 lines
b) it best maximizes buffer memory. A request could need a buffer of any size so with variable-sized buffers, there's extra space in the buffer that is still being wasted. For example, for large payload requests, the big buffers would need to be the size of the max payload size (eg default 1 MB) but a lot of requests will fall under that. With incremental buffer consumption, only however many bytes used by the request are reserved in the buffer.
Doesn't that cause fragmentation?
With incremetnal buffer consumption, there's not fragmentation in the classical sense (eg scattered unusable holes). The buffer gets recycled back into the ring as a whole once all the requests in it have completed (tracked by refcounting).
I think the concern is that if the server is very slow to fulfill requests and the workload pattern has it so that slow requests are packed into the same buffer as fast requests across all the buffers in the queue and that queue has all its buffers saturated, then the next buffer is available only once the slow request has completed. We can mitigate this by assigning the request to a queue on the nearest numa node as a fallback if we detect that case. We could also do the same thing to mitigate the variable-sized buffer scenario where there's not enough big buffers for the queue, but I think that logic ends up a bit more complex.
I think overall we're able to support both incremental buffer consumption + variable-sized buffers if there's a need for it in the future where the server would like to choose.
Quoted 16 lines
Quoted 7 lines
c) there's no overhead with having to (as you pointed out) keep the buffers tracked and sorted into per-sized lists. If we wanted to use variable-sized buffers with kbufs instead of using incremental buffer consumption, the best way to do that would be to allocate a separate kbufring to support payload requests vs metadata requests.
Yeah, I had thought of multiple kbuf rings, with different sizes.
Quoted 6 lines
Quoted 5 lines
register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has
Quoted 6 lines
Quoted 2 lines
Non-privileged fuse servers use kbufs as well. It's only zero-copying that is not possible for non-privileged servers.
Non-privileged cannot pin, at least by default mlock size is 8MB. I was under the impression that kbuf would be always pinned, but I need to read over it again.
Quoted 5 lines
The kbufs get accounted to the user's mlock usage (this happens in
__io_account_mem()). If the user running the unprivileged server doesn't belong to a group that has high enough mlock limits, they'll have to use regular fuse over-io-uring buffers instead of kbufs for most of their queues.
That is exactly what I mean - in reality unprivileged servers will not be able to use kbufs. And there it would be good, if that server could use unpinned pbufs.
Thanks,
Bernd
Quoted 109 lines · Bernd Schubert
Quoted 99 lines · Joanne Koong
Quoted 36 lines · Bernd Schubert
Quoted 11 lines · Joanne Koong
Quoted 2 lines · Bernd Schubert
On 3/6/26 01:32, Joanne Koong wrote:
Hi Bernd,
Quoted 5 lines
Hi Joanne,
I'm a bit late, but could we have a design discussion about fuse here? From my point of view it would be good if we could have different request sizes for the ring buffers. Without kbuf I thought we would just
Quoted 9 lines
Is your motivation for wanting different request sizes for the ring buffers so that it can optimize the memory costs of the buffers? I agree that trying to reduce the memory footprint of the buffers is very important. The main reason I ended up going with the buffer ring design was for that purpose. When kbuf incremental buffer consumption is added in the future (I plan to submit it separately once all the io-uring pieces of the fuse-zero-copy patchset land), this will allow non-overlapping regions of the individual buffer to be used across multiple different-sized requests concurrently.
That is also fine.
Quoted 10 lines
From my point of view, this is better than allocating variable-sized buffers upfront because:
a) entries are fully maximized. With variable-sized buffers, the big buffers would be reserved specifically for payload requests while the small buffers would be reserved specifically for metadata requests. We could allocate '# entries' amount of small buffers, but for big buffers there would be less than '# entries'. If the server needs to service a lot of concurrent I/O requests, then the ring gets throttled on the limited number of big buffers available.Quoted 1 line
I would like to see something like 8K, 16K, 32K, 128K.
My worry is that for I/O heavy workloads with large read/write payloads (eg client access patterns reading/writing MBs at a time), the limited number of big enough buffers becomes the throttling bottleneck.
Quoted 11 lines
Quoted 9 lines
b) it best maximizes buffer memory. A request could need a buffer of any size so with variable-sized buffers, there's extra space in the buffer that is still being wasted. For example, for large payload requests, the big buffers would need to be the size of the max payload size (eg default 1 MB) but a lot of requests will fall under that. With incremental buffer consumption, only however many bytes used by the request are reserved in the buffer.
Doesn't that cause fragmentation?
With incremetnal buffer consumption, there's not fragmentation in the classical sense (eg scattered unusable holes). The buffer gets recycled back into the ring as a whole once all the requests in it have completed (tracked by refcounting).
I think the concern is that if the server is very slow to fulfill requests and the workload pattern has it so that slow requests are packed into the same buffer as fast requests across all the buffers in the queue and that queue has all its buffers saturated, then the next buffer is available only once the slow request has completed. We can mitigate this by assigning the request to a queue on the nearest numa node as a fallback if we detect that case. We could also do the same thing to mitigate the variable-sized buffer scenario where there's not enough big buffers for the queue, but I think that logic ends up a bit more complex.
I think overall we're able to support both incremental buffer consumption + variable-sized buffers if there's a need for it in the future where the server would like to choose.
Quoted 16 lines
Quoted 7 lines
c) there's no overhead with having to (as you pointed out) keep the buffers tracked and sorted into per-sized lists. If we wanted to use variable-sized buffers with kbufs instead of using incremental buffer consumption, the best way to do that would be to allocate a separate kbufring to support payload requests vs metadata requests.
Yeah, I had thought of multiple kbuf rings, with different sizes.
Quoted 6 lines
Quoted 5 lines
register entries with different sizes, which would then get sorted into per size lists. Now with kbuf that will not work anymore and we need different kbuf sizes. But then kbuf is not suitable for non-privileged users. So in order to support different request sizes one basically has
Quoted 6 lines
Quoted 2 lines
Non-privileged fuse servers use kbufs as well. It's only zero-copying that is not possible for non-privileged servers.
Non-privileged cannot pin, at least by default mlock size is 8MB. I was under the impression that kbuf would be always pinned, but I need to read over it again.
Quoted 5 lines
The kbufs get accounted to the user's mlock usage (this happens in
__io_account_mem()). If the user running the unprivileged server doesn't belong to a group that has high enough mlock limits, they'll have to use regular fuse over-io-uring buffers instead of kbufs for most of their queues.
Quoted 3 lines
That is exactly what I mean - in reality unprivileged servers will not be able to use kbufs. And there it would be good, if that server could use unpinned pbufs.
I think it's fundamentally wrong to use pbufs here. The buffers are completely under the sovereignty of userspace. If userspace chooses to free the buffers at any point or reuse the buffer for other purposes (or a free() and later malloc() ends up mapping to the same adddres), the kernel should not still be accessing the data in that buffer. When it's userspace that recycles back the buffer, this isn't a problem because userspace maintains direct control of the lifecycle and kernel usage of the buffer (eg if the buffer shouldn't be used by the kernel anymore since it's freed, then userspace simply won't add it back to the ring). This gets worse with incremental buffer consumption where multiple requests now simultaneously use the buffer - the server can't know when the kernel finishes reading each sub-region and on the kernel side, refcount tracking the pbuf becomes impractical. The pbuf buffers are a shared mapping that userspace can dynamically modify at any time, they can add and remove buffers from the ring at will. We can't add the refcount to the backing buffer struct directly since that is modifiable by userspace; we would have to track it separately in its own kernel-private list, but keeping that list in sync is difficult when userspace can add or remove buffers through the shared mapping at any time invisibly to the kernel.
I agree that for unprivileged servers constrained by low mlock limits, it's annoying that they can't use these features, but I don't think pbufs are the right direction. I think for those unprivileged servers, if the memory consumption is a pressing issue, then we should either just add the variable-sized buffers you mentioned or the servers should just use fewer queues and entries and accept the tradeoff that running unprivileged has a lower performance ceiling.
Thanks,
Joanne
Quoted 4 lines
Thanks,
Bernd
Quoted 6 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.
Quoted 7 lines
There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
I'm looking at the next-20260324 tree and trying to understand what happens if normal userspace (without FUSE) tries to use IOBL_KERNEL_MANAGED.
For a buffer list with IOBL_KERNEL_MANAGED, io_ring_buffer_select() will write a kernel pointer into sel.kaddr, but nothing in this series seems to ever read out of sel.kaddr; that only happens in fuse_uring_select_buffer() in the FUSE series
(https://github.com/joannekoong/linux/commit/fae19be22ab629b1301f37f2a942d5d84b45cc5c).
Instead, looking at the reverse call graph of io_ring_buffer_select():
io_ring_buffer_select
io_buffer_select
io_recvmsg
io_recv_buf_select
__io_import_rw_buffer
- io_recvmsg() passes sel.addr into iov_iter_ubuf(), which creates an ITER_UBUF
- io_recv_buf_select() passes sel.addr into import_ubuf(), which does
an access_ok() check before creating an ITER_UBUF
- __io_import_rw_buffer() also passes sel.addr into import_ubuf()
I think that means io_recv_buf_select() and __io_import_rw_buffer() will first access the union through the wrong member, then fail on access_ok().io_recvmsg() will create an ITER_UBUF pointing to kernel memory (which AFAIK isn't supposed to happen?), which I think will then cause a later failure when you actually try to access the iterator (because copy_to_user_iter() checks access_ok()).
Am I missing something that prevents normal io_uring operations from grabbing IOBL_KERNEL_MANAGED buffers and accessing the wrong union member?
Quoted 47 lines · Jann Horn
Quoted 6 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings where the kernel allocates and manages the buffers on behalf of the application. From the uapi side, this goes through the pbuf ring interface, through theIOU_PBUF_RING_KERNEL_MANAGEDflag.Quoted 7 lines
There was a long discussion with Pavel on v1 [1] regarding the design. The alternatives were to have the buffers allocated and registered through a memory region or through the registered buffers interface and have fuse implement ring buffer logic internally outside of io-uring. However, because the buffers need to be contiguous for DMA and some high-performance fuse servers may need non-fuse io-uring requests to use the buffer ring directly, v3 keeps the design.
I'm looking at the next-20260324 tree and trying to understand what happens if normal userspace (without FUSE) tries to use
IOBL_KERNEL_MANAGED.For a buffer list with
IOBL_KERNEL_MANAGED,io_ring_buffer_select()will write a kernel pointer into sel.kaddr, but nothing in this series seems to ever read out of sel.kaddr; that only happens infuse_uring_select_buffer()in the FUSE series
(https://github.com/joannekoong/linux/commit/fae19be22ab629b1301f37f2a942d5d84b45cc5c).Instead, looking at the reverse call graph of
io_ring_buffer_select():
io_ring_buffer_select
io_buffer_select
io_recvmsg
io_recv_buf_select
__io_import_rw_buffer-
io_recvmsg()passes sel.addr intoiov_iter_ubuf(), which creates anITER_UBUF
-io_recv_buf_select()passes sel.addr intoimport_ubuf(), which does
anaccess_ok()check before creating anITER_UBUF
-__io_import_rw_buffer()also passes sel.addr intoimport_ubuf()I think that means
io_recv_buf_select()and__io_import_rw_buffer()will first access the union through the wrong member, then fail onaccess_ok().io_recvmsg()will create anITER_UBUFpointing to kernel memory (which AFAIK isn't supposed to happen?), which I think will then cause a later failure when you actually try to access the iterator (becausecopy_to_user_iter()checksaccess_ok()).
Quoted 3 lines
Am I missing something that prevents normal
io_uringoperations from grabbingIOBL_KERNEL_MANAGEDbuffers and accessing the wrong union member?
Hi Jann,
I am going to be submitting the changes for kernel-managed pbufring compatibility with normal (non-fuse) io-uring requests as part of a separate patchset. You're right that there is a functional gap right now where trying to use kernel-managed pbuf rings fails with errors. In those patches, an iter_kvec will be constructed for IOBL_KERNEL_MANAGED rings instead of an iter_ubuf. I'm intending to submit that patchset upstream in time for the 7.1 merge window before it closes in mid-April.
Thanks,
Joanne
Hi!
Quoted 7 lines · Joanne Koong
Quoted 3 lines · Jann Horn
Am I missing something that prevents normal
io_uringoperations from grabbingIOBL_KERNEL_MANAGEDbuffers and accessing the wrong union member?Hi Jann,
Quoted 8 lines
I am going to be submitting the changes for kernel-managed pbufring compatibility with normal (non-fuse) io-uring requests as part of a separate patchset. You're right that there is a functional gap right now where trying to use kernel-managed pbuf rings fails with errors. In those patches, an
iter_kvecwill be constructed forIOBL_KERNEL_MANAGEDrings instead of aniter_ubuf. I'm intending to submit that patchset upstream in time for the 7.1 merge window before it closes in mid-April.
Ah, thanks for the explanation. Please CC me on that patchset, I'd be interested in taking a look at how that will be implemented.
Thanks,
Jann
Quoted 20 lines · Jann Horn
Hi!
Quoted 7 lines · Joanne Koong
Quoted 3 lines · Jann Horn
Am I missing something that prevents normal
io_uringoperations from grabbingIOBL_KERNEL_MANAGEDbuffers and accessing the wrong union member?Hi Jann,
Quoted 8 lines
I am going to be submitting the changes for kernel-managed pbufring compatibility with normal (non-fuse) io-uring requests as part of a separate patchset. You're right that there is a functional gap right now where trying to use kernel-managed pbuf rings fails with errors. In those patches, an
iter_kvecwill be constructed forIOBL_KERNEL_MANAGEDrings instead of aniter_ubuf. I'm intending to submit that patchset upstream in time for the 7.1 merge window before it closes in mid-April.
Quoted 2 lines
Ah, thanks for the explanation. Please CC me on that patchset, I'd be interested in taking a look at how that will be implemented.
I will make sure to CC you on the patchset.
Thanks for taking a look at this.
Thanks,
Joanne
Quoted 3 lines
Thanks,
Jann