io_uring: add kernel-managed buffer rings
v2 cover io_uring: add kernel-managed buffer rings
Currently, io_uring buffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings, where the kernel allocates and manages the buffers on behalf of the application.
This is split out from the fuse over io_uring series in [1], which needs the kernel to own and manage buffers shared between the fuse server and the kernel.
This series is on top of commit 73cf88d775b1f55 in the for-next branch in Jens' io-uring tree. The corresponding liburing changes are in [2] and will be submitted after the changes in this patchset are accepted.
There was a discussion on v1 about having kernel-managed buffer rings go through a user-provided registered memory region. This changes proposed in this patchset add a simple straightforward interface where the kernel allocates the buffers and the buffers are tied to the lifecycle of the ring, which suffices for the majority of use cases. If/when in the future it is useful for the buffers to be backed by a prior user registered mem region (eg for PMD optimization gains), the changes in this patchset do not preclude support for that from being added.
The link to the fuse commits that use the changes in this series is in [3].
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://github.com/joannekoong/liburing/commits/pbuf_kernel_managed/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy/
Changelog
---------
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t
* Incorporate Jens' feedback, including fixing wraparound int promotion bug
* memmap: drop allocation per buf + have everything go through io_create_region (Pavel), add 2MB chunking workaround for large allocations
* uapi: merge kmbuf into pbuf interface/apis as IOU_PBUF_RING_KERNEL_MANAGED flag (Pavel)
Changes since [1]:
* add "if (bl)" check for recycling API (Bernd)
* check mul overflow, use GFP_USER, use PTR as return type (Christoph)
* fix bl->ring leak (me)
Joanne Koong (9):
io_uring/memmap: chunk allocations in io_region_allocate_pages() io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: return buffer id in buffer selection io_uring/kbuf: add recycling for kernel managed buffer rings io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: export io_ring_buffer_select() io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
include/linux/io_uring/cmd.h | 53 ++++++- include/linux/io_uring_types.h | 10 +- include/uapi/linux/io_uring.h | 14 +- io_uring/kbuf.c | 248 +++++++++++++++++++++++++++++---- io_uring/kbuf.h | 11 +- io_uring/memmap.c | 87 +++++++++--- io_uring/uring_cmd.c | 6 +- 7 files changed, 373 insertions(+), 56 deletions(-)
Signature · 1 line
2.47.3
PATCH v2 1/9 io_uring/memmap: chunk allocations in io_region_allocate_pages()
Currently, io_region_allocate_pages() tries a single compound allocation for the entire region, and falls back to alloc_pages_bulk_node() if that fails.
When allocating a large region, trying to do a single compound allocation may be unrealistic while allocating page by page may be inefficient and cause worse TLB performance.
Rework io_region_allocate_pages() to allocate memory in 2MB chunks, attempting a compound allocation for each chunk.
Replace IO_REGION_F_SINGLE_REF with IO_REGION_F_COMPOUND_PAGES to reflect that the page array may contain tail pages from multiple compound allocations.
Currently, alloc_pages_bulk_node() fails when the GFP_KERNEL_ACCOUNT gfp flag is set. This makes this commit a necessary change in order to use kernel-managed ring buffers (which will allocate regions of large sizes), at least until that issue is fixed.
--- io_uring/memmap.c | 87 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 64 insertions(+), 23 deletions(-)
io_uring/memmap.c +64 -23
@@ -15,6 +15,28 @@#include "rsrc.h" #include "zcrx.h" +static void release_compound_pages (struct page **pages,unsigned long nr_pages) +{ +struct page *page; +unsigned int nr, i =0 ; + +while (nr_pages) { + page = pages[i]; + +if (!page ||WARN_ON_ONCE (page !=compound_head (page))) +return ; + + nr =compound_nr (page); +put_page (page); + +if (nr >= nr_pages) +return ; + + i += nr; + nr_pages -= nr; + } +} +static bool io_mem_alloc_compound (struct page **pages,int nr_pages,size_t size,gfp_t gfp) {
@@ -84,22 +106,19 @@ enum {IO_REGION_F_VMAP =1 ,/* memory is provided by user and pinned by the kernel */ IO_REGION_F_USER_PROVIDED =2 , -/* only the first page in the array is ref'ed */ - IO_REGION_F_SINGLE_REF = 4, +/* memory may contain tail pages from compound allocations */ + IO_REGION_F_COMPOUND_PAGES = 4, };void io_free_region (struct user_struct *user,struct io_mapped_region *mr) {if (mr->pages) { -long nr_refs = mr->nr_pages; - -if (mr->flags &IO_REGION_F_SINGLE_REF ) - nr_refs =1 ; -if (mr->flags &IO_REGION_F_USER_PROVIDED ) -unpin_user_pages (mr->pages, nr_refs); +unpin_user_pages (mr->pages, mr->nr_pages); +else if (mr->flags &IO_REGION_F_COMPOUND_PAGES ) +release_compound_pages (mr->pages, mr->nr_pages);else -release_pages (mr->pages, nr_refs); +release_pages (mr->pages, mr->nr_pages);kvfree (mr->pages); }
@@ -154,28 +173,50 @@ static int io_region_allocate_pages(struct io_mapped_region *mr, unsigned long mmap_offset) {gfp_t gfp =GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN; -size_t size =io_region_size (mr);unsigned long nr_allocated; -struct page **pages; +struct page **pages, **cur_pages; +unsigned chunk_size, chunk_nr_pages; +unsigned int pages_left; pages =kvmalloc_array (mr->nr_pages,sizeof (*pages), gfp);if (!pages)return -ENOMEM ; -if (io_mem_alloc_compound (pages, mr->nr_pages, size, gfp)) { - mr->flags |=IO_REGION_F_SINGLE_REF ; - goto done; - } + chunk_size =SZ_2M ; + chunk_nr_pages = chunk_size >>PAGE_SHIFT ; + pages_left = mr->nr_pages; + cur_pages = pages; + +while (pages_left) { +unsigned int nr_pages =min (pages_left, + chunk_nr_pages); + +if (io_mem_alloc_compound (cur_pages, nr_pages, + nr_pages <<PAGE_SHIFT , gfp)) { + mr->flags |=IO_REGION_F_COMPOUND_PAGES ; + cur_pages += nr_pages; + pages_left -= nr_pages; +continue ; + } - nr_allocated =alloc_pages_bulk_node (gfp,NUMA_NO_NODE , - mr->nr_pages, pages); -if (nr_allocated != mr->nr_pages) { -if (nr_allocated) -release_pages (pages, nr_allocated); -kvfree (pages); -return -ENOMEM ; + nr_allocated =alloc_pages_bulk_node (gfp,NUMA_NO_NODE , + nr_pages, cur_pages); +if (nr_allocated != nr_pages) { +unsigned int total = + (cur_pages - pages) + nr_allocated; + +if (mr->flags &IO_REGION_F_COMPOUND_PAGES ) +release_compound_pages (pages, total); +else +release_pages (pages, total); +kvfree (pages); +return -ENOMEM ; + } + + cur_pages += nr_pages; + pages_left -= nr_pages; } -done: + reg->mmap_offset = mmap_offset; mr->pages = pages;return 0 ;
Signature · 1 line
2.47.3
PATCH v2 2/9 io_uring/kbuf: add support for kernel-managed buffer rings
In reply to Joanne KoongAdd support for kernel-managed buffer rings, which allow the kernel to allocate and manage the backing buffers for a buffer ring, rather than requiring the application to provide and manage them.
Internally, the IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.
At the uapi level, kernel-managed buffer rings are created through the pbuf interface with the IOU_PBUF_RING_KERNEL_MANAGED flag set. The io_uring_buf_reg struct is modified to allow taking in a buf_size instead of a ring_addr. To create a kernel-managed buffer ring, the caller must set the IOU_PBUF_RING_MMAP flag as well to indicate that the kernel will allocate the memory for the ring. When the caller mmaps the ring, they will get back a virtual mapping to the buffer memory.
--- include/uapi/linux/io_uring.h | 14 +++++- io_uring/kbuf.c | 95 +++++++++++++++++++++++++++++------ io_uring/kbuf.h | 6 ++- 3 files changed, 97 insertions(+), 18 deletions(-)
include/uapi/linux/io_uring.h +13 -1
@@ -885,15 +885,27 @@ struct io_uring_buf_ring {* use of it will consume only as much as it needs. This * requires that both the kernel and application keep * track of where the current read/recv index is at. +* IOU_PBUF_RING_KERNEL_MANAGED: If set, kernel allocates the memory for the +* ring and its buffers. The application must set the +* buffer size through reg->buf_size. The buffers are +* recycled by the kernel. IOU_PBUF_RING_MMAP must be set +* as well. When the caller makes a subsequent mmap call, +* the virtual mapping returned is a contiguous mapping of +* the buffers. IOU_PBUF_RING_INC is not yet supported. */ enum io_uring_register_pbuf_ring_flags {IOU_PBUF_RING_MMAP =1 ,IOU_PBUF_RING_INC =2 , +IOU_PBUF_RING_KERNEL_MANAGED =4 , };/* argument for IORING_(UN)REGISTER_PBUF_RING */ struct io_uring_buf_reg { -__u64 ring_addr; +union { +__u64 ring_addr; +/* used if reg->flags & IOU_PBUF_RING_KERNEL_MANAGED */ +__u32 buf_size; + };__u32 ring_entries;__u16 bgid;__u16 flags;
io_uring/kbuf.c +80 -15
141 lines not on this page · raw message
io_uring/kbuf.h +4 -2
@@ -7,9 +7,11 @@enum {/* ring mapped provided buffers */ -IOBL_BUF_RING =1 , +IOBL_BUF_RING =1 ,/* buffers are consumed incrementally rather than always fully */ -IOBL_INC =2 , +IOBL_INC =2 , +/* buffers are kernel managed */ +IOBL_KERNEL_MANAGED =4 , };struct io_buffer_list {
Signature · 1 line
2.47.3
PATCH v2 3/9 io_uring/kbuf: support kernel-managed buffer rings in buffer selection
In reply to Joanne KoongAllow kernel-managed buffers to be selected. This requires modifying the io_br_sel struct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.
Auto-commit any selected kernel-managed buffer.
--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 16 ++++++++++++---- 2 files changed, 16 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +4 -4
@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };
io_uring/kbuf.c +12 -4
@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the
@@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true; -/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD )return true;return false;
@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; - sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr); +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; }
Signature · 1 line
2.47.3
PATCH v2 4/9 io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Joanne KoongAdd kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.
This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
--- include/linux/io_uring/cmd.h | 17 +++++++++++ io_uring/kbuf.c | 55 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 5 ++++ 3 files changed, 77 insertions(+)
include/linux/io_uring/cmd.h +17 -0
@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,bool io_uring_mshot_cmd_post_cqe (struct io_uring_cmd *ioucmd,struct io_br_sel *sel,unsigned int issue_flags); +int io_uring_buf_ring_pin (struct io_uring_cmd *cmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **out_bl); +int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd,unsigned buf_group, +unsigned issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, {return true; } +static inline int io_uring_buf_ring_pin (struct io_uring_cmd *cmd, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +55 -0
73 lines not on this page · raw message
io_uring/kbuf.h +5 -0
@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {
Signature · 1 line
2.47.3
PATCH v2 5/9 io_uring/kbuf: return buffer id in buffer selection
In reply to Joanne KoongReturn the id of the selected buffer in io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.
--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +1 -1
@@ -71,7 +71,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);/* * Select a buffer from the provided buffer group for multishot uring_cmd. -* Returns the selected buffer address and size. +* Returns the selected buffer address, size, and id. */ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, unsigned buf_group, size_t *len,
include/linux/io_uring_types.h +2 -0
@@ -100,6 +100,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id; };
io_uring/kbuf.c +5 -2
@@ -206,6 +206,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; + sel.buf_id = req->buf_index;if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)(uintptr_t )READ_ONCE (buf->addr);else
@@ -229,10 +230,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index; + } }io_ring_submit_unlock (req->ctx, issue_flags);return sel;
Signature · 1 line
2.47.3
PATCH v2 6/9 io_uring/kbuf: add recycling for kernel managed buffer rings
In reply to Joanne KoongAdd an interface for buffers to be recycled back into a kernel-managed buffer ring.
--- include/linux/io_uring/cmd.h | 11 +++++++++ io_uring/kbuf.c | 48 ++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
include/linux/io_uring/cmd.h +11 -0
@@ -88,6 +88,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **out_bl);int io_uring_buf_ring_unpin (struct io_uring_cmd *cmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd,unsigned int buf_group, +u64 addr,unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +147,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +48 -0
54 lines not on this page · raw message
Signature · 1 line
2.47.3
PATCH v2 7/9 io_uring/kbuf: add io_uring_is_kmbuf_ring()
In reply to Joanne Koongio_uring_is_kmbuf_ring() returns true if there is a kernel-managed buffer ring at the specified buffer group.
This is a preparatory patch for upcoming fuse kernel-managed buffer support, which needs to ensure the buffer ring registered by the server is a kernel-managed buffer ring.
--- include/linux/io_uring/cmd.h | 9 +++++++++ io_uring/kbuf.c | 20 ++++++++++++++++++++ 2 files changed, 29 insertions(+)
include/linux/io_uring/cmd.h +9 -0
@@ -92,6 +92,9 @@ int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,int io_uring_kmbuf_recycle (struct io_uring_cmd *cmd,unsigned int buf_group,u64 addr,unsigned int len,unsigned int bid,unsigned int issue_flags); + +bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -154,6 +157,12 @@ static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd, +unsigned int buf_group, +unsigned int issue_flags) +{ +return false; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +20 -0
@@ -917,3 +917,23 @@ struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx,return NULL ;return &bl->region; } + +bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct io_buffer_list *bl; +bool is_kmbuf_ring = false; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) && (bl->flags &IOBL_KERNEL_MANAGED )) { +WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )); + is_kmbuf_ring = true; + } + +io_ring_submit_unlock (ctx, issue_flags); +return is_kmbuf_ring; +} +EXPORT_SYMBOL_GPL (io_uring_is_kmbuf_ring);
Signature · 1 line
2.47.3
PATCH v2 8/9 io_uring/kbuf: export io_ring_buffer_select()
In reply to Joanne KoongExport io_ring_buffer_select() so that it may be used by callers who pass in a pinned bufring without needing to grab the io_uring mutex.
This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
--- include/linux/io_uring/cmd.h | 14 ++++++++++++++ io_uring/kbuf.c | 7 ++++--- 2 files changed, 18 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +14 -0
@@ -95,6 +95,10 @@ int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,bool io_uring_is_kmbuf_ring (struct io_uring_cmd *cmd,unsigned int buf_group,unsigned int issue_flags); + +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -163,6 +167,16 @@ static inline bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, {return false; } +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; +return sel; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +4 -3
@@ -230,9 +230,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,return false; } -static struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -unsigned int issue_flags) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) {struct io_uring_buf_ring *br = bl->buf_ring;__u16 tail, head = bl->head;
@@ -266,6 +266,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }return sel; } +EXPORT_SYMBOL_GPL (io_ring_buffer_select);struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags)
Signature · 1 line
2.47.3
PATCH v2 9/9 io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
In reply to Joanne KoongWhen uring_cmd operations select a buffer, the completion queue entry should indicate which buffer was selected.
Set IORING_CQE_F_BUFFER on the completed entry and encode the buffer index if a buffer was selected.
This change is needed in order to relay to userspace which selected buffer contains the data.
--- io_uring/uring_cmd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/uring_cmd.c +5 -1
@@ -151,6 +151,7 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); +u32 cflags =0 ;if (WARN_ON_ONCE (req->flags &REQ_F_APOLL_MULTISHOT ))return ;
@@ -160,7 +161,10 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (ret <0 )req_set_fail (req); -io_req_set_res (req, ret,0 ); +if (req->flags & (REQ_F_BUFFER_SELECTED |REQ_F_BUFFER_RING )) + cflags |=IORING_CQE_F_BUFFER | + (req->buf_index <<IORING_CQE_BUFFER_SHIFT ); +io_req_set_res (req, ret, cflags);if (is_cqe32) {if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;
Signature · 1 line
2.47.3
syzbot ci has tested the following series
syzbot ci has tested the following series
[v2] io_uring: add kernel-managed buffer rings
https://lore.kernel.org/all/[email protected]
* [PATCH v2 1/9] io_uring/memmap: chunk allocations in io_region_allocate_pages()
* [PATCH v2 2/9] io_uring/kbuf: add support for kernel-managed buffer rings
* [PATCH v2 3/9] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
* [PATCH v2 4/9] io_uring/kbuf: add buffer ring pinning/unpinning
* [PATCH v2 5/9] io_uring/kbuf: return buffer id in buffer selection
* [PATCH v2 6/9] io_uring/kbuf: add recycling for kernel managed buffer rings
* [PATCH v2 7/9] io_uring/kbuf: add io_uring_is_kmbuf_ring()
* [PATCH v2 8/9] io_uring/kbuf: export io_ring_buffer_select()
* [PATCH v2 9/9] io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
and found the following issue:
general protection fault in io_remove_buffers_legacy
Full report is available here:
https://ci.syzbot.org/series/ddeaf464-c69b-4166-b0cf-53c9d51e4820
***
general protection fault in io_remove_buffers_legacy
tree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: 2961f841b025fb234860bac26dfb7fa7cb0fb122 arch: amd64 compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8 config: https://ci.syzbot.org/builds/ab5ad5aa-2757-4d66-a2c5-391a8417535d/config C repro: https://ci.syzbot.org/findings/061747e2-36f1-499b-ac34-38cefffbce63/c_repro syz repro: https://ci.syzbot.org/findings/061747e2-36f1-499b-ac34-38cefffbce63/syz_repro
Oops: general protection fault, probably for non-canonical address 0xdffffc0000000001: 0000 [#1] SMP KASAN PTI KASAN: null-ptr-deref in range [0x0000000000000008-0x000000000000000f] CPU: 1 UID: 0 PID: 5967 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:__list_del_entry_valid_or_report+0x25/0x190 lib/list_debug.c:49 Code: 90 90 90 90 90 f3 0f 1e fa 41 57 41 56 41 55 41 54 53 48 89 fb 49 bd 00 00 00 00 00 fc ff df 48 83 c7 08 48 89 f8 48 c1 e8 03 <42> 80 3c 28 00 74 05 e8 df 8c 77 fd 4c 8b 7b 08 48 89 d8 48 c1 e8 RSP: 0018:ffffc900040a7b68 EFLAGS: 00010202 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 1ffff11035ee2732 RDX: 1ffff11035ee2730 RSI: 00000000ffffffff RDI: 0000000000000008 RBP: dffffc0000000000 R08: ffff8881af7139b7 R09: 0000000000000000 R10: ffff8881af7139a0 R11: ffffed1035ee2737 R12: ffff8881af713980 R13: dffffc0000000000 R14: 00000000ffffffff R15: 0000000000000000 FS: 0000555560587500(0000) GS:ffff8882a9466000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000200000001000 CR3: 0000000175cd2000 CR4: 00000000000006f0 Call Trace: <TASK> __list_del_entry_valid include/linux/list.h:132 [inline] __list_del_entry include/linux/list.h:223 [inline] list_del include/linux/list.h:237 [inline] io_remove_buffers_legacy+0x139/0x310 io_uring/kbuf.c:533 io_put_bl+0x62/0x120 io_uring/kbuf.c:548 io_register_pbuf_ring+0x6c0/0x7d0 io_uring/kbuf.c:855 __io_uring_register io_uring/register.c:838 [inline] __do_sys_io_uring_register io_uring/register.c:1024 [inline] __se_sys_io_uring_register+0xc3e/0x19a0 io_uring/register.c:1001 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0x14d/0xf80 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f056859bf79 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fffd8dcfaf8 EFLAGS: 00000246 ORIG_RAX: 00000000000001ab RAX: ffffffffffffffda RBX: 00007f0568815fa0 RCX: 00007f056859bf79 RDX: 0000200000000040 RSI: 0000000000000016 RDI: 0000000000000004 RBP: 00007f05686327e0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f0568815fac R14: 00007f0568815fa0 R15: 00007f0568815fa0 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:__list_del_entry_valid_or_report+0x25/0x190 lib/list_debug.c:49 Code: 90 90 90 90 90 f3 0f 1e fa 41 57 41 56 41 55 41 54 53 48 89 fb 49 bd 00 00 00 00 00 fc ff df 48 83 c7 08 48 89 f8 48 c1 e8 03 <42> 80 3c 28 00 74 05 e8 df 8c 77 fd 4c 8b 7b 08 48 89 d8 48 c1 e8 RSP: 0018:ffffc900040a7b68 EFLAGS: 00010202 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 1ffff11035ee2732 RDX: 1ffff11035ee2730 RSI: 00000000ffffffff RDI: 0000000000000008 RBP: dffffc0000000000 R08: ffff8881af7139b7 R09: 0000000000000000 R10: ffff8881af7139a0 R11: ffffed1035ee2737 R12: ffff8881af713980 R13: dffffc0000000000 R14: 00000000ffffffff R15: 0000000000000000 FS: 0000555560587500(0000) GS:ffff8882a9466000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f25f1e17095 CR3: 0000000175cd2000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess): 0: 90 nop 1: 90 nop 2: 90 nop 3: 90 nop 4: 90 nop 5: f3 0f 1e fa endbr64 9: 41 57 push %r15 b: 41 56 push %r14 d: 41 55 push %r13 f: 41 54 push %r12 11: 53 push %rbx 12: 48 89 fb mov %rdi,%rbx 15: 49 bd 00 00 00 00 00 movabs $0xdffffc0000000000,%r13 1c: fc ff df 1f: 48 83 c7 08 add $0x8,%rdi 23: 48 89 f8 mov %rdi,%rax 26: 48 c1 e8 03 shr $0x3,%rax * 2a: 42 80 3c 28 00 cmpb $0x0,(%rax,%r13,1) <-- trapping instruction 2f: 74 05 je 0x36 31: e8 df 8c 77 fd call 0xfd778d15 36: 4c 8b 7b 08 mov 0x8(%rbx),%r15 3a: 48 89 d8 mov %rbx,%rax 3d: 48 rex.W 3e: c1 .byte 0xc1 3f: e8 .byte 0xe8
***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Quoted 22 lines · Joanne Koong
Currently,
io_uringbuffer rings require the application to allocate and manage the backing buffers. This series introduces buffer rings, where the kernel allocates and manages the buffers on behalf of the application.This is split out from the fuse over
io_uringseries in [1], which needs the kernel to own and manage buffers shared between the fuse server and the kernel.This series is on top of commit
73cf88d775b1f55in the for-next branch in Jens' io-uring tree. The corresponding liburing changes are in [2] and will be submitted after the changes in this patchset are accepted.There was a discussion on v1 about having kernel-managed buffer rings go through a user-provided registered memory region. This changes proposed in this patchset add a simple straightforward interface where the kernel allocates the buffers and the buffers are tied to the lifecycle of the ring, which suffices for the majority of use cases. If/when in the future it is useful for the buffers to be backed by a prior user registered mem region (eg for PMD optimization gains), the changes in this patchset do not preclude support for that from being added.
Quoted 4 lines
The link to the fuse commits that use the changes in this series is in [3].
Thanks,
Joanne
I'm going to update v2 with another version, as per the conversation in this thread [1].
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/T/#t