fuse/io-uring: add kernel-managed buffer rings and zero-copy
Subsystems (7)
Joanne Koong
maintainerv3 cover fuse/io-uring: add kernel-managed buffer rings and zero-copy
This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.
On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.
The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.
There was a previous submission for supporting registered buffers in fuse [1] but that was abandoned in favor of using kernel-managed buffer rings, which, once incremental buffer consumption is added in a later patchset, gives significant memory usage advantages in allowing the full buffer capacity to be utilized across multiple requests, as well as offers more flexibility for future additions. As well, it also makes the userspace side setup simpler. The relevant refactoring fuse patches from the previous submission are carried over into this one.
Benchmarks for zero-copy (patch 24) show approximately the following differences in throughput for bs=1M:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s) buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)
The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
This series is on top of commit 40fbbd64bba6 in the io-uring tree.
The libfuse patch used for testing / verifying functionality is in [2]. This has a dependency on the liburing changes in [3].
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://github.com/joannekoong/libfuse/commit/f15094b1881f9488b45026ae51f18d13ced4a554
[3] https://github.com/joannekoong/liburing/tree/kmbuf
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* fix casting between void * and u64 for 32-bit architectures (kernel test robot)
* add newline for documentation bullet points (kernel test robot)
* fix unrecognized "boolean" (kernel test robot), switch it to a flag (me)
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* drop fuse buffer cleanup on ring death, which makes things a lot simpler (Jens)
- this drops a lot of things (eg needing ring_ctx tracking, needing callback for ring death, etc)
* drop fixed buffer pinning altogether and just do lookup every time (Jens) (didn't significantly affect the benchmark results seen)
* fix spelling mistake in docs (Askar)
* use -EALREADY for pinning already pinned bufring, return PTR_ERR for registration instead of err, move initializations to outside locks (Caleb)
* drop fuse patches for zero-ed out headers (me)
Joanne Koong (25):
io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic
helpers
io_uring/kbuf: rename io_unregister_pbuf_ring() to
io_unregister_buf_ring()
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: add mmap support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: add recycling for kernel managed buffer rings
io_uring: add io_uring_cmd_fixed_index_get() and
io_uring_cmd_fixed_index_put()
io_uring/kbuf: add io_uring_cmd_is_kmbuf_ring() io_uring/kbuf: export io_ring_buffer_select() io_uring/kbuf: return buffer id in buffer selection io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: support buffer copying for kernel addresses
fuse: add io-uring kernel-managed buffer ring io_uring/rsrc: rename
io_buffer_register_bvec()/io_buffer_unregister_bvec()
io_uring/rsrc: split io_buffer_register_request() logic io_uring/rsrc: Allow buffer release callback to be optional io_uring/rsrc: add io_buffer_register_bvec()
fuse: add zero-copy over io-uring
docs: fuse: add io-uring bufring and zero-copy documentation
Documentation/block/ublk.rst | 14 +- .../filesystems/fuse/fuse-io-uring.rst | 59 +- drivers/block/ublk_drv.c | 18 +- fs/fuse/dev.c | 30 +- fs/fuse/dev_uring.c | 711 ++++++++++++++---- fs/fuse/dev_uring_i.h | 41 +- fs/fuse/fuse_dev_i.h | 8 +- include/linux/io_uring/buf.h | 25 + include/linux/io_uring/cmd.h | 99 ++- include/linux/io_uring_types.h | 10 +- include/uapi/linux/fuse.h | 17 +- include/uapi/linux/io_uring.h | 17 +- io_uring/kbuf.c | 350 +++++++-- io_uring/kbuf.h | 29 +- io_uring/memmap.c | 117 ++- io_uring/memmap.h | 4 + io_uring/register.c | 9 +- io_uring/rsrc.c | 190 ++++- io_uring/rsrc.h | 5 + io_uring/uring_cmd.c | 65 +- 20 files changed, 1541 insertions(+), 277 deletions(-) create mode 100644 include/linux/io_uring/buf.h
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 1/25 io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic helpers
Refactor the logic in io_register_pbuf_ring() into generic helpers:
- io_validate_buf_reg(): Validate user input and buffer registration parameters
- io_alloc_new_buffer_list(): Allocate and initialize a new buffer list for the given buffer group ID
- io_setup_pbuf_ring(): Sets up the physical buffer ring region and handles memory mapping for provided buffer rings
This is a preparatory change for upcoming kernel-managed buffer ring support which will need to reuse some of these helpers.
--- io_uring/kbuf.c | 123 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 82 insertions(+), 41 deletions(-)
io_uring/kbuf.c +82 -41
@@ -596,55 +596,71 @@ int io_manage_buffers_legacy(struct io_kiocb *req, unsigned int issue_flags)return IOU_COMPLETE ; } -int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +static int io_validate_buf_reg (struct io_uring_buf_reg *reg, +unsigned int permitted_flags) { -struct io_uring_buf_reg reg; -struct io_buffer_list *bl; -struct io_uring_region_desc rd; -struct io_uring_buf_ring *br; -unsigned long mmap_offset; -unsigned long ring_size; -int ret; - -lockdep_assert_held (&ctx->uring_lock); - -if (copy_from_user (®, arg,sizeof (reg))) -return -EFAULT ; -if (!mem_is_zero (reg.resv,sizeof (reg.resv))) +if (!mem_is_zero (reg->resv,sizeof (reg->resv)))return -EINVAL ; -if (reg.flags & ~(IOU_PBUF_RING_MMAP |IOU_PBUF_RING_INC )) +if (reg->flags & ~permitted_flags)return -EINVAL ; -if (!is_power_of_2 (reg.ring_entries)) +if (!is_power_of_2 (reg->ring_entries))return -EINVAL ;/* cannot disambiguate full vs empty due to head/tail size */ -if (reg.ring_entries >=65536 ) +if (reg->ring_entries >=65536 )return -EINVAL ; +return 0 ; +} - bl =io_buffer_get_list (ctx, reg.bgid); -if (bl) { +static int io_alloc_new_buffer_list (struct io_ring_ctx *ctx, +struct io_uring_buf_reg *reg, +struct io_buffer_list **bl) +{ +struct io_buffer_list *list; + + list =io_buffer_get_list (ctx, reg->bgid); +if (list) {/* if mapped buffer ring OR classic exists, don't allow */ -if (bl->flags &IOBL_BUF_RING || !list_empty (&bl->buf_list)) +if (list->flags &IOBL_BUF_RING || !list_empty (&list->buf_list))return -EEXIST ; -io_destroy_bl (ctx, bl); +io_destroy_bl (ctx, list); } - bl =kzalloc (sizeof (*bl),GFP_KERNEL_ACCOUNT ); -if (!bl) + list =kzalloc (sizeof (*list),GFP_KERNEL_ACCOUNT ); +if (!list)return -ENOMEM ; - mmap_offset = (unsigned long )reg.bgid <<IORING_OFF_PBUF_SHIFT ; - ring_size =flex_array_size (br, bufs, reg.ring_entries); + list->nr_entries = reg->ring_entries; + list->mask = reg->ring_entries -1 ; + list->flags =IOBL_BUF_RING ; + + *bl = list; + +return 0 ; +} + +static int io_setup_pbuf_ring (struct io_ring_ctx *ctx, +struct io_uring_buf_reg *reg, +struct io_buffer_list *bl) +{ +struct io_uring_region_desc rd; +unsigned long mmap_offset; +unsigned long ring_size; +int ret; + + mmap_offset = (unsigned long )reg->bgid <<IORING_OFF_PBUF_SHIFT ; + ring_size =flex_array_size (bl->buf_ring, bufs, reg->ring_entries);memset (&rd,0 ,sizeof (rd)); rd.size =PAGE_ALIGN (ring_size); -if (!(reg.flags &IOU_PBUF_RING_MMAP )) { - rd.user_addr = reg.ring_addr; +if (!(reg->flags &IOU_PBUF_RING_MMAP )) { + rd.user_addr = reg->ring_addr; rd.flags |=IORING_MEM_REGION_TYPE_USER ; } + ret =io_create_region (ctx, &bl->region, &rd, mmap_offset);if (ret) - goto fail; - br =io_region_get_ptr (&bl->region); +return ret; + bl->buf_ring =io_region_get_ptr (&bl->region);#ifdef SHM_COLOUR /*
@@ -656,25 +672,50 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)* should use IOU_PBUF_RING_MMAP instead, and liburing will handle * this transparently. */ -if (!(reg.flags &IOU_PBUF_RING_MMAP ) && - ((reg.ring_addr | (unsigned long )br) & (SHM_COLOUR -1 ))) { - ret = -EINVAL ; - goto fail; +if (!(reg->flags &IOU_PBUF_RING_MMAP ) && + ((reg->ring_addr | (unsigned long )bl->buf_ring) & + (SHM_COLOUR -1 ))) { +io_free_region (ctx->user, &bl->region); +return -EINVAL ; }#endif - bl->nr_entries = reg.ring_entries; - bl->mask = reg.ring_entries -1 ; - bl->flags |=IOBL_BUF_RING ; - bl->buf_ring = br; +return 0 ; +} + +int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +{ +unsigned int permitted_flags; +struct io_uring_buf_reg reg; +struct io_buffer_list *bl; +int ret; + +lockdep_assert_held (&ctx->uring_lock); + +if (copy_from_user (®, arg,sizeof (reg))) +return -EFAULT ; + + permitted_flags =IOU_PBUF_RING_MMAP |IOU_PBUF_RING_INC ; + ret =io_validate_buf_reg (®, permitted_flags); +if (ret) +return ret; + + ret =io_alloc_new_buffer_list (ctx, ®, &bl); +if (ret) +return ret; + + ret =io_setup_pbuf_ring (ctx, ®, bl); +if (ret) { +kfree (bl); +return ret; + } +if (reg.flags &IOU_PBUF_RING_INC ) bl->flags |=IOBL_INC ; +io_buffer_add_list (ctx, bl, reg.bgid); +return 0 ; -fail: -io_free_region (ctx->user, &bl->region); -kfree (bl); -return ret; }int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg)
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 2/25 io_uring/kbuf: rename io_unregister_pbuf_ring() to io_unregister_buf_ring()
In reply to Joanne KoongUse the more generic name io_unregister_buf_ring() as this function will be used for unregistering both provided buffer rings and kernel-managed buffer rings.
This is a preparatory change for upcoming kernel-managed buffer ring support.
--- io_uring/kbuf.c | 2 +- io_uring/kbuf.h | 2 +- io_uring/register.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
io_uring/kbuf.c +1 -1
@@ -718,7 +718,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } -int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg) {struct io_uring_buf_reg reg;struct io_buffer_list *bl;
io_uring/kbuf.h +1 -1
@@ -74,7 +74,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqeint io_manage_buffers_legacy (struct io_kiocb *req,unsigned int issue_flags);int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); -int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); +int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_register_pbuf_status (struct io_ring_ctx *ctx,void __user *arg);bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags);
io_uring/register.c +1 -1
@@ -750,7 +750,7 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, ret = -EINVAL ;if (!arg || nr_args !=1 )break ; - ret =io_unregister_pbuf_ring (ctx, arg); + ret =io_unregister_buf_ring (ctx, arg);break ;case IORING_REGISTER_SYNC_CANCEL : ret = -EINVAL ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 3/25 io_uring/kbuf: add support for kernel-managed buffer rings
In reply to Joanne KoongAdd support for kernel-managed buffer rings (kmbuf rings), which allow the kernel to allocate and manage the backing buffers for a buffer ring, rather than requiring the application to provide and manage them.
This introduces two new registration opcodes:
- IORING_REGISTER_KMBUF_RING: Register a kernel-managed buffer ring
- IORING_UNREGISTER_KMBUF_RING: Unregister a kernel-managed buffer ring
The existing io_uring_buf_reg structure is extended with a union to support both application-provided buffer rings (pbuf) and kernel-managed buffer rings (kmbuf):
- For pbuf rings: ring_addr specifies the user-provided ring address
- For kmbuf rings: buf_size specifies the size of each buffer. buf_size must be non-zero and page-aligned.
The implementation follows the same pattern as pbuf ring registration, reusing the validation and buffer list allocation helpers introduced in earlier refactoring. The IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.
--- include/uapi/linux/io_uring.h | 15 ++++- io_uring/kbuf.c | 76 +++++++++++++++++++++++ io_uring/kbuf.h | 7 ++- io_uring/memmap.c | 112 ++++++++++++++++++++++++++++++++++ io_uring/memmap.h | 4 ++ io_uring/register.c | 7 +++ 6 files changed, 217 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +13 -2
@@ -700,6 +700,10 @@ enum io_uring_register_op {/* auxiliary zcrx configuration, see enum zcrx_ctrl_op */ IORING_REGISTER_ZCRX_CTRL =36 , +/* register/unregister kernel-managed ring buffer group */ +IORING_REGISTER_KMBUF_RING =37 , +IORING_UNREGISTER_KMBUF_RING =38 , +/* this goes last */ IORING_REGISTER_LAST,
@@ -869,9 +873,16 @@ enum io_uring_register_pbuf_ring_flags { IOU_PBUF_RING_INC = 2, }; -/* argument for IORING_(UN)REGISTER_PBUF_RING */ +/* argument for IORING_(UN)REGISTER_PBUF_RING and +* IORING_(UN)REGISTER_KMBUF_RING +*/ struct io_uring_buf_reg { -__u64 ring_addr; +union { +/* used for pbuf rings */ +__u64 ring_addr; +/* used for kmbuf rings */ +__u32 buf_size; + };__u32 ring_entries;__u16 bgid;__u16 flags;
io_uring/kbuf.c +76 -0
79 lines not on this page · raw message
io_uring/kbuf.h +5 -2
@@ -7,9 +7,11 @@enum {/* ring mapped provided buffers */ -IOBL_BUF_RING =1 , +IOBL_BUF_RING =1 ,/* buffers are consumed incrementally rather than always fully */ -IOBL_INC =2 , +IOBL_INC =2 , +/* buffers are kernel managed */ +IOBL_KERNEL_MANAGED =4 , };struct io_buffer_list {
@@ -74,6 +76,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqeint io_manage_buffers_legacy (struct io_kiocb *req,unsigned int issue_flags);int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); +int io_register_kmbuf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_register_pbuf_status (struct io_ring_ctx *ctx,void __user *arg);
io_uring/memmap.c +112 -0
142 lines not on this page · raw message
io_uring/memmap.h +4 -0
@@ -21,6 +21,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr,struct io_uring_region_desc *reg, unsigned long mmap_offset); +int io_create_region_multi_buf (struct io_ring_ctx *ctx, +struct io_mapped_region *mr, +unsigned int nr_bufs,unsigned int buf_size); +static inline void *io_region_get_ptr (struct io_mapped_region *mr) {return mr->ptr;
io_uring/register.c +7 -0
@@ -746,7 +746,14 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_pbuf_ring (ctx, arg);break ; +case IORING_REGISTER_KMBUF_RING : + ret = -EINVAL ; +if (!arg || nr_args !=1 ) +break ; + ret =io_register_kmbuf_ring (ctx, arg); +break ;case IORING_UNREGISTER_PBUF_RING : +case IORING_UNREGISTER_KMBUF_RING : ret = -EINVAL ;if (!arg || nr_args !=1 )break ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 4/25 io_uring/kbuf: add mmap support for kernel-managed buffer rings
In reply to Joanne KoongAdd support for mmapping kernel-managed buffer rings (kmbuf) to userspace, allowing applications to access the kernel-allocated buffers.
Similar to application-provided buffer rings (pbuf), kmbuf rings use the buffer group ID encoded in the mmap offset to identify which buffer ring to map. The implementation follows the same pattern as pbuf rings.
New mmap offset constants are introduced:
- IORING_OFF_KMBUF_RING (0x88000000): Base offset for kmbuf mappings
- IORING_OFF_KMBUF_SHIFT (16): Shift value to encode buffer group ID
The mmap offset is calculated during registration, encoding the bgid shifted by IORING_OFF_KMBUF_SHIFT. The io_buf_get_region() helper retrieves the appropriate region.
This allows userspace to mmap the kernel-allocated buffer region and access the buffers directly.
--- include/uapi/linux/io_uring.h | 2 ++ io_uring/kbuf.c | 11 +++++++++-- io_uring/kbuf.h | 5 +++-- io_uring/memmap.c | 5 ++++- 4 files changed, 18 insertions(+), 5 deletions(-)
include/uapi/linux/io_uring.h +2 -0
@@ -533,6 +533,8 @@ struct io_uring_cqe {#define IORING_OFF_SQES 0x10000000ULL#define IORING_OFF_PBUF_RING 0x80000000ULL#define IORING_OFF_PBUF_SHIFT 16 +#define IORING_OFF_KMBUF_RING 0x88000000ULL +#define IORING_OFF_KMBUF_SHIFT 16#define IORING_OFF_MMAP_MASK 0xf8000000ULL/*
io_uring/kbuf.c +9 -2
@@ -766,16 +766,23 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } -struct io_mapped_region *io_pbuf_get_region (struct io_ring_ctx *ctx, -unsigned int bgid) +struct io_mapped_region *io_buf_get_region (struct io_ring_ctx *ctx, +unsigned int bgid, +bool kernel_managed) {struct io_buffer_list *bl; +bool is_kernel_managed;lockdep_assert_held (&ctx->mmap_lock); bl =xa_load (&ctx->io_bl_xa, bgid);if (!bl || !(bl->flags &IOBL_BUF_RING ))return NULL ; + + is_kernel_managed = !!(bl->flags &IOBL_KERNEL_MANAGED ); +if (is_kernel_managed != kernel_managed) +return NULL ; +return &bl->region; }
io_uring/kbuf.h +3 -2
@@ -88,8 +88,9 @@ unsigned int __io_put_kbufs(struct io_kiocb *req, struct io_buffer_list *bl,bool io_kbuf_commit (struct io_kiocb *req,struct io_buffer_list *bl,int len,int nr); -struct io_mapped_region *io_pbuf_get_region (struct io_ring_ctx *ctx, -unsigned int bgid); +struct io_mapped_region *io_buf_get_region (struct io_ring_ctx *ctx, +unsigned int bgid, +bool kernel_managed);static inline bool io_kbuf_recycle_ring (struct io_kiocb *req,struct io_buffer_list *bl)
io_uring/memmap.c +4 -1
@@ -357,7 +357,10 @@ static struct io_mapped_region *io_mmap_get_region(struct io_ring_ctx *ctx,return &ctx->sq_region;case IORING_OFF_PBUF_RING : id = (offset & ~IORING_OFF_MMAP_MASK ) >>IORING_OFF_PBUF_SHIFT ; -return io_pbuf_get_region (ctx, id); +return io_buf_get_region (ctx, id, false); +case IORING_OFF_KMBUF_RING : + id = (offset & ~IORING_OFF_MMAP_MASK ) >>IORING_OFF_KMBUF_SHIFT ; +return io_buf_get_region (ctx, id, true);case IORING_MAP_OFF_PARAM_REGION :return &ctx->param_region;case IORING_MAP_OFF_ZCRX_REGION :
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 5/25 io_uring/kbuf: support kernel-managed buffer rings in buffer selection
In reply to Joanne KoongAllow kernel-managed buffers to be selected. This requires modifying the io_br_sel struct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.
Auto-commit any selected kernel-managed buffer.
--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 15 ++++++++++++--- 2 files changed, 16 insertions(+), 7 deletions(-)
include/linux/io_uring_types.h +4 -4
@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };
io_uring/kbuf.c +12 -3
@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the
@@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true; -/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD )return true;return false;
@@ -201,8 +206,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )buf->addr; +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; }
Signature · 1 line
2.47.3
Re: [PATCH v3 05/25] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
Quoted 64 lines · Joanne Koong
Allow kernel-managed buffers to be selected. This requires modifying the
io_br_selstruct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.Auto-commit any selected kernel-managed buffer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 15 ++++++++++++--- 2 files changed, 16 insertions(+), 7 deletions(-)include/linux/io_uring_types.h +4
-4@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };io_uring/kbuf.c +7
-2@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the @@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true; -/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD )return true;return false;
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 208
Quoted 3 lines
io_uring/kbuf.c +0
-0@@ -201,8 +206,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr));
Drop this assignment as it's overwritten by the assignments below?
Best,
Caleb
Quoted 13 lines
io_uring/kbuf.c +5
-2+if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )buf->addr; +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; } --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 05/25] io_uring/kbuf: support kernel-managed buffer rings in buffer selection
Quoted 56 lines · Caleb Sander Mateos
Quoted 54 lines · Joanne Koong
Allow kernel-managed buffers to be selected. This requires modifying the
io_br_selstruct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.Auto-commit any selected kernel-managed buffer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 15 ++++++++++++--- 2 files changed, 16 insertions(+), 7 deletions(-)include/linux/io_uring_types.h +4
-4@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };io_uring/kbuf.c +2
-1@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the @@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true;
Quoted 15 lines
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 208
Quoted 13 lines
io_uring/kbuf.c +5
-1-/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD )return true;return false;@@ -201,8 +206,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr));Drop this assignment as it's overwritten by the assignments below?
Ah yes, this is a duplicate. I'll remove this for v4.
Thanks,
Joanne
Quoted 17 lines
Best,
CalebQuoted 13 lines
io_uring/kbuf.c +5
-2+if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)(uintptr_t )buf->addr; +else + sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; } --2.47.3
Joanne Koong
maintainerPATCH v3 6/25 io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Joanne KoongAdd kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.
This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
--- include/linux/io_uring/cmd.h | 17 +++++++++++++ io_uring/kbuf.c | 46 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 10 ++++++++ io_uring/uring_cmd.c | 18 ++++++++++++++ 4 files changed, 91 insertions(+)
include/linux/io_uring/cmd.h +17 -0
@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,bool io_uring_mshot_cmd_post_cqe (struct io_uring_cmd *ioucmd,struct io_br_sel *sel,unsigned int issue_flags); +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, {return true; } +static inline int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +46 -0
58 lines not on this page · raw message
io_uring/kbuf.h +10 -0
@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {
@@ -136,4 +141,9 @@ static inline unsigned int io_put_kbufs(struct io_kiocb *req, int len,return 0 ; return __io_put_kbufs(req, bl, len, nbufs); } + +int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags);#endif
io_uring/uring_cmd.c +18 -0
@@ -398,3 +398,21 @@ bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,return true; }EXPORT_SYMBOL_GPL (io_uring_mshot_cmd_post_cqe); + +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_pin (req, buf_group, issue_flags, bl); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_pin); + +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_unpin (req, buf_group, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin);
Signature · 1 line
2.47.3
Re: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 251
Quoted 11 lines · Joanne Koong
io_uring/kbuf.c +11
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) {
FWIW, the likely construct is unnecessary here. At least, it should encompass the entire expression:
if (likely(buffer_list && buffer_list->flags & IOBL_BUF_RING))
But you can just drop it.
Quoted 15 lines
io_uring/kbuf.c +15
-0+if (unlikely (buffer_list->flags &IOBL_PINNED )) { + ret = -EALREADY ; + }else { + buffer_list->flags |=IOBL_PINNED ; + ret =0 ; + *bl = buffer_list; + } + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} + +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags)
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 276
Quoted 10 lines
io_uring/kbuf.c +10
-0+{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) &&likely (bl->flags &IOBL_BUF_RING ) && +likely (bl->flags &IOBL_PINNED )) {
likewise.
Quoted 72 lines
io_uring/kbuf.c +10
-0+ bl->flags &= ~IOBL_PINNED ; + ret =0 ; + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +/* cap it at a reasonable 256, will be one page even for 4K */ #define PEEK_MAX_IMPORT 256@@ -744,6 +788,8 @@ int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg)return -ENOENT ;if (!(bl->flags &IOBL_BUF_RING ))return -EINVAL ; +if (bl->flags &IOBL_PINNED ) +return -EBUSY ; scoped_guard(mutex, &ctx->mmap_lock)xa_erase (&ctx->io_bl_xa, bl->bgid);io_uring/kbuf.h +10
-0@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {@@ -136,4 +141,9 @@ static inline unsigned int io_put_kbufs(struct io_kiocb *req, int len,return 0 ; return __io_put_kbufs(req, bl, len, nbufs); } + +int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags);#endif io_uring/uring_cmd.c +18
-0@@ -398,3 +398,21 @@ bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,return true; }EXPORT_SYMBOL_GPL (io_uring_mshot_cmd_post_cqe); + +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_pin (req, buf_group, issue_flags, bl); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_pin); + +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_unpin (req, buf_group, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin);
Signature · 1 line
Gabriel Krisman Bertazi
Joanne Koong
maintainerRe: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 18 lines · Gabriel Krisman Bertazi
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 251
Quoted 11 lines · Joanne Koong
io_uring/kbuf.c +11
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) {FWIW, the likely construct is unnecessary here. At least, it should encompass the entire expression:
Quoted 3 lines
if (likely(buffer_list && buffer_list->flags & IOBL_BUF_RING))But you can just drop it.
I see, thanks. Could you explain when likelys/unlikelys should be used vs not? It's unclear to me when they need to be included vs can be dropped. I see some other io-uring code use likely() for similar-ish logic, but is the idea that it's unnecessary because the compiler already infers it?
Thanks,
Joanne
Quoted 29 lines
Quoted 16 lines
io_uring/kbuf.c +15
-0+if (unlikely (buffer_list->flags &IOBL_PINNED )) { + ret = -EALREADY ; + }else { + buffer_list->flags |=IOBL_PINNED ; + ret =0 ; + *bl = buffer_list; + } + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} + +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags)io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 276
Quoted 10 lines
io_uring/kbuf.c +10
-0+{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) &&likely (bl->flags &IOBL_BUF_RING ) && +likely (bl->flags &IOBL_PINNED )) {likewise.
Re: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 22 lines · Joanne Koong
Quoted 20 lines · Gabriel Krisman Bertazi
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 251
Quoted 11 lines · Joanne Koong
io_uring/kbuf.c +11
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) {FWIW, the likely construct is unnecessary here. At least, it should encompass the entire expression:
if (likely(buffer_list && buffer_list->flags & IOBL_BUF_RING))
Quoted 7 lines
Quoted 1 line
But you can just drop it.
I see, thanks. Could you explain when likelys/unlikelys should be used vs not? It's unclear to me when they need to be included vs can be dropped. I see some other io-uring code use
likely()for similar-ish logic, but is the idea that it's unnecessary because the compiler already infers it?
likely/unlikely help the compiler decide whether it should reverse the jump to optimize branch prediction and code spacial locality for icache. The compiler is usually great in figuring it out by itself and, in general, these should only be used after profilings shows the specific jump is problematic, or when you know the jump will or will not be taken almost every time. The compiler decision depends on heuristics (which I guess considers the leg size and favors the if leg), but it usually gets it right.
One obvious case where *unlikely* is useful is to handle error paths. The logic behind it is that the error path is obviously not the hot-path, so a branch misprediction or a cache miss in that path is just fine.
The usage of likely is more rare, and some usages are just cargo-cult. Here you could use it, as the hot path is definitely the if leg. But if you look at the generated code, it most likely doesn't make any difference, because gcc is smart enough to handle it.
A problem arises when likely/unlikely are used improperly, or the code changes and the frequency when each leg is taken changes. Now the likely/unlikely is introducing mispredictions the compiler could have avoided and harming performance.
I wasn't gonna comment in the review, since the likely() seems harmless in your patch. But what got my attention was that each separate expression was under a single likely() expression. I don't think that makes much sense, since the hint is useful for the placement of the if/else legs, it should encompass the whole condition. That's how it is used almost anywhere else in the kernel (there are a few occurrences drivers/scsi/ that also look a bit fishy, IMO).
Signature · 1 line
Gabriel Krisman Bertazi
Joanne Koong
maintainerRe: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 57 lines · Gabriel Krisman Bertazi
Quoted 22 lines · Joanne Koong
Quoted 20 lines · Gabriel Krisman Bertazi
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 251
Quoted 11 lines · Joanne Koong
io_uring/kbuf.c +11
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) {FWIW, the likely construct is unnecessary here. At least, it should encompass the entire expression:
if (likely(buffer_list && buffer_list->flags & IOBL_BUF_RING))Quoted 7 lines
Quoted 1 line
But you can just drop it.
I see, thanks. Could you explain when likelys/unlikelys should be used vs not? It's unclear to me when they need to be included vs can be dropped. I see some other io-uring code use
likely()for similar-ish logic, but is the idea that it's unnecessary because the compiler already infers it?likely/unlikely help the compiler decide whether it should reverse the jump to optimize branch prediction and code spacial locality for icache. The compiler is usually great in figuring it out by itself and, in general, these should only be used after profilings shows the specific jump is problematic, or when you know the jump will or will not be taken almost every time. The compiler decision depends on heuristics (which I guess considers the leg size and favors the if leg), but it usually gets it right.
One obvious case where *unlikely* is useful is to handle error paths. The logic behind it is that the error path is obviously not the hot-path, so a branch misprediction or a cache miss in that path is just fine.
The usage of likely is more rare, and some usages are just cargo-cult. Here you could use it, as the hot path is definitely the if leg. But if you look at the generated code, it most likely doesn't make any difference, because gcc is smart enough to handle it.
A problem arises when likely/unlikely are used improperly, or the code changes and the frequency when each leg is taken changes. Now the likely/unlikely is introducing mispredictions the compiler could have avoided and harming performance.
Quoted 7 lines
I wasn't gonna comment in the review, since the
likely()seems harmless in your patch. But what got my attention was that each separate expression was under a singlelikely()expression. I don't think that makes much sense, since the hint is useful for the placement of the if/else legs, it should encompass the whole condition. That's how it is used almost anywhere else in the kernel (there are a few occurrencesdrivers/scsi/that also look a bit fishy, IMO).
That makes sense. Thanks for the elaboration.
Quoted 3 lines
--
Gabriel Krisman Bertazi
Re: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Gabriel Krisman BertaziQuoted 33 lines · Gabriel Krisman Bertazi
Quoted 22 lines · Joanne Koong
Quoted 20 lines · Gabriel Krisman Bertazi
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 251
Quoted 11 lines · Joanne Koong
io_uring/kbuf.c +11
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) {FWIW, the likely construct is unnecessary here. At least, it should encompass the entire expression:
if (likely(buffer_list && buffer_list->flags & IOBL_BUF_RING))Quoted 7 lines
Quoted 1 line
But you can just drop it.
I see, thanks. Could you explain when likelys/unlikelys should be used vs not? It's unclear to me when they need to be included vs can be dropped. I see some other io-uring code use
likely()for similar-ish logic, but is the idea that it's unnecessary because the compiler already infers it?
Quoted 8 lines
likely/unlikely help the compiler decide whether it should reverse the jump to optimize branch prediction and code spacial locality for icache. The compiler is usually great in figuring it out by itself and, in general, these should only be used after profilings shows the specific jump is problematic, or when you know the jump will or will not be taken almost every time. The compiler decision depends on heuristics (which I guess considers the leg size and favors the if leg), but it usually gets it right.
Yeah, the compiler can make a guess at the likely branch direction, but it's not magic. Anecdotally, early return paths generally seem to be inferred to be unlikely and if branches seemed to be inferred as likely (and else branches therefore as unlikely). That's a good starting point, but a programmer may well have more context to know which direction branches are likely to go. Additionally, explicit likely()/unlikely() annotations seem to encourage compilers to optimize more aggressively for the likely path (e.g. splitting the unlikely path into a separate function). I guess compilers want to hedge their bets if they're relying on heuristics to predict the branch direction.
The performance benefit of each likely()/unlikely() is generally small, so I tend to agree adding them is usually a premature optimization. But it can certainly be helpful to improve the code generation of a hot path. And it's used widely throughout io_uring due to the focus on maximizing performance.
Quoted 5 lines
One obvious case where *unlikely* is useful is to handle error paths. The logic behind it is that the error path is obviously not the hot-path, so a branch misprediction or a cache miss in that path is just fine.
And that's the case here; the if condition is only false in the error case where buffer_group doesn't specify a valid kernel buf ring.
Quoted 5 lines
The usage of likely is more rare, and some usages are just cargo-cult. Here you could use it, as the hot path is definitely the if leg. But if you look at the generated code, it most likely doesn't make any difference, because gcc is smart enough to handle it.
Not sure I agree about this distinction between likely() and unlikely() in general. For example, it can make sense to annotate an early return path with likely() to override the compiler's heuristic that early returns are unlikely.
Quoted 5 lines
A problem arises when likely/unlikely are used improperly, or the code changes and the frequency when each leg is taken changes. Now the likely/unlikely is introducing mispredictions the compiler could have avoided and harming performance.
Yes, this is probably the biggest pitfall of likely()/unlikely(). The additional macro call also makes the code a bit harder for a human to parse. Commit 4ec703ec0c38 ("io_uring: fix incorrect unlikely() usage in io_waitid_prep()") is a good example of both these problems.
Quoted 8 lines
I wasn't gonna comment in the review, since the
likely()seems harmless in your patch. But what got my attention was that each separate expression was under a singlelikely()expression. I don't think that makes much sense, since the hint is useful for the placement of the if/else legs, it should encompass the whole condition. That's how it is used almost anywhere else in the kernel (there are a few occurrencesdrivers/scsi/that also look a bit fishy, IMO).
It can absolutely make sense to use likely()/unlikely() to annotate individual components of an && or || expression, not just a full if/while/for/ternary condition. Since && and || are short-circuiting operators, the compiler will often emit a branch after each boolean operand. (For example, absent optimizations, if (a && b) ... is translated into if (a) { if (b) ... }.)
But you're totally correct that likely(A && B) implies likely(A) && likely(B), so there's no need to separately annotate each of the operands to the &&s here.
Best,
Caleb
Re: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Joanne KoongQuoted 62 lines · Joanne Koong
Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return
-EBUSY.This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 17 +++++++++++++ io_uring/kbuf.c | 46 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 10 ++++++++ io_uring/uring_cmd.c | 18 ++++++++++++++ 4 files changed, 91 insertions(+)include/linux/io_uring/cmd.h +17
-0@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,bool io_uring_mshot_cmd_post_cqe (struct io_uring_cmd *ioucmd,struct io_br_sel *sel,unsigned int issue_flags); +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, {return true; } +static inline int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +0
-0
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 245
Quoted 8 lines
io_uring/kbuf.c +5
-0@@ -238,6 +238,50 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,return sel; } +int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx;
Would it make sense to take struct io_ring_ctx *ctx instead of struct io_kiocb *req since this doesn't otherwise use the req? Same comment for several of the other kbuf_ring helpers (io_kbuf_ring_unpin(), io_reg_buf_index_get(), io_reg_buf_index_put(), io_is_kmbuf_ring()).
Best,
Caleb
Quoted 106 lines
io_uring/kbuf.c +41
-0+int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) &&likely (buffer_list->flags &IOBL_BUF_RING )) { +if (unlikely (buffer_list->flags &IOBL_PINNED )) { + ret = -EALREADY ; + }else { + buffer_list->flags |=IOBL_PINNED ; + ret =0 ; + *bl = buffer_list; + } + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} + +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) &&likely (bl->flags &IOBL_BUF_RING ) && +likely (bl->flags &IOBL_PINNED )) { + bl->flags &= ~IOBL_PINNED ; + ret =0 ; + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +/* cap it at a reasonable 256, will be one page even for 4K */ #define PEEK_MAX_IMPORT 256@@ -744,6 +788,8 @@ int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg)return -ENOENT ;if (!(bl->flags &IOBL_BUF_RING ))return -EINVAL ; +if (bl->flags &IOBL_PINNED ) +return -EBUSY ; scoped_guard(mutex, &ctx->mmap_lock)xa_erase (&ctx->io_bl_xa, bl->bgid);io_uring/kbuf.h +10
-0@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {@@ -136,4 +141,9 @@ static inline unsigned int io_put_kbufs(struct io_kiocb *req, int len,return 0 ; return __io_put_kbufs(req, bl, len, nbufs); } + +int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags);#endif io_uring/uring_cmd.c +18
-1@@ -398,3 +398,21 @@ bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,return true; }EXPORT_SYMBOL_GPL (io_uring_mshot_cmd_post_cqe); + +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_pin (req, buf_group, issue_flags, bl); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_pin); + +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kbuf_ring_unpin (req, buf_group, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin); --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 06/25] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 67 lines · Caleb Sander Mateos
Quoted 65 lines · Joanne Koong
Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return
-EBUSY.This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 17 +++++++++++++ io_uring/kbuf.c | 46 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 10 ++++++++ io_uring/uring_cmd.c | 18 ++++++++++++++ 4 files changed, 91 insertions(+)include/linux/io_uring/cmd.h +17
-0@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,bool io_uring_mshot_cmd_post_cqe (struct io_uring_cmd *ioucmd,struct io_br_sel *sel,unsigned int issue_flags); +int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group, +unsigned issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, {return true; } +static inline int io_uring_cmd_buf_ring_pin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +0
-0@@ -238,6 +238,50 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,return sel; }
Quoted 11 lines
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 245
Quoted 5 lines
io_uring/kbuf.c +5
-0+int io_kbuf_ring_pin (struct io_kiocb *req,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +struct io_ring_ctx *ctx = req->ctx;Would it make sense to take struct
io_ring_ctx*ctx instead of structio_kiocb*req since this doesn't otherwise use the req? Same comment for several of the otherkbuf_ringhelpers (io_kbuf_ring_unpin(),io_reg_buf_index_get(),io_reg_buf_index_put(),io_is_kmbuf_ring()).
I think this makes sense and is a good idea. I'll make this change for v4.
Thanks,
Joanne
Quoted 3 lines
Best,
Caleb
Joanne Koong
maintainerPATCH v3 7/25 io_uring/kbuf: add recycling for kernel managed buffer rings
In reply to Joanne KoongAdd an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)
include/linux/io_uring/cmd.h +13 -0
@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +42 -0
48 lines not on this page · raw message
io_uring/kbuf.h +3 -0
@@ -146,4 +146,7 @@ int io_kbuf_ring_pin(struct io_kiocb *req, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group,unsigned issue_flags); +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#endif
io_uring/uring_cmd.c +11 -0
@@ -416,3 +416,14 @@ int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group,return io_kbuf_ring_unpin (req, buf_group, issue_flags); }EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *ioucmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kmbuf_recycle (req, buf_group, addr, len, bid, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_kmbuffer_recycle);
Signature · 1 line
2.47.3
Re: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
Quoted 62 lines · Joanne Koong
Add an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)include/linux/io_uring/cmd.h +13
-0@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +10
-0@@ -101,6 +101,48 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_uring_buf_ring *br; +struct io_uring_buf *buf; +struct io_buffer_list *bl; +int ret = -EINVAL ; +
io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 123
Quoted 10 lines
io_uring/kbuf.c +10
-0+if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT )) +return ret; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, bgid); + +if (WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) + goto done;
Hi Joanne,
WARN_ONs are not supposed to be reached by the user, but I think that is possible here, i.e. by passing the bgid of legacy provided buffers.
Quoted 57 lines
io_uring/kbuf.c +22
-0+ + br = bl->buf_ring; + +if (WARN_ON_ONCE ((br->tail - bl->head) >= bl->nr_entries)) + goto done; + + buf = &br->bufs[(br->tail) & bl->mask]; + + buf->addr = addr; + buf->len = len; + buf->bid = bid; + + req->flags &= ~REQ_F_BUFFER_RING ; + + br->tail++; + ret =0 ; + +done: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags) {io_uring/kbuf.c +0
-0struct io_ring_ctx *ctx = req->ctx;io_uring/kbuf.h +3
-0@@ -146,4 +146,7 @@ int io_kbuf_ring_pin(struct io_kiocb *req, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group,unsigned issue_flags); +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#endif io_uring/uring_cmd.c +11
-0@@ -416,3 +416,14 @@ int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group,return io_kbuf_ring_unpin (req, buf_group, issue_flags); }EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *ioucmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kmbuf_recycle (req, buf_group, addr, len, bid, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_kmbuffer_recycle);
Signature · 1 line
Gabriel Krisman Bertazi
Re: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
Quoted 77 lines · Gabriel Krisman Bertazi
Quoted 62 lines · Joanne Koong
Add an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)include/linux/io_uring/cmd.h +13
-0@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +10
-0@@ -101,6 +101,48 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_uring_buf_ring *br; +struct io_uring_buf *buf; +struct io_buffer_list *bl; +int ret = -EINVAL ; +io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 123
Quoted 10 lines
io_uring/kbuf.c +10
-0+if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT )) +return ret; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, bgid); + +if (WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) + goto done;Hi Joanne,
Quoted 2 lines
WARN_ONsare not supposed to be reached by the user, but I think that is possible here, i.e. by passing the bgid of legacy provided buffers.
But now I see this is never exposed to userspace as an io_uring_cmd command itself, it is only used internally by other fuse operations. Nevertheless, it's implemented as an io_uring_cmd by io_uring_cmd_kmbuffer_recycle.
Is it eventually going to be exposed as operations to userspace? If not, I'd suggest to stay out of the io_uring_cmd namespace (perhaps call io_kmbuf_recycle directly from fs/fuse). Do we need to have this io_uring_cmd abstraction for some reason I'm missing?
Thanks,
Signature · 1 line
Gabriel Krisman Bertazi
Joanne Koong
maintainerRe: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
Quoted 88 lines · Gabriel Krisman Bertazi
Quoted 77 lines · Gabriel Krisman Bertazi
Quoted 62 lines · Joanne Koong
Add an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)include/linux/io_uring/cmd.h +13
-0@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +10
-0@@ -101,6 +101,48 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_uring_buf_ring *br; +struct io_uring_buf *buf; +struct io_buffer_list *bl; +int ret = -EINVAL ; +io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 123
Quoted 10 lines
io_uring/kbuf.c +10
-0+if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT )) +return ret; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, bgid); + +if (WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) + goto done;Hi Joanne,
Quoted 2 lines
WARN_ONsare not supposed to be reached by the user, but I think that is possible here, i.e. by passing the bgid of legacy provided buffers.But now I see this is never exposed to userspace as an
io_uring_cmdcommand itself, it is only used internally by other fuse operations. Nevertheless, it's implemented as anio_uring_cmdbyio_uring_cmd_kmbuffer_recycle.
Quoted 4 lines
Is it eventually going to be exposed as operations to userspace? If not, I'd suggest to stay out of the
io_uring_cmdnamespace (perhaps callio_kmbuf_recycledirectly from fs/fuse). Do we need to have thisio_uring_cmdabstraction for some reason I'm missing?
Hi Gabriel,
Thanks for taking a look at the patchset.
This is not going to be exposed as an operation to userspace. Only the kernel will be able to recycle kmbufs.
I was under the impression the io_uring_cmd_* abstraction was preferred as the API for interfacing with io_uring from another subsystem. In that case then, I'll get rid of the io_uring_cmd layers for the calls then, that will make things simpler.
Thanks,
Joanne
Quoted 5 lines
Thanks,
--
Gabriel Krisman Bertazi
Re: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
Quoted 12 lines · Joanne Koong
Quoted 9 lines
But now I see this is never exposed to userspace as an
io_uring_cmdcommand itself, it is only used internally by other fuse operations. Nevertheless, it's implemented as anio_uring_cmdbyio_uring_cmd_kmbuffer_recycle.Is it eventually going to be exposed as operations to userspace? If not, I'd suggest to stay out of the
io_uring_cmdnamespace (perhaps callio_kmbuf_recycledirectly from fs/fuse). Do we need to have thisio_uring_cmdabstraction for some reason I'm missing?Hi Gabriel,
Quoted 8 lines
Thanks for taking a look at the patchset.
This is not going to be exposed as an operation to userspace. Only the kernel will be able to recycle kmbufs.
I was under the impression the io_uring_cmd_* abstraction was preferred as the API for interfacing with
io_uringfrom another subsystem.
Hello,
I don't think that's the case, no. io_uring_cmd are used to expose backend-specific file operations to userspace, similar to an asynchronous ioctl. Moreover, it would just add an extra layer, without bringing any benefits, IMO.
Quoted 2 lines
In that case then, I'll get rid of the
io_uring_cmdlayers for the calls then, that will make things simpler.
Signature · 1 line
Gabriel Krisman Bertazi
Re: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
In reply to Joanne KoongQuoted 117 lines · Joanne Koong
Add an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)include/linux/io_uring/cmd.h +13
-0@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +42
-0@@ -101,6 +101,48 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_uring_buf_ring *br; +struct io_uring_buf *buf; +struct io_buffer_list *bl; +int ret = -EINVAL ; + +if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT )) +return ret; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, bgid); + +if (WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) + goto done; + + br = bl->buf_ring; + +if (WARN_ON_ONCE ((br->tail - bl->head) >= bl->nr_entries)) + goto done; + + buf = &br->bufs[(br->tail) & bl->mask]; + + buf->addr = addr; + buf->len = len; + buf->bid = bid; + + req->flags &= ~REQ_F_BUFFER_RING ; + + br->tail++; + ret =0 ; + +done: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags) {struct io_ring_ctx *ctx = req->ctx;io_uring/kbuf.h +3
-0@@ -146,4 +146,7 @@ int io_kbuf_ring_pin(struct io_kiocb *req, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group,unsigned issue_flags); +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#endif io_uring/uring_cmd.c +0
-0@@ -416,3 +416,14 @@ int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group,return io_kbuf_ring_unpin (req, buf_group, issue_flags); }
io_uring/uring_cmd.c int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group, · line 427
Quoted 10 lines
io_uring/uring_cmd.c +9
-0EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *ioucmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kmbuf_recycle (req, buf_group, addr, len, bid, issue_flags);
Total nit, but is there a reason for the inconsistency between "kmbuffer" and "kmbuf"? I would prefer to use a single name for the concept.
Best,
Caleb
Quoted 5 lines
io_uring/uring_cmd.c +2
-1+} +EXPORT_SYMBOL_GPL (io_uring_cmd_kmbuffer_recycle); --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings
Quoted 125 lines · Caleb Sander Mateos
Quoted 123 lines · Joanne Koong
Add an interface for buffers to be recycled back into a kernel-managed buffer ring.
This is a preparatory patch for fuse over io-uring.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 13 +++++++++++ io_uring/kbuf.c | 42 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 3 +++ io_uring/uring_cmd.c | 11 ++++++++++ 4 files changed, 69 insertions(+)include/linux/io_uring/cmd.h +13
-0@@ -88,6 +88,11 @@ int io_uring_cmd_buf_ring_pin(struct io_uring_cmd *ioucmd, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_cmd_buf_ring_unpin (struct io_uring_cmd *ioucmd,unsigned buf_group,unsigned issue_flags); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -143,6 +148,14 @@ static inline int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *cmd, +unsigned int buf_group, +u64 addr,unsigned int len, +unsigned int bid, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)io_uring/kbuf.c +42
-0@@ -101,6 +101,48 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_uring_buf_ring *br; +struct io_uring_buf *buf; +struct io_buffer_list *bl; +int ret = -EINVAL ; + +if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT )) +return ret; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, bgid); + +if (WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) + goto done; + + br = bl->buf_ring; + +if (WARN_ON_ONCE ((br->tail - bl->head) >= bl->nr_entries)) + goto done; + + buf = &br->bufs[(br->tail) & bl->mask]; + + buf->addr = addr; + buf->len = len; + buf->bid = bid; + + req->flags &= ~REQ_F_BUFFER_RING ; + + br->tail++; + ret =0 ; + +done: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags) {struct io_ring_ctx *ctx = req->ctx;io_uring/kbuf.h +3
-0@@ -146,4 +146,7 @@ int io_kbuf_ring_pin(struct io_kiocb *req, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_kbuf_ring_unpin (struct io_kiocb *req,unsigned buf_group,unsigned issue_flags); +int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags);#endif io_uring/uring_cmd.c +5
-0@@ -416,3 +416,14 @@ int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group,return io_kbuf_ring_unpin (req, buf_group, issue_flags); }EXPORT_SYMBOL_GPL (io_uring_cmd_buf_ring_unpin); + +int io_uring_cmd_kmbuffer_recycle (struct io_uring_cmd *ioucmd, +unsigned int buf_group,u64 addr, +unsigned int len,unsigned int bid, +unsigned int issue_flags)
Quoted 8 lines
io_uring/uring_cmd.c int io_uring_cmd_buf_ring_unpin(struct io_uring_cmd *ioucmd, unsigned buf_group, · line 427
Quoted 4 lines
io_uring/uring_cmd.c +4
-0+{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_kmbuf_recycle (req, buf_group, addr, len, bid, issue_flags);Total nit, but is there a reason for the inconsistency between "kmbuffer" and "kmbuf"? I would prefer to use a single name for the concept.
Ah totally valid question. I'll stick with the kmbuf naming convention and rename this for v4.
Thanks for taking a look at this patchset series.
Quoted 9 lines
Best,
CalebQuoted 5 lines
io_uring/uring_cmd.c +2
-1+} +EXPORT_SYMBOL_GPL (io_uring_cmd_kmbuffer_recycle); --2.47.3
Joanne Koong
maintainerPATCH v3 8/25 io_uring: add io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put()
In reply to Joanne KoongAdd two new helpers, io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put(). io_uring_cmd_fixed_index_get() constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node. io_uring_cmd_fixed_index_put() decrements this refcount. The caller is responsible for ensuring io_uring_cmd_fixed_index_put() is properly called for releasing the refcount after it is done using the iter it obtained through io_uring_cmd_fixed_index_get().
This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.
--- include/linux/io_uring/cmd.h | 20 +++++++++++ io_uring/rsrc.c | 65 ++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 +++ io_uring/uring_cmd.c | 21 ++++++++++++ 4 files changed, 111 insertions(+)
include/linux/io_uring/cmd.h +20 -0
@@ -44,6 +44,12 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,size_t uvec_segs, int ddir, struct iov_iter *iter, unsigned issue_flags); +int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int off,size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags); +int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int issue_flags);/* * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd
@@ -109,6 +115,20 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd, +u16 buf_index,unsigned int off, +size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd, +u16 buf_index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline void __io_uring_cmd_done (struct io_uring_cmd *cmd,s32 ret,u64 ret2,unsigned issue_flags,bool is_cqe32) {
io_uring/rsrc.c +65 -0
71 lines not on this page · raw message
io_uring/rsrc.h +5 -0
@@ -64,6 +64,11 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags); +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);
io_uring/uring_cmd.c +21 -0
@@ -314,6 +314,27 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec); +int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int off,size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_reg_buf_index_get (req, iter, buf_index, off, len, ddir, + issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_fixed_index_get); + +int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_reg_buf_index_put (req, buf_index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_fixed_index_put); +void io_uring_cmd_issue_blocking (struct io_uring_cmd *ioucmd) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd);
Signature · 1 line
2.47.3
Re: [PATCH v3 08/25] io_uring: add io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put()
Quoted 111 lines · Joanne Koong
Add two new helpers,
io_uring_cmd_fixed_index_get()andio_uring_cmd_fixed_index_put().io_uring_cmd_fixed_index_get()constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node.io_uring_cmd_fixed_index_put()decrements this refcount. The caller is responsible for ensuringio_uring_cmd_fixed_index_put()is properly called for releasing the refcount after it is done using the iter it obtained throughio_uring_cmd_fixed_index_get().This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 20 +++++++++++ io_uring/rsrc.c | 65 ++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 +++ io_uring/uring_cmd.c | 21 ++++++++++++ 4 files changed, 111 insertions(+)include/linux/io_uring/cmd.h +20
-0@@ -44,6 +44,12 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,size_t uvec_segs, int ddir, struct iov_iter *iter, unsigned issue_flags); +int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int off,size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags); +int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int issue_flags);/* * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd @@ -109,6 +115,20 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd, +u16 buf_index,unsigned int off, +size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd, +u16 buf_index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline void __io_uring_cmd_done (struct io_uring_cmd *cmd,s32 ret,u64 ret2,unsigned issue_flags,bool is_cqe32) {io_uring/rsrc.c +43
-0@@ -1151,6 +1151,71 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +u64 addr; +int err; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +if (!node) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + + node->refs++; + +io_ring_submit_unlock (ctx, issue_flags); + + imu = node->buf; +if (!imu) { + err = -EFAULT ; + goto error; + } + +if (check_add_overflow (imu->ubuf, off, &addr)) { + err = -EINVAL ; + goto error; + } + + err =io_import_fixed (ddir, iter, imu, addr, len); +if (err) + goto error; + +return 0 ; + +error: +io_reg_buf_index_put (req, buf_index, issue_flags); +return err; +}
io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1206
Quoted 10 lines
io_uring/rsrc.c +10
-0+ +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index);
Hmm, I don't think it's safe to assume this node looked up by buf_index matches the one obtained in io_reg_buf_index_get(). Since the uring_lock is released between io_reg_buf_index_get() and io_reg_buf_index_put(), an intervening IORING_REGISTER_BUFFERS_UPDATE operation may modify the node at buf_index in the buf_table. That could result in a reference decrement on the wrong node, resulting in the new node being freed while still in use.
Can we return the struct io_rsrc_node * from io_reg_buf_index_get() and pass it to io_reg_buf_index_put() in place of buf_index?
Thanks,
Caleb
Quoted 66 lines
io_uring/rsrc.c +12
-0+if (WARN_ON_ONCE (!node)) { +io_ring_submit_unlock (ctx, issue_flags); +return -EFAULT ; + } + +io_put_rsrc_node (ctx, node); + +io_ring_submit_unlock (ctx, issue_flags); + +return 0 ; +} +/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2) {io_uring/rsrc.h +5
-0@@ -64,6 +64,11 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags); +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);io_uring/uring_cmd.c +21
-1@@ -314,6 +314,27 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec); +int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int off,size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_reg_buf_index_get (req, iter, buf_index, off, len, ddir, + issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_fixed_index_get); + +int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_reg_buf_index_put (req, buf_index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_fixed_index_put); +void io_uring_cmd_issue_blocking (struct io_uring_cmd *ioucmd) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); --2.47.3
Re: [PATCH v3 08/25] io_uring: add io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put()
Quoted 125 lines · Caleb Sander Mateos
Quoted 111 lines · Joanne Koong
Add two new helpers,
io_uring_cmd_fixed_index_get()andio_uring_cmd_fixed_index_put().io_uring_cmd_fixed_index_get()constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node.io_uring_cmd_fixed_index_put()decrements this refcount. The caller is responsible for ensuringio_uring_cmd_fixed_index_put()is properly called for releasing the refcount after it is done using the iter it obtained throughio_uring_cmd_fixed_index_get().This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 20 +++++++++++ io_uring/rsrc.c | 65 ++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 +++ io_uring/uring_cmd.c | 21 ++++++++++++ 4 files changed, 111 insertions(+)include/linux/io_uring/cmd.h +20
-0@@ -44,6 +44,12 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,size_t uvec_segs, int ddir, struct iov_iter *iter, unsigned issue_flags); +int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int off,size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags); +int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd,u16 buf_index, +unsigned int issue_flags);/* * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd @@ -109,6 +115,20 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_fixed_index_get (struct io_uring_cmd *ioucmd, +u16 buf_index,unsigned int off, +size_t len,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_cmd_fixed_index_put (struct io_uring_cmd *ioucmd, +u16 buf_index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline void __io_uring_cmd_done (struct io_uring_cmd *cmd,s32 ret,u64 ret2,unsigned issue_flags,bool is_cqe32) {io_uring/rsrc.c +43
-0@@ -1151,6 +1151,71 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +u64 addr; +int err; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +if (!node) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + + node->refs++; + +io_ring_submit_unlock (ctx, issue_flags); + + imu = node->buf; +if (!imu) { + err = -EFAULT ; + goto error; + } + +if (check_add_overflow (imu->ubuf, off, &addr)) { + err = -EINVAL ; + goto error; + } + + err =io_import_fixed (ddir, iter, imu, addr, len); +if (err) + goto error; + +return 0 ; + +error: +io_reg_buf_index_put (req, buf_index, issue_flags); +return err; +}io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1206
Quoted 10 lines
io_uring/rsrc.c +10
-0+ +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index);Hmm, I don't think it's safe to assume this node looked up by
Quoted 8 lines
buf_indexmatches the one obtained inio_reg_buf_index_get(). Since theuring_lockis released betweenio_reg_buf_index_get()andio_reg_buf_index_put(), an interveningIORING_REGISTER_BUFFERS_UPDATEoperation may modify the node atbuf_indexin thebuf_table. That could result in a reference decrement on the wrong node, resulting in the new node being freed while still in use.
Can we return the structio_rsrc_node* fromio_reg_buf_index_get()and pass it toio_reg_buf_index_put()in place ofbuf_index?
Alternatively, store the struct io_rsrc_node * in the struct io_kiocb, as io_find_buf_node() does? Then you wouldn't even need to call io_reg_buf_index_put(); io_uring would automatically release the buffer node reference upon completion of the request. The only reason I can see why that wouldn't work is if a single fuse io_uring_cmd needs to import multiple registered buffers. But it doesn't look like that's the case from my quick skim of the fuse code.
Joanne Koong
maintainerRe: [PATCH v3 08/25] io_uring: add io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put()
Quoted 99 lines · Caleb Sander Mateos
Quoted 87 lines · Caleb Sander Mateos
Quoted 73 lines · Joanne Koong
Add two new helpers,
io_uring_cmd_fixed_index_get()andio_uring_cmd_fixed_index_put().io_uring_cmd_fixed_index_get()constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node.io_uring_cmd_fixed_index_put()decrements this refcount. The caller is responsible for ensuringio_uring_cmd_fixed_index_put()is properly called for releasing the refcount after it is done using the iter it obtained throughio_uring_cmd_fixed_index_get().This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 20 +++++++++++ io_uring/rsrc.c | 65 ++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 +++ io_uring/uring_cmd.c | 21 ++++++++++++ 4 files changed, 111 insertions(+)io_uring/rsrc.c +43
-0@@ -1151,6 +1151,71 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +u64 addr; +int err; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +if (!node) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + + node->refs++; + +io_ring_submit_unlock (ctx, issue_flags); + + imu = node->buf; +if (!imu) { + err = -EFAULT ; + goto error; + } + +if (check_add_overflow (imu->ubuf, off, &addr)) { + err = -EINVAL ; + goto error; + } + + err =io_import_fixed (ddir, iter, imu, addr, len); +if (err) + goto error; + +return 0 ; + +error: +io_reg_buf_index_put (req, buf_index, issue_flags); +return err; +}io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1206
Quoted 10 lines
io_uring/rsrc.c +10
-0+ +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index);Hmm, I don't think it's safe to assume this node looked up by
Quoted 8 lines
buf_indexmatches the one obtained inio_reg_buf_index_get(). Since theuring_lockis released betweenio_reg_buf_index_get()andio_reg_buf_index_put(), an interveningIORING_REGISTER_BUFFERS_UPDATEoperation may modify the node atbuf_indexin thebuf_table. That could result in a reference decrement on the wrong node, resulting in the new node being freed while still in use.
Can we return the structio_rsrc_node* fromio_reg_buf_index_get()and pass it toio_reg_buf_index_put()in place ofbuf_index?
Quoted 7 lines
Alternatively, store the struct
io_rsrc_node* in the structio_kiocb, asio_find_buf_node()does? Then you wouldn't even need to callio_reg_buf_index_put();io_uringwould automatically release the buffer node reference upon completion of the request. The only reason I can see why that wouldn't work is if a single fuseio_uring_cmdneeds to import multiple registered buffers. But it doesn't look like that's the case from my quick skim of the fuse code.
Good catch, I think I'll need to store that rsrc node pointer inside the fuse code then. I think we could store it inside the struct io_kiocb for the fuse use case but I think at an api level that ends up getting ugly with introducing unobvious relationships between io_uring_cmd_fixed_index_get() and io_uring_cmd_import_fixed().
Thanks,
Joanne
Re: [PATCH v3 08/25] io_uring: add io_uring_cmd_fixed_index_get() and io_uring_cmd_fixed_index_put()
Quoted 110 lines · Joanne Koong
Quoted 99 lines · Caleb Sander Mateos
Quoted 87 lines · Caleb Sander Mateos
Quoted 73 lines · Joanne Koong
Add two new helpers,
io_uring_cmd_fixed_index_get()andio_uring_cmd_fixed_index_put().io_uring_cmd_fixed_index_get()constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node.io_uring_cmd_fixed_index_put()decrements this refcount. The caller is responsible for ensuringio_uring_cmd_fixed_index_put()is properly called for releasing the refcount after it is done using the iter it obtained throughio_uring_cmd_fixed_index_get().This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 20 +++++++++++ io_uring/rsrc.c | 65 ++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 +++ io_uring/uring_cmd.c | 21 ++++++++++++ 4 files changed, 111 insertions(+)io_uring/rsrc.c +43
-0@@ -1151,6 +1151,71 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_reg_buf_index_get (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,unsigned int off,size_t len, +int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +u64 addr; +int err; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +if (!node) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + + node->refs++; + +io_ring_submit_unlock (ctx, issue_flags); + + imu = node->buf; +if (!imu) { + err = -EFAULT ; + goto error; + } + +if (check_add_overflow (imu->ubuf, off, &addr)) { + err = -EINVAL ; + goto error; + } + + err =io_import_fixed (ddir, iter, imu, addr, len); +if (err) + goto error; + +return 0 ; + +error: +io_reg_buf_index_put (req, buf_index, issue_flags); +return err; +}io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1206
Quoted 10 lines
io_uring/rsrc.c +10
-0+ +int io_reg_buf_index_put (struct io_kiocb *req,u16 buf_index, +unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; + +io_ring_submit_lock (ctx, issue_flags); + + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index);Hmm, I don't think it's safe to assume this node looked up by
Quoted 8 lines
buf_indexmatches the one obtained inio_reg_buf_index_get(). Since theuring_lockis released betweenio_reg_buf_index_get()andio_reg_buf_index_put(), an interveningIORING_REGISTER_BUFFERS_UPDATEoperation may modify the node atbuf_indexin thebuf_table. That could result in a reference decrement on the wrong node, resulting in the new node being freed while still in use.
Can we return the structio_rsrc_node* fromio_reg_buf_index_get()and pass it toio_reg_buf_index_put()in place ofbuf_index?Quoted 7 lines
Alternatively, store the struct
io_rsrc_node* in the structio_kiocb, asio_find_buf_node()does? Then you wouldn't even need to callio_reg_buf_index_put();io_uringwould automatically release the buffer node reference upon completion of the request. The only reason I can see why that wouldn't work is if a single fuseio_uring_cmdneeds to import multiple registered buffers. But it doesn't look like that's the case from my quick skim of the fuse code.
Quoted 5 lines
Good catch, I think I'll need to store that rsrc node pointer inside the fuse code then. I think we could store it inside the struct
io_kiocbfor the fuse use case but I think at an api level that ends up getting ugly with introducing unobvious relationships betweenio_uring_cmd_fixed_index_get()andio_uring_cmd_import_fixed().
Perhaps renaming it to something more similar ("io_uring_cmd_import_fixed_index()" or something) would help hint that these 2 functions are related? But I'm fine with storing the struct io_rsrc_node * externally if you'd prefer that.
Best,
Caleb
Joanne Koong
maintainerPATCH v3 9/25 io_uring/kbuf: add io_uring_cmd_is_kmbuf_ring()
In reply to Joanne Koongio_uring_cmd_is_kmbuf_ring() returns true if there is a kernel-managed buffer ring at the specified buffer group.
This is a preparatory patch for upcoming fuse kernel-managed buffer support, which needs to ensure the buffer ring registered by the server is a kernel-managed buffer ring.
--- include/linux/io_uring/cmd.h | 9 +++++++++ io_uring/kbuf.c | 19 +++++++++++++++++++ io_uring/kbuf.h | 2 ++ io_uring/uring_cmd.c | 9 +++++++++ 4 files changed, 39 insertions(+)
include/linux/io_uring/cmd.h +9 -0
@@ -99,6 +99,9 @@ int io_uring_cmd_kmbuffer_recycle(struct io_uring_cmd *cmd, unsigned int buf_group, u64 addr, unsigned int len, unsigned int bid, unsigned int issue_flags); + +int io_uring_cmd_is_kmbuf_ring (struct io_uring_cmd *ioucmd, +unsigned int buf_group,unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -176,6 +179,12 @@ static inline int io_uring_cmd_kmbuffer_recycle(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_is_kmbuf_ring (struct io_uring_cmd *ioucmd, +unsigned int buf_group, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
io_uring/kbuf.c +19 -0
@@ -958,3 +958,22 @@ int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } + +bool io_is_kmbuf_ring (struct io_kiocb *req,unsigned int buf_group, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_buffer_list *bl; +bool is_kmbuf_ring = false; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) && (bl->flags &IOBL_KERNEL_MANAGED )) { +WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )); + is_kmbuf_ring = true; + } + +io_ring_submit_unlock (ctx, issue_flags); +return is_kmbuf_ring; +}
io_uring/kbuf.h +2 -0
@@ -149,4 +149,6 @@ int io_kbuf_ring_unpin(struct io_kiocb *req, unsigned buf_group,int io_kmbuf_recycle (struct io_kiocb *req,unsigned int bgid,u64 addr,unsigned int len,unsigned int bid,unsigned int issue_flags); +bool io_is_kmbuf_ring (struct io_kiocb *req,unsigned int buf_group, +unsigned int issue_flags);#endif
io_uring/uring_cmd.c +9 -0
@@ -448,3 +448,12 @@ int io_uring_cmd_kmbuffer_recycle(struct io_uring_cmd *ioucmd,return io_kmbuf_recycle (req, buf_group, addr, len, bid, issue_flags); }EXPORT_SYMBOL_GPL (io_uring_cmd_kmbuffer_recycle); + +int io_uring_cmd_is_kmbuf_ring (struct io_uring_cmd *ioucmd, +unsigned int buf_group,unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_is_kmbuf_ring (req, buf_group, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_is_kmbuf_ring);
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 10/25 io_uring/kbuf: export io_ring_buffer_select()
In reply to Joanne KoongExport io_ring_buffer_select() so that it may be used by callers who pass in a pinned bufring without needing to grab the io_uring mutex.
This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
--- include/linux/io_uring/buf.h | 25 +++++++++++++++++++++++++ io_uring/kbuf.c | 8 +++++--- 2 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 include/linux/io_uring/buf.h
include/linux/io_uring/buf.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H + +#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags); +#else +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; + +return sel; +} +#endif /* CONFIG_IO_URING */ + +#endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +5 -3
@@ -9,6 +9,7 @@#include <linux/poll.h> #include <linux/vmalloc.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <uapi/linux/io_uring.h>
@@ -223,9 +224,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,return false; } -static struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -unsigned int issue_flags) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) {struct io_uring_buf_ring *br = bl->buf_ring;__u16 tail, head = bl->head;
@@ -259,6 +260,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }return sel; } +EXPORT_SYMBOL_GPL (io_ring_buffer_select);struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags)
Signature · 1 line
2.47.3
Re: [PATCH v3 10/25] io_uring/kbuf: export io_ring_buffer_select()
Quoted 22 lines · Joanne Koong
Export
io_ring_buffer_select()so that it may be used by callers who pass in a pinned bufring without needing to grab theio_uringmutex.This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 25 +++++++++++++++++++++++++ io_uring/kbuf.c | 8 +++++--- 2 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 include/linux/io_uring/buf.hinclude/linux/io_uring/buf.h new file +0
-0
include/linux/io_uring/buf.h line 8
Quoted 8 lines
include/linux/io_uring/buf.h new file +8
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H + +#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
I think struct io_kiocb isn't intended to be exposed outside of io_uring internal code. Is there a reason not to instead expose a wrapper function that takes struct io_uring_cmd * instead?
Best,
Caleb
Quoted 53 lines
include/linux/io_uring/buf.h new file +17
-0+struct io_buffer_list *bl, + unsigned int issue_flags); +#else +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; + +return sel; +} +#endif /* CONFIG_IO_URING */ + +#endif /* _LINUX_IO_URING_BUF_H */ io_uring/kbuf.c +5
-4@@ -9,6 +9,7 @@#include <linux/poll.h> #include <linux/vmalloc.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <uapi/linux/io_uring.h> @@ -223,9 +224,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,return false; } -static struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -unsigned int issue_flags) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) {struct io_uring_buf_ring *br = bl->buf_ring;__u16 tail, head = bl->head;@@ -259,6 +260,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }return sel; } +EXPORT_SYMBOL_GPL (io_ring_buffer_select);struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags) --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 10/25] io_uring/kbuf: export io_ring_buffer_select()
Quoted 28 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Export
io_ring_buffer_select()so that it may be used by callers who pass in a pinned bufring without needing to grab theio_uringmutex.This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 25 +++++++++++++++++++++++++ io_uring/kbuf.c | 8 +++++--- 2 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 include/linux/io_uring/buf.hinclude/linux/io_uring/buf.h new file +4
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H +
Quoted 8 lines
include/linux/io_uring/buf.h line 8
Quoted 4 lines
include/linux/io_uring/buf.h new file +4
-0+#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,I think struct
io_kiocbisn't intended to be exposed outside ofio_uringinternal code. Is there a reason not to instead expose a wrapper function that takes structio_uring_cmd* instead?
Oh interesting... I see struct io_kiocb defined in include/linux/io_uring_types.h, so I assumed this was fine to use. Hmm, we could wrap this in io_uring_cmd * instead, but that adds an extra layer and I think it clashes with the philosophy of io_uring_cmd being a "general user interface" (or maybe my interpretation of io_uring_cmd is incorrect) whereas this api is pretty io-uring internal specific (eg bypasses the io ring lock which means it'll be responsible for having to do its own synchronization, passing the io_buffer_list pointer directly, etc.).
Thanks,
Joanne
Quoted 19 lines
Best,
CalebQuoted 15 lines
include/linux/io_uring/buf.h new file +15
-0+struct io_buffer_list *bl, + unsigned int issue_flags); +#else +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; + +return sel; +} +#endif /* CONFIG_IO_URING */
Re: [PATCH v3 10/25] io_uring/kbuf: export io_ring_buffer_select()
Quoted 40 lines · Joanne Koong
Quoted 28 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Export
io_ring_buffer_select()so that it may be used by callers who pass in a pinned bufring without needing to grab theio_uringmutex.This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 25 +++++++++++++++++++++++++ io_uring/kbuf.c | 8 +++++--- 2 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 include/linux/io_uring/buf.hinclude/linux/io_uring/buf.h new file +4
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H +Quoted 8 lines
include/linux/io_uring/buf.h line 8
Quoted 4 lines
include/linux/io_uring/buf.h new file +4
-0+#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,I think struct
io_kiocbisn't intended to be exposed outside ofio_uringinternal code. Is there a reason not to instead expose a wrapper function that takes structio_uring_cmd* instead?
Quoted 2 lines
Oh interesting... I see struct
io_kiocbdefined ininclude/linux/io_uring_types.h, so I assumed this was fine to use.
Yeah, I see a lot of types that look io_uring-internal in include/linux/io_uring_types.h. I'm not sure why they're visible to the rest of the kernel rather than forward declared. But Jens please correct me if I'm misunderstanding whether these types are meant to be private to the io_uring subsystem.
Quoted 4 lines
Hmm, we could wrap this in
io_uring_cmd* instead, but that adds an extra layer and I think it clashes with the philosophy ofio_uring_cmdbeing a "general user interface" (or maybe my interpretation ofio_uring_cmdis incorrect) whereas this api is pretty io-uring
That's my understanding of io_uring_cmd too.
Quoted 3 lines
internal specific (eg bypasses the io ring lock which means it'll be responsible for having to do its own synchronization, passing the
io_buffer_listpointer directly, etc.).
That's a good point. I wonder if there's a better way to encapsulate this for general usage outside of io_uring? I'm not that familiar with io_uring buffer rings, though.
Best,
Caleb
Joanne Koong
maintainerPATCH v3 11/25 io_uring/kbuf: return buffer id in buffer selection
In reply to Joanne KoongReturn the id of the selected buffer in io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.
--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 8 +++++--- 3 files changed, 8 insertions(+), 4 deletions(-)
include/linux/io_uring/cmd.h +1 -1
@@ -77,7 +77,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);/* * Select a buffer from the provided buffer group for multishot uring_cmd. -* Returns the selected buffer address and size. +* Returns the selected buffer address, size, and id. */ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, unsigned buf_group, size_t *len,
include/linux/io_uring_types.h +2 -0
@@ -100,6 +100,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id; };
io_uring/kbuf.c +5 -3
@@ -248,7 +248,7 @@ struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index =READ_ONCE (buf->bid); sel.buf_list = bl; - sel.addr =u64_to_user_ptr (READ_ONCE (buf->addr)); + sel.buf_id = req->buf_index;if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)(uintptr_t )buf->addr;else
@@ -273,10 +273,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index; + } }io_ring_submit_unlock (req->ctx, issue_flags);return sel;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 12/25 io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
In reply to Joanne KoongWhen uring_cmd operations select a buffer, the completion queue entry should indicate which buffer was selected.
Set IORING_CQE_F_BUFFER on the completed entry and encode the buffer index if a buffer was selected.
This will be needed for fuse, which needs to relay to userspace which selected buffer contains the data.
--- io_uring/uring_cmd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/uring_cmd.c +5 -1
@@ -142,6 +142,7 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); +u32 cflags =0 ;if (WARN_ON_ONCE (req->flags &REQ_F_APOLL_MULTISHOT ))return ;
@@ -151,7 +152,10 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (ret <0 )req_set_fail (req); -io_req_set_res (req, ret,0 ); +if (req->flags & (REQ_F_BUFFER_SELECTED |REQ_F_BUFFER_RING )) + cflags |=IORING_CQE_F_BUFFER | + (req->buf_index <<IORING_CQE_BUFFER_SHIFT ); +io_req_set_res (req, ret, cflags);if (is_cqe32) {if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 13/25 fuse: refactor io-uring logic for getting next fuse request
In reply to Joanne KoongSimplify the logic for getting the next fuse request.
--- fs/fuse/dev_uring.c | 78 ++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
@@ -714,34 +714,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,return err; } -/* -* Write data to the ring buffer and send the request to userspace, -* userspace will read it -* This is comparable with classical read(/dev/fuse) -*/ -static int fuse_uring_send_next_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req, -unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; -int err; -struct io_uring_cmd *cmd; - - err =fuse_uring_prepare_send (ent, req); -if (err) -return err; - -spin_lock (&queue->lock); - cmd = ent->cmd; - ent->cmd =NULL ; - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd,0 , issue_flags); -return 0 ; -} -/* * Make a ring entry available for fuse_req assignment */
@@ -838,11 +810,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, }/* -* Get the next fuse req and send it +* Get the next fuse req. +* +* Returns true if the next fuse request has been assigned to the ent. +* Else, there is no next fuse request and this returns false. */ -static void fuse_uring_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue, -unsigned int issue_flags) +static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, +struct fuse_ring_queue *queue) {int err;struct fuse_req *req;
@@ -854,10 +828,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_send_next_to_ring (ent, req, issue_flags); + err =fuse_uring_prepare_send (ent, req);if (err) goto retry; } + +return req !=NULL ; }static int fuse_ring_ent_set_commit (struct fuse_ring_ent *ent)
@@ -875,6 +851,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)return 0 ; } +static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, +ssize_t ret,unsigned int issue_flags) +{ +struct fuse_ring_queue *queue = ent->queue; + +spin_lock (&queue->lock); + ent->state =FRRS_USERSPACE ; +list_move_tail (&ent->list, &queue->ent_in_userspace); + ent->cmd =NULL ; +spin_unlock (&queue->lock); + +io_uring_cmd_done (cmd, ret, issue_flags); +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_conn *fc)
@@ -946,7 +936,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -fuse_uring_next_fuse_req (ent, queue, issue_flags); +if (fuse_uring_get_next_fuse_req (ent, queue)) +fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1194,20 +1185,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)return -EIOCBQUEUED ; } -static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, -ssize_t ret,unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; - -spin_lock (&queue->lock); - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); - ent->cmd =NULL ; -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd, ret, issue_flags); -} -/* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission
@@ -1224,8 +1201,9 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)if (!tw.cancel) { err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -fuse_uring_next_fuse_req (ent, queue, issue_flags); -return ; +if (!fuse_uring_get_next_fuse_req (ent, queue)) +return ; + err =0 ; } }else { err = -ECANCELED ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 14/25 fuse: refactor io-uring header copying to ring
In reply to Joanne KoongMove header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +21 -18
@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -637,13 +649,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;
@@ -659,9 +669,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -690,14 +699,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err =copy_to_user (&ent->headers->in_out, &req->in.h, -sizeof (req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.47.3
Re: [PATCH v3 14/25] fuse: refactor io-uring header copying to ring
Quoted 13 lines · Joanne Koong
Move header copying to ring logic into a new
copy_header_to_ring()function. This consolidates error handling.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)fs/fuse/dev_uring.c +0
-0
fs/fuse/dev_uring.c static int fuse_uring_out_header_has_err(struct fuse_out_header *oh, · line 580
Quoted 6 lines
fs/fuse/dev_uring.c +3
-0@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size)
Minor nit: The only part I don't like too much is the __always_inline. I had at least two times a debug issue where I didn't get much out of the trace and then used for fuse.ko
+ccflags-y += -g -O1\
+ -fno-inline-functions \ + -fno-omit-frame-pointer \ + -fno-optimize-sibling-calls \ + -fno-strict-aliasing \ + -fno-delete-null-pointer-checks \ + -fno-common \
After that the trace became very clear within 5min, before that I couldn't decode the trace.
Quoted 50 lines
fs/fuse/dev_uring.c +16
-13+{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)@@ -637,13 +649,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;@@ -659,9 +669,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -690,14 +699,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err =copy_to_user (&ent->headers->in_out, &req->in.h, -sizeof (req->in.h)); -if (err) {
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 706
Quoted 10 lines
fs/fuse/dev_uring.c +2
-5- err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Joanne Koong
maintainerRe: [PATCH v3 14/25] fuse: refactor io-uring header copying to ring
Quoted 18 lines · Bernd Schubert
Quoted 16 lines · Joanne Koong
Move header copying to ring logic into a new
copy_header_to_ring()function. This consolidates error handling.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)fs/fuse/dev_uring.c +0
-0@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; }
Quoted 7 lines
fs/fuse/dev_uring.c static int fuse_uring_out_header_has_err(struct fuse_out_header *oh, · line 580
Quoted 3 lines
fs/fuse/dev_uring.c +3
-0+static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size)Minor nit: The only part I don't like too much is the
__always_inline. I had at least two times a debug issue where I didn't get much out of the trace and then used for fuse.ko
Unfortunately the __always_inline here is necessary else builds with CONFIG_HARDENED_USERCOPY will complain because there's no metadata visibility into the header object which means __builtin_object_size() can't correctly determine the header size.
Thanks,
Joanne
Quoted 20 lines
(quoted diff) +7
-0+ccflags-y += -g -O1\ + -fno-inline-functions \ + -fno-omit-frame-pointer \ + -fno-optimize-sibling-calls \ + -fno-strict-aliasing \ + -fno-delete-null-pointer-checks \ + -fno-common \After that the trace became very clear within 5min, before that I couldn't decode the trace.
Quoted 8 lines
fs/fuse/dev_uring.c +8
-0+{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +}
Bernd Schubert
maintainerRe: [PATCH v3 14/25] fuse: refactor io-uring header copying to ring
Quoted 27 lines · Joanne Koong
Quoted 18 lines · Bernd Schubert
Quoted 16 lines · Joanne Koong
Move header copying to ring logic into a new
copy_header_to_ring()function. This consolidates error handling.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)fs/fuse/dev_uring.c +0
-0@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; }Quoted 7 lines
fs/fuse/dev_uring.c static int fuse_uring_out_header_has_err(struct fuse_out_header *oh, · line 580
Quoted 3 lines
fs/fuse/dev_uring.c +3
-0+static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size)Minor nit: The only part I don't like too much is the
__always_inline. I had at least two times a debug issue where I didn't get much out of the trace and then used for fuse.ko
Quoted 4 lines
Unfortunately the
__always_inlinehere is necessary else builds withCONFIG_HARDENED_USERCOPYwill complain because there's no metadata visibility into the header object which means__builtin_object_size()can't correctly determine the header size.
Oh I see, thanks for pointing to CONFIG_HARDENED_USERCOPY.
Thanks,
Bernd
Joanne Koong
maintainerPATCH v3 15/25 fuse: refactor io-uring header copying from ring
In reply to Joanne KoongMove header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +18 -6
@@ -587,6 +587,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } +static __always_inline int copy_header_from_ring(void *header, +const void __user *ring, +size_t header_size) +{ +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -597,10 +609,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_from_user (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, +sizeof (ring_in_out));if (err) -return -EFAULT ; +return err; err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, &iter);
@@ -794,10 +806,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_from_user (&req->out.h, &ent->headers->in_out, -sizeof (req->out.h)); + err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, +sizeof (req->out.h));if (err) { - req->out.h.error = -EFAULT ; + req->out.h.error = err; goto out; }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 16/25 fuse: use enum types for header copying
In reply to Joanne KoongUse enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when kernel-managed buffer rings are added.
--- fs/fuse/dev_uring.c | 57 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +47 -10
@@ -31,6 +31,15 @@ struct fuse_uring_pdu {static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +/* struct fuse_in_header / struct fuse_out_header */ +FUSE_URING_HEADER_IN_OUT , +/* per op code header */ +FUSE_URING_HEADER_OP , +/* struct fuse_uring_ent_in_out header */ +FUSE_URING_HEADER_RING_ENT , +}; +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -575,10 +584,32 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } -static __always_inline int copy_header_to_ring(void __user *ring, +static void __user *get_user_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} + +static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent, +enum fuse_uring_header_type type,const void *header,size_t header_size) { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) {pr_info_ratelimited ("Copying header to ring failed.\n" );return -EFAULT ;
@@ -587,10 +618,16 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } -static __always_inline int copy_header_from_ring(void *header, -const void __user *ring, +static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +void *header,size_t header_size) { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) {pr_info_ratelimited ("Copying header from ring failed.\n" );return -EFAULT ;
@@ -609,8 +646,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ring_in_out,sizeof (ring_in_out));if (err)return err;
@@ -661,7 +698,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_header_to_ring (&ent->headers->op_in, + err =copy_header_to_ring (ent,FUSE_URING_HEADER_OP , in_args->value, in_args->size);if (err)
@@ -681,8 +718,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; -return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ent_in_out,sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -711,7 +748,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ -return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +return copy_header_to_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->in.h,sizeof (req->in.h)); }
@@ -806,7 +843,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 17/25 fuse: refactor setting up copy state for payload copying
In reply to Joanne KoongAdd a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.
--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +24 -14
@@ -636,6 +636,27 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; } +static int setup_fuse_copy_state (struct fuse_copy_state *cs, +struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent *ent,int dir, +struct iov_iter *iter) +{ +int err; + + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } + +fuse_copy_init (cs, dir ==ITER_DEST , iter); + + cs->is_uring = true; + cs->req = req; + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -651,15 +672,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, - &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter);if (err)return err; -fuse_copy_init (&cs, false, &iter); - cs.is_uring = true; - cs.req = req; - err =fuse_copy_out_args (&cs, args, ring_in_out.payload_sz);fuse_copy_finish (&cs);return err;
@@ -682,15 +698,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter); +if (err)return err; - } - -fuse_copy_init (&cs, true, &iter); - cs.is_uring = true; - cs.req = req;if (num_args >0 ) {/*
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 18/25 fuse: support buffer copying for kernel addresses
In reply to Joanne KoongThis is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.
Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.
--- fs/fuse/dev.c | 23 +++++++++++++++++------ fs/fuse/fuse_dev_i.h | 7 ++++++- 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,/* Unmap and put previous page of userspace buffer */ void fuse_copy_finish (struct fuse_copy_state *cs) { +if (cs->is_kaddr) +return ; +if (cs->currbuf) {struct pipe_buffer *buf = cs->currbuf;
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)struct page *page;int err; +if (cs->is_kaddr) +return 0 ; + err =unlock_request (cs->req);if (err)return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size) {unsigned ncpy =min (*size, cs->len);if (val) { -void *pgaddr =kmap_local_page (cs->pg); -void *buf = pgaddr + cs->offset; +void *pgaddr, *buf; +if (!cs->is_kaddr) { + pgaddr =kmap_local_page (cs->pg); + buf = pgaddr + cs->offset; + }else { + buf = cs->kaddr + cs->offset; + }if (cs->write)memcpy (buf, *val, ncpy);else memcpy (*val, buf, ncpy); - -kunmap_local (pgaddr); +if (!cs->is_kaddr) +kunmap_local (pgaddr); *val += ncpy; } *size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, }while (count) { -if (cs->write && cs->pipebufs && folio) { +if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {/* * Can't control lifetime of pipe buffers, so always * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, }else {return fuse_ref_folio (cs, folio, offset, count); } - }else if (!cs->len) { + }else if (!cs->len && !cs->is_kaddr) {if (cs->move_folios && folio && offset ==0 && count == size) { err =fuse_try_move_folio (cs, foliop);
fs/fuse/fuse_dev_i.h +6 -1
@@ -28,12 +28,17 @@ struct fuse_copy_state {struct pipe_buffer *currbuf;struct pipe_inode_info *pipe;unsigned long nr_segs; -struct page *pg; +union { +struct page *pg; +void *kaddr; + };unsigned int len;unsigned int offset; bool write:1; bool move_folios:1; bool is_uring:1; +/* if set, use kaddr; otherwise use pg */ + bool is_kaddr:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 19/25 fuse: add io-uring kernel-managed buffer ring
In reply to Joanne KoongAdd io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction
b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.
--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)
fs/fuse/dev_uring.c +358 -65
@@ -10,6 +10,8 @@#include "fuse_trace.h" #include <linux/fs.h> +#include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> static bool __read_mostly enable_uring;
@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,#define FUSE_URING_IOV_SEGS 2/* header and payload */ +#define FUSE_URING_RINGBUF_GROUP 0 +#define FUSE_URING_FIXED_HEADERS_OFFSET 0bool fuse_uring_enabled (void ) {
@@ -276,20 +280,46 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +static int fuse_uring_buf_ring_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue, +unsigned int issue_flags) +{ +int err; + + err =io_uring_cmd_buf_ring_pin (cmd,FUSE_URING_RINGBUF_GROUP , + issue_flags, &queue->bufring); +if (err) +return err; + +if (!io_uring_cmd_is_kmbuf_ring (cmd,FUSE_URING_RINGBUF_GROUP , + issue_flags)) { +io_uring_cmd_buf_ring_unpin (cmd, +FUSE_URING_RINGBUF_GROUP , + issue_flags); +return -EINVAL ; + } + + queue->use_bufring = true; + +return 0 ; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,bool use_bufring,unsigned int issue_flags) {struct fuse_conn *fc = ring->fc;struct fuse_ring_queue *queue;struct list_head *pq; +int err; queue =kzalloc (sizeof (*queue),GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =kcalloc (FUSE_PQ_HASH_SIZE ,sizeof (struct list_head ),GFP_KERNEL );if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;
@@ -307,6 +337,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, queue->fpq.processing = pq;fuse_pqueue_init (&queue->fpq); +if (use_bufring) { + err =fuse_uring_buf_ring_setup (cmd, queue, issue_flags); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fc->lock);if (ring->queues[qid]) {spin_unlock (&fc->lock);
@@ -584,6 +623,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +struct iov_iter *headers_iter) +{ +size_t offset; + +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +/* No offset - start of header */ + offset =0 ; +break ; +case FUSE_URING_HEADER_OP : + offset = offsetof(struct fuse_uring_req_header , op_in); +break ; +case FUSE_URING_HEADER_RING_ENT : + offset = offsetof(struct fuse_uring_req_header , ring_ent_in_out); +break ; +default : +WARN_ONCE (1 ,"Invalid header type: %d\n" , type); +return -EINVAL ; + } + + *headers_iter = ent->headers_iter; +if (offset) +iov_iter_advance (headers_iter, offset); + +return 0 ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {
@@ -605,17 +673,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; + + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; + +if (copy_to_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (copy_to_user (ring, header, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header to ring failed: " +"header_type=%u, header_size=%zu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
@@ -623,17 +712,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,void *header, size_t header_size) { -const void __user *ring =get_user_ring_header (ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; + + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; + +if (copy_from_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (copy_from_user (header, ring, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header from ring failed: " +"header_type=%u, header_size=%zu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static int setup_fuse_copy_state(struct fuse_copy_state *cs,
@@ -643,14 +753,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, {int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (!ent->queue->use_bufring) { + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer " +"failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter); +if (ent->queue->use_bufring) { + cs->is_kaddr = true; + cs->len = ent->payload_kvec.iov_len; + cs->kaddr = ent->payload_kvec.iov_base; + } + cs->is_uring = true; cs->req = req;
@@ -762,6 +881,103 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent, +unsigned int issue_flags) + __must_hold(&queue->lock) +{ +struct io_br_sel sel; +size_t len =0 ; + +lockdep_assert_held (&ent->queue->lock); + +/* Get a buffer to use for the payload */ + sel =io_ring_buffer_select (cmd_to_io_kiocb (ent->cmd), &len, + ent->queue->bufring, issue_flags); +if (sel.val) +return sel.val; +if (!sel.kaddr) +return -ENOENT ; + + ent->payload_kvec.iov_base = sel.kaddr; + ent->payload_kvec.iov_len = len; + ent->ringbuf_buf_id = sel.buf_id; + +return 0 ; +} + +static void fuse_uring_clean_up_buffer (struct fuse_ring_ent *ent, +unsigned int issue_flags) + __must_hold(&queue->lock) +{ +struct kvec *kvec = &ent->payload_kvec; + +lockdep_assert_held (&ent->queue->lock); + +if (!ent->queue->use_bufring || !kvec->iov_base) +return ; + +WARN_ON_ONCE (io_uring_cmd_kmbuffer_recycle (ent->cmd, +FUSE_URING_RINGBUF_GROUP , + (u64 )kvec->iov_base, + kvec->iov_len, + ent->ringbuf_buf_id, + issue_flags)); + +memset (kvec,0 ,sizeof (*kvec)); +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req, +unsigned int issue_flags) +{ +bool buffer_selected; +bool has_payload; + +if (!ent->queue->use_bufring) +return 0 ; + + ent->headers_iter.data_source = false; + + buffer_selected = ent->payload_kvec.iov_base !=0 ; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent, issue_flags); + +if (!has_payload && buffer_selected) +fuse_uring_clean_up_buffer (ent, issue_flags); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req,unsigned int dir, +unsigned issue_flags) +{ +if (!ent->queue->use_bufring) +return 0 ; + +if (dir ==ITER_SOURCE ) { + ent->headers_iter.data_source = true; +return 0 ; + } + + ent->headers_iter.data_source = false; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent, issue_flags); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {
@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + } + } -return req; +fuse_uring_clean_up_buffer (ent, issue_flags); +return NULL ; }/*
@@ -878,7 +1102,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,* Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +unsigned int issue_flags) {int err;struct fuse_req *req;
@@ -886,7 +1111,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent, retry:spin_lock (&queue->lock);fuse_uring_ent_avail (ent, queue); - req =fuse_uring_ent_assign_req (ent); + req =fuse_uring_ent_assign_req (ent, issue_flags);spin_unlock (&queue->lock);if (req) {
@@ -927,6 +1152,38 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,io_uring_cmd_done (cmd, ret, issue_flags); } +static void fuse_uring_headers_cleanup (struct fuse_ring_ent *ent, +unsigned int issue_flags) +{ +if (!ent->queue->use_bufring) +return ; + +WARN_ON_ONCE (io_uring_cmd_fixed_index_put (ent->cmd, +FUSE_URING_FIXED_HEADERS_OFFSET , + issue_flags)); +} + +static int fuse_uring_headers_prep (struct fuse_ring_ent *ent,unsigned int dir, +unsigned int issue_flags) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct io_uring_cmd *cmd = ent->cmd; +unsigned int offset; +int err; + +if (!ent->queue->use_bufring) +return 0 ; + + offset = ent->fixed_buf_id * header_size; + + err =io_uring_cmd_fixed_index_get (cmd,FUSE_URING_FIXED_HEADERS_OFFSET , + offset, header_size, dir, + &ent->headers_iter, issue_flags); + +WARN_ON_ONCE (err); +return err; +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_conn *fc)
@@ -940,6 +1197,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,unsigned int qid =READ_ONCE (cmd_req->qid);struct fuse_pqueue *fpq;struct fuse_req *req; +bool send; err = -ENOTCONN ;if (!ring)
@@ -990,7 +1248,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); -fuse_uring_commit (ent, req, issue_flags); + + err =fuse_uring_headers_prep (ent,ITER_SOURCE , issue_flags); +if (err) +fuse_uring_req_end (ent, req, err); +else +fuse_uring_commit (ent, req, issue_flags);/* * Fetching the next request is absolutely required as queued
@@ -998,7 +1261,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) + send =fuse_uring_get_next_fuse_req (ent, queue, issue_flags); +fuse_uring_headers_cleanup (ent, issue_flags); +if (send)fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1094,39 +1359,48 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (-ENOMEM ); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + +if (queue->use_bufring) { + ent->fixed_buf_id =READ_ONCE (cmd->sqe->buf_index); +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->headers = iov[0 ].iov_base; ent->payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/*
@@ -1137,6 +1411,8 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, unsigned int issue_flags, struct fuse_conn *fc) { const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe_cmd(cmd->sqe); +unsigned int init_flags =READ_ONCE (cmd_req->init.flags); +bool use_bufring = init_flags &FUSE_URING_BUF_RING ;struct fuse_ring *ring =smp_load_acquire (&fc->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;
@@ -1157,9 +1433,13 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, queue = ring->queues[qid];if (!queue) { - queue =fuse_uring_create_queue (ring, qid); -if (!queue) -return err; + queue =fuse_uring_create_queue (cmd, ring, qid, use_bufring, + issue_flags); +if (IS_ERR (queue)) +return PTR_ERR (queue); + }else { +if (queue->use_bufring != use_bufring) +return -EINVAL ; }/*
@@ -1258,15 +1538,19 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)struct io_uring_cmd *cmd =io_uring_cmd_from_tw (tw_req);struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send = true;int err;if (!tw.cancel) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } +if (fuse_uring_headers_prep (ent,ITER_DEST , issue_flags)) +return ; + +if (fuse_uring_prepare_send (ent, ent->fuse_req)) + send =fuse_uring_get_next_fuse_req (ent, queue, issue_flags); +fuse_uring_headers_cleanup (ent, issue_flags); +if (!send) +return ; + err =0 ; }else { err = -ECANCELED ; }
@@ -1325,14 +1609,20 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent =list_first_entry_or_null (&queue->ent_avail_queue, struct fuse_ring_ent, list); -if (ent) -fuse_uring_add_req_to_ring_ent (ent, req); -else -list_add_tail (&req->list, &queue->fuse_req_queue); -spin_unlock (&queue->lock); +if (ent) { + err =fuse_uring_prep_buffer (ent, req,ITER_DEST , +IO_URING_F_UNLOCKED ); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return ; + } +WARN_ON_ONCE (err != -ENOENT ); + } -if (ent) -fuse_uring_dispatch_ent (ent); +list_add_tail (&req->list, &queue->fuse_req_queue); +spin_unlock (&queue->lock);return ;
@@ -1350,6 +1640,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)struct fuse_ring *ring = fc->ring;struct fuse_ring_queue *queue;struct fuse_ring_ent *ent =NULL ; +int err; queue =fuse_uring_task_to_queue (ring);if (!queue)
@@ -1382,14 +1673,16 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list);if (ent && req) { -fuse_uring_add_req_to_ring_ent (ent, req); -spin_unlock (&queue->lock); - -fuse_uring_dispatch_ent (ent); - }else { -spin_unlock (&queue->lock); + err =fuse_uring_prep_buffer (ent, req,ITER_DEST , +IO_URING_F_UNLOCKED ); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return true; + } } - +spin_unlock (&queue->lock);return true; }
fs/fuse/dev_uring_i.h +27 -3
@@ -7,6 +7,8 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> +#include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +40,25 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +union { +/* queue->use_bufring == false */ +struct { +/* userspace buffers */ +struct fuse_uring_req_header __user *headers; +void __user *payload; + }; +/* queue->use_bufring == true */ +struct { +struct iov_iter headers_iter; +struct kvec payload_kvec; +/* +* This needs to be tracked in order to properly recycle +* the buffer when done with it +*/ +unsigned int ringbuf_buf_id; +unsigned int fixed_buf_id; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
@@ -99,6 +117,12 @@ struct fuse_ring_queue {unsigned int active_background;bool stopped; + +/* true if kernel-managed buffer ring is used */ + bool use_bufring: 1; + +/* synchronized by the queue lock */ +struct io_buffer_list *bufring; };/**
include/uapi/linux/fuse.h +14 -1
@@ -240,6 +240,9 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add fuse_uring_cmd_req init flags */ #ifndef _LINUX_FUSE_H
@@ -1294,6 +1297,9 @@ enum fuse_uring_cmd { FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, }; +/* fuse_uring_cmd_req init flags */ +#define FUSE_URING_BUF_RING (1 << 0) +/** * In the 80B command area of the SQE. */
@@ -1305,7 +1311,14 @@ struct fuse_uring_cmd_req {/* queue the command is for (queue index) */ uint16_t qid; - uint8_t padding[6]; + +union { +struct { +uint16_t flags; + } init; + }; + + uint8_t padding[4]; };#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
Bernd Schubert
maintainerRe: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 390 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +241
-23@@ -10,6 +10,8 @@#include "fuse_trace.h" #include <linux/fs.h> +#include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> static bool __read_mostly enable_uring;@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,#define FUSE_URING_IOV_SEGS 2/* header and payload */ +#define FUSE_URING_RINGBUF_GROUP 0 +#define FUSE_URING_FIXED_HEADERS_OFFSET 0bool fuse_uring_enabled (void ) {@@ -276,20 +280,46 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +static int fuse_uring_buf_ring_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue, +unsigned int issue_flags) +{ +int err; + + err =io_uring_cmd_buf_ring_pin (cmd,FUSE_URING_RINGBUF_GROUP , + issue_flags, &queue->bufring); +if (err) +return err; + +if (!io_uring_cmd_is_kmbuf_ring (cmd,FUSE_URING_RINGBUF_GROUP , + issue_flags)) { +io_uring_cmd_buf_ring_unpin (cmd, +FUSE_URING_RINGBUF_GROUP , + issue_flags); +return -EINVAL ; + } + + queue->use_bufring = true; + +return 0 ; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,bool use_bufring,unsigned int issue_flags) {struct fuse_conn *fc = ring->fc;struct fuse_ring_queue *queue;struct list_head *pq; +int err; queue =kzalloc (sizeof (*queue),GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =kcalloc (FUSE_PQ_HASH_SIZE ,sizeof (struct list_head ),GFP_KERNEL );if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;@@ -307,6 +337,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, queue->fpq.processing = pq;fuse_pqueue_init (&queue->fpq); +if (use_bufring) { + err =fuse_uring_buf_ring_setup (cmd, queue, issue_flags); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fc->lock);if (ring->queues[qid]) {spin_unlock (&fc->lock);@@ -584,6 +623,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +struct iov_iter *headers_iter) +{ +size_t offset; + +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +/* No offset - start of header */ + offset =0 ; +break ; +case FUSE_URING_HEADER_OP : + offset = offsetof(struct fuse_uring_req_header , op_in); +break ; +case FUSE_URING_HEADER_RING_ENT : + offset = offsetof(struct fuse_uring_req_header , ring_ent_in_out); +break ; +default : +WARN_ONCE (1 ,"Invalid header type: %d\n" , type); +return -EINVAL ; + } + + *headers_iter = ent->headers_iter; +if (offset) +iov_iter_advance (headers_iter, offset); + +return 0 ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {@@ -605,17 +673,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; + + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; + +if (copy_to_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (copy_to_user (ring, header, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header to ring failed: " +"header_type=%u, header_size=%zu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,@@ -623,17 +712,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,void *header, size_t header_size) { -const void __user *ring =get_user_ring_header (ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; + + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; + +if (copy_from_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (copy_from_user (header, ring, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header from ring failed: " +"header_type=%u, header_size=%zu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static int setup_fuse_copy_state(struct fuse_copy_state *cs,@@ -643,14 +753,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, {int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (!ent->queue->use_bufring) { + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer " +"failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter); +if (ent->queue->use_bufring) { + cs->is_kaddr = true; + cs->len = ent->payload_kvec.iov_len; + cs->kaddr = ent->payload_kvec.iov_base; + } + cs->is_uring = true; cs->req = req;@@ -762,6 +881,103 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent, +unsigned int issue_flags) + __must_hold(&queue->lock) +{ +struct io_br_sel sel; +size_t len =0 ; + +lockdep_assert_held (&ent->queue->lock); + +/* Get a buffer to use for the payload */ + sel =io_ring_buffer_select (cmd_to_io_kiocb (ent->cmd), &len, + ent->queue->bufring, issue_flags); +if (sel.val) +return sel.val; +if (!sel.kaddr) +return -ENOENT ; + + ent->payload_kvec.iov_base = sel.kaddr; + ent->payload_kvec.iov_len = len; + ent->ringbuf_buf_id = sel.buf_id; + +return 0 ; +} + +static void fuse_uring_clean_up_buffer (struct fuse_ring_ent *ent, +unsigned int issue_flags) + __must_hold(&queue->lock) +{ +struct kvec *kvec = &ent->payload_kvec; + +lockdep_assert_held (&ent->queue->lock); + +if (!ent->queue->use_bufring || !kvec->iov_base) +return ; + +WARN_ON_ONCE (io_uring_cmd_kmbuffer_recycle (ent->cmd, +FUSE_URING_RINGBUF_GROUP , + (u64 )kvec->iov_base, + kvec->iov_len, + ent->ringbuf_buf_id, + issue_flags)); + +memset (kvec,0 ,sizeof (*kvec)); +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req, +unsigned int issue_flags) +{ +bool buffer_selected; +bool has_payload; + +if (!ent->queue->use_bufring) +return 0 ; + + ent->headers_iter.data_source = false; + + buffer_selected = ent->payload_kvec.iov_base !=0 ; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent, issue_flags); + +if (!has_payload && buffer_selected) +fuse_uring_clean_up_buffer (ent, issue_flags); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req,unsigned int dir, +unsigned issue_flags) +{ +if (!ent->queue->use_bufring) +return 0 ; + +if (dir ==ITER_SOURCE ) { + ent->headers_iter.data_source = true; +return 0 ; + } + + ent->headers_iter.data_source = false; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent, issue_flags); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }
Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into FRRS_AVAILABLE and nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 46 lines · Bernd Schubert
Quoted 42 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +3
-1@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);
Quoted 16 lines
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into
FRRS_AVAILABLEand nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.
If the queue only has one ring ent and this fails, the request gets reassigned to the ent whenever ->send_req() is next triggered. I don't think this is a new edge case introduced by kmbufs; in the existing code, fuse_uring_commit_fetch() -> fuse_uring_get_next_fuse_req() -> fuse_uring_send_next_to_ring() -> fuse_uring_prepare_send() could fail if any of the copying fails, in which case we end up in the same position of the ent getting assigned the next request whenever ->send_req() is next triggered.
Thanks,
Joanne
Quoted 3 lines
Thanks,
Bernd
Re: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 64 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 42 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +3
-1@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);Quoted 16 lines
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into
FRRS_AVAILABLEand nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.
Quoted 8 lines
If the queue only has one ring ent and this fails, the request gets reassigned to the ent whenever ->send_req() is next triggered. I don't think this is a new edge case introduced by kmbufs; in the existing code,
fuse_uring_commit_fetch()->fuse_uring_get_next_fuse_req()->fuse_uring_send_next_to_ring()->fuse_uring_prepare_send()could fail if any of the copying fails, in which case we end up in the same position of the ent getting assigned the next request whenever ->send_req() is next triggered.
I don't manage to check right now (need to solve another imbalance with reduced rings right now), but every failed copy is *supposed* to end up in a request failure. Why should it block, if the copy failed? It would be a bug if it does not right now and should be solved.
Regarding your copy, I don't think waiting for for the next ->send_req() is an option, it might be never.
One solution might be a single entry in any of the queues or in a separate queue that doesn't have buf-rings - i.e. it can go slowly, but it must not block. Some wake-up task retry might also work, but would be timeout based.
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 77 lines · Bernd Schubert
Quoted 64 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 42 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +3
-1@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);Quoted 16 lines
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into
FRRS_AVAILABLEand nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.Quoted 8 lines
If the queue only has one ring ent and this fails, the request gets reassigned to the ent whenever ->send_req() is next triggered. I don't think this is a new edge case introduced by kmbufs; in the existing code,
fuse_uring_commit_fetch()->fuse_uring_get_next_fuse_req()->fuse_uring_send_next_to_ring()->fuse_uring_prepare_send()could fail if any of the copying fails, in which case we end up in the same position of the ent getting assigned the next request whenever ->send_req() is next triggered.
Quoted 2 lines
I don't manage to check right now (need to solve another imbalance with reduced rings right now), but every failed copy is *supposed* to end up
Thanks for your work on the reduced rings, I'm looking forward to seeing your patchset.
Quoted 9 lines
in a request failure. Why should it block, if the copy failed? It would be a bug if it does not right now and should be solved.
Regarding your copy, I don't think waiting for for the next ->send_req() is an option, it might be never.
One solution might be a single entry in any of the queues or in a separate queue that doesn't have buf-rings - i.e. it can go slowly, but it must not block. Some wake-up task retry might also work, but would be timeout based.
Ah, so your concern is about the request taking too long to complete, not about the ent not being available again to send requests. In the existing code, yes if the next request can't be sent after a commit then that next request is immediately terminated. For the kmbuf case, the fuse_uring_next_req_update_buffer() call only fails if all the buffers are currently being used. The request will be picked up when the next buffer becomes available / is recycled back, which happens when the request(s) sent out to the server completes and is committed, if a ->send_req() hasn't already been triggered by then.
Thanks,
Joanne
Quoted 5 lines
Thanks,
Bernd
Re: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 94 lines · Joanne Koong
Quoted 77 lines · Bernd Schubert
Quoted 64 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 42 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +3
-1@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);Quoted 16 lines
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into
FRRS_AVAILABLEand nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.Quoted 8 lines
If the queue only has one ring ent and this fails, the request gets reassigned to the ent whenever ->send_req() is next triggered. I don't think this is a new edge case introduced by kmbufs; in the existing code,
fuse_uring_commit_fetch()->fuse_uring_get_next_fuse_req()->fuse_uring_send_next_to_ring()->fuse_uring_prepare_send()could fail if any of the copying fails, in which case we end up in the same position of the ent getting assigned the next request whenever ->send_req() is next triggered.Quoted 2 lines
I don't manage to check right now (need to solve another imbalance with reduced rings right now), but every failed copy is *supposed* to end up
Thanks for your work on the reduced rings, I'm looking forward to seeing your patchset.
Quoted 9 lines
in a request failure. Why should it block, if the copy failed? It would be a bug if it does not right now and should be solved.
Regarding your copy, I don't think waiting for for the next ->send_req() is an option, it might be never.
One solution might be a single entry in any of the queues or in a separate queue that doesn't have buf-rings - i.e. it can go slowly, but it must not block. Some wake-up task retry might also work, but would be timeout based.
Quoted 9 lines
Ah, so your concern is about the request taking too long to complete, not about the ent not being available again to send requests. In the existing code, yes if the next request can't be sent after a commit then that next request is immediately terminated. For the kmbuf case, the
fuse_uring_next_req_update_buffer()call only fails if all the buffers are currently being used. The request will be picked up when the next buffer becomes available / is recycled back, which happens when the request(s) sent out to the server completes and is committed, if a ->send_req() hasn't already been triggered by then.
In the simple one ring entry example there wouldn't be another in-fly request - the request would basically hang forever, if some reasons the ring buffer is not available. It *shouldn't* happen, but what if for example the same buffer would be used for zery-copy another subsystem now consumes them? No idea if that could happen - with these buffers there an additional complexity, which I don't understand from fuse point of view. Let's just assume there would be some kind of ring buffer bug and requests would now hang - how would we debug this if it just hangs without any log or failure message?
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v3 19/25] fuse: add io-uring kernel-managed buffer ring
Quoted 109 lines · Bernd Schubert
Quoted 94 lines · Joanne Koong
Quoted 77 lines · Bernd Schubert
Quoted 64 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 42 lines · Joanne Koong
Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interactionb) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "
use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 423 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 30 ++- include/uapi/linux/fuse.h | 15 +- 3 files changed, 399 insertions(+), 69 deletions(-)fs/fuse/dev_uring.c +3
-1@@ -824,21 +1040,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);Quoted 16 lines
fs/fuse/dev_uring.c static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, · line 1061
Quoted 10 lines
fs/fuse/dev_uring.c +6
-2/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { + err =fuse_uring_next_req_update_buffer (ent, req, issue_flags); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + }Hmm, who/what is going to handle the request if this fails? Let's say we have just one ring entry per queue and now it fails here - this ring entry will go into
FRRS_AVAILABLEand nothing will pull from the queue anymore. I guess it _should_ not happen, some protection would be good. In order to handle it, at least one other ent needs to be in flight.Quoted 8 lines
If the queue only has one ring ent and this fails, the request gets reassigned to the ent whenever ->send_req() is next triggered. I don't think this is a new edge case introduced by kmbufs; in the existing code,
fuse_uring_commit_fetch()->fuse_uring_get_next_fuse_req()->fuse_uring_send_next_to_ring()->fuse_uring_prepare_send()could fail if any of the copying fails, in which case we end up in the same position of the ent getting assigned the next request whenever ->send_req() is next triggered.Quoted 2 lines
I don't manage to check right now (need to solve another imbalance with reduced rings right now), but every failed copy is *supposed* to end up
Thanks for your work on the reduced rings, I'm looking forward to seeing your patchset.
Quoted 9 lines
in a request failure. Why should it block, if the copy failed? It would be a bug if it does not right now and should be solved.
Regarding your copy, I don't think waiting for for the next ->send_req() is an option, it might be never.
One solution might be a single entry in any of the queues or in a separate queue that doesn't have buf-rings - i.e. it can go slowly, but it must not block. Some wake-up task retry might also work, but would be timeout based.Quoted 9 lines
Ah, so your concern is about the request taking too long to complete, not about the ent not being available again to send requests. In the existing code, yes if the next request can't be sent after a commit then that next request is immediately terminated. For the kmbuf case, the
fuse_uring_next_req_update_buffer()call only fails if all the buffers are currently being used. The request will be picked up when the next buffer becomes available / is recycled back, which happens when the request(s) sent out to the server completes and is committed, if a ->send_req() hasn't already been triggered by then.
Quoted 2 lines
In the simple one ring entry example there wouldn't be another in-fly request - the request would basically hang forever, if some reasons the
If there's no in-flight request, then the buffer will always be available since there's no other request using it. For the one ring-entry example with 1 registered buffer in the bufring,
request A -> reserves buffer 1 -> request A gets sent to the server
request B gets enqueued -> no ent is available -> request is added to the list for later servicing
server sends a reply for request A -> kernel processes the reply in fuse_uring_commit_fetch() -> buffer 1 is recycled -> server gets the next request (request B) -> server uses buffer 1 to service request B
Quoted 3 lines
ring buffer is not available. It *shouldn't* happen, but what if for example the same buffer would be used for zery-copy another subsystem now consumes them? No idea if that could happen - with these buffers
The buffer is registered by libfuse for the fuse connection and can *only* be used by that fuse connection. It cannot be used by another subsystem.
Quoted 4 lines
there an additional complexity, which I don't understand from fuse point of view. Let's just assume there would be some kind of ring buffer bug and requests would now hang - how would we debug this if it just hangs without any log or failure message?
The ring buffer can only run out of entries if there are in-flight requests that are currently using all the buffers. Those requests will complete and recycle back the buffer they used and fetch the next request, which will use the buffer that was just recycled back.
Thanks,
Joanne
Quoted 4 lines
Thanks,
Bernd
Joanne Koong
maintainerPATCH v3 20/25 io_uring/rsrc: rename io_buffer_register_bvec()/io_buffer_unregister_bvec()
In reply to Joanne KoongCurrently, io_buffer_register_bvec() takes in a request. In preparation for supporting kernel-populated buffers in fuse io-uring (which will need to register bvecs directly, not through a struct request), rename this to io_buffer_register_request().
A subsequent patch will commandeer the "io_buffer_register_bvec()" function name to support registering bvecs directly.
Rename io_buffer_unregister_bvec() to a more generic name, io_buffer_unregister(), as both io_buffer_register_request() and io_buffer_register_bvec() callers will use it for unregistration.
--- Documentation/block/ublk.rst | 14 +++++++------- drivers/block/ublk_drv.c | 18 +++++++++--------- include/linux/io_uring/cmd.h | 26 ++++++++++++++++++++------ io_uring/rsrc.c | 14 +++++++------- 4 files changed, 43 insertions(+), 29 deletions(-)
Documentation/block/ublk.rst +7 -7
@@ -326,17 +326,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_buffer_register_request()` and `io_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request +`io_buffer_register_request()` for ublk server to register client request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the -operation is completed. +calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed +to be live between calling `io_buffer_register_request()` and +`io_buffer_unregister()`. Any io_uring operation which supports this kind of +kernel buffer will grab one reference of the buffer until the operation is +completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and be trusted, because it is ublk server's responsibility to make sure IO buffer
drivers/block/ublk_drv.c +9 -9
@@ -1202,8 +1202,8 @@ __ublk_do_auto_buf_reg(const struct ublk_queue *ubq, struct request *req, {int ret; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, - io->buf.auto_reg.index, issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, + io->buf.auto_reg.index, issue_flags);if (ret) {if (io->buf.auto_reg.flags &UBLK_AUTO_BUF_REG_FALLBACK ) {ublk_auto_buf_reg_fallback (ubq, req->tag);
@@ -2166,8 +2166,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,if (!req)return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, index, + issue_flags);if (ret) {ublk_put_req_ref (io, req);return ret;
@@ -2198,8 +2198,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,if (!ublk_dev_support_zero_copy (ub) || !ublk_rq_has_data (req))return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, index, + issue_flags);if (ret)return ret;
@@ -2214,7 +2214,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,if (!(ub->dev_info.flags &UBLK_F_SUPPORT_ZERO_COPY ))return -EINVAL ; -return io_buffer_unregister_bvec (cmd, index, issue_flags); +return io_buffer_unregister (cmd, index, issue_flags); }static int ublk_check_fetch_buf (const struct ublk_device *ub,__u64 buf_addr)
@@ -2350,7 +2350,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out;/* -* io_buffer_unregister_bvec() doesn't access the ubq or io, +* io_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR (cmd_op) ==UBLK_IO_UNREGISTER_IO_BUF )
@@ -2420,7 +2420,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,/* can't touch 'ublk_io' any more */ if (buf_idx !=UBLK_INVALID_BUF_IDX ) -io_buffer_unregister_bvec (cmd, buf_idx, issue_flags); +io_buffer_unregister (cmd, buf_idx, issue_flags);if (req_op (req) ==REQ_OP_ZONE_APPEND ) req->__sector = addr;if (compl)
include/linux/io_uring/cmd.h +20 -6
@@ -102,6 +102,12 @@ int io_uring_cmd_kmbuffer_recycle(struct io_uring_cmd *cmd,int io_uring_cmd_is_kmbuf_ring (struct io_uring_cmd *ioucmd,unsigned int buf_group,unsigned int issue_flags); + +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags); +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -185,6 +191,20 @@ static inline int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_buffer_unregister (struct io_uring_cmd *cmd, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)
@@ -234,10 +254,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags); -#endif /* _LINUX_IO_URING_CMD_H */
io_uring/rsrc.c +7 -7
@@ -936,9 +936,9 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) {struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;
@@ -998,10 +998,10 @@ int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_register_bvec); +EXPORT_SYMBOL_GPL (io_buffer_register_request); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags) +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) {struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;
@@ -1031,7 +1031,7 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_unregister_bvec); +EXPORT_SYMBOL_GPL (io_buffer_unregister);static int validate_fixed_range (u64 buf_addr,size_t len,const struct io_mapped_ubuf *imu)
Signature · 1 line
2.47.3
Re: [PATCH v3 20/25] io_uring/rsrc: rename io_buffer_register_bvec()/io_buffer_unregister_bvec()
Quoted 6 lines · Joanne Koong
Currently,
io_buffer_register_bvec()takes in a request. In preparation for supporting kernel-populated buffers in fuse io-uring (which will need to register bvecs directly, not through a struct request), rename this toio_buffer_register_request().
Quoted 8 lines
A subsequent patch will commandeer the "
io_buffer_register_bvec()" function name to support registering bvecs directly.Rename
io_buffer_unregister_bvec()to a more generic name,io_buffer_unregister(), as bothio_buffer_register_request()andio_buffer_register_bvec()callers will use it for unregistration.Signed-off-by Joanne Koong <[email protected]>
Quoted 192 lines
--- Documentation/block/ublk.rst | 14 +++++++------- drivers/block/ublk_drv.c | 18 +++++++++--------- include/linux/io_uring/cmd.h | 26 ++++++++++++++++++++------ io_uring/rsrc.c | 14 +++++++------- 4 files changed, 43 insertions(+), 29 deletions(-)Documentation/block/ublk.rst +7
-7@@ -326,17 +326,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_buffer_register_request()` and `io_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request +`io_buffer_register_request()` for ublk server to register client request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the -operation is completed. +calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed +to be live between calling `io_buffer_register_request()` and +`io_buffer_unregister()`. Any io_uring operation which supports this kind of +kernel buffer will grab one reference of the buffer until the operation is +completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and be trusted, because it is ublk server's responsibility to make sure IO bufferdrivers/block/ublk_drv.c +9
-9@@ -1202,8 +1202,8 @@ __ublk_do_auto_buf_reg(const struct ublk_queue *ubq, struct request *req, {int ret; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, - io->buf.auto_reg.index, issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, + io->buf.auto_reg.index, issue_flags);if (ret) {if (io->buf.auto_reg.flags &UBLK_AUTO_BUF_REG_FALLBACK ) {ublk_auto_buf_reg_fallback (ubq, req->tag);@@ -2166,8 +2166,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,if (!req)return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, index, + issue_flags);if (ret) {ublk_put_req_ref (io, req);return ret;@@ -2198,8 +2198,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,if (!ublk_dev_support_zero_copy (ub) || !ublk_rq_has_data (req))return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_buffer_register_request (cmd, req, ublk_io_release, index, + issue_flags);if (ret)return ret;@@ -2214,7 +2214,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,if (!(ub->dev_info.flags &UBLK_F_SUPPORT_ZERO_COPY ))return -EINVAL ; -return io_buffer_unregister_bvec (cmd, index, issue_flags); +return io_buffer_unregister (cmd, index, issue_flags); }static int ublk_check_fetch_buf (const struct ublk_device *ub,__u64 buf_addr)@@ -2350,7 +2350,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out;/* -* io_buffer_unregister_bvec() doesn't access the ubq or io, +* io_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR (cmd_op) ==UBLK_IO_UNREGISTER_IO_BUF )@@ -2420,7 +2420,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,/* can't touch 'ublk_io' any more */ if (buf_idx !=UBLK_INVALID_BUF_IDX ) -io_buffer_unregister_bvec (cmd, buf_idx, issue_flags); +io_buffer_unregister (cmd, buf_idx, issue_flags);if (req_op (req) ==REQ_OP_ZONE_APPEND ) req->__sector = addr;if (compl)include/linux/io_uring/cmd.h +20
-6@@ -102,6 +102,12 @@ int io_uring_cmd_kmbuffer_recycle(struct io_uring_cmd *cmd,int io_uring_cmd_is_kmbuf_ring (struct io_uring_cmd *ioucmd,unsigned int buf_group,unsigned int issue_flags); + +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags); +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags);#else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,@@ -185,6 +191,20 @@ static inline int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_buffer_unregister (struct io_uring_cmd *cmd, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif static inline struct io_uring_cmd *io_uring_cmd_from_tw (struct io_tw_req tw_req)@@ -234,10 +254,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags); -#endif /* _LINUX_IO_URING_CMD_H */ io_uring/rsrc.c +7
-8@@ -936,9 +936,9 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) {struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;@@ -998,10 +998,10 @@ int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_register_bvec); +EXPORT_SYMBOL_GPL (io_buffer_register_request); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags) +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) {struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;@@ -1031,7 +1031,7 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_unregister_bvec); +EXPORT_SYMBOL_GPL (io_buffer_unregister);static int validate_fixed_range (u64 buf_addr,size_t len,const struct io_mapped_ubuf *imu) --2.47.3
Joanne Koong
maintainerPATCH v3 21/25 io_uring/rsrc: split io_buffer_register_request() logic
In reply to Joanne KoongSplit the main initialization logic in io_buffer_register_request() into a helper function.
This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
--- io_uring/rsrc.c | 89 ++++++++++++++++++++++++++++++------------------- 1 file changed, 54 insertions(+), 35 deletions(-)
io_uring/rsrc.c +54 -35
@@ -936,67 +936,86 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static struct io_mapped_ubuf *io_kernel_buffer_init (struct io_ring_ctx *ctx, +unsigned int nr_bvecs, +unsigned int total_bytes, +u8 dir, +void (*release)(void *), +void *priv, +unsigned int index) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return ERR_PTR (-EINVAL ); index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return ERR_PTR (-EBUSY ); node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return ERR_PTR (-ENOMEM ); -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return ERR_PTR (-ENOMEM ); } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + node->buf = imu; + data->nodes[index] = node; + +return imu; +} + +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes; + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq); + +io_ring_submit_lock (ctx, issue_flags); + + imu =io_kernel_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), + release, rq, index); +if (IS_ERR (imu)) { +io_ring_submit_unlock (ctx, issue_flags); +return PTR_ERR (imu); + } + + nr_bvecs =0 ;rq_for_each_bvec (bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; -unlock:io_ring_submit_unlock (ctx, issue_flags); -return ret; +return 0 ; }EXPORT_SYMBOL_GPL (io_buffer_register_request);
Signature · 1 line
2.47.3
Re: [PATCH v3 21/25] io_uring/rsrc: split io_buffer_register_request() logic
Quoted 104 lines · Joanne Koong
Split the main initialization logic in
io_buffer_register_request()into a helper function.This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
Signed-off-by Joanne Koong <[email protected]>--- io_uring/rsrc.c | 89 ++++++++++++++++++++++++++++++------------------- 1 file changed, 54 insertions(+), 35 deletions(-)io_uring/rsrc.c +32
-31@@ -936,67 +936,86 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static struct io_mapped_ubuf *io_kernel_buffer_init (struct io_ring_ctx *ctx, +unsigned int nr_bvecs, +unsigned int total_bytes, +u8 dir, +void (*release)(void *), +void *priv, +unsigned int index) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return ERR_PTR (-EINVAL ); index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return ERR_PTR (-EBUSY ); node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return ERR_PTR (-ENOMEM ); -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return ERR_PTR (-ENOMEM ); } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + node->buf = imu; + data->nodes[index] = node; + +return imu; +} + +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu;
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1001
Quoted 10 lines
io_uring/rsrc.c +10
-0+struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes; + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq);
Could combine these initializations with the variable declarations
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1008
Quoted 7 lines
io_uring/rsrc.c +7
-0+ +io_ring_submit_lock (ctx, issue_flags); + + imu =io_kernel_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), + release, rq, index); +if (IS_ERR (imu)) { +io_ring_submit_unlock (ctx, issue_flags);
I would prefer to leave the existing goto unlock; pattern. The goto pattern is easier to extend in the future with additional resource acquisitions. And keeping it would make the diff slightly smaller. For new functions with a single early return path, I don't feel all that strongly, but it seems like unnecessary refactoring of this existing code.
Other than that,
Quoted 20 lines
io_uring/rsrc.c +5
-5+return PTR_ERR (imu); + } + + nr_bvecs =0 ;rq_for_each_bvec (bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; -unlock:io_ring_submit_unlock (ctx, issue_flags); -return ret; +return 0 ; }EXPORT_SYMBOL_GPL (io_buffer_register_request); --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 21/25] io_uring/rsrc: split io_buffer_register_request() logic
Quoted 110 lines · Caleb Sander Mateos
Quoted 108 lines · Joanne Koong
Split the main initialization logic in
io_buffer_register_request()into a helper function.This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
Signed-off-by Joanne Koong <[email protected]>--- io_uring/rsrc.c | 89 ++++++++++++++++++++++++++++++------------------- 1 file changed, 54 insertions(+), 35 deletions(-)io_uring/rsrc.c +36
-31@@ -936,67 +936,86 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static struct io_mapped_ubuf *io_kernel_buffer_init (struct io_ring_ctx *ctx, +unsigned int nr_bvecs, +unsigned int total_bytes, +u8 dir, +void (*release)(void *), +void *priv, +unsigned int index) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return ERR_PTR (-EINVAL ); index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return ERR_PTR (-EBUSY ); node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return ERR_PTR (-ENOMEM ); -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return ERR_PTR (-ENOMEM ); } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + node->buf = imu; + data->nodes[index] = node; + +return imu; +} + +int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes; +
Quoted 8 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1001
Quoted 6 lines
io_uring/rsrc.c +6
-0+/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq);Could combine these initializations with the variable declarations
I don't have a preference but was thinking it would look unaesthetic with having that comment block in the declaration section. I don't think it matters though, so I'm happy to combine the two sections for v4.
Quoted 15 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1008
Quoted 8 lines
io_uring/rsrc.c +7
-0+ +io_ring_submit_lock (ctx, issue_flags); + + imu =io_kernel_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), + release, rq, index); +if (IS_ERR (imu)) { +io_ring_submit_unlock (ctx, issue_flags);I would prefer to leave the existing goto unlock; pattern. The goto pattern is easier to extend in the future with additional resource acquisitions. And keeping it would make the diff slightly smaller. For new functions with a single early return path, I don't feel all that strongly, but it seems like unnecessary refactoring of this existing code.
Sounds good, I'll leave in the existing goto unlock; pattern.
Thanks for reviewing this patchset.
Quoted 24 lines
Other than that,
Reviewed-by Caleb Sander Mateos <[email protected]>Quoted 20 lines
io_uring/rsrc.c +5
-5+return PTR_ERR (imu); + } + + nr_bvecs =0 ;rq_for_each_bvec (bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; -unlock:io_ring_submit_unlock (ctx, issue_flags); -return ret; +return 0 ; }EXPORT_SYMBOL_GPL (io_buffer_register_request); --2.47.3
Joanne Koong
maintainerPATCH v3 22/25 io_uring/rsrc: Allow buffer release callback to be optional
In reply to Joanne KoongThis is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which does not need a release callback.
--- io_uring/rsrc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
io_uring/rsrc.c +2 -1
@@ -148,7 +148,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)if (imu->acct_pages)io_unaccount_mem (ctx->user, ctx->mm_account, imu->acct_pages); - imu->release (imu->priv); +if (imu->release) + imu->release (imu->priv);io_free_imu (ctx, imu); }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v3 23/25 io_uring/rsrc: add io_buffer_register_bvec()
In reply to Joanne KoongAdd io_buffer_register_bvec() for registering a bvec array.
This is a preparatory patch for fuse-over-io-uring zero-copy.
--- include/linux/io_uring/cmd.h | 12 ++++++++++++ io_uring/rsrc.c | 27 +++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/linux/io_uring/cmd.h +12 -0
@@ -106,6 +106,9 @@ int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd,int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq,void (*release)(void *),unsigned int index,unsigned int issue_flags); +int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index,unsigned int issue_flags);int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags);#else
@@ -199,6 +202,15 @@ static inline int io_buffer_register_request(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline int io_buffer_register_bvec (struct io_uring_cmd *cmd, +struct bio_vec *bvs, +unsigned int nr_bvecs, +unsigned int total_bytes,u8 dir, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags)
io_uring/rsrc.c +27 -0
@@ -1020,6 +1020,33 @@ int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, }EXPORT_SYMBOL_GPL (io_buffer_register_request); +int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct io_mapped_ubuf *imu; +struct bio_vec *bvec; +int i; + +io_ring_submit_lock (ctx, issue_flags); + imu =io_kernel_buffer_init (ctx, nr_bvecs, total_bytes, dir,NULL , +NULL , index); +if (IS_ERR (imu)) { +io_ring_submit_unlock (ctx, issue_flags); +return PTR_ERR (imu); + } + + bvec = imu->bvec; +for (i =0 ; i < nr_bvecs; i++) + bvec[i] = bvs[i]; + +io_ring_submit_unlock (ctx, issue_flags); +return 0 ; +} +EXPORT_SYMBOL_GPL (io_buffer_register_bvec); +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags) {
Signature · 1 line
2.47.3
Re: [PATCH v3 23/25] io_uring/rsrc: add io_buffer_register_bvec()
Quoted 16 lines · Joanne Koong
Add
io_buffer_register_bvec()for registering a bvec array.This is a preparatory patch for fuse-over-io-uring zero-copy.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 12 ++++++++++++ io_uring/rsrc.c | 27 +++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)include/linux/io_uring/cmd.h +0
-0
include/linux/io_uring/cmd.h int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd, · line 109
Quoted 4 lines
include/linux/io_uring/cmd.h +1
-0@@ -106,6 +106,9 @@ int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd,int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq,void (*release)(void *),unsigned int index,unsigned int issue_flags); +int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct bio_vec *bvs,
Could take const struct bio_vec *? Might also be helpful to document that this internally makes a copy of the bio_vec array, so the memory bvs points to can be deallocated as soon as io_buffer_register_bvec() returns.
Quoted 28 lines
include/linux/io_uring/cmd.h +11
-0+ unsigned int nr_bvecs, unsigned int total_bytes, + u8 dir, unsigned int index, unsigned int issue_flags);int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags);#else @@ -199,6 +202,15 @@ static inline int io_buffer_register_request(struct io_uring_cmd *cmd, {return -EOPNOTSUPP ; } +static inline int io_buffer_register_bvec (struct io_uring_cmd *cmd, +struct bio_vec *bvs, +unsigned int nr_bvecs, +unsigned int total_bytes,u8 dir, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags)io_uring/rsrc.c +0
-0@@ -1020,6 +1020,33 @@ int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, }EXPORT_SYMBOL_GPL (io_buffer_register_request);
io_uring/rsrc.c int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, · line 1031
Quoted 10 lines
io_uring/rsrc.c +9
-0+int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct io_mapped_ubuf *imu; +struct bio_vec *bvec; +int i;
unsigned?
Other than that,
Quoted 24 lines
io_uring/rsrc.c +18
-1+ +io_ring_submit_lock (ctx, issue_flags); + imu =io_kernel_buffer_init (ctx, nr_bvecs, total_bytes, dir,NULL , +NULL , index); +if (IS_ERR (imu)) { +io_ring_submit_unlock (ctx, issue_flags); +return PTR_ERR (imu); + } + + bvec = imu->bvec; +for (i =0 ; i < nr_bvecs; i++) + bvec[i] = bvs[i]; + +io_ring_submit_unlock (ctx, issue_flags); +return 0 ; +} +EXPORT_SYMBOL_GPL (io_buffer_register_bvec); +int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags) { --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 23/25] io_uring/rsrc: add io_buffer_register_bvec()
Quoted 13 lines · Caleb Sander Mateos
Quoted 11 lines · Joanne Koong
Add
io_buffer_register_bvec()for registering a bvec array.This is a preparatory patch for fuse-over-io-uring zero-copy.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 12 ++++++++++++ io_uring/rsrc.c | 27 +++++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
Quoted 14 lines
include/linux/io_uring/cmd.h int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd, · line 109
Quoted 9 lines
include/linux/io_uring/cmd.h +1
-0@@ -106,6 +106,9 @@ int io_uring_cmd_is_kmbuf_ring(struct io_uring_cmd *ioucmd,int io_buffer_register_request (struct io_uring_cmd *cmd,struct request *rq,void (*release)(void *),unsigned int index,unsigned int issue_flags); +int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct bio_vec *bvs,Could take const struct
bio_vec*? Might also be helpful to document that this internally makes a copy of thebio_vecarray, so the memory bvs points to can be deallocated as soon asio_buffer_register_bvec()returns.
I'll add the "const" and your suggested documentation for v4.
Quoted 13 lines
Quoted 13 lines
include/linux/io_uring/cmd.h +2
-0+ unsigned int nr_bvecs, unsigned int total_bytes, + u8 dir, unsigned int index, unsigned int issue_flags);int io_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index,unsigned int issue_flags);io_uring/rsrc.c +0
-0@@ -1020,6 +1020,33 @@ int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, }EXPORT_SYMBOL_GPL (io_buffer_register_request);
Quoted 11 lines
io_uring/rsrc.c int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, · line 1031
Quoted 9 lines
io_uring/rsrc.c +9
-0+int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; +struct io_mapped_ubuf *imu; +struct bio_vec *bvec; +int i;unsigned?
Will do for v4.
Thanks,
Joanne
Quoted 3 lines
Other than that,
Reviewed-by Caleb Sander Mateos <[email protected]>
Joanne Koong
maintainerPATCH v3 24/25 fuse: add zero-copy over io-uring
In reply to Joanne KoongImplement zero-copy data transfer for fuse over io-uring, eliminating memory copies between kernel and userspace for read/write operations.
This is only allowed on privileged servers and requires the server to preregister the following:
a) a sparse buffer corresponding to the queue depth
b) a fixed buffer at index queue_depth (the tail of the buffers)
c) a kernel-managed buffer ring
The sparse buffer is where the client's pages reside. The fixed buffer at the tail is where the headers (struct fuse_uring_req_header) are placed. The kernel-managed buffer ring is where any non-zero-copied args reside (eg out headers).
Benchmarks with bs=1M showed approximately the following differences in throughput:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s)
buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)
The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 176 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 11 +++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 6 +- 5 files changed, 164 insertions(+), 37 deletions(-)
fs/fuse/dev.c +5 -2
@@ -1229,8 +1229,11 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) - err =fuse_copy_folios (cs, arg->size, zeroing); +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; +return fuse_copy_folios (cs, arg->size, zeroing); + }else err =fuse_copy_one (cs, arg->value, arg->size); }
fs/fuse/dev_uring.c +143 -33
358 lines not on this page · raw message
fs/fuse/dev_uring_i.h +11 -0
@@ -57,6 +57,9 @@ struct fuse_ring_ent {*/ unsigned int ringbuf_buf_id;unsigned int fixed_buf_id; + +/* True if the request's pages are being zero-copied */ +bool zero_copied; }; };
@@ -123,6 +126,14 @@ struct fuse_ring_queue {/* synchronized by the queue lock */ struct io_buffer_list *bufring; + +/* +* True if zero copy should be used for payloads. This is only enabled +* on privileged servers. Kernel-managed ring buffers must be enabled +* in order to use zero copy. +*/ + bool use_zero_copy : 1; +unsigned int zero_copy_depth; };/**
fs/fuse/fuse_dev_i.h +1 -0
@@ -39,6 +39,7 @@ struct fuse_copy_state { bool is_uring:1;/* if set, use kaddr; otherwise use pg */ bool is_kaddr:1; + bool skip_folio_copy:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;
include/uapi/linux/fuse.h +4 -2
@@ -242,7 +242,7 @@* - add FUSE_NOTIFY_PRUNE * * 7.46 -* - add fuse_uring_cmd_req init flags +* - add fuse_uring_cmd_req init flags and queue_depth */ #ifndef _LINUX_FUSE_H
@@ -1299,6 +1299,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req init flags */ #define FUSE_URING_BUF_RING (1 << 0) +#define FUSE_URING_ZERO_COPY (1 << 1)/** * In the 80B command area of the SQE.
@@ -1315,10 +1316,11 @@ struct fuse_uring_cmd_req {union {struct {uint16_t flags; +uint16_t queue_depth; } init; }; - uint8_t padding[4]; + uint8_t padding[2]; };#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
Re: [PATCH v3 24/25] fuse: add zero-copy over io-uring
Quoted 74 lines · Joanne Koong
Implement zero-copy data transfer for fuse over io-uring, eliminating memory copies between kernel and userspace for read/write operations.
This is only allowed on privileged servers and requires the server to preregister the following:
a) a sparse buffer corresponding to the queue depth
b) a fixed buffer at indexqueue_depth(the tail of the buffers)
c) a kernel-managed buffer ringThe sparse buffer is where the client's pages reside. The fixed buffer at the tail is where the headers (struct
fuse_uring_req_header) are placed. The kernel-managed buffer ring is where any non-zero-copied args reside (eg out headers).Benchmarks with bs=1M showed approximately the following differences in throughput:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s)
buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)The benchmark was run using fio on the
passthrough_hpserver:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 176 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 11 +++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 6 +- 5 files changed, 164 insertions(+), 37 deletions(-)fs/fuse/dev.c +5
-2@@ -1229,8 +1229,11 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) - err =fuse_copy_folios (cs, arg->size, zeroing); +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; +return fuse_copy_folios (cs, arg->size, zeroing); + }else err =fuse_copy_one (cs, arg->value, arg->size); }fs/fuse/dev_uring.c +7
-1@@ -89,8 +89,14 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) } } +static bool can_zero_copy_req (struct fuse_ring_ent *ent,struct fuse_req *req) +{ +return ent->queue->use_zero_copy && + (req->args->in_pages || req->args->out_pages); +} +static void fuse_uring_req_end (struct fuse_ring_ent *ent,struct fuse_req *req, -int error) +int error,unsigned issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;
fs/fuse/dev_uring.c static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, · line 119
Quoted 5 lines
fs/fuse/dev_uring.c +2
-0@@ -109,6 +115,12 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,spin_unlock (&queue->lock); +if (ent->zero_copied) { + WARN_ON_ONCE(io_buffer_unregister(ent->cmd, ent->fixed_buf_id,
io_buffer_unregister() can fail if the registered buffer index has been unregistered or updated with a userspace buffer since the call to io_buffer_register_bvec(). So this WARN_ON_ONCE() can be triggered from userspace. I think it would be preferable to ignore the error or report it to the fuse server.
Best,
Caleb
Quoted 437 lines
fs/fuse/dev_uring.c +134
-32+ issue_flags)); + ent->zero_copied = false; + } +if (error) req->out.h.error = error;@@ -282,6 +294,7 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)static int fuse_uring_buf_ring_setup (struct io_uring_cmd *cmd,struct fuse_ring_queue *queue, +bool zero_copy,unsigned int issue_flags) {int err;@@ -291,22 +304,39 @@ static int fuse_uring_buf_ring_setup(struct io_uring_cmd *cmd,if (err)return err; + err = -EINVAL ; +if (!io_uring_cmd_is_kmbuf_ring (cmd,FUSE_URING_RINGBUF_GROUP , - issue_flags)) { -io_uring_cmd_buf_ring_unpin (cmd, -FUSE_URING_RINGBUF_GROUP , - issue_flags); -return -EINVAL ; + issue_flags)) + goto error; + +if (zero_copy) { +const struct fuse_uring_cmd_req *cmd_req = +io_uring_sqe_cmd (cmd->sqe); + +if (!capable (CAP_SYS_ADMIN )) + goto error; + + queue->use_zero_copy = true; + queue->zero_copy_depth =READ_ONCE (cmd_req->init.queue_depth); +if (!queue->zero_copy_depth) + goto error; } queue->use_bufring = true;return 0 ; + +error: +io_uring_cmd_buf_ring_unpin (cmd,FUSE_URING_RINGBUF_GROUP , + issue_flags); +return err; }static struct fuse_ring_queue *fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, -int qid,bool use_bufring,unsigned int issue_flags) +int qid,bool use_bufring,bool zero_copy, +unsigned int issue_flags) {struct fuse_conn *fc = ring->fc;struct fuse_ring_queue *queue;@@ -338,12 +368,13 @@ fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring,fuse_pqueue_init (&queue->fpq);if (use_bufring) { - err =fuse_uring_buf_ring_setup (cmd, queue, issue_flags); -if (err) { -kfree (pq); -kfree (queue); -return ERR_PTR (err); - } + err =fuse_uring_buf_ring_setup (cmd, queue, zero_copy, + issue_flags); +if (err) + goto cleanup; + }else if (zero_copy) { + err = -EINVAL ; + goto cleanup; }spin_lock (&fc->lock);@@ -361,6 +392,11 @@ fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring,spin_unlock (&fc->lock);return queue; + +cleanup: +kfree (pq); +kfree (queue); +return ERR_PTR (err); }static void fuse_uring_stop_fuse_req_end (struct fuse_req *req)@@ -768,6 +804,7 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, cs->is_kaddr = true; cs->len = ent->payload_kvec.iov_len; cs->kaddr = ent->payload_kvec.iov_base; + cs->skip_folio_copy =can_zero_copy_req (ent, req); } cs->is_uring = true;@@ -800,11 +837,53 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,return err; } +static int fuse_uring_set_up_zero_copy (struct fuse_ring_ent *ent, +struct fuse_req *req, +unsigned issue_flags) +{ +struct fuse_args_pages *ap; +size_t total_bytes =0 ; +struct bio_vec *bvs; +int err, ddir, i; + +/* out_pages indicates a read, in_pages indicates a write */ + ddir = req->args->out_pages ?ITER_DEST :ITER_SOURCE ; + + ap =container_of (req->args,typeof (*ap), args); + +/* +* We can avoid having to allocate the bvs array when folios and +* descriptors are internally represented by bvs in fuse +*/ + bvs =kcalloc (ap->num_folios,sizeof (*bvs),GFP_KERNEL_ACCOUNT ); +if (!bvs) +return -ENOMEM ; + +for (i =0 ; i < ap->num_folios; i++) { + total_bytes += ap->descs[i].length; + bvs[i].bv_page =folio_page (ap->folios[i],0 ); + bvs[i].bv_offset = ap->descs[i].offset; + bvs[i].bv_len = ap->descs[i].length; + } + + err =io_buffer_register_bvec (ent->cmd, bvs, ap->num_folios, + total_bytes, ddir, ent->fixed_buf_id, + issue_flags); +kfree (bvs); +if (err) +return err; + + ent->zero_copied = true; + +return 0 ; +} +/* * Copy data from the req to the ring buffer */ static int fuse_uring_args_to_ring (struct fuse_ring *ring,struct fuse_req *req, -struct fuse_ring_ent *ent) +struct fuse_ring_ent *ent, +unsigned int issue_flags) {struct fuse_copy_state cs;struct fuse_args *args = req->args;@@ -837,6 +916,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, num_args--; } +if (can_zero_copy_req (ent, req)) { + err =fuse_uring_set_up_zero_copy (ent, req, issue_flags); +if (err) +return err; + }/* copy the payload */ err =fuse_copy_args (&cs, num_args, args->in_pages, (struct fuse_arg *)in_args,0 );@@ -847,12 +931,17 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; +if (cs.skip_folio_copy && args->in_pages) + ent_in_out.payload_sz += + args->in_args[args->in_numargs -1 ].size; +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , &ent_in_out,sizeof (ent_in_out)); }static int fuse_uring_copy_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;@@ -870,7 +959,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,return err;/* copy the request */ - err =fuse_uring_args_to_ring (ring, req, ent); + err =fuse_uring_args_to_ring (ring, req, ent, issue_flags);if (unlikely (err)) {pr_info_ratelimited ("Copy to ring failed: %d\n" , err);return err;@@ -881,11 +970,20 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } -static bool fuse_uring_req_has_payload (struct fuse_req *req) +static bool fuse_uring_req_has_copyable_payload (struct fuse_ring_ent *ent, +struct fuse_req *req) {struct fuse_args *args = req->args; -return args->in_numargs >1 || args->out_numargs; +if (!can_zero_copy_req (ent, req)) +return args->in_numargs >1 || args->out_numargs; + +if ((args->in_numargs >1 ) && (!args->in_pages || args->in_numargs >2 )) +return true; +if (args->out_numargs && (!args->out_pages || args->out_numargs >1 )) +return true; + +return false; } static int fuse_uring_select_buffer(struct fuse_ring_ent *ent,@@ -946,7 +1044,7 @@ static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent, ent->headers_iter.data_source = false; buffer_selected = ent->payload_kvec.iov_base !=0 ; - has_payload =fuse_uring_req_has_payload (req); + has_payload =fuse_uring_req_has_copyable_payload (ent, req);if (has_payload && !buffer_selected)return fuse_uring_select_buffer (ent, issue_flags);@@ -972,22 +1070,23 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent, ent->headers_iter.data_source = false;/* no payload to copy, can skip selecting a buffer */ -if (!fuse_uring_req_has_payload (req)) +if (!fuse_uring_req_has_copyable_payload (ent, req))return 0 ;return fuse_uring_select_buffer (ent, issue_flags); }static int fuse_uring_prepare_send (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {int err; - err =fuse_uring_copy_to_ring (ent, req); + err =fuse_uring_copy_to_ring (ent, req, issue_flags);if (!err)set_bit (FR_SENT , &req->flags);else -fuse_uring_req_end (ent, req, err); +fuse_uring_req_end (ent, req, err, issue_flags);return err; }@@ -1092,7 +1191,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, err =fuse_uring_copy_from_ring (ring, req, ent); out: -fuse_uring_req_end (ent, req, err); +fuse_uring_req_end (ent, req, err, issue_flags); }/* @@ -1115,7 +1214,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_prepare_send (ent, req); + err =fuse_uring_prepare_send (ent, req, issue_flags);if (err) goto retry; }@@ -1155,11 +1254,15 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,static void fuse_uring_headers_cleanup (struct fuse_ring_ent *ent,unsigned int issue_flags) { +u16 headers_index =FUSE_URING_FIXED_HEADERS_OFFSET ; +if (!ent->queue->use_bufring)return ; -WARN_ON_ONCE (io_uring_cmd_fixed_index_put (ent->cmd, -FUSE_URING_FIXED_HEADERS_OFFSET , +if (ent->queue->use_zero_copy) + headers_index += ent->queue->zero_copy_depth; + +WARN_ON_ONCE (io_uring_cmd_fixed_index_put (ent->cmd, headers_index, issue_flags)); }@@ -1167,6 +1270,7 @@ static int fuse_uring_headers_prep(struct fuse_ring_ent *ent, unsigned int dir, unsigned int issue_flags) {size_t header_size =sizeof (struct fuse_uring_req_header ); +u16 headers_index =FUSE_URING_FIXED_HEADERS_OFFSET ;struct io_uring_cmd *cmd = ent->cmd;unsigned int offset;int err;@@ -1176,11 +1280,15 @@ static int fuse_uring_headers_prep(struct fuse_ring_ent *ent, unsigned int dir, offset = ent->fixed_buf_id * header_size; - err =io_uring_cmd_fixed_index_get (cmd,FUSE_URING_FIXED_HEADERS_OFFSET , - offset, header_size, dir, +if (ent->queue->use_zero_copy) + headers_index += ent->queue->zero_copy_depth; + + err =io_uring_cmd_fixed_index_get (cmd, headers_index, offset, + header_size, dir, &ent->headers_iter, issue_flags);WARN_ON_ONCE (err); +return err; }@@ -1251,7 +1359,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, err =fuse_uring_headers_prep (ent,ITER_SOURCE , issue_flags);if (err) -fuse_uring_req_end (ent, req, err); +fuse_uring_req_end (ent, req, err, issue_flags);else fuse_uring_commit (ent, req, issue_flags);@@ -1413,6 +1521,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,const struct fuse_uring_cmd_req *cmd_req =io_uring_sqe_cmd (cmd->sqe);unsigned int init_flags =READ_ONCE (cmd_req->init.flags);bool use_bufring = init_flags &FUSE_URING_BUF_RING ; +bool zero_copy = init_flags &FUSE_URING_ZERO_COPY ;struct fuse_ring *ring =smp_load_acquire (&fc->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;@@ -1434,11 +1543,12 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, queue = ring->queues[qid];if (!queue) { queue =fuse_uring_create_queue (cmd, ring, qid, use_bufring, - issue_flags); + zero_copy, issue_flags);if (IS_ERR (queue))return PTR_ERR (queue); }else { -if (queue->use_bufring != use_bufring) +if ((queue->use_bufring != use_bufring) || + (queue->use_zero_copy != zero_copy))return -EINVAL ; }@@ -1545,7 +1655,7 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)if (fuse_uring_headers_prep (ent,ITER_DEST , issue_flags))return ; -if (fuse_uring_prepare_send (ent, ent->fuse_req)) +if (fuse_uring_prepare_send (ent, ent->fuse_req, issue_flags)) send =fuse_uring_get_next_fuse_req (ent, queue, issue_flags);fuse_uring_headers_cleanup (ent, issue_flags);if (!send)fs/fuse/dev_uring_i.h +11
-0@@ -57,6 +57,9 @@ struct fuse_ring_ent {*/ unsigned int ringbuf_buf_id;unsigned int fixed_buf_id; + +/* True if the request's pages are being zero-copied */ +bool zero_copied; }; };@@ -123,6 +126,14 @@ struct fuse_ring_queue {/* synchronized by the queue lock */ struct io_buffer_list *bufring; + +/* +* True if zero copy should be used for payloads. This is only enabled +* on privileged servers. Kernel-managed ring buffers must be enabled +* in order to use zero copy. +*/ + bool use_zero_copy : 1; +unsigned int zero_copy_depth; };/** fs/fuse/fuse_dev_i.h +1
-0@@ -39,6 +39,7 @@ struct fuse_copy_state { bool is_uring:1;/* if set, use kaddr; otherwise use pg */ bool is_kaddr:1; + bool skip_folio_copy:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;include/uapi/linux/fuse.h +4
-3@@ -242,7 +242,7 @@* - add FUSE_NOTIFY_PRUNE * * 7.46 -* - add fuse_uring_cmd_req init flags +* - add fuse_uring_cmd_req init flags and queue_depth */ #ifndef _LINUX_FUSE_H@@ -1299,6 +1299,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req init flags */ #define FUSE_URING_BUF_RING (1 << 0) +#define FUSE_URING_ZERO_COPY (1 << 1)/** * In the 80B command area of the SQE. @@ -1315,10 +1316,11 @@ struct fuse_uring_cmd_req {union {struct {uint16_t flags; +uint16_t queue_depth; } init; }; -uint8_t padding[4 ]; + uint8_t padding[2]; };#endif /* _LINUX_FUSE_H */ --2.47.3
Joanne Koong
maintainerRe: [PATCH v3 24/25] fuse: add zero-copy over io-uring
Quoted 79 lines · Caleb Sander Mateos
Quoted 77 lines · Joanne Koong
Implement zero-copy data transfer for fuse over io-uring, eliminating memory copies between kernel and userspace for read/write operations.
This is only allowed on privileged servers and requires the server to preregister the following:
a) a sparse buffer corresponding to the queue depth
b) a fixed buffer at indexqueue_depth(the tail of the buffers)
c) a kernel-managed buffer ringThe sparse buffer is where the client's pages reside. The fixed buffer at the tail is where the headers (struct
fuse_uring_req_header) are placed. The kernel-managed buffer ring is where any non-zero-copied args reside (eg out headers).Benchmarks with bs=1M showed approximately the following differences in throughput:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s)
buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)The benchmark was run using fio on the
passthrough_hpserver:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 176 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 11 +++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 6 +- 5 files changed, 164 insertions(+), 37 deletions(-)fs/fuse/dev.c +5
-2@@ -1229,8 +1229,11 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) - err =fuse_copy_folios (cs, arg->size, zeroing); +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; +return fuse_copy_folios (cs, arg->size, zeroing); + }else err =fuse_copy_one (cs, arg->value, arg->size); }fs/fuse/dev_uring.c +7
-1@@ -89,8 +89,14 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) } } +static bool can_zero_copy_req (struct fuse_ring_ent *ent,struct fuse_req *req) +{ +return ent->queue->use_zero_copy && + (req->args->in_pages || req->args->out_pages); +} +static void fuse_uring_req_end (struct fuse_ring_ent *ent,struct fuse_req *req, -int error) +int error,unsigned issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;@@ -109,6 +115,12 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,spin_unlock (&queue->lock);
Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, · line 119
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+if (ent->zero_copied) { + WARN_ON_ONCE(io_buffer_unregister(ent->cmd, ent->fixed_buf_id,
io_buffer_unregister()can fail if the registered buffer index has been unregistered or updated with a userspace buffer since the call toio_buffer_register_bvec(). So thisWARN_ON_ONCE()can be triggered from userspace. I think it would be preferable to ignore the error or report it to the fuse server.
Sounds good, this was a remnant from when registered buffers had been pinned. I'll remove this WARN_ON in v4.
Thanks,
Joanne
Quoted 4 lines
Best,
Caleb
Joanne Koong
maintainerPATCH v3 25/25 docs: fuse: add io-uring bufring and zero-copy documentation
In reply to Joanne KoongAdd documentation for fuse over io-uring usage of kernel-managed bufrings and zero-copy.
--- .../filesystems/fuse/fuse-io-uring.rst | 59 ++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)
Documentation/filesystems/fuse/fuse-io-uring.rst +58 -1
@@ -95,5 +95,62 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | +Kernel-managed buffer rings +=========================== - +Kernel-managed buffer rings have two main advantages: + +* eliminates the overhead of pinning/unpinning user pages and translating + virtual addresses for every server-kernel interaction +* reduces buffer memory allocation requirements + +In order to use buffer rings, the server must preregister the following: + +* a fixed buffer at index 0. This is where the headers will reside +* a kernel-managed buffer ring. This is where the payload will reside + +At a high-level, this is how fuse uses buffer rings: + +* The server registers a kernel-managed buffer ring. In the kernel this + allocates the pages needed for the buffers and vmaps them. The server + obtains the virtual address for the buffers through an mmap call on the ring + fd. +* When there is a request from a client, fuse will select a buffer from the + ring if there is any payload that needs to be copied, copy over the payload + to the selected buffer, and copy over the headers to the fixed buffer at + index 0, at the buffer id that corresponds to the server (which the server + needs to specify through sqe->buf_index). +* The server obtains a cqe representing the request. The cqe flag will have + IORING_CQE_F_BUFFER set if a selected buffer was used for the payload. The + buffer id is stashed in cqe->flags (through IORING_CQE_BUFFER_SHIFT). The + server can directly access the payload by using that buffer id to calculate + the offset into the virtual address obtained for the buffers. +* The server processes the request and then sends a + FUSE_URING_CMD_COMMIT_AND_FETCH sqe with the reply. +* When the kernel handles the sqe, it will process the reply and if there is a + next request, it will reuse the same selected buffer for the request. If + there is no next request, it will recycle the buffer back to the ring. + +Zero-copy +========= + +Fuse io-uring zero-copy allows the server to directly read from / write to the +client's pages and bypass any intermediary buffer copies. This is only allowed +on privileged servers. + +In order to use zero-copy, the server must pregister the following: + +* a sparse buffer for every entry in the queue. This is where the client's + pages will reside +* a fixed buffer at index queue_depth (tailing the sparse buffer). + This is where the headers will reside +* a kernel-managed buffer ring. This is where any non-zero-copied payload (eg + out headers) will reside + +When the client issues a read/write, fuse stores the client's underlying pages +in the sparse buffer entry corresponding to the ent in the queue. The server +can then issue reads/writes on these pages through io_uring rw operations. +Please note that the server is not able to directly access these pages, it +must go through the io-uring interface to read/write to them. The pages are +unregistered once the server replies to the request. Non-zero-copyable +payload (if needed) is placed in a buffer from the kernel-managed buffer ring.
Signature · 1 line
2.47.3