fuse/io-uring: add kernel-managed buffer rings and zero-copy
Subsystems (7)
Joanne Koong
maintainercover fuse/io-uring: add kernel-managed buffer rings and zero-copy
This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.
On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.
Patches 6 and 7 add the capability to pin buffer rings and the fixed buffer table. While originally desired as an optimization, this is a necessity for fuse because the ent headers reside at a different index than the sqe's buf index, which would require having to track the refcount for the imported buffer in a gnarlier way. There are some cases where fuse needs to select buffers from the buffer ring in atomic contexts where the uring mutex is not held, and pinning the buffer ring allows the selection of buffers using the underlying buffer list pointer with synchronization from the fuse queue spinlock.
The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 21: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.
There was a previous submission for supporting registered buffers in fuse [1] but that was abandoned in favor of using kernel-managed buffer rings, which, once incremental buffer consumption is added in a later patchset, gives significant memory usage advantages in allowing the full buffer capacity to be utilized across multiple requests, as well as offers more flexibility for future additions. As well, it also makes the userspace side setup simpler. The relevant refactoring fuse patches from the previous submission are carried over into this one.
Benchmarks for zero-copy (patch 29) show approximately the following differences in throughput for bs=1M:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s) buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)
The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
This series is on top of commit 5d24321e4c15 ("io_uring: Introduce sockname...") in the io-uring tree, and on top of two locally patched fixups
[2] and [3].
Thanks,
Joanne
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://lore.kernel.org/linux-fsdevel/[email protected]/
[3] https://lore.kernel.org/linux-fsdevel/[email protected]/
Joanne Koong (30):
io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic
helpers
io_uring/kbuf: rename io_unregister_pbuf_ring() to
io_unregister_buf_ring()
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: add mmap support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/rsrc: add fixed buffer table pinning/unpinning io_uring/kbuf: add recycling for pinned kernel managed buffer rings
io_uring: add io_uring_cmd_import_fixed_index() io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: return buffer id in buffer selection io_uring/kbuf: export io_ring_buffer_select() io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
io_uring: add release callback for ring death
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: support buffer copying for kernel addresses
fuse: add io-uring kernel-managed buffer ring io_uring/rsrc: refactor
io_buffer_register_bvec()/io_buffer_unregister_bvec()
io_uring/rsrc: split io_buffer_register_request() logic io_uring/rsrc: Allow buffer release callback to be optional io_uring/rsrc: add io_buffer_register_bvec() io_uring/rsrc: export io_buffer_unregister
fuse: rename fuse_set_zero_arg0() to fuse_zero_in_arg0()
fuse: enforce op header for every payload reply
fuse: add zero-copy over io-uring
docs: fuse: add io-uring bufring and zero-copy documentation
Documentation/block/ublk.rst | 15 +- .../filesystems/fuse/fuse-io-uring.rst | 55 +- drivers/block/ublk_drv.c | 20 +- fs/fuse/dax.c | 2 +- fs/fuse/dev.c | 32 +- fs/fuse/dev_uring.c | 775 +++++++++++++++--- fs/fuse/dev_uring_i.h | 47 +- fs/fuse/dir.c | 13 +- fs/fuse/file.c | 11 +- fs/fuse/fuse_dev_i.h | 8 +- fs/fuse/fuse_i.h | 8 +- fs/fuse/readdir.c | 2 +- fs/fuse/xattr.c | 18 +- include/linux/io_uring.h | 9 + include/linux/io_uring/buf.h | 98 +++ include/linux/io_uring/cmd.h | 25 +- include/linux/io_uring_types.h | 21 +- include/uapi/linux/fuse.h | 15 +- include/uapi/linux/io_uring.h | 17 +- io_uring/io_uring.c | 15 + io_uring/kbuf.c | 337 ++++++-- io_uring/kbuf.h | 19 +- io_uring/memmap.c | 117 ++- io_uring/memmap.h | 4 + io_uring/register.c | 9 +- io_uring/rsrc.c | 188 ++++- io_uring/rsrc.h | 6 + io_uring/uring_cmd.c | 39 +- 28 files changed, 1632 insertions(+), 293 deletions(-) create mode 100644 include/linux/io_uring/buf.h
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 1/30 io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic helpers
Refactor the logic in io_register_pbuf_ring() into generic helpers:
- io_validate_buf_reg(): Validate user input and buffer registration parameters
- io_alloc_new_buffer_list(): Allocate and initialize a new buffer list for the given buffer group ID
- io_setup_pbuf_ring(): Sets up the physical buffer ring region and handles memory mapping for provided buffer rings
This is a preparatory change for upcoming kernel-managed buffer ring support which will need to reuse some of these helpers.
--- io_uring/kbuf.c | 123 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 82 insertions(+), 41 deletions(-)
io_uring/kbuf.c +82 -41
@@ -596,55 +596,71 @@ int io_manage_buffers_legacy(struct io_kiocb *req, unsigned int issue_flags)return IOU_COMPLETE ; } -int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +static int io_validate_buf_reg (struct io_uring_buf_reg *reg, +unsigned int permitted_flags) { -struct io_uring_buf_reg reg; -struct io_buffer_list *bl; -struct io_uring_region_desc rd; -struct io_uring_buf_ring *br; -unsigned long mmap_offset; -unsigned long ring_size; -int ret; - -lockdep_assert_held (&ctx->uring_lock); - -if (copy_from_user (®, arg,sizeof (reg))) -return -EFAULT ; -if (!mem_is_zero (reg.resv,sizeof (reg.resv))) +if (!mem_is_zero (reg->resv,sizeof (reg->resv)))return -EINVAL ; -if (reg.flags & ~(IOU_PBUF_RING_MMAP |IOU_PBUF_RING_INC )) +if (reg->flags & ~permitted_flags)return -EINVAL ; -if (!is_power_of_2 (reg.ring_entries)) +if (!is_power_of_2 (reg->ring_entries))return -EINVAL ;/* cannot disambiguate full vs empty due to head/tail size */ -if (reg.ring_entries >=65536 ) +if (reg->ring_entries >=65536 )return -EINVAL ; +return 0 ; +} - bl =io_buffer_get_list (ctx, reg.bgid); -if (bl) { +static int io_alloc_new_buffer_list (struct io_ring_ctx *ctx, +struct io_uring_buf_reg *reg, +struct io_buffer_list **bl) +{ +struct io_buffer_list *list; + + list =io_buffer_get_list (ctx, reg->bgid); +if (list) {/* if mapped buffer ring OR classic exists, don't allow */ -if (bl->flags &IOBL_BUF_RING || !list_empty (&bl->buf_list)) +if (list->flags &IOBL_BUF_RING || !list_empty (&list->buf_list))return -EEXIST ; -io_destroy_bl (ctx, bl); +io_destroy_bl (ctx, list); } - bl =kzalloc (sizeof (*bl),GFP_KERNEL_ACCOUNT ); -if (!bl) + list =kzalloc (sizeof (*list),GFP_KERNEL_ACCOUNT ); +if (!list)return -ENOMEM ; - mmap_offset = (unsigned long )reg.bgid <<IORING_OFF_PBUF_SHIFT ; - ring_size =flex_array_size (br, bufs, reg.ring_entries); + list->nr_entries = reg->ring_entries; + list->mask = reg->ring_entries -1 ; + list->flags =IOBL_BUF_RING ; + + *bl = list; + +return 0 ; +} + +static int io_setup_pbuf_ring (struct io_ring_ctx *ctx, +struct io_uring_buf_reg *reg, +struct io_buffer_list *bl) +{ +struct io_uring_region_desc rd; +unsigned long mmap_offset; +unsigned long ring_size; +int ret; + + mmap_offset = (unsigned long )reg->bgid <<IORING_OFF_PBUF_SHIFT ; + ring_size =flex_array_size (bl->buf_ring, bufs, reg->ring_entries);memset (&rd,0 ,sizeof (rd)); rd.size =PAGE_ALIGN (ring_size); -if (!(reg.flags &IOU_PBUF_RING_MMAP )) { - rd.user_addr = reg.ring_addr; +if (!(reg->flags &IOU_PBUF_RING_MMAP )) { + rd.user_addr = reg->ring_addr; rd.flags |=IORING_MEM_REGION_TYPE_USER ; } + ret =io_create_region (ctx, &bl->region, &rd, mmap_offset);if (ret) - goto fail; - br =io_region_get_ptr (&bl->region); +return ret; + bl->buf_ring =io_region_get_ptr (&bl->region);#ifdef SHM_COLOUR /*
@@ -656,25 +672,50 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)* should use IOU_PBUF_RING_MMAP instead, and liburing will handle * this transparently. */ -if (!(reg.flags &IOU_PBUF_RING_MMAP ) && - ((reg.ring_addr | (unsigned long )br) & (SHM_COLOUR -1 ))) { - ret = -EINVAL ; - goto fail; +if (!(reg->flags &IOU_PBUF_RING_MMAP ) && + ((reg->ring_addr | (unsigned long )bl->buf_ring) & + (SHM_COLOUR -1 ))) { +io_free_region (ctx->user, &bl->region); +return -EINVAL ; }#endif - bl->nr_entries = reg.ring_entries; - bl->mask = reg.ring_entries -1 ; - bl->flags |=IOBL_BUF_RING ; - bl->buf_ring = br; +return 0 ; +} + +int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +{ +unsigned int permitted_flags; +struct io_uring_buf_reg reg; +struct io_buffer_list *bl; +int ret; + +lockdep_assert_held (&ctx->uring_lock); + +if (copy_from_user (®, arg,sizeof (reg))) +return -EFAULT ; + + permitted_flags =IOU_PBUF_RING_MMAP |IOU_PBUF_RING_INC ; + ret =io_validate_buf_reg (®, permitted_flags); +if (ret) +return ret; + + ret =io_alloc_new_buffer_list (ctx, ®, &bl); +if (ret) +return ret; + + ret =io_setup_pbuf_ring (ctx, ®, bl); +if (ret) { +kfree (bl); +return ret; + } +if (reg.flags &IOU_PBUF_RING_INC ) bl->flags |=IOBL_INC ; +io_buffer_add_list (ctx, bl, reg.bgid); +return 0 ; -fail: -io_free_region (ctx->user, &bl->region); -kfree (bl); -return ret; }int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg)
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 2/30 io_uring/kbuf: rename io_unregister_pbuf_ring() to io_unregister_buf_ring()
In reply to Joanne KoongUse the more generic name io_unregister_buf_ring() as this function will be used for unregistering both provided buffer rings and kernel-managed buffer rings.
This is a preparatory change for upcoming kernel-managed buffer ring support.
--- io_uring/kbuf.c | 2 +- io_uring/kbuf.h | 2 +- io_uring/register.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
io_uring/kbuf.c +1 -1
@@ -718,7 +718,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } -int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) +int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg) {struct io_uring_buf_reg reg;struct io_buffer_list *bl;
io_uring/kbuf.h +1 -1
@@ -74,7 +74,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqeint io_manage_buffers_legacy (struct io_kiocb *req,unsigned int issue_flags);int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); -int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); +int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_register_pbuf_status (struct io_ring_ctx *ctx,void __user *arg);bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags);
io_uring/register.c +1 -1
@@ -750,7 +750,7 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, ret = -EINVAL ;if (!arg || nr_args !=1 )break ; - ret =io_unregister_pbuf_ring (ctx, arg); + ret =io_unregister_buf_ring (ctx, arg);break ;case IORING_REGISTER_SYNC_CANCEL : ret = -EINVAL ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 3/30 io_uring/kbuf: add support for kernel-managed buffer rings
In reply to Joanne KoongAdd support for kernel-managed buffer rings (kmbuf rings), which allow the kernel to allocate and manage the backing buffers for a buffer ring, rather than requiring the application to provide and manage them.
This introduces two new registration opcodes:
- IORING_REGISTER_KMBUF_RING: Register a kernel-managed buffer ring
- IORING_UNREGISTER_KMBUF_RING: Unregister a kernel-managed buffer ring
The existing io_uring_buf_reg structure is extended with a union to support both application-provided buffer rings (pbuf) and kernel-managed buffer rings (kmbuf):
- For pbuf rings: ring_addr specifies the user-provided ring address
- For kmbuf rings: buf_size specifies the size of each buffer. buf_size must be non-zero and page-aligned.
The implementation follows the same pattern as pbuf ring registration, reusing the validation and buffer list allocation helpers introduced in earlier refactoring. The IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.
--- include/uapi/linux/io_uring.h | 15 ++++- io_uring/kbuf.c | 76 +++++++++++++++++++++++ io_uring/kbuf.h | 7 ++- io_uring/memmap.c | 112 ++++++++++++++++++++++++++++++++++ io_uring/memmap.h | 4 ++ io_uring/register.c | 7 +++ 6 files changed, 217 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +13 -2
@@ -700,6 +700,10 @@ enum io_uring_register_op {/* auxiliary zcrx configuration, see enum zcrx_ctrl_op */ IORING_REGISTER_ZCRX_CTRL =36 , +/* register/unregister kernel-managed ring buffer group */ +IORING_REGISTER_KMBUF_RING =37 , +IORING_UNREGISTER_KMBUF_RING =38 , +/* this goes last */ IORING_REGISTER_LAST,
@@ -869,9 +873,16 @@ enum io_uring_register_pbuf_ring_flags { IOU_PBUF_RING_INC = 2, }; -/* argument for IORING_(UN)REGISTER_PBUF_RING */ +/* argument for IORING_(UN)REGISTER_PBUF_RING and +* IORING_(UN)REGISTER_KMBUF_RING +*/ struct io_uring_buf_reg { -__u64 ring_addr; +union { +/* used for pbuf rings */ +__u64 ring_addr; +/* used for kmbuf rings */ +__u32 buf_size; + };__u32 ring_entries;__u16 bgid;__u16 flags;
io_uring/kbuf.c +76 -0
79 lines not on this page · raw message
io_uring/kbuf.h +5 -2
@@ -7,9 +7,11 @@enum {/* ring mapped provided buffers */ -IOBL_BUF_RING =1 , +IOBL_BUF_RING =1 ,/* buffers are consumed incrementally rather than always fully */ -IOBL_INC =2 , +IOBL_INC =2 , +/* buffers are kernel managed */ +IOBL_KERNEL_MANAGED =4 , };struct io_buffer_list {
@@ -74,6 +76,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqeint io_manage_buffers_legacy (struct io_kiocb *req,unsigned int issue_flags);int io_register_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg); +int io_register_kmbuf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_unregister_buf_ring (struct io_ring_ctx *ctx,void __user *arg);int io_register_pbuf_status (struct io_ring_ctx *ctx,void __user *arg);
io_uring/memmap.c +112 -0
142 lines not on this page · raw message
io_uring/memmap.h +4 -0
@@ -21,6 +21,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr,struct io_uring_region_desc *reg, unsigned long mmap_offset); +int io_create_region_multi_buf (struct io_ring_ctx *ctx, +struct io_mapped_region *mr, +unsigned int nr_bufs,unsigned int buf_size); +static inline void *io_region_get_ptr (struct io_mapped_region *mr) {return mr->ptr;
io_uring/register.c +7 -0
@@ -746,7 +746,14 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_pbuf_ring (ctx, arg);break ; +case IORING_REGISTER_KMBUF_RING : + ret = -EINVAL ; +if (!arg || nr_args !=1 ) +break ; + ret =io_register_kmbuf_ring (ctx, arg); +break ;case IORING_UNREGISTER_PBUF_RING : +case IORING_UNREGISTER_KMBUF_RING : ret = -EINVAL ;if (!arg || nr_args !=1 )break ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 4/30 io_uring/kbuf: add mmap support for kernel-managed buffer rings
In reply to Joanne KoongAdd support for mmapping kernel-managed buffer rings (kmbuf) to userspace, allowing applications to access the kernel-allocated buffers.
Similar to application-provided buffer rings (pbuf), kmbuf rings use the buffer group ID encoded in the mmap offset to identify which buffer ring to map. The implementation follows the same pattern as pbuf rings.
New mmap offset constants are introduced:
- IORING_OFF_KMBUF_RING (0x88000000): Base offset for kmbuf mappings
- IORING_OFF_KMBUF_SHIFT (16): Shift value to encode buffer group ID
The mmap offset is calculated during registration, encoding the bgid shifted by IORING_OFF_KMBUF_SHIFT. The io_buf_get_region() helper retrieves the appropriate region.
This allows userspace to mmap the kernel-allocated buffer region and access the buffers directly.
--- include/uapi/linux/io_uring.h | 2 ++ io_uring/kbuf.c | 11 +++++++++-- io_uring/kbuf.h | 5 +++-- io_uring/memmap.c | 5 ++++- 4 files changed, 18 insertions(+), 5 deletions(-)
include/uapi/linux/io_uring.h +2 -0
@@ -533,6 +533,8 @@ struct io_uring_cqe {#define IORING_OFF_SQES 0x10000000ULL#define IORING_OFF_PBUF_RING 0x80000000ULL#define IORING_OFF_PBUF_SHIFT 16 +#define IORING_OFF_KMBUF_RING 0x88000000ULL +#define IORING_OFF_KMBUF_SHIFT 16#define IORING_OFF_MMAP_MASK 0xf8000000ULL/*
io_uring/kbuf.c +9 -2
@@ -766,16 +766,23 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } -struct io_mapped_region *io_pbuf_get_region (struct io_ring_ctx *ctx, -unsigned int bgid) +struct io_mapped_region *io_buf_get_region (struct io_ring_ctx *ctx, +unsigned int bgid, +bool kernel_managed) {struct io_buffer_list *bl; +bool is_kernel_managed;lockdep_assert_held (&ctx->mmap_lock); bl =xa_load (&ctx->io_bl_xa, bgid);if (!bl || !(bl->flags &IOBL_BUF_RING ))return NULL ; + + is_kernel_managed = !!(bl->flags &IOBL_KERNEL_MANAGED ); +if (is_kernel_managed != kernel_managed) +return NULL ; +return &bl->region; }
io_uring/kbuf.h +3 -2
@@ -88,8 +88,9 @@ unsigned int __io_put_kbufs(struct io_kiocb *req, struct io_buffer_list *bl,bool io_kbuf_commit (struct io_kiocb *req,struct io_buffer_list *bl,int len,int nr); -struct io_mapped_region *io_pbuf_get_region (struct io_ring_ctx *ctx, -unsigned int bgid); +struct io_mapped_region *io_buf_get_region (struct io_ring_ctx *ctx, +unsigned int bgid, +bool kernel_managed);static inline bool io_kbuf_recycle_ring (struct io_kiocb *req,struct io_buffer_list *bl)
io_uring/memmap.c +4 -1
@@ -357,7 +357,10 @@ static struct io_mapped_region *io_mmap_get_region(struct io_ring_ctx *ctx,return &ctx->sq_region;case IORING_OFF_PBUF_RING : id = (offset & ~IORING_OFF_MMAP_MASK ) >>IORING_OFF_PBUF_SHIFT ; -return io_pbuf_get_region (ctx, id); +return io_buf_get_region (ctx, id, false); +case IORING_OFF_KMBUF_RING : + id = (offset & ~IORING_OFF_MMAP_MASK ) >>IORING_OFF_KMBUF_SHIFT ; +return io_buf_get_region (ctx, id, true);case IORING_MAP_OFF_PARAM_REGION :return &ctx->param_region;case IORING_MAP_OFF_ZCRX_REGION :
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 5/30 io_uring/kbuf: support kernel-managed buffer rings in buffer selection
In reply to Joanne KoongAllow kernel-managed buffers to be selected. This requires modifying the io_br_sel struct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.
Auto-commit any selected kernel-managed buffer.
--- include/linux/io_uring_types.h | 8 ++++---- io_uring/kbuf.c | 16 ++++++++++++---- 2 files changed, 16 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +4 -4
@@ -93,13 +93,13 @@ struct io_mapped_region {*/ struct io_br_sel {struct io_buffer_list *buf_list; -/* -* Some selection parts return the user address, others return an error. -*/ union { +/* for classic/ring provided buffers */ void __user *addr; -ssize_t val; +/* for kernel-managed buffers */ +void *kaddr; }; +ssize_t val; };
io_uring/kbuf.c +12 -4
@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,return 1 ; } -static bool io_should_commit (struct io_kiocb *req,unsigned int issue_flags) +static bool io_should_commit (struct io_kiocb *req,struct io_buffer_list *bl, +unsigned int issue_flags) {/* * If we came in unlocked, we have no choice but to consume the
@@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)if (issue_flags &IO_URING_F_UNLOCKED )return true; -/* uring_cmd commits kbuf upfront, no need to auto-commit */ +/* kernel-managed buffers are auto-committed */ +if (bl->flags &IOBL_KERNEL_MANAGED ) +return true; + +/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */ if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD )return true;return false;
@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index = buf->bid; sel.buf_list = bl; - sel.addr =u64_to_user_ptr (buf->addr); +if (bl->flags &IOBL_KERNEL_MANAGED ) + sel.kaddr = (void *)buf->addr; +else + sel.addr =u64_to_user_ptr (buf->addr); -if (io_should_commit (req, issue_flags)) { +if (io_should_commit (req, bl, issue_flags)) {io_kbuf_commit (req, sel.buf_list, *len,1 ); sel.buf_list =NULL ; }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 6/30 io_uring/kbuf: add buffer ring pinning/unpinning
In reply to Joanne KoongAdd kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY. Pinning an already-pinned bufring is acceptable and returns 0.
The API accepts a "struct io_ring_ctx *ctx" rather than a cmd pointer, as the buffer ring may need to be unpinned in contexts where a cmd is not readily available.
This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
--- include/linux/io_uring/buf.h | 28 +++++++++++++++++++++++ io_uring/kbuf.c | 43 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 5 +++++ 3 files changed, 76 insertions(+) create mode 100644 include/linux/io_uring/buf.h
include/linux/io_uring/buf.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H + +#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +int io_uring_buf_ring_pin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags); +#else +static inline int io_uring_buf_ring_pin (struct io_ring_ctx *ctx, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl); +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +#endif /* CONFIG_IO_URING */ + +#endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +43 -0
61 lines not on this page · raw message
io_uring/kbuf.h +5 -0
@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list {
Signature · 1 line
2.47.3
Re: [PATCH v1 06/30] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 77 lines · Joanne Koong
Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return
-EBUSY. Pinning an already-pinned bufring is acceptable and returns 0.The API accepts a "struct
io_ring_ctx*ctx" rather than a cmd pointer, as the buffer ring may need to be unpinned in contexts where a cmd is not readily available.This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 28 +++++++++++++++++++++++ io_uring/kbuf.c | 43 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 5 +++++ 3 files changed, 76 insertions(+) create mode 100644 include/linux/io_uring/buf.hinclude/linux/io_uring/buf.h new file +28
-0@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_IO_URING_BUF_H +#define _LINUX_IO_URING_BUF_H + +#include <linux/io_uring_types.h> + +#if defined(CONFIG_IO_URING ) +int io_uring_buf_ring_pin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl); +int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags); +#else +static inline int io_uring_buf_ring_pin (struct io_ring_ctx *ctx, +unsigned buf_group, +unsigned issue_flags, +struct io_buffer_list **bl); +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx, +unsigned buf_group, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +#endif /* CONFIG_IO_URING */ + +#endif /* _LINUX_IO_URING_BUF_H */ io_uring/kbuf.c +1
-0@@ -9,6 +9,7 @@#include <linux/poll.h> #include <linux/vmalloc.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <uapi/linux/io_uring.h> @@ -237,6 +238,46 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,return sel; }
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 250
Quoted 10 lines
io_uring/kbuf.c +10
-0+int io_uring_buf_ring_pin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) && (buffer_list->flags &IOBL_BUF_RING )) {
Since there's no reference-counting of pins, I think it might make more sense to fail io_uring_buf_ring_pin() if the buffer ring is already pinned. Otherwise, the buffer ring will be unpinned in the first call to io_uring_buf_ring_unpin(), when it might still be in use by another caller of io_uring_buf_ring_pin().
Best,
Caleb
Quoted 61 lines
io_uring/kbuf.c +32
-0+ buffer_list->flags |=IOBL_PINNED ; + ret =0 ; + *bl = buffer_list; + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +EXPORT_SYMBOL_GPL (io_uring_buf_ring_pin); + +int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags) +{ +struct io_buffer_list *bl; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) && (bl->flags &IOBL_BUF_RING ) && + (bl->flags &IOBL_PINNED )) { + bl->flags &= ~IOBL_PINNED ; + ret =0 ; + } + +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +EXPORT_SYMBOL_GPL (io_uring_buf_ring_unpin); +/* cap it at a reasonable 256, will be one page even for 4K */ #define PEEK_MAX_IMPORT 256@@ -743,6 +784,8 @@ int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg)return -ENOENT ;if (!(bl->flags &IOBL_BUF_RING ))return -EINVAL ; +if (bl->flags &IOBL_PINNED ) +return -EBUSY ; scoped_guard(mutex, &ctx->mmap_lock)xa_erase (&ctx->io_bl_xa, bl->bgid);io_uring/kbuf.h +5
-1@@ -12,6 +12,11 @@ enum {IOBL_INC =2 ,/* buffers are kernel managed */ IOBL_KERNEL_MANAGED =4 , +/* +* buffer ring is pinned and cannot be unregistered by userspace until +* it has been unpinned +*/ + IOBL_PINNED = 8, }; struct io_buffer_list { --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 06/30] io_uring/kbuf: add buffer ring pinning/unpinning
Quoted 45 lines · Caleb Sander Mateos
Quoted 43 lines · Joanne Koong
Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.
This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return
-EBUSY. Pinning an already-pinned bufring is acceptable and returns 0.The API accepts a "struct
io_ring_ctx*ctx" rather than a cmd pointer, as the buffer ring may need to be unpinned in contexts where a cmd is not readily available.This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 28 +++++++++++++++++++++++ io_uring/kbuf.c | 43 ++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 5 +++++ 3 files changed, 76 insertions(+) create mode 100644 include/linux/io_uring/buf.hio_uring/kbuf.c +1
-0@@ -9,6 +9,7 @@#include <linux/poll.h> #include <linux/vmalloc.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <uapi/linux/io_uring.h> @@ -237,6 +238,46 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,return sel; }
Quoted 16 lines
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 250
Quoted 10 lines
io_uring/kbuf.c +10
-0+int io_uring_buf_ring_pin (struct io_ring_ctx *ctx,unsigned buf_group, +unsigned issue_flags,struct io_buffer_list **bl) +{ +struct io_buffer_list *buffer_list; +int ret = -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + + buffer_list =io_buffer_get_list (ctx, buf_group); +if (likely (buffer_list) && (buffer_list->flags &IOBL_BUF_RING )) {Since there's no reference-counting of pins, I think it might make more sense to fail
io_uring_buf_ring_pin()if the buffer ring is already pinned. Otherwise, the buffer ring will be unpinned in the first call toio_uring_buf_ring_unpin(), when it might still be in use by another caller ofio_uring_buf_ring_pin().
That makes sense, I'll change this to return -EALREADY then if it's already pinned.
Thanks,
Joanne
Quoted 4 lines
Best,
Caleb
Joanne Koong
maintainerPATCH 7/30 io_uring/rsrc: add fixed buffer table pinning/unpinning
In reply to Joanne KoongAdd kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the io_kiocb so that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.
This is a preparatory patch for fuse io-uring's usage of fixed buffers.
--- include/linux/io_uring/buf.h | 13 +++++++++++ include/linux/io_uring_types.h | 9 ++++++++ io_uring/rsrc.c | 42 ++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+)
include/linux/io_uring/buf.h +13 -0
@@ -9,6 +9,9 @@ int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group,unsigned issue_flags); + +int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags); +int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -23,6 +26,16 @@ static inline int io_uring_buf_ring_unpin(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } +static inline int io_uring_buf_table_pin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_table_unpin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
include/linux/io_uring_types.h +9 -0
@@ -57,8 +57,17 @@ struct io_wq_work {int cancel_seq; }; +/* +* struct io_rsrc_data flag values: +* +* IO_RSRC_DATA_PINNED: data is pinned and cannot be unregistered by userspace +* until it has been unpinned. Currently this is only possible on buffer tables. +*/ +#define IO_RSRC_DATA_PINNED BIT(0) +struct io_rsrc_data {unsigned int nr; +u8 flags;struct io_rsrc_node **nodes; };
io_uring/rsrc.c +42 -0
63 lines not on this page · raw message
Signature · 1 line
2.47.3
Re: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 11 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.
Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing io_rsrc_node refs field.
Quoted 76 lines
This is a preparatory patch for fuse io-uring's usage of fixed buffers.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 13 +++++++++++ include/linux/io_uring_types.h | 9 ++++++++ io_uring/rsrc.c | 42 ++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+)include/linux/io_uring/buf.h +13
-0@@ -9,6 +9,9 @@ int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group,unsigned issue_flags); + +int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags); +int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,@@ -23,6 +26,16 @@ static inline int io_uring_buf_ring_unpin(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } +static inline int io_uring_buf_table_pin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_table_unpin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */ include/linux/io_uring_types.h +9
-0@@ -57,8 +57,17 @@ struct io_wq_work {int cancel_seq; }; +/* +* struct io_rsrc_data flag values: +* +* IO_RSRC_DATA_PINNED: data is pinned and cannot be unregistered by userspace +* until it has been unpinned. Currently this is only possible on buffer tables. +*/ +#define IO_RSRC_DATA_PINNED BIT(0) +struct io_rsrc_data {unsigned int nr; +u8 flags;struct io_rsrc_node **nodes; };io_uring/rsrc.c +1
-0@@ -9,6 +9,7 @@#include <linux/hugetlb.h> #include <linux/compat.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> #include <uapi/linux/io_uring.h>
io_uring/rsrc.c static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, · line 309
Quoted 5 lines
io_uring/rsrc.c +2
-0@@ -304,6 +305,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,return -ENXIO ;if (up->offset + nr_args > ctx->buf_table.nr)return -EINVAL ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
IORING_REGISTER_CLONE_BUFFERS can also be used to unregister existing buffers, so it may need the check too?
io_uring/rsrc.c int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) · line 622
Quoted 9 lines
io_uring/rsrc.c +2
-0for (done =0 ; done < nr_args; done++) {struct io_rsrc_node *node;@@ -615,6 +618,8 @@ int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) {if (!ctx->buf_table.nr)return -ENXIO ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
io_buffer_unregister_bvec() can also be used to unregister ublk zero-copy buffers (also under control of userspace), so it may need the check too? But maybe fuse ensures that it never uses a ublk zero-copy buffer?
Best,
Caleb
Quoted 47 lines
io_uring/rsrc.c +37
-1io_rsrc_data_free (ctx, &ctx->buf_table);return 0 ; }@@ -1580,3 +1585,40 @@ int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv, req->flags |=REQ_F_IMPORT_BUFFER ;return 0 ; } + +int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags) +{ +struct io_rsrc_data *data; +int err =0 ; + +io_ring_submit_lock (ctx, issue_flags); + + data = &ctx->buf_table; +/* There was nothing registered. There is nothing to pin */ +if (!data->nr) + err = -ENXIO ; +else + data->flags |=IO_RSRC_DATA_PINNED ; + +io_ring_submit_unlock (ctx, issue_flags); +return err; +} +EXPORT_SYMBOL_GPL (io_uring_buf_table_pin); + +int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags) +{ +struct io_rsrc_data *data; +int err =0 ; + +io_ring_submit_lock (ctx, issue_flags); + + data = &ctx->buf_table; +if (WARN_ON_ONCE (!(data->flags &IO_RSRC_DATA_PINNED ))) + err = -EINVAL ; +else + data->flags &= ~IO_RSRC_DATA_PINNED ; + +io_ring_submit_unlock (ctx, issue_flags); +return err; +} +EXPORT_SYMBOL_GPL (io_uring_buf_table_unpin); --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 20 lines · Caleb Sander Mateos
Quoted 11 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.
Quoted 3 lines
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing
io_rsrc_noderefs field.
How would this work with userspace buffer unregistration (which works at the table level)? If buffer unregistration should still succeed then fuse would need a way to be notified that the buffer has been unregistered since the buffer belongs to userspace (eg it would be wrong if fuse continues using it even though fuse retains a refcount on it). If buffer unregistration should fail, then we would need to track this pinned state inside the node instead of relying just on the refs field, as buffers can be unregistered even if there are in-flight refs (eg we would need to differentiate the ref being from a pin vs from not a pin), and I think this would make unregistration more cumbersome as well (eg we would have to iterate through all the entries looking to see if any are pinned before iterating through them again to do the actual unregistration).
Quoted 75 lines
Quoted 75 lines
This is a preparatory patch for fuse io-uring's usage of fixed buffers.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 13 +++++++++++ include/linux/io_uring_types.h | 9 ++++++++ io_uring/rsrc.c | 42 ++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+)include/linux/io_uring/buf.h +13
-0@@ -9,6 +9,9 @@ int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group,unsigned issue_flags); + +int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags); +int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,@@ -23,6 +26,16 @@ static inline int io_uring_buf_ring_unpin(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } +static inline int io_uring_buf_table_pin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_table_unpin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */ include/linux/io_uring_types.h +9
-0@@ -57,8 +57,17 @@ struct io_wq_work {int cancel_seq; }; +/* +* struct io_rsrc_data flag values: +* +* IO_RSRC_DATA_PINNED: data is pinned and cannot be unregistered by userspace +* until it has been unpinned. Currently this is only possible on buffer tables. +*/ +#define IO_RSRC_DATA_PINNED BIT(0) +struct io_rsrc_data {unsigned int nr; +u8 flags;struct io_rsrc_node **nodes; };io_uring/rsrc.c +1
-0@@ -9,6 +9,7 @@#include <linux/hugetlb.h> #include <linux/compat.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h>
Quoted 10 lines
io_uring/rsrc.c static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, · line 309
Quoted 7 lines
io_uring/rsrc.c +2
-0#include <uapi/linux/io_uring.h> @@ -304,6 +305,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,return -ENXIO ;if (up->offset + nr_args > ctx->buf_table.nr)return -EINVAL ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
IORING_REGISTER_CLONE_BUFFERScan also be used to unregister existing buffers, so it may need the check too?
Ah I didn't realize this existed, thanks. imo I think it's okay to clone the buffers in a source ring's pinned buffer table to the destination ring (where the destination ring's buffer table is unpinned) since the clone acquires its own refcounts on the underlying nodes and the clone is its own entity. Do you think this makes sense or do you think it's better to just not allow this?
Quoted 15 lines
io_uring/rsrc.c int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) · line 622
Quoted 10 lines
io_uring/rsrc.c +2
-0for (done =0 ; done < nr_args; done++) {struct io_rsrc_node *node;@@ -615,6 +618,8 @@ int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) {if (!ctx->buf_table.nr)return -ENXIO ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
io_buffer_unregister_bvec()can also be used to unregister ublk zero-copy buffers (also under control of userspace), so it may need the check too? But maybe fuse ensures that it never uses a ublk zero-copy buffer?
fuse doesn't expose a way for userspace to unregister a zero-copy buffer, but thanks for considering this possibility.
Thanks,
Joanne
Quoted 3 lines
Best,
Caleb
Re: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 27 lines · Joanne Koong
Quoted 20 lines · Caleb Sander Mateos
Quoted 11 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.Quoted 3 lines
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing
io_rsrc_noderefs field.
Quoted 13 lines
How would this work with userspace buffer unregistration (which works at the table level)? If buffer unregistration should still succeed then fuse would need a way to be notified that the buffer has been unregistered since the buffer belongs to userspace (eg it would be wrong if fuse continues using it even though fuse retains a refcount on it). If buffer unregistration should fail, then we would need to track this pinned state inside the node instead of relying just on the refs field, as buffers can be unregistered even if there are in-flight refs (eg we would need to differentiate the ref being from a pin vs from not a pin), and I think this would make unregistration more cumbersome as well (eg we would have to iterate through all the entries looking to see if any are pinned before iterating through them again to do the actual unregistration).
Not sure I would say buffer unregistration operates on the table as a whole. Each registered buffer node is unregistered individually and stores its own reference count. io_put_rsrc_node() will be called on each buffer node in the table. However, io_put_rsrc_node() just removes the one reference from the buffer node. If there are other references on the buffer node (such as an inflight io_uring request using it), io_free_rsrc_node() won't be called to free the buffer node until all those references are dropped too. So fuse holding a reference on the buffer node would allow it to be unregistered, but prevent it from being freed until fuse dropped its reference. I'm not sure I understand the problem with fuse continuing to hold onto a registered buffer node after userspace has unregistered it from the buffer table. (It looks like the buffer node in question is the one at FUSE_URING_FIXED_HEADERS_INDEX?) Wouldn't pinning the buffer table present similar issues? How would userspace get fuse to drop its pin if it wants to modify the buffer registrations? I would imagine the code path that calls io_uring_buf_table_unpin() currently could instead call into io_put_rsrc_node() (maybe by completing an io_uring request that has imported the registered buffer) to release its reference on the buffer node. For ublk, userspace can request to stop a ublk device or the kernel will do so automatically if userspace drops its file handle (e.g. if the process exits), which will release any io_uring resources the ublk device is using.
Quoted 87 lines
Quoted 76 lines
Quoted 76 lines
This is a preparatory patch for fuse io-uring's usage of fixed buffers.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/buf.h | 13 +++++++++++ include/linux/io_uring_types.h | 9 ++++++++ io_uring/rsrc.c | 42 ++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+)include/linux/io_uring/buf.h +13
-0@@ -9,6 +9,9 @@ int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group, unsigned issue_flags, struct io_buffer_list **bl);int io_uring_buf_ring_unpin (struct io_ring_ctx *ctx,unsigned buf_group,unsigned issue_flags); + +int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags); +int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,@@ -23,6 +26,16 @@ static inline int io_uring_buf_ring_unpin(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } +static inline int io_uring_buf_table_pin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +} +static inline int io_uring_buf_table_unpin (struct io_ring_ctx *ctx, +unsigned issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */ include/linux/io_uring_types.h +9
-0@@ -57,8 +57,17 @@ struct io_wq_work {int cancel_seq; }; +/* +* struct io_rsrc_data flag values: +* +* IO_RSRC_DATA_PINNED: data is pinned and cannot be unregistered by userspace +* until it has been unpinned. Currently this is only possible on buffer tables. +*/ +#define IO_RSRC_DATA_PINNED BIT(0) +struct io_rsrc_data {unsigned int nr; +u8 flags;struct io_rsrc_node **nodes; };io_uring/rsrc.c +1
-0@@ -9,6 +9,7 @@#include <linux/hugetlb.h> #include <linux/compat.h> #include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> Quoted 10 lines
io_uring/rsrc.c static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, · line 309
Quoted 7 lines
io_uring/rsrc.c +2
-0#include <uapi/linux/io_uring.h> @@ -304,6 +305,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,return -ENXIO ;if (up->offset + nr_args > ctx->buf_table.nr)return -EINVAL ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
IORING_REGISTER_CLONE_BUFFERScan also be used to unregister existing buffers, so it may need the check too?
Quoted 6 lines
Ah I didn't realize this existed, thanks. imo I think it's okay to clone the buffers in a source ring's pinned buffer table to the destination ring (where the destination ring's buffer table is unpinned) since the clone acquires its own refcounts on the underlying nodes and the clone is its own entity. Do you think this makes sense or do you think it's better to just not allow this?
I think cloning buffers to unused buffer table slots on another ring is fine (analogous to registering new buffers in unused slots). But with IORING_REGISTER_DST_REPLACE set, it can also be used to unregister whatever existing buffers happen to be registered in those slots.
Best,
Caleb
Quoted 25 lines
Quoted 16 lines
io_uring/rsrc.c int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) · line 622
Quoted 11 lines
io_uring/rsrc.c +2
-0for (done =0 ; done < nr_args; done++) {struct io_rsrc_node *node;@@ -615,6 +618,8 @@ int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) {if (!ctx->buf_table.nr)return -ENXIO ; +if (ctx->buf_table.flags &IO_RSRC_DATA_PINNED ) +return -EBUSY ;
io_buffer_unregister_bvec()can also be used to unregister ublk zero-copy buffers (also under control of userspace), so it may need the check too? But maybe fuse ensures that it never uses a ublk zero-copy buffer?fuse doesn't expose a way for userspace to unregister a zero-copy buffer, but thanks for considering this possibility.
Thanks,
JoanneQuoted 3 lines
Best,
Caleb
Joanne Koong
maintainerRe: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 43 lines · Caleb Sander Mateos
Quoted 27 lines · Joanne Koong
Quoted 20 lines · Caleb Sander Mateos
Quoted 11 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.Quoted 3 lines
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing
io_rsrc_noderefs field.Quoted 13 lines
How would this work with userspace buffer unregistration (which works at the table level)? If buffer unregistration should still succeed then fuse would need a way to be notified that the buffer has been unregistered since the buffer belongs to userspace (eg it would be wrong if fuse continues using it even though fuse retains a refcount on it). If buffer unregistration should fail, then we would need to track this pinned state inside the node instead of relying just on the refs field, as buffers can be unregistered even if there are in-flight refs (eg we would need to differentiate the ref being from a pin vs from not a pin), and I think this would make unregistration more cumbersome as well (eg we would have to iterate through all the entries looking to see if any are pinned before iterating through them again to do the actual unregistration).
Quoted 2 lines
Not sure I would say buffer unregistration operates on the table as a whole. Each registered buffer node is unregistered individually and
I'm looking at the liburing interface for it and I'm only seeing io_uring_unregister_buffers() / IORING_UNREGISTER_BUFFERS which works on the entire table, so I'm wondering how that interface would work if pinning/unpinning was at the entry level?
Quoted 11 lines
stores its own reference count.
io_put_rsrc_node()will be called on each buffer node in the table. However,io_put_rsrc_node()just removes the one reference from the buffer node. If there are other references on the buffer node (such as an inflightio_uringrequest using it),io_free_rsrc_node()won't be called to free the buffer node until all those references are dropped too. So fuse holding a reference on the buffer node would allow it to be unregistered, but prevent it from being freed until fuse dropped its reference. I'm not sure I understand the problem with fuse continuing to hold onto a registered buffer node after userspace has unregistered it from the buffer table. (It looks like the buffer node in question is the
For fuse, it holds the reference to the buffer for the lifetime of the connection, which could be a very long time. I'm not seeing how we could let userspace succeed in unregistering with fuse continuing to hold that reference, since as I understand it conceptually, unregistering the buffer should give ownership of the buffer completely back to userspace.
Quoted 1 line
one at
FUSE_URING_FIXED_HEADERS_INDEX?) Wouldn't pinning the buffer
Yep you have that right, the buffer node in question is the one at FUSE_URING_FIXED_HEADERS_INDEX which is where all the headers for requests are placed.
Quoted 1 line
table present similar issues? How would userspace get fuse to drop its
I don't think pinning the buffer table has a similar issue because we disallow unregistration if it's pinned.
Quoted 1 line
pin if it wants to modify the buffer registrations? I would imagine
For the fuse use case, the server never really modifies its buffer registrations as it sets up everything before initiating the connection. But if it wanted to in the future, the server could send a fuse notification to the kernel to unpin the buf table.
Quoted 7 lines
the code path that calls
io_uring_buf_table_unpin()currently could instead call intoio_put_rsrc_node()(maybe by completing anio_uringrequest that has imported the registered buffer) to release its reference on the buffer node. For ublk, userspace can request to stop a ublk device or the kernel will do so automatically if userspace drops its file handle (e.g. if the process exits), which will release anyio_uringresources the ublk device is using.
Fuse has something similar where the server can abort the connection, and that will release the pin / other io uring resources.
Thanks,
Joanne
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Re: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 46 lines · Joanne Koong
Quoted 43 lines · Caleb Sander Mateos
Quoted 40 lines · Joanne Koong
Quoted 20 lines · Caleb Sander Mateos
Quoted 11 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.Quoted 3 lines
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing
io_rsrc_noderefs field.How would this work with userspace buffer unregistration (which works at the table level)? If buffer unregistration should still succeed then fuse would need a way to be notified that the buffer has been unregistered since the buffer belongs to userspace (eg it would be wrong if fuse continues using it even though fuse retains a refcount on it). If buffer unregistration should fail, then we would need to track this pinned state inside the node instead of relying just on the refs field, as buffers can be unregistered even if there are in-flight refs (eg we would need to differentiate the ref being from a pin vs from not a pin), and I think this would make unregistration more cumbersome as well (eg we would have to iterate through all the entries looking to see if any are pinned before iterating through them again to do the actual unregistration).
Quoted 7 lines
Quoted 2 lines
Not sure I would say buffer unregistration operates on the table as a whole. Each registered buffer node is unregistered individually and
I'm looking at the liburing interface for it and I'm only seeing
io_uring_unregister_buffers()/IORING_UNREGISTER_BUFFERSwhich works on the entire table, so I'm wondering how that interface would work if pinning/unpinning was at the entry level?
IORING_REGISTER_BUFFERS_UPDATE can be used to update individual registered buffers. For each updated slot, if there is an existing buffer, it will be unregistered (decrementing the buffer node's reference count). A new buffer may or may not be registered in its place; it can be skipped by specifying a struct iovec with a NULL iov_base.
Quoted 13 lines
Quoted 12 lines
stores its own reference count.
io_put_rsrc_node()will be called on each buffer node in the table. However,io_put_rsrc_node()just removes the one reference from the buffer node. If there are other references on the buffer node (such as an inflightio_uringrequest using it),io_free_rsrc_node()won't be called to free the buffer node until all those references are dropped too. So fuse holding a reference on the buffer node would allow it to be unregistered, but prevent it from being freed until fuse dropped its reference. I'm not sure I understand the problem with fuse continuing to hold onto a registered buffer node after userspace has unregistered it from the buffer table. (It looks like the buffer node in question is the
Quoted 6 lines
For fuse, it holds the reference to the buffer for the lifetime of the connection, which could be a very long time. I'm not seeing how we could let userspace succeed in unregistering with fuse continuing to hold that reference, since as I understand it conceptually, unregistering the buffer should give ownership of the buffer completely back to userspace.
I'm not quite sure what you mean by "give ownership of the buffer completely back to userspace". My understanding is that registering a buffer with io_uring just pins the physical pages as a perf optimization. I'm not aware of a way for userspace to observe directly whether or not certain physical pages are pinned. There's already no guarantee that the physical pages are unpinned as soon as a buffer is unregistered; if there are any inflight io_uring requests using the registered buffer, they will continue to hold a reference count on the buffer, preventing it from being released. The only guarantee is that the unregistered slot in the buffer table is now empty.
Presumably fuse must release its reference count (or pin) eventually, or otherwise there would be a resource leak? I don't see an issue with holding references to registered buffers for the lifetime of a fuse connection. As long as there's a way for the fuse server to tell the kernel to release those resources if they are no longer needed (which it sounds like already exists from your description of aborting a fuse connection).
Quoted 11 lines
Quoted 2 lines
one at
FUSE_URING_FIXED_HEADERS_INDEX?) Wouldn't pinning the bufferYep you have that right, the buffer node in question is the one at
FUSE_URING_FIXED_HEADERS_INDEXwhich is where all the headers for requests are placed.Quoted 1 line
table present similar issues? How would userspace get fuse to drop its
I don't think pinning the buffer table has a similar issue because we disallow unregistration if it's pinned.
It sounds like you're saying that buffer unregistration just isn't expected to work together with fuse's use of registered buffers, is that accurate? Does it matter then whether the buffer unregistration returns -EBUSY because the buffer table is pinned vs. succeeding but not actually releasing the buffer resource because fuse still holds a reference on it? My preference would be to just use the existing reference counting mechanism rather than introducing another mechanism if both provide safety equally well.
Quoted 7 lines
Quoted 2 lines
pin if it wants to modify the buffer registrations? I would imagine
For the fuse use case, the server never really modifies its buffer registrations as it sets up everything before initiating the connection. But if it wanted to in the future, the server could send a fuse notification to the kernel to unpin the buf table.
This seems to assume that all the registered buffers are used exclusively for fuse purposes. But it seems plausible that a fuse server might want to use io_uring for other non-fuse purposes and want to use registered buffers there too as a perf optimization? Then it would be nice for the process to be able to update the other registered buffers even while the kernel pins (or holds a reference count on) the ones used for fuse.
Best,
Caleb
Quoted 19 lines
Quoted 8 lines
the code path that calls
io_uring_buf_table_unpin()currently could instead call intoio_put_rsrc_node()(maybe by completing anio_uringrequest that has imported the registered buffer) to release its reference on the buffer node. For ublk, userspace can request to stop a ublk device or the kernel will do so automatically if userspace drops its file handle (e.g. if the process exits), which will release anyio_uringresources the ublk device is using.Fuse has something similar where the server can abort the connection, and that will release the pin / other io uring resources.
Thanks,
JoanneQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Joanne Koong
maintainerRe: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning
Quoted 8 lines · Caleb Sander Mateos
Quoted 6 lines · Joanne Koong
Quoted 3 lines · Caleb Sander Mateos
Quoted 1 line · Joanne Koong
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Caleb Sander Mateos
Quoted 4 lines · Joanne Koong
Add kernel APIs to pin and unpin the buffer table for fixed buffers, preventing userspace from unregistering or updating the fixed buffers table while it is pinned by the kernel.
After discussing this a bit with Jens at lpc, he's not a fan of the pinning idea for fixed buffers, so for v2 I am going to abandon this and instead just do the lookup every time.
This ends up making the discussion below irrelevant, but adding my comments below anyways for completeness.
Quoted 40 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 14 lines
Quoted 7 lines
This has two advantages:
a) Eliminating the overhead of having to fetch and construct an iter for a fixed buffer per every cmd. Instead, the caller can pin the buffer table, fetch/construct the iter once, and use that across cmds for however long it needs to until it is ready to unpin the buffer table.Quoted 6 lines
b) Allowing a fixed buffer lookup at any index. The buffer table must be pinned in order to allow this, otherwise we would have to keep track of all the nodes that have been looked up by the
io_kiocbso that we can properly adjust the refcounts for those nodes. Ensuring that the buffer table must first be pinned before being able to fetch a buffer at any index makes things logistically a lot neater.Quoted 3 lines
Why is it necessary to pin the entire buffer table rather than specific entries? That's the purpose of the existing
io_rsrc_noderefs field.How would this work with userspace buffer unregistration (which works at the table level)? If buffer unregistration should still succeed then fuse would need a way to be notified that the buffer has been unregistered since the buffer belongs to userspace (eg it would be wrong if fuse continues using it even though fuse retains a refcount on it). If buffer unregistration should fail, then we would need to track this pinned state inside the node instead of relying just on the refs field, as buffers can be unregistered even if there are in-flight refs (eg we would need to differentiate the ref being from a pin vs from not a pin), and I think this would make unregistration more cumbersome as well (eg we would have to iterate through all the entries looking to see if any are pinned before iterating through them again to do the actual unregistration).
Quoted 7 lines
Quoted 2 lines
Not sure I would say buffer unregistration operates on the table as a whole. Each registered buffer node is unregistered individually and
I'm looking at the liburing interface for it and I'm only seeing
io_uring_unregister_buffers()/IORING_UNREGISTER_BUFFERSwhich works on the entire table, so I'm wondering how that interface would work if pinning/unpinning was at the entry level?
Quoted 7 lines
IORING_REGISTER_BUFFERS_UPDATEcan be used to update individual registered buffers. For each updated slot, if there is an existing buffer, it will be unregistered (decrementing the buffer node's reference count). A new buffer may or may not be registered in its place; it can be skipped by specifying a struct iovec with a NULLiov_base.
I think we would still need to account for the IORING_UNREGISTER_BUFFERS path since that is callable by users.
Quoted 31 lines
Quoted 13 lines
Quoted 12 lines
stores its own reference count.
io_put_rsrc_node()will be called on each buffer node in the table. However,io_put_rsrc_node()just removes the one reference from the buffer node. If there are other references on the buffer node (such as an inflightio_uringrequest using it),io_free_rsrc_node()won't be called to free the buffer node until all those references are dropped too. So fuse holding a reference on the buffer node would allow it to be unregistered, but prevent it from being freed until fuse dropped its reference. I'm not sure I understand the problem with fuse continuing to hold onto a registered buffer node after userspace has unregistered it from the buffer table. (It looks like the buffer node in question is theQuoted 6 lines
For fuse, it holds the reference to the buffer for the lifetime of the connection, which could be a very long time. I'm not seeing how we could let userspace succeed in unregistering with fuse continuing to hold that reference, since as I understand it conceptually, unregistering the buffer should give ownership of the buffer completely back to userspace.
I'm not quite sure what you mean by "give ownership of the buffer completely back to userspace". My understanding is that registering a buffer with
io_uringjust pins the physical pages as a perf optimization. I'm not aware of a way for userspace to observe directly whether or not certain physical pages are pinned. There's already no guarantee that the physical pages are unpinned as soon as a buffer is unregistered; if there are any inflightio_uringrequests using the registered buffer, they will continue to hold a reference count on the buffer, preventing it from being released. The only guarantee is that the unregistered slot in the buffer table is now empty.
Quoted 7 lines
Presumably fuse must release its reference count (or pin) eventually, or otherwise there would be a resource leak? I don't see an issue with holding references to registered buffers for the lifetime of a fuse connection. As long as there's a way for the fuse server to tell the kernel to release those resources if they are no longer needed (which it sounds like already exists from your description of aborting a fuse connection).
The way I view it is that conceptually, userspace "owns" the buffer. Userspace allocated the buffer and is later responsible for freeing it when it's done using it. When userspace registers the buffer to the ring, it can expect the buffer to be written to / read from by the kernel code, but it should also be able to expect that when it unregisters the buffer from the ring and any of the concurrent inflight requests are fulfilled, the kernel will not be accessing that buffer anymore and userspace has complete sovereignty over the buffer.
With fuse continuing to hold the reference on the buffer (for the lifetime of the connection) and continuing to read to / write from it after unregistration, that violates that contract.
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines
one at
FUSE_URING_FIXED_HEADERS_INDEX?) Wouldn't pinning the bufferQuoted 8 lines
Yep you have that right, the buffer node in question is the one at
FUSE_URING_FIXED_HEADERS_INDEXwhich is where all the headers for requests are placed.Quoted 1 line
table present similar issues? How would userspace get fuse to drop its
I don't think pinning the buffer table has a similar issue because we disallow unregistration if it's pinned.
Quoted 6 lines
It sounds like you're saying that buffer unregistration just isn't expected to work together with fuse's use of registered buffers, is that accurate? Does it matter then whether the buffer unregistration returns
-EBUSYbecause the buffer table is pinned vs. succeeding but not actually releasing the buffer resource because fuse still holds a reference on it? My preference would be to just use the existing
Yes, I think the difference is that returning -EBUSY signals to the caller that the buffer cannot be unregistered and will still be used. Whereas succeeding but not actually releasing the buffer resource and still continuing to use it beyond the inflight cmds is contrary/deceptive behavior to what the user should be able to expect.
Quoted 11 lines
reference counting mechanism rather than introducing another mechanism if both provide safety equally well.
Quoted 7 lines
Quoted 2 lines
pin if it wants to modify the buffer registrations? I would imagine
For the fuse use case, the server never really modifies its buffer registrations as it sets up everything before initiating the connection. But if it wanted to in the future, the server could send a fuse notification to the kernel to unpin the buf table.
Quoted 7 lines
This seems to assume that all the registered buffers are used exclusively for fuse purposes. But it seems plausible that a fuse server might want to use
io_uringfor other non-fuse purposes and want to use registered buffers there too as a perf optimization? Then it would be nice for the process to be able to update the other registered buffers even while the kernel pins (or holds a reference count on) the ones used for fuse.
I don't think this is likely. I think it's more likely that a high-performance fuse server would use a separate ring for non-fuse-related I/O, as doing it on the same ring it's using to communicate with the fuse kernel would lead to more contention for ring entries which would result in higher latency for servicing incoming/outgoing fuse requests. I guess the caller could make the ring very big to accommodate for that, but then that leads to more memory overhead (eg allocating the headers buffers for all the potential entries).
Quoted 3 lines
Best,
Caleb
Thanks,
Joanne
Joanne Koong
maintainerPATCH 8/30 io_uring/kbuf: add recycling for pinned kernel managed buffer rings
In reply to Joanne KoongAdd an interface for buffers to be recycled back into a pinned kernel-managed buffer ring. This requires the caller to synchronize recycling and selecting.
This is a preparatory patch for fuse io-uring, which requires buffers to be recycled without contending for the uring mutex, as the buffer may need to be selected/recycled in an atomic context.
--- include/linux/io_uring/buf.h | 10 ++++++++++ io_uring/kbuf.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+)
include/linux/io_uring/buf.h +10 -0
@@ -12,6 +12,10 @@ int io_uring_buf_ring_unpin(struct io_ring_ctx *ctx, unsigned buf_group,int io_uring_buf_table_pin (struct io_ring_ctx *ctx,unsigned issue_flags);int io_uring_buf_table_unpin (struct io_ring_ctx *ctx,unsigned issue_flags); + +int io_uring_kmbuf_recycle_pinned (struct io_kiocb *req, +struct io_buffer_list *bl,u64 addr, +unsigned int len,unsigned int bid);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -36,6 +40,12 @@ static inline int io_uring_buf_table_unpin(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } +int io_uring_kmbuf_recycle_pinned (struct io_kiocb *req, +struct io_buffer_list *bl,u64 addr, +unsigned int len,unsigned int bid) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +33 -0
@@ -102,6 +102,39 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; } +/* The caller is responsible for synchronizing recycling and selection */ +int io_uring_kmbuf_recycle_pinned (struct io_kiocb *req, +struct io_buffer_list *bl,u64 addr, +unsigned int len,unsigned int bid) +{ +struct io_uring_buf *buf; +struct io_uring_buf_ring *br; + +if (WARN_ON_ONCE (req->flags &REQ_F_BUFFERS_COMMIT ) || +WARN_ON_ONCE (!(bl->flags &IOBL_PINNED )) || +WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )) || +WARN_ON_ONCE (!(bl->flags &IOBL_KERNEL_MANAGED ))) +return -EINVAL ; + + br = bl->buf_ring; + +if (WARN_ON_ONCE ((br->tail - bl->head) >= bl->nr_entries)) +return -EINVAL ; + + buf = &br->bufs[(br->tail) & bl->mask]; + + buf->addr = addr; + buf->len = len; + buf->bid = bid; + + req->flags &= ~REQ_F_BUFFER_RING ; + + br->tail++; + +return 0 ; +} +EXPORT_SYMBOL_GPL (io_uring_kmbuf_recycle_pinned); +bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags) {struct io_ring_ctx *ctx = req->ctx;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 9/30 io_uring: add io_uring_cmd_import_fixed_index()
In reply to Joanne KoongAdd a new helper, io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.
This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)
include/linux/io_uring/cmd.h +10 -0
@@ -44,6 +44,9 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,size_t uvec_segs, int ddir, struct iov_iter *iter, unsigned issue_flags); +int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags);/* * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd
@@ -100,6 +103,13 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd, +u16 buf_index,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline void __io_uring_cmd_done (struct io_uring_cmd *cmd,s32 ret,u64 ret2,unsigned issue_flags,bool is_cqe32) {
io_uring/rsrc.c +31 -0
@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + +if (buf_index >= req->ctx->buf_table.nr || + !(ctx->buf_table.flags &IO_RSRC_DATA_PINNED )) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + +/* +* We don't have to grab the reference on the node because the buffer +* table is pinned. The caller is responsible for ensuring the iter +* isn't used after the buffer table has been unpinned. +*/ + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +io_ring_submit_unlock (ctx, issue_flags); + +if (!node || !node->buf) +return -EFAULT ; + + imu = node->buf; + +return io_import_fixed (ddir, iter, imu, imu->ubuf, imu->len); +} +/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2) {
io_uring/rsrc.h +2 -0
@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);
io_uring/uring_cmd.c +11 -0
@@ -314,6 +314,17 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec); +int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_import_reg_buf_index (req, iter, buf_index, ddir, + issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_index); +void io_uring_cmd_issue_blocking (struct io_uring_cmd *ioucmd) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd);
Signature · 1 line
2.47.3
Re: [PATCH v1 09/30] io_uring: add io_uring_cmd_import_fixed_index()
Quoted 54 lines · Joanne Koong
Add a new helper,
io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)include/linux/io_uring/cmd.h +10
-0@@ -44,6 +44,9 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,size_t uvec_segs, int ddir, struct iov_iter *iter, unsigned issue_flags); +int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags);/* * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd @@ -100,6 +103,13 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, {return -EOPNOTSUPP ; } +static inline int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd, +u16 buf_index,int ddir, +struct iov_iter *iter, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}static inline void __io_uring_cmd_done (struct io_uring_cmd *cmd,s32 ret,u64 ret2,unsigned issue_flags,bool is_cqe32) {io_uring/rsrc.c +0
-0@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); }
io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1168
Quoted 10 lines
io_uring/rsrc.c +10
-0+int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + + if (buf_index >= req->ctx->buf_table.nr ||
This condition is already checked in io_rsrc_node_lookup() below.
Quoted 44 lines
io_uring/rsrc.c +21
-0+ !(ctx->buf_table.flags & IO_RSRC_DATA_PINNED)) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + } + +/* +* We don't have to grab the reference on the node because the buffer +* table is pinned. The caller is responsible for ensuring the iter +* isn't used after the buffer table has been unpinned. +*/ + node =io_rsrc_node_lookup (&ctx->buf_table, buf_index); +io_ring_submit_unlock (ctx, issue_flags); + +if (!node || !node->buf) +return -EFAULT ; + + imu = node->buf; + +return io_import_fixed (ddir, iter, imu, imu->ubuf, imu->len); +} +/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2) {io_uring/rsrc.h +2
-0@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);io_uring/uring_cmd.c +0
-0@@ -314,6 +314,17 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec);
io_uring/uring_cmd.c int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, · line 325
Quoted 10 lines
io_uring/uring_cmd.c +9
-0+int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_import_reg_buf_index (req, iter, buf_index, ddir, + issue_flags); +}
Probably would make sense to make this an inline function, since it immediately defers to io_import_reg_buf_index().
Best,
Caleb
Quoted 8 lines
io_uring/uring_cmd.c +2
-1+EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_index); +void io_uring_cmd_issue_blocking (struct io_uring_cmd *ioucmd) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 09/30] io_uring: add io_uring_cmd_import_fixed_index()
Quoted 28 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Add a new helper,
io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)io_uring/rsrc.c +0
-0@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); }
Quoted 12 lines
io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1168
Quoted 10 lines
io_uring/rsrc.c +10
-0+int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + + if (buf_index >= req->ctx->buf_table.nr ||This condition is already checked in
io_rsrc_node_lookup()below.
I think we still need this check here to differentiate between -EINVAL if buf_index is out of bounds and -EFAULT if the buf index was not out of bounds but the lookup returned NULL.
Quoted 13 lines
Quoted 13 lines
io_uring/rsrc.c +4
-0+ !(ctx->buf_table.flags & IO_RSRC_DATA_PINNED)) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + }io_uring/uring_cmd.c +0
-0@@ -314,6 +314,17 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec);
Quoted 12 lines
io_uring/uring_cmd.c int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, · line 325
Quoted 9 lines
io_uring/uring_cmd.c +9
-0+int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_import_reg_buf_index (req, iter, buf_index, ddir, + issue_flags); +}Probably would make sense to make this an inline function, since it immediately defers to
io_import_reg_buf_index().
That makes sense to me, I'll make this change for v2.
Thanks,
Joanne
Quoted 3 lines
Best,
Caleb
Re: [PATCH v1 09/30] io_uring: add io_uring_cmd_import_fixed_index()
Quoted 44 lines · Joanne Koong
Quoted 28 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Add a new helper,
io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)io_uring/rsrc.c +0
-0@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); }Quoted 12 lines
io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1168
Quoted 10 lines
io_uring/rsrc.c +10
-0+int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + + if (buf_index >= req->ctx->buf_table.nr ||This condition is already checked in
io_rsrc_node_lookup()below.
Quoted 3 lines
I think we still need this check here to differentiate between
-EINVALifbuf_indexis out of bounds and-EFAULTif the buf index was not out of bounds but the lookup returned NULL.
Is there a reason you prefer EINVAL over EFAULT? EFAULT seems consistent with the errors returned from registered buffer lookups in other cases.
Best,
Caleb
Quoted 35 lines
Quoted 14 lines
Quoted 14 lines
io_uring/rsrc.c +4
-0+ !(ctx->buf_table.flags & IO_RSRC_DATA_PINNED)) { +io_ring_submit_unlock (ctx, issue_flags); +return -EINVAL ; + }io_uring/uring_cmd.c +0
-0@@ -314,6 +314,17 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_vec);Quoted 12 lines
io_uring/uring_cmd.c int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, · line 325
Quoted 9 lines
io_uring/uring_cmd.c +9
-0+int io_uring_cmd_import_fixed_index (struct io_uring_cmd *ioucmd,u16 buf_index, +int ddir,struct iov_iter *iter, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +return io_import_reg_buf_index (req, iter, buf_index, ddir, + issue_flags); +}Probably would make sense to make this an inline function, since it immediately defers to
io_import_reg_buf_index().That makes sense to me, I'll make this change for v2.
Thanks,
JoanneQuoted 3 lines
Best,
Caleb
Joanne Koong
maintainerRe: [PATCH v1 09/30] io_uring: add io_uring_cmd_import_fixed_index()
Quoted 46 lines · Caleb Sander Mateos
Quoted 44 lines · Joanne Koong
Quoted 40 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Add a new helper,
io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)io_uring/rsrc.c +0
-0@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); }io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1168
Quoted 10 lines
io_uring/rsrc.c +10
-0+int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + + if (buf_index >= req->ctx->buf_table.nr ||This condition is already checked in
io_rsrc_node_lookup()below.
Quoted 7 lines
Quoted 3 lines
I think we still need this check here to differentiate between
-EINVALifbuf_indexis out of bounds and-EFAULTif the buf index was not out of bounds but the lookup returned NULL.Is there a reason you prefer EINVAL over EFAULT? EFAULT seems consistent with the errors returned from registered buffer lookups in other cases.
To me -EINVAL makes sense because the error stems from the user passing in an invalid argument (eg a buffer index that exceeds the number of buffers registered to the table). The comment in errno-base.h for EINVAL is "Invalid argument". The EFAULT use for the other cases (eg io_import_reg_buf) makes sense because it might be the case that for whatever reason the req->buf_index isn't found in the table but isn't attributable to having passed in an invalid index.
Thanks,
Joanne
Quoted 4 lines
Best,
Caleb
Re: [PATCH v1 09/30] io_uring: add io_uring_cmd_import_fixed_index()
Quoted 57 lines · Joanne Koong
Quoted 46 lines · Caleb Sander Mateos
Quoted 44 lines · Joanne Koong
Quoted 40 lines · Caleb Sander Mateos
Quoted 26 lines · Joanne Koong
Add a new helper,
io_uring_cmd_import_fixed_index(). This takes in a buffer index. This requires the buffer table to have been pinned beforehand. The caller is responsible for ensuring it does not use the returned iter after the buffer table has been unpinned.This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the sqe's buffer index.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 10 ++++++++++ io_uring/rsrc.c | 31 +++++++++++++++++++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 11 +++++++++++ 4 files changed, 54 insertions(+)io_uring/rsrc.c +0
-0@@ -1156,6 +1156,37 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); }io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1168
Quoted 10 lines
io_uring/rsrc.c +10
-0+int io_import_reg_buf_index (struct io_kiocb *req,struct iov_iter *iter, +u16 buf_index,int ddir,unsigned issue_flags) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + +io_ring_submit_lock (ctx, issue_flags); + + if (buf_index >= req->ctx->buf_table.nr ||This condition is already checked in
io_rsrc_node_lookup()below.Quoted 7 lines
Quoted 3 lines
I think we still need this check here to differentiate between
-EINVALifbuf_indexis out of bounds and-EFAULTif the buf index was not out of bounds but the lookup returned NULL.Is there a reason you prefer EINVAL over EFAULT? EFAULT seems consistent with the errors returned from registered buffer lookups in other cases.
Quoted 7 lines
To me
-EINVALmakes sense because the error stems from the user passing in an invalid argument (eg a buffer index that exceeds the number of buffers registered to the table). The comment inerrno-base.hfor EINVAL is "Invalid argument". The EFAULT use for the other cases (egio_import_reg_buf) makes sense because it might be the case that for whatever reason thereq->buf_indexisn't found in the table but isn't attributable to having passed in an invalid index.
req->buf_index generally comes from the buf_index field of the io_uring SQE, so you could make the same argument about EINVAL making sense for other failed buffer lookups. I don't feel strongly either way, but it seems a bit more consistent (and less code) to just propagate EFAULT from io_rsrc_node_lookup().
Best,
Caleb
Joanne Koong
maintainerPATCH 10/30 io_uring/kbuf: add io_uring_is_kmbuf_ring()
In reply to Joanne KoongAdd a function io_uring_is_kmbuf_ring() that returns true if there is a kernel-managed buffer ring at the specified buffer group.
This is a preparatory patch for upcoming fuse kernel-managed buffer support, which needs to ensure the buffer ring registered by the server is a kernel-managed buffer ring.
--- include/linux/io_uring/buf.h | 9 +++++++++ io_uring/kbuf.c | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+)
include/linux/io_uring/buf.h +9 -0
@@ -16,6 +16,9 @@ int io_uring_buf_table_unpin(struct io_ring_ctx *ctx, unsigned issue_flags);int io_uring_kmbuf_recycle_pinned (struct io_kiocb *req,struct io_buffer_list *bl,u64 addr,unsigned int len,unsigned int bid); + +bool io_uring_is_kmbuf_ring (struct io_ring_ctx *ctx,unsigned int buf_group, +unsigned int issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -46,6 +49,12 @@ int io_uring_kmbuf_recycle_pinned(struct io_kiocb *req, {return -EOPNOTSUPP ; } +static inline bool io_uring_is_kmbuf_ring (struct io_ring_ctx *ctx, +unsigned int buf_group, +unsigned int issue_flags) +{ +return false; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +19 -0
@@ -945,3 +945,22 @@ int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg)return 0 ; } + +bool io_uring_is_kmbuf_ring (struct io_ring_ctx *ctx,unsigned int buf_group, +unsigned int issue_flags) +{ +struct io_buffer_list *bl; +bool is_kmbuf_ring = false; + +io_ring_submit_lock (ctx, issue_flags); + + bl =io_buffer_get_list (ctx, buf_group); +if (likely (bl) && (bl->flags &IOBL_KERNEL_MANAGED )) { +WARN_ON_ONCE (!(bl->flags &IOBL_BUF_RING )); + is_kmbuf_ring = true; + } + +io_ring_submit_unlock (ctx, issue_flags); +return is_kmbuf_ring; +} +EXPORT_SYMBOL_GPL (io_uring_is_kmbuf_ring);
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 11/30 io_uring/kbuf: return buffer id in buffer selection
In reply to Joanne KoongReturn the id of the selected buffer in io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.
--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +1 -1
@@ -74,7 +74,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);/* * Select a buffer from the provided buffer group for multishot uring_cmd. -* Returns the selected buffer address and size. +* Returns the selected buffer address, size, and id. */ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, unsigned buf_group, size_t *len,
include/linux/io_uring_types.h +2 -0
@@ -109,6 +109,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id; };
io_uring/kbuf.c +5 -2
@@ -239,6 +239,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index = buf->bid; sel.buf_list = bl; + sel.buf_id = buf->bid;if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)buf->addr;else
@@ -262,10 +263,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index; + } }io_ring_submit_unlock (req->ctx, issue_flags);return sel;
Signature · 1 line
2.47.3
Re: [PATCH v1 11/30] io_uring/kbuf: return buffer id in buffer selection
Quoted 30 lines · Joanne Koong
Return the id of the selected buffer in
io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)include/linux/io_uring/cmd.h +1
-1@@ -74,7 +74,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);/* * Select a buffer from the provided buffer group for multishot uring_cmd. -* Returns the selected buffer address and size. +* Returns the selected buffer address, size, and id. */ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, unsigned buf_group, size_t *len,include/linux/io_uring_types.h +0
-0
include/linux/io_uring_types.h struct io_br_sel { · line 113
Quoted 5 lines
include/linux/io_uring_types.h +2
-0@@ -109,6 +109,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id;
Looks like this could be unioned with val? I think val's size can be reduced to an int since only int values are assigned to it.
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 242
Quoted 12 lines
include/linux/io_uring_types.h +0
-0};io_uring/kbuf.c +1
-0@@ -239,6 +239,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index = buf->bid; sel.buf_list = bl; + sel.buf_id = buf->bid;
This is userspace mapped, so probably should be using READ_ONCE() and reusing the value between req->buf_index and buf->bid? Looks like an existing bug that the reads of buf->bid and buf->addr aren't using READ_ONCE().
Quoted 4 lines
io_uring/kbuf.c +0
-0if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)buf->addr;else
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 270
Quoted 10 lines
io_uring/kbuf.c +3
-2@@ -262,10 +263,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index;
Could this cover both IOBL_BUF_RING and !IOBL_BUF_RING cases to avoid the additional logic in io_ring_buffer_select()?
Best,
Caleb
Quoted 7 lines
io_uring/kbuf.c +1
-1+ } }io_ring_submit_unlock (req->ctx, issue_flags);return sel; --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 11/30] io_uring/kbuf: return buffer id in buffer selection
Quoted 14 lines · Caleb Sander Mateos
Quoted 12 lines · Joanne Koong
Return the id of the selected buffer in
io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)
Quoted 14 lines
include/linux/io_uring_types.h struct io_br_sel { · line 113
Quoted 10 lines
include/linux/io_uring_types.h +2
-0@@ -109,6 +109,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id;Looks like this could be unioned with val? I think val's size can be reduced to an int since only int values are assigned to it.
I'm not sure I see the advantage of this. imo I think it makes the interface a bit more confusing, as val also represents the error code and is logically its own separate entity from buf_id. I don't see the struct io_br_sel being stored anywhere where it seems important to save a few bytes, but maybe I'm missing something here.
Quoted 17 lines
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 242
Quoted 12 lines
include/linux/io_uring_types.h +0
-0};io_uring/kbuf.c +1
-0@@ -239,6 +239,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index = buf->bid; sel.buf_list = bl; + sel.buf_id = buf->bid;This is userspace mapped, so probably should be using
READ_ONCE()and reusing the value betweenreq->buf_indexandbuf->bid? Looks like an existing bug that the reads ofbuf->bidandbuf->addraren't usingREAD_ONCE().
Agreed. I think that existing bug should be its own patch. Are you planning to submit that fix? since you found that bug, I think you should get the authorship for it, but if it's annoying for whatever reason and you prefer not to, I can roll that up into this patchset. EIther way, I'll rebase v2 on top of that and change this line to "sel.buf_id = req->buf_index".
Quoted 6 lines
Quoted 6 lines
io_uring/kbuf.c +0
-0if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)buf->addr;else @@ -262,10 +263,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,
Quoted 12 lines
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 270
Quoted 9 lines
io_uring/kbuf.c +3
-2bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index;Could this cover both
IOBL_BUF_RINGand !IOBL_BUF_RING cases to avoid the additional logic inio_ring_buffer_select()?
Ah the next patch (patch 12/30: io_uring/kbuf: export io_ring_buffer_select() [1]) exports io_ring_buffer_select() to be directly called, so that was my motivation for having that sel.buf_id assignment logic in io_ring_buffer_select(). I should have swapped the ordering of those 2 patches to make that more clear, I'll do that for v2.
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Thanks,
Joanne
Quoted 11 lines
Best,
CalebQuoted 7 lines
io_uring/kbuf.c +1
-1+ } }io_ring_submit_unlock (req->ctx, issue_flags);return sel; --2.47.3
Re: [PATCH v1 11/30] io_uring/kbuf: return buffer id in buffer selection
Quoted 32 lines · Joanne Koong
Quoted 14 lines · Caleb Sander Mateos
Quoted 12 lines · Joanne Koong
Return the id of the selected buffer in
io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 2 +- include/linux/io_uring_types.h | 2 ++ io_uring/kbuf.c | 7 +++++-- 3 files changed, 8 insertions(+), 3 deletions(-)Quoted 14 lines
include/linux/io_uring_types.h struct io_br_sel { · line 113
Quoted 10 lines
include/linux/io_uring_types.h +2
-0@@ -109,6 +109,8 @@ struct io_br_sel {void *kaddr; };ssize_t val; +/* id of the selected buffer */ +unsigned buf_id;Looks like this could be unioned with val? I think val's size can be reduced to an int since only int values are assigned to it.
Quoted 5 lines
I'm not sure I see the advantage of this. imo I think it makes the interface a bit more confusing, as val also represents the error code and is logically its own separate entity from
buf_id. I don't see the structio_br_selbeing stored anywhere where it seems important to save a few bytes, but maybe I'm missing something here.
Yeah, fair enough. Looks like it's only stored on the stack and returned from functions. Splitting ssize_t val out of the union has already increased its size beyond 16 bytes, so it was already too large to fit in a register.
Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
include/linux/io_uring_types.h +0
-0};Quoted 14 lines
io_uring/kbuf.c static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, · line 242
Quoted 9 lines
io_uring/kbuf.c +1
-0@@ -239,6 +239,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->flags |=REQ_F_BUFFER_RING |REQ_F_BUFFERS_COMMIT ; req->buf_index = buf->bid; sel.buf_list = bl; + sel.buf_id = buf->bid;This is userspace mapped, so probably should be using
READ_ONCE()and reusing the value betweenreq->buf_indexandbuf->bid? Looks like an existing bug that the reads ofbuf->bidandbuf->addraren't usingREAD_ONCE().
Quoted 6 lines
Agreed. I think that existing bug should be its own patch. Are you planning to submit that fix? since you found that bug, I think you should get the authorship for it, but if it's annoying for whatever reason and you prefer not to, I can roll that up into this patchset. EIther way, I'll rebase v2 on top of that and change this line to "sel.buf_id =
req->buf_index".
Sure, happy to send a patch.
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
io_uring/kbuf.c +0
-0if (bl->flags &IOBL_KERNEL_MANAGED ) sel.kaddr = (void *)buf->addr;else @@ -262,10 +263,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,Quoted 12 lines
io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 270
Quoted 9 lines
io_uring/kbuf.c +3
-2bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) { -if (bl->flags &IOBL_BUF_RING ) +if (bl->flags &IOBL_BUF_RING ) { sel =io_ring_buffer_select (req, len, bl, issue_flags); -else + }else { sel.addr =io_provided_buffer_select (req, len, bl); + sel.buf_id = req->buf_index;Could this cover both
IOBL_BUF_RINGand !IOBL_BUF_RING cases to avoid the additional logic inio_ring_buffer_select()?
Quoted 6 lines
Ah the next patch (patch 12/30:
io_uring/kbuf: exportio_ring_buffer_select()[1]) exportsio_ring_buffer_select()to be directly called, so that was my motivation for having that sel.buf_id assignment logic inio_ring_buffer_select(). I should have swapped the ordering of those 2 patches to make that more clear, I'll do that for v2.
Makes sense. No need to switch the order of the patches.
Best,
Caleb
Joanne Koong
maintainerPATCH 12/30 io_uring/kbuf: export io_ring_buffer_select()
In reply to Joanne KoongExport io_ring_buffer_select() so that it may be used by callers who pass in a pinned bufring without needing to grab the io_uring mutex.
This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.
--- include/linux/io_uring/buf.h | 15 +++++++++++++++ io_uring/kbuf.c | 7 ++++--- 2 files changed, 19 insertions(+), 3 deletions(-)
include/linux/io_uring/buf.h +15 -0
@@ -19,6 +19,10 @@ int io_uring_kmbuf_recycle_pinned(struct io_kiocb *req,bool io_uring_is_kmbuf_ring (struct io_ring_ctx *ctx,unsigned int buf_group,unsigned int issue_flags); + +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -55,6 +59,17 @@ static inline bool io_uring_is_kmbuf_ring(struct io_ring_ctx *ctx, {return false; } +static inline struct io_br_sel io_ring_buffer_select (struct io_kiocb *req, +size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) +{ +struct io_br_sel sel = { + .val = -EOPNOTSUPP , + }; + +return sel; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +4 -3
@@ -215,9 +215,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,return false; } -static struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, -struct io_buffer_list *bl, -unsigned int issue_flags) +struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len, +struct io_buffer_list *bl, +unsigned int issue_flags) {struct io_uring_buf_ring *br = bl->buf_ring;__u16 tail, head = bl->head;
@@ -251,6 +251,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }return sel; } +EXPORT_SYMBOL_GPL (io_ring_buffer_select);struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags)
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 13/30 io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
In reply to Joanne KoongWhen uring_cmd operations select a buffer, the completion queue entry should indicate which buffer was selected.
Set IORING_CQE_F_BUFFER on the completed entry and encode the buffer index if a buffer was selected.
This will be needed for fuse, which needs to relay to userspace which selected buffer contains the data.
--- io_uring/uring_cmd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/uring_cmd.c +5 -1
@@ -142,6 +142,7 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32) {struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); +u32 cflags =0 ;if (WARN_ON_ONCE (req->flags &REQ_F_APOLL_MULTISHOT ))return ;
@@ -151,7 +152,10 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (ret <0 )req_set_fail (req); -io_req_set_res (req, ret,0 ); +if (req->flags & (REQ_F_BUFFER_SELECTED |REQ_F_BUFFER_RING )) + cflags |=IORING_CQE_F_BUFFER | + (req->buf_index <<IORING_CQE_BUFFER_SHIFT ); +io_req_set_res (req, ret, cflags);if (is_cqe32) {if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 14/30 io_uring: add release callback for ring death
In reply to Joanne KoongAllow registering a release callback on a ring context that will be called when the ring is about to be destroyed.
This is a preparatory patch for fuse. Fuse will be pinning buffers and registering bvecs, which requires cleanup whenever a server disconnects. It needs to know if the ring is alive when the server has disconnected, to avoid double-freeing or accessing invalid memory.
--- include/linux/io_uring.h | 9 +++++++++ include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 15 +++++++++++++++ 3 files changed, 26 insertions(+)
include/linux/io_uring.h +9 -0
@@ -2,6 +2,7 @@#ifndef _LINUX_IO_URING_H#define _LINUX_IO_URING_H +#include <linux/io_uring_types.h> #include <linux/sched.h> #include <linux/xarray.h> #include <uapi/linux/io_uring.h>
@@ -28,6 +29,9 @@ static inline void io_uring_free(struct task_struct *tsk)if (tsk->io_uring) __io_uring_free(tsk); } +void io_uring_set_release_callback (struct io_ring_ctx *ctx, +void (*release)(void *),void *priv, +unsigned int issue_flags);#else static inline void io_uring_task_cancel (void ) {
@@ -46,6 +50,11 @@ static inline bool io_is_uring_fops(struct file *file) {return false; } +static inline void +io_uring_set_release_callback (struct io_ring_ctx *ctx,void (*release)(void *), +void *priv,unsigned int issue_flags) +{ +}#endif #endif
include/linux/io_uring_types.h +2 -0
@@ -441,6 +441,8 @@ struct io_ring_ctx {struct work_struct exit_work;struct list_head tctx_list;struct completion ref_comp; +void (*release)(void *); +void *priv;/* io-wq management, e.g. thread count */ u32 iowq_limits[2 ];
io_uring/io_uring.c +15 -0
@@ -2952,6 +2952,19 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait)return mask; } +void io_uring_set_release_callback (struct io_ring_ctx *ctx, +void (*release)(void *),void *priv, +unsigned int issue_flags) +{ +io_ring_submit_lock (ctx, issue_flags); + + ctx->release = release; + ctx->priv = priv; + +io_ring_submit_unlock (ctx, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_set_release_callback); +struct io_tctx_exit {struct callback_head task_work;struct completion completion;
@@ -3099,6 +3112,8 @@ static int io_uring_release(struct inode *inode, struct file *file)struct io_ring_ctx *ctx = file->private_data; file->private_data =NULL ; +if (ctx->release) + ctx->release (ctx->priv);io_ring_ctx_wait_and_kill (ctx);return 0 ; }
Signature · 1 line
2.47.3
Re: [PATCH v1 14/30] io_uring: add release callback for ring death
Quoted 69 lines · Joanne Koong
Allow registering a release callback on a ring context that will be called when the ring is about to be destroyed.
This is a preparatory patch for fuse. Fuse will be pinning buffers and registering bvecs, which requires cleanup whenever a server disconnects. It needs to know if the ring is alive when the server has disconnected, to avoid double-freeing or accessing invalid memory.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring.h | 9 +++++++++ include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 15 +++++++++++++++ 3 files changed, 26 insertions(+)include/linux/io_uring.h +9
-0@@ -2,6 +2,7 @@#ifndef _LINUX_IO_URING_H#define _LINUX_IO_URING_H +#include <linux/io_uring_types.h> #include <linux/sched.h> #include <linux/xarray.h> #include <uapi/linux/io_uring.h> @@ -28,6 +29,9 @@ static inline void io_uring_free(struct task_struct *tsk)if (tsk->io_uring) __io_uring_free(tsk); } +void io_uring_set_release_callback (struct io_ring_ctx *ctx, +void (*release)(void *),void *priv, +unsigned int issue_flags);#else static inline void io_uring_task_cancel (void ) {@@ -46,6 +50,11 @@ static inline bool io_is_uring_fops(struct file *file) {return false; } +static inline void +io_uring_set_release_callback (struct io_ring_ctx *ctx,void (*release)(void *), +void *priv,unsigned int issue_flags) +{ +}#endif #endif include/linux/io_uring_types.h +2
-0@@ -441,6 +441,8 @@ struct io_ring_ctx {struct work_struct exit_work;struct list_head tctx_list;struct completion ref_comp; +void (*release)(void *); +void *priv;/* io-wq management, e.g. thread count */ u32 iowq_limits[2 ];io_uring/io_uring.c +0
-0@@ -2952,6 +2952,19 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait)return mask;
io_uring/io_uring.c static __poll_t io_uring_poll(struct file *file, poll_table *wait) · line 2962
Quoted 10 lines
io_uring/io_uring.c +8
-0} +void io_uring_set_release_callback (struct io_ring_ctx *ctx, +void (*release)(void *),void *priv, +unsigned int issue_flags) +{ +io_ring_submit_lock (ctx, issue_flags); + + ctx->release = release; + ctx->priv = priv;
Looks like this doesn't support the registration of multiple release callbacks. Should there be a WARN_ON() to that effect?
Best,
Caleb
Quoted 20 lines
io_uring/io_uring.c +7
-1+ +io_ring_submit_unlock (ctx, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_set_release_callback); +struct io_tctx_exit {struct callback_head task_work;struct completion completion;@@ -3099,6 +3112,8 @@ static int io_uring_release(struct inode *inode, struct file *file)struct io_ring_ctx *ctx = file->private_data; file->private_data =NULL ; +if (ctx->release) + ctx->release (ctx->priv);io_ring_ctx_wait_and_kill (ctx);return 0 ; } --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 14/30] io_uring: add release callback for ring death
Quoted 21 lines · Caleb Sander Mateos
Quoted 19 lines · Joanne Koong
Allow registering a release callback on a ring context that will be called when the ring is about to be destroyed.
This is a preparatory patch for fuse. Fuse will be pinning buffers and registering bvecs, which requires cleanup whenever a server disconnects. It needs to know if the ring is alive when the server has disconnected, to avoid double-freeing or accessing invalid memory.
Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring.h | 9 +++++++++ include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 15 +++++++++++++++ 3 files changed, 26 insertions(+)include/linux/io_uring.h +0
-0
Quoted 11 lines
include/linux/io_uring.h
Quoted 8 lines
include/linux/io_uring.h +8
-0+void io_uring_set_release_callback (struct io_ring_ctx *ctx, +void (*release)(void *),void *priv, +unsigned int issue_flags) +{ +io_ring_submit_lock (ctx, issue_flags); + + ctx->release = release; + ctx->priv = priv;Looks like this doesn't support the registration of multiple release callbacks. Should there be a
WARN_ON()to that effect?
Great idea, I'll add that in for v2.
Thanks,
Joanne
Quoted 3 lines
Best,
Caleb
Joanne Koong
maintainerPATCH 15/30 fuse: refactor io-uring logic for getting next fuse request
In reply to Joanne KoongSimplify the logic for getting the next fuse request.
--- fs/fuse/dev_uring.c | 78 ++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
@@ -714,34 +714,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,return err; } -/* -* Write data to the ring buffer and send the request to userspace, -* userspace will read it -* This is comparable with classical read(/dev/fuse) -*/ -static int fuse_uring_send_next_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req, -unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; -int err; -struct io_uring_cmd *cmd; - - err =fuse_uring_prepare_send (ent, req); -if (err) -return err; - -spin_lock (&queue->lock); - cmd = ent->cmd; - ent->cmd =NULL ; - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd,0 , issue_flags); -return 0 ; -} -/* * Make a ring entry available for fuse_req assignment */
@@ -838,11 +810,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, }/* -* Get the next fuse req and send it +* Get the next fuse req. +* +* Returns true if the next fuse request has been assigned to the ent. +* Else, there is no next fuse request and this returns false. */ -static void fuse_uring_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue, -unsigned int issue_flags) +static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, +struct fuse_ring_queue *queue) {int err;struct fuse_req *req;
@@ -854,10 +828,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_send_next_to_ring (ent, req, issue_flags); + err =fuse_uring_prepare_send (ent, req);if (err) goto retry; } + +return req !=NULL ; }static int fuse_ring_ent_set_commit (struct fuse_ring_ent *ent)
@@ -875,6 +851,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)return 0 ; } +static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, +ssize_t ret,unsigned int issue_flags) +{ +struct fuse_ring_queue *queue = ent->queue; + +spin_lock (&queue->lock); + ent->state =FRRS_USERSPACE ; +list_move_tail (&ent->list, &queue->ent_in_userspace); + ent->cmd =NULL ; +spin_unlock (&queue->lock); + +io_uring_cmd_done (cmd, ret, issue_flags); +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_conn *fc)
@@ -946,7 +936,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -fuse_uring_next_fuse_req (ent, queue, issue_flags); +if (fuse_uring_get_next_fuse_req (ent, queue)) +fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1194,20 +1185,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)return -EIOCBQUEUED ; } -static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, -ssize_t ret,unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; - -spin_lock (&queue->lock); - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); - ent->cmd =NULL ; -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd, ret, issue_flags); -} -/* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission
@@ -1224,8 +1201,9 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)if (!tw.cancel) { err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -fuse_uring_next_fuse_req (ent, queue, issue_flags); -return ; +if (!fuse_uring_get_next_fuse_req (ent, queue)) +return ; + err =0 ; } }else { err = -ECANCELED ;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 16/30 fuse: refactor io-uring header copying to ring
In reply to Joanne KoongMove header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +21 -18
@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -637,13 +649,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;
@@ -659,9 +669,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -690,14 +699,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err =copy_to_user (&ent->headers->in_out, &req->in.h, -sizeof (req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 17/30 fuse: refactor io-uring header copying from ring
In reply to Joanne KoongMove header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +18 -6
@@ -587,6 +587,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } +static __always_inline int copy_header_from_ring(void *header, +const void __user *ring, +size_t header_size) +{ +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -597,10 +609,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_from_user (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, +sizeof (ring_in_out));if (err) -return -EFAULT ; +return err; err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, &iter);
@@ -794,10 +806,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_from_user (&req->out.h, &ent->headers->in_out, -sizeof (req->out.h)); + err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, +sizeof (req->out.h));if (err) { - req->out.h.error = -EFAULT ; + req->out.h.error = err; goto out; }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 18/30 fuse: use enum types for header copying
In reply to Joanne KoongUse enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when kernel-managed buffer rings are added.
--- fs/fuse/dev_uring.c | 57 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +47 -10
@@ -31,6 +31,15 @@ struct fuse_uring_pdu {static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +/* struct fuse_in_header / struct fuse_out_header */ +FUSE_URING_HEADER_IN_OUT , +/* per op code header */ +FUSE_URING_HEADER_OP , +/* struct fuse_uring_ent_in_out header */ +FUSE_URING_HEADER_RING_ENT , +}; +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -575,10 +584,32 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } -static __always_inline int copy_header_to_ring(void __user *ring, +static void __user *get_user_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} + +static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent, +enum fuse_uring_header_type type,const void *header,size_t header_size) { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) {pr_info_ratelimited ("Copying header to ring failed.\n" );return -EFAULT ;
@@ -587,10 +618,16 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } -static __always_inline int copy_header_from_ring(void *header, -const void __user *ring, +static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +void *header,size_t header_size) { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) {pr_info_ratelimited ("Copying header from ring failed.\n" );return -EFAULT ;
@@ -609,8 +646,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ring_in_out,sizeof (ring_in_out));if (err)return err;
@@ -661,7 +698,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_header_to_ring (&ent->headers->op_in, + err =copy_header_to_ring (ent,FUSE_URING_HEADER_OP , in_args->value, in_args->size);if (err)
@@ -681,8 +718,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; -return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ent_in_out,sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -711,7 +748,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ -return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +return copy_header_to_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->in.h,sizeof (req->in.h)); }
@@ -806,7 +843,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 19/30 fuse: refactor setting up copy state for payload copying
In reply to Joanne KoongAdd a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.
--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +24 -14
@@ -636,6 +636,27 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; } +static int setup_fuse_copy_state (struct fuse_copy_state *cs, +struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent *ent,int dir, +struct iov_iter *iter) +{ +int err; + + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } + +fuse_copy_init (cs, dir ==ITER_DEST , iter); + + cs->is_uring = true; + cs->req = req; + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -651,15 +672,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, - &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter);if (err)return err; -fuse_copy_init (&cs, false, &iter); - cs.is_uring = true; - cs.req = req; - err =fuse_copy_out_args (&cs, args, ring_in_out.payload_sz);fuse_copy_finish (&cs);return err;
@@ -682,15 +698,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter); +if (err)return err; - } - -fuse_copy_init (&cs, true, &iter); - cs.is_uring = true; - cs.req = req;if (num_args >0 ) {/*
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 20/30 fuse: support buffer copying for kernel addresses
In reply to Joanne KoongThis is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.
Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.
--- fs/fuse/dev.c | 23 +++++++++++++++++------ fs/fuse/fuse_dev_i.h | 7 ++++++- 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,/* Unmap and put previous page of userspace buffer */ void fuse_copy_finish (struct fuse_copy_state *cs) { +if (cs->is_kaddr) +return ; +if (cs->currbuf) {struct pipe_buffer *buf = cs->currbuf;
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)struct page *page;int err; +if (cs->is_kaddr) +return 0 ; + err =unlock_request (cs->req);if (err)return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size) {unsigned ncpy =min (*size, cs->len);if (val) { -void *pgaddr =kmap_local_page (cs->pg); -void *buf = pgaddr + cs->offset; +void *pgaddr, *buf; +if (!cs->is_kaddr) { + pgaddr =kmap_local_page (cs->pg); + buf = pgaddr + cs->offset; + }else { + buf = cs->kaddr + cs->offset; + }if (cs->write)memcpy (buf, *val, ncpy);else memcpy (*val, buf, ncpy); - -kunmap_local (pgaddr); +if (!cs->is_kaddr) +kunmap_local (pgaddr); *val += ncpy; } *size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, }while (count) { -if (cs->write && cs->pipebufs && folio) { +if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {/* * Can't control lifetime of pipe buffers, so always * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, }else {return fuse_ref_folio (cs, folio, offset, count); } - }else if (!cs->len) { + }else if (!cs->len && !cs->is_kaddr) {if (cs->move_folios && folio && offset ==0 && count == size) { err =fuse_try_move_folio (cs, foliop);
fs/fuse/fuse_dev_i.h +6 -1
@@ -28,12 +28,17 @@ struct fuse_copy_state {struct pipe_buffer *currbuf;struct pipe_inode_info *pipe;unsigned long nr_segs; -struct page *pg; +union { +struct page *pg; +void *kaddr; + };unsigned int len;unsigned int offset; bool write:1; bool move_folios:1; bool is_uring:1; +/* if set, use kaddr; otherwise use pg */ + bool is_kaddr:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 21/30 fuse: add io-uring kernel-managed buffer ring
In reply to Joanne KoongAdd io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.
This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction
b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.
Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring and fixed buffer will be pinned for the lifetime of the connection.
--- fs/fuse/dev_uring.c | 452 +++++++++++++++++++++++++++++++++----- fs/fuse/dev_uring_i.h | 35 ++- include/uapi/linux/fuse.h | 12 +- 3 files changed, 437 insertions(+), 62 deletions(-)
fs/fuse/dev_uring.c +394 -58
@@ -10,6 +10,8 @@#include "fuse_trace.h" #include <linux/fs.h> +#include <linux/io_uring.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> static bool __read_mostly enable_uring;
@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,#define FUSE_URING_IOV_SEGS 2/* header and payload */ +#define FUSE_URING_RINGBUF_GROUP 0 +#define FUSE_URING_FIXED_HEADERS_INDEX 0bool fuse_uring_enabled (void ) {
@@ -194,6 +198,37 @@ bool fuse_uring_request_expired(struct fuse_conn *fc)return false; } +static void fuse_uring_teardown_buffers (struct fuse_ring_queue *queue, +unsigned int issue_flags) +{ +if (!queue->use_bufring) +return ; + +spin_lock (&queue->lock); + +if (queue->ring_killed) { +spin_unlock (&queue->lock); +return ; + } + +/* +* Try to get a reference on it so the ctx isn't killed while we're +* unpinning +*/ +if (!percpu_ref_tryget_live (&queue->ring_ctx->refs)) { +spin_unlock (&queue->lock); +return ; + } + +spin_unlock (&queue->lock); + +WARN_ON_ONCE (io_uring_buf_table_unpin (queue->ring_ctx, issue_flags)); +WARN_ON_ONCE (io_uring_buf_ring_unpin (queue->ring_ctx, +FUSE_URING_RINGBUF_GROUP , + issue_flags)); +percpu_ref_put (&queue->ring_ctx->refs); +} +void fuse_uring_destruct (struct fuse_conn *fc) {struct fuse_ring *ring = fc->ring;
@@ -276,20 +311,76 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +static void io_ring_killed (void *priv) +{ +struct fuse_ring_queue *queue = (struct fuse_ring_queue *)priv; + +spin_lock (&queue->lock); + queue->ring_killed = true; +spin_unlock (&queue->lock); +} + +static int fuse_uring_buf_ring_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue, +unsigned int issue_flags) +{ +struct io_ring_ctx *ring_ctx =cmd_to_io_kiocb (cmd)->ctx; +int err; + + err =io_uring_buf_ring_pin (ring_ctx,FUSE_URING_RINGBUF_GROUP , + issue_flags, &queue->bufring); +if (err) +return err; + +if (!io_uring_is_kmbuf_ring (ring_ctx,FUSE_URING_RINGBUF_GROUP , + issue_flags)) { + err = -EINVAL ; + goto error; + } + + err =io_uring_buf_table_pin (ring_ctx, issue_flags); +if (err) + goto error; + + err =io_uring_cmd_import_fixed_index (cmd, +FUSE_URING_FIXED_HEADERS_INDEX , +ITER_DEST , &queue->headers_iter, + issue_flags); +if (err) { +io_uring_buf_table_unpin (ring_ctx, issue_flags); + goto error; + } + +io_uring_set_release_callback (ring_ctx, io_ring_killed, queue, + issue_flags); + queue->ring_ctx = ring_ctx; + + queue->use_bufring = true; + +return 0 ; + +error: +io_uring_buf_ring_unpin (ring_ctx,FUSE_URING_RINGBUF_GROUP , + issue_flags); +return err; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,bool use_bufring,unsigned int issue_flags) {struct fuse_conn *fc = ring->fc;struct fuse_ring_queue *queue;struct list_head *pq; +int err =0 ; queue =kzalloc (sizeof (*queue),GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =kcalloc (FUSE_PQ_HASH_SIZE ,sizeof (struct list_head ),GFP_KERNEL );if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;
@@ -307,6 +398,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, queue->fpq.processing = pq;fuse_pqueue_init (&queue->fpq); +if (use_bufring) { + err =fuse_uring_buf_ring_setup (cmd, queue, issue_flags); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fc->lock);if (ring->queues[qid]) {spin_unlock (&fc->lock);
@@ -452,6 +552,7 @@ static void fuse_uring_async_stop_queues(struct work_struct *work)continue ;fuse_uring_teardown_entries (queue); +fuse_uring_teardown_buffers (queue,IO_URING_F_UNLOCKED ); }/*
@@ -487,6 +588,7 @@ void fuse_uring_stop_queues(struct fuse_ring *ring)continue ;fuse_uring_teardown_entries (queue); +fuse_uring_teardown_buffers (queue,IO_URING_F_UNLOCKED ); }if (atomic_read (&ring->queue_refs) >0 ) {
@@ -584,6 +686,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +struct iov_iter *headers_iter) +{ +size_t offset; + +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +/* No offset - start of header */ + offset =0 ; +break ; +case FUSE_URING_HEADER_OP : + offset = offsetof(struct fuse_uring_req_header , op_in); +break ; +case FUSE_URING_HEADER_RING_ENT : + offset = offsetof(struct fuse_uring_req_header , ring_ent_in_out); +break ; +default : +WARN_ONCE (1 ,"Invalid header type: %d\n" , type); +return -EINVAL ; + } + + *headers_iter = ent->headers_iter; +if (offset) +iov_iter_advance (headers_iter, offset); + +return 0 ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {
@@ -605,17 +736,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; + + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (copy_to_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } + +if (copy_to_user (ring, header, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header to ring failed: " +"header_type=%u, header_size=%lu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
@@ -623,17 +775,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,void *header, size_t header_size) { -const void __user *ring =get_user_ring_header (ent, type); + bool use_bufring = ent->queue->use_bufring; +int err =0 ; -if (!ring) -return -EINVAL ; +if (use_bufring) { +struct iov_iter iter; -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; + err =get_kernel_ring_header (ent, type, &iter); +if (err) + goto done; + +if (copy_from_iter (header, header_size, &iter) != header_size) + err = -EFAULT ; + }else { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) { + err = -EINVAL ; + goto done; + } + +if (copy_from_user (header, ring, header_size)) + err = -EFAULT ; } -return 0 ; +done: +if (err) +pr_info_ratelimited ("Copying header from ring failed: " +"header_type=%u, header_size=%lu, " +"use_bufring=%d\n" , type, header_size, + use_bufring); + +return err; } static int setup_fuse_copy_state(struct fuse_copy_state *cs,
@@ -643,14 +816,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, {int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (!ent->queue->use_bufring) { + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer " +"failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter); +if (ent->queue->use_bufring) { + cs->is_kaddr = true; + cs->len = ent->payload_kvec.iov_len; + cs->kaddr = ent->payload_kvec.iov_base; + } + cs->is_uring = true; cs->req = req;
@@ -762,6 +944,108 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent, +unsigned int issue_flags) + __must_hold(&queue->lock) +{ +struct io_br_sel sel; +size_t len =0 ; + +lockdep_assert_held (&ent->queue->lock); + +/* Get a buffer to use for the payload */ + sel =io_ring_buffer_select (cmd_to_io_kiocb (ent->cmd), &len, + ent->queue->bufring, issue_flags); +if (sel.val) +return sel.val; +if (!sel.kaddr) +return -ENOENT ; + + ent->payload_kvec.iov_base = sel.kaddr; + ent->payload_kvec.iov_len = len; + ent->ringbuf_buf_id = sel.buf_id; + +return 0 ; +} + +static int fuse_uring_clean_up_buffer (struct fuse_ring_ent *ent, +struct io_uring_cmd *cmd) + __must_hold(&queue->lock) +{ +struct kvec *kvec = &ent->payload_kvec; +int err; + +lockdep_assert_held (&ent->queue->lock); + +if (!ent->queue->use_bufring) +return 0 ; + +if (kvec->iov_base) { + err =io_uring_kmbuf_recycle_pinned (cmd_to_io_kiocb (ent->cmd), + ent->queue->bufring, + (u64 )kvec->iov_base, + kvec->iov_len, + ent->ringbuf_buf_id); +if (WARN_ON_ONCE (err)) +return err; +memset (kvec,0 ,sizeof (*kvec)); + } + +return 0 ; +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req, +unsigned int issue_flags) +{ +bool buffer_selected; +bool has_payload; + +if (!ent->queue->use_bufring) +return 0 ; + + ent->headers_iter.data_source = false; + + buffer_selected = ent->payload_kvec.iov_base !=0 ; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent, issue_flags); + +if (!has_payload && buffer_selected) +fuse_uring_clean_up_buffer (ent, ent->cmd); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req,unsigned int dir, +unsigned issue_flags) +{ +if (!ent->queue->use_bufring) +return 0 ; + +if (dir ==ITER_SOURCE ) { + ent->headers_iter.data_source = true; +return 0 ; + } + + ent->headers_iter.data_source = false; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent, issue_flags); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {
@@ -824,7 +1108,8 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent, }/* Fetch the next fuse request if available */ -static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent) +static struct fuse_req *fuse_uring_ent_assign_req (struct fuse_ring_ent *ent, +unsigned int issue_flags) __must_hold(&queue->lock) {struct fuse_req *req;
@@ -835,8 +1120,13 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) +if (req) { +if (fuse_uring_next_req_update_buffer (ent, req, issue_flags)) +return NULL ;fuse_uring_add_req_to_ring_ent (ent, req); + }else { +fuse_uring_clean_up_buffer (ent, ent->cmd); + }return req; }
@@ -878,7 +1168,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,* Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +unsigned int issue_flags) {int err;struct fuse_req *req;
@@ -886,7 +1177,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent, retry:spin_lock (&queue->lock);fuse_uring_ent_avail (ent, queue); - req =fuse_uring_ent_assign_req (ent); + req =fuse_uring_ent_assign_req (ent, issue_flags);spin_unlock (&queue->lock);if (req) {
@@ -990,7 +1281,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); -fuse_uring_commit (ent, req, issue_flags); + + err =fuse_uring_prep_buffer (ent, req,ITER_SOURCE , issue_flags); +if (WARN_ON_ONCE (err)) +fuse_uring_req_end (ent, req, err); +else +fuse_uring_commit (ent, req, issue_flags);/* * Fetching the next request is absolutely required as queued
@@ -998,7 +1294,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) +if (fuse_uring_get_next_fuse_req (ent, queue, issue_flags))fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1094,39 +1390,64 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err; + err = -ENOMEM ; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (err); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + + err = -EINVAL ; +if (queue->use_bufring) { +size_t header_size =sizeof (struct fuse_uring_req_header ); +u16 buf_index; + +if (!(cmd->flags &IORING_URING_CMD_FIXED )) + goto error; + + buf_index =READ_ONCE (cmd->sqe->buf_index); + +/* set up the headers */ + ent->headers_iter = queue->headers_iter; +iov_iter_advance (&ent->headers_iter, buf_index * header_size); +iov_iter_truncate (&ent->headers_iter, header_size); +if (iov_iter_count (&ent->headers_iter) != header_size) + goto error; + +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->headers = iov[0 ].iov_base; ent->payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/*
@@ -1137,6 +1458,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, unsigned int issue_flags, struct fuse_conn *fc) { const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe_cmd(cmd->sqe); +bool use_bufring =READ_ONCE (cmd_req->init.use_bufring);struct fuse_ring *ring =smp_load_acquire (&fc->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;
@@ -1157,9 +1479,13 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, queue = ring->queues[qid];if (!queue) { - queue =fuse_uring_create_queue (ring, qid); -if (!queue) -return err; + queue =fuse_uring_create_queue (cmd, ring, qid, use_bufring, + issue_flags); +if (IS_ERR (queue)) +return PTR_ERR (queue); + }else { +if (queue->use_bufring != use_bufring) +return -EINVAL ; }/*
@@ -1263,7 +1589,8 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)if (!tw.cancel) { err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) +if (!fuse_uring_get_next_fuse_req (ent, queue, + issue_flags))return ; err =0 ; }
@@ -1325,14 +1652,20 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent =list_first_entry_or_null (&queue->ent_avail_queue, struct fuse_ring_ent, list); -if (ent) -fuse_uring_add_req_to_ring_ent (ent, req); -else -list_add_tail (&req->list, &queue->fuse_req_queue); -spin_unlock (&queue->lock); +if (ent) { + err =fuse_uring_prep_buffer (ent, req,ITER_DEST , +IO_URING_F_UNLOCKED ); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return ; + } +WARN_ON_ONCE (err != -ENOENT ); + } -if (ent) -fuse_uring_dispatch_ent (ent); +list_add_tail (&req->list, &queue->fuse_req_queue); +spin_unlock (&queue->lock);return ;
@@ -1350,6 +1683,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)struct fuse_ring *ring = fc->ring;struct fuse_ring_queue *queue;struct fuse_ring_ent *ent =NULL ; +int err; queue =fuse_uring_task_to_queue (ring);if (!queue)
@@ -1382,14 +1716,16 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list);if (ent && req) { -fuse_uring_add_req_to_ring_ent (ent, req); -spin_unlock (&queue->lock); - -fuse_uring_dispatch_ent (ent); - }else { -spin_unlock (&queue->lock); + err =fuse_uring_prep_buffer (ent, req,ITER_DEST , +IO_URING_F_UNLOCKED ); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return true; + } } - +spin_unlock (&queue->lock);return true; }
fs/fuse/dev_uring_i.h +32 -3
@@ -7,6 +7,8 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> +#include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +40,24 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +union { +/* queue->use_bufring == false */ +struct { +/* userspace buffers */ +struct fuse_uring_req_header __user *headers; +void __user *payload; + }; +/* queue->use_bufring == true */ +struct { +struct iov_iter headers_iter; +struct kvec payload_kvec; +/* +* This needs to be tracked in order to properly recycle +* the buffer when done with it +*/ +unsigned int ringbuf_buf_id; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
@@ -99,6 +116,18 @@ struct fuse_ring_queue {unsigned int active_background;bool stopped; + + bool ring_killed : 1; + +/* true if kernel-managed buffer ring is used */ + bool use_bufring: 1; + +/* the below fields are only used if the bufring is used */ +struct io_ring_ctx *ring_ctx; +/* iter for the headers buffer for all the ents */ +struct iov_iter headers_iter; +/* synchronized by the queue lock */ +struct io_buffer_list *bufring; };/**
include/uapi/linux/fuse.h +11 -1
@@ -240,6 +240,9 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add fuse_uring_cmd_req use_bufring */ #ifndef _LINUX_FUSE_H
@@ -1305,7 +1308,14 @@ struct fuse_uring_cmd_req {/* queue the command is for (queue index) */ uint16_t qid; - uint8_t padding[6]; + +union { +struct { +bool use_bufring; + } init; + }; + + uint8_t padding[5]; };#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 22/30 io_uring/rsrc: refactor io_buffer_register_bvec()/io_buffer_unregister_bvec()
In reply to Joanne KoongChanges:
- Rename io_buffer_register_bvec() to io_buffer_register_request()
- Rename io_buffer_unregister_bvec() to io_buffer_unregister()
- Add cmd wrappers for io_buffer_register_request() and io_buffer_unregister() for ublk to use
This is in preparation for supporting kernel-populated buffers in fuse io-uring, which will need to register bvecs directly (not through a block-based request) and will need to do unregistration through an io_ring_ctx directly.
--- Documentation/block/ublk.rst | 15 ++++++++------- drivers/block/ublk_drv.c | 20 +++++++++++--------- include/linux/io_uring/cmd.h | 13 ++++++++----- io_uring/rsrc.c | 14 +++++--------- io_uring/rsrc.h | 7 +++++++ io_uring/uring_cmd.c | 21 +++++++++++++++++++++ 6 files changed, 60 insertions(+), 30 deletions(-)
Documentation/block/ublk.rst +8 -7
@@ -326,16 +326,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_uring_cmd_buffer_register_request()` and +`io_uring_cmd_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request -buffer into io_uring buffer table, then ublk server can submit io_uring +`io_uring_cmd_buffer_register_request()` for ublk server to register client +request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the +calls `io_uring_cmd_buffer_unregister()` to unregister the buffer, which is +guaranteed to be live between calling `io_uring_cmd_buffer_register_request()` +and `io_uring_cmd_buffer_unregister()`. Any io_uring operation which supports +this kind of kernel buffer will grab one reference of the buffer until the operation is completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and
drivers/block/ublk_drv.c +11 -9
@@ -1246,8 +1246,9 @@ static bool ublk_auto_buf_reg(const struct ublk_queue *ubq, struct request *req, {int ret; - ret =io_buffer_register_bvec (io->cmd, req, ublk_io_release, - io->buf.index, issue_flags); + ret =io_uring_cmd_buffer_register_request (io->cmd, req, + ublk_io_release, + io->buf.index, issue_flags);if (ret) {if (io->buf.flags &UBLK_AUTO_BUF_REG_FALLBACK ) {ublk_auto_buf_reg_fallback (ubq, io);
@@ -2204,8 +2205,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,if (!req)return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_uring_cmd_buffer_register_request (cmd, req, ublk_io_release, + index, issue_flags);if (ret) {ublk_put_req_ref (io, req);return ret;
@@ -2236,8 +2237,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,if (!ublk_dev_support_zero_copy (ub) || !ublk_rq_has_data (req))return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_uring_cmd_buffer_register_request (cmd, req, ublk_io_release, + index, issue_flags);if (ret)return ret;
@@ -2252,7 +2253,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,if (!(ub->dev_info.flags &UBLK_F_SUPPORT_ZERO_COPY ))return -EINVAL ; -return io_buffer_unregister_bvec (cmd, index, issue_flags); +return io_uring_cmd_buffer_unregister (cmd, index, issue_flags); }static int ublk_check_fetch_buf (const struct ublk_device *ub,__u64 buf_addr)
@@ -2386,7 +2387,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out;/* -* io_buffer_unregister_bvec() doesn't access the ubq or io, +* io_uring_cmd_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR (cmd_op) ==UBLK_IO_UNREGISTER_IO_BUF )
@@ -2456,7 +2457,8 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,/* can't touch 'ublk_io' any more */ if (buf_idx !=UBLK_INVALID_BUF_IDX ) -io_buffer_unregister_bvec (cmd, buf_idx, issue_flags); +io_uring_cmd_buffer_unregister (cmd, buf_idx, + issue_flags);if (req_op (req) ==REQ_OP_ZONE_APPEND ) req->__sector = addr;if (compl)
include/linux/io_uring/cmd.h +8 -5
@@ -185,10 +185,13 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags); +int io_uring_cmd_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags); + +int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags);#endif /* _LINUX_IO_URING_CMD_H */
io_uring/rsrc.c +5 -9
@@ -941,11 +941,10 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;struct req_iterator rq_iter;struct io_mapped_ubuf *imu;
@@ -1003,12 +1002,10 @@ int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_register_bvec); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags) +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, +unsigned int issue_flags) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;struct io_rsrc_node *node;int ret =0 ;
@@ -1036,7 +1033,6 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_unregister_bvec);static int validate_fixed_range (u64 buf_addr,size_t len,const struct io_mapped_ubuf *imu)
io_uring/rsrc.h +7 -0
@@ -91,6 +91,13 @@ int io_validate_user_buf_range(u64 uaddr, u64 ulen);bool io_check_coalesce_buffer (struct page **page_array,int nr_pages,struct io_imu_folio_data *data); +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags); + +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, +unsigned int issue_flags); +static inline struct io_rsrc_node *io_rsrc_node_lookup (struct io_rsrc_data *data,int index) {
io_uring/uring_cmd.c +21 -0
@@ -383,6 +383,27 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_select); +int io_uring_cmd_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_register_request (ctx, rq, release, index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_register_request); + +int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_unregister (ctx, index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_unregister); +/* * Return true if this multishot uring_cmd needs to be completed, otherwise * the event CQE is posted successfully.
Signature · 1 line
2.47.3
Re: [PATCH v1 22/30] io_uring/rsrc: refactor io_buffer_register_bvec()/io_buffer_unregister_bvec()
Quoted 6 lines · Joanne Koong
Changes:
- Renameio_buffer_register_bvec()toio_buffer_register_request()
- Renameio_buffer_unregister_bvec()toio_buffer_unregister()
- Add cmd wrappers forio_buffer_register_request()andio_buffer_unregister()for ublk to use
I agree these names seem clearer.
Quoted 212 lines
This is in preparation for supporting kernel-populated buffers in fuse io-uring, which will need to register bvecs directly (not through a block-based request) and will need to do unregistration through an
io_ring_ctxdirectly.Signed-off-by Joanne Koong <[email protected]>--- Documentation/block/ublk.rst | 15 ++++++++------- drivers/block/ublk_drv.c | 20 +++++++++++--------- include/linux/io_uring/cmd.h | 13 ++++++++----- io_uring/rsrc.c | 14 +++++--------- io_uring/rsrc.h | 7 +++++++ io_uring/uring_cmd.c | 21 +++++++++++++++++++++ 6 files changed, 60 insertions(+), 30 deletions(-)Documentation/block/ublk.rst +8
-7@@ -326,16 +326,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_uring_cmd_buffer_register_request()` and +`io_uring_cmd_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request -buffer into io_uring buffer table, then ublk server can submit io_uring +`io_uring_cmd_buffer_register_request()` for ublk server to register client +request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the +calls `io_uring_cmd_buffer_unregister()` to unregister the buffer, which is +guaranteed to be live between calling `io_uring_cmd_buffer_register_request()` +and `io_uring_cmd_buffer_unregister()`. Any io_uring operation which supports +this kind of kernel buffer will grab one reference of the buffer until the operation is completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN anddrivers/block/ublk_drv.c +11
-9@@ -1246,8 +1246,9 @@ static bool ublk_auto_buf_reg(const struct ublk_queue *ubq, struct request *req, {int ret; - ret =io_buffer_register_bvec (io->cmd, req, ublk_io_release, - io->buf.index, issue_flags); + ret =io_uring_cmd_buffer_register_request (io->cmd, req, + ublk_io_release, + io->buf.index, issue_flags);if (ret) {if (io->buf.flags &UBLK_AUTO_BUF_REG_FALLBACK ) {ublk_auto_buf_reg_fallback (ubq, io);@@ -2204,8 +2205,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,if (!req)return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_uring_cmd_buffer_register_request (cmd, req, ublk_io_release, + index, issue_flags);if (ret) {ublk_put_req_ref (io, req);return ret;@@ -2236,8 +2237,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,if (!ublk_dev_support_zero_copy (ub) || !ublk_rq_has_data (req))return -EINVAL ; - ret =io_buffer_register_bvec (cmd, req, ublk_io_release, index, - issue_flags); + ret =io_uring_cmd_buffer_register_request (cmd, req, ublk_io_release, + index, issue_flags);if (ret)return ret;@@ -2252,7 +2253,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,if (!(ub->dev_info.flags &UBLK_F_SUPPORT_ZERO_COPY ))return -EINVAL ; -return io_buffer_unregister_bvec (cmd, index, issue_flags); +return io_uring_cmd_buffer_unregister (cmd, index, issue_flags); }static int ublk_check_fetch_buf (const struct ublk_device *ub,__u64 buf_addr)@@ -2386,7 +2387,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out;/* -* io_buffer_unregister_bvec() doesn't access the ubq or io, +* io_uring_cmd_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR (cmd_op) ==UBLK_IO_UNREGISTER_IO_BUF )@@ -2456,7 +2457,8 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,/* can't touch 'ublk_io' any more */ if (buf_idx !=UBLK_INVALID_BUF_IDX ) -io_buffer_unregister_bvec (cmd, buf_idx, issue_flags); +io_uring_cmd_buffer_unregister (cmd, buf_idx, + issue_flags);if (req_op (req) ==REQ_OP_ZONE_APPEND ) req->__sector = addr;if (compl)include/linux/io_uring/cmd.h +8
-5@@ -185,10 +185,13 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags); +int io_uring_cmd_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags); + +int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags);#endif /* _LINUX_IO_URING_CMD_H */ io_uring/rsrc.c +5
-9@@ -941,11 +941,10 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_bvec (struct io_uring_cmd *cmd,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;struct req_iterator rq_iter;struct io_mapped_ubuf *imu;@@ -1003,12 +1002,10 @@ int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_register_bvec); -int io_buffer_unregister_bvec (struct io_uring_cmd *cmd,unsigned int index, -unsigned int issue_flags) +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, +unsigned int issue_flags) { -struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx;struct io_rsrc_data *data = &ctx->buf_table;struct io_rsrc_node *node;int ret =0 ;@@ -1036,7 +1033,6 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,io_ring_submit_unlock (ctx, issue_flags);return ret; } -EXPORT_SYMBOL_GPL (io_buffer_unregister_bvec);static int validate_fixed_range (u64 buf_addr,size_t len,const struct io_mapped_ubuf *imu)io_uring/rsrc.h +7
-0@@ -91,6 +91,13 @@ int io_validate_user_buf_range(u64 uaddr, u64 ulen);bool io_check_coalesce_buffer (struct page **page_array,int nr_pages,struct io_imu_folio_data *data); +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags); + +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, +unsigned int issue_flags); +static inline struct io_rsrc_node *io_rsrc_node_lookup (struct io_rsrc_data *data,int index) {io_uring/uring_cmd.c +10
-0@@ -383,6 +383,27 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, }EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_select); +int io_uring_cmd_buffer_register_request (struct io_uring_cmd *cmd, +struct request *rq, +void (*release)(void *), +unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_register_request (ctx, rq, release, index, issue_flags); +}
io_uring/uring_cmd.c struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, · line 405
Quoted 10 lines
io_uring/uring_cmd.c +10
-0+EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_register_request); + +int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_unregister (ctx, index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_unregister);
It would be nice to avoid these additional function calls that can't be inlined. I guess we probably don't want to include the io_uring-internal header io_uring/rsrc.h in the external header linux/io_uring/cmd.h, which is probably why the functions were declared in linux/io_uring/cmd.h but defined in io_uring/rsrc.c previously. Maybe it would make sense to move the definitions of io_uring_cmd_buffer_register_request() and io_uring_cmd_buffer_unregister() to io_uring/rsrc.c so io_buffer_register_request()/io_buffer_unregister() can be inlined into them?
Best,
Caleb
Quoted 7 lines
io_uring/uring_cmd.c +1
-1+/* * Return true if this multishot uring_cmd needs to be completed, otherwise * the event CQE is posted successfully. -- 2.47.3
Joanne Koong
maintainerRe: [PATCH v1 22/30] io_uring/rsrc: refactor io_buffer_register_bvec()/io_buffer_unregister_bvec()
Quoted 14 lines · Caleb Sander Mateos
io_uring/uring_cmd.c struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, · line 405
Quoted 9 lines · Joanne Koong
io_uring/uring_cmd.c +8
-0+int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_unregister (ctx, index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_unregister);It would be nice to avoid these additional function calls that can't be inlined. I guess we probably don't want to include the
Quoted 8 lines
io_uring-internal header
io_uring/rsrc.hin the external headerlinux/io_uring/cmd.h, which is probably why the functions were declared inlinux/io_uring/cmd.hbut defined inio_uring/rsrc.cpreviously. Maybe it would make sense to move the definitions ofio_uring_cmd_buffer_register_request()andio_uring_cmd_buffer_unregister()toio_uring/rsrc.cso io_buffer_register_request()/io_buffer_unregister() can be inlined into them?
imo I think having the code organized more logically outweighs the minimal improvement we would get from inlining (especially as io_buffer_register_request() is not a small function), but I'm happy to make this change if you feel strongly about it.
Thanks,
Joanne
Quoted 11 lines
Best,
CalebQuoted 7 lines
+
/*
* Return true if this multishoturing_cmdneeds to be completed, otherwise
* the event CQE is posted successfully.
--
2.47.3
Re: [PATCH v1 22/30] io_uring/rsrc: refactor io_buffer_register_bvec()/io_buffer_unregister_bvec()
Quoted 26 lines · Joanne Koong
Quoted 14 lines · Caleb Sander Mateos
io_uring/uring_cmd.c struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, · line 405
Quoted 9 lines · Joanne Koong
io_uring/uring_cmd.c +8
-0+int io_uring_cmd_buffer_unregister (struct io_uring_cmd *cmd,unsigned int index, +unsigned int issue_flags) +{ +struct io_ring_ctx *ctx =cmd_to_io_kiocb (cmd)->ctx; + +return io_buffer_unregister (ctx, index, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_buffer_unregister);It would be nice to avoid these additional function calls that can't be inlined. I guess we probably don't want to include the
Quoted 8 lines
io_uring-internal header
io_uring/rsrc.hin the external headerlinux/io_uring/cmd.h, which is probably why the functions were declared inlinux/io_uring/cmd.hbut defined inio_uring/rsrc.cpreviously. Maybe it would make sense to move the definitions ofio_uring_cmd_buffer_register_request()andio_uring_cmd_buffer_unregister()toio_uring/rsrc.cso io_buffer_register_request()/io_buffer_unregister() can be inlined into them?
Quoted 4 lines
imo I think having the code organized more logically outweighs the minimal improvement we would get from inlining (especially as
io_buffer_register_request()is not a small function), but I'm happy to make this change if you feel strongly about it.
Yes, io_buffer_register_request() is a large function, but this io_uring_cmd_buffer_unregister() wrapper is small. It would be nice to either inline it into its caller or have io_buffer_register_request() inlined into it. I don't feel that strongly, but this is in the ublk zero-copy I/O path. You make a good point that this organization is a cleaner abstraction. And I guess there are already plenty of thin wrapper functions in io_uring/uring_cmd.c. Let's hope LTO comes to the rescue!
Thanks,
Caleb
Quoted 15 lines
Thanks,
JoanneQuoted 11 lines
Best,
CalebQuoted 7 lines
+
/*
* Return true if this multishoturing_cmdneeds to be completed, otherwise
* the event CQE is posted successfully.
--
2.47.3
Joanne Koong
maintainerPATCH 23/30 io_uring/rsrc: split io_buffer_register_request() logic
In reply to Joanne KoongSplit the main initialization logic in io_buffer_register_request() into a helper function.
This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
--- io_uring/rsrc.c | 80 +++++++++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 32 deletions(-)
io_uring/rsrc.c +48 -32
@@ -941,63 +941,79 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static int io_buffer_init (struct io_ring_ctx *ctx,unsigned int nr_bvecs, +unsigned int total_bytes,u8 dir, +void (*release)(void *),void *priv, +unsigned int index) {struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return -EINVAL ; index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return -EBUSY ; node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return -ENOMEM ; -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return -ENOMEM ; } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + + node->buf = imu; + data->nodes[index] = node; + +return 0 ; +} + +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes; +int ret; + +io_ring_submit_lock (ctx, issue_flags); + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq); + ret =io_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), release, rq, + index); +if (ret) + goto unlock; + imu = ctx->buf_table.nodes[index]->buf; + nr_bvecs =0 ;rq_for_each_bvec (bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; unlock:io_ring_submit_unlock (ctx, issue_flags);return ret;
Signature · 1 line
2.47.3
Re: [PATCH v1 23/30] io_uring/rsrc: split io_buffer_register_request() logic
Quoted 17 lines · Joanne Koong
Split the main initialization logic in
io_buffer_register_request()into a helper function.This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
Signed-off-by Joanne Koong <[email protected]>--- io_uring/rsrc.c | 80 +++++++++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 32 deletions(-)io_uring/rsrc.c +0
-0
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 944
Quoted 7 lines
io_uring/rsrc.c +1
-3@@ -941,63 +941,79 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static int io_buffer_init(struct io_ring_ctx *ctx, unsigned int nr_bvecs,
Consider adding "kernel" somewhere in the name to distinguish this from the userspace registered buffer initialization
Quoted 78 lines
io_uring/rsrc.c +31
-27+ unsigned int total_bytes, u8 dir, + void (*release)(void *), void *priv, + unsigned int index) { struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return -EINVAL ; index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return -EBUSY ; node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return -ENOMEM ; -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return -ENOMEM ; } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + + node->buf = imu; + data->nodes[index] = node; + +return 0 ; +} + +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes;
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1005
Quoted 10 lines
io_uring/rsrc.c +10
-0+int ret; + +io_ring_submit_lock (ctx, issue_flags); + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq);
These could be initialized before io_ring_submit_lock()
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1011
Quoted 6 lines
io_uring/rsrc.c +5
-0+ ret =io_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), release, rq, + index); +if (ret) + goto unlock; + imu = ctx->buf_table.nodes[index]->buf;
It would be nice to avoid all these additional dereferences. Could io_buffer_init() return the struct io_mapped_ubuf *, using ERR_PTR() to return any error code?
Best,
Caleb
Quoted 13 lines
io_uring/rsrc.c +1
-3+ nr_bvecs =0 ;rq_for_each_bvec (bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; unlock:io_ring_submit_unlock (ctx, issue_flags);return ret; --2.47.3
Joanne Koong
maintainerRe: [PATCH v1 23/30] io_uring/rsrc: split io_buffer_register_request() logic
Quoted 22 lines · Caleb Sander Mateos
Quoted 20 lines · Joanne Koong
Split the main initialization logic in
io_buffer_register_request()into a helper function.This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
Signed-off-by Joanne Koong <[email protected]>--- io_uring/rsrc.c | 80 +++++++++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 32 deletions(-)io_uring/rsrc.c +0
-0@@ -941,63 +941,79 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; }
Quoted 7 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 944
Quoted 4 lines
io_uring/rsrc.c +1
-3-int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static int io_buffer_init(struct io_ring_ctx *ctx, unsigned int nr_bvecs,Consider adding "kernel" somewhere in the name to distinguish this from the userspace registered buffer initialization
I can rename this to io_kernel_buffer_init().
Quoted 97 lines
Quoted 79 lines
io_uring/rsrc.c +31
-27+ unsigned int total_bytes, u8 dir, + void (*release)(void *), void *priv, + unsigned int index) { struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return -EINVAL ; index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return -EBUSY ; node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return -ENOMEM ; -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return -ENOMEM ; } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + + node->buf = imu; + data->nodes[index] = node; + +return 0 ; +} + +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes;io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1005
Quoted 10 lines
io_uring/rsrc.c +10
-0+int ret; + +io_ring_submit_lock (ctx, issue_flags); + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq);These could be initialized before
io_ring_submit_lock()Quoted 5 lines
io_uring/rsrc.c +4
-0+ ret =io_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), release, rq, + index); +if (ret) + goto unlock;
Quoted 5 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1011
Quoted 1 line
io_uring/rsrc.c +1
-0+ imu = ctx->buf_table.nodes[index]->buf;It would be nice to avoid all these additional dereferences. Could
io_buffer_init()return the structio_mapped_ubuf*, usingERR_PTR()to return any error code?
I'll make this change for v2.
Thanks,
Joanne
Quoted 3 lines
Best,
Caleb
Re: [PATCH v1 23/30] io_uring/rsrc: split io_buffer_register_request() logic
Quoted 33 lines · Joanne Koong
Quoted 22 lines · Caleb Sander Mateos
Quoted 20 lines · Joanne Koong
Split the main initialization logic in
io_buffer_register_request()into a helper function.This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.
Signed-off-by Joanne Koong <[email protected]>--- io_uring/rsrc.c | 80 +++++++++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 32 deletions(-)io_uring/rsrc.c +0
-0@@ -941,63 +941,79 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,return ret; }Quoted 7 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 944
Quoted 4 lines
io_uring/rsrc.c +1
-3-int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, -void (*release)(void *),unsigned int index, -unsigned int issue_flags) +static int io_buffer_init(struct io_ring_ctx *ctx, unsigned int nr_bvecs,Consider adding "kernel" somewhere in the name to distinguish this from the userspace registered buffer initialization
Quoted 1 line
I can rename this to
io_kernel_buffer_init().
Sounds good.
Thanks,
Caleb
Quoted 111 lines
Quoted 98 lines
Quoted 80 lines
io_uring/rsrc.c +31
-27+ unsigned int total_bytes, u8 dir, + void (*release)(void *), void *priv, + unsigned int index) { struct io_rsrc_data *data = &ctx->buf_table; -struct req_iterator rq_iter;struct io_mapped_ubuf *imu;struct io_rsrc_node *node; -struct bio_vec bv; -unsigned int nr_bvecs =0 ; -int ret =0 ; -io_ring_submit_lock (ctx, issue_flags); -if (index >= data->nr) { - ret = -EINVAL ; - goto unlock; - } +if (index >= data->nr) +return -EINVAL ; index =array_index_nospec (index, data->nr); -if (data->nodes[index]) { - ret = -EBUSY ; - goto unlock; - } +if (data->nodes[index]) +return -EBUSY ; node =io_rsrc_node_alloc (ctx,IORING_RSRC_BUFFER ); -if (!node) { - ret = -ENOMEM ; - goto unlock; - } +if (!node) +return -ENOMEM ; -/* -* blk_rq_nr_phys_segments() may overestimate the number of bvecs -* but avoids needing to iterate over the bvecs -*/ - imu =io_alloc_imu (ctx,blk_rq_nr_phys_segments (rq)); + imu =io_alloc_imu (ctx, nr_bvecs);if (!imu) {kfree (node); - ret = -ENOMEM ; - goto unlock; +return -ENOMEM ; } imu->ubuf =0 ; - imu->len =blk_rq_bytes (rq); + imu->len = total_bytes; imu->acct_pages =0 ; imu->folio_shift =PAGE_SHIFT ; + imu->nr_bvecs = nr_bvecs;refcount_set (&imu->refs,1 ); imu->release = release; - imu->priv = rq; + imu->priv = priv; imu->is_kbuf = true; - imu->dir =1 <<rq_data_dir (rq); + imu->dir =1 << dir; + + node->buf = imu; + data->nodes[index] = node; + +return 0 ; +} + +int io_buffer_register_request (struct io_ring_ctx *ctx,struct request *rq, +void (*release)(void *),unsigned int index, +unsigned int issue_flags) +{ +struct req_iterator rq_iter; +struct io_mapped_ubuf *imu; +struct bio_vec bv; +unsigned int nr_bvecs; +unsigned int total_bytes;io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1005
Quoted 10 lines
io_uring/rsrc.c +10
-0+int ret; + +io_ring_submit_lock (ctx, issue_flags); + +/* +* blk_rq_nr_phys_segments() may overestimate the number of bvecs +* but avoids needing to iterate over the bvecs +*/ + nr_bvecs =blk_rq_nr_phys_segments (rq); + total_bytes =blk_rq_bytes (rq);These could be initialized before
io_ring_submit_lock()Quoted 5 lines
io_uring/rsrc.c +4
-0+ ret =io_buffer_init (ctx, nr_bvecs, total_bytes,rq_data_dir (rq), release, rq, + index); +if (ret) + goto unlock;Quoted 5 lines
io_uring/rsrc.c int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, · line 1011
Quoted 1 line
io_uring/rsrc.c +1
-0+ imu = ctx->buf_table.nodes[index]->buf;It would be nice to avoid all these additional dereferences. Could
io_buffer_init()return the structio_mapped_ubuf*, usingERR_PTR()to return any error code?I'll make this change for v2.
Thanks,
JoanneQuoted 3 lines
Best,
Caleb
Joanne Koong
maintainerPATCH 24/30 io_uring/rsrc: Allow buffer release callback to be optional
In reply to Joanne KoongThis is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which does not need a release callback.
--- io_uring/rsrc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
io_uring/rsrc.c +2 -1
@@ -149,7 +149,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)if (imu->acct_pages)io_unaccount_mem (ctx->user, ctx->mm_account, imu->acct_pages); - imu->release (imu->priv); +if (imu->release) + imu->release (imu->priv);io_free_imu (ctx, imu); }
Signature · 1 line
2.47.3
Re: [PATCH v1 24/30] io_uring/rsrc: Allow buffer release callback to be optional
Quoted 5 lines · Joanne Koong
This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which does not need a release callback.
Signed-off-by Joanne Koong <[email protected]>
Quoted 21 lines
--- io_uring/rsrc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)io_uring/rsrc.c +2
-2@@ -149,7 +149,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)if (imu->acct_pages)io_unaccount_mem (ctx->user, ctx->mm_account, imu->acct_pages); - imu->release (imu->priv); +if (imu->release) + imu->release (imu->priv);io_free_imu (ctx, imu); } --2.47.3
Joanne Koong
maintainerPATCH 25/30 io_uring/rsrc: add io_buffer_register_bvec()
In reply to Joanne KoongAdd io_buffer_register_bvec() for registering a bvec array.
This is a preparatory patch for fuse-over-io-uring zero-copy.
--- include/linux/io_uring/buf.h | 14 ++++++++++++++ io_uring/rsrc.c | 25 +++++++++++++++++++++++++ 2 files changed, 39 insertions(+)
include/linux/io_uring/buf.h +14 -0
@@ -23,6 +23,11 @@ bool io_uring_is_kmbuf_ring(struct io_ring_ctx *ctx, unsigned int buf_group,struct io_br_sel io_ring_buffer_select (struct io_kiocb *req,size_t *len,struct io_buffer_list *bl,unsigned int issue_flags); + +int io_buffer_register_bvec (struct io_ring_ctx *ctx,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index, +unsigned int issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -70,6 +75,15 @@ static inline struct io_br_sel io_ring_buffer_select(struct io_kiocb *req,return sel; } +static inline int io_buffer_register_bvec (struct io_ring_ctx *ctx, +struct bio_vec *bvs, +unsigned int nr_bvecs, +unsigned int total_bytes,u8 dir, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
io_uring/rsrc.c +25 -0
@@ -1020,6 +1020,31 @@ int io_buffer_register_request(struct io_ring_ctx *ctx, struct request *rq,return ret; } +int io_buffer_register_bvec (struct io_ring_ctx *ctx,struct bio_vec *bvs, +unsigned int nr_bvecs,unsigned int total_bytes, +u8 dir,unsigned int index, +unsigned int issue_flags) +{ +struct io_rsrc_data *data = &ctx->buf_table; +struct bio_vec *bvec; +int ret, i; + +io_ring_submit_lock (ctx, issue_flags); + ret =io_buffer_init (ctx, nr_bvecs, total_bytes, dir,NULL ,NULL , + index); +if (ret) + goto unlock; + + bvec = data->nodes[index]->buf->bvec; +for (i =0 ; i < nr_bvecs; i++) + bvec[i] = bvs[i]; + +unlock: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +} +EXPORT_SYMBOL_GPL (io_buffer_register_bvec); +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index,unsigned int issue_flags) {
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 26/30 io_uring/rsrc: export io_buffer_unregister
In reply to Joanne KoongThis is a preparatory patch for fuse-over-io-uring zero-copy.
--- include/linux/io_uring/buf.h | 9 +++++++++ io_uring/rsrc.c | 1 + io_uring/rsrc.h | 3 --- io_uring/uring_cmd.c | 1 + 4 files changed, 11 insertions(+), 3 deletions(-)
include/linux/io_uring/buf.h +9 -0
@@ -28,6 +28,8 @@ int io_buffer_register_bvec(struct io_ring_ctx *ctx, struct bio_vec *bvs, unsigned int nr_bvecs, unsigned int total_bytes, u8 dir, unsigned int index, unsigned int issue_flags); +int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, +unsigned int issue_flags);#else static inline int io_uring_buf_ring_pin(struct io_ring_ctx *ctx, unsigned buf_group,
@@ -84,6 +86,13 @@ static inline int io_buffer_register_bvec(struct io_ring_ctx *ctx, {return -EOPNOTSUPP ; } + +static inline int io_buffer_unregister (struct io_ring_ctx *ctx, +unsigned int index, +unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +}#endif /* CONFIG_IO_URING */ #endif /* _LINUX_IO_URING_BUF_H */
io_uring/rsrc.c +1 -0
@@ -1075,6 +1075,7 @@ int io_buffer_unregister(struct io_ring_ctx *ctx, unsigned int index,io_ring_submit_unlock (ctx, issue_flags);return ret; } +EXPORT_SYMBOL_GPL (io_buffer_unregister);static int validate_fixed_range (u64 buf_addr,size_t len,const struct io_mapped_ubuf *imu)
io_uring/rsrc.h +0 -3
@@ -95,9 +95,6 @@ int io_buffer_register_request(struct io_ring_ctx *ctx, struct request *rq, void (*release)(void *), unsigned int index, unsigned int issue_flags); -int io_buffer_unregister (struct io_ring_ctx *ctx,unsigned int index, -unsigned int issue_flags); -static inline struct io_rsrc_node *io_rsrc_node_lookup (struct io_rsrc_data *data,int index) {
io_uring/uring_cmd.c +1 -0
@@ -2,6 +2,7 @@#include <linux/kernel.h> #include <linux/errno.h> #include <linux/file.h> +#include <linux/io_uring/buf.h> #include <linux/io_uring/cmd.h> #include <linux/security.h> #include <linux/nospec.h>
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 27/30 fuse: rename fuse_set_zero_arg0() to fuse_zero_in_arg0()
In reply to Joanne KoongThe fuse_set_zero_arg0() function is used for setting a no-op header for in args but to support fuse io-uring zero-copy, the first parameter of outargs will also need to be set to a no-op header if the request contains payload but no out heeader.
Rename fuse_set_zero_arg0() to fuse_zero_in_arg0() to indicate this is for the in arg header. Later, fuse_zero_out_arg0() will be added.
--- fs/fuse/dax.c | 2 +- fs/fuse/dev.c | 2 +- fs/fuse/dir.c | 8 ++++---- fs/fuse/fuse_i.h | 2 +- fs/fuse/xattr.c | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-)
fs/fuse/dax.c +1 -1
@@ -240,7 +240,7 @@ static int fuse_send_removemapping(struct inode *inode, args.opcode =FUSE_REMOVEMAPPING ; args.nodeid = fi->nodeid; args.in_numargs =3 ; -fuse_set_zero_arg0 (&args); +fuse_zero_in_arg0 (&args); args.in_args[1 ].size =sizeof (*inargp); args.in_args[1 ].value = inargp; args.in_args[2 ].size = inargp->count *sizeof (*remove_one);
fs/fuse/dev.c +1 -1
@@ -1943,7 +1943,7 @@ static int fuse_retrieve(struct fuse_mount *fm, struct inode *inode, } ra->inarg.offset = outarg->offset; ra->inarg.size = total_len; -fuse_set_zero_arg0 (args); +fuse_zero_in_arg0 (args); args->in_args[1 ].size =sizeof (ra->inarg); args->in_args[1 ].value = &ra->inarg; args->in_args[2 ].size = total_len;
fs/fuse/dir.c +4 -4
@@ -176,7 +176,7 @@ static void fuse_lookup_init(struct fuse_conn *fc, struct fuse_args *args, args->opcode =FUSE_LOOKUP ; args->nodeid = nodeid; args->in_numargs =3 ; -fuse_set_zero_arg0 (args); +fuse_zero_in_arg0 (args); args->in_args[1 ].size = name->len; args->in_args[1 ].value = name->name; args->in_args[2 ].size =1 ;
@@ -943,7 +943,7 @@ static int fuse_symlink(struct mnt_idmap *idmap, struct inode *dir, args.opcode =FUSE_SYMLINK ; args.in_numargs =3 ; -fuse_set_zero_arg0 (&args); +fuse_zero_in_arg0 (&args); args.in_args[1 ].size = entry->d_name.len +1 ; args.in_args[1 ].value = entry->d_name.name; args.in_args[2 ].size = len;
@@ -1008,7 +1008,7 @@ static int fuse_unlink(struct inode *dir, struct dentry *entry) args.opcode =FUSE_UNLINK ; args.nodeid =get_node_id (dir); args.in_numargs =2 ; -fuse_set_zero_arg0 (&args); +fuse_zero_in_arg0 (&args); args.in_args[1 ].size = entry->d_name.len +1 ; args.in_args[1 ].value = entry->d_name.name; err =fuse_simple_request (fm, &args);
@@ -1032,7 +1032,7 @@ static int fuse_rmdir(struct inode *dir, struct dentry *entry) args.opcode =FUSE_RMDIR ; args.nodeid =get_node_id (dir); args.in_numargs =2 ; -fuse_set_zero_arg0 (&args); +fuse_zero_in_arg0 (&args); args.in_args[1 ].size = entry->d_name.len +1 ; args.in_args[1 ].value = entry->d_name.name; err =fuse_simple_request (fm, &args);
fs/fuse/fuse_i.h +1 -1
@@ -1020,7 +1020,7 @@ struct fuse_mount {*/ struct fuse_zero_header {}; -static inline void fuse_set_zero_arg0 (struct fuse_args *args) +static inline void fuse_zero_in_arg0 (struct fuse_args *args) { args->in_args[0 ].size =sizeof (struct fuse_zero_header ); args->in_args[0 ].value =NULL ;
fs/fuse/xattr.c +1 -1
@@ -165,7 +165,7 @@ int fuse_removexattr(struct inode *inode, const char *name) args.opcode =FUSE_REMOVEXATTR ; args.nodeid =get_node_id (inode); args.in_numargs =2 ; -fuse_set_zero_arg0 (&args); +fuse_zero_in_arg0 (&args); args.in_args[1 ].size =strlen (name) +1 ; args.in_args[1 ].value = name; err =fuse_simple_request (fm, &args);
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 28/30 fuse: enforce op header for every payload reply
In reply to Joanne KoongIn order to support fuse io-uring zero-copy, the payload and the headers for a request/reply must reside in separate buffers since any zero-copied payload will be transparent to the daemon but the headers need to be accessible.
Currently, a fuse reply can be either:
* arg[0] = op header, arg[1] = payload
* arg[0] = payload
* arg[0] = NULL
Fuse io-uring needs to differentiate between the first two for copying to/from the ring.
Enforce that all fuse replies that have a payload also have an op header. If there is is only a payload to send in the reply, then the header will be a zero-size no-op header.
--- fs/fuse/dir.c | 5 +++-- fs/fuse/file.c | 11 ++++++----- fs/fuse/fuse_i.h | 6 ++++++ fs/fuse/readdir.c | 2 +- fs/fuse/xattr.c | 16 ++++++++++------ 5 files changed, 26 insertions(+), 14 deletions(-)
fs/fuse/dir.c +3 -2
@@ -1630,8 +1630,9 @@ static int fuse_readlink_folio(struct inode *inode, struct folio *folio) ap.args.out_pages = true; ap.args.out_argvar = true; ap.args.page_zeroing = true; - ap.args.out_numargs =1 ; - ap.args.out_args[0 ].size = desc.length; + ap.args.out_numargs =2 ; +fuse_zero_out_arg0 (&ap.args); + ap.args.out_args[1 ].size = desc.length; res =fuse_simple_request (fm, &ap.args);fuse_invalidate_atime (inode);
fs/fuse/file.c +6 -5
@@ -581,8 +581,9 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos, args->in_args[0 ].size =sizeof (ia->read.in); args->in_args[0 ].value = &ia->read.in; args->out_argvar = true; - args->out_numargs =1 ; - args->out_args[0 ].size = count; + args->out_numargs =2 ; +fuse_zero_out_arg0 (args); + args->out_args[1 ].size = count; } static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
@@ -711,7 +712,7 @@ static void fuse_aio_complete_req(struct fuse_mount *fm, struct fuse_args *args, ia->write.out.size; } }else { -u32 outsize = args->out_args[0 ].size; +u32 outsize = args->out_args[1 ].size; nres = outsize;if (ia->read.in.size != outsize)
@@ -870,7 +871,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,struct fuse_io_args *ia =container_of (args,typeof (*ia), ap.args);struct fuse_args_pages *ap = &ia->ap;size_t count = ia->read.in.size; -size_t num_read = args->out_args[0 ].size; +size_t num_read = args->out_args[1 ].size;struct address_space *mapping;struct inode *inode;
@@ -1506,7 +1507,7 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii,if (write) ap->args.in_args[1 ].value = user_addr;else - ap->args.out_args[0 ].value = user_addr; + ap->args.out_args[1 ].value = user_addr;iov_iter_advance (ii, frag_size); *nbytesp = frag_size;
fs/fuse/fuse_i.h +6 -0
@@ -1026,6 +1026,12 @@ static inline void fuse_zero_in_arg0(struct fuse_args *args) args->in_args[0 ].value =NULL ; } +static inline void fuse_zero_out_arg0 (struct fuse_args *args) +{ + args->out_args[0 ].size =sizeof (struct fuse_zero_header ); + args->out_args[0 ].value =NULL ; +} +static inline struct fuse_mount *get_fuse_mount_super (struct super_block *sb) {return sb->s_fs_info;
fs/fuse/readdir.c +1 -1
@@ -349,7 +349,7 @@ static int fuse_readdir_uncached(struct file *file, struct dir_context *ctx)if (!buf)return -ENOMEM ; - args->out_args[0 ].value = buf; + args->out_args[1 ].value = buf; plus =fuse_use_readdirplus (inode, ctx);if (plus) {
fs/fuse/xattr.c +10 -6
@@ -70,12 +70,14 @@ ssize_t fuse_getxattr(struct inode *inode, const char *name, void *value, args.in_args[1 ].size =strlen (name) +1 ; args.in_args[1 ].value = name;/* This is really two different operations rolled into one */ - args.out_numargs =1 ;if (size) { args.out_argvar = true; - args.out_args[0 ].size = size; - args.out_args[0 ].value = value; + args.out_numargs =2 ; +fuse_zero_out_arg0 (&args); + args.out_args[1 ].size = size; + args.out_args[1 ].value = value; }else { + args.out_numargs =1 ; args.out_args[0 ].size =sizeof (outarg); args.out_args[0 ].value = &outarg; }
@@ -132,12 +134,14 @@ ssize_t fuse_listxattr(struct dentry *entry, char *list, size_t size) args.in_args[0 ].size =sizeof (inarg); args.in_args[0 ].value = &inarg;/* This is really two different operations rolled into one */ - args.out_numargs =1 ;if (size) { args.out_argvar = true; - args.out_args[0 ].size = size; - args.out_args[0 ].value = list; + args.out_numargs =2 ; +fuse_zero_out_arg0 (&args); + args.out_args[1 ].size = size; + args.out_args[1 ].value = list; }else { + args.out_numargs =1 ; args.out_args[0 ].size =sizeof (outarg); args.out_args[0 ].value = &outarg; }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 29/30 fuse: add zero-copy over io-uring
In reply to Joanne KoongImplement zero-copy data transfer for fuse over io-uring, eliminating memory copies between kernel and userspace for read/write operations.
This is only allowed on privileged servers and requires the server to preregister the following:
a) a sparse buffer corresponding to the queue depth
b) a fixed buffer at index queue_depth (the tail of the buffers)
c) a kernel-managed buffer ring
The sparse buffer is where the client's pages reside. The fixed buffer at the tail is where the headers (struct fuse_uring_req_header) are placed. The kernel-managed buffer ring is where any non-zero-copied args reside (eg out headers).
Benchmarks with bs=1M showed approximately the following differences in throughput:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s)
buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)
The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 191 ++++++++++++++++++++++++++++++++------ fs/fuse/dev_uring_i.h | 12 +++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 5 +- 5 files changed, 187 insertions(+), 29 deletions(-)
fs/fuse/dev.c +5 -2
@@ -1229,8 +1229,11 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) - err =fuse_copy_folios (cs, arg->size, zeroing); +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; +return fuse_copy_folios (cs, arg->size, zeroing); + }else err =fuse_copy_one (cs, arg->value, arg->size); }
fs/fuse/dev_uring.c +165 -26
370 lines not on this page · raw message
fs/fuse/dev_uring_i.h +12 -0
@@ -56,6 +56,11 @@ struct fuse_ring_ent {* the buffer when done with it */ unsigned int ringbuf_buf_id; + +/* True if the request's pages are being zero-copied */ +bool zero_copied; +/* Buf id for this ent's zero-copied pages */ +unsigned int zero_copy_buf_id; }; };
@@ -128,6 +133,13 @@ struct fuse_ring_queue {struct iov_iter headers_iter;/* synchronized by the queue lock */ struct io_buffer_list *bufring; +/* +* True if zero copy should be used for payloads. This is only enabled +* on privileged servers. Kernel-managed ring buffers must be enabled +* in order to use zero copy. +*/ + bool use_zero_copy : 1; +unsigned int depth; };/**
fs/fuse/fuse_dev_i.h +1 -0
@@ -39,6 +39,7 @@ struct fuse_copy_state { bool is_uring:1;/* if set, use kaddr; otherwise use pg */ bool is_kaddr:1; + bool skip_folio_copy:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;
include/uapi/linux/fuse.h +4 -1
@@ -243,6 +243,7 @@* * 7.46 * - add fuse_uring_cmd_req use_bufring +* - add fuse_uring_cmd_req zero_copy and queue_depth */ #ifndef _LINUX_FUSE_H
@@ -1312,10 +1313,12 @@ struct fuse_uring_cmd_req {union {struct {bool use_bufring; +bool zero_copy; +uint16_t queue_depth; } init; }; - uint8_t padding[5]; + uint8_t padding[2]; };#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH 30/30 docs: fuse: add io-uring bufring and zero-copy documentation
In reply to Joanne KoongAdd documentation for fuse over io-uring usage of kernel-managed bufrings and zero-copy.
--- .../filesystems/fuse/fuse-io-uring.rst | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-)
Documentation/filesystems/fuse/fuse-io-uring.rst +54 -1
@@ -95,5 +95,58 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | +Kernel-managed buffer rings +=========================== - +Kernel-managed buffer rings have two main advantages: +* eliminates the overhead of pinning/unpinning user pages and translating + virtual addresses for evey server-kernel interaction +* reduces buffer memory allocation requirements + +In order to use buffer rings, the server must preregister the following: +* a fixed buffer at index 0. This is where the headers will reside +* a kernel-managed buffer ring. This is where the payload will reside + +At a high-level, this is how fuse uses buffer rings: +* The server registers a kernel-managed buffer ring. In the kernel this + allocates the pages needed for the buffers and vmaps them. The server + obtains the virtual address for the buffers through an mmap call on the ring + fd. +* When there is a request from a client, fuse will select a buffer from the + ring if there is any payload that needs to be copied, copy over the payload + to the selected buffer, and copy over the headers to the fixed buffer at + index 0, at the buffer id that corresponds to the server (which the server + needs to specify through sqe->buf_index). +* The server obtains a cqe representing the request. The cqe flag will have + IORING_CQE_F_BUFFER set if a selected buffer was used for the payload. The + buffer id is stashed in cqe->flags (through IORING_CQE_BUFFER_SHIFT). The + server can directly access the payload by using that buffer id to calculate + the offset into the virtual address obtained for the buffers. +* The server processes the request and then sends a + FUSE_URING_CMD_COMMIT_AND_FETCH sqe with the reply. +* When the kernel handles the sqe, it will process the reply and if there is a + next request, it will reuse the same selected buffer for the request. If + there is no next request, it will recycle the buffer back to the ring. + +Zero-copy +========= + +Fuse io-uring zero-copy allows the server to directly read from / write to the +client's pages and bypass any intermediary buffer copies. This is only allowed +on privileged servers. + +In order to use zero-copy, the server must pregister the following: +* a sparse buffer for every entry in the queue. This is where the client's + pages will reside +* a fixed buffer at index queue_depth (tailing the sparse buffer). + This is where the headers will reside +* a kernel-managed buffer ring. This is where any non-zero-copied payload (eg + out headers) will reside + +When the client issues a read/write, fuse stores the client's underlying pages +in the sparse buffer entry corresponding to the ent in the queue. The server +can then issue reads/writes on these pages through io_uring rw operations. +Please note that the server is not able to directly access these pages, it +must go through the io-uring interface to read/write to them. The pages are +unregistered once the server replies to the request. Non-zero-copyable +payload (if needed) is placed in a buffer from the kernel-managed buffer ring.
Signature · 1 line
2.47.3
Re: [PATCH v1 30/30] docs: fuse: add io-uring bufring and zero-copy documentation
Joanne Koong <[email protected]>:
Quoted 1 line
+ virtual addresses for evey server-kernel interaction
I think you meant "every"
Signature · 1 line
Askar Safin
Joanne Koong
maintainerRe: [PATCH v1 30/30] docs: fuse: add io-uring bufring and zero-copy documentation
Quoted 5 lines · Askar Safin
Joanne Koong <[email protected]>:
Quoted 1 line
+ virtual addresses for evey server-kernel interaction
I think you meant "every"
Yes you're right, this should be "every". I'll fix this in v2.
Thanks,
Joanne
Quoted 3 lines
--
Askar Safin
Joanne Koong <[email protected]>:
Quoted 1 line
This series adds buffer ring and zero-copy capabilities to fuse over io-uring.
So this is superior modern io-uring-based zero-copy for fuse? I. e. modern alternative to splice fuse hacks here:
https://elixir.bootlin.com/linux/v6.18-rc6/source/fs/fuse/dev.c#L979
https://elixir.bootlin.com/linux/v6.18-rc6/source/fs/fuse/dev.c#L2291
?
If so, then may I remove that fuse splice special-casing?
I assume that splice for fuse is not that fast anyway despite that special-casing linked above. So code linked above introduce complexity to the kernel without giving any actual benefits. As opposed to truly fast modern uring interfaces.
Fuse is the only user of "pipe_buf_confirm" outside of fs/pipe.c and fs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/pipe_buf_confirm
It is the only user of "PIPE_BUF_FLAG_LRU" outside of fs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/PIPE_BUF_FLAG_LRU .
It is the only user of "PIPE_BUF_FLAG_GIFT" outside of fs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/PIPE_BUF_FLAG_GIFT .
It is one of the few users of "SPLICE_F_MOVE":
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/SPLICE_F_MOVE .
It is one of two callers of "pipe_buf_try_steal":
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/pipe_buf_try_steal
(the other is virtio-console).
(Side note: Linus on pipe_buf_try_steal/SPLICE_F_GIFT: "Now, I would
actually not disagree with removing that part. It's
scary. But I think we don't really have any users (ok, fuse and some
random console driver?)"
- https://lore.kernel.org/all/CAHk-=whYWEUU69nY6k4j1_EQnQDNPy4TqAMvpf1UA111UDdmYg@mail.gmail.com/
)
So, removing special handling of splice in fuse will lead to simplifications in core pipe/splice code. And I think that we need to do this, because splice is not fast anyway, compared to uring.
Also from Linus:
Quoted 3 lines
Side note: maybe I should clarify. I have grown to pretty much hate
splice()over the years, just because it's been a constant source of sorrow in so many ways.
...
Quoted 4 lines
It's just that it was never as lovely and as useful as it promised to be. So I'd actually be more than happy to just say "let's decommission splice entirely, just keeping the interfaces alive for backwards compatibility"
...
Quoted 10 lines
I'd be willing to *simplify*
splice()by just
saying "it was all a mistake", and just turning it into wrappers around read/write. But those patches would have to be radical simplifications, not adding yet more crud on top of the pain that issplice().Because it will hurt performance. And I'm ok with that as long as it comes with huge simplifications. What I'm *not* ok with is "I mis-used splice, now I want splice to act differently, so let's make it even more complicated".
- https://lore.kernel.org/all/CAHk-=wgG_2cmHgZwKjydi7=iimyHyN8aessnbM9XQ9ufbaUz9g@mail.gmail.com/
- https://lore.kernel.org/all/CAHk-=wjixHw6n_R5TQWW1r0a+GgFAPGw21KMj6obkzr3qXXbYA@mail.gmail.com/
For all these reasons, may I remove fuse splice special casing?
Signature · 1 line
Askar Safin
Joanne Koong
maintainerQuoted 59 lines · Askar Safin
Joanne Koong <[email protected]>:
Quoted 1 line
This series adds buffer ring and zero-copy capabilities to fuse over io-uring.
So this is superior modern io-uring-based zero-copy for fuse? I. e. modern alternative to splice fuse hacks here:
https://elixir.bootlin.com/linux/v6.18-rc6/source/fs/fuse/dev.c#L979
https://elixir.bootlin.com/linux/v6.18-rc6/source/fs/fuse/dev.c#L2291
?If so, then may I remove that fuse splice special-casing?
I assume that splice for fuse is not that fast anyway despite that special-casing linked above. So code linked above introduce complexity to the kernel without giving any actual benefits. As opposed to truly fast modern uring interfaces.
Fuse is the only user of "
pipe_buf_confirm" outside offs/pipe.candfs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/pipe_buf_confirmIt is the only user of "
PIPE_BUF_FLAG_LRU" outside offs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/PIPE_BUF_FLAG_LRU .It is the only user of "
PIPE_BUF_FLAG_GIFT" outside offs/splice.c:
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/PIPE_BUF_FLAG_GIFT .It is one of the few users of "
SPLICE_F_MOVE":
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/SPLICE_F_MOVE .It is one of two callers of "
pipe_buf_try_steal":
https://elixir.bootlin.com/linux/v6.18-rc6/C/ident/pipe_buf_try_steal
(the other is virtio-console).(Side note: Linus on pipe_buf_try_steal/SPLICE_F_GIFT: "Now, I would
actually not disagree with removing that part. It's
scary. But I think we don't really have any users (ok, fuse and some
random console driver?)"
- https://lore.kernel.org/all/CAHk-=whYWEUU69nY6k4j1_EQnQDNPy4TqAMvpf1UA111UDdmYg@mail.gmail.com/
)So, removing special handling of splice in fuse will lead to simplifications in core pipe/splice code. And I think that we need to do this, because splice is not fast anyway, compared to uring.
Also from Linus:
Quoted 3 lines
Side note: maybe I should clarify. I have grown to pretty much hate
splice()over the years, just because it's been a constant source of sorrow in so many ways....
Quoted 4 lines
It's just that it was never as lovely and as useful as it promised to be. So I'd actually be more than happy to just say "let's decommission splice entirely, just keeping the interfaces alive for backwards compatibility"
...
Quoted 6 lines
I'd be willing to *simplify*
splice()by just
saying "it was all a mistake", and just turning it into wrappers around read/write. But those patches would have to be radical simplifications, not adding yet more crud on top of the pain that issplice().
Quoted 8 lines
Quoted 4 lines
Because it will hurt performance. And I'm ok with that as long as it comes with huge simplifications. What I'm *not* ok with is "I mis-used splice, now I want splice to act differently, so let's make it even more complicated".
- https://lore.kernel.org/all/CAHk-=wgG_2cmHgZwKjydi7=iimyHyN8aessnbM9XQ9ufbaUz9g@mail.gmail.com/
- https://lore.kernel.org/all/CAHk-=wjixHw6n_R5TQWW1r0a+GgFAPGw21KMj6obkzr3qXXbYA@mail.gmail.com/For all these reasons, may I remove fuse splice special casing?
fuse uring is opt-in, not the default. Removing the splice optimizations would regress performance for users who aren't using uring, which is currently most of them.
Thanks,
Joanne
Quoted 3 lines
--
Askar Safin