Skip to contentSkip to conversation

fuse/io-uring: add kernel-managed buffer rings and zero-copy

PATCH In review Joanne Koong 7 participants 58 messages Jan 16 – Feb 26

Subsystems (7)
lore
Share & tools
lore

Joanne Koong

author maintainer
#0 lore raw

v4 cover fuse/io-uring: add kernel-managed buffer rings and zero-copy

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.

There was a previous submission for supporting registered buffers in fuse [1] but that was abandoned in favor of using kernel-managed buffer rings, which, once incremental buffer consumption is added in a later patchset, gives significant memory usage advantages in allowing the full buffer capacity to be utilized across multiple requests, as well as offers more flexibility for future additions. As well, it also makes the userspace side setup simpler. The relevant refactoring fuse patches from the previous submission are carried over into this one.

Benchmarks for zero-copy (patch 24) show approximately the following differences in throughput for bs=1M:

direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s) buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)

The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1

This series is on top of commit b71e635feefc in the io-uring tree.

The libfuse changes can be found in [2]. This has a dependency on the liburing changes in [3]. To test the server, you can run it with:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -o io_uring_zero_copy -o io_uring_q_depth=8

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://github.com/joannekoong/libfuse/tree/zero_copy
[3] https://github.com/joannekoong/liburing/tree/kmbuf

v3: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3 -> v4:
* Get rid of likely()s and get rid of going through cmd interface layer (Gabriel)
* Fix io_uring_cmd_fixed_index_get() to return back the node pointer (Caleb)
* Add documentation for io_buffer_register_bvec (Caleb)
* Remove WARN_ON_ONCE() for io_buffer_unregister() call (Caleb)

v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* fix casting between void * and u64 for 32-bit architectures (kernel test robot)
* add newline for documentation bullet points (kernel test robot)
* fix unrecognized "boolean" (kernel test robot), switch it to a flag (me)

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* drop fuse buffer cleanup on ring death, which makes things a lot simpler (Jens)
- this drops a lot of things (eg needing ring_ctx tracking, needing callback for ring death, etc)
* drop fixed buffer pinning altogether and just do lookup every time (Jens) (didn't significantly affect the benchmark results seen)
* fix spelling mistake in docs (Askar)
* use -EALREADY for pinning already pinned bufring, return PTR_ERR for registration instead of err, move initializations to outside locks (Caleb)
* drop fuse patches for zero-ed out headers (me)

Joanne Koong (25):
io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic
helpers
io_uring/kbuf: rename io_unregister_pbuf_ring() to
io_unregister_buf_ring()
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: add mmap support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: add recycling for kernel managed buffer rings
io_uring: add io_uring_fixed_index_get() and
io_uring_fixed_index_put()
io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: export io_ring_buffer_select() io_uring/kbuf: return buffer id in buffer selection io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: support buffer copying for kernel addresses
fuse: add io-uring kernel-managed buffer ring io_uring/rsrc: rename
io_buffer_register_bvec()/io_buffer_unregister_bvec()
io_uring/rsrc: split io_buffer_register_request() logic io_uring/rsrc: Allow buffer release callback to be optional io_uring/rsrc: add io_buffer_register_bvec()
fuse: add zero-copy over io-uring
docs: fuse: add io-uring bufring and zero-copy documentation

 Documentation/block/ublk.rst                  |  14 +-
 .../filesystems/fuse/fuse-io-uring.rst        |  59 +-
 drivers/block/ublk_drv.c                      |  18 +-
 fs/fuse/dev.c                                 |  30 +-
 fs/fuse/dev_uring.c                           | 692 ++++++++++++++----
 fs/fuse/dev_uring_i.h                         |  42 +-
 fs/fuse/fuse_dev_i.h                          |   8 +-
 include/linux/io_uring/buf.h                  |  25 +
 include/linux/io_uring/cmd.h                  |  97 ++-
 include/linux/io_uring_types.h                |  10 +-
 include/uapi/linux/fuse.h                     |  17 +-
 include/uapi/linux/io_uring.h                 |  17 +-
 io_uring/kbuf.c                               | 355 +++++++--
 io_uring/kbuf.h                               |  19 +-
 io_uring/memmap.c                             | 117 ++-
 io_uring/memmap.h                             |   4 +
 io_uring/register.c                           |   9 +-
 io_uring/rsrc.c                               | 183 ++++-
 io_uring/uring_cmd.c                          |   6 +-
 19 files changed, 1447 insertions(+), 275 deletions(-)
 create mode 100644 include/linux/io_uring/buf.h
Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH v4 1/25 io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic helpers

Refactor the logic in io_register_pbuf_ring() into generic helpers:
- io_validate_buf_reg(): Validate user input and buffer registration parameters
- io_alloc_new_buffer_list(): Allocate and initialize a new buffer list for the given buffer group ID
- io_setup_pbuf_ring(): Sets up the physical buffer ring region and handles memory mapping for provided buffer rings

This is a preparatory change for upcoming kernel-managed buffer ring support which will need to reuse some of these helpers.

---
 io_uring/kbuf.c | 123 ++++++++++++++++++++++++++++++++----------------
 1 file changed, 82 insertions(+), 41 deletions(-)
io_uring/kbuf.c +82 -41
index 796d131107dd..100367bb510b 100644
@@ -596,55 +596,71 @@ int io_manage_buffers_legacy(struct io_kiocb *req, unsigned int issue_flags)
 	return IOU_COMPLETE;
 }
 
-int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
+static int io_validate_buf_reg(struct io_uring_buf_reg *reg,
+			       unsigned int permitted_flags)
 {
-	struct io_uring_buf_reg reg;
-	struct io_buffer_list *bl;
-	struct io_uring_region_desc rd;
-	struct io_uring_buf_ring *br;
-	unsigned long mmap_offset;
-	unsigned long ring_size;
-	int ret;
-
-	lockdep_assert_held(&ctx->uring_lock);
-
-	if (copy_from_user(&reg, arg, sizeof(reg)))
-		return -EFAULT;
-	if (!mem_is_zero(reg.resv, sizeof(reg.resv)))
+	if (!mem_is_zero(reg->resv, sizeof(reg->resv)))
 		return -EINVAL;
-	if (reg.flags & ~(IOU_PBUF_RING_MMAP | IOU_PBUF_RING_INC))
+	if (reg->flags & ~permitted_flags)
 		return -EINVAL;
-	if (!is_power_of_2(reg.ring_entries))
+	if (!is_power_of_2(reg->ring_entries))
 		return -EINVAL;
 	/* cannot disambiguate full vs empty due to head/tail size */
-	if (reg.ring_entries >= 65536)
+	if (reg->ring_entries >= 65536)
 		return -EINVAL;
+	return 0;
+}
 
-	bl = io_buffer_get_list(ctx, reg.bgid);
-	if (bl) {
+static int io_alloc_new_buffer_list(struct io_ring_ctx *ctx,
+				    struct io_uring_buf_reg *reg,
+				    struct io_buffer_list **bl)
+{
+	struct io_buffer_list *list;
+
+	list = io_buffer_get_list(ctx, reg->bgid);
+	if (list) {
 		/* if mapped buffer ring OR classic exists, don't allow */
-		if (bl->flags & IOBL_BUF_RING || !list_empty(&bl->buf_list))
+		if (list->flags & IOBL_BUF_RING || !list_empty(&list->buf_list))
 			return -EEXIST;
-		io_destroy_bl(ctx, bl);
+		io_destroy_bl(ctx, list);
 	}
 
-	bl = kzalloc(sizeof(*bl), GFP_KERNEL_ACCOUNT);
-	if (!bl)
+	list = kzalloc(sizeof(*list), GFP_KERNEL_ACCOUNT);
+	if (!list)
 		return -ENOMEM;
 
-	mmap_offset = (unsigned long)reg.bgid << IORING_OFF_PBUF_SHIFT;
-	ring_size = flex_array_size(br, bufs, reg.ring_entries);
+	list->nr_entries = reg->ring_entries;
+	list->mask = reg->ring_entries - 1;
+	list->flags = IOBL_BUF_RING;
+
+	*bl = list;
+
+	return 0;
+}
+
+static int io_setup_pbuf_ring(struct io_ring_ctx *ctx,
+			      struct io_uring_buf_reg *reg,
+			      struct io_buffer_list *bl)
+{
+	struct io_uring_region_desc rd;
+	unsigned long mmap_offset;
+	unsigned long ring_size;
+	int ret;
+
+	mmap_offset = (unsigned long)reg->bgid << IORING_OFF_PBUF_SHIFT;
+	ring_size = flex_array_size(bl->buf_ring, bufs, reg->ring_entries);
 
 	memset(&rd, 0, sizeof(rd));
 	rd.size = PAGE_ALIGN(ring_size);
-	if (!(reg.flags & IOU_PBUF_RING_MMAP)) {
-		rd.user_addr = reg.ring_addr;
+	if (!(reg->flags & IOU_PBUF_RING_MMAP)) {
+		rd.user_addr = reg->ring_addr;
 		rd.flags |= IORING_MEM_REGION_TYPE_USER;
 	}
+
 	ret = io_create_region(ctx, &bl->region, &rd, mmap_offset);
 	if (ret)
-		goto fail;
-	br = io_region_get_ptr(&bl->region);
+		return ret;
+	bl->buf_ring = io_region_get_ptr(&bl->region);
 
 #ifdef SHM_COLOUR
 	/*
@@ -656,25 +672,50 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
 	 * should use IOU_PBUF_RING_MMAP instead, and liburing will handle
 	 * this transparently.
 	 */
-	if (!(reg.flags & IOU_PBUF_RING_MMAP) &&
-	    ((reg.ring_addr | (unsigned long)br) & (SHM_COLOUR - 1))) {
-		ret = -EINVAL;
-		goto fail;
+	if (!(reg->flags & IOU_PBUF_RING_MMAP) &&
+	    ((reg->ring_addr | (unsigned long)bl->buf_ring) &
+	     (SHM_COLOUR - 1))) {
+		io_free_region(ctx->user, &bl->region);
+		return -EINVAL;
 	}
 #endif
 
-	bl->nr_entries = reg.ring_entries;
-	bl->mask = reg.ring_entries - 1;
-	bl->flags |= IOBL_BUF_RING;
-	bl->buf_ring = br;
+	return 0;
+}
+
+int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
+{
+	unsigned int permitted_flags;
+	struct io_uring_buf_reg reg;
+	struct io_buffer_list *bl;
+	int ret;
+
+	lockdep_assert_held(&ctx->uring_lock);
+
+	if (copy_from_user(&reg, arg, sizeof(reg)))
+		return -EFAULT;
+
+	permitted_flags = IOU_PBUF_RING_MMAP | IOU_PBUF_RING_INC;
+	ret = io_validate_buf_reg(&reg, permitted_flags);
+	if (ret)
+		return ret;
+
+	ret = io_alloc_new_buffer_list(ctx, &reg, &bl);
+	if (ret)
+		return ret;
+
+	ret = io_setup_pbuf_ring(ctx, &reg, bl);
+	if (ret) {
+		kfree(bl);
+		return ret;
+	}
+
 	if (reg.flags & IOU_PBUF_RING_INC)
 		bl->flags |= IOBL_INC;
+
 	io_buffer_add_list(ctx, bl, reg.bgid);
+
 	return 0;
-fail:
-	io_free_region(ctx->user, &bl->region);
-	kfree(bl);
-	return ret;
 }
 
 int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
Signature · 1 line
2.47.3
author maintainer
#2 lore raw

PATCH v4 2/25 io_uring/kbuf: rename io_unregister_pbuf_ring() to io_unregister_buf_ring()

In reply to Joanne Koong

Use the more generic name io_unregister_buf_ring() as this function will be used for unregistering both provided buffer rings and kernel-managed buffer rings.

This is a preparatory change for upcoming kernel-managed buffer ring support.

---
 io_uring/kbuf.c     | 2 +-
 io_uring/kbuf.h     | 2 +-
 io_uring/register.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
io_uring/kbuf.c +1 -1
index 100367bb510b..cbe477db7b86 100644
@@ -718,7 +718,7 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
 	return 0;
 }
 
-int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
+int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg)
 {
 	struct io_uring_buf_reg reg;
 	struct io_buffer_list *bl;
io_uring/kbuf.h +1 -1
index bf15e26520d3..40b44f4fdb15 100644
@@ -74,7 +74,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
 int io_manage_buffers_legacy(struct io_kiocb *req, unsigned int issue_flags);
 
 int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg);
-int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg);
+int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg);
 int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg);
 
 bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags);
io_uring/register.c +1 -1
index 3d3822ff3fd9..888c8172818f 100644
@@ -752,7 +752,7 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 		ret = -EINVAL;
 		if (!arg || nr_args != 1)
 			break;
-		ret = io_unregister_pbuf_ring(ctx, arg);
+		ret = io_unregister_buf_ring(ctx, arg);
 		break;
 	case IORING_REGISTER_SYNC_CANCEL:
 		ret = -EINVAL;
Signature · 1 line
2.47.3
author maintainer
#3 lore raw

PATCH v4 3/25 io_uring/kbuf: add support for kernel-managed buffer rings

In reply to Joanne Koong

Add support for kernel-managed buffer rings (kmbuf rings), which allow the kernel to allocate and manage the backing buffers for a buffer ring, rather than requiring the application to provide and manage them.

This introduces two new registration opcodes:
- IORING_REGISTER_KMBUF_RING: Register a kernel-managed buffer ring
- IORING_UNREGISTER_KMBUF_RING: Unregister a kernel-managed buffer ring

The existing io_uring_buf_reg structure is extended with a union to support both application-provided buffer rings (pbuf) and kernel-managed buffer rings (kmbuf):
- For pbuf rings: ring_addr specifies the user-provided ring address
- For kmbuf rings: buf_size specifies the size of each buffer. buf_size must be non-zero and page-aligned.

The implementation follows the same pattern as pbuf ring registration, reusing the validation and buffer list allocation helpers introduced in earlier refactoring. The IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.

---
 include/uapi/linux/io_uring.h |  15 ++++-
 io_uring/kbuf.c               |  76 +++++++++++++++++++++++
 io_uring/kbuf.h               |   7 ++-
 io_uring/memmap.c             | 112 ++++++++++++++++++++++++++++++++++
 io_uring/memmap.h             |   4 ++
 io_uring/register.c           |   7 +++
 6 files changed, 217 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +13 -2
index b5b23c0d5283..589755a4e2b4 100644
@@ -700,6 +700,10 @@ enum io_uring_register_op {
 	/* auxiliary zcrx configuration, see enum zcrx_ctrl_op */
 	IORING_REGISTER_ZCRX_CTRL		= 36,
 
+	/* register/unregister kernel-managed ring buffer group */
+	IORING_REGISTER_KMBUF_RING		= 37,
+	IORING_UNREGISTER_KMBUF_RING		= 38,
+
 	/* this goes last */
 	IORING_REGISTER_LAST,
 
@@ -869,9 +873,16 @@ enum io_uring_register_pbuf_ring_flags {
 	IOU_PBUF_RING_INC	= 2,
 };
 
-/* argument for IORING_(UN)REGISTER_PBUF_RING */
+/* argument for IORING_(UN)REGISTER_PBUF_RING and
+ * IORING_(UN)REGISTER_KMBUF_RING
+ */
 struct io_uring_buf_reg {
-	__u64	ring_addr;
+	union {
+		/* used for pbuf rings */
+		__u64	ring_addr;
+		/* used for kmbuf rings */
+		__u32   buf_size;
+	};
 	__u32	ring_entries;
 	__u16	bgid;
 	__u16	flags;
io_uring/kbuf.c +76 -0

79 lines not on this page · raw message

io_uring/kbuf.h +5 -2
index 40b44f4fdb15..62c80a1ebf03 100644
@@ -7,9 +7,11 @@
 
 enum {
 	/* ring mapped provided buffers */
-	IOBL_BUF_RING	= 1,
+	IOBL_BUF_RING		= 1,
 	/* buffers are consumed incrementally rather than always fully */
-	IOBL_INC	= 2,
+	IOBL_INC		= 2,
+	/* buffers are kernel managed */
+	IOBL_KERNEL_MANAGED	= 4,
 };
 
 struct io_buffer_list {
@@ -74,6 +76,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
 int io_manage_buffers_legacy(struct io_kiocb *req, unsigned int issue_flags);
 
 int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg);
+int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg);
 int io_unregister_buf_ring(struct io_ring_ctx *ctx, void __user *arg);
 int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg);
 
io_uring/memmap.c +112 -0

142 lines not on this page · raw message

io_uring/memmap.h +4 -0
index a39d9e518905..b09fc34d5eb9 100644
@@ -21,6 +21,10 @@ int io_create_region(struct io_ring_ctx *ctx, struct io_mapped_region *mr,
 		     struct io_uring_region_desc *reg,
 		     unsigned long mmap_offset);
 
+int io_create_region_multi_buf(struct io_ring_ctx *ctx,
+			       struct io_mapped_region *mr,
+			       unsigned int nr_bufs, unsigned int buf_size);
+
 static inline void *io_region_get_ptr(struct io_mapped_region *mr)
 {
 	return mr->ptr;
io_uring/register.c +7 -0
index 888c8172818f..7f51b44b0e1d 100644
@@ -748,7 +748,14 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 			break;
 		ret = io_register_pbuf_ring(ctx, arg);
 		break;
+	case IORING_REGISTER_KMBUF_RING:
+		ret = -EINVAL;
+		if (!arg || nr_args != 1)
+			break;
+		ret = io_register_kmbuf_ring(ctx, arg);
+		break;
 	case IORING_UNREGISTER_PBUF_RING:
+	case IORING_UNREGISTER_KMBUF_RING:
 		ret = -EINVAL;
 		if (!arg || nr_args != 1)
 			break;
Signature · 1 line
2.47.3
#4 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

Quoted 4 lines · Joanne Koong

The implementation follows the same pattern as pbuf ring registration, reusing the validation and buffer list allocation helpers introduced in earlier refactoring. The IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.

Do you have a man page or other documentation for the uapi somewhere?

Quoted 4 lines
io_uring/kbuf.c +4 -0
+int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
+{
+	struct io_uring_buf_reg reg;
+	struct io_buffer_list *bl;

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 836

Quoted 10 lines
io_uring/kbuf.c +10 -0
+	int ret;
+
+	lockdep_assert_held(&ctx->uring_lock);
+
+	if (copy_from_user(&reg, arg, sizeof(reg)))
+		return -EFAULT;
+
+	ret = io_validate_buf_reg(&reg, 0);
+	if (ret)
+		return ret;

Probably more a comment for patch 1, but wouldn't it make sense to combine copy from user and vaidation into a single helper?

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 843

Quoted 3 lines
io_uring/kbuf.c +3 -0
+	ret = io_alloc_new_buffer_list(ctx, &reg, &bl);
+	if (ret)
+		return ret;

Return the buffer list from io_alloc_new_buffer_list or an ERR_PTR to simplify this a bit?

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 851

Quoted 7 lines
io_uring/kbuf.c +7 -0
+	ret = io_setup_kmbuf_ring(ctx, bl, &reg);
+	if (ret) {
+		kfree(bl);
+		return ret;
+	}
+
+	bl->flags |= IOBL_KERNEL_MANAGED;

Should io_setup_kmbuf_ring set IOBL_KERNEL_MANAGED as it is the one creating the kernel managed buffers?

io_uring/memmap.c static int io_region_pin_pages(struct io_mapped_region *mr, · line 182

Quoted 2 lines
io_uring/memmap.c +2 -0
+{
+	gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN;

Isn't this really a GFP_USER allocation and should account towardas the callers memory cgroup limit?

io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 273

Quoted 9 lines
io_uring/memmap.c +9 -0
+	if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages))
+		return -EFAULT;
+
+	if (WARN_ON_ONCE(!nr_bufs || !buf_size))
+		return -EINVAL;
+
+	nr_pages = ((size_t)buf_size * nr_bufs) >> PAGE_SHIFT;
+	if (nr_pages > UINT_MAX)
+		return -E2BIG;

This looks overflow prone, and probably should use check_mul_overflow.

author maintainer
#5 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Joanne Koong

The implementation follows the same pattern as pbuf ring registration, reusing the validation and buffer list allocation helpers introduced in earlier refactoring. The IOBL_KERNEL_MANAGED flag marks buffer lists as kernel-managed for appropriate handling in the I/O path.

Do you have a man page or other documentation for the uapi somewhere?

No, but I will tidy up the liburing side changes I have for this and add the documentation to liburing/man.

Quoted 10 lines
Quoted 10 lines
io_uring/kbuf.c +9 -0
+int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
+{
+     struct io_uring_buf_reg reg;
+     struct io_buffer_list *bl;
+     int ret;
+
+     lockdep_assert_held(&ctx->uring_lock);
+
+     if (copy_from_user(&reg, arg, sizeof(reg)))
Quoted 8 lines

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 836

Quoted 5 lines
io_uring/kbuf.c +5 -0
+             return -EFAULT;
+
+     ret = io_validate_buf_reg(&reg, 0);
+     if (ret)
+             return ret;

Probably more a comment for patch 1, but wouldn't it make sense to combine copy from user and vaidation into a single helper?

I'll rename this function to io_copy_and_validate_buf_reg() and move the copying into that.

Quoted 7 lines

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 843

Quoted 4 lines
io_uring/kbuf.c +3 -0
+     ret = io_alloc_new_buffer_list(ctx, &reg, &bl);
+     if (ret)
+             return ret;

Return the buffer list from io_alloc_new_buffer_list or an ERR_PTR to simplify this a bit?

Sounds good, I'll make this change.

Quoted 11 lines

io_uring/kbuf.c struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, · line 851

Quoted 8 lines
io_uring/kbuf.c +7 -0
+     ret = io_setup_kmbuf_ring(ctx, bl, &reg);
+     if (ret) {
+             kfree(bl);
+             return ret;
+     }
+
+     bl->flags |= IOBL_KERNEL_MANAGED;

Should io_setup_kmbuf_ring set IOBL_KERNEL_MANAGED as it is the one creating the kernel managed buffers?

That's a good point, I'll move this line into io_setup_kmbuf_ring() (and do the same for moving pbuf's IOBL_INC flag setting to io_setup_pbuf_ring()).

Quoted 6 lines

io_uring/memmap.c static int io_region_pin_pages(struct io_mapped_region *mr, · line 182

Quoted 3 lines
io_uring/memmap.c +2 -0
+{
+     gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_NOWARN;

Isn't this really a GFP_USER allocation and should account towardas the callers memory cgroup limit?

I think __GFP_ACCOUNT is what accounts the memory towards the caller's mem cgroup limit and GFP_KERNEL_ACCOUNT sets that whereas GFP_USER doesn't. GFP_USER sets __GFP_HARDWALL though which seems like it's also useful, but elsewhere in the io uring code, GFP_KERNEL_ACCOUNT is used for other userspace-facing allocations (eg pbufring's buffer metadata). But maybe this should be GFP_USER | __GFP_ACCOUNT.

Quoted 12 lines

io_uring/memmap.c static int io_region_allocate_pages(struct io_mapped_region *mr, · line 273

Quoted 10 lines
io_uring/memmap.c +9 -0
+     if (WARN_ON_ONCE(mr->pages || mr->ptr || mr->nr_pages))
+             return -EFAULT;
+
+     if (WARN_ON_ONCE(!nr_bufs || !buf_size))
+             return -EINVAL;
+
+     nr_pages = ((size_t)buf_size * nr_bufs) >> PAGE_SHIFT;
+     if (nr_pages > UINT_MAX)
+             return -E2BIG;

This looks overflow prone, and probably should use check_mul_overflow.

Okay yeah I think you're right that the (size_t) casting only protects against overflow on 64-bit systems but could still overflow 32-bit systems. I'll change this to use check_mul_overflow.

Thanks,
Joanne

Quoted 1 line
#6 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

In reply to Joanne Koong

Joanne Koong <[email protected]>:

Quoted 1 line

Add support for kernel-managed buffer rings (kmbuf rings)

Is it true that these kbufs solve same problem splice originally meant for? I. e. is it true that kbuf is modern uring-based replacement for splice?

Linus said in 2006 in https://lore.kernel.org/all/[email protected]/ :

Quoted 4 lines

The pipe is just the standard in-kernel buffer between two arbitrary points. Think of it as a scatter-gather list with a wait-queue. That's what a pipe _is_. Trying to get rid of the pipe totally misses the whole point of splice().

So, kbuf is modern version of exactly this?

Signature · 1 line
Askar Safin
author maintainer
#7 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

Quoted 7 lines · Askar Safin

Joanne Koong <[email protected]>:

Quoted 1 line

Add support for kernel-managed buffer rings (kmbuf rings)

Is it true that these kbufs solve same problem splice originally meant for? I. e. is it true that kbuf is modern uring-based replacement for splice?

Quoted 8 lines

Linus said in 2006 in https://lore.kernel.org/all/[email protected]/ :

Quoted 4 lines

The pipe is just the standard in-kernel buffer between two arbitrary points. Think of it as a scatter-gather list with a wait-queue. That's what a pipe _is_. Trying to get rid of the pipe totally misses the whole point of splice().

So, kbuf is modern version of exactly this?

I don't think this is related to kmbufs. Zero-copying is done through registered buffers (eg userspace registers sparse buffers for the ring ahead of time and then on the kernel side, those sparse buffers can be filled out by the kernel to point to relevant folios), which is separate from kmbufs. In fuse, kmbufs are used for non-zero-copyable payloads, and leveraged primarily as a way to minimize the memory footprint (especially once incremental buffer consumption for kmbufs is added, which I'm planning to do next).

Thanks,
Joanne

Quoted 3 lines

--
Askar Safin

#8 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

Quoted 2 lines · Joanne Koong

I don't think this is related to kmbufs. Zero-copying is done through registered buffers (eg userspace registers sparse buffers for the ring

Thank you for your answer.

Please, don't CC me when sending future versions of this patchset.

Signature · 1 line
Askar Safin
author maintainer
#9 lore raw

Re: [PATCH v4 03/25] io_uring/kbuf: add support for kernel-managed buffer rings

Quoted 8 lines · Askar Safin
Quoted 2 lines · Joanne Koong

I don't think this is related to kmbufs. Zero-copying is done through registered buffers (eg userspace registers sparse buffers for the ring

Thank you for your answer.

Please, don't CC me when sending future versions of this patchset.

For context, your email address was cc-ed because you had left a comment [1] on v1 of this patchset series. I'll make sure to remove you from the cc list going forward.

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Quoted 3 lines

--
Askar Safin

author maintainer
#10 lore raw

PATCH v4 4/25 io_uring/kbuf: add mmap support for kernel-managed buffer rings

In reply to Joanne Koong

Add support for mmapping kernel-managed buffer rings (kmbuf) to userspace, allowing applications to access the kernel-allocated buffers.

Similar to application-provided buffer rings (pbuf), kmbuf rings use the buffer group ID encoded in the mmap offset to identify which buffer ring to map. The implementation follows the same pattern as pbuf rings.

New mmap offset constants are introduced:
- IORING_OFF_KMBUF_RING (0x88000000): Base offset for kmbuf mappings
- IORING_OFF_KMBUF_SHIFT (16): Shift value to encode buffer group ID

The mmap offset is calculated during registration, encoding the bgid shifted by IORING_OFF_KMBUF_SHIFT. The io_buf_get_region() helper retrieves the appropriate region.

This allows userspace to mmap the kernel-allocated buffer region and access the buffers directly.

---
 include/uapi/linux/io_uring.h |  2 ++
 io_uring/kbuf.c               | 11 +++++++++--
 io_uring/kbuf.h               |  5 +++--
 io_uring/memmap.c             |  5 ++++-
 4 files changed, 18 insertions(+), 5 deletions(-)
include/uapi/linux/io_uring.h +2 -0
index 589755a4e2b4..96e936503ef6 100644
@@ -533,6 +533,8 @@ struct io_uring_cqe {
 #define IORING_OFF_SQES			0x10000000ULL
 #define IORING_OFF_PBUF_RING		0x80000000ULL
 #define IORING_OFF_PBUF_SHIFT		16
+#define IORING_OFF_KMBUF_RING		0x88000000ULL
+#define IORING_OFF_KMBUF_SHIFT		16
 #define IORING_OFF_MMAP_MASK		0xf8000000ULL
 
 /*
io_uring/kbuf.c +9 -2
index 8d253724754e..68469efe5552 100644
@@ -766,16 +766,23 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg)
 	return 0;
 }
 
-struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx,
-					    unsigned int bgid)
+struct io_mapped_region *io_buf_get_region(struct io_ring_ctx *ctx,
+					   unsigned int bgid,
+					   bool kernel_managed)
 {
 	struct io_buffer_list *bl;
+	bool is_kernel_managed;
 
 	lockdep_assert_held(&ctx->mmap_lock);
 
 	bl = xa_load(&ctx->io_bl_xa, bgid);
 	if (!bl || !(bl->flags & IOBL_BUF_RING))
 		return NULL;
+
+	is_kernel_managed = !!(bl->flags & IOBL_KERNEL_MANAGED);
+	if (is_kernel_managed != kernel_managed)
+		return NULL;
+
 	return &bl->region;
 }
 
io_uring/kbuf.h +3 -2
index 62c80a1ebf03..11d165888b8e 100644
@@ -88,8 +88,9 @@ unsigned int __io_put_kbufs(struct io_kiocb *req, struct io_buffer_list *bl,
 bool io_kbuf_commit(struct io_kiocb *req,
 		    struct io_buffer_list *bl, int len, int nr);
 
-struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx,
-					    unsigned int bgid);
+struct io_mapped_region *io_buf_get_region(struct io_ring_ctx *ctx,
+					   unsigned int bgid,
+					   bool kernel_managed);
 
 static inline bool io_kbuf_recycle_ring(struct io_kiocb *req,
 					struct io_buffer_list *bl)
io_uring/memmap.c +4 -1
index 6ecb6f1da59c..861038c1df49 100644
@@ -357,7 +357,10 @@ static struct io_mapped_region *io_mmap_get_region(struct io_ring_ctx *ctx,
 		return &ctx->sq_region;
 	case IORING_OFF_PBUF_RING:
 		id = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_PBUF_SHIFT;
-		return io_pbuf_get_region(ctx, id);
+		return io_buf_get_region(ctx, id, false);
+	case IORING_OFF_KMBUF_RING:
+		id = (offset & ~IORING_OFF_MMAP_MASK) >> IORING_OFF_KMBUF_SHIFT;
+		return io_buf_get_region(ctx, id, true);
 	case IORING_MAP_OFF_PARAM_REGION:
 		return &ctx->param_region;
 	case IORING_MAP_OFF_ZCRX_REGION:
Signature · 1 line
2.47.3
author maintainer
#11 lore raw

PATCH v4 5/25 io_uring/kbuf: support kernel-managed buffer rings in buffer selection

In reply to Joanne Koong

Allow kernel-managed buffers to be selected. This requires modifying the io_br_sel struct to separate the fields for address and val, since a kernel address cannot be distinguished from a negative val when error checking.

Auto-commit any selected kernel-managed buffer.

---
 include/linux/io_uring_types.h |  8 ++++----
 io_uring/kbuf.c                | 16 ++++++++++++----
 2 files changed, 16 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +4 -4
index a3e8ddc9b380..0b8880cdda8b 100644
@@ -93,13 +93,13 @@ struct io_mapped_region {
  */
 struct io_br_sel {
 	struct io_buffer_list *buf_list;
-	/*
-	 * Some selection parts return the user address, others return an error.
-	 */
 	union {
+		/* for classic/ring provided buffers */
 		void __user *addr;
-		ssize_t val;
+		/* for kernel-managed buffers */
+		void *kaddr;
 	};
+	ssize_t val;
 };
 
 
io_uring/kbuf.c +12 -4
index 68469efe5552..d9bdb2be5f13 100644
@@ -155,7 +155,8 @@ static int io_provided_buffers_select(struct io_kiocb *req, size_t *len,
 	return 1;
 }
 
-static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)
+static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,
+			     unsigned int issue_flags)
 {
 	/*
 	* If we came in unlocked, we have no choice but to consume the
@@ -170,7 +171,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)
 	if (issue_flags & IO_URING_F_UNLOCKED)
 		return true;
 
-	/* uring_cmd commits kbuf upfront, no need to auto-commit */
+	/* kernel-managed buffers are auto-committed */
+	if (bl->flags & IOBL_KERNEL_MANAGED)
+		return true;
+
+	/* multishot uring_cmd commits kbuf upfront, no need to auto-commit */
 	if (!io_file_can_poll(req) && req->opcode != IORING_OP_URING_CMD)
 		return true;
 	return false;
@@ -200,9 +205,12 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 	req->flags |= REQ_F_BUFFER_RING | REQ_F_BUFFERS_COMMIT;
 	req->buf_index = READ_ONCE(buf->bid);
 	sel.buf_list = bl;
-	sel.addr = u64_to_user_ptr(READ_ONCE(buf->addr));
+	if (bl->flags & IOBL_KERNEL_MANAGED)
+		sel.kaddr = (void *)(uintptr_t)READ_ONCE(buf->addr);
+	else
+		sel.addr = u64_to_user_ptr(READ_ONCE(buf->addr));
 
-	if (io_should_commit(req, issue_flags)) {
+	if (io_should_commit(req, bl, issue_flags)) {
 		io_kbuf_commit(req, sel.buf_list, *len, 1);
 		sel.buf_list = NULL;
 	}
Signature · 1 line
2.47.3
author maintainer
#12 lore raw

PATCH v4 6/25 io_uring/kbuf: add buffer ring pinning/unpinning

In reply to Joanne Koong

Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.

This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.

This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.

---
 include/linux/io_uring/cmd.h | 17 +++++++++++++
 io_uring/kbuf.c              | 48 ++++++++++++++++++++++++++++++++++++
 io_uring/kbuf.h              |  5 ++++
 3 files changed, 70 insertions(+)
include/linux/io_uring/cmd.h +17 -0
index 375fd048c4cb..702b1903e6ee 100644
@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,
 bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
 				 struct io_br_sel *sel, unsigned int issue_flags);
 
+int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
+			  unsigned issue_flags, struct io_buffer_list **bl);
+int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
+			    unsigned issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
 {
 	return true;
 }
+static inline int io_uring_buf_ring_pin(struct io_uring_cmd *cmd,
+					unsigned buf_group,
+					unsigned issue_flags,
+					struct io_buffer_list **bl)
+{
+	return -EOPNOTSUPP;
+}
+static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
+					  unsigned buf_group,
+					  unsigned issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +48 -0

66 lines not on this page · raw message

io_uring/kbuf.h +5 -0
index 11d165888b8e..781630c2cc10 100644
@@ -12,6 +12,11 @@ enum {
 	IOBL_INC		= 2,
 	/* buffers are kernel managed */
 	IOBL_KERNEL_MANAGED	= 4,
+	/*
+	 * buffer ring is pinned and cannot be unregistered by userspace until
+	 * it has been unpinned
+	 */
+	IOBL_PINNED		= 8,
 };
 
 struct io_buffer_list {
Signature · 1 line
2.47.3
maintainer
#13 lore raw

Re: [PATCH v4 06/25] io_uring/kbuf: add buffer ring pinning/unpinning

Quoted 68 lines · Joanne Koong

Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.

This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.

This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.

---
 include/linux/io_uring/cmd.h | 17 +++++++++++++
 io_uring/kbuf.c              | 48 ++++++++++++++++++++++++++++++++++++
 io_uring/kbuf.h              |  5 ++++
 3 files changed, 70 insertions(+)
include/linux/io_uring/cmd.h +17 -0
index 375fd048c4cb..702b1903e6ee 100644
@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,
 bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
 				 struct io_br_sel *sel, unsigned int issue_flags);
 
+int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
+			  unsigned issue_flags, struct io_buffer_list **bl);
+int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
+			    unsigned issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
 {
 	return true;
 }
+static inline int io_uring_buf_ring_pin(struct io_uring_cmd *cmd,
+					unsigned buf_group,
+					unsigned issue_flags,
+					struct io_buffer_list **bl)
+{
+	return -EOPNOTSUPP;
+}
+static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
+					  unsigned buf_group,
+					  unsigned issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +1 -0
index d9bdb2be5f13..94ab23400721 100644
@@ -9,6 +9,7 @@
 #include <linux/poll.h>
 #include <linux/vmalloc.h>
 #include <linux/io_uring.h>
+#include <linux/io_uring/cmd.h>
 
 #include <uapi/linux/io_uring.h>
 

io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 242

Quoted 5 lines
io_uring/kbuf.c +2 -0
@@ -237,6 +238,51 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,
 	return sel;
 }
 
+int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
+			  unsigned issue_flags, struct io_buffer_list **bl)

I'm just looking at the fuse part and I'm actually not sure what the "buf_group" parameter is. I guess it is a the buffer group set up by userspace? Does io-uring have some documentation like fuse has under Documentation/filesystems/fuse/?

Thanks,
Bernd

author maintainer
#14 lore raw

Re: [PATCH v4 06/25] io_uring/kbuf: add buffer ring pinning/unpinning

Quoted 73 lines · Bernd Schubert
Quoted 71 lines · Joanne Koong

Add kernel APIs to pin and unpin buffer rings, preventing userspace from unregistering a buffer ring while it is pinned by the kernel.

This provides a mechanism for kernel subsystems to safely access buffer ring contents while ensuring the buffer ring remains valid. A pinned buffer ring cannot be unregistered until explicitly unpinned. On the userspace side, trying to unregister a pinned buffer will return -EBUSY.

This is a preparatory change for upcoming fuse usage of kernel-managed buffer rings. It is necessary for fuse to pin the buffer ring because fuse may need to select a buffer in atomic contexts, which it can only do so by using the underlying buffer list pointer.

---
 include/linux/io_uring/cmd.h | 17 +++++++++++++
 io_uring/kbuf.c              | 48 ++++++++++++++++++++++++++++++++++++
 io_uring/kbuf.h              |  5 ++++
 3 files changed, 70 insertions(+)
include/linux/io_uring/cmd.h +17 -0
index 375fd048c4cb..702b1903e6ee 100644
@@ -84,6 +84,10 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,
 bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
                               struct io_br_sel *sel, unsigned int issue_flags);
 
+int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
+                       unsigned issue_flags, struct io_buffer_list **bl);
+int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
+                         unsigned issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -126,6 +130,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
 {
      return true;
 }
+static inline int io_uring_buf_ring_pin(struct io_uring_cmd *cmd,
+                                     unsigned buf_group,
+                                     unsigned issue_flags,
+                                     struct io_buffer_list **bl)
+{
+     return -EOPNOTSUPP;
+}
+static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
+                                       unsigned buf_group,
+                                       unsigned issue_flags)
+{
+     return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +1 -0
index d9bdb2be5f13..94ab23400721 100644
@@ -9,6 +9,7 @@
 #include <linux/poll.h>
 #include <linux/vmalloc.h>
 #include <linux/io_uring.h>
+#include <linux/io_uring/cmd.h>
 
 #include <uapi/linux/io_uring.h>
 
@@ -237,6 +238,51 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,
      return sel;
 }
 
Quoted 7 lines

io_uring/kbuf.c struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len, · line 242

Quoted 2 lines
io_uring/kbuf.c +2 -0
+int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
+                       unsigned issue_flags, struct io_buffer_list **bl)

I'm just looking at the fuse part and I'm actually not sure what the "buf_group" parameter is. I guess it is a the buffer group set up by userspace? Does io-uring have some documentation like fuse has under Documentation/filesystems/fuse/?

Hi Bernd,

Yes, buf_group is the id of the buffer group set up by userspace. I don't see anything documentation related for it under Documentation/, I think the closest equivalent is the man page for the buffer registration API in [1] where it says " bgid is the buffer group ID associated with this ring. SQEs that select a buffer have a buffer group associated with them in their buf_group field".

Thanks,
Joanne

[1] https://man7.org/linux/man-pages/man3/io_uring_register_buf_ring.3.html

Quoted 4 lines

Thanks,
Bernd

author maintainer
#15 lore raw

PATCH v4 7/25 io_uring/kbuf: add recycling for kernel managed buffer rings

In reply to Joanne Koong

Add an interface for buffers to be recycled back into a kernel-managed buffer ring.

This is a preparatory patch for fuse over io-uring.

---
 include/linux/io_uring/cmd.h | 11 +++++++++
 io_uring/kbuf.c              | 44 ++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
include/linux/io_uring/cmd.h +11 -0
index 702b1903e6ee..a488e945f883 100644
@@ -88,6 +88,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
 			  unsigned issue_flags, struct io_buffer_list **bl);
 int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
 			    unsigned issue_flags);
+
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+			   u64 addr, unsigned int len, unsigned int bid,
+			   unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +147,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
 {
 	return -EOPNOTSUPP;
 }
+static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd,
+					 unsigned int buf_group, u64 addr,
+					 unsigned int len, unsigned int bid,
+					 unsigned int issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +44 -0

50 lines not on this page · raw message

Signature · 1 line
2.47.3
maintainer
#16 lore raw

Re: [PATCH v4 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings

Quoted 59 lines · Joanne Koong

Add an interface for buffers to be recycled back into a kernel-managed buffer ring.

This is a preparatory patch for fuse over io-uring.

---
 include/linux/io_uring/cmd.h | 11 +++++++++
 io_uring/kbuf.c              | 44 ++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
include/linux/io_uring/cmd.h +11 -0
index 702b1903e6ee..a488e945f883 100644
@@ -88,6 +88,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
 			  unsigned issue_flags, struct io_buffer_list **bl);
 int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
 			    unsigned issue_flags);
+
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+			   u64 addr, unsigned int len, unsigned int bid,
+			   unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +147,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
 {
 	return -EOPNOTSUPP;
 }
+static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd,
+					 unsigned int buf_group, u64 addr,
+					 unsigned int len, unsigned int bid,
+					 unsigned int issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +11 -0
index 94ab23400721..a7d7d2c6b42c 100644
@@ -102,6 +102,50 @@ void io_kbuf_drop_legacy(struct io_kiocb *req)
 	req->kbuf = NULL;
 }
 
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+			   u64 addr, unsigned int len, unsigned int bid,
+			   unsigned int issue_flags)
+{
+	struct io_kiocb *req = cmd_to_io_kiocb(cmd);
+	struct io_ring_ctx *ctx = req->ctx;
+	struct io_uring_buf_ring *br;
+	struct io_uring_buf *buf;
+	struct io_buffer_list *bl;
+	int ret = -EINVAL;
+

io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 125

Quoted 10 lines
io_uring/kbuf.c +10 -0
+	if (WARN_ON_ONCE(req->flags & REQ_F_BUFFERS_COMMIT))
+		return ret;
+
+	io_ring_submit_lock(ctx, issue_flags);
+
+	bl = io_buffer_get_list(ctx, buf_group);
+
+	if (WARN_ON_ONCE(!(bl->flags & IOBL_BUF_RING)) ||
+	    WARN_ON_ONCE(!(bl->flags & IOBL_KERNEL_MANAGED)))
+		goto done;

Misses "if bl"?

Thanks,
Bernd

author maintainer
#17 lore raw

Re: [PATCH v4 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings

Quoted 65 lines · Bernd Schubert
Quoted 63 lines · Joanne Koong

Add an interface for buffers to be recycled back into a kernel-managed buffer ring.

This is a preparatory patch for fuse over io-uring.

---
 include/linux/io_uring/cmd.h | 11 +++++++++
 io_uring/kbuf.c              | 44 ++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
include/linux/io_uring/cmd.h +11 -0
index 702b1903e6ee..a488e945f883 100644
@@ -88,6 +88,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
                        unsigned issue_flags, struct io_buffer_list **bl);
 int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
                          unsigned issue_flags);
+
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+                        u64 addr, unsigned int len, unsigned int bid,
+                        unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +147,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
 {
      return -EOPNOTSUPP;
 }
+static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd,
+                                      unsigned int buf_group, u64 addr,
+                                      unsigned int len, unsigned int bid,
+                                      unsigned int issue_flags)
+{
+     return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +15 -0
index 94ab23400721..a7d7d2c6b42c 100644
@@ -102,6 +102,50 @@ void io_kbuf_drop_legacy(struct io_kiocb *req)
      req->kbuf = NULL;
 }
 
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+                        u64 addr, unsigned int len, unsigned int bid,
+                        unsigned int issue_flags)
+{
+     struct io_kiocb *req = cmd_to_io_kiocb(cmd);
+     struct io_ring_ctx *ctx = req->ctx;
+     struct io_uring_buf_ring *br;
+     struct io_uring_buf *buf;
+     struct io_buffer_list *bl;
+     int ret = -EINVAL;
+
+     if (WARN_ON_ONCE(req->flags & REQ_F_BUFFERS_COMMIT))
+             return ret;
+
+     io_ring_submit_lock(ctx, issue_flags);
Quoted 8 lines

io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 125

Quoted 6 lines
io_uring/kbuf.c +6 -0
+
+     bl = io_buffer_get_list(ctx, buf_group);
+
+     if (WARN_ON_ONCE(!(bl->flags & IOBL_BUF_RING)) ||
+         WARN_ON_ONCE(!(bl->flags & IOBL_KERNEL_MANAGED)))
+             goto done;

Misses "if bl"?

A buffer shouldn't be recycled back into the bufring if the bufring has already been unregistered. I'll add a WARN_ON to make this more clear. For the fuse case, this is ensured by the kmbuf ring pin which prevents unregistration.

Thanks,
Joanne

Quoted 5 lines

Thanks,
Bernd

#18 lore raw

Re: [PATCH v4 07/25] io_uring/kbuf: add recycling for kernel managed buffer rings

Quoted 75 lines · Joanne Koong
Quoted 65 lines · Bernd Schubert
Quoted 63 lines · Joanne Koong

Add an interface for buffers to be recycled back into a kernel-managed buffer ring.

This is a preparatory patch for fuse over io-uring.

---
 include/linux/io_uring/cmd.h | 11 +++++++++
 io_uring/kbuf.c              | 44 ++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
include/linux/io_uring/cmd.h +11 -0
index 702b1903e6ee..a488e945f883 100644
@@ -88,6 +88,10 @@ int io_uring_buf_ring_pin(struct io_uring_cmd *cmd, unsigned buf_group,
                        unsigned issue_flags, struct io_buffer_list **bl);
 int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
                          unsigned issue_flags);
+
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+                        u64 addr, unsigned int len, unsigned int bid,
+                        unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -143,6 +147,13 @@ static inline int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd,
 {
      return -EOPNOTSUPP;
 }
+static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd,
+                                      unsigned int buf_group, u64 addr,
+                                      unsigned int len, unsigned int bid,
+                                      unsigned int issue_flags)
+{
+     return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +15 -0
index 94ab23400721..a7d7d2c6b42c 100644
@@ -102,6 +102,50 @@ void io_kbuf_drop_legacy(struct io_kiocb *req)
      req->kbuf = NULL;
 }
 
+int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
+                        u64 addr, unsigned int len, unsigned int bid,
+                        unsigned int issue_flags)
+{
+     struct io_kiocb *req = cmd_to_io_kiocb(cmd);
+     struct io_ring_ctx *ctx = req->ctx;
+     struct io_uring_buf_ring *br;
+     struct io_uring_buf *buf;
+     struct io_buffer_list *bl;
+     int ret = -EINVAL;
+
+     if (WARN_ON_ONCE(req->flags & REQ_F_BUFFERS_COMMIT))
+             return ret;
+
+     io_ring_submit_lock(ctx, issue_flags);
Quoted 8 lines

io_uring/kbuf.c void io_kbuf_drop_legacy(struct io_kiocb *req) · line 125

Quoted 6 lines
io_uring/kbuf.c +6 -0
+
+     bl = io_buffer_get_list(ctx, buf_group);
+
+     if (WARN_ON_ONCE(!(bl->flags & IOBL_BUF_RING)) ||
+         WARN_ON_ONCE(!(bl->flags & IOBL_KERNEL_MANAGED)))
+             goto done;

Misses "if bl"?

Quoted 4 lines

A buffer shouldn't be recycled back into the bufring if the bufring has already been unregistered. I'll add a WARN_ON to make this more clear. For the fuse case, this is ensured by the kmbuf ring pin which prevents unregistration.

It is a bit confusing, because all the other conditions first check if "bl" isn't a null pointer, this one here is the only exception, I think.

Thanks,
Bernd

author maintainer
#19 lore raw

PATCH v4 8/25 io_uring: add io_uring_fixed_index_get() and io_uring_fixed_index_put()

In reply to Joanne Koong

Add two new helpers, io_uring_fixed_index_get() and io_uring_fixed_index_put(). io_uring_fixed_index_get() constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node. io_uring_fixed_index_put() decrements this refcount. The caller is responsible for ensuring io_uring_fixed_index_put() is properly called for releasing the refcount after it is done using the iter it obtained through io_uring_fixed_index_get().

The struct io_rsrc_node pointer needs to be returned in io_uring_fixed_index_get() because the buffer at the index may be unregistered/replaced in the meantime between this and the io_uring_fixed_index_put() call. io_uring_fixed_index_put() takes in the struct io_rsrc_node pointer as an arg.

This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.

---
 include/linux/io_uring/cmd.h | 20 ++++++++++++
 io_uring/rsrc.c              | 59 ++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)
include/linux/io_uring/cmd.h +20 -0
index a488e945f883..de3f550598cf 100644
@@ -44,6 +44,14 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
 				  size_t uvec_segs,
 				  int ddir, struct iov_iter *iter,
 				  unsigned issue_flags);
+struct io_rsrc_node *io_uring_fixed_index_get(struct io_uring_cmd *cmd,
+					      int buf_index, unsigned int off,
+					      size_t len, int ddir,
+					      struct iov_iter *iter,
+					      unsigned int issue_flags);
+void io_uring_fixed_index_put(struct io_uring_cmd *cmd,
+			      struct io_rsrc_node *node,
+			      unsigned int issue_flags);
 
 /*
  * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd
@@ -108,6 +116,18 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
 {
 	return -EOPNOTSUPP;
 }
+static inline struct io_rsrc_node *
+io_uring_fixed_index_get(struct io_uring_cmd *cmd, int buf_index,
+			 unsigned int off, size_t len, int ddir,
+			 struct iov_iter *iter, unsigned int issue_flags)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+static inline void io_uring_fixed_index_put(struct io_uring_cmd *cmd,
+					    struct io_rsrc_node *node,
+					    unsigned int issue_flags)
+{
+}
 static inline void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret,
 		u64 ret2, unsigned issue_flags, bool is_cqe32)
 {
io_uring/rsrc.c +59 -0

65 lines not on this page · raw message

Signature · 1 line
2.47.3
Reviewed-by on v4 8/25
#20 lore raw

Re: [PATCH v4 08/25] io_uring: add io_uring_fixed_index_get() and io_uring_fixed_index_put()

Quoted 88 lines · Joanne Koong

Add two new helpers, io_uring_fixed_index_get() and io_uring_fixed_index_put(). io_uring_fixed_index_get() constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node. io_uring_fixed_index_put() decrements this refcount. The caller is responsible for ensuring io_uring_fixed_index_put() is properly called for releasing the refcount after it is done using the iter it obtained through io_uring_fixed_index_get().

The struct io_rsrc_node pointer needs to be returned in io_uring_fixed_index_get() because the buffer at the index may be unregistered/replaced in the meantime between this and the io_uring_fixed_index_put() call. io_uring_fixed_index_put() takes in the struct io_rsrc_node pointer as an arg.

This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.

---
 include/linux/io_uring/cmd.h | 20 ++++++++++++
 io_uring/rsrc.c              | 59 ++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)
include/linux/io_uring/cmd.h +20 -0
index a488e945f883..de3f550598cf 100644
@@ -44,6 +44,14 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
                                  size_t uvec_segs,
                                  int ddir, struct iov_iter *iter,
                                  unsigned issue_flags);
+struct io_rsrc_node *io_uring_fixed_index_get(struct io_uring_cmd *cmd,
+                                             int buf_index, unsigned int off,
+                                             size_t len, int ddir,
+                                             struct iov_iter *iter,
+                                             unsigned int issue_flags);
+void io_uring_fixed_index_put(struct io_uring_cmd *cmd,
+                             struct io_rsrc_node *node,
+                             unsigned int issue_flags);
 
 /*
  * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd
@@ -108,6 +116,18 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
 {
        return -EOPNOTSUPP;
 }
+static inline struct io_rsrc_node *
+io_uring_fixed_index_get(struct io_uring_cmd *cmd, int buf_index,
+                        unsigned int off, size_t len, int ddir,
+                        struct iov_iter *iter, unsigned int issue_flags)
+{
+       return ERR_PTR(-EOPNOTSUPP);
+}
+static inline void io_uring_fixed_index_put(struct io_uring_cmd *cmd,
+                                           struct io_rsrc_node *node,
+                                           unsigned int issue_flags)
+{
+}
 static inline void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret,
                u64 ret2, unsigned issue_flags, bool is_cqe32)
 {
io_uring/rsrc.c +16 -0
index 41c89f5c616d..fa41cae5e922 100644
@@ -1152,6 +1152,65 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
        return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
 }
 
+struct io_rsrc_node *io_uring_fixed_index_get(struct io_uring_cmd *cmd,
+                                             int buf_index, unsigned int off,
+                                             size_t len, int ddir,
+                                             struct iov_iter *iter,
+                                             unsigned int issue_flags)
+{
+       struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+       struct io_rsrc_node *node;
+       struct io_mapped_ubuf *imu;
+       u64 addr;
+       int err;
+
+       io_ring_submit_lock(ctx, issue_flags);
+
+       node = io_rsrc_node_lookup(&ctx->buf_table, buf_index);
+       if (!node) {

io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1180

Quoted 10 lines
io_uring/rsrc.c +10 -0
+               io_ring_submit_unlock(ctx, issue_flags);
+               return ERR_PTR(-EINVAL);
+       }
+
+       node->refs++;
+
+       io_ring_submit_unlock(ctx, issue_flags);
+
+       imu = node->buf;
+       if (!imu) {

How is this possible?

Other than that,

Quoted 39 lines
io_uring/rsrc.c +33 -1
+               err = -EFAULT;
+               goto error;
+       }
+
+       if (check_add_overflow(imu->ubuf, off, &addr)) {
+               err = -EINVAL;
+               goto error;
+       }
+
+       err = io_import_fixed(ddir, iter, imu, addr, len);
+       if (err)
+               goto error;
+
+       return node;
+
+error:
+       io_uring_fixed_index_put(cmd, node, issue_flags);
+       return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(io_uring_fixed_index_get);
+
+void io_uring_fixed_index_put(struct io_uring_cmd *cmd,
+                             struct io_rsrc_node *node,
+                             unsigned int issue_flags)
+{
+       struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+
+       io_ring_submit_lock(ctx, issue_flags);
+       io_put_rsrc_node(ctx, node);
+       io_ring_submit_unlock(ctx, issue_flags);
+}
+EXPORT_SYMBOL_GPL(io_uring_fixed_index_put);
+
 /* Lock two rings at once. The rings must be different! */
 static void lock_two_rings(struct io_ring_ctx *ctx1, struct io_ring_ctx *ctx2)
 {
--

2.47.3

author maintainer
#21 lore raw

Re: [PATCH v4 08/25] io_uring: add io_uring_fixed_index_get() and io_uring_fixed_index_put()

Quoted 56 lines · Caleb Sander Mateos
Quoted 54 lines · Joanne Koong

Add two new helpers, io_uring_fixed_index_get() and io_uring_fixed_index_put(). io_uring_fixed_index_get() constructs an iter for a fixed buffer at a given index and acquires a refcount on the underlying node. io_uring_fixed_index_put() decrements this refcount. The caller is responsible for ensuring io_uring_fixed_index_put() is properly called for releasing the refcount after it is done using the iter it obtained through io_uring_fixed_index_get().

The struct io_rsrc_node pointer needs to be returned in io_uring_fixed_index_get() because the buffer at the index may be unregistered/replaced in the meantime between this and the io_uring_fixed_index_put() call. io_uring_fixed_index_put() takes in the struct io_rsrc_node pointer as an arg.

This is a preparatory patch needed for fuse-over-io-uring support, as the metadata for fuse requests will be stored at the last index, which will be different from the buf index set on the sqe.

---
 include/linux/io_uring/cmd.h | 20 ++++++++++++
 io_uring/rsrc.c              | 59 ++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+)
io_uring/rsrc.c +20 -0
index 41c89f5c616d..fa41cae5e922 100644
@@ -1152,6 +1152,65 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
        return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
 }
 
+struct io_rsrc_node *io_uring_fixed_index_get(struct io_uring_cmd *cmd,
+                                             int buf_index, unsigned int off,
+                                             size_t len, int ddir,
+                                             struct iov_iter *iter,
+                                             unsigned int issue_flags)
+{
+       struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+       struct io_rsrc_node *node;
+       struct io_mapped_ubuf *imu;
+       u64 addr;
+       int err;
+
+       io_ring_submit_lock(ctx, issue_flags);
+
+       node = io_rsrc_node_lookup(&ctx->buf_table, buf_index);
+       if (!node) {
+               io_ring_submit_unlock(ctx, issue_flags);
+               return ERR_PTR(-EINVAL);
+       }
+
Quoted 8 lines

io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1180

Quoted 6 lines
io_uring/rsrc.c +6 -0
+       node->refs++;
+
+       io_ring_submit_unlock(ctx, issue_flags);
+
+       imu = node->buf;
+       if (!imu) {

How is this possible?

You're right, this null check is unnecessary. I'll drop it.

Thank you for reviewing the patches, Caleb.

Quoted 7 lines

Other than that,

Quoted 3 lines
io_uring/rsrc.c +3 -0
+               err = -EFAULT;
+               goto error;
+       }
author maintainer
#22 lore raw

PATCH v4 9/25 io_uring/kbuf: add io_uring_is_kmbuf_ring()

In reply to Joanne Koong

io_uring_is_kmbuf_ring() returns true if there is a kernel-managed buffer ring at the specified buffer group.

This is a preparatory patch for upcoming fuse kernel-managed buffer support, which needs to ensure the buffer ring registered by the server is a kernel-managed buffer ring.

---
 include/linux/io_uring/cmd.h |  9 +++++++++
 io_uring/kbuf.c              | 20 ++++++++++++++++++++
 2 files changed, 29 insertions(+)
include/linux/io_uring/cmd.h +9 -0
index de3f550598cf..6ff5c0386d0a 100644
@@ -100,6 +100,9 @@ int io_uring_buf_ring_unpin(struct io_uring_cmd *cmd, unsigned buf_group,
 int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
 			   u64 addr, unsigned int len, unsigned int bid,
 			   unsigned int issue_flags);
+
+bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, unsigned int buf_group,
+			    unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -174,6 +177,12 @@ static inline int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd,
 {
 	return -EOPNOTSUPP;
 }
+static inline bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd,
+					  unsigned int buf_group,
+					  unsigned int issue_flags)
+{
+	return false;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
io_uring/kbuf.c +20 -0
index a7d7d2c6b42c..d86199ac3377 100644
@@ -961,3 +961,23 @@ int io_register_kmbuf_ring(struct io_ring_ctx *ctx, void __user *arg)
 
 	return 0;
 }
+
+bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, unsigned int buf_group,
+			    unsigned int issue_flags)
+{
+	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+	struct io_buffer_list *bl;
+	bool is_kmbuf_ring = false;
+
+	io_ring_submit_lock(ctx, issue_flags);
+
+	bl = io_buffer_get_list(ctx, buf_group);
+	if (likely(bl) && (bl->flags & IOBL_KERNEL_MANAGED)) {
+		WARN_ON_ONCE(!(bl->flags & IOBL_BUF_RING));
+		is_kmbuf_ring = true;
+	}
+
+	io_ring_submit_unlock(ctx, issue_flags);
+	return is_kmbuf_ring;
+}
+EXPORT_SYMBOL_GPL(io_uring_is_kmbuf_ring);
Signature · 1 line
2.47.3
author maintainer
#23 lore raw

PATCH v4 10/25 io_uring/kbuf: export io_ring_buffer_select()

In reply to Joanne Koong

Export io_ring_buffer_select() so that it may be used by callers who pass in a pinned bufring without needing to grab the io_uring mutex.

This is a preparatory patch that will be needed by fuse io-uring, which will need to select a buffer from a kernel-managed bufring while the uring mutex may already be held by in-progress commits, and may need to select a buffer in atomic contexts.

---
 include/linux/io_uring/buf.h | 25 +++++++++++++++++++++++++
 io_uring/kbuf.c              |  8 +++++---
 2 files changed, 30 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/io_uring/buf.h
include/linux/io_uring/buf.h new file +25 -0
new file mode 100644
index 000000000000..3f7426ced3eb
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _LINUX_IO_URING_BUF_H
+#define _LINUX_IO_URING_BUF_H
+
+#include <linux/io_uring_types.h>
+
+#if defined(CONFIG_IO_URING)
+struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
+				       struct io_buffer_list *bl,
+				       unsigned int issue_flags);
+#else
+static inline struct io_br_sel io_ring_buffer_select(struct io_kiocb *req,
+						     size_t *len,
+						     struct io_buffer_list *bl,
+						     unsigned int issue_flags)
+{
+	struct io_br_sel sel = {
+		.val = -EOPNOTSUPP,
+	};
+
+	return sel;
+}
+#endif /* CONFIG_IO_URING */
+
+#endif /* _LINUX_IO_URING_BUF_H */
io_uring/kbuf.c +5 -3
index d86199ac3377..d2a660ecc7ac 100644
@@ -9,6 +9,7 @@
 #include <linux/poll.h>
 #include <linux/vmalloc.h>
 #include <linux/io_uring.h>
+#include <linux/io_uring/buf.h>
 #include <linux/io_uring/cmd.h>
 
 #include <uapi/linux/io_uring.h>
@@ -226,9 +227,9 @@ static bool io_should_commit(struct io_kiocb *req, struct io_buffer_list *bl,
 	return false;
 }
 
-static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
-					      struct io_buffer_list *bl,
-					      unsigned int issue_flags)
+struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
+				       struct io_buffer_list *bl,
+				       unsigned int issue_flags)
 {
 	struct io_uring_buf_ring *br = bl->buf_ring;
 	__u16 tail, head = bl->head;
@@ -261,6 +262,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 	}
 	return sel;
 }
+EXPORT_SYMBOL_GPL(io_ring_buffer_select);
 
 struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,
 				  unsigned buf_group, unsigned int issue_flags)
Signature · 1 line
2.47.3
author maintainer
#24 lore raw

PATCH v4 11/25 io_uring/kbuf: return buffer id in buffer selection

In reply to Joanne Koong

Return the id of the selected buffer in io_buffer_select(). This is needed for kernel-managed buffer rings to later recycle the selected buffer.

---
 include/linux/io_uring/cmd.h   | 2 +-
 include/linux/io_uring_types.h | 2 ++
 io_uring/kbuf.c                | 7 +++++--
 3 files changed, 8 insertions(+), 3 deletions(-)
include/linux/io_uring/cmd.h +1 -1
index 6ff5c0386d0a..8881fb8da5e6 100644
@@ -79,7 +79,7 @@ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd);
 
 /*
  * Select a buffer from the provided buffer group for multishot uring_cmd.
- * Returns the selected buffer address and size.
+ * Returns the selected buffer address, size, and id.
  */
 struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,
 					    unsigned buf_group, size_t *len,
include/linux/io_uring_types.h +2 -0
index 0b8880cdda8b..157eaf92d893 100644
@@ -100,6 +100,8 @@ struct io_br_sel {
 		void *kaddr;
 	};
 	ssize_t val;
+	/* id of the selected buffer */
+	unsigned buf_id;
 };
 
 
io_uring/kbuf.c +5 -2
index d2a660ecc7ac..ada4bdaae79d 100644
@@ -251,6 +251,7 @@ struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
 	req->flags |= REQ_F_BUFFER_RING | REQ_F_BUFFERS_COMMIT;
 	req->buf_index = READ_ONCE(buf->bid);
 	sel.buf_list = bl;
+	sel.buf_id = req->buf_index;
 	if (bl->flags & IOBL_KERNEL_MANAGED)
 		sel.kaddr = (void *)(uintptr_t)READ_ONCE(buf->addr);
 	else
@@ -275,10 +276,12 @@ struct io_br_sel io_buffer_select(struct io_kiocb *req, size_t *len,
 
 	bl = io_buffer_get_list(ctx, buf_group);
 	if (likely(bl)) {
-		if (bl->flags & IOBL_BUF_RING)
+		if (bl->flags & IOBL_BUF_RING) {
 			sel = io_ring_buffer_select(req, len, bl, issue_flags);
-		else
+		} else {
 			sel.addr = io_provided_buffer_select(req, len, bl);
+			sel.buf_id = req->buf_index;
+		}
 	}
 	io_ring_submit_unlock(req->ctx, issue_flags);
 	return sel;
Signature · 1 line
2.47.3
author maintainer
#25 lore raw

PATCH v4 12/25 io_uring/cmd: set selected buffer index in __io_uring_cmd_done()

In reply to Joanne Koong

When uring_cmd operations select a buffer, the completion queue entry should indicate which buffer was selected.

Set IORING_CQE_F_BUFFER on the completed entry and encode the buffer index if a buffer was selected.

This will be needed for fuse, which needs to relay to userspace which selected buffer contains the data.

---
 io_uring/uring_cmd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/uring_cmd.c +5 -1
index 197474911f04..8eaea40231ff 100644
@@ -142,6 +142,7 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,
 		       unsigned issue_flags, bool is_cqe32)
 {
 	struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
+	u32 cflags = 0;
 
 	if (WARN_ON_ONCE(req->flags & REQ_F_APOLL_MULTISHOT))
 		return;
@@ -151,7 +152,10 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,
 	if (ret < 0)
 		req_set_fail(req);
 
-	io_req_set_res(req, ret, 0);
+	if (req->flags & (REQ_F_BUFFER_SELECTED | REQ_F_BUFFER_RING))
+		cflags |= IORING_CQE_F_BUFFER |
+			(req->buf_index << IORING_CQE_BUFFER_SHIFT);
+	io_req_set_res(req, ret, cflags);
 	if (is_cqe32) {
 		if (req->ctx->flags & IORING_SETUP_CQE_MIXED)
 			req->cqe.flags |= IORING_CQE_F_32;
Signature · 1 line
2.47.3
author maintainer
#26 lore raw

PATCH v4 13/25 fuse: refactor io-uring logic for getting next fuse request

In reply to Joanne Koong

Simplify the logic for getting the next fuse request.

---
 fs/fuse/dev_uring.c | 78 ++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
index 5ceb217ced1b..1efee4391af5 100644
@@ -714,34 +714,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 	return err;
 }
 
-/*
- * Write data to the ring buffer and send the request to userspace,
- * userspace will read it
- * This is comparable with classical read(/dev/fuse)
- */
-static int fuse_uring_send_next_to_ring(struct fuse_ring_ent *ent,
-					struct fuse_req *req,
-					unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-	int err;
-	struct io_uring_cmd *cmd;
-
-	err = fuse_uring_prepare_send(ent, req);
-	if (err)
-		return err;
-
-	spin_lock(&queue->lock);
-	cmd = ent->cmd;
-	ent->cmd = NULL;
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, 0, issue_flags);
-	return 0;
-}
-
 /*
  * Make a ring entry available for fuse_req assignment
  */
@@ -838,11 +810,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 }
 
 /*
- * Get the next fuse req and send it
+ * Get the next fuse req.
+ *
+ * Returns true if the next fuse request has been assigned to the ent.
+ * Else, there is no next fuse request and this returns false.
  */
-static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
-				     struct fuse_ring_queue *queue,
-				     unsigned int issue_flags)
+static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
+					 struct fuse_ring_queue *queue)
 {
 	int err;
 	struct fuse_req *req;
@@ -854,10 +828,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
 	spin_unlock(&queue->lock);
 
 	if (req) {
-		err = fuse_uring_send_next_to_ring(ent, req, issue_flags);
+		err = fuse_uring_prepare_send(ent, req);
 		if (err)
 			goto retry;
 	}
+
+	return req != NULL;
 }
 
 static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
@@ -875,6 +851,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
 	return 0;
 }
 
+static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
+			    ssize_t ret, unsigned int issue_flags)
+{
+	struct fuse_ring_queue *queue = ent->queue;
+
+	spin_lock(&queue->lock);
+	ent->state = FRRS_USERSPACE;
+	list_move_tail(&ent->list, &queue->ent_in_userspace);
+	ent->cmd = NULL;
+	spin_unlock(&queue->lock);
+
+	io_uring_cmd_done(cmd, ret, issue_flags);
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -946,7 +936,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	fuse_uring_next_fuse_req(ent, queue, issue_flags);
+	if (fuse_uring_get_next_fuse_req(ent, queue))
+		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
 
@@ -1194,20 +1185,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
 	return -EIOCBQUEUED;
 }
 
-static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
-			    ssize_t ret, unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-
-	spin_lock(&queue->lock);
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	ent->cmd = NULL;
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, ret, issue_flags);
-}
-
 /*
  * This prepares and sends the ring request in fuse-uring task context.
  * User buffers are not mapped yet - the application does not have permission
@@ -1224,8 +1201,9 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
 	if (!tw.cancel) {
 		err = fuse_uring_prepare_send(ent, ent->fuse_req);
 		if (err) {
-			fuse_uring_next_fuse_req(ent, queue, issue_flags);
-			return;
+			if (!fuse_uring_get_next_fuse_req(ent, queue))
+				return;
+			err = 0;
 		}
 	} else {
 		err = -ECANCELED;
Signature · 1 line
2.47.3
author maintainer
#27 lore raw

PATCH v4 14/25 fuse: refactor io-uring header copying to ring

In reply to Joanne Koong

Move header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------
 1 file changed, 21 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +21 -18
index 1efee4391af5..7962a9876031 100644
@@ -575,6 +575,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static __always_inline int copy_header_to_ring(void __user *ring,
+					       const void *header,
+					       size_t header_size)
+{
+	if (copy_to_user(ring, header, header_size)) {
+		pr_info_ratelimited("Copying header to ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -637,13 +649,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_to_user(&ent->headers->op_in, in_args->value,
-					   in_args->size);
-			if (err) {
-				pr_info_ratelimited(
-					"Copying the header failed.\n");
-				return -EFAULT;
-			}
+			err = copy_header_to_ring(&ent->headers->op_in,
+						  in_args->value,
+						  in_args->size);
+			if (err)
+				return err;
 		}
 		in_args++;
 		num_args--;
@@ -659,9 +669,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	err = copy_to_user(&ent->headers->ring_ent_in_out, &ent_in_out,
-			   sizeof(ent_in_out));
-	return err ? -EFAULT : 0;
+	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+				   sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -690,14 +699,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	err = copy_to_user(&ent->headers->in_out, &req->in.h,
-			   sizeof(req->in.h));
-	if (err) {
-		err = -EFAULT;
-		return err;
-	}
-
-	return 0;
+	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+				   sizeof(req->in.h));
 }
 
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.47.3
author maintainer
#28 lore raw

PATCH v4 15/25 fuse: refactor io-uring header copying from ring

In reply to Joanne Koong

Move header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +18 -6
index 7962a9876031..e8ee51bfa5fc 100644
@@ -587,6 +587,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,
 	return 0;
 }
 
+static __always_inline int copy_header_from_ring(void *header,
+						 const void __user *ring,
+						 size_t header_size)
+{
+	if (copy_from_user(header, ring, header_size)) {
+		pr_info_ratelimited("Copying header from ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -597,10 +609,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_from_user(&ring_in_out, &ent->headers->ring_ent_in_out,
-			     sizeof(ring_in_out));
+	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
+				    sizeof(ring_in_out));
 	if (err)
-		return -EFAULT;
+		return err;
 
 	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
 			  &iter);
@@ -794,10 +806,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 
-	err = copy_from_user(&req->out.h, &ent->headers->in_out,
-			     sizeof(req->out.h));
+	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+				    sizeof(req->out.h));
 	if (err) {
-		req->out.h.error = -EFAULT;
+		req->out.h.error = err;
 		goto out;
 	}
 
Signature · 1 line
2.47.3
Reviewed-by on v4 15/25
#29 lore raw

Re: [PATCH v4 15/25] fuse: refactor io-uring header copying from ring

Quoted 49 lines · Joanne Koong

Move header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +15 -3
index 7962a9876031..e8ee51bfa5fc 100644
@@ -587,6 +587,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,
 	return 0;
 }
 
+static __always_inline int copy_header_from_ring(void *header,
+						 const void __user *ring,
+						 size_t header_size)
+{
+	if (copy_from_user(header, ring, header_size)) {
+		pr_info_ratelimited("Copying header from ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -597,10 +609,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_from_user(&ring_in_out, &ent->headers->ring_ent_in_out,
-			     sizeof(ring_in_out));
+	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
+				    sizeof(ring_in_out));
 	if (err)
-		return -EFAULT;
+		return err;
 
 	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
 			  &iter);
@@ -794,10 +806,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 

fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 815

Quoted 10 lines
fs/fuse/dev_uring.c +3 -3
-	err = copy_from_user(&req->out.h, &ent->headers->in_out,
-			     sizeof(req->out.h));
+	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+				    sizeof(req->out.h));
 	if (err) {
-		req->out.h.error = -EFAULT;
+		req->out.h.error = err;
 		goto out;
 	}
 
author maintainer
#30 lore raw

PATCH v4 16/25 fuse: use enum types for header copying

In reply to Joanne Koong

Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when kernel-managed buffer rings are added.

---
 fs/fuse/dev_uring.c | 57 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 47 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +47 -10
index e8ee51bfa5fc..d16f6b3489c1 100644
@@ -31,6 +31,15 @@ struct fuse_uring_pdu {
 
 static const struct fuse_iqueue_ops fuse_io_uring_ops;
 
+enum fuse_uring_header_type {
+	/* struct fuse_in_header / struct fuse_out_header */
+	FUSE_URING_HEADER_IN_OUT,
+	/* per op code header */
+	FUSE_URING_HEADER_OP,
+	/* struct fuse_uring_ent_in_out header */
+	FUSE_URING_HEADER_RING_ENT,
+};
+
 static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
 				   struct fuse_ring_ent *ring_ent)
 {
@@ -575,10 +584,32 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
-static __always_inline int copy_header_to_ring(void __user *ring,
+static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
+					 enum fuse_uring_header_type type)
+{
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		return &ent->headers->in_out;
+	case FUSE_URING_HEADER_OP:
+		return &ent->headers->op_in;
+	case FUSE_URING_HEADER_RING_ENT:
+		return &ent->headers->ring_ent_in_out;
+	}
+
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
+static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,
+					       enum fuse_uring_header_type type,
 					       const void *header,
 					       size_t header_size)
 {
+	void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_to_user(ring, header, header_size)) {
 		pr_info_ratelimited("Copying header to ring failed.\n");
 		return -EFAULT;
@@ -587,10 +618,16 @@ static __always_inline int copy_header_to_ring(void __user *ring,
 	return 0;
 }
 
-static __always_inline int copy_header_from_ring(void *header,
-						 const void __user *ring,
+static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
+						 enum fuse_uring_header_type type,
+						 void *header,
 						 size_t header_size)
 {
+	const void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_from_user(header, ring, header_size)) {
 		pr_info_ratelimited("Copying header from ring failed.\n");
 		return -EFAULT;
@@ -609,8 +646,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
-				    sizeof(ring_in_out));
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				    &ring_in_out, sizeof(ring_in_out));
 	if (err)
 		return err;
 
@@ -661,7 +698,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_header_to_ring(&ent->headers->op_in,
+			err = copy_header_to_ring(ent, FUSE_URING_HEADER_OP,
 						  in_args->value,
 						  in_args->size);
 			if (err)
@@ -681,8 +718,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
-				   sizeof(ent_in_out));
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				   &ent_in_out, sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -711,7 +748,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,
 				   sizeof(req->in.h));
 }
 
@@ -806,7 +843,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 
-	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
 				    sizeof(req->out.h));
 	if (err) {
 		req->out.h.error = err;
Signature · 1 line
2.47.3
author maintainer
#31 lore raw

PATCH v4 17/25 fuse: refactor setting up copy state for payload copying

In reply to Joanne Koong

Add a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.

---
 fs/fuse/dev_uring.c | 38 ++++++++++++++++++++++++--------------
 1 file changed, 24 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +24 -14
index d16f6b3489c1..b57871f92d08 100644
@@ -636,6 +636,27 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
 	return 0;
 }
 
+static int setup_fuse_copy_state(struct fuse_copy_state *cs,
+				 struct fuse_ring *ring, struct fuse_req *req,
+				 struct fuse_ring_ent *ent, int dir,
+				 struct iov_iter *iter)
+{
+	int err;
+
+	err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
+	if (err) {
+		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+		return err;
+	}
+
+	fuse_copy_init(cs, dir == ITER_DEST, iter);
+
+	cs->is_uring = true;
+	cs->req = req;
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -651,15 +672,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	if (err)
 		return err;
 
-	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
-			  &iter);
+	err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter);
 	if (err)
 		return err;
 
-	fuse_copy_init(&cs, false, &iter);
-	cs.is_uring = true;
-	cs.req = req;
-
 	err = fuse_copy_out_args(&cs, args, ring_in_out.payload_sz);
 	fuse_copy_finish(&cs);
 	return err;
@@ -682,15 +698,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		.commit_id = req->in.h.unique,
 	};
 
-	err = import_ubuf(ITER_DEST, ent->payload, ring->max_payload_sz, &iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+	err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter);
+	if (err)
 		return err;
-	}
-
-	fuse_copy_init(&cs, true, &iter);
-	cs.is_uring = true;
-	cs.req = req;
 
 	if (num_args > 0) {
 		/*
Signature · 1 line
2.47.3
author maintainer
#32 lore raw

PATCH v4 18/25 fuse: support buffer copying for kernel addresses

In reply to Joanne Koong

This is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.

Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.

---
 fs/fuse/dev.c        | 23 +++++++++++++++++------
 fs/fuse/fuse_dev_i.h |  7 ++++++-
 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
index 6d59cbc877c6..ceb5d6a553c0 100644
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,
 /* Unmap and put previous page of userspace buffer */
 void fuse_copy_finish(struct fuse_copy_state *cs)
 {
+	if (cs->is_kaddr)
+		return;
+
 	if (cs->currbuf) {
 		struct pipe_buffer *buf = cs->currbuf;
 
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
 	struct page *page;
 	int err;
 
+	if (cs->is_kaddr)
+		return 0;
+
 	err = unlock_request(cs->req);
 	if (err)
 		return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 {
 	unsigned ncpy = min(*size, cs->len);
 	if (val) {
-		void *pgaddr = kmap_local_page(cs->pg);
-		void *buf = pgaddr + cs->offset;
+		void *pgaddr, *buf;
+		if (!cs->is_kaddr) {
+			pgaddr = kmap_local_page(cs->pg);
+			buf = pgaddr + cs->offset;
+		} else {
+			buf = cs->kaddr + cs->offset;
+		}
 
 		if (cs->write)
 			memcpy(buf, *val, ncpy);
 		else
 			memcpy(*val, buf, ncpy);
-
-		kunmap_local(pgaddr);
+		if (!cs->is_kaddr)
+			kunmap_local(pgaddr);
 		*val += ncpy;
 	}
 	*size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
 	}
 
 	while (count) {
-		if (cs->write && cs->pipebufs && folio) {
+		if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {
 			/*
 			 * Can't control lifetime of pipe buffers, so always
 			 * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
 			} else {
 				return fuse_ref_folio(cs, folio, offset, count);
 			}
-		} else if (!cs->len) {
+		} else if (!cs->len && !cs->is_kaddr) {
 			if (cs->move_folios && folio &&
 			    offset == 0 && count == size) {
 				err = fuse_try_move_folio(cs, foliop);
fs/fuse/fuse_dev_i.h +6 -1
index 134bf44aff0d..aa1d25421054 100644
@@ -28,12 +28,17 @@ struct fuse_copy_state {
 	struct pipe_buffer *currbuf;
 	struct pipe_inode_info *pipe;
 	unsigned long nr_segs;
-	struct page *pg;
+	union {
+		struct page *pg;
+		void *kaddr;
+	};
 	unsigned int len;
 	unsigned int offset;
 	bool write:1;
 	bool move_folios:1;
 	bool is_uring:1;
+	/* if set, use kaddr; otherwise use pg */
+	bool is_kaddr:1;
 	struct {
 		unsigned int copied_sz; /* copied size into the user buffer */
 	} ring;
Signature · 1 line
2.47.3
#33 lore raw

Re: [PATCH v4 18/25] fuse: support buffer copying for kernel addresses

Quoted 94 lines · Joanne Koong

This is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.

Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.

---
 fs/fuse/dev.c        | 23 +++++++++++++++++------
 fs/fuse/fuse_dev_i.h |  7 ++++++-
 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
index 6d59cbc877c6..ceb5d6a553c0 100644
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,
 /* Unmap and put previous page of userspace buffer */
 void fuse_copy_finish(struct fuse_copy_state *cs)
 {
+	if (cs->is_kaddr)
+		return;
+
 	if (cs->currbuf) {
 		struct pipe_buffer *buf = cs->currbuf;
 
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
 	struct page *page;
 	int err;
 
+	if (cs->is_kaddr)
+		return 0;
+
 	err = unlock_request(cs->req);
 	if (err)
 		return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 {
 	unsigned ncpy = min(*size, cs->len);
 	if (val) {
-		void *pgaddr = kmap_local_page(cs->pg);
-		void *buf = pgaddr + cs->offset;
+		void *pgaddr, *buf;
+		if (!cs->is_kaddr) {
+			pgaddr = kmap_local_page(cs->pg);
+			buf = pgaddr + cs->offset;
+		} else {
+			buf = cs->kaddr + cs->offset;
+		}
 
 		if (cs->write)
 			memcpy(buf, *val, ncpy);
 		else
 			memcpy(*val, buf, ncpy);
-
-		kunmap_local(pgaddr);
+		if (!cs->is_kaddr)
+			kunmap_local(pgaddr);
 		*val += ncpy;
 	}
 	*size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
 	}
 
 	while (count) {
-		if (cs->write && cs->pipebufs && folio) {
+		if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {
 			/*
 			 * Can't control lifetime of pipe buffers, so always
 			 * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
 			} else {
 				return fuse_ref_folio(cs, folio, offset, count);
 			}
-		} else if (!cs->len) {
+		} else if (!cs->len && !cs->is_kaddr) {
 			if (cs->move_folios && folio &&
 			    offset == 0 && count == size) {
 				err = fuse_try_move_folio(cs, foliop);
fs/fuse/fuse_dev_i.h +4 -1
index 134bf44aff0d..aa1d25421054 100644
@@ -28,12 +28,17 @@ struct fuse_copy_state {
 	struct pipe_buffer *currbuf;
 	struct pipe_inode_info *pipe;
 	unsigned long nr_segs;
-	struct page *pg;
+	union {
+		struct page *pg;
+		void *kaddr;
+	};

fs/fuse/fuse_dev_i.h struct fuse_copy_state { · line 44

Quoted 10 lines
fs/fuse/fuse_dev_i.h +2 -0
 	unsigned int len;
 	unsigned int offset;
 	bool write:1;
 	bool move_folios:1;
 	bool is_uring:1;
+	/* if set, use kaddr; otherwise use pg */
+	bool is_kaddr:1;
 	struct {
 		unsigned int copied_sz; /* copied size into the user buffer */
 	} ring;

I'm confused here, how cs->len will get initialized. So far that was done from fuse_copy_fill?

Thanks,
Bernd

author maintainer
#34 lore raw

Re: [PATCH v4 18/25] fuse: support buffer copying for kernel addresses

Quoted 102 lines · Bernd Schubert
Quoted 100 lines · Joanne Koong

This is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.

Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.

---
 fs/fuse/dev.c        | 23 +++++++++++++++++------
 fs/fuse/fuse_dev_i.h |  7 ++++++-
 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
index 6d59cbc877c6..ceb5d6a553c0 100644
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,
 /* Unmap and put previous page of userspace buffer */
 void fuse_copy_finish(struct fuse_copy_state *cs)
 {
+     if (cs->is_kaddr)
+             return;
+
      if (cs->currbuf) {
              struct pipe_buffer *buf = cs->currbuf;
 
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
      struct page *page;
      int err;
 
+     if (cs->is_kaddr)
+             return 0;
+
      err = unlock_request(cs->req);
      if (err)
              return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 {
      unsigned ncpy = min(*size, cs->len);
      if (val) {
-             void *pgaddr = kmap_local_page(cs->pg);
-             void *buf = pgaddr + cs->offset;
+             void *pgaddr, *buf;
+             if (!cs->is_kaddr) {
+                     pgaddr = kmap_local_page(cs->pg);
+                     buf = pgaddr + cs->offset;
+             } else {
+                     buf = cs->kaddr + cs->offset;
+             }
 
              if (cs->write)
                      memcpy(buf, *val, ncpy);
              else
                      memcpy(*val, buf, ncpy);
-
-             kunmap_local(pgaddr);
+             if (!cs->is_kaddr)
+                     kunmap_local(pgaddr);
              *val += ncpy;
      }
      *size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
      }
 
      while (count) {
-             if (cs->write && cs->pipebufs && folio) {
+             if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {
                      /*
                       * Can't control lifetime of pipe buffers, so always
                       * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
                      } else {
                              return fuse_ref_folio(cs, folio, offset, count);
                      }
-             } else if (!cs->len) {
+             } else if (!cs->len && !cs->is_kaddr) {
                      if (cs->move_folios && folio &&
                          offset == 0 && count == size) {
                              err = fuse_try_move_folio(cs, foliop);
fs/fuse/fuse_dev_i.h +5 -1
index 134bf44aff0d..aa1d25421054 100644
@@ -28,12 +28,17 @@ struct fuse_copy_state {
      struct pipe_buffer *currbuf;
      struct pipe_inode_info *pipe;
      unsigned long nr_segs;
-     struct page *pg;
+     union {
+             struct page *pg;
+             void *kaddr;
+     };
      unsigned int len;
      unsigned int offset;
      bool write:1;
      bool move_folios:1;
      bool is_uring:1;
+     /* if set, use kaddr; otherwise use pg */
Quoted 8 lines

fs/fuse/fuse_dev_i.h struct fuse_copy_state { · line 44

Quoted 4 lines
fs/fuse/fuse_dev_i.h +1 -0
+     bool is_kaddr:1;
      struct {
              unsigned int copied_sz; /* copied size into the user buffer */
      } ring;

I'm confused here, how cs->len will get initialized. So far that was done from fuse_copy_fill?

With kaddrs, cs->len is initialized when the copy state is set up (in setup_fuse_copy_state()) before we do any copying to/from the ring. The changes for that are in the later patch that adds the ringbuffer logic ("fuse: add io-uring kernel-managed buffer ring"). The kaddr and len correspond to the address and length of the buffer that was selected from the ring buffer (in fuse_uring_select_buffer()).

Thanks,
Joanne

Quoted 6 lines

Thanks,
Bernd

maintainer Reviewed-by
#35 lore raw

Re: [PATCH v4 18/25] fuse: support buffer copying for kernel addresses

Quoted 112 lines · Joanne Koong
Quoted 102 lines · Bernd Schubert
Quoted 100 lines · Joanne Koong

This is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.

Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.

---
 fs/fuse/dev.c        | 23 +++++++++++++++++------
 fs/fuse/fuse_dev_i.h |  7 ++++++-
 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
index 6d59cbc877c6..ceb5d6a553c0 100644
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,
 /* Unmap and put previous page of userspace buffer */
 void fuse_copy_finish(struct fuse_copy_state *cs)
 {
+     if (cs->is_kaddr)
+             return;
+
      if (cs->currbuf) {
              struct pipe_buffer *buf = cs->currbuf;
 
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
      struct page *page;
      int err;
 
+     if (cs->is_kaddr)
+             return 0;
+
      err = unlock_request(cs->req);
      if (err)
              return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 {
      unsigned ncpy = min(*size, cs->len);
      if (val) {
-             void *pgaddr = kmap_local_page(cs->pg);
-             void *buf = pgaddr + cs->offset;
+             void *pgaddr, *buf;
+             if (!cs->is_kaddr) {
+                     pgaddr = kmap_local_page(cs->pg);
+                     buf = pgaddr + cs->offset;
+             } else {
+                     buf = cs->kaddr + cs->offset;
+             }
 
              if (cs->write)
                      memcpy(buf, *val, ncpy);
              else
                      memcpy(*val, buf, ncpy);
-
-             kunmap_local(pgaddr);
+             if (!cs->is_kaddr)
+                     kunmap_local(pgaddr);
              *val += ncpy;
      }
      *size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
      }
 
      while (count) {
-             if (cs->write && cs->pipebufs && folio) {
+             if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {
                      /*
                       * Can't control lifetime of pipe buffers, so always
                       * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
                      } else {
                              return fuse_ref_folio(cs, folio, offset, count);
                      }
-             } else if (!cs->len) {
+             } else if (!cs->len && !cs->is_kaddr) {
                      if (cs->move_folios && folio &&
                          offset == 0 && count == size) {
                              err = fuse_try_move_folio(cs, foliop);
fs/fuse/fuse_dev_i.h +5 -1
index 134bf44aff0d..aa1d25421054 100644
@@ -28,12 +28,17 @@ struct fuse_copy_state {
      struct pipe_buffer *currbuf;
      struct pipe_inode_info *pipe;
      unsigned long nr_segs;
-     struct page *pg;
+     union {
+             struct page *pg;
+             void *kaddr;
+     };
      unsigned int len;
      unsigned int offset;
      bool write:1;
      bool move_folios:1;
      bool is_uring:1;
+     /* if set, use kaddr; otherwise use pg */
Quoted 8 lines

fs/fuse/fuse_dev_i.h struct fuse_copy_state { · line 44

Quoted 4 lines
fs/fuse/fuse_dev_i.h +1 -0
+     bool is_kaddr:1;
      struct {
              unsigned int copied_sz; /* copied size into the user buffer */
      } ring;

I'm confused here, how cs->len will get initialized. So far that was done from fuse_copy_fill?

Quoted 6 lines

With kaddrs, cs->len is initialized when the copy state is set up (in setup_fuse_copy_state()) before we do any copying to/from the ring. The changes for that are in the later patch that adds the ringbuffer logic ("fuse: add io-uring kernel-managed buffer ring"). The kaddr and len correspond to the address and length of the buffer that was selected from the ring buffer (in fuse_uring_select_buffer()).

Maybe we could add a sanity check into fuse_copy_do() or even into fuse_copy_fill in the cs->is_kaddr condition that cs->len is > 0?

Otherwise looks good.

author maintainer
#36 lore raw

Re: [PATCH v4 18/25] fuse: support buffer copying for kernel addresses

Quoted 125 lines · Bernd Schubert
Quoted 112 lines · Joanne Koong
Quoted 102 lines · Bernd Schubert
Quoted 100 lines · Joanne Koong

This is a preparatory patch needed to support kernel-managed ring buffers in fuse-over-io-uring. For kernel-managed ring buffers, we get the vmapped address of the buffer which we can directly use.

Currently, buffer copying in fuse only supports extracting underlying pages from an iov iter and kmapping them. This commit allows buffer copying to work directly on a kaddr.

---
 fs/fuse/dev.c        | 23 +++++++++++++++++------
 fs/fuse/fuse_dev_i.h |  7 ++++++-
 2 files changed, 23 insertions(+), 7 deletions(-)
fs/fuse/dev.c +17 -6
index 6d59cbc877c6..ceb5d6a553c0 100644
@@ -848,6 +848,9 @@ void fuse_copy_init(struct fuse_copy_state *cs, bool write,
 /* Unmap and put previous page of userspace buffer */
 void fuse_copy_finish(struct fuse_copy_state *cs)
 {
+     if (cs->is_kaddr)
+             return;
+
      if (cs->currbuf) {
              struct pipe_buffer *buf = cs->currbuf;
 
@@ -873,6 +876,9 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
      struct page *page;
      int err;
 
+     if (cs->is_kaddr)
+             return 0;
+
      err = unlock_request(cs->req);
      if (err)
              return err;
@@ -931,15 +937,20 @@ static int fuse_copy_do(struct fuse_copy_state *cs, void **val, unsigned *size)
 {
      unsigned ncpy = min(*size, cs->len);
      if (val) {
-             void *pgaddr = kmap_local_page(cs->pg);
-             void *buf = pgaddr + cs->offset;
+             void *pgaddr, *buf;
+             if (!cs->is_kaddr) {
+                     pgaddr = kmap_local_page(cs->pg);
+                     buf = pgaddr + cs->offset;
+             } else {
+                     buf = cs->kaddr + cs->offset;
+             }
 
              if (cs->write)
                      memcpy(buf, *val, ncpy);
              else
                      memcpy(*val, buf, ncpy);
-
-             kunmap_local(pgaddr);
+             if (!cs->is_kaddr)
+                     kunmap_local(pgaddr);
              *val += ncpy;
      }
      *size -= ncpy;
@@ -1127,7 +1138,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
      }
 
      while (count) {
-             if (cs->write && cs->pipebufs && folio) {
+             if (cs->write && cs->pipebufs && folio && !cs->is_kaddr) {
                      /*
                       * Can't control lifetime of pipe buffers, so always
                       * copy user pages.
@@ -1139,7 +1150,7 @@ static int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
                      } else {
                              return fuse_ref_folio(cs, folio, offset, count);
                      }
-             } else if (!cs->len) {
+             } else if (!cs->len && !cs->is_kaddr) {
                      if (cs->move_folios && folio &&
                          offset == 0 && count == size) {
                              err = fuse_try_move_folio(cs, foliop);
fs/fuse/fuse_dev_i.h +5 -1
index 134bf44aff0d..aa1d25421054 100644
@@ -28,12 +28,17 @@ struct fuse_copy_state {
      struct pipe_buffer *currbuf;
      struct pipe_inode_info *pipe;
      unsigned long nr_segs;
-     struct page *pg;
+     union {
+             struct page *pg;
+             void *kaddr;
+     };
      unsigned int len;
      unsigned int offset;
      bool write:1;
      bool move_folios:1;
      bool is_uring:1;
+     /* if set, use kaddr; otherwise use pg */
Quoted 8 lines

fs/fuse/fuse_dev_i.h struct fuse_copy_state { · line 44

Quoted 4 lines
fs/fuse/fuse_dev_i.h +1 -0
+     bool is_kaddr:1;
      struct {
              unsigned int copied_sz; /* copied size into the user buffer */
      } ring;

I'm confused here, how cs->len will get initialized. So far that was done from fuse_copy_fill?

Quoted 6 lines

With kaddrs, cs->len is initialized when the copy state is set up (in setup_fuse_copy_state()) before we do any copying to/from the ring. The changes for that are in the later patch that adds the ringbuffer logic ("fuse: add io-uring kernel-managed buffer ring"). The kaddr and len correspond to the address and length of the buffer that was selected from the ring buffer (in fuse_uring_select_buffer()).

Quoted 2 lines

Maybe we could add a sanity check into fuse_copy_do() or even into fuse_copy_fill in the cs->is_kaddr condition that cs->len is > 0?

I will add a WARN_ON for this in the next version. Thanks for reviewing this.

Thanks,
Joanne

Quoted 4 lines

Otherwise looks good.

author maintainer
#37 lore raw

PATCH v4 19/25 fuse: add io-uring kernel-managed buffer ring

In reply to Joanne Koong

Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.

This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction

b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.

Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.

---
 fs/fuse/dev_uring.c       | 412 ++++++++++++++++++++++++++++++++------
 fs/fuse/dev_uring_i.h     |  31 ++-
 include/uapi/linux/fuse.h |  15 +-
 3 files changed, 389 insertions(+), 69 deletions(-)
fs/fuse/dev_uring.c +347 -65
index b57871f92d08..40e8c2e6b77c 100644
@@ -10,6 +10,8 @@
 #include "fuse_trace.h"
 
 #include <linux/fs.h>
+#include <linux/io_uring.h>
+#include <linux/io_uring/buf.h>
 #include <linux/io_uring/cmd.h>
 
 static bool __read_mostly enable_uring;
@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,
 
 #define FUSE_URING_IOV_SEGS 2 /* header and payload */
 
+#define FUSE_URING_RINGBUF_GROUP 0
+#define FUSE_URING_FIXED_HEADERS_OFFSET 0
 
 bool fuse_uring_enabled(void)
 {
@@ -276,20 +280,45 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)
 	return res;
 }
 
-static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
-						       int qid)
+static int fuse_uring_buf_ring_setup(struct io_uring_cmd *cmd,
+				     struct fuse_ring_queue *queue,
+				     unsigned int issue_flags)
+{
+	int err;
+
+	err = io_uring_buf_ring_pin(cmd, FUSE_URING_RINGBUF_GROUP, issue_flags,
+				    &queue->bufring);
+	if (err)
+		return err;
+
+	if (!io_uring_is_kmbuf_ring(cmd, FUSE_URING_RINGBUF_GROUP,
+				    issue_flags)) {
+		io_uring_buf_ring_unpin(cmd, FUSE_URING_RINGBUF_GROUP,
+					issue_flags);
+		return -EINVAL;
+	}
+
+	queue->use_bufring = true;
+
+	return 0;
+}
+
+static struct fuse_ring_queue *
+fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring,
+			int qid, bool use_bufring, unsigned int issue_flags)
 {
 	struct fuse_conn *fc = ring->fc;
 	struct fuse_ring_queue *queue;
 	struct list_head *pq;
+	int err;
 
 	queue = kzalloc(sizeof(*queue), GFP_KERNEL_ACCOUNT);
 	if (!queue)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	pq = kcalloc(FUSE_PQ_HASH_SIZE, sizeof(struct list_head), GFP_KERNEL);
 	if (!pq) {
 		kfree(queue);
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	}
 
 	queue->qid = qid;
@@ -307,6 +336,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
 	queue->fpq.processing = pq;
 	fuse_pqueue_init(&queue->fpq);
 
+	if (use_bufring) {
+		err = fuse_uring_buf_ring_setup(cmd, queue, issue_flags);
+		if (err) {
+			kfree(pq);
+			kfree(queue);
+			return ERR_PTR(err);
+		}
+	}
+
 	spin_lock(&fc->lock);
 	if (ring->queues[qid]) {
 		spin_unlock(&fc->lock);
@@ -584,6 +622,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static int get_kernel_ring_header(struct fuse_ring_ent *ent,
+				  enum fuse_uring_header_type type,
+				  struct iov_iter *headers_iter)
+{
+	size_t offset;
+
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		/* No offset - start of header */
+		offset = 0;
+		break;
+	case FUSE_URING_HEADER_OP:
+		offset = offsetof(struct fuse_uring_req_header, op_in);
+		break;
+	case FUSE_URING_HEADER_RING_ENT:
+		offset = offsetof(struct fuse_uring_req_header, ring_ent_in_out);
+		break;
+	default:
+		WARN_ONCE(1, "Invalid header type: %d\n", type);
+		return -EINVAL;
+	}
+
+	*headers_iter = ent->headers_iter;
+	if (offset)
+		iov_iter_advance(headers_iter, offset);
+
+	return 0;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 					 enum fuse_uring_header_type type)
 {
@@ -605,17 +672,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,
 					       const void *header,
 					       size_t header_size)
 {
-	void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err =  get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_to_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		void __user *ring = get_user_ring_header(ent, type);
+
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
 
-	if (copy_to_user(ring, header, header_size)) {
-		pr_info_ratelimited("Copying header to ring failed.\n");
-		return -EFAULT;
+		if (copy_to_user(ring, header, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header to ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
@@ -623,17 +711,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
 						 void *header,
 						 size_t header_size)
 {
-	const void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err = get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_from_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		const void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_from_user(header, ring, header_size)) {
-		pr_info_ratelimited("Copying header from ring failed.\n");
-		return -EFAULT;
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
+
+		if (copy_from_user(header, ring, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header from ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static int setup_fuse_copy_state(struct fuse_copy_state *cs,
@@ -643,14 +752,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
 {
 	int err;
 
-	err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
-		return err;
+	if (!ent->queue->use_bufring) {
+		err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
+		if (err) {
+			pr_info_ratelimited("fuse: Import of user buffer "
+					    "failed\n");
+			return err;
+		}
 	}
 
 	fuse_copy_init(cs, dir == ITER_DEST, iter);
 
+	if (ent->queue->use_bufring) {
+		cs->is_kaddr = true;
+		cs->len = ent->payload_kvec.iov_len;
+		cs->kaddr = ent->payload_kvec.iov_base;
+	}
+
 	cs->is_uring = true;
 	cs->req = req;
 
@@ -762,6 +880,94 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 				   sizeof(req->in.h));
 }
 
+static bool fuse_uring_req_has_payload(struct fuse_req *req)
+{
+	struct fuse_args *args = req->args;
+
+	return args->in_numargs > 1 || args->out_numargs;
+}
+
+static int fuse_uring_select_buffer(struct fuse_ring_ent *ent,
+				    unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct io_br_sel sel;
+	size_t len = 0;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	/* Get a buffer to use for the payload */
+	sel = io_ring_buffer_select(cmd_to_io_kiocb(ent->cmd), &len,
+				    ent->queue->bufring, issue_flags);
+	if (sel.val)
+		return sel.val;
+	if (!sel.kaddr)
+		return -ENOENT;
+
+	ent->payload_kvec.iov_base = sel.kaddr;
+	ent->payload_kvec.iov_len = len;
+	ent->ringbuf_buf_id = sel.buf_id;
+
+	return 0;
+}
+
+static void fuse_uring_clean_up_buffer(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct kvec *kvec = &ent->payload_kvec;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	if (!ent->queue->use_bufring || !kvec->iov_base)
+		return;
+
+	WARN_ON_ONCE(io_uring_kmbuf_recycle(ent->cmd, FUSE_URING_RINGBUF_GROUP,
+					    (u64)kvec->iov_base, kvec->iov_len,
+					    ent->ringbuf_buf_id, issue_flags));
+
+	memset(kvec, 0, sizeof(*kvec));
+}
+
+static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
+					     struct fuse_req *req,
+					     unsigned int issue_flags)
+{
+	bool buffer_selected;
+	bool has_payload;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	buffer_selected = ent->payload_kvec.iov_base != NULL;
+	has_payload = fuse_uring_req_has_payload(req);
+
+	if (has_payload && !buffer_selected)
+		return fuse_uring_select_buffer(ent, issue_flags);
+
+	if (!has_payload && buffer_selected)
+		fuse_uring_clean_up_buffer(ent, issue_flags);
+
+	return 0;
+}
+
+static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
+				  struct fuse_req *req, unsigned issue_flags)
+{
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	/* no payload to copy, can skip selecting a buffer */
+	if (!fuse_uring_req_has_payload(req))
+		return 0;
+
+	return fuse_uring_select_buffer(ent, issue_flags);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 				   struct fuse_req *req)
 {
@@ -824,21 +1030,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent,
 }
 
 /* Fetch the next fuse request if available */
-static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
+static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent,
+						  unsigned int issue_flags)
 	__must_hold(&queue->lock)
 {
 	struct fuse_req *req;
 	struct fuse_ring_queue *queue = ent->queue;
 	struct list_head *req_queue = &queue->fuse_req_queue;
+	int err;
 
 	lockdep_assert_held(&queue->lock);
 
 	/* get and assign the next entry while it is still holding the lock */
 	req = list_first_entry_or_null(req_queue, struct fuse_req, list);
-	if (req)
-		fuse_uring_add_req_to_ring_ent(ent, req);
+	if (req) {
+		err = fuse_uring_next_req_update_buffer(ent, req, issue_flags);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			return req;
+		}
+	}
 
-	return req;
+	fuse_uring_clean_up_buffer(ent, issue_flags);
+	return NULL;
 }
 
 /*
@@ -878,7 +1092,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
  * Else, there is no next fuse request and this returns false.
  */
 static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
-					 struct fuse_ring_queue *queue)
+					 struct fuse_ring_queue *queue,
+					 unsigned int issue_flags)
 {
 	int err;
 	struct fuse_req *req;
@@ -886,7 +1101,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
 retry:
 	spin_lock(&queue->lock);
 	fuse_uring_ent_avail(ent, queue);
-	req = fuse_uring_ent_assign_req(ent);
+	req = fuse_uring_ent_assign_req(ent, issue_flags);
 	spin_unlock(&queue->lock);
 
 	if (req) {
@@ -927,6 +1142,39 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
 	io_uring_cmd_done(cmd, ret, issue_flags);
 }
 
+static void fuse_uring_headers_cleanup(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+{
+	if (!ent->queue->use_bufring || !ent->headers_node)
+		return;
+
+	io_uring_fixed_index_put(ent->cmd, ent->headers_node, issue_flags);
+	ent->headers_node = NULL;
+}
+
+static int fuse_uring_headers_prep(struct fuse_ring_ent *ent, unsigned int dir,
+				   unsigned int issue_flags)
+{
+	size_t header_size = sizeof(struct fuse_uring_req_header);
+	struct io_uring_cmd *cmd = ent->cmd;
+	struct io_rsrc_node *node;
+	unsigned int offset;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	offset = ent->fixed_buf_id * header_size;
+
+	node = io_uring_fixed_index_get(cmd, FUSE_URING_FIXED_HEADERS_OFFSET,
+					offset, header_size, dir,
+					&ent->headers_iter, issue_flags);
+	if (IS_ERR(node))
+		return PTR_ERR(node);
+
+	ent->headers_node = node;
+	return 0;
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	unsigned int qid = READ_ONCE(cmd_req->qid);
 	struct fuse_pqueue *fpq;
 	struct fuse_req *req;
+	bool send;
 
 	err = -ENOTCONN;
 	if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
 	/* without the queue lock, as other locks are taken */
 	fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-	fuse_uring_commit(ent, req, issue_flags);
+
+	err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+	if (err)
+		fuse_uring_req_end(ent, req, err);
+	else
+		fuse_uring_commit(ent, req, issue_flags);
 
 	/*
 	 * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	if (fuse_uring_get_next_fuse_req(ent, queue))
+	send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+	fuse_uring_headers_cleanup(ent, issue_flags);
+	if (send)
 		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
@@ -1094,39 +1350,48 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	struct iovec iov[FUSE_URING_IOV_SEGS];
 	int err;
 
+	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+	if (!ent)
+		return ERR_PTR(-ENOMEM);
+
+	INIT_LIST_HEAD(&ent->list);
+
+	ent->queue = queue;
+
+	if (queue->use_bufring) {
+		ent->fixed_buf_id = READ_ONCE(cmd->sqe->buf_index);
+		atomic_inc(&ring->queue_refs);
+		return ent;
+	}
+
 	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
 	if (err) {
 		pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
 				    err);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	err = -EINVAL;
 	if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
 		pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	payload_size = iov[1].iov_len;
 	if (payload_size < ring->max_payload_sz) {
 		pr_info_ratelimited("Invalid req payload len %zu\n",
 				    payload_size);
-		return ERR_PTR(err);
+		goto error;
 	}
-
-	err = -ENOMEM;
-	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-	if (!ent)
-		return ERR_PTR(err);
-
-	INIT_LIST_HEAD(&ent->list);
-
-	ent->queue = queue;
 	ent->headers = iov[0].iov_base;
 	ent->payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
+
+error:
+	kfree(ent);
+	return ERR_PTR(err);
 }
 
 /*
@@ -1137,6 +1402,8 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
 			       unsigned int issue_flags, struct fuse_conn *fc)
 {
 	const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe_cmd(cmd->sqe);
+	unsigned int init_flags = READ_ONCE(cmd_req->init.flags);
+	bool use_bufring = init_flags & FUSE_URING_BUF_RING;
 	struct fuse_ring *ring = smp_load_acquire(&fc->ring);
 	struct fuse_ring_queue *queue;
 	struct fuse_ring_ent *ent;
@@ -1157,9 +1424,13 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
 
 	queue = ring->queues[qid];
 	if (!queue) {
-		queue = fuse_uring_create_queue(ring, qid);
-		if (!queue)
-			return err;
+		queue = fuse_uring_create_queue(cmd, ring, qid, use_bufring,
+						issue_flags);
+		if (IS_ERR(queue))
+			return PTR_ERR(queue);
+	} else {
+		if (queue->use_bufring != use_bufring)
+			return -EINVAL;
 	}
 
 	/*
@@ -1258,15 +1529,19 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
 	struct io_uring_cmd *cmd = io_uring_cmd_from_tw(tw_req);
 	struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
 	struct fuse_ring_queue *queue = ent->queue;
+	bool send = true;
 	int err;
 
 	if (!tw.cancel) {
-		err = fuse_uring_prepare_send(ent, ent->fuse_req);
-		if (err) {
-			if (!fuse_uring_get_next_fuse_req(ent, queue))
-				return;
-			err = 0;
-		}
+		if (fuse_uring_headers_prep(ent, ITER_DEST, issue_flags))
+			return;
+
+		if (fuse_uring_prepare_send(ent, ent->fuse_req))
+			send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+		fuse_uring_headers_cleanup(ent, issue_flags);
+		if (!send)
+			return;
+		err = 0;
 	} else {
 		err = -ECANCELED;
 	}
@@ -1325,14 +1600,19 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req)
 	req->ring_queue = queue;
 	ent = list_first_entry_or_null(&queue->ent_avail_queue,
 				       struct fuse_ring_ent, list);
-	if (ent)
-		fuse_uring_add_req_to_ring_ent(ent, req);
-	else
-		list_add_tail(&req->list, &queue->fuse_req_queue);
-	spin_unlock(&queue->lock);
+	if (ent) {
+		err = fuse_uring_prep_buffer(ent, req, IO_URING_F_UNLOCKED);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			spin_unlock(&queue->lock);
+			fuse_uring_dispatch_ent(ent);
+			return;
+		}
+		WARN_ON_ONCE(err != -ENOENT);
+	}
 
-	if (ent)
-		fuse_uring_dispatch_ent(ent);
+	list_add_tail(&req->list, &queue->fuse_req_queue);
+	spin_unlock(&queue->lock);
 
 	return;
 
@@ -1350,6 +1630,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
 	struct fuse_ring *ring = fc->ring;
 	struct fuse_ring_queue *queue;
 	struct fuse_ring_ent *ent = NULL;
+	int err;
 
 	queue = fuse_uring_task_to_queue(ring);
 	if (!queue)
@@ -1382,14 +1663,15 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
 	req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
 				       list);
 	if (ent && req) {
-		fuse_uring_add_req_to_ring_ent(ent, req);
-		spin_unlock(&queue->lock);
-
-		fuse_uring_dispatch_ent(ent);
-	} else {
-		spin_unlock(&queue->lock);
+		err = fuse_uring_prep_buffer(ent, req, IO_URING_F_UNLOCKED);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			spin_unlock(&queue->lock);
+			fuse_uring_dispatch_ent(ent);
+			return true;
+		}
 	}
-
+	spin_unlock(&queue->lock);
 	return true;
 }
 
fs/fuse/dev_uring_i.h +28 -3
index 51a563922ce1..ac6da80c3d70 100644
@@ -7,6 +7,8 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
+
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +40,26 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-	/* userspace buffer */
-	struct fuse_uring_req_header __user *headers;
-	void __user *payload;
+	union {
+		/* queue->use_bufring == false */
+		struct {
+			/* userspace buffers */
+			struct fuse_uring_req_header __user *headers;
+			void __user *payload;
+		};
+		/* queue->use_bufring == true */
+		struct {
+			struct iov_iter headers_iter;
+			struct io_rsrc_node *headers_node;
+			struct kvec payload_kvec;
+			/*
+			 * This needs to be tracked in order to properly recycle
+			 * the buffer when done with it
+			 */
+			unsigned int ringbuf_buf_id;
+			unsigned int fixed_buf_id;
+		};
+	};
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;
@@ -99,6 +118,12 @@ struct fuse_ring_queue {
 	unsigned int active_background;
 
 	bool stopped;
+
+	/* true if kernel-managed buffer ring is used */
+	bool use_bufring: 1;
+
+	/* synchronized by the queue lock */
+	struct io_buffer_list *bufring;
 };
 
 /**
include/uapi/linux/fuse.h +14 -1
index c13e1f9a2f12..b49c8d3b9ab6 100644
@@ -240,6 +240,9 @@
  *  - add FUSE_COPY_FILE_RANGE_64
  *  - add struct fuse_copy_file_range_out
  *  - add FUSE_NOTIFY_PRUNE
+ *
+ *  7.46
+ *  - add fuse_uring_cmd_req init flags
  */
 
 #ifndef _LINUX_FUSE_H
@@ -1294,6 +1297,9 @@ enum fuse_uring_cmd {
 	FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
 };
 
+/* fuse_uring_cmd_req init flags */
+#define FUSE_URING_BUF_RING	(1 << 0)
+
 /**
  * In the 80B command area of the SQE.
  */
@@ -1305,7 +1311,14 @@ struct fuse_uring_cmd_req {
 
 	/* queue the command is for (queue index) */
 	uint16_t qid;
-	uint8_t padding[6];
+
+	union {
+		struct {
+			uint16_t flags;
+		} init;
+	};
+
+	uint8_t padding[4];
 };
 
 #endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
#38 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

kernel test robot noticed the following build warnings:

Hi Joanne,

kernel test robot noticed the following build warnings:

[auto build test WARNING on axboe/for-next]
[also build test WARNING on mszeredi-fuse/for-next linus/master v6.19-rc5 next-20260116] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Joanne-Koong/io_uring-kbuf-refactor-io_buf_pbuf_register-logic-into-generic-helpers/20260117-073512
base:   https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link:    https://lore.kernel.org/r/20260116233044.1532965-20-joannelkoong%40gmail.com
patch subject: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring
config: sh-randconfig-001-20260117 (https://download.01.org/0day-ci/archive/20260117/[email protected]/config)
compiler: sh4-linux-gcc (GCC) 10.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260117/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from arch/sh/include/asm/bug.h:112,

                    from include/linux/bug.h:5,
                    from include/linux/random.h:6,
                    from include/linux/nodemask.h:94,
                    from include/linux/list_lru.h:12,
                    from include/linux/fs/super_types.h:7,
                    from include/linux/fs/super.h:5,
                    from include/linux/fs.h:5,
                    from fs/fuse/fuse_i.h:17,
                    from fs/fuse/dev_uring.c:7:
   fs/fuse/dev_uring.c: In function 'fuse_uring_clean_up_buffer':
>> fs/fuse/dev_uring.c:926:10: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
     926 |          (u64)kvec->iov_base, kvec->iov_len,
         |          ^
   include/asm-generic/bug.h:205:25: note: in definition of macro 'WARN_ON'
     205 |  int __ret_warn_on = !!(condition);    \
         |                         ^~~~~~~~~
   fs/fuse/dev_uring.c:925:2: note: in expansion of macro 'WARN_ON_ONCE'
     925 |  WARN_ON_ONCE(io_uring_kmbuf_recycle(ent->cmd, FUSE_URING_RINGBUF_GROUP,
         |  ^~~~~~~~~~~~

vim +926 fs/fuse/dev_uring.c

913
914 static void fuse_uring_clean_up_buffer(struct fuse_ring_ent *ent,
915 unsigned int issue_flags)
916 __must_hold(&queue->lock)
917 {
918 struct kvec *kvec = &ent->payload_kvec;
919
920 lockdep_assert_held(&ent->queue->lock);
921
922 if (!ent->queue->use_bufring || !kvec->iov_base)
923 return;
924
925 WARN_ON_ONCE(io_uring_kmbuf_recycle(ent->cmd, FUSE_URING_RINGBUF_GROUP,

 > 926						    (u64)kvec->iov_base, kvec->iov_len,

927 ent->ringbuf_buf_id, issue_flags));
928
929 memset(kvec, 0, sizeof(*kvec));
930 }
931

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#39 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

In reply to Joanne Koong
Quoted 743 lines · Joanne Koong

Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.

This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction

b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.

Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.

---
 fs/fuse/dev_uring.c       | 412 ++++++++++++++++++++++++++++++++------
 fs/fuse/dev_uring_i.h     |  31 ++-
 include/uapi/linux/fuse.h |  15 +-
 3 files changed, 389 insertions(+), 69 deletions(-)
fs/fuse/dev_uring.c +347 -65
index b57871f92d08..40e8c2e6b77c 100644
@@ -10,6 +10,8 @@
 #include "fuse_trace.h"
 
 #include <linux/fs.h>
+#include <linux/io_uring.h>
+#include <linux/io_uring/buf.h>
 #include <linux/io_uring/cmd.h>
 
 static bool __read_mostly enable_uring;
@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,
 
 #define FUSE_URING_IOV_SEGS 2 /* header and payload */
 
+#define FUSE_URING_RINGBUF_GROUP 0
+#define FUSE_URING_FIXED_HEADERS_OFFSET 0
 
 bool fuse_uring_enabled(void)
 {
@@ -276,20 +280,45 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)
 	return res;
 }
 
-static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
-						       int qid)
+static int fuse_uring_buf_ring_setup(struct io_uring_cmd *cmd,
+				     struct fuse_ring_queue *queue,
+				     unsigned int issue_flags)
+{
+	int err;
+
+	err = io_uring_buf_ring_pin(cmd, FUSE_URING_RINGBUF_GROUP, issue_flags,
+				    &queue->bufring);
+	if (err)
+		return err;
+
+	if (!io_uring_is_kmbuf_ring(cmd, FUSE_URING_RINGBUF_GROUP,
+				    issue_flags)) {
+		io_uring_buf_ring_unpin(cmd, FUSE_URING_RINGBUF_GROUP,
+					issue_flags);
+		return -EINVAL;
+	}
+
+	queue->use_bufring = true;
+
+	return 0;
+}
+
+static struct fuse_ring_queue *
+fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring,
+			int qid, bool use_bufring, unsigned int issue_flags)
 {
 	struct fuse_conn *fc = ring->fc;
 	struct fuse_ring_queue *queue;
 	struct list_head *pq;
+	int err;
 
 	queue = kzalloc(sizeof(*queue), GFP_KERNEL_ACCOUNT);
 	if (!queue)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	pq = kcalloc(FUSE_PQ_HASH_SIZE, sizeof(struct list_head), GFP_KERNEL);
 	if (!pq) {
 		kfree(queue);
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	}
 
 	queue->qid = qid;
@@ -307,6 +336,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
 	queue->fpq.processing = pq;
 	fuse_pqueue_init(&queue->fpq);
 
+	if (use_bufring) {
+		err = fuse_uring_buf_ring_setup(cmd, queue, issue_flags);
+		if (err) {
+			kfree(pq);
+			kfree(queue);
+			return ERR_PTR(err);
+		}
+	}
+
 	spin_lock(&fc->lock);
 	if (ring->queues[qid]) {
 		spin_unlock(&fc->lock);
@@ -584,6 +622,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static int get_kernel_ring_header(struct fuse_ring_ent *ent,
+				  enum fuse_uring_header_type type,
+				  struct iov_iter *headers_iter)
+{
+	size_t offset;
+
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		/* No offset - start of header */
+		offset = 0;
+		break;
+	case FUSE_URING_HEADER_OP:
+		offset = offsetof(struct fuse_uring_req_header, op_in);
+		break;
+	case FUSE_URING_HEADER_RING_ENT:
+		offset = offsetof(struct fuse_uring_req_header, ring_ent_in_out);
+		break;
+	default:
+		WARN_ONCE(1, "Invalid header type: %d\n", type);
+		return -EINVAL;
+	}
+
+	*headers_iter = ent->headers_iter;
+	if (offset)
+		iov_iter_advance(headers_iter, offset);
+
+	return 0;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 					 enum fuse_uring_header_type type)
 {
@@ -605,17 +672,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,
 					       const void *header,
 					       size_t header_size)
 {
-	void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err =  get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_to_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		void __user *ring = get_user_ring_header(ent, type);
+
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
 
-	if (copy_to_user(ring, header, header_size)) {
-		pr_info_ratelimited("Copying header to ring failed.\n");
-		return -EFAULT;
+		if (copy_to_user(ring, header, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header to ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
@@ -623,17 +711,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
 						 void *header,
 						 size_t header_size)
 {
-	const void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err = get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_from_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		const void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_from_user(header, ring, header_size)) {
-		pr_info_ratelimited("Copying header from ring failed.\n");
-		return -EFAULT;
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
+
+		if (copy_from_user(header, ring, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header from ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static int setup_fuse_copy_state(struct fuse_copy_state *cs,
@@ -643,14 +752,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
 {
 	int err;
 
-	err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
-		return err;
+	if (!ent->queue->use_bufring) {
+		err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
+		if (err) {
+			pr_info_ratelimited("fuse: Import of user buffer "
+					    "failed\n");
+			return err;
+		}
 	}
 
 	fuse_copy_init(cs, dir == ITER_DEST, iter);
 
+	if (ent->queue->use_bufring) {
+		cs->is_kaddr = true;
+		cs->len = ent->payload_kvec.iov_len;
+		cs->kaddr = ent->payload_kvec.iov_base;
+	}
+
 	cs->is_uring = true;
 	cs->req = req;
 
@@ -762,6 +880,94 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 				   sizeof(req->in.h));
 }
 
+static bool fuse_uring_req_has_payload(struct fuse_req *req)
+{
+	struct fuse_args *args = req->args;
+
+	return args->in_numargs > 1 || args->out_numargs;
+}
+
+static int fuse_uring_select_buffer(struct fuse_ring_ent *ent,
+				    unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct io_br_sel sel;
+	size_t len = 0;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	/* Get a buffer to use for the payload */
+	sel = io_ring_buffer_select(cmd_to_io_kiocb(ent->cmd), &len,
+				    ent->queue->bufring, issue_flags);
+	if (sel.val)
+		return sel.val;
+	if (!sel.kaddr)
+		return -ENOENT;
+
+	ent->payload_kvec.iov_base = sel.kaddr;
+	ent->payload_kvec.iov_len = len;
+	ent->ringbuf_buf_id = sel.buf_id;
+
+	return 0;
+}
+
+static void fuse_uring_clean_up_buffer(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct kvec *kvec = &ent->payload_kvec;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	if (!ent->queue->use_bufring || !kvec->iov_base)
+		return;
+
+	WARN_ON_ONCE(io_uring_kmbuf_recycle(ent->cmd, FUSE_URING_RINGBUF_GROUP,
+					    (u64)kvec->iov_base, kvec->iov_len,
+					    ent->ringbuf_buf_id, issue_flags));
+
+	memset(kvec, 0, sizeof(*kvec));
+}
+
+static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
+					     struct fuse_req *req,
+					     unsigned int issue_flags)
+{
+	bool buffer_selected;
+	bool has_payload;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	buffer_selected = ent->payload_kvec.iov_base != NULL;
+	has_payload = fuse_uring_req_has_payload(req);
+
+	if (has_payload && !buffer_selected)
+		return fuse_uring_select_buffer(ent, issue_flags);
+
+	if (!has_payload && buffer_selected)
+		fuse_uring_clean_up_buffer(ent, issue_flags);
+
+	return 0;
+}
+
+static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
+				  struct fuse_req *req, unsigned issue_flags)
+{
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	/* no payload to copy, can skip selecting a buffer */
+	if (!fuse_uring_req_has_payload(req))
+		return 0;
+
+	return fuse_uring_select_buffer(ent, issue_flags);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 				   struct fuse_req *req)
 {
@@ -824,21 +1030,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent,
 }
 
 /* Fetch the next fuse request if available */
-static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
+static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent,
+						  unsigned int issue_flags)
 	__must_hold(&queue->lock)
 {
 	struct fuse_req *req;
 	struct fuse_ring_queue *queue = ent->queue;
 	struct list_head *req_queue = &queue->fuse_req_queue;
+	int err;
 
 	lockdep_assert_held(&queue->lock);
 
 	/* get and assign the next entry while it is still holding the lock */
 	req = list_first_entry_or_null(req_queue, struct fuse_req, list);
-	if (req)
-		fuse_uring_add_req_to_ring_ent(ent, req);
+	if (req) {
+		err = fuse_uring_next_req_update_buffer(ent, req, issue_flags);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			return req;
+		}
+	}
 
-	return req;
+	fuse_uring_clean_up_buffer(ent, issue_flags);
+	return NULL;
 }
 
 /*
@@ -878,7 +1092,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
  * Else, there is no next fuse request and this returns false.
  */
 static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
-					 struct fuse_ring_queue *queue)
+					 struct fuse_ring_queue *queue,
+					 unsigned int issue_flags)
 {
 	int err;
 	struct fuse_req *req;
@@ -886,7 +1101,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
 retry:
 	spin_lock(&queue->lock);
 	fuse_uring_ent_avail(ent, queue);
-	req = fuse_uring_ent_assign_req(ent);
+	req = fuse_uring_ent_assign_req(ent, issue_flags);
 	spin_unlock(&queue->lock);
 
 	if (req) {
@@ -927,6 +1142,39 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
 	io_uring_cmd_done(cmd, ret, issue_flags);
 }
 
+static void fuse_uring_headers_cleanup(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+{
+	if (!ent->queue->use_bufring || !ent->headers_node)
+		return;
+
+	io_uring_fixed_index_put(ent->cmd, ent->headers_node, issue_flags);
+	ent->headers_node = NULL;
+}
+
+static int fuse_uring_headers_prep(struct fuse_ring_ent *ent, unsigned int dir,
+				   unsigned int issue_flags)
+{
+	size_t header_size = sizeof(struct fuse_uring_req_header);
+	struct io_uring_cmd *cmd = ent->cmd;
+	struct io_rsrc_node *node;
+	unsigned int offset;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	offset = ent->fixed_buf_id * header_size;
+
+	node = io_uring_fixed_index_get(cmd, FUSE_URING_FIXED_HEADERS_OFFSET,
+					offset, header_size, dir,
+					&ent->headers_iter, issue_flags);
+	if (IS_ERR(node))
+		return PTR_ERR(node);
+
+	ent->headers_node = node;
+	return 0;
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	unsigned int qid = READ_ONCE(cmd_req->qid);
 	struct fuse_pqueue *fpq;
 	struct fuse_req *req;
+	bool send;
 
 	err = -ENOTCONN;
 	if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
 	/* without the queue lock, as other locks are taken */
 	fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-	fuse_uring_commit(ent, req, issue_flags);
+
+	err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+	if (err)
+		fuse_uring_req_end(ent, req, err);
+	else
+		fuse_uring_commit(ent, req, issue_flags);
 
 	/*
 	 * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	if (fuse_uring_get_next_fuse_req(ent, queue))
+	send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+	fuse_uring_headers_cleanup(ent, issue_flags);
+	if (send)
 		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
@@ -1094,39 +1350,48 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	struct iovec iov[FUSE_URING_IOV_SEGS];
 	int err;
 
+	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+	if (!ent)
+		return ERR_PTR(-ENOMEM);
+
+	INIT_LIST_HEAD(&ent->list);
+
+	ent->queue = queue;
+
+	if (queue->use_bufring) {
+		ent->fixed_buf_id = READ_ONCE(cmd->sqe->buf_index);
+		atomic_inc(&ring->queue_refs);
+		return ent;
+	}
+
 	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
 	if (err) {
 		pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
 				    err);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	err = -EINVAL;
 	if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
 		pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	payload_size = iov[1].iov_len;
 	if (payload_size < ring->max_payload_sz) {
 		pr_info_ratelimited("Invalid req payload len %zu\n",
 				    payload_size);
-		return ERR_PTR(err);
+		goto error;
 	}
-
-	err = -ENOMEM;
-	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-	if (!ent)
-		return ERR_PTR(err);
-
-	INIT_LIST_HEAD(&ent->list);
-
-	ent->queue = queue;
 	ent->headers = iov[0].iov_base;
 	ent->payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
+
+error:
+	kfree(ent);
+	return ERR_PTR(err);
 }
 
 /*
@@ -1137,6 +1402,8 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
 			       unsigned int issue_flags, struct fuse_conn *fc)
 {
 	const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe_cmd(cmd->sqe);
+	unsigned int init_flags = READ_ONCE(cmd_req->init.flags);
+	bool use_bufring = init_flags & FUSE_URING_BUF_RING;
 	struct fuse_ring *ring = smp_load_acquire(&fc->ring);
 	struct fuse_ring_queue *queue;
 	struct fuse_ring_ent *ent;
@@ -1157,9 +1424,13 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
 
 	queue = ring->queues[qid];
 	if (!queue) {
-		queue = fuse_uring_create_queue(ring, qid);
-		if (!queue)
-			return err;
+		queue = fuse_uring_create_queue(cmd, ring, qid, use_bufring,
+						issue_flags);
+		if (IS_ERR(queue))
+			return PTR_ERR(queue);
+	} else {
+		if (queue->use_bufring != use_bufring)
+			return -EINVAL;
 	}
 
 	/*
@@ -1258,15 +1529,19 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
 	struct io_uring_cmd *cmd = io_uring_cmd_from_tw(tw_req);
 	struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
 	struct fuse_ring_queue *queue = ent->queue;
+	bool send = true;
 	int err;
 
 	if (!tw.cancel) {
-		err = fuse_uring_prepare_send(ent, ent->fuse_req);
-		if (err) {
-			if (!fuse_uring_get_next_fuse_req(ent, queue))
-				return;
-			err = 0;
-		}
+		if (fuse_uring_headers_prep(ent, ITER_DEST, issue_flags))
+			return;
+
+		if (fuse_uring_prepare_send(ent, ent->fuse_req))
+			send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+		fuse_uring_headers_cleanup(ent, issue_flags);
+		if (!send)
+			return;
+		err = 0;
 	} else {
 		err = -ECANCELED;
 	}
@@ -1325,14 +1600,19 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req)
 	req->ring_queue = queue;
 	ent = list_first_entry_or_null(&queue->ent_avail_queue,
 				       struct fuse_ring_ent, list);
-	if (ent)
-		fuse_uring_add_req_to_ring_ent(ent, req);
-	else
-		list_add_tail(&req->list, &queue->fuse_req_queue);
-	spin_unlock(&queue->lock);
+	if (ent) {
+		err = fuse_uring_prep_buffer(ent, req, IO_URING_F_UNLOCKED);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			spin_unlock(&queue->lock);
+			fuse_uring_dispatch_ent(ent);
+			return;
+		}
+		WARN_ON_ONCE(err != -ENOENT);
+	}
 
-	if (ent)
-		fuse_uring_dispatch_ent(ent);
+	list_add_tail(&req->list, &queue->fuse_req_queue);
+	spin_unlock(&queue->lock);
 
 	return;
 
@@ -1350,6 +1630,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
 	struct fuse_ring *ring = fc->ring;
 	struct fuse_ring_queue *queue;
 	struct fuse_ring_ent *ent = NULL;
+	int err;
 
 	queue = fuse_uring_task_to_queue(ring);
 	if (!queue)
@@ -1382,14 +1663,15 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
 	req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
 				       list);
 	if (ent && req) {
-		fuse_uring_add_req_to_ring_ent(ent, req);
-		spin_unlock(&queue->lock);
-
-		fuse_uring_dispatch_ent(ent);
-	} else {
-		spin_unlock(&queue->lock);
+		err = fuse_uring_prep_buffer(ent, req, IO_URING_F_UNLOCKED);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			spin_unlock(&queue->lock);
+			fuse_uring_dispatch_ent(ent);
+			return true;
+		}
 	}
-
+	spin_unlock(&queue->lock);
 	return true;
 }
 
fs/fuse/dev_uring_i.h +28 -3
index 51a563922ce1..ac6da80c3d70 100644
@@ -7,6 +7,8 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
+
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +40,26 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-	/* userspace buffer */
-	struct fuse_uring_req_header __user *headers;
-	void __user *payload;
+	union {
+		/* queue->use_bufring == false */
+		struct {
+			/* userspace buffers */
+			struct fuse_uring_req_header __user *headers;
+			void __user *payload;
+		};
+		/* queue->use_bufring == true */
+		struct {
+			struct iov_iter headers_iter;
+			struct io_rsrc_node *headers_node;
+			struct kvec payload_kvec;
+			/*
+			 * This needs to be tracked in order to properly recycle
+			 * the buffer when done with it
+			 */
+			unsigned int ringbuf_buf_id;
+			unsigned int fixed_buf_id;
+		};
+	};
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;
@@ -99,6 +118,12 @@ struct fuse_ring_queue {
 	unsigned int active_background;
 
 	bool stopped;
+
+	/* true if kernel-managed buffer ring is used */
+	bool use_bufring: 1;
+
+	/* synchronized by the queue lock */
+	struct io_buffer_list *bufring;
 };
 
 /**
include/uapi/linux/fuse.h +6 -0
index c13e1f9a2f12..b49c8d3b9ab6 100644
@@ -240,6 +240,9 @@
  *  - add FUSE_COPY_FILE_RANGE_64
  *  - add struct fuse_copy_file_range_out
  *  - add FUSE_NOTIFY_PRUNE
+ *
+ *  7.46
+ *  - add fuse_uring_cmd_req init flags
  */
 
 #ifndef _LINUX_FUSE_H
@@ -1294,6 +1297,9 @@ enum fuse_uring_cmd {
 	FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
 };
 
+/* fuse_uring_cmd_req init flags */
+#define FUSE_URING_BUF_RING	(1 << 0)
+
 /**
  * In the 80B command area of the SQE.
  */
@@ -1305,7 +1311,14 @@ struct fuse_uring_cmd_req {
 

include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1320

Quoted 10 lines
include/uapi/linux/fuse.h +7 -1
 	/* queue the command is for (queue index) */
 	uint16_t qid;
-	uint8_t padding[6];
+
+	union {
+		struct {
+			uint16_t flags;
+		} init;
+	};
+

I won't manage to review everything of this patch today, but just noticed this. There is already an unused flags, why don't you use that? I had edded it exactly for such things.

include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1324

Quoted 4 lines
include/uapi/linux/fuse.h +1 -0
+	uint8_t padding[4];
 };
 
 #endif /* _LINUX_FUSE_H */

Thanks,
Bernd

author maintainer
#40 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

Quoted 36 lines · Bernd Schubert
Quoted 32 lines · Joanne Koong

Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.

This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction

b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.

Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.

---
 fs/fuse/dev_uring.c       | 412 ++++++++++++++++++++++++++++++++------
 fs/fuse/dev_uring_i.h     |  31 ++-
 include/uapi/linux/fuse.h |  15 +-
 3 files changed, 389 insertions(+), 69 deletions(-)
fs/fuse/dev_uring.c +0 -0
index b57871f92d08..40e8c2e6b77c 100644
@@ -1305,7 +1311,14 @@ struct fuse_uring_cmd_req {
 
Quoted 14 lines

fs/fuse/dev_uring.c struct fuse_uring_cmd_req { · line 1320

Quoted 10 lines
fs/fuse/dev_uring.c +7 -1
      /* queue the command is for (queue index) */
      uint16_t qid;
-     uint8_t padding[6];
+
+     union {
+             struct {
+                     uint16_t flags;
+             } init;
+     };
+

I won't manage to review everything of this patch today, but just noticed this. There is already an unused flags, why don't you use that? I had edded it exactly for such things.

Oh nice, I missed seeing that. I'll just use that flags variable then.

Thanks,
Joanne

Quoted 8 lines

fs/fuse/dev_uring.c struct fuse_uring_cmd_req { · line 1324

Quoted 5 lines
fs/fuse/dev_uring.c +1 -0
+     uint8_t padding[4];
 };
 
 #endif /* _LINUX_FUSE_H */

Thanks,
Bernd

maintainer
#41 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

In reply to Bernd Schubert
Quoted 480 lines · Bernd Schubert
Quoted 477 lines · Joanne Koong

Add io-uring kernel-managed buffer ring capability for fuse daemons communicating through the io-uring interface.

This has two benefits:
a) eliminates the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction

b) reduces the amount of memory needed for the buffers per queue and allows buffers to be reused across entries. Incremental buffer consumption, when added, will allow a buffer to be used across multiple requests.

Buffer ring usage is set on a per-queue basis. In order to use this, the daemon needs to have preregistered a kernel-managed buffer ring and a fixed buffer at index 0 that will hold all the headers, and set the "use_bufring" field during registration. The kernel-managed buffer ring will be pinned for the lifetime of the connection.

---
 fs/fuse/dev_uring.c       | 412 ++++++++++++++++++++++++++++++++------
 fs/fuse/dev_uring_i.h     |  31 ++-
 include/uapi/linux/fuse.h |  15 +-
 3 files changed, 389 insertions(+), 69 deletions(-)
fs/fuse/dev_uring.c +283 -29
index b57871f92d08..40e8c2e6b77c 100644
@@ -10,6 +10,8 @@
 #include "fuse_trace.h"
 
 #include <linux/fs.h>
+#include <linux/io_uring.h>
+#include <linux/io_uring/buf.h>
 #include <linux/io_uring/cmd.h>
 
 static bool __read_mostly enable_uring;
@@ -19,6 +21,8 @@ MODULE_PARM_DESC(enable_uring,
 
 #define FUSE_URING_IOV_SEGS 2 /* header and payload */
 
+#define FUSE_URING_RINGBUF_GROUP 0
+#define FUSE_URING_FIXED_HEADERS_OFFSET 0
 
 bool fuse_uring_enabled(void)
 {
@@ -276,20 +280,45 @@ static struct fuse_ring *fuse_uring_create(struct fuse_conn *fc)
 	return res;
 }
 
-static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
-						       int qid)
+static int fuse_uring_buf_ring_setup(struct io_uring_cmd *cmd,
+				     struct fuse_ring_queue *queue,
+				     unsigned int issue_flags)
+{
+	int err;
+
+	err = io_uring_buf_ring_pin(cmd, FUSE_URING_RINGBUF_GROUP, issue_flags,
+				    &queue->bufring);
+	if (err)
+		return err;
+
+	if (!io_uring_is_kmbuf_ring(cmd, FUSE_URING_RINGBUF_GROUP,
+				    issue_flags)) {
+		io_uring_buf_ring_unpin(cmd, FUSE_URING_RINGBUF_GROUP,
+					issue_flags);
+		return -EINVAL;
+	}
+
+	queue->use_bufring = true;
+
+	return 0;
+}
+
+static struct fuse_ring_queue *
+fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring,
+			int qid, bool use_bufring, unsigned int issue_flags)
 {
 	struct fuse_conn *fc = ring->fc;
 	struct fuse_ring_queue *queue;
 	struct list_head *pq;
+	int err;
 
 	queue = kzalloc(sizeof(*queue), GFP_KERNEL_ACCOUNT);
 	if (!queue)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	pq = kcalloc(FUSE_PQ_HASH_SIZE, sizeof(struct list_head), GFP_KERNEL);
 	if (!pq) {
 		kfree(queue);
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	}
 
 	queue->qid = qid;
@@ -307,6 +336,15 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
 	queue->fpq.processing = pq;
 	fuse_pqueue_init(&queue->fpq);
 
+	if (use_bufring) {
+		err = fuse_uring_buf_ring_setup(cmd, queue, issue_flags);
+		if (err) {
+			kfree(pq);
+			kfree(queue);
+			return ERR_PTR(err);
+		}
+	}
+
 	spin_lock(&fc->lock);
 	if (ring->queues[qid]) {
 		spin_unlock(&fc->lock);
@@ -584,6 +622,35 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static int get_kernel_ring_header(struct fuse_ring_ent *ent,
+				  enum fuse_uring_header_type type,
+				  struct iov_iter *headers_iter)
+{
+	size_t offset;
+
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		/* No offset - start of header */
+		offset = 0;
+		break;
+	case FUSE_URING_HEADER_OP:
+		offset = offsetof(struct fuse_uring_req_header, op_in);
+		break;
+	case FUSE_URING_HEADER_RING_ENT:
+		offset = offsetof(struct fuse_uring_req_header, ring_ent_in_out);
+		break;
+	default:
+		WARN_ONCE(1, "Invalid header type: %d\n", type);
+		return -EINVAL;
+	}
+
+	*headers_iter = ent->headers_iter;
+	if (offset)
+		iov_iter_advance(headers_iter, offset);
+
+	return 0;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 					 enum fuse_uring_header_type type)
 {
@@ -605,17 +672,38 @@ static __always_inline int copy_header_to_ring(struct fuse_ring_ent *ent,
 					       const void *header,
 					       size_t header_size)
 {
-	void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err =  get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_to_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		void __user *ring = get_user_ring_header(ent, type);
+
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
 
-	if (copy_to_user(ring, header, header_size)) {
-		pr_info_ratelimited("Copying header to ring failed.\n");
-		return -EFAULT;
+		if (copy_to_user(ring, header, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header to ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
@@ -623,17 +711,38 @@ static __always_inline int copy_header_from_ring(struct fuse_ring_ent *ent,
 						 void *header,
 						 size_t header_size)
 {
-	const void __user *ring = get_user_ring_header(ent, type);
+	bool use_bufring = ent->queue->use_bufring;
+	int err = 0;
 
-	if (!ring)
-		return -EINVAL;
+	if (use_bufring) {
+		struct iov_iter iter;
+
+		err = get_kernel_ring_header(ent, type, &iter);
+		if (err)
+			goto done;
+
+		if (copy_from_iter(header, header_size, &iter) != header_size)
+			err = -EFAULT;
+	} else {
+		const void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_from_user(header, ring, header_size)) {
-		pr_info_ratelimited("Copying header from ring failed.\n");
-		return -EFAULT;
+		if (!ring) {
+			err = -EINVAL;
+			goto done;
+		}
+
+		if (copy_from_user(header, ring, header_size))
+			err = -EFAULT;
 	}
 
-	return 0;
+done:
+	if (err)
+		pr_info_ratelimited("Copying header from ring failed: "
+				    "header_type=%u, header_size=%zu, "
+				    "use_bufring=%d\n", type, header_size,
+				    use_bufring);
+
+	return err;
 }
 
 static int setup_fuse_copy_state(struct fuse_copy_state *cs,
@@ -643,14 +752,23 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
 {
 	int err;
 
-	err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
-		return err;
+	if (!ent->queue->use_bufring) {
+		err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
+		if (err) {
+			pr_info_ratelimited("fuse: Import of user buffer "
+					    "failed\n");
+			return err;
+		}
 	}
 
 	fuse_copy_init(cs, dir == ITER_DEST, iter);
 
+	if (ent->queue->use_bufring) {
+		cs->is_kaddr = true;
+		cs->len = ent->payload_kvec.iov_len;
+		cs->kaddr = ent->payload_kvec.iov_base;
+	}
+
 	cs->is_uring = true;
 	cs->req = req;
 
@@ -762,6 +880,94 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 				   sizeof(req->in.h));
 }
 
+static bool fuse_uring_req_has_payload(struct fuse_req *req)
+{
+	struct fuse_args *args = req->args;
+
+	return args->in_numargs > 1 || args->out_numargs;
+}
+
+static int fuse_uring_select_buffer(struct fuse_ring_ent *ent,
+				    unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct io_br_sel sel;
+	size_t len = 0;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	/* Get a buffer to use for the payload */
+	sel = io_ring_buffer_select(cmd_to_io_kiocb(ent->cmd), &len,
+				    ent->queue->bufring, issue_flags);
+	if (sel.val)
+		return sel.val;
+	if (!sel.kaddr)
+		return -ENOENT;
+
+	ent->payload_kvec.iov_base = sel.kaddr;
+	ent->payload_kvec.iov_len = len;
+	ent->ringbuf_buf_id = sel.buf_id;
+
+	return 0;
+}
+
+static void fuse_uring_clean_up_buffer(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+	__must_hold(&queue->lock)
+{
+	struct kvec *kvec = &ent->payload_kvec;
+
+	lockdep_assert_held(&ent->queue->lock);
+
+	if (!ent->queue->use_bufring || !kvec->iov_base)
+		return;
+
+	WARN_ON_ONCE(io_uring_kmbuf_recycle(ent->cmd, FUSE_URING_RINGBUF_GROUP,
+					    (u64)kvec->iov_base, kvec->iov_len,
+					    ent->ringbuf_buf_id, issue_flags));
+
+	memset(kvec, 0, sizeof(*kvec));
+}
+
+static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
+					     struct fuse_req *req,
+					     unsigned int issue_flags)
+{
+	bool buffer_selected;
+	bool has_payload;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	buffer_selected = ent->payload_kvec.iov_base != NULL;
+	has_payload = fuse_uring_req_has_payload(req);
+
+	if (has_payload && !buffer_selected)
+		return fuse_uring_select_buffer(ent, issue_flags);
+
+	if (!has_payload && buffer_selected)
+		fuse_uring_clean_up_buffer(ent, issue_flags);
+
+	return 0;
+}
+
+static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
+				  struct fuse_req *req, unsigned issue_flags)
+{
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	ent->headers_iter.data_source = false;
+
+	/* no payload to copy, can skip selecting a buffer */
+	if (!fuse_uring_req_has_payload(req))
+		return 0;
+
+	return fuse_uring_select_buffer(ent, issue_flags);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 				   struct fuse_req *req)
 {
@@ -824,21 +1030,29 @@ static void fuse_uring_add_req_to_ring_ent(struct fuse_ring_ent *ent,
 }
 
 /* Fetch the next fuse request if available */
-static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
+static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent,
+						  unsigned int issue_flags)
 	__must_hold(&queue->lock)
 {
 	struct fuse_req *req;
 	struct fuse_ring_queue *queue = ent->queue;
 	struct list_head *req_queue = &queue->fuse_req_queue;
+	int err;
 
 	lockdep_assert_held(&queue->lock);
 
 	/* get and assign the next entry while it is still holding the lock */
 	req = list_first_entry_or_null(req_queue, struct fuse_req, list);
-	if (req)
-		fuse_uring_add_req_to_ring_ent(ent, req);
+	if (req) {
+		err = fuse_uring_next_req_update_buffer(ent, req, issue_flags);
+		if (!err) {
+			fuse_uring_add_req_to_ring_ent(ent, req);
+			return req;
+		}
+	}
 
-	return req;
+	fuse_uring_clean_up_buffer(ent, issue_flags);
+	return NULL;
 }
 
 /*
@@ -878,7 +1092,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
  * Else, there is no next fuse request and this returns false.
  */
 static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
-					 struct fuse_ring_queue *queue)
+					 struct fuse_ring_queue *queue,
+					 unsigned int issue_flags)
 {
 	int err;
 	struct fuse_req *req;
@@ -886,7 +1101,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
 retry:
 	spin_lock(&queue->lock);
 	fuse_uring_ent_avail(ent, queue);
-	req = fuse_uring_ent_assign_req(ent);
+	req = fuse_uring_ent_assign_req(ent, issue_flags);
 	spin_unlock(&queue->lock);
 
 	if (req) {
@@ -927,6 +1142,39 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
 	io_uring_cmd_done(cmd, ret, issue_flags);
 }
 
+static void fuse_uring_headers_cleanup(struct fuse_ring_ent *ent,
+				       unsigned int issue_flags)
+{
+	if (!ent->queue->use_bufring || !ent->headers_node)
+		return;
+
+	io_uring_fixed_index_put(ent->cmd, ent->headers_node, issue_flags);
+	ent->headers_node = NULL;
+}
+
+static int fuse_uring_headers_prep(struct fuse_ring_ent *ent, unsigned int dir,
+				   unsigned int issue_flags)
+{
+	size_t header_size = sizeof(struct fuse_uring_req_header);
+	struct io_uring_cmd *cmd = ent->cmd;
+	struct io_rsrc_node *node;
+	unsigned int offset;
+
+	if (!ent->queue->use_bufring)
+		return 0;
+
+	offset = ent->fixed_buf_id * header_size;
+
+	node = io_uring_fixed_index_get(cmd, FUSE_URING_FIXED_HEADERS_OFFSET,
+					offset, header_size, dir,
+					&ent->headers_iter, issue_flags);
+	if (IS_ERR(node))
+		return PTR_ERR(node);
+
+	ent->headers_node = node;
+	return 0;
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	unsigned int qid = READ_ONCE(cmd_req->qid);
 	struct fuse_pqueue *fpq;
 	struct fuse_req *req;
+	bool send;
 
 	err = -ENOTCONN;
 	if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
 	/* without the queue lock, as other locks are taken */
 	fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-	fuse_uring_commit(ent, req, issue_flags);
+
+	err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+	if (err)
+		fuse_uring_req_end(ent, req, err);
+	else
+		fuse_uring_commit(ent, req, issue_flags);
 
Quoted 12 lines

fs/fuse/dev_uring.c static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · line 1259

Quoted 12 lines
fs/fuse/dev_uring.c +3 -1
 	/*
 	 * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	if (fuse_uring_get_next_fuse_req(ent, queue))
+	send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+	fuse_uring_headers_cleanup(ent, issue_flags);
+	if (send)
 		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;

Hello Joanne,

couldn't it call fuse_uring_headers_cleanup() before the fuse_uring_get_next_fuse_req()? I find it a bit confusing that it firsts gets the next request and then cleans up the buffer from the previous request.

As I understand it, the the patch basically adds the feature of 0-byte payloads. Maybe worth mentioning in the commit message?
I also wonder if it would be worth to document as code comment that fuse_uring_ent_assign_req / fuse_uring_next_req_update_buffer are allowed to fail for a buffer upgrade (i.e. 0 to max-payload). At least the current comment of "Fetching the next request is absolutely required" is actually not entirely true anymore.

Thanks,
Bernd

author maintainer
#42 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

Quoted 39 lines · Bernd Schubert
Quoted 22 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
fs/fuse/dev_uring.c +7 -1
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
     unsigned int qid = READ_ONCE(cmd_req->qid);
     struct fuse_pqueue *fpq;
     struct fuse_req *req;
+    bool send;
 
     err = -ENOTCONN;
     if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
     /* without the queue lock, as other locks are taken */
     fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-    fuse_uring_commit(ent, req, issue_flags);
+
+    err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+    if (err)
+            fuse_uring_req_end(ent, req, err);
+    else
+            fuse_uring_commit(ent, req, issue_flags);
 
Quoted 12 lines

fs/fuse/dev_uring.c static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · line 1259

Quoted 12 lines
fs/fuse/dev_uring.c +3 -1
     /*
      * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
      * and fetching is done in one step vs legacy fuse, which has separated
      * read (fetch request) and write (commit result).
      */
-    if (fuse_uring_get_next_fuse_req(ent, queue))
+    send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+    fuse_uring_headers_cleanup(ent, issue_flags);
+    if (send)
             fuse_uring_send(ent, cmd, 0, issue_flags);
     return 0;

Hello Joanne,

Quoted 4 lines

couldn't it call fuse_uring_headers_cleanup() before the fuse_uring_get_next_fuse_req()? I find it a bit confusing that it firsts gets the next request and then cleans up the buffer from the previous request.

Hi Bernd,

Thanks for taking a look.

The fuse_uring_headers_cleanup() call has to happen after the fuse_uring_get_next_fuse_req() call because fuse_uring_get_next_fuse_req() copies payload to the header, so we can't yet relinquish the refcount on the headers buffer / clean it up yet. I can add a comment about this to make this more clear.

Quoted 3 lines

As I understand it, the the patch basically adds the feature of 0-byte payloads. Maybe worth mentioning in the commit message?

Hmm I'm not really sure I am seeing where the 0-byte payload gets added. On the server side, they don't receive payloads that are 0-bytes. If there is no next fuse request to send, then nothing gets sent. But maybe I'm not interpreting your comment about 0-byte payloads correctly?

Quoted 3 lines

I also wonder if it would be worth to document as code comment that fuse_uring_ent_assign_req / fuse_uring_next_req_update_buffer are allowed to fail for a buffer upgrade (i.e. 0 to max-payload). At least

Good idea, I'll add a comment about this.

Quoted 3 lines

the current comment of "Fetching the next request is absolutely required" is actually not entirely true anymore.

I don't think this patch introduces new behavior on this front. fuse_uring_get_next_fuse_req() is still called to fetch the next request AFAICS.

Thanks,
Joanne

Quoted 3 lines

Thanks,
Bernd

maintainer
#43 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

Quoted 47 lines · Joanne Koong
Quoted 39 lines · Bernd Schubert
Quoted 22 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
fs/fuse/dev_uring.c +7 -1
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
     unsigned int qid = READ_ONCE(cmd_req->qid);
     struct fuse_pqueue *fpq;
     struct fuse_req *req;
+    bool send;
 
     err = -ENOTCONN;
     if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
     /* without the queue lock, as other locks are taken */
     fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-    fuse_uring_commit(ent, req, issue_flags);
+
+    err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+    if (err)
+            fuse_uring_req_end(ent, req, err);
+    else
+            fuse_uring_commit(ent, req, issue_flags);
 
Quoted 12 lines

fs/fuse/dev_uring.c static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · line 1259

Quoted 12 lines
fs/fuse/dev_uring.c +3 -1
     /*
      * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
      * and fetching is done in one step vs legacy fuse, which has separated
      * read (fetch request) and write (commit result).
      */
-    if (fuse_uring_get_next_fuse_req(ent, queue))
+    send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+    fuse_uring_headers_cleanup(ent, issue_flags);
+    if (send)
             fuse_uring_send(ent, cmd, 0, issue_flags);
     return 0;

Hello Joanne,

Quoted 4 lines

couldn't it call fuse_uring_headers_cleanup() before the fuse_uring_get_next_fuse_req()? I find it a bit confusing that it firsts gets the next request and then cleans up the buffer from the previous request.

Hi Bernd,

Quoted 7 lines

Thanks for taking a look.

The fuse_uring_headers_cleanup() call has to happen after the fuse_uring_get_next_fuse_req() call because fuse_uring_get_next_fuse_req() copies payload to the header, so we can't yet relinquish the refcount on the headers buffer / clean it up yet. I can add a comment about this to make this more clear.

I only found time right now and already super late (or early) here.

I guess that is fuse_uring_copy_to_ring -> copy_header_to_ring, but why can it then call fuse_uring_headers_cleanup() -> io_uring_fixed_index_put(). I.e. doesn't it put buffer it just copied to? Why not the sequence of

err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags); fuse_uring_commit(ent, req, issue_flags); fuse_uring_headers_cleanup(ent, issue_flags);

And then fuse_uring_get_next_fuse_req() does another fuse_uring_headers_prep() with ITER_DEST?

Quoted 10 lines
Quoted 4 lines

As I understand it, the the patch basically adds the feature of 0-byte payloads. Maybe worth mentioning in the commit message?

Hmm I'm not really sure I am seeing where the 0-byte payload gets added. On the server side, they don't receive payloads that are 0-bytes. If there is no next fuse request to send, then nothing gets sent. But maybe I'm not interpreting your comment about 0-byte payloads correctly?

There is fuse_uring_req_has_payload() and fuse_uring_select_buffer()/fuse_uring_next_req_update_buffer() using that function. When a request doesn't have a payload the ring entries runs without a payload - effectively that introduces 0-byte payloads, doesn't it?

Quoted 7 lines
Quoted 4 lines

I also wonder if it would be worth to document as code comment that fuse_uring_ent_assign_req / fuse_uring_next_req_update_buffer are allowed to fail for a buffer upgrade (i.e. 0 to max-payload). At least

Good idea, I'll add a comment about this.

Quoted 8 lines
Quoted 3 lines

the current comment of "Fetching the next request is absolutely required" is actually not entirely true anymore.

I don't think this patch introduces new behavior on this front. fuse_uring_get_next_fuse_req() is still called to fetch the next request AFAICS.

It still does, but if the request didn't have a payload it might not have a buffer and if it didn't have a buffer and doesn't manage to get a buffer, it doesn't handle a request - that a bit change of 'commit-and-fetch always fetches a new request if there is any request queued'.

Thanks,
Bernd

author maintainer
#44 lore raw

Re: [PATCH v4 19/25] fuse: add io-uring kernel-managed buffer ring

Quoted 64 lines · Bernd Schubert
Quoted 47 lines · Joanne Koong
Quoted 39 lines · Bernd Schubert
Quoted 22 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
fs/fuse/dev_uring.c +7 -1
@@ -940,6 +1188,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
     unsigned int qid = READ_ONCE(cmd_req->qid);
     struct fuse_pqueue *fpq;
     struct fuse_req *req;
+    bool send;
 
     err = -ENOTCONN;
     if (!ring)
@@ -990,7 +1239,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
     /* without the queue lock, as other locks are taken */
     fuse_uring_prepare_cancel(cmd, issue_flags, ent);
-    fuse_uring_commit(ent, req, issue_flags);
+
+    err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags);
+    if (err)
+            fuse_uring_req_end(ent, req, err);
+    else
+            fuse_uring_commit(ent, req, issue_flags);
 
Quoted 12 lines

fs/fuse/dev_uring.c static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · line 1259

Quoted 12 lines
fs/fuse/dev_uring.c +3 -1
     /*
      * Fetching the next request is absolutely required as queued
@@ -998,7 +1252,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
      * and fetching is done in one step vs legacy fuse, which has separated
      * read (fetch request) and write (commit result).
      */
-    if (fuse_uring_get_next_fuse_req(ent, queue))
+    send = fuse_uring_get_next_fuse_req(ent, queue, issue_flags);
+    fuse_uring_headers_cleanup(ent, issue_flags);
+    if (send)
             fuse_uring_send(ent, cmd, 0, issue_flags);
     return 0;

Hello Joanne,

Quoted 4 lines

couldn't it call fuse_uring_headers_cleanup() before the fuse_uring_get_next_fuse_req()? I find it a bit confusing that it firsts gets the next request and then cleans up the buffer from the previous request.

Hi Bernd,

Quoted 7 lines

Thanks for taking a look.

The fuse_uring_headers_cleanup() call has to happen after the fuse_uring_get_next_fuse_req() call because fuse_uring_get_next_fuse_req() copies payload to the header, so we can't yet relinquish the refcount on the headers buffer / clean it up yet. I can add a comment about this to make this more clear.

I only found time right now and already super late (or early) here.

I guess that is fuse_uring_copy_to_ring -> copy_header_to_ring, but why can it then call fuse_uring_headers_cleanup() -> io_uring_fixed_index_put(). I.e. doesn't it put buffer it just copied to? Why not the sequence of

Quoted 6 lines

err = fuse_uring_headers_prep(ent, ITER_SOURCE, issue_flags); fuse_uring_commit(ent, req, issue_flags); fuse_uring_headers_cleanup(ent, issue_flags);

And then fuse_uring_get_next_fuse_req() does another fuse_uring_headers_prep() with ITER_DEST?

The headers buffer is the same buffer for the request that's being committed and the next request that is fetched (just like how the ent is the same ent for the request that's committed and the next fetched request). Because of this we can just reuse the same headers_iter and call io_uring_fixed_index_put() after we're done copying over the next request. I can add a comment about this to make this more clear.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 2 lines

As I understand it, the the patch basically adds the feature of 0-byte payloads. Maybe worth mentioning in the commit message?

Hmm I'm not really sure I am seeing where the 0-byte payload gets added. On the server side, they don't receive payloads that are 0-bytes. If there is no next fuse request to send, then nothing gets sent. But maybe I'm not interpreting your comment about 0-byte payloads correctly?

Quoted 5 lines

There is fuse_uring_req_has_payload() and fuse_uring_select_buffer()/fuse_uring_next_req_update_buffer() using that function. When a request doesn't have a payload the ring entries runs without a payload - effectively that introduces 0-byte payloads, doesn't it?

When the request doesn't have a payload, no ring entry gets used for the request. The check for this happens in fuse_uring_prep_buffer()

static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
struct fuse_req *req, unsigned issue_flags)

{
        ...
        /* no payload to copy, can skip selecting a buffer */
        if (!fuse_uring_req_has_payload(req))
                return 0;

        return fuse_uring_select_buffer(ent, issue_flags);
}
Quoted 17 lines
Quoted 8 lines
Quoted 5 lines

I also wonder if it would be worth to document as code comment that fuse_uring_ent_assign_req / fuse_uring_next_req_update_buffer are allowed to fail for a buffer upgrade (i.e. 0 to max-payload). At least

Good idea, I'll add a comment about this.

Quoted 8 lines
Quoted 3 lines

the current comment of "Fetching the next request is absolutely required" is actually not entirely true anymore.

I don't think this patch introduces new behavior on this front. fuse_uring_get_next_fuse_req() is still called to fetch the next request AFAICS.

Quoted 5 lines

It still does, but if the request didn't have a payload it might not have a buffer and if it didn't have a buffer and doesn't manage to get a buffer, it doesn't handle a request - that a bit change of 'commit-and-fetch always fetches a new request if there is any request queued'.

Ok, I'll update the "Fetching the next request is absolutely required" comment.

Thanks,
Joanne

Quoted 4 lines

Thanks,
Bernd

author maintainer
#45 lore raw

PATCH v4 20/25 io_uring/rsrc: rename io_buffer_register_bvec()/io_buffer_unregister_bvec()

In reply to Joanne Koong

Currently, io_buffer_register_bvec() takes in a request. In preparation for supporting kernel-populated buffers in fuse io-uring (which will need to register bvecs directly, not through a struct request), rename this to io_buffer_register_request().

A subsequent patch will commandeer the "io_buffer_register_bvec()" function name to support registering bvecs directly.

Rename io_buffer_unregister_bvec() to a more generic name, io_buffer_unregister(), as both io_buffer_register_request() and io_buffer_register_bvec() callers will use it for unregistration.

---
 Documentation/block/ublk.rst | 14 +++++++-------
 drivers/block/ublk_drv.c     | 18 +++++++++---------
 include/linux/io_uring/cmd.h | 26 ++++++++++++++++++++------
 io_uring/rsrc.c              | 14 +++++++-------
 4 files changed, 43 insertions(+), 29 deletions(-)
Documentation/block/ublk.rst +7 -7
index 8c4030bcabb6..aa6e0bf9405b 100644
@@ -326,17 +326,17 @@ Zero copy
 ---------
 
 ublk zero copy relies on io_uring's fixed kernel buffer, which provides
-two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`.
+two APIs: `io_buffer_register_request()` and `io_buffer_unregister`.
 
 ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call
-`io_buffer_register_bvec()` for ublk server to register client request
+`io_buffer_register_request()` for ublk server to register client request
 buffer into io_uring buffer table, then ublk server can submit io_uring
 IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF`
-calls `io_buffer_unregister_bvec()` to unregister the buffer, which is
-guaranteed to be live between calling `io_buffer_register_bvec()` and
-`io_buffer_unregister_bvec()`. Any io_uring operation which supports this
-kind of kernel buffer will grab one reference of the buffer until the
-operation is completed.
+calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed
+to be live between calling `io_buffer_register_request()` and
+`io_buffer_unregister()`. Any io_uring operation which supports this kind of
+kernel buffer will grab one reference of the buffer until the operation is
+completed.
 
 ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and
 be trusted, because it is ublk server's responsibility to make sure IO buffer
drivers/block/ublk_drv.c +9 -9
index f6e5a0766721..03652d9ce5a4 100644
@@ -1227,8 +1227,8 @@ __ublk_do_auto_buf_reg(const struct ublk_queue *ubq, struct request *req,
 {
 	int ret;
 
-	ret = io_buffer_register_bvec(cmd, req, ublk_io_release,
-				      io->buf.auto_reg.index, issue_flags);
+	ret = io_buffer_register_request(cmd, req, ublk_io_release,
+					 io->buf.auto_reg.index, issue_flags);
 	if (ret) {
 		if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
 			ublk_auto_buf_reg_fallback(ubq, req->tag);
@@ -2212,8 +2212,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,
 	if (!req)
 		return -EINVAL;
 
-	ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
-				      issue_flags);
+	ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
+					 issue_flags);
 	if (ret) {
 		ublk_put_req_ref(io, req);
 		return ret;
@@ -2244,8 +2244,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,
 	if (!ublk_dev_support_zero_copy(ub) || !ublk_rq_has_data(req))
 		return -EINVAL;
 
-	ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
-				      issue_flags);
+	ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
+					 issue_flags);
 	if (ret)
 		return ret;
 
@@ -2260,7 +2260,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,
 	if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY))
 		return -EINVAL;
 
-	return io_buffer_unregister_bvec(cmd, index, issue_flags);
+	return io_buffer_unregister(cmd, index, issue_flags);
 }
 
 static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
@@ -2396,7 +2396,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
 		goto out;
 
 	/*
-	 * io_buffer_unregister_bvec() doesn't access the ubq or io,
+	 * io_buffer_unregister() doesn't access the ubq or io,
 	 * so no need to validate the q_id, tag, or task
 	 */
 	if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF)
@@ -2466,7 +2466,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
 
 		/* can't touch 'ublk_io' any more */
 		if (buf_idx != UBLK_INVALID_BUF_IDX)
-			io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
+			io_buffer_unregister(cmd, buf_idx, issue_flags);
 		if (req_op(req) == REQ_OP_ZONE_APPEND)
 			req->__sector = addr;
 		if (compl)
include/linux/io_uring/cmd.h +20 -6
index 8881fb8da5e6..73f8ff9317d7 100644
@@ -103,6 +103,12 @@ int io_uring_kmbuf_recycle(struct io_uring_cmd *cmd, unsigned int buf_group,
 
 bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, unsigned int buf_group,
 			    unsigned int issue_flags);
+
+int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
+			       void (*release)(void *), unsigned int index,
+			       unsigned int issue_flags);
+int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
+			 unsigned int issue_flags);
 #else
 static inline int
 io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -183,6 +189,20 @@ static inline bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd,
 {
 	return false;
 }
+static inline int io_buffer_register_request(struct io_uring_cmd *cmd,
+					     struct request *rq,
+					     void (*release)(void *),
+					     unsigned int index,
+					     unsigned int issue_flags)
+{
+	return -EOPNOTSUPP;
+}
+static inline int io_buffer_unregister(struct io_uring_cmd *cmd,
+				       unsigned int index,
+				       unsigned int issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
@@ -232,10 +252,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret,
 	return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true);
 }
 
-int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
-			    void (*release)(void *), unsigned int index,
-			    unsigned int issue_flags);
-int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
-			      unsigned int issue_flags);
-
 #endif /* _LINUX_IO_URING_CMD_H */
io_uring/rsrc.c +7 -7
index fa41cae5e922..2aac2778e5c1 100644
@@ -936,9 +936,9 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
 	return ret;
 }
 
-int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
-			    void (*release)(void *), unsigned int index,
-			    unsigned int issue_flags)
+int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
+			       void (*release)(void *), unsigned int index,
+			       unsigned int issue_flags)
 {
 	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
 	struct io_rsrc_data *data = &ctx->buf_table;
@@ -998,10 +998,10 @@ int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
 	io_ring_submit_unlock(ctx, issue_flags);
 	return ret;
 }
-EXPORT_SYMBOL_GPL(io_buffer_register_bvec);
+EXPORT_SYMBOL_GPL(io_buffer_register_request);
 
-int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
-			      unsigned int issue_flags)
+int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
+			 unsigned int issue_flags)
 {
 	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
 	struct io_rsrc_data *data = &ctx->buf_table;
@@ -1031,7 +1031,7 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
 	io_ring_submit_unlock(ctx, issue_flags);
 	return ret;
 }
-EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec);
+EXPORT_SYMBOL_GPL(io_buffer_unregister);
 
 static int validate_fixed_range(u64 buf_addr, size_t len,
 				const struct io_mapped_ubuf *imu)
Signature · 1 line
2.47.3
author maintainer
#46 lore raw

PATCH v4 21/25 io_uring/rsrc: split io_buffer_register_request() logic

In reply to Joanne Koong

Split the main initialization logic in io_buffer_register_request() into a helper function.

This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which will be reusing this logic.

---
 io_uring/rsrc.c | 84 ++++++++++++++++++++++++++++++-------------------
 1 file changed, 51 insertions(+), 33 deletions(-)
io_uring/rsrc.c +51 -33
index 2aac2778e5c1..63ddadca116b 100644
@@ -936,64 +936,82 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
 	return ret;
 }
 
-int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
-			       void (*release)(void *), unsigned int index,
-			       unsigned int issue_flags)
+static struct io_mapped_ubuf *io_kernel_buffer_init(struct io_ring_ctx *ctx,
+						    unsigned int nr_bvecs,
+						    unsigned int total_bytes,
+						    u8 dir,
+						    void (*release)(void *),
+						    void *priv,
+						    unsigned int index)
 {
-	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
 	struct io_rsrc_data *data = &ctx->buf_table;
-	struct req_iterator rq_iter;
 	struct io_mapped_ubuf *imu;
 	struct io_rsrc_node *node;
-	struct bio_vec bv;
-	unsigned int nr_bvecs = 0;
-	int ret = 0;
 
-	io_ring_submit_lock(ctx, issue_flags);
-	if (index >= data->nr) {
-		ret = -EINVAL;
-		goto unlock;
-	}
+	if (index >= data->nr)
+		return ERR_PTR(-EINVAL);
 	index = array_index_nospec(index, data->nr);
 
-	if (data->nodes[index]) {
-		ret = -EBUSY;
-		goto unlock;
-	}
+	if (data->nodes[index])
+		return ERR_PTR(-EBUSY);
 
 	node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
-	if (!node) {
-		ret = -ENOMEM;
-		goto unlock;
-	}
+	if (!node)
+		return ERR_PTR(-ENOMEM);
 
-	/*
-	 * blk_rq_nr_phys_segments() may overestimate the number of bvecs
-	 * but avoids needing to iterate over the bvecs
-	 */
-	imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq));
+	imu = io_alloc_imu(ctx, nr_bvecs);
 	if (!imu) {
 		kfree(node);
-		ret = -ENOMEM;
-		goto unlock;
+		return ERR_PTR(-ENOMEM);
 	}
 
 	imu->ubuf = 0;
-	imu->len = blk_rq_bytes(rq);
+	imu->len = total_bytes;
 	imu->acct_pages = 0;
 	imu->folio_shift = PAGE_SHIFT;
+	imu->nr_bvecs = nr_bvecs;
 	refcount_set(&imu->refs, 1);
 	imu->release = release;
-	imu->priv = rq;
+	imu->priv = priv;
 	imu->is_kbuf = true;
-	imu->dir = 1 << rq_data_dir(rq);
+	imu->dir = 1 << dir;
+
+	node->buf = imu;
+	data->nodes[index] = node;
+
+	return imu;
+}
+
+int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
+			       void (*release)(void *), unsigned int index,
+			       unsigned int issue_flags)
+{
+	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+	struct req_iterator rq_iter;
+	struct io_mapped_ubuf *imu;
+	struct bio_vec bv;
+	/*
+	 * blk_rq_nr_phys_segments() may overestimate the number of bvecs
+	 * but avoids needing to iterate over the bvecs
+	 */
+	unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq);
+	unsigned int total_bytes = blk_rq_bytes(rq);
+	int ret = 0;
 
+	io_ring_submit_lock(ctx, issue_flags);
+
+	imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, rq_data_dir(rq),
+				    release, rq, index);
+	if (IS_ERR(imu)) {
+		ret = PTR_ERR(imu);
+		goto unlock;
+	}
+
+	nr_bvecs = 0;
 	rq_for_each_bvec(bv, rq, rq_iter)
 		imu->bvec[nr_bvecs++] = bv;
 	imu->nr_bvecs = nr_bvecs;
 
-	node->buf = imu;
-	data->nodes[index] = node;
 unlock:
 	io_ring_submit_unlock(ctx, issue_flags);
 	return ret;
Signature · 1 line
2.47.3
author maintainer
#47 lore raw

PATCH v4 22/25 io_uring/rsrc: Allow buffer release callback to be optional

In reply to Joanne Koong

This is a preparatory patch for supporting kernel-populated buffers in fuse io-uring, which does not need a release callback.

---
 io_uring/rsrc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
io_uring/rsrc.c +2 -1
index 63ddadca116b..dc43aab0f019 100644
@@ -148,7 +148,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)
 
 	if (imu->acct_pages)
 		io_unaccount_mem(ctx->user, ctx->mm_account, imu->acct_pages);
-	imu->release(imu->priv);
+	if (imu->release)
+		imu->release(imu->priv);
 	io_free_imu(ctx, imu);
 }
 
Signature · 1 line
2.47.3
author maintainer
#48 lore raw

PATCH v4 23/25 io_uring/rsrc: add io_buffer_register_bvec()

In reply to Joanne Koong

Add io_buffer_register_bvec() for registering a bvec array.

This is a preparatory patch for fuse-over-io-uring zero-copy.

---
 include/linux/io_uring/cmd.h | 12 ++++++++++++
 io_uring/rsrc.c              | 31 +++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
include/linux/io_uring/cmd.h +12 -0
index 73f8ff9317d7..7dde6e2af05b 100644
@@ -107,6 +107,9 @@ bool io_uring_is_kmbuf_ring(struct io_uring_cmd *cmd, unsigned int buf_group,
 int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
 			       void (*release)(void *), unsigned int index,
 			       unsigned int issue_flags);
+int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs,
+			    unsigned int nr_bvecs, unsigned int total_bytes,
+			    u8 dir, unsigned int index, unsigned int issue_flags);
 int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
 			 unsigned int issue_flags);
 #else
@@ -197,6 +200,15 @@ static inline int io_buffer_register_request(struct io_uring_cmd *cmd,
 {
 	return -EOPNOTSUPP;
 }
+static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd,
+					  const struct bio_vec *bvs,
+					  unsigned int nr_bvecs,
+					  unsigned int total_bytes, u8 dir,
+					  unsigned int index,
+					  unsigned int issue_flags)
+{
+	return -EOPNOTSUPP;
+}
 static inline int io_buffer_unregister(struct io_uring_cmd *cmd,
 				       unsigned int index,
 				       unsigned int issue_flags)
io_uring/rsrc.c +31 -0
index dc43aab0f019..b6350812255b 100644
@@ -1019,6 +1019,37 @@ int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
 }
 EXPORT_SYMBOL_GPL(io_buffer_register_request);
 
+/*
+ * This internally makes a copy of the bio_vec array. The memory bvs points to
+ * can be freed as soon as this returns.
+ */
+int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs,
+			    unsigned int nr_bvecs, unsigned int total_bytes,
+			    u8 dir, unsigned int index,
+			    unsigned int issue_flags)
+{
+	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+	struct io_mapped_ubuf *imu;
+	struct bio_vec *bvec;
+	unsigned int i;
+
+	io_ring_submit_lock(ctx, issue_flags);
+	imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, NULL,
+				    NULL, index);
+	if (IS_ERR(imu)) {
+		io_ring_submit_unlock(ctx, issue_flags);
+		return PTR_ERR(imu);
+	}
+
+	bvec = imu->bvec;
+	for (i = 0; i < nr_bvecs; i++)
+		bvec[i] = bvs[i];
+
+	io_ring_submit_unlock(ctx, issue_flags);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(io_buffer_register_bvec);
+
 int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
 			 unsigned int issue_flags)
 {
Signature · 1 line
2.47.3
author maintainer
#49 lore raw

PATCH v4 24/25 fuse: add zero-copy over io-uring

In reply to Joanne Koong

Implement zero-copy data transfer for fuse over io-uring, eliminating memory copies between kernel and userspace for read/write operations.

This is only allowed on privileged servers and requires the server to preregister the following:
a) a sparse buffer corresponding to the queue depth
b) a fixed buffer at index queue_depth (the tail of the buffers)
c) a kernel-managed buffer ring

The sparse buffer is where the client's pages reside. The fixed buffer at the tail is where the headers (struct fuse_uring_req_header) are placed. The kernel-managed buffer ring is where any non-zero-copied args reside (eg out headers).

Benchmarks with bs=1M showed approximately the following differences in throughput:
direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s)
buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)

The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1

---
 fs/fuse/dev.c             |   7 +-
 fs/fuse/dev_uring.c       | 166 ++++++++++++++++++++++++++++++--------
 fs/fuse/dev_uring_i.h     |  11 +++
 fs/fuse/fuse_dev_i.h      |   1 +
 include/uapi/linux/fuse.h |   6 +-
 5 files changed, 155 insertions(+), 36 deletions(-)
fs/fuse/dev.c +5 -2
index ceb5d6a553c0..bdb2b0d01e0c 100644
@@ -1229,10 +1229,13 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,
 
 	for (i = 0; !err && i < numargs; i++)  {
 		struct fuse_arg *arg = &args[i];
-		if (i == numargs - 1 && argpages)
+		if (i == numargs - 1 && argpages) {
+			if (cs->skip_folio_copy)
+				return 0;
 			err = fuse_copy_folios(cs, arg->size, zeroing);
-		else
+		} else {
 			err = fuse_copy_one(cs, arg->value, arg->size);
+		}
 	}
 	return err;
 }
fs/fuse/dev_uring.c +134 -32

335 lines not on this page · raw message

fs/fuse/dev_uring_i.h +11 -0
index ac6da80c3d70..e32405d7b3bd 100644
@@ -58,6 +58,9 @@ struct fuse_ring_ent {
 			 */
 			unsigned int ringbuf_buf_id;
 			unsigned int fixed_buf_id;
+
+			/* True if the request's pages are being zero-copied */
+			bool zero_copied;
 		};
 	};
 
@@ -124,6 +127,14 @@ struct fuse_ring_queue {
 
 	/* synchronized by the queue lock */
 	struct io_buffer_list *bufring;
+
+	/*
+	 * True if zero copy should be used for payloads. This is only enabled
+	 * on privileged servers. Kernel-managed ring buffers must be enabled
+	 * in order to use zero copy.
+	 */
+	bool use_zero_copy : 1;
+	unsigned int zero_copy_depth;
 };
 
 /**
fs/fuse/fuse_dev_i.h +1 -0
index aa1d25421054..67b5bed451fe 100644
@@ -39,6 +39,7 @@ struct fuse_copy_state {
 	bool is_uring:1;
 	/* if set, use kaddr; otherwise use pg */
 	bool is_kaddr:1;
+	bool skip_folio_copy:1;
 	struct {
 		unsigned int copied_sz; /* copied size into the user buffer */
 	} ring;
include/uapi/linux/fuse.h +4 -2
index b49c8d3b9ab6..2c44219f0062 100644
@@ -242,7 +242,7 @@
  *  - add FUSE_NOTIFY_PRUNE
  *
  *  7.46
- *  - add fuse_uring_cmd_req init flags
+ *  - add fuse_uring_cmd_req init flags and queue_depth
  */
 
 #ifndef _LINUX_FUSE_H
@@ -1299,6 +1299,7 @@ enum fuse_uring_cmd {
 
 /* fuse_uring_cmd_req init flags */
 #define FUSE_URING_BUF_RING	(1 << 0)
+#define FUSE_URING_ZERO_COPY	(1 << 1)
 
 /**
  * In the 80B command area of the SQE.
@@ -1315,10 +1316,11 @@ struct fuse_uring_cmd_req {
 	union {
 		struct {
 			uint16_t flags;
+			uint16_t queue_depth;
 		} init;
 	};
 
-	uint8_t padding[4];
+	uint8_t padding[2];
 };
 
 #endif /* _LINUX_FUSE_H */
Signature · 1 line
2.47.3
author maintainer
#50 lore raw

PATCH v4 25/25 docs: fuse: add io-uring bufring and zero-copy documentation

In reply to Joanne Koong

Add documentation for fuse over io-uring usage of kernel-managed bufrings and zero-copy.

---
 .../filesystems/fuse/fuse-io-uring.rst        | 59 ++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)
Documentation/filesystems/fuse/fuse-io-uring.rst +58 -1
index d73dd0dbd238..11c244b63d25 100644
@@ -95,5 +95,62 @@ Sending requests with CQEs
  |    <fuse_unlink()                         |
  |  <sys_unlink()                            |
 
+Kernel-managed buffer rings
+===========================
 
-
+Kernel-managed buffer rings have two main advantages:
+
+* eliminates the overhead of pinning/unpinning user pages and translating
+  virtual addresses for every server-kernel interaction
+* reduces buffer memory allocation requirements
+
+In order to use buffer rings, the server must preregister the following:
+
+* a fixed buffer at index 0. This is where the headers will reside
+* a kernel-managed buffer ring. This is where the payload will reside
+
+At a high-level, this is how fuse uses buffer rings:
+
+* The server registers a kernel-managed buffer ring. In the kernel this
+  allocates the pages needed for the buffers and vmaps them. The server
+  obtains the virtual address for the buffers through an mmap call on the ring
+  fd.
+* When there is a request from a client, fuse will select a buffer from the
+  ring if there is any payload that needs to be copied, copy over the payload
+  to the selected buffer, and copy over the headers to the fixed buffer at
+  index 0, at the buffer id that corresponds to the server (which the server
+  needs to specify through sqe->buf_index).
+* The server obtains a cqe representing the request. The cqe flag will have
+  IORING_CQE_F_BUFFER set if a selected buffer was used for the payload. The
+  buffer id is stashed in cqe->flags (through IORING_CQE_BUFFER_SHIFT). The
+  server can directly access the payload by using that buffer id to calculate
+  the offset into the virtual address obtained for the buffers.
+* The server processes the request and then sends a
+  FUSE_URING_CMD_COMMIT_AND_FETCH sqe with the reply.
+* When the kernel handles the sqe, it will process the reply and if there is a
+  next request, it will reuse the same selected buffer for the request. If
+  there is no next request, it will recycle the buffer back to the ring.
+
+Zero-copy
+=========
+
+Fuse io-uring zero-copy allows the server to directly read from / write to the
+client's pages and bypass any intermediary buffer copies. This is only allowed
+on privileged servers.
+
+In order to use zero-copy, the server must pregister the following:
+
+* a sparse buffer for every entry in the queue. This is where the client's
+  pages will reside
+* a fixed buffer at index queue_depth (tailing the sparse buffer).
+  This is where the headers will reside
+* a kernel-managed buffer ring. This is where any non-zero-copied payload (eg
+  out headers) will reside
+
+When the client issues a read/write, fuse stores the client's underlying pages
+in the sparse buffer entry corresponding to the ent in the queue. The server
+can then issue reads/writes on these pages through io_uring rw operations.
+Please note that the server is not able to directly access these pages, it
+must go through the io-uring interface to read/write to them. The pages are
+unregistered once the server replies to the request. Non-zero-copyable
+payload (if needed) is placed in a buffer from the kernel-managed buffer ring.
Signature · 1 line
2.47.3
maintainer
#51 lore raw

Re: [PATCH v4 25/25] docs: fuse: add io-uring bufring and zero-copy documentation

Quoted 20 lines · Joanne Koong

Add documentation for fuse over io-uring usage of kernel-managed bufrings and zero-copy.

---
 .../filesystems/fuse/fuse-io-uring.rst        | 59 ++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)
Documentation/filesystems/fuse/fuse-io-uring.rst +2 -1
index d73dd0dbd238..11c244b63d25 100644
@@ -95,5 +95,62 @@ Sending requests with CQEs
  |    <fuse_unlink()                         |
  |  <sys_unlink()                            |
 
+Kernel-managed buffer rings
+===========================
 
-

Documentation/filesystems/fuse/fuse-io-uring.rst Sending requests with CQEs · line 110

Quoted 10 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +10 -0
+Kernel-managed buffer rings have two main advantages:
+
+* eliminates the overhead of pinning/unpinning user pages and translating
+  virtual addresses for every server-kernel interaction
+* reduces buffer memory allocation requirements
+
+In order to use buffer rings, the server must preregister the following:
+
+* a fixed buffer at index 0. This is where the headers will reside
+* a kernel-managed buffer ring. This is where the payload will reside

Would you mind to add the actual liburing call for this? I think it would be helpful for anyone who wants to implement it.

Quoted 46 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +46 -0
+
+At a high-level, this is how fuse uses buffer rings:
+
+* The server registers a kernel-managed buffer ring. In the kernel this
+  allocates the pages needed for the buffers and vmaps them. The server
+  obtains the virtual address for the buffers through an mmap call on the ring
+  fd.
+* When there is a request from a client, fuse will select a buffer from the
+  ring if there is any payload that needs to be copied, copy over the payload
+  to the selected buffer, and copy over the headers to the fixed buffer at
+  index 0, at the buffer id that corresponds to the server (which the server
+  needs to specify through sqe->buf_index).
+* The server obtains a cqe representing the request. The cqe flag will have
+  IORING_CQE_F_BUFFER set if a selected buffer was used for the payload. The
+  buffer id is stashed in cqe->flags (through IORING_CQE_BUFFER_SHIFT). The
+  server can directly access the payload by using that buffer id to calculate
+  the offset into the virtual address obtained for the buffers.
+* The server processes the request and then sends a
+  FUSE_URING_CMD_COMMIT_AND_FETCH sqe with the reply.
+* When the kernel handles the sqe, it will process the reply and if there is a
+  next request, it will reuse the same selected buffer for the request. If
+  there is no next request, it will recycle the buffer back to the ring.
+
+Zero-copy
+=========
+
+Fuse io-uring zero-copy allows the server to directly read from / write to the
+client's pages and bypass any intermediary buffer copies. This is only allowed
+on privileged servers.
+
+In order to use zero-copy, the server must pregister the following:
+
+* a sparse buffer for every entry in the queue. This is where the client's
+  pages will reside
+* a fixed buffer at index queue_depth (tailing the sparse buffer).
+  This is where the headers will reside
+* a kernel-managed buffer ring. This is where any non-zero-copied payload (eg
+  out headers) will reside
+
+When the client issues a read/write, fuse stores the client's underlying pages
+in the sparse buffer entry corresponding to the ent in the queue. The server
+can then issue reads/writes on these pages through io_uring rw operations.
+Please note that the server is not able to directly access these pages, it
+must go through the io-uring interface to read/write to them. The pages are
+unregistered once the server replies to the request. Non-zero-copyable
+payload (if needed) is placed in a buffer from the kernel-managed buffer ring.
author maintainer
#52 lore raw

Re: [PATCH v4 25/25] docs: fuse: add io-uring bufring and zero-copy documentation

Quoted 23 lines · Bernd Schubert
Quoted 19 lines · Joanne Koong

Add documentation for fuse over io-uring usage of kernel-managed bufrings and zero-copy.

---
 .../filesystems/fuse/fuse-io-uring.rst        | 59 ++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)
Documentation/filesystems/fuse/fuse-io-uring.rst +2 -0
index d73dd0dbd238..11c244b63d25 100644
@@ -95,5 +95,62 @@ Sending requests with CQEs
  |    <fuse_unlink()                         |
  |  <sys_unlink()                            |
 
+Kernel-managed buffer rings
+===========================
 
Quoted 14 lines

Documentation/filesystems/fuse/fuse-io-uring.rst Sending requests with CQEs · line 110

Quoted 11 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +10 -1
-
+Kernel-managed buffer rings have two main advantages:
+
+* eliminates the overhead of pinning/unpinning user pages and translating
+  virtual addresses for every server-kernel interaction
+* reduces buffer memory allocation requirements
+
+In order to use buffer rings, the server must preregister the following:
+
+* a fixed buffer at index 0. This is where the headers will reside
+* a kernel-managed buffer ring. This is where the payload will reside

Would you mind to add the actual liburing call for this? I think it would be helpful for anyone who wants to implement it.

Good idea, I think what might be even more helpful is to refer them to the libfuse lib/lowlevel_fuse.c file for reference on how to set this up. Then they could just copy/paste the code directly from libfuse if they're trying to do something similar. I'll add this in for the next version.

Thanks,
Joanne

Quoted 3 lines
Quoted 3 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +2 -0
+
+At a high-level, this is how fuse uses buffer rings:
author maintainer
#53 lore raw
In reply to Joanne Koong
Quoted 11 lines · Joanne Koong

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

Quoted 4 lines

The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.

Could anyone on the fuse side review the fuse changes in patches 19 and 24?

Thanks,
Joanne

Quoted 117 lines

There was a previous submission for supporting registered buffers in fuse [1] but that was abandoned in favor of using kernel-managed buffer rings, which, once incremental buffer consumption is added in a later patchset, gives significant memory usage advantages in allowing the full buffer capacity to be utilized across multiple requests, as well as offers more flexibility for future additions. As well, it also makes the userspace side setup simpler. The relevant refactoring fuse patches from the previous submission are carried over into this one.

Benchmarks for zero-copy (patch 24) show approximately the following differences in throughput for bs=1M:

direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s) buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)

The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1

This series is on top of commit b71e635feefc in the io-uring tree.

The libfuse changes can be found in [2]. This has a dependency on the liburing changes in [3]. To test the server, you can run it with:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -o io_uring_zero_copy -o io_uring_q_depth=8

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://github.com/joannekoong/libfuse/tree/zero_copy
[3] https://github.com/joannekoong/liburing/tree/kmbuf

v3: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3 -> v4:
* Get rid of likely()s and get rid of going through cmd interface layer (Gabriel)
* Fix io_uring_cmd_fixed_index_get() to return back the node pointer (Caleb)
* Add documentation for io_buffer_register_bvec (Caleb)
* Remove WARN_ON_ONCE() for io_buffer_unregister() call (Caleb)

v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* fix casting between void * and u64 for 32-bit architectures (kernel test robot)
* add newline for documentation bullet points (kernel test robot)
* fix unrecognized "boolean" (kernel test robot), switch it to a flag (me)

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* drop fuse buffer cleanup on ring death, which makes things a lot simpler (Jens)
- this drops a lot of things (eg needing ring_ctx tracking, needing callback for ring death, etc)
* drop fixed buffer pinning altogether and just do lookup every time (Jens) (didn't significantly affect the benchmark results seen)
* fix spelling mistake in docs (Askar)
* use -EALREADY for pinning already pinned bufring, return PTR_ERR for registration instead of err, move initializations to outside locks (Caleb)
* drop fuse patches for zero-ed out headers (me)

Joanne Koong (25):
io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic
helpers
io_uring/kbuf: rename io_unregister_pbuf_ring() to
io_unregister_buf_ring()
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: add mmap support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: add recycling for kernel managed buffer rings
io_uring: add io_uring_fixed_index_get() and
io_uring_fixed_index_put()
io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: export io_ring_buffer_select() io_uring/kbuf: return buffer id in buffer selection io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: support buffer copying for kernel addresses
fuse: add io-uring kernel-managed buffer ring io_uring/rsrc: rename
io_buffer_register_bvec()/io_buffer_unregister_bvec()
io_uring/rsrc: split io_buffer_register_request() logic io_uring/rsrc: Allow buffer release callback to be optional io_uring/rsrc: add io_buffer_register_bvec()
fuse: add zero-copy over io-uring
docs: fuse: add io-uring bufring and zero-copy documentation

 Documentation/block/ublk.rst                  |  14 +-
 .../filesystems/fuse/fuse-io-uring.rst        |  59 +-
 drivers/block/ublk_drv.c                      |  18 +-
 fs/fuse/dev.c                                 |  30 +-
 fs/fuse/dev_uring.c                           | 692 ++++++++++++++----
 fs/fuse/dev_uring_i.h                         |  42 +-
 fs/fuse/fuse_dev_i.h                          |   8 +-
 include/linux/io_uring/buf.h                  |  25 +
 include/linux/io_uring/cmd.h                  |  97 ++-
 include/linux/io_uring_types.h                |  10 +-
 include/uapi/linux/fuse.h                     |  17 +-
 include/uapi/linux/io_uring.h                 |  17 +-
 io_uring/kbuf.c                               | 355 +++++++--
 io_uring/kbuf.h                               |  19 +-
 io_uring/memmap.c                             | 117 ++-
 io_uring/memmap.h                             |   4 +
 io_uring/register.c                           |   9 +-
 io_uring/rsrc.c                               | 183 ++++-
 io_uring/uring_cmd.c                          |   6 +-
 19 files changed, 1447 insertions(+), 275 deletions(-)
 create mode 100644 include/linux/io_uring/buf.h

--
2.47.3

maintainer
#54 lore raw
Quoted 12 lines · Joanne Koong
Quoted 11 lines · Joanne Koong

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

Quoted 6 lines
Quoted 4 lines

The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.

Could anyone on the fuse side review the fuse changes in patches 19 and 24?

I will really do this week, getting persistently other "urgent" work :/

Sorry for late reviews,
Bernd

author maintainer
#55 lore raw
Quoted 14 lines · Bernd Schubert
Quoted 12 lines · Joanne Koong
Quoted 11 lines · Joanne Koong

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.

Could anyone on the fuse side review the fuse changes in patches 19 and 24?

I will really do this week, getting persistently other "urgent" work :/

No worries, thank you Bernd!

Quoted 3 lines

Sorry for late reviews,
Bernd

#56 lore raw
In reply to Joanne Koong
Quoted 9 lines · Joanne Koong

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

Can you split that series out as it also has other applications and smaller series might be easier to review?

author maintainer
#57 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Joanne Koong

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

Quoted 7 lines
Quoted 4 lines

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

Can you split that series out as it also has other applications and smaller series might be easier to review?

I'll split this out and send it by itself.

Thanks,
Joanne

Quoted 1 line