Skip to content

fuse/io-uring: add kernel-managed buffer rings and zero-copy

Joanne Koong io-uring linux-fsdevel oe-kbuild-all 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest25 patches 31 repliesIn review
  2. v325 patches 38 repliesSuperseded
  3. v225 patches 1 replySuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v130 patches29 repliesSuperseded

Changes in v4 from the cover letter

* Get rid of likely()s and get rid of going through cmd interface layer (Gabriel)
* Fix io_uring_cmd_fixed_index_get() to return back the node pointer (Caleb)
* Add documentation for io_buffer_register_bvec (Caleb)
* Remove WARN_ON_ONCE() for io_buffer_unregister() call (Caleb)

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/25io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic helpersio_uring/kbuf.c+82 -410 replies
2/25io_uring/kbuf: rename io_unregister_pbuf_ring() to io_unregister_buf_ring()io_uring/kbuf.c io_uring/kbuf.h io_uring/register.c+3 -30 replies
3/25io_uring/kbuf: add support for kernel-managed buffer ringsinclude/uapi/linux/io_uring.h io_uring/kbuf.c io_uring/kbuf.h +3 more+217 -46 replies
4/25io_uring/kbuf: add mmap support for kernel-managed buffer ringsinclude/uapi/linux/io_uring.h io_uring/kbuf.c io_uring/kbuf.h +1 more+18 -50 replies
5/25io_uring/kbuf: support kernel-managed buffer rings in buffer selectioninclude/linux/io_uring_types.h io_uring/kbuf.c+16 -80 replies
6/25io_uring/kbuf: add buffer ring pinning/unpinninginclude/linux/io_uring/cmd.h io_uring/kbuf.c io_uring/kbuf.h+70 -02 replies
7/25io_uring/kbuf: add recycling for kernel managed buffer ringsinclude/linux/io_uring/cmd.h io_uring/kbuf.c+55 -03 replies
8/25io_uring: add io_uring_fixed_index_get() and io_uring_fixed_index_put()include/linux/io_uring/cmd.h io_uring/rsrc.c+79 -02 replies
9/25io_uring/kbuf: add io_uring_is_kmbuf_ring()include/linux/io_uring/cmd.h io_uring/kbuf.c+29 -00 replies
10/25io_uring/kbuf: export io_ring_buffer_select()include/linux/io_uring/buf.h io_uring/kbuf.c+30 -30 replies
11/25io_uring/kbuf: return buffer id in buffer selectioninclude/linux/io_uring/cmd.h include/linux/io_uring_types.h io_uring/kbuf.c+8 -30 replies
12/25io_uring/cmd: set selected buffer index in __io_uring_cmd_done()io_uring/uring_cmd.c+5 -10 replies
13/25fuse: refactor io-uring logic for getting next fuse requestfs/fuse/dev_uring.c+28 -500 replies
14/25fuse: refactor io-uring header copying to ringfs/fuse/dev_uring.c+21 -180 replies
15/25fuse: refactor io-uring header copying from ringfs/fuse/dev_uring.c+18 -61 reply
16/25fuse: use enum types for header copyingfs/fuse/dev_uring.c+47 -100 replies
17/25fuse: refactor setting up copy state for payload copyingfs/fuse/dev_uring.c+24 -140 replies
18/25fuse: support buffer copying for kernel addressesfs/fuse/dev.c fs/fuse/fuse_dev_i.h+23 -74 replies
19/25fuse: add io-uring kernel-managed buffer ringfs/fuse/dev_uring.c fs/fuse/dev_uring_i.h include/uapi/linux/fuse.h+389 -696 replies
20/25io_uring/rsrc: rename io_buffer_register_bvec()/io_buffer_unregister_bvec()Documentation/block/ublk.rst drivers/block/ublk_drv.c include/linux/io_uring/cmd.h +1 more+43 -290 replies
21/25io_uring/rsrc: split io_buffer_register_request() logicio_uring/rsrc.c+51 -330 replies
22/25io_uring/rsrc: Allow buffer release callback to be optionalio_uring/rsrc.c+2 -10 replies
23/25io_uring/rsrc: add io_buffer_register_bvec()include/linux/io_uring/cmd.h io_uring/rsrc.c+43 -00 replies
24/25fuse: add zero-copy over io-uringfs/fuse/dev.c fs/fuse/dev_uring.c fs/fuse/dev_uring_i.h +2 more+155 -360 replies
25/25docs: fuse: add io-uring bufring and zero-copy documentationDocumentation/filesystems/fuse/fuse-io-uring.rst+58 -12 replies

Reviews trailers per patch

11 of 25 patches have a Reviewed-by or Acked-by.

PatchBernd SchubertCaleb Sander Mateos
1/25
2/25
3/25
4/25
5/25
6/25
7/25
8/25R-b
9/25
10/25
11/25
12/25
13/25R-b
14/25R-b
15/25R-b
16/25R-b
17/25R-b
18/25R-b
19/25
20/25R-b
21/25R-b
22/25R-b
23/25R-b
24/25
25/25

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCH v4 00/25] fuse/io-uring: add kernel-managed buffer rings and zero-copy

This series adds buffer ring and zero-copy capabilities to fuse over io-uring. This requires adding a new kernel-managed buf (kmbuf) ring type to io-uring where the buffers are provided and managed by the kernel instead of by userspace.

On the io-uring side, the kmbuf interface is basically identical to pbufs. They differ mostly in how the memory region is set up and whether it is userspace or kernel that recycles back the buffer. Internally, the IOBL_KERNEL_MANAGED flag is used to mark the buffer ring as kernel-managed.

The zero-copy work builds on top of the infrastructure added for kernel-managed buffer rings (the bulk of which is in patch 19: "fuse: add io-uring kernel-managed buffer ring") and that informs some of the design choices for how fuse uses the kernel-managed buffer ring without zero-copy.

There was a previous submission for supporting registered buffers in fuse [1] but that was abandoned in favor of using kernel-managed buffer rings, which, once incremental buffer consumption is added in a later patchset, gives significant memory usage advantages in allowing the full buffer capacity to be utilized across multiple requests, as well as offers more flexibility for future additions. As well, it also makes the userspace side setup simpler. The relevant refactoring fuse patches from the previous submission are carried over into this one.

Benchmarks for zero-copy (patch 24) show approximately the following differences in throughput for bs=1M:

direct randreads: ~20% increase (~2100 MB/s -> ~2600 MB/s) buffered randreads: ~25% increase (~1900 MB/s -> 2400 MB/s) direct randwrites: no difference (~750 MB/s) buffered randwrites: ~10% increase (950 MB/s -> 1050 MB/s)

The benchmark was run using fio on the passthrough_hp server:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1

This series is on top of commit b71e635feefc in the io-uring tree.

The libfuse changes can be found in [2]. This has a dependency on the liburing changes in [3]. To test the server, you can run it with:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -o io_uring_zero_copy -o io_uring_q_depth=8

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
[2] https://github.com/joannekoong/libfuse/tree/zero_copy
[3] https://github.com/joannekoong/liburing/tree/kmbuf

v3: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3 -> v4:
* Get rid of likely()s and get rid of going through cmd interface layer (Gabriel)
* Fix io_uring_cmd_fixed_index_get() to return back the node pointer (Caleb)
* Add documentation for io_buffer_register_bvec (Caleb)
* Remove WARN_ON_ONCE() for io_buffer_unregister() call (Caleb)

v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* fix casting between void * and u64 for 32-bit architectures (kernel test robot)
* add newline for documentation bullet points (kernel test robot)
* fix unrecognized "boolean" (kernel test robot), switch it to a flag (me)

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* drop fuse buffer cleanup on ring death, which makes things a lot simpler (Jens)
- this drops a lot of things (eg needing ring_ctx tracking, needing callback for ring death, etc)
* drop fixed buffer pinning altogether and just do lookup every time (Jens) (didn't significantly affect the benchmark results seen)
* fix spelling mistake in docs (Askar)
* use -EALREADY for pinning already pinned bufring, return PTR_ERR for registration instead of err, move initializations to outside locks (Caleb)
* drop fuse patches for zero-ed out headers (me)

Joanne Koong (25):
io_uring/kbuf: refactor io_buf_pbuf_register() logic into generic
helpers
io_uring/kbuf: rename io_unregister_pbuf_ring() to
io_unregister_buf_ring()
io_uring/kbuf: add support for kernel-managed buffer rings io_uring/kbuf: add mmap support for kernel-managed buffer rings io_uring/kbuf: support kernel-managed buffer rings in buffer selection io_uring/kbuf: add buffer ring pinning/unpinning io_uring/kbuf: add recycling for kernel managed buffer rings
io_uring: add io_uring_fixed_index_get() and
io_uring_fixed_index_put()
io_uring/kbuf: add io_uring_is_kmbuf_ring()
io_uring/kbuf: export io_ring_buffer_select() io_uring/kbuf: return buffer id in buffer selection io_uring/cmd: set selected buffer index in __io_uring_cmd_done()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: support buffer copying for kernel addresses
fuse: add io-uring kernel-managed buffer ring io_uring/rsrc: rename
io_buffer_register_bvec()/io_buffer_unregister_bvec()
io_uring/rsrc: split io_buffer_register_request() logic io_uring/rsrc: Allow buffer release callback to be optional io_uring/rsrc: add io_buffer_register_bvec()
fuse: add zero-copy over io-uring
docs: fuse: add io-uring bufring and zero-copy documentation

 Documentation/block/ublk.rst                  |  14 +-
 .../filesystems/fuse/fuse-io-uring.rst        |  59 +-
 drivers/block/ublk_drv.c                      |  18 +-
 fs/fuse/dev.c                                 |  30 +-
 fs/fuse/dev_uring.c                           | 692 ++++++++++++++----
 fs/fuse/dev_uring_i.h                         |  42 +-
 fs/fuse/fuse_dev_i.h                          |   8 +-
 include/linux/io_uring/buf.h                  |  25 +
 include/linux/io_uring/cmd.h                  |  97 ++-
 include/linux/io_uring_types.h                |  10 +-
 include/uapi/linux/fuse.h                     |  17 +-
 include/uapi/linux/io_uring.h                 |  17 +-
 io_uring/kbuf.c                               | 355 +++++++--
 io_uring/kbuf.h                               |  19 +-
 io_uring/memmap.c                             | 117 ++-
 io_uring/memmap.h                             |   4 +
 io_uring/register.c                           |   9 +-
 io_uring/rsrc.c                               | 183 ++++-
 io_uring/uring_cmd.c                          |   6 +-
 19 files changed, 1447 insertions(+), 275 deletions(-)
 create mode 100644 include/linux/io_uring/buf.h
Signature · 1 line
2.47.3