fuse: add io-uring buffer pools and zero-copy
Subsystems (3)
Joanne Koong
maintainerv3 cover fuse: add io-uring buffer pools and zero-copy
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires CAP_SYS_ADMIN privileges and using registered buffers.
This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED operation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additional io_uring operation may negate the savings from eliminating the memcpy.
In benchmarks using passthrough_hp on a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).
Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -o io_uring_zero_copy -o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.
Thanks,
Joanne
[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/
Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers infrastructure instead of doing pinning logic in fuse. Get rid of header pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different naming
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Drop kernel managed buffers from io-uring infrastructure and move it to fuse
* Add visual diagrams and more documentatoin to commit messages and documentation patch
Joanne Koong (10):
fuse: separate next request fetching from sending logic
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: refactor setting up copy state for payload copying
fuse: use named constants for io-uring iovec indices
fuse: add io-uring buffer pools (implemented as buffer ring)
fuse: support registered buffers as buffer pools in io-uring
fuse: add zero-copy over io-uring
docs: fuse: add io-uring buffer pool and zero-copy documentation
.../filesystems/fuse/fuse-io-uring.rst | 194 +++++ fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 753 ++++++++++++++---- fs/fuse/dev_uring_i.h | 43 +- fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 43 +- 6 files changed, 876 insertions(+), 165 deletions(-)
base-commit: aec58539941927e8e0d1b74f9e134b0298c1e437
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 1/10 fuse: separate next request fetching from sending logic
Simplify the logic for fetching + sending off the next request.
This gets rid of fuse_uring_send_next_to_ring() which contained duplicated logic from fuse_uring_send(). This decouples request fetching from the send operation, which makes the control flow clearer and reduces unnecessary parameter passing.
--- fs/fuse/dev_uring.c | 78 ++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
@@ -709,34 +709,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,return err; } -/* -* Write data to the ring buffer and send the request to userspace, -* userspace will read it -* This is comparable with classical read(/dev/fuse) -*/ -static int fuse_uring_send_next_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req, -unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; -int err; -struct io_uring_cmd *cmd; - - err =fuse_uring_prepare_send (ent, req); -if (err) -return err; - -spin_lock (&queue->lock); - cmd = ent->cmd; - ent->cmd =NULL ; - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd,0 , issue_flags); -return 0 ; -} -/* * Make a ring entry available for fuse_req assignment */
@@ -832,11 +804,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, }/* -* Get the next fuse req and send it +* Get the next fuse req. +* +* Returns true if the next fuse request has been assigned to the ent. +* Else, there is no next fuse request and this returns false. */ -static void fuse_uring_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue, -unsigned int issue_flags) +static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, +struct fuse_ring_queue *queue) {int err;struct fuse_req *req;
@@ -848,10 +822,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_send_next_to_ring (ent, req, issue_flags); + err =fuse_uring_prepare_send (ent, req);if (err) goto retry; } + +return req !=NULL ; }static int fuse_ring_ent_set_commit (struct fuse_ring_ent *ent)
@@ -869,6 +845,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)return 0 ; } +static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, +ssize_t ret,unsigned int issue_flags) +{ +struct fuse_ring_queue *queue = ent->queue; + +spin_lock (&queue->lock); + ent->state =FRRS_USERSPACE ; +list_move_tail (&ent->list, &queue->ent_in_userspace); + ent->cmd =NULL ; +spin_unlock (&queue->lock); + +io_uring_cmd_done (cmd, ret, issue_flags); +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_chan *fch)
@@ -941,7 +931,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -fuse_uring_next_fuse_req (ent, queue, issue_flags); +if (fuse_uring_get_next_fuse_req (ent, queue)) +fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1190,20 +1181,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)return -EIOCBQUEUED ; } -static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, -ssize_t ret,unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; - -spin_lock (&queue->lock); - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); - ent->cmd =NULL ; -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd, ret, issue_flags); -} -/* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission
@@ -1220,8 +1197,9 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)if (!tw.cancel) { err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -fuse_uring_next_fuse_req (ent, queue, issue_flags); -return ; +if (!fuse_uring_get_next_fuse_req (ent, queue)) +return ; + err =0 ; } }else { err = -ECANCELED ;
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 2/10 fuse: refactor io-uring header copying to ring
In reply to Joanne KoongMove header copying to ring logic into a new copy_header_to_ring() function. This makes the copy_to_user() logic more clear and centralizes error handling / rate-limited logging.
--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +21 -18
@@ -570,6 +570,18 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static __always_inline int copy_header_to_ring(void __user *ring, +const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -632,13 +644,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;
@@ -654,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -685,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err =copy_to_user (&ent->headers->in_out, &req->in.h, -sizeof (req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 3/10 fuse: refactor io-uring header copying from ring
In reply to Joanne KoongMove header copying from ring logic into a new copy_header_from_ring() function. This makes the copy_from_user() logic more clear and centralizes error handling / rate-limited logging.
--- fs/fuse/dev_uring.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +18 -6
@@ -582,6 +582,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } +static __always_inline int copy_header_from_ring(void *header, +const void __user *ring, +size_t header_size) +{ +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -592,10 +604,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_from_user (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, +sizeof (ring_in_out));if (err) -return -EFAULT ; +return err; err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, &iter);
@@ -788,10 +800,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_ring *ring = ent->queue->ring;ssize_t err =0 ; - err =copy_from_user (&req->out.h, &ent->headers->in_out, -sizeof (req->out.h)); + err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, +sizeof (req->out.h));if (err) { - req->out.h.error = -EFAULT ; + req->out.h.error = err; goto out; }
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 4/10 fuse: use enum types for header copying
In reply to Joanne KoongUse enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses copying when buffer rings are added.
--- fs/fuse/dev_uring.c | 66 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 53 insertions(+), 13 deletions(-)
fs/fuse/dev_uring.c +53 -13
@@ -31,6 +31,15 @@ struct fuse_uring_pdu {static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +/* struct fuse_in_header / struct fuse_out_header */ +FUSE_URING_HEADER_IN_OUT , +/* per op code header */ +FUSE_URING_HEADER_OP , +/* struct fuse_uring_ent_in_out header */ +FUSE_URING_HEADER_RING_ENT , +}; +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -570,10 +579,33 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } -static __always_inline int copy_header_to_ring(void __user *ring, -const void *header, -size_t header_size) +static int ring_header_type_offset (enum fuse_uring_header_type type) { +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return 0 ; +case FUSE_URING_HEADER_OP : +return offsetof(struct fuse_uring_req_header , op_in); +case FUSE_URING_HEADER_RING_ENT : +return offsetof(struct fuse_uring_req_header , ring_ent_in_out); +default : +WARN_ONCE (1 ,"Invalid header type: %d\n" , type); +return -EINVAL ; + } +} + +static int copy_header_to_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +const void *header,size_t header_size) +{ +int offset =ring_header_type_offset (type); +void __user *ring; + +if (offset <0 ) +return offset; + + ring = (void __user *)ent->headers + offset; +if (copy_to_user (ring, header, header_size)) {pr_info_ratelimited ("Copying header to ring failed.\n" );return -EFAULT ;
@@ -582,10 +614,18 @@ static __always_inline int copy_header_to_ring(void __user *ring,return 0 ; } -static __always_inline int copy_header_from_ring(void *header, -const void __user *ring, -size_t header_size) +static int copy_header_from_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type,void *header, +size_t header_size) { +int offset =ring_header_type_offset (type); +const void __user *ring; + +if (offset <0 ) +return offset; + + ring = (void __user *)ent->headers + offset; +if (copy_from_user (header, ring, header_size)) {pr_info_ratelimited ("Copying header from ring failed.\n" );return -EFAULT ;
@@ -604,8 +644,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ring_in_out,sizeof (ring_in_out));if (err)return err;
@@ -656,7 +696,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_header_to_ring (&ent->headers->op_in, + err =copy_header_to_ring (ent,FUSE_URING_HEADER_OP , in_args->value, in_args->size);if (err)
@@ -676,8 +716,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; -return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ent_in_out,sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -706,7 +746,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ -return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +return copy_header_to_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->in.h,sizeof (req->in.h)); }
@@ -800,7 +840,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_ring *ring = ent->queue->ring;ssize_t err =0 ; - err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 5/10 fuse: refactor setting up copy state for payload copying
In reply to Joanne KoongAdd a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.
--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +24 -14
@@ -634,6 +634,27 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; } +static int setup_fuse_copy_state (struct fuse_copy_state *cs, +struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent *ent,int dir, +struct iov_iter *iter) +{ +int err; + + err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } + +fuse_copy_init (cs, dir ==ITER_DEST , iter); + + cs->is_uring = true; + cs->req = req; + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -649,15 +670,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, - &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter);if (err)return err; -fuse_copy_init (&cs, false, &iter); - cs.is_uring = true; - cs.req = req; - err =fuse_copy_out_args (&cs, args, ring_in_out.payload_sz);fuse_copy_finish (&cs);return err;
@@ -680,15 +696,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter); +if (err)return err; - } - -fuse_copy_init (&cs, true, &iter); - cs.is_uring = true; - cs.req = req;if (num_args >0 ) {/*
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 6/10 fuse: use named constants for io-uring iovec indices
In reply to Joanne KoongReplace magic indices 0 and 1 for the iovec array with named constants FUSE_URING_IOV_HEADERS and FUSE_URING_IOV_PAYLOAD. This makes the usages self-documenting and prepares for buffer ring support which will also reference these iovec slots by index.
--- fs/fuse/dev_uring.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +13 -11
@@ -18,7 +18,8 @@ MODULE_PARM_DESC(enable_uring, "Enable userspace communication through io-uring");#define FUSE_URING_IOV_SEGS 2/* header and payload */ - +#define FUSE_URING_IOV_HEADERS 0 +#define FUSE_URING_IOV_PAYLOAD 1bool fuse_uring_enabled (void ) {
@@ -1057,8 +1058,8 @@ static void fuse_uring_do_register(struct fuse_ring_ent *ent, }/* -* sqe->addr is a ptr to an iovec array, iov[0] has the headers, iov[1] -* the payload +* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the +* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload */ static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe,struct iovec iov[FUSE_URING_IOV_SEGS ])
@@ -1088,8 +1089,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, {struct fuse_ring *ring = queue->ring;struct fuse_ring_ent *ent; -size_t payload_size;struct iovec iov[FUSE_URING_IOV_SEGS ]; +struct iovec *headers, *payload;int err; err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);
@@ -1100,15 +1101,16 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, } err = -EINVAL ; -if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) { -pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); + headers = &iov[FUSE_URING_IOV_HEADERS ]; +if (headers->iov_len <sizeof (struct fuse_uring_req_header )) { +pr_info_ratelimited ("Invalid header len %zu\n" , headers->iov_len);return ERR_PTR (err); } - payload_size = iov[1 ].iov_len; -if (payload_size < ring->max_payload_sz) { + payload = &iov[FUSE_URING_IOV_PAYLOAD ]; +if (payload->iov_len < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , - payload_size); + payload->iov_len);return ERR_PTR (err); }
@@ -1120,8 +1122,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,INIT_LIST_HEAD (&ent->list); ent->queue = queue; - ent->headers = iov[0 ].iov_base; - ent->payload = iov[1 ].iov_base; + ent->headers = headers->iov_base; + ent->payload = payload->iov_base;atomic_inc (&ring->queue_refs);return ent;
Signature · 1 line
2.52.0
Joanne Koong
maintainerPATCH v3 7/10 fuse: add io-uring buffer pools (implemented as buffer ring)
In reply to Joanne KoongRight now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
To use this, the server must set the FUSE_URING_BUFPOOL flag during registration and provide the payload buffer pool through the iovec payload in the first sqe. The server must set init.buf_size and init.nr_bufs in the struct fuse_uring_cmd_req it passes to the kernel to describe the buffer pool configuration. Buffer pool usage is set on a per-ring basis. All subsequent registration SQEs for the same ring must pass in consistent flags.
Internally, a buffer ring data structure is used for managing the buffer pool. The kernel manages buffer selection and recycling through this ring structure. A payload buffer is selected from the ring on demand if the request requires a payload buffer and recycled after the request completes if used.
The contiguous buffer pool passed in is partitioned into individual buffers according to the buf_size passed in at registration time.
PAYLOAD BUFFER POOL (contiguous, provided via iov[1] in first sqe):
|<-------------- payload_size ------------>|
+--------- --+-----------+-----------+-----+
| buf [0] | buf [1] | buf [2] | ... |
| buf_size | buf_size | buf_size | ... |
+--------- --+-----------+-----------+-----+The buffer ring logic is as follows:
select: buf = bufs[head % nbufs]; head++ recycle: bufs[tail % nbufs] = buf; tail++ empty: tail == head (no buffers available) full: tail - head >= nbufs
Buffer ring request flow
------------------------
| Kernel | FUSE daemon | | | [client request arrives] | | >fuse_uring_send() | | [select payload buf from pool] | | >fuse_uring_select_buffer() | | [copy headers to ent] | | >copy_header_to_ring() | | [copy payload to selected buf] | | >fuse_uring_copy_to_ring() | | [set buf_id in ent_in_out header] | | >io_uring_cmd_done() | | | [CQE received] | | [read headers] | | [read payload from buf_id] | | [process request] | | [write reply to header] | | [write reply payload to | | buf] | | >io_uring_submit() | | COMMIT_AND_FETCH | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | [copy reply from ring] | | >fuse_uring_recycle_buffer() | | >fuse_uring_get_next_fuse_req() |
--- fs/fuse/dev_uring.c | 356 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 35 +++- include/uapi/linux/fuse.h | 29 +++- 3 files changed, 353 insertions(+), 67 deletions(-)
fs/fuse/dev_uring.c +294 -62
@@ -41,6 +41,11 @@ enum fuse_uring_header_type { FUSE_URING_HEADER_RING_ENT, }; +static inline bool bufpool_enabled (struct fuse_ring_queue *queue) +{ +return queue->bufpool !=NULL ; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -222,6 +227,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) }kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); ring->queues[qid] =NULL ; }
@@ -277,20 +283,113 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +/* +* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the +* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload +*/ +static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, +struct iovec iov[FUSE_URING_IOV_SEGS ]) +{ +struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); +struct iov_iter iter; +ssize_t ret; + +if (sqe->len !=FUSE_URING_IOV_SEGS ) +return -EINVAL ; + +/* +* Direction for buffer access will actually be READ and WRITE, +* using write for the import should include READ access as well. +*/ + ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , +FUSE_URING_IOV_SEGS , &iov, &iter); +if (ret <0 ) +return ret; + +return 0 ; +} + +static int fuse_uring_bufpool_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue) +{ +const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); +unsigned int buf_size =READ_ONCE (cmd_req->init.buf_size); +unsigned int nr_bufs =READ_ONCE (cmd_req->init.nr_bufs); +struct iovec iov[FUSE_URING_IOV_SEGS ]; +uintptr_t payload_addr; +size_t payload_size, ring_size; +struct fuse_bufpool *pool; +unsigned int i; +int err; + +if (!buf_size || !nr_bufs) +return -EINVAL ; + + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov); +if (err) +return err; + + payload_addr = (uintptr_t )iov[FUSE_URING_IOV_PAYLOAD ].iov_base; + payload_size = iov[FUSE_URING_IOV_PAYLOAD ].iov_len; + +if (payload_size < (uint64_t )buf_size * nr_bufs) +return -EINVAL ; + +/* create the pool, which is managed as a ring buffer */ + ring_size =struct_size (pool, bufs, nr_bufs); + pool =kzalloc (ring_size,GFP_KERNEL_ACCOUNT ); +if (!pool) +return -ENOMEM ; + +/* populate the ring buffer */ +for (i =0 ; i < nr_bufs; i++, payload_addr += buf_size) { +struct fuse_bufpool_buf *buf = &pool->bufs[i]; + + buf->addr = payload_addr; + buf->len = buf_size; + buf->id = i; + } + + pool->nbufs = nr_bufs; + pool->tail = nr_bufs; + + queue->bufpool = pool; + +return 0 ; +} + +/* +* If the queue is already registered, check that the queue was initialized with +* the same init flags set for this FUSE_IO_URING_CMD_REGISTER cmd. all +* FUSE_IO_URING_CMD_REGISTER cmds should have the same init fields set on a +* per-queue basis. +*/ +static bool queue_init_flags_consistent (struct fuse_ring_queue *queue, +u64 init_flags) +{ +bool bufpool = init_flags &FUSE_URING_BUFPOOL ; + +return bufpool_enabled (queue) == bufpool; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,u64 init_flags) { +bool use_bufpool = init_flags &FUSE_URING_BUFPOOL ;struct fuse_chan *fch = ring->chan;struct fuse_ring_queue *queue;struct list_head *pq; +int err; queue =kzalloc_obj (*queue,GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =fuse_pqueue_alloc ();if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;
@@ -308,12 +407,27 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,fuse_pqueue_init (&queue->fpq); queue->fpq.processing = pq; +if (use_bufpool) { + err =fuse_uring_bufpool_setup (cmd, queue); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fch->lock); +/* check if the queue creation raced with another thread */ if (ring->queues[qid]) {spin_unlock (&fch->lock);kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); -return ring->queues[qid]; + + queue = ring->queues[qid]; +if (!queue_init_flags_consistent (queue, init_flags)) +return ERR_PTR (-EINVAL ); +return queue; }/*
@@ -640,12 +754,26 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, struct fuse_ring_ent *ent, int dir, struct iov_iter *iter) { + void __user *payload; +size_t len;int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +memset (iter,0 ,sizeof (*iter)); + +if (bufpool_enabled (ent->queue)) { + payload = (void __user *)ent->payload_buf.addr; + len = ent->payload_buf.len; + }else { + payload = ent->payload; + len = ring->max_payload_sz; + } + +if (payload) { + err =import_ubuf (dir, payload, len, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter);
@@ -697,6 +825,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; +if (bufpool_enabled (ent->queue)) + ent_in_out.buf_id = ent->payload_buf.id; + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter);if (err)return err;
@@ -761,6 +892,92 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool *pool = queue->bufpool; +struct fuse_bufpool_buf *buf; +unsigned int tail = pool->tail, head = pool->head; + +lockdep_assert_held (&queue->lock); + +if (tail == head) +return -ENOBUFS ; + + buf = &pool->bufs[head % pool->nbufs]; + pool->head++; + + ent->payload_buf = *buf; + +return 0 ; +} + +static void fuse_uring_recycle_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_bufpool_buf *ent_payload = &ent->payload_buf; +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool_buf *buf; +struct fuse_bufpool *pool; + +lockdep_assert_held (&queue->lock); + +if (!bufpool_enabled (queue) || !ent_payload->addr) +return ; + + pool = queue->bufpool; + +/* pool should never be full */ +WARN_ON_ONCE (pool->tail - pool->head >= pool->nbufs); + + buf = &pool->bufs[(pool->tail) % pool->nbufs]; + *buf = *ent_payload; + + pool->tail++; + +memset (ent_payload,0 ,sizeof (*ent_payload)); +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +bool buffer_selected; +bool has_payload; + +if (!bufpool_enabled (ent->queue)) +return 0 ; + + buffer_selected = !!ent->payload_buf.addr; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent); + +if (!has_payload && buffer_selected) +fuse_uring_recycle_buffer (ent); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +if (!bufpool_enabled (ent->queue)) +return 0 ; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {
@@ -829,15 +1046,26 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { +/* +* Buffer selection may fail if all the buffers are currently saturated. +* The request will be serviced when a buffer is freed up. +*/ + err =fuse_uring_next_req_update_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + } + } -return req; +fuse_uring_recycle_buffer (ent); +return NULL ; }/*
@@ -996,6 +1224,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* fuse requests would otherwise not get processed - committing * and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). +* +* If the server is using bufpools and has populated the pool with less +* payload buffers than ents, it is possible that there may not be an +* available buffer for the next request. If so, then the fetch is a +* no-op and the next request will be serviced when a buffer becomes +* available. */ if (fuse_uring_get_next_fuse_req (ent, queue))fuse_uring_send (ent, cmd,0 , issue_flags);
@@ -1057,32 +1291,6 @@ static void fuse_uring_do_register(struct fuse_ring_ent *ent, } } -/* -* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the -* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload -*/ -static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, -struct iovec iov[FUSE_URING_IOV_SEGS ]) -{ -struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); -struct iov_iter iter; -ssize_t ret; - -if (sqe->len !=FUSE_URING_IOV_SEGS ) -return -EINVAL ; - -/* -* Direction for buffer access will actually be READ and WRITE, -* using write for the import should include READ access as well. -*/ - ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , -FUSE_URING_IOV_SEGS , &iov, &iter); -if (ret <0 ) -return ret; - -return 0 ; -} -static struct fuse_ring_ent *fuse_uring_create_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_queue *queue)
@@ -1107,13 +1315,6 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,return ERR_PTR (err); } - payload = &iov[FUSE_URING_IOV_PAYLOAD ]; -if (payload->iov_len < ring->max_payload_sz) { -pr_info_ratelimited ("Invalid req payload len %zu\n" , - payload->iov_len); -return ERR_PTR (err); - } - err = -ENOMEM ; ent =kzalloc_obj (*ent,GFP_KERNEL_ACCOUNT );if (!ent)
@@ -1123,12 +1324,30 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + + err = -EINVAL ; +if (!bufpool_enabled (queue)) { + payload = &iov[FUSE_URING_IOV_PAYLOAD ]; +if (payload->iov_len < ring->max_payload_sz) { +pr_info_ratelimited ("Invalid req payload len %zu\n" , + payload->iov_len); +kfree (ent); +return ERR_PTR (err); + } + ent->payload = payload->iov_base; + }atomic_inc (&ring->queue_refs);return ent; } +static bool init_flags_valid (u64 init_flags) +{ +u64 valid_flags =FUSE_URING_BUFPOOL ; + +return !(init_flags & ~valid_flags); +} +/* * Register header and payload buffer with the kernel and puts the * entry as "ready to get fuse requests" on the queue
@@ -1138,6 +1357,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, {const struct fuse_uring_cmd_req *cmd_req =io_uring_sqe128_cmd (cmd->sqe, struct fuse_uring_cmd_req); +u64 init_flags =READ_ONCE (cmd_req->flags);struct fuse_ring *ring =smp_load_acquire (&fch->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;
@@ -1156,11 +1376,16 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,return -EINVAL ; } +if (!init_flags_valid (init_flags)) +return -EINVAL ; + queue = ring->queues[qid];if (!queue) { - queue =fuse_uring_create_queue (ring, qid); -if (!queue) -return err; + queue =fuse_uring_create_queue (cmd, ring, qid, init_flags); +if (IS_ERR (queue)) +return PTR_ERR (queue); + }else if (!queue_init_flags_consistent (queue, init_flags)) { +return -EINVAL ; }/*
@@ -1325,14 +1550,18 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent =list_first_entry_or_null (&queue->ent_avail_queue, struct fuse_ring_ent, list); -if (ent) -fuse_uring_add_req_to_ring_ent (ent, req); -else -list_add_tail (&req->list, &queue->fuse_req_queue); -spin_unlock (&queue->lock); +if (ent) { + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return ; + } + } -if (ent) -fuse_uring_dispatch_ent (ent); +list_add_tail (&req->list, &queue->fuse_req_queue); +spin_unlock (&queue->lock);return ;
@@ -1350,6 +1579,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)struct fuse_ring *ring = fch->ring;struct fuse_ring_queue *queue;struct fuse_ring_ent *ent =NULL ; +int err; queue =fuse_uring_task_to_queue (ring);if (!queue)
@@ -1382,14 +1612,16 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list);if (ent && req) { -fuse_uring_add_req_to_ring_ent (ent, req); -spin_unlock (&queue->lock); - -fuse_uring_dispatch_ent (ent); - }else { -spin_unlock (&queue->lock); + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return true; + } } +spin_unlock (&queue->lock);return true; }
fs/fuse/dev_uring_i.h +33 -2
@@ -36,11 +36,36 @@ enum fuse_ring_req_state { FRRS_RELEASED, }; +struct fuse_bufpool_buf { +uintptr_t addr; +unsigned int len; +unsigned int id; +}; + +/* implemented as a buffer ring */ +struct fuse_bufpool { +/* metadata tracking state of the bufpool */ +unsigned int nbufs; +unsigned int head; +unsigned int tail; + +/* the buffers backing the pool */ + __DECLARE_FLEX_ARRAY(struct fuse_bufpool_buf, bufs); +}; +/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ struct fuse_uring_req_header __user *headers; -void __user *payload; +union { +/* if bufpools are not used */ +struct { +void __user *payload; + }; +/* if bufpools are used */ +struct { +struct fuse_bufpool_buf payload_buf; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
@@ -99,6 +124,12 @@ struct fuse_ring_queue {unsigned int active_background;bool stopped; + +/* +* only allocated if the server registers the ring with +* FUSE_URING_BUFPOOL. +*/ +struct fuse_bufpool *bufpool; };/*
include/uapi/linux/fuse.h +26 -3
@@ -240,6 +240,11 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add FUSE_URING_BUFPOOL flag +* - add fuse_uring_cmd_req init struct +* - add buf_id field to fuse_uring_ent_in_out struct */ #ifndef _LINUX_FUSE_H
@@ -275,7 +280,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46/** The node ID of the root inode */ #define FUSE_ROOT_ID 1
@@ -1263,7 +1268,13 @@ struct fuse_uring_ent_in_out {/* size of user payload buffer */ uint32_t payload_sz; -uint32_t padding; + +/* +* if using bufpools, this is the id of the selected buffer. +* the selected buffer holds the request payload +*/ +uint16_t buf_id; +uint16_t padding;uint64_t reserved; };
@@ -1294,6 +1305,9 @@ enum fuse_uring_cmd { FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, }; +/* fuse_uring_cmd_req flags */ +#define FUSE_URING_BUFPOOL (1 << 0) +/** * In the 80B command area of the SQE. */
@@ -1305,7 +1319,16 @@ struct fuse_uring_cmd_req {/* queue the command is for (queue index) */ uint16_t qid; - uint8_t padding[6]; +uint16_t padding; +uint32_t reserved; + +union { +struct { +/* FUSE_URING_BUFPOOL: buffer pool configuration */ +uint32_t buf_size; +uint16_t nr_bufs; + } init; + }; };#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.52.0
Re: [PATCH v3 07/10] fuse: add io-uring buffer pools (implemented as buffer ring)
Quoted 79 lines · Joanne Koong
Right now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
To use this, the server must set the
FUSE_URING_BUFPOOLflag during registration and provide the payload buffer pool through the iovec payload in the first sqe. The server must set init.buf_size and init.nr_bufs in the structfuse_uring_cmd_reqit passes to the kernel to describe the buffer pool configuration. Buffer pool usage is set on a per-ring basis. All subsequent registration SQEs for the same ring must pass in consistent flags.Internally, a buffer ring data structure is used for managing the buffer pool. The kernel manages buffer selection and recycling through this ring structure. A payload buffer is selected from the ring on demand if the request requires a payload buffer and recycled after the request completes if used.
The contiguous buffer pool passed in is partitioned into individual buffers according to the
buf_sizepassed in at registration time.PAYLOAD BUFFER POOL (contiguous, provided via iov[1] in first sqe):
|<-------------- payload_size ------------>| +--------- --+-----------+-----------+-----+ | buf [0] | buf [1] | buf [2] | ... | | buf_size | buf_size | buf_size | ... | +--------- --+-----------+-----------+-----+The buffer ring logic is as follows:
select: buf = bufs[head % nbufs]; head++ recycle: bufs[tail % nbufs] = buf; tail++ empty: tail == head (no buffers available) full: tail - head >= nbufsBuffer ring request flow
------------------------| Kernel | FUSE daemon | | | [client request arrives] | | >fuse_uring_send() | | [select payload buf from pool] | | >fuse_uring_select_buffer() | | [copy headers to ent] | | >copy_header_to_ring() | | [copy payload to selected buf] | | >fuse_uring_copy_to_ring() | | [set buf_id in ent_in_out header] | | >io_uring_cmd_done() | | | [CQE received] | | [read headers] | | [read payload from buf_id] | | [process request] | | [write reply to header] | | [write reply payload to | | buf] | | >io_uring_submit() | | COMMIT_AND_FETCH | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | [copy reply from ring] | | >fuse_uring_recycle_buffer() | | >fuse_uring_get_next_fuse_req() |
Quoted 2 lines
Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>
Looks good!
Quoted 642 lines
--- fs/fuse/dev_uring.c | 356 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 35 +++- include/uapi/linux/fuse.h | 29 +++- 3 files changed, 353 insertions(+), 67 deletions(-)fs/fuse/dev_uring.c +294
-62@@ -41,6 +41,11 @@ enum fuse_uring_header_type { FUSE_URING_HEADER_RING_ENT, }; +static inline bool bufpool_enabled (struct fuse_ring_queue *queue) +{ +return queue->bufpool !=NULL ; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {@@ -222,6 +227,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) }kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); ring->queues[qid] =NULL ; }@@ -277,20 +283,113 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +/* +* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the +* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload +*/ +static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, +struct iovec iov[FUSE_URING_IOV_SEGS ]) +{ +struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); +struct iov_iter iter; +ssize_t ret; + +if (sqe->len !=FUSE_URING_IOV_SEGS ) +return -EINVAL ; + +/* +* Direction for buffer access will actually be READ and WRITE, +* using write for the import should include READ access as well. +*/ + ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , +FUSE_URING_IOV_SEGS , &iov, &iter); +if (ret <0 ) +return ret; + +return 0 ; +} + +static int fuse_uring_bufpool_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue) +{ +const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); +unsigned int buf_size =READ_ONCE (cmd_req->init.buf_size); +unsigned int nr_bufs =READ_ONCE (cmd_req->init.nr_bufs); +struct iovec iov[FUSE_URING_IOV_SEGS ]; +uintptr_t payload_addr; +size_t payload_size, ring_size; +struct fuse_bufpool *pool; +unsigned int i; +int err; + +if (!buf_size || !nr_bufs) +return -EINVAL ; + + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov); +if (err) +return err; + + payload_addr = (uintptr_t )iov[FUSE_URING_IOV_PAYLOAD ].iov_base; + payload_size = iov[FUSE_URING_IOV_PAYLOAD ].iov_len; + +if (payload_size < (uint64_t )buf_size * nr_bufs) +return -EINVAL ; + +/* create the pool, which is managed as a ring buffer */ + ring_size =struct_size (pool, bufs, nr_bufs); + pool =kzalloc (ring_size,GFP_KERNEL_ACCOUNT ); +if (!pool) +return -ENOMEM ; + +/* populate the ring buffer */ +for (i =0 ; i < nr_bufs; i++, payload_addr += buf_size) { +struct fuse_bufpool_buf *buf = &pool->bufs[i]; + + buf->addr = payload_addr; + buf->len = buf_size; + buf->id = i; + } + + pool->nbufs = nr_bufs; + pool->tail = nr_bufs; + + queue->bufpool = pool; + +return 0 ; +} + +/* +* If the queue is already registered, check that the queue was initialized with +* the same init flags set for this FUSE_IO_URING_CMD_REGISTER cmd. all +* FUSE_IO_URING_CMD_REGISTER cmds should have the same init fields set on a +* per-queue basis. +*/ +static bool queue_init_flags_consistent (struct fuse_ring_queue *queue, +u64 init_flags) +{ +bool bufpool = init_flags &FUSE_URING_BUFPOOL ; + +return bufpool_enabled (queue) == bufpool; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,u64 init_flags) { +bool use_bufpool = init_flags &FUSE_URING_BUFPOOL ;struct fuse_chan *fch = ring->chan;struct fuse_ring_queue *queue;struct list_head *pq; +int err; queue =kzalloc_obj (*queue,GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =fuse_pqueue_alloc ();if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;@@ -308,12 +407,27 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,fuse_pqueue_init (&queue->fpq); queue->fpq.processing = pq; +if (use_bufpool) { + err =fuse_uring_bufpool_setup (cmd, queue); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fch->lock); +/* check if the queue creation raced with another thread */ if (ring->queues[qid]) {spin_unlock (&fch->lock);kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); -return ring->queues[qid]; + + queue = ring->queues[qid]; +if (!queue_init_flags_consistent (queue, init_flags)) +return ERR_PTR (-EINVAL ); +return queue; }/* @@ -640,12 +754,26 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, struct fuse_ring_ent *ent, int dir, struct iov_iter *iter) { + void __user *payload; +size_t len;int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +memset (iter,0 ,sizeof (*iter)); + +if (bufpool_enabled (ent->queue)) { + payload = (void __user *)ent->payload_buf.addr; + len = ent->payload_buf.len; + }else { + payload = ent->payload; + len = ring->max_payload_sz; + } + +if (payload) { + err =import_ubuf (dir, payload, len, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter);@@ -697,6 +825,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; +if (bufpool_enabled (ent->queue)) + ent_in_out.buf_id = ent->payload_buf.id; + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter);if (err)return err;@@ -761,6 +892,92 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool *pool = queue->bufpool; +struct fuse_bufpool_buf *buf; +unsigned int tail = pool->tail, head = pool->head; + +lockdep_assert_held (&queue->lock); + +if (tail == head) +return -ENOBUFS ; + + buf = &pool->bufs[head % pool->nbufs]; + pool->head++; + + ent->payload_buf = *buf; + +return 0 ; +} + +static void fuse_uring_recycle_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_bufpool_buf *ent_payload = &ent->payload_buf; +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool_buf *buf; +struct fuse_bufpool *pool; + +lockdep_assert_held (&queue->lock); + +if (!bufpool_enabled (queue) || !ent_payload->addr) +return ; + + pool = queue->bufpool; + +/* pool should never be full */ +WARN_ON_ONCE (pool->tail - pool->head >= pool->nbufs); + + buf = &pool->bufs[(pool->tail) % pool->nbufs]; + *buf = *ent_payload; + + pool->tail++; + +memset (ent_payload,0 ,sizeof (*ent_payload)); +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +bool buffer_selected; +bool has_payload; + +if (!bufpool_enabled (ent->queue)) +return 0 ; + + buffer_selected = !!ent->payload_buf.addr; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent); + +if (!has_payload && buffer_selected) +fuse_uring_recycle_buffer (ent); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +if (!bufpool_enabled (ent->queue)) +return 0 ; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -829,15 +1046,26 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { +/* +* Buffer selection may fail if all the buffers are currently saturated. +* The request will be serviced when a buffer is freed up. +*/ + err =fuse_uring_next_req_update_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + } + } -return req; +fuse_uring_recycle_buffer (ent); +return NULL ; }/* @@ -996,6 +1224,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* fuse requests would otherwise not get processed - committing * and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). +* +* If the server is using bufpools and has populated the pool with less +* payload buffers than ents, it is possible that there may not be an +* available buffer for the next request. If so, then the fetch is a +* no-op and the next request will be serviced when a buffer becomes +* available. */ if (fuse_uring_get_next_fuse_req (ent, queue))fuse_uring_send (ent, cmd,0 , issue_flags);@@ -1057,32 +1291,6 @@ static void fuse_uring_do_register(struct fuse_ring_ent *ent, } } -/* -* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the -* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload -*/ -static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, -struct iovec iov[FUSE_URING_IOV_SEGS ]) -{ -struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); -struct iov_iter iter; -ssize_t ret; - -if (sqe->len !=FUSE_URING_IOV_SEGS ) -return -EINVAL ; - -/* -* Direction for buffer access will actually be READ and WRITE, -* using write for the import should include READ access as well. -*/ - ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , -FUSE_URING_IOV_SEGS , &iov, &iter); -if (ret <0 ) -return ret; - -return 0 ; -} -static struct fuse_ring_ent *fuse_uring_create_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_queue *queue)@@ -1107,13 +1315,6 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,return ERR_PTR (err); } - payload = &iov[FUSE_URING_IOV_PAYLOAD ]; -if (payload->iov_len < ring->max_payload_sz) { -pr_info_ratelimited ("Invalid req payload len %zu\n" , - payload->iov_len); -return ERR_PTR (err); - } - err = -ENOMEM ; ent =kzalloc_obj (*ent,GFP_KERNEL_ACCOUNT );if (!ent)@@ -1123,12 +1324,30 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + + err = -EINVAL ; +if (!bufpool_enabled (queue)) { + payload = &iov[FUSE_URING_IOV_PAYLOAD ]; +if (payload->iov_len < ring->max_payload_sz) { +pr_info_ratelimited ("Invalid req payload len %zu\n" , + payload->iov_len); +kfree (ent); +return ERR_PTR (err); + } + ent->payload = payload->iov_base; + }atomic_inc (&ring->queue_refs);return ent; } +static bool init_flags_valid (u64 init_flags) +{ +u64 valid_flags =FUSE_URING_BUFPOOL ; + +return !(init_flags & ~valid_flags); +} +/* * Register header and payload buffer with the kernel and puts the * entry as "ready to get fuse requests" on the queue @@ -1138,6 +1357,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, {const struct fuse_uring_cmd_req *cmd_req =io_uring_sqe128_cmd (cmd->sqe, struct fuse_uring_cmd_req); +u64 init_flags =READ_ONCE (cmd_req->flags);struct fuse_ring *ring =smp_load_acquire (&fch->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;@@ -1156,11 +1376,16 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,return -EINVAL ; } +if (!init_flags_valid (init_flags)) +return -EINVAL ; + queue = ring->queues[qid];if (!queue) { - queue =fuse_uring_create_queue (ring, qid); -if (!queue) -return err; + queue =fuse_uring_create_queue (cmd, ring, qid, init_flags); +if (IS_ERR (queue)) +return PTR_ERR (queue); + }else if (!queue_init_flags_consistent (queue, init_flags)) { +return -EINVAL ; }/* @@ -1325,14 +1550,18 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent =list_first_entry_or_null (&queue->ent_avail_queue, struct fuse_ring_ent, list); -if (ent) -fuse_uring_add_req_to_ring_ent (ent, req); -else -list_add_tail (&req->list, &queue->fuse_req_queue); -spin_unlock (&queue->lock); +if (ent) { + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return ; + } + } -if (ent) -fuse_uring_dispatch_ent (ent); +list_add_tail (&req->list, &queue->fuse_req_queue); +spin_unlock (&queue->lock);return ;@@ -1350,6 +1579,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)struct fuse_ring *ring = fch->ring;struct fuse_ring_queue *queue;struct fuse_ring_ent *ent =NULL ; +int err; queue =fuse_uring_task_to_queue (ring);if (!queue)@@ -1382,14 +1612,16 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list);if (ent && req) { -fuse_uring_add_req_to_ring_ent (ent, req); -spin_unlock (&queue->lock); - -fuse_uring_dispatch_ent (ent); - }else { -spin_unlock (&queue->lock); + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return true; + } } +spin_unlock (&queue->lock);return true; }fs/fuse/dev_uring_i.h +33
-2@@ -36,11 +36,36 @@ enum fuse_ring_req_state { FRRS_RELEASED, }; +struct fuse_bufpool_buf { +uintptr_t addr; +unsigned int len; +unsigned int id; +}; + +/* implemented as a buffer ring */ +struct fuse_bufpool { +/* metadata tracking state of the bufpool */ +unsigned int nbufs; +unsigned int head; +unsigned int tail; + +/* the buffers backing the pool */ + __DECLARE_FLEX_ARRAY(struct fuse_bufpool_buf, bufs); +}; +/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ struct fuse_uring_req_header __user *headers; -void __user *payload; +union { +/* if bufpools are not used */ +struct { +void __user *payload; + }; +/* if bufpools are used */ +struct { +struct fuse_bufpool_buf payload_buf; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;@@ -99,6 +124,12 @@ struct fuse_ring_queue {unsigned int active_background;bool stopped; + +/* +* only allocated if the server registers the ring with +* FUSE_URING_BUFPOOL. +*/ +struct fuse_bufpool *bufpool; };/* include/uapi/linux/fuse.h +26
-3@@ -240,6 +240,11 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add FUSE_URING_BUFPOOL flag +* - add fuse_uring_cmd_req init struct +* - add buf_id field to fuse_uring_ent_in_out struct */ #ifndef _LINUX_FUSE_H@@ -275,7 +280,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -1263,7 +1268,13 @@ struct fuse_uring_ent_in_out {/* size of user payload buffer */ uint32_t payload_sz; -uint32_t padding; + +/* +* if using bufpools, this is the id of the selected buffer. +* the selected buffer holds the request payload +*/ +uint16_t buf_id; +uint16_t padding;uint64_t reserved; };@@ -1294,6 +1305,9 @@ enum fuse_uring_cmd { FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, }; +/* fuse_uring_cmd_req flags */ +#define FUSE_URING_BUFPOOL (1 << 0) +/** * In the 80B command area of the SQE. */ @@ -1305,7 +1319,16 @@ struct fuse_uring_cmd_req {/* queue the command is for (queue index) */ uint16_t qid; - uint8_t padding[6]; +uint16_t padding; +uint32_t reserved; + +union { +struct { +/* FUSE_URING_BUFPOOL: buffer pool configuration */ +uint32_t buf_size; +uint16_t nr_bufs; + } init; + }; };#endif /* _LINUX_FUSE_H */
Bernd Schubert
maintainerRe: [PATCH v3 07/10] fuse: add io-uring buffer pools (implemented as buffer ring)
In reply to Joanne KoongHi Joanne,
I one generic question, could you also add in for which kernel branch the series is? I think I need to do the same for larger patches series as well (recent reduced-queue was based on 'miklos/for-next'). Basically 'b4 shazam' fails to apply on linux master and also fails to on 'miklos/for-next' and I don't even know what the base for the series here is
Quoted 489 lines · Joanne Koong
Right now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
To use this, the server must set the
FUSE_URING_BUFPOOLflag during registration and provide the payload buffer pool through the iovec payload in the first sqe. The server must set init.buf_size and init.nr_bufs in the structfuse_uring_cmd_reqit passes to the kernel to describe the buffer pool configuration. Buffer pool usage is set on a per-ring basis. All subsequent registration SQEs for the same ring must pass in consistent flags.Internally, a buffer ring data structure is used for managing the buffer pool. The kernel manages buffer selection and recycling through this ring structure. A payload buffer is selected from the ring on demand if the request requires a payload buffer and recycled after the request completes if used.
The contiguous buffer pool passed in is partitioned into individual buffers according to the
buf_sizepassed in at registration time.PAYLOAD BUFFER POOL (contiguous, provided via iov[1] in first sqe):
|<-------------- payload_size ------------>| +--------- --+-----------+-----------+-----+ | buf [0] | buf [1] | buf [2] | ... | | buf_size | buf_size | buf_size | ... | +--------- --+-----------+-----------+-----+The buffer ring logic is as follows:
select: buf = bufs[head % nbufs]; head++ recycle: bufs[tail % nbufs] = buf; tail++ empty: tail == head (no buffers available) full: tail - head >= nbufsBuffer ring request flow
------------------------| Kernel | FUSE daemon | | | [client request arrives] | | >fuse_uring_send() | | [select payload buf from pool] | | >fuse_uring_select_buffer() | | [copy headers to ent] | | >copy_header_to_ring() | | [copy payload to selected buf] | | >fuse_uring_copy_to_ring() | | [set buf_id in ent_in_out header] | | >io_uring_cmd_done() | | | [CQE received] | | [read headers] | | [read payload from buf_id] | | [process request] | | [write reply to header] | | [write reply payload to | | buf] | | >io_uring_submit() | | COMMIT_AND_FETCH | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | [copy reply from ring] | | >fuse_uring_recycle_buffer() | | >fuse_uring_get_next_fuse_req() |Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 356 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 35 +++- include/uapi/linux/fuse.h | 29 +++- 3 files changed, 353 insertions(+), 67 deletions(-)fs/fuse/dev_uring.c +248
-46@@ -41,6 +41,11 @@ enum fuse_uring_header_type { FUSE_URING_HEADER_RING_ENT, }; +static inline bool bufpool_enabled (struct fuse_ring_queue *queue) +{ +return queue->bufpool !=NULL ; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {@@ -222,6 +227,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) }kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); ring->queues[qid] =NULL ; }@@ -277,20 +283,113 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)return res; } -static struct fuse_ring_queue *fuse_uring_create_queue (struct fuse_ring *ring, -int qid) +/* +* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the +* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload +*/ +static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, +struct iovec iov[FUSE_URING_IOV_SEGS ]) +{ +struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); +struct iov_iter iter; +ssize_t ret; + +if (sqe->len !=FUSE_URING_IOV_SEGS ) +return -EINVAL ; + +/* +* Direction for buffer access will actually be READ and WRITE, +* using write for the import should include READ access as well. +*/ + ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , +FUSE_URING_IOV_SEGS , &iov, &iter); +if (ret <0 ) +return ret; + +return 0 ; +} + +static int fuse_uring_bufpool_setup (struct io_uring_cmd *cmd, +struct fuse_ring_queue *queue) +{ +const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); +unsigned int buf_size =READ_ONCE (cmd_req->init.buf_size); +unsigned int nr_bufs =READ_ONCE (cmd_req->init.nr_bufs); +struct iovec iov[FUSE_URING_IOV_SEGS ]; +uintptr_t payload_addr; +size_t payload_size, ring_size; +struct fuse_bufpool *pool; +unsigned int i; +int err; + +if (!buf_size || !nr_bufs) +return -EINVAL ; + + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov); +if (err) +return err; + + payload_addr = (uintptr_t )iov[FUSE_URING_IOV_PAYLOAD ].iov_base; + payload_size = iov[FUSE_URING_IOV_PAYLOAD ].iov_len; + +if (payload_size < (uint64_t )buf_size * nr_bufs) +return -EINVAL ; + +/* create the pool, which is managed as a ring buffer */ + ring_size =struct_size (pool, bufs, nr_bufs); + pool =kzalloc (ring_size,GFP_KERNEL_ACCOUNT ); +if (!pool) +return -ENOMEM ; + +/* populate the ring buffer */ +for (i =0 ; i < nr_bufs; i++, payload_addr += buf_size) { +struct fuse_bufpool_buf *buf = &pool->bufs[i]; + + buf->addr = payload_addr; + buf->len = buf_size; + buf->id = i; + } + + pool->nbufs = nr_bufs; + pool->tail = nr_bufs; + + queue->bufpool = pool; + +return 0 ; +} + +/* +* If the queue is already registered, check that the queue was initialized with +* the same init flags set for this FUSE_IO_URING_CMD_REGISTER cmd. all +* FUSE_IO_URING_CMD_REGISTER cmds should have the same init fields set on a +* per-queue basis. +*/ +static bool queue_init_flags_consistent (struct fuse_ring_queue *queue, +u64 init_flags) +{ +bool bufpool = init_flags &FUSE_URING_BUFPOOL ; + +return bufpool_enabled (queue) == bufpool; +} + +static struct fuse_ring_queue * +fuse_uring_create_queue (struct io_uring_cmd *cmd,struct fuse_ring *ring, +int qid,u64 init_flags) { +bool use_bufpool = init_flags &FUSE_URING_BUFPOOL ;struct fuse_chan *fch = ring->chan;struct fuse_ring_queue *queue;struct list_head *pq; +int err; queue =kzalloc_obj (*queue,GFP_KERNEL_ACCOUNT );if (!queue) -return NULL ; +return ERR_PTR (-ENOMEM ); pq =fuse_pqueue_alloc ();if (!pq) {kfree (queue); -return NULL ; +return ERR_PTR (-ENOMEM ); } queue->qid = qid;@@ -308,12 +407,27 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,fuse_pqueue_init (&queue->fpq); queue->fpq.processing = pq; +if (use_bufpool) { + err =fuse_uring_bufpool_setup (cmd, queue); +if (err) { +kfree (pq); +kfree (queue); +return ERR_PTR (err); + } + } +spin_lock (&fch->lock); +/* check if the queue creation raced with another thread */ if (ring->queues[qid]) {spin_unlock (&fch->lock);kfree (queue->fpq.processing); +kfree (queue->bufpool);kfree (queue); -return ring->queues[qid]; + + queue = ring->queues[qid]; +if (!queue_init_flags_consistent (queue, init_flags)) +return ERR_PTR (-EINVAL ); +return queue; }/* @@ -640,12 +754,26 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs, struct fuse_ring_ent *ent, int dir, struct iov_iter *iter) { + void __user *payload; +size_t len;int err; - err =import_ubuf (dir, ent->payload, ring->max_payload_sz, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +memset (iter,0 ,sizeof (*iter)); + +if (bufpool_enabled (ent->queue)) { + payload = (void __user *)ent->payload_buf.addr; + len = ent->payload_buf.len; + }else { + payload = ent->payload; + len = ring->max_payload_sz; + } + +if (payload) { + err =import_ubuf (dir, payload, len, iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, dir ==ITER_DEST , iter);@@ -697,6 +825,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; +if (bufpool_enabled (ent->queue)) + ent_in_out.buf_id = ent->payload_buf.id; + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter);if (err)return err;@@ -761,6 +892,92 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +static bool fuse_uring_req_has_payload (struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +return args->in_numargs >1 || args->out_numargs; +} + +static int fuse_uring_select_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool *pool = queue->bufpool; +struct fuse_bufpool_buf *buf; +unsigned int tail = pool->tail, head = pool->head; + +lockdep_assert_held (&queue->lock); + +if (tail == head) +return -ENOBUFS ; + + buf = &pool->bufs[head % pool->nbufs]; + pool->head++; + + ent->payload_buf = *buf; + +return 0 ; +} + +static void fuse_uring_recycle_buffer (struct fuse_ring_ent *ent) +{ +struct fuse_bufpool_buf *ent_payload = &ent->payload_buf; +struct fuse_ring_queue *queue = ent->queue; +struct fuse_bufpool_buf *buf; +struct fuse_bufpool *pool; + +lockdep_assert_held (&queue->lock); + +if (!bufpool_enabled (queue) || !ent_payload->addr) +return ; + + pool = queue->bufpool; + +/* pool should never be full */ +WARN_ON_ONCE (pool->tail - pool->head >= pool->nbufs); + + buf = &pool->bufs[(pool->tail) % pool->nbufs]; + *buf = *ent_payload; + + pool->tail++; + +memset (ent_payload,0 ,sizeof (*ent_payload)); +} + +static int fuse_uring_next_req_update_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +bool buffer_selected; +bool has_payload; + +if (!bufpool_enabled (ent->queue)) +return 0 ; + + buffer_selected = !!ent->payload_buf.addr; + has_payload =fuse_uring_req_has_payload (req); + +if (has_payload && !buffer_selected) +return fuse_uring_select_buffer (ent); + +if (!has_payload && buffer_selected) +fuse_uring_recycle_buffer (ent); + +return 0 ; +} + +static int fuse_uring_prep_buffer (struct fuse_ring_ent *ent, +struct fuse_req *req) +{ +if (!bufpool_enabled (ent->queue)) +return 0 ; + +/* no payload to copy, can skip selecting a buffer */ +if (!fuse_uring_req_has_payload (req)) +return 0 ; + +return fuse_uring_select_buffer (ent); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -829,15 +1046,26 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)struct fuse_req *req;struct fuse_ring_queue *queue = ent->queue;struct list_head *req_queue = &queue->fuse_req_queue; +int err;lockdep_assert_held (&queue->lock);/* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); -if (req) -fuse_uring_add_req_to_ring_ent (ent, req); +if (req) { +/* +* Buffer selection may fail if all the buffers are currently saturated. +* The request will be serviced when a buffer is freed up. +*/ + err =fuse_uring_next_req_update_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +return req; + } + } -return req; +fuse_uring_recycle_buffer (ent); +return NULL ; }/* @@ -996,6 +1224,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* fuse requests would otherwise not get processed - committing * and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). +* +* If the server is using bufpools and has populated the pool with less +* payload buffers than ents, it is possible that there may not be an +* available buffer for the next request. If so, then the fetch is a +* no-op and the next request will be serviced when a buffer becomes +* available. */ if (fuse_uring_get_next_fuse_req (ent, queue))fuse_uring_send (ent, cmd,0 , issue_flags);@@ -1057,32 +1291,6 @@ static void fuse_uring_do_register(struct fuse_ring_ent *ent, } } -/* -* sqe->addr is a ptr to an iovec array, iov[FUSE_URING_IOV_HEADERS] has the -* headers, iov[FUSE_URING_IOV_PAYLOAD] the payload -*/ -static int fuse_uring_get_iovec_from_sqe (const struct io_uring_sqe *sqe, -struct iovec iov[FUSE_URING_IOV_SEGS ]) -{ -struct iovec __user *uiov =u64_to_user_ptr (READ_ONCE (sqe->addr)); -struct iov_iter iter; -ssize_t ret; - -if (sqe->len !=FUSE_URING_IOV_SEGS ) -return -EINVAL ; - -/* -* Direction for buffer access will actually be READ and WRITE, -* using write for the import should include READ access as well. -*/ - ret =import_iovec (WRITE , uiov,FUSE_URING_IOV_SEGS , -FUSE_URING_IOV_SEGS , &iov, &iter); -if (ret <0 ) -return ret; - -return 0 ; -} -static struct fuse_ring_ent *fuse_uring_create_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_queue *queue)@@ -1107,13 +1315,6 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,return ERR_PTR (err); } - payload = &iov[FUSE_URING_IOV_PAYLOAD ]; -if (payload->iov_len < ring->max_payload_sz) { -pr_info_ratelimited ("Invalid req payload len %zu\n" , - payload->iov_len); -return ERR_PTR (err); - } - err = -ENOMEM ; ent =kzalloc_obj (*ent,GFP_KERNEL_ACCOUNT );if (!ent)@@ -1123,12 +1324,30 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + + err = -EINVAL ;
fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1338
Quoted 10 lines
fs/fuse/dev_uring.c +10
-0+if (!bufpool_enabled (queue)) { + payload = &iov[FUSE_URING_IOV_PAYLOAD ]; +if (payload->iov_len < ring->max_payload_sz) { +pr_info_ratelimited ("Invalid req payload len %zu\n" , + payload->iov_len); +kfree (ent); +return ERR_PTR (err); + } + ent->payload = payload->iov_base; + }
Let's say a confused user library would not know how to set up things and would in one SQE register a buf pool and in another register payload and header - should this fail with an error here? We could add that sanity check later, though.
Quoted 207 lines
fs/fuse/dev_uring.c +36
-16atomic_inc (&ring->queue_refs);return ent; } +static bool init_flags_valid (u64 init_flags) +{ +u64 valid_flags =FUSE_URING_BUFPOOL ; + +return !(init_flags & ~valid_flags); +} +/* * Register header and payload buffer with the kernel and puts the * entry as "ready to get fuse requests" on the queue @@ -1138,6 +1357,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, {const struct fuse_uring_cmd_req *cmd_req =io_uring_sqe128_cmd (cmd->sqe, struct fuse_uring_cmd_req); +u64 init_flags =READ_ONCE (cmd_req->flags);struct fuse_ring *ring =smp_load_acquire (&fch->ring);struct fuse_ring_queue *queue;struct fuse_ring_ent *ent;@@ -1156,11 +1376,16 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,return -EINVAL ; } +if (!init_flags_valid (init_flags)) +return -EINVAL ; + queue = ring->queues[qid];if (!queue) { - queue =fuse_uring_create_queue (ring, qid); -if (!queue) -return err; + queue =fuse_uring_create_queue (cmd, ring, qid, init_flags); +if (IS_ERR (queue)) +return PTR_ERR (queue); + }else if (!queue_init_flags_consistent (queue, init_flags)) { +return -EINVAL ; }/* @@ -1325,14 +1550,18 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent =list_first_entry_or_null (&queue->ent_avail_queue, struct fuse_ring_ent, list); -if (ent) -fuse_uring_add_req_to_ring_ent (ent, req); -else -list_add_tail (&req->list, &queue->fuse_req_queue); -spin_unlock (&queue->lock); +if (ent) { + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return ; + } + } -if (ent) -fuse_uring_dispatch_ent (ent); +list_add_tail (&req->list, &queue->fuse_req_queue); +spin_unlock (&queue->lock);return ;@@ -1350,6 +1579,7 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)struct fuse_ring *ring = fch->ring;struct fuse_ring_queue *queue;struct fuse_ring_ent *ent =NULL ; +int err; queue =fuse_uring_task_to_queue (ring);if (!queue)@@ -1382,14 +1612,16 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list);if (ent && req) { -fuse_uring_add_req_to_ring_ent (ent, req); -spin_unlock (&queue->lock); - -fuse_uring_dispatch_ent (ent); - }else { -spin_unlock (&queue->lock); + err =fuse_uring_prep_buffer (ent, req); +if (!err) { +fuse_uring_add_req_to_ring_ent (ent, req); +spin_unlock (&queue->lock); +fuse_uring_dispatch_ent (ent); +return true; + } } +spin_unlock (&queue->lock);return true; }fs/fuse/dev_uring_i.h +33
-2@@ -36,11 +36,36 @@ enum fuse_ring_req_state { FRRS_RELEASED, }; +struct fuse_bufpool_buf { +uintptr_t addr; +unsigned int len; +unsigned int id; +}; + +/* implemented as a buffer ring */ +struct fuse_bufpool { +/* metadata tracking state of the bufpool */ +unsigned int nbufs; +unsigned int head; +unsigned int tail; + +/* the buffers backing the pool */ + __DECLARE_FLEX_ARRAY(struct fuse_bufpool_buf, bufs); +}; +/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ struct fuse_uring_req_header __user *headers; -void __user *payload; +union { +/* if bufpools are not used */ +struct { +void __user *payload; + }; +/* if bufpools are used */ +struct { +struct fuse_bufpool_buf payload_buf; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;@@ -99,6 +124,12 @@ struct fuse_ring_queue {unsigned int active_background;bool stopped; + +/* +* only allocated if the server registers the ring with +* FUSE_URING_BUFPOOL. +*/ +struct fuse_bufpool *bufpool; };/* include/uapi/linux/fuse.h +16
-2@@ -240,6 +240,11 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add FUSE_URING_BUFPOOL flag +* - add fuse_uring_cmd_req init struct +* - add buf_id field to fuse_uring_ent_in_out struct */ #ifndef _LINUX_FUSE_H@@ -275,7 +280,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -1263,7 +1268,13 @@ struct fuse_uring_ent_in_out {/* size of user payload buffer */ uint32_t payload_sz; -uint32_t padding; + +/* +* if using bufpools, this is the id of the selected buffer. +* the selected buffer holds the request payload +*/ +uint16_t buf_id; +uint16_t padding;uint64_t reserved; };@@ -1294,6 +1305,9 @@ enum fuse_uring_cmd { FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, }; +/* fuse_uring_cmd_req flags */ +#define FUSE_URING_BUFPOOL (1 << 0) +/** * In the 80B command area of the SQE. */
include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1323
Quoted 6 lines
include/uapi/linux/fuse.h +2
-1@@ -1305,7 +1319,16 @@ struct fuse_uring_cmd_req {/* queue the command is for (queue index) */ uint16_t qid; -uint8_t padding[6 ]; +uint16_t padding; +uint32_t reserved;
Dunno, it was a padding up to 64 bit? Non-needed change?
include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1330
Quoted 7 lines
include/uapi/linux/fuse.h +7
-0+ +union { +struct { +/* FUSE_URING_BUFPOOL: buffer pool configuration */ +uint32_t buf_size; +uint16_t nr_bufs; + } init;
If we want to support multiple buffer pools, this struct should add in a buffer index or at a minimum reserve a field for it. Adding in
uint16_t buf_idx; /* currently unused */ would also fill up the hole at the end. libfuse could even already set buf_idx to 0 and then kernel could check if zero and fail something else is set. Would save us another flag later on.
Thanks,
Bernd
Bernd Schubert
maintainerRe: [PATCH v3 07/10] fuse: add io-uring buffer pools (implemented as buffer ring)
Quoted 26 lines · Bernd Schubert
Hi Joanne,
I one generic question, could you also add in for which kernel branch the series is? I think I need to do the same for larger patches series as well (recent reduced-queue was based on 'miklos/for-next'). Basically 'b4 shazam' fails to apply on linux master and also fails to on 'miklos/for-next' and I don't even know what the base for the series here is
Quoted 21 lines · Joanne Koong
Right now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
To use this, the server must set the
FUSE_URING_BUFPOOLflag during
Sorry, had missed that in the earlier mail. How about renaming this to FUSE_URING_BUFPOOL_PAYLOAD (or so), so that it would clearly differentiated from a possible header buf pool in the future?
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v3 07/10] fuse: add io-uring buffer pools (implemented as buffer ring)
Quoted 27 lines · Bernd Schubert
Quoted 25 lines · Bernd Schubert
Hi Joanne,
I one generic question, could you also add in for which kernel branch the series is? I think I need to do the same for larger patches series as well (recent reduced-queue was based on 'miklos/for-next'). Basically 'b4 shazam' fails to apply on linux master and also fails to on 'miklos/for-next' and I don't even know what the base for the series here is
Quoted 20 lines · Joanne Koong
Right now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
To use this, the server must set the
FUSE_URING_BUFPOOLflag duringSorry, had missed that in the earlier mail. How about renaming this to
FUSE_URING_BUFPOOL_PAYLOAD(or so), so that it would clearly differentiated from a possible header buf pool in the future?
I don't see a use case where buf pools for headers will be needed because every ent needs a header and the header is always the same fixed-size (288 bytes).
Thanks,
Joanne
Quoted 4 lines
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v3 07/10] fuse: add io-uring buffer pools (implemented as buffer ring)
In reply to Bernd SchubertQuoted 4 lines · Bernd Schubert
Hi Joanne,
I one generic question, could you also add in for which kernel branch the series is? I think I need to do the same for larger patches series as well (recent reduced-queue was based on 'miklos/for-next'). Basically 'b4 shazam' fails to apply on linux master and also fails to on 'miklos/for-next' and I don't even know what the base for the series here is
Hi Bernd,
This was rebased on top of Miklos's for-next tree (base commit eee3f1a62). This has a dependency on the io-uring registered bvec buffers changes that were added in [1].
Quoted 108 lines · Joanne Koong
Quoted 106 lines
Right now, ents and buffers are tightly coupled in fuse io-uring where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types (lookup, open, release, getattr, etc) require vastly less bytes than the maximum payload size and some requests (unlink, rmdir, fsync, flush, etc) do not require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers, allow the server to pass in a buffer pool (a contiguous allocation of buffers) that the kernel will use as it wishes for servicing ents/requests. Buffers are now pooled and selected on demand. Entries only "reserve" a buffer while actively processing a request that requires a payload buffer. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption which will allow non-overlapping regions of a single buffer to be used simultaneously across multiple ents/requests, and the possibility of multiple buffer sizes to be supported per ring in the future.
To use this, the server must set the
FUSE_URING_BUFPOOLflag during registration and provide the payload buffer pool through the iovec payload in the first sqe. The server must set init.buf_size and init.nr_bufs in the structfuse_uring_cmd_reqit passes to the kernel to describe the buffer pool configuration. Buffer pool usage is set on a per-ring basis. All subsequent registration SQEs for the same ring must pass in consistent flags.Internally, a buffer ring data structure is used for managing the buffer pool. The kernel manages buffer selection and recycling through this ring structure. A payload buffer is selected from the ring on demand if the request requires a payload buffer and recycled after the request completes if used.
The contiguous buffer pool passed in is partitioned into individual buffers according to the
buf_sizepassed in at registration time.PAYLOAD BUFFER POOL (contiguous, provided via iov[1] in first sqe):
|<-------------- payload_size ------------>| +--------- --+-----------+-----------+-----+ | buf [0] | buf [1] | buf [2] | ... | | buf_size | buf_size | buf_size | ... | +--------- --+-----------+-----------+-----+The buffer ring logic is as follows:
select: buf = bufs[head % nbufs]; head++ recycle: bufs[tail % nbufs] = buf; tail++ empty: tail == head (no buffers available) full: tail - head >= nbufsBuffer ring request flow
------------------------| Kernel | FUSE daemon | | | [client request arrives] | | >fuse_uring_send() | | [select payload buf from pool] | | >fuse_uring_select_buffer() | | [copy headers to ent] | | >copy_header_to_ring() | | [copy payload to selected buf] | | >fuse_uring_copy_to_ring() | | [set buf_id in ent_in_out header] | | >io_uring_cmd_done() | | | [CQE received] | | [read headers] | | [read payload from buf_id] | | [process request] | | [write reply to header] | | [write reply payload to | | buf] | | >io_uring_submit() | | COMMIT_AND_FETCH | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | [copy reply from ring] | | >fuse_uring_recycle_buffer() | | >fuse_uring_get_next_fuse_req() |Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 356 +++++++++++++++++++++++++++++++------- fs/fuse/dev_uring_i.h | 35 +++- include/uapi/linux/fuse.h | 29 +++- 3 files changed, 353 insertions(+), 67 deletions(-)fs/fuse/dev_uring.c +9
-1@@ -41,6 +41,11 @@ enum fuse_uring_header_type {@@ -1123,12 +1324,30 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + + err = -EINVAL ; +if (!bufpool_enabled (queue)) { + payload = &iov[FUSE_URING_IOV_PAYLOAD ]; +if (payload->iov_len < ring->max_payload_sz) { +pr_info_ratelimited ("Invalid req payload len %zu\n" , + payload->iov_len); +kfree (ent); +return ERR_PTR (err);
Quoted 8 lines
fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1338
Quoted 3 lines
fs/fuse/dev_uring.c +3
-0+ } + ent->payload = payload->iov_base; + }Let's say a confused user library would not know how to set up things and would in one SQE register a buf pool and in another register payload and header - should this fail with an error here? We could add that sanity check later, though.
This is already checked right now with the queue_init_flags_consistent() checking we do at registration time. That enforces that all or none of the registration cmds for that queue must have the FUSE_URING_BUFPOOL flag set.
Quoted 65 lines
Quoted 65 lines
fs/fuse/dev_uring.c +7
-0atomic_inc (&ring->queue_refs);return ent; } +static bool init_flags_valid (u64 init_flags) +{ +u64 valid_flags =FUSE_URING_BUFPOOL ; + +return !(init_flags & ~valid_flags); +} +include/uapi/linux/fuse.h +16
-2@@ -240,6 +240,11 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - add FUSE_URING_BUFPOOL flag +* - add fuse_uring_cmd_req init struct +* - add buf_id field to fuse_uring_ent_in_out struct */ #ifndef _LINUX_FUSE_H@@ -275,7 +280,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -1263,7 +1268,13 @@ struct fuse_uring_ent_in_out {/* size of user payload buffer */ uint32_t payload_sz; -uint32_t padding; + +/* +* if using bufpools, this is the id of the selected buffer. +* the selected buffer holds the request payload +*/ +uint16_t buf_id; +uint16_t padding;uint64_t reserved; };@@ -1294,6 +1305,9 @@ enum fuse_uring_cmd { FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, }; +/* fuse_uring_cmd_req flags */ +#define FUSE_URING_BUFPOOL (1 << 0) +/** * In the 80B command area of the SQE. */ @@ -1305,7 +1319,16 @@ struct fuse_uring_cmd_req {
Quoted 7 lines
include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1323
Quoted 5 lines
include/uapi/linux/fuse.h +2
-1/* queue the command is for (queue index) */ uint16_t qid; -uint8_t padding[6 ]; +uint16_t padding; +uint32_t reserved;Dunno, it was a padding up to 64 bit? Non-needed change?
I'll change the padding back to uint8_t padding[6].
Quoted 9 lines
include/uapi/linux/fuse.h struct fuse_uring_cmd_req { · line 1330
Quoted 8 lines
include/uapi/linux/fuse.h +7
-0+ +union { +struct { +/* FUSE_URING_BUFPOOL: buffer pool configuration */ +uint32_t buf_size; +uint16_t nr_bufs; + } init;
Quoted 6 lines
If we want to support multiple buffer pools, this struct should add in a buffer index or at a minimum reserve a field for it. Adding in
uint16_tbuf_idx; /* currently unused */ would also fill up the hole at the end. libfuse could even already setbuf_idxto 0 and then kernel could check if zero and fail something else is set. Would save us another flag later on.
I think a new flag would need to be added regardless for multi-pool support. I'm not confident we'd need an explicit pool_idx field in the init config for multi-pool support, so I think it's better to just leave that to the future if it does turn out to be needed. As for filling up the hole, the later uint16_t zero_copy_index fills that up.
Thanks,
Joanne
Joanne Koong
maintainerPATCH v3 8/10 fuse: support registered buffers as buffer pools in io-uring
In reply to Joanne KoongAllow servers to use an io-uring registered buffer as the buffer pool. When the server registers buffers with io-uring, the pages backing the buffers are pinned upfront. This eliminates the overhead of pinning/unpinning user pages and translating virtual addresses per i/o request. This also allows servers to use the same registered buffers for subsequent backing store I/O (eg read_fixed/write_fixed), keeping data in the same pinned pages without additional pinning or mapping overhead required.
Registered buffers require the buffer pool (FUSE_URING_BUFPOOL) to be enabled. To use registered buffers, the server needs to register the payload buffers through io-uring and during registration, set the FUSE_URING_REGISTERED_BUFFERS init flag (alongside the FUSE_URING_BUFPOOL flag) in the registration sqe. For all sqes the server passes in (including the registration ones), it needs to follow standard protocol for using registered buffers (eg set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED, and pass in the index where the registered buffers reside to sqe->buf_index).
In benchmarks using passthrough_hp on a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).
--- fs/fuse/dev_uring.c | 97 ++++++++++++++++++++++++++------------- fs/fuse/dev_uring_i.h | 2 + include/uapi/linux/fuse.h | 2 + 3 files changed, 70 insertions(+), 31 deletions(-)
fs/fuse/dev_uring.c +66 -31
@@ -46,6 +46,11 @@ static inline bool bufpool_enabled(struct fuse_ring_queue *queue)return queue->bufpool !=NULL ; } +static inline bool bufpool_registered_buffers (struct fuse_ring_queue *queue) +{ +return queue->bufpool && queue->bufpool->registered_buffers; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -310,7 +315,8 @@ static int fuse_uring_get_iovec_from_sqe(const struct io_uring_sqe *sqe, }static int fuse_uring_bufpool_setup (struct io_uring_cmd *cmd, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +u64 init_flags) {const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
@@ -353,6 +359,7 @@ static int fuse_uring_bufpool_setup(struct io_uring_cmd *cmd, pool->nbufs = nr_bufs; pool->tail = nr_bufs; + pool->registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; queue->bufpool = pool;
@@ -369,8 +376,10 @@ static bool queue_init_flags_consistent(struct fuse_ring_queue *queue, u64 init_flags) {bool bufpool = init_flags &FUSE_URING_BUFPOOL ; +bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; -return bufpool_enabled (queue) == bufpool; +return bufpool_enabled (queue) == bufpool && +bufpool_registered_buffers (queue) == registered_buffers; } static struct fuse_ring_queue *
@@ -408,7 +417,7 @@ fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring, queue->fpq.processing = pq;if (use_bufpool) { - err =fuse_uring_bufpool_setup (cmd, queue); + err =fuse_uring_bufpool_setup (cmd, queue, init_flags);if (err) {kfree (pq);kfree (queue);
@@ -752,28 +761,41 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,static int setup_fuse_copy_state (struct fuse_copy_state *cs,struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent,int dir, -struct iov_iter *iter) +struct iov_iter *iter, +unsigned int issue_flags) { +bool bufpool =bufpool_enabled (ent->queue); +bool registered_buffers =bufpool_registered_buffers (ent->queue);void __user *payload;size_t len; -int err; +int err =0 ;memset (iter,0 ,sizeof (*iter)); -if (bufpool_enabled (ent->queue)) { - payload = (void __user *)ent->payload_buf.addr; - len = ent->payload_buf.len; +if (registered_buffers) { +if (ent->payload_buf.addr) + err =io_uring_cmd_import_fixed (ent->payload_buf.addr, + ent->payload_buf.len, + dir, iter, ent->cmd, + issue_flags); }else { - payload = ent->payload; - len = ring->max_payload_sz; +if (bufpool) { + payload = (void __user *)ent->payload_buf.addr; + len = ent->payload_buf.len; + }else { + payload = ent->payload; + len = ring->max_payload_sz; + } + +if (payload) + err =import_ubuf (dir, payload, len, iter); } -if (payload) { - err =import_ubuf (dir, payload, len, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; - } +if (err) { +pr_info_ratelimited ("fuse: Importing iter failed, bufpool=%d, " +"registered_buffers=%d\n" , bufpool, + registered_buffers); +return err; }fuse_copy_init (cs, dir ==ITER_DEST , iter);
@@ -786,7 +808,8 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req, -struct fuse_ring_ent *ent) +struct fuse_ring_ent *ent, +unsigned int issue_flags) {struct fuse_copy_state cs;struct fuse_args *args = req->args;
@@ -799,7 +822,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter, + issue_flags);if (err)return err;
@@ -812,7 +836,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,* Copy data from the req to the ring buffer */ static int fuse_uring_args_to_ring (struct fuse_ring *ring,struct fuse_req *req, -struct fuse_ring_ent *ent) +struct fuse_ring_ent *ent, +unsigned int issue_flags) {struct fuse_copy_state cs;struct fuse_args *args = req->args;
@@ -828,7 +853,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,if (bufpool_enabled (ent->queue)) ent_in_out.buf_id = ent->payload_buf.id; - err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter, + issue_flags);if (err)return err;
@@ -863,7 +889,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, }static int fuse_uring_copy_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;
@@ -881,7 +908,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,return err;/* copy the request */ - err =fuse_uring_args_to_ring (ring, req, ent); + err =fuse_uring_args_to_ring (ring, req, ent, issue_flags);if (unlikely (err)) {pr_info_ratelimited ("Copy to ring failed: %d\n" , err);return err;
@@ -979,11 +1006,12 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent, }static int fuse_uring_prepare_send (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {int err; - err =fuse_uring_copy_to_ring (ent, req); + err =fuse_uring_copy_to_ring (ent, req, issue_flags);if (!err)set_bit (FR_SENT , &req->flags);else
@@ -1092,7 +1120,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, goto out; } - err =fuse_uring_copy_from_ring (ring, req, ent); + err =fuse_uring_copy_from_ring (ring, req, ent, issue_flags); out:fuse_uring_req_end (ent, req, err); }
@@ -1104,7 +1132,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,* Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +unsigned int issue_flags) {int err;struct fuse_req *req;
@@ -1116,7 +1145,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_prepare_send (ent, req); + err =fuse_uring_prepare_send (ent, req, issue_flags);if (err) goto retry; }
@@ -1231,7 +1260,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* no-op and the next request will be serviced when a buffer becomes * available. */ -if (fuse_uring_get_next_fuse_req (ent, queue)) +if (fuse_uring_get_next_fuse_req (ent, queue, issue_flags))fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1343,7 +1372,12 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,static bool init_flags_valid (u64 init_flags) { -u64 valid_flags =FUSE_URING_BUFPOOL ; +u64 valid_flags =FUSE_URING_BUFPOOL |FUSE_URING_REGISTERED_BUFFERS ; +bool bufpool = init_flags &FUSE_URING_BUFPOOL ; +bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; + +if (registered_buffers && !bufpool) +return false;return !(init_flags & ~valid_flags); }
@@ -1487,9 +1521,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)int err;if (!tw.cancel) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); + err =fuse_uring_prepare_send (ent, ent->fuse_req, issue_flags);if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) +if (!fuse_uring_get_next_fuse_req (ent, queue, + issue_flags))return ; err =0 ; }
fs/fuse/dev_uring_i.h +2 -0
@@ -44,6 +44,8 @@ struct fuse_bufpool_buf {/* implemented as a buffer ring */ struct fuse_bufpool { +bool registered_buffers; +/* metadata tracking state of the bufpool */ unsigned int nbufs;unsigned int head;
include/uapi/linux/fuse.h +2 -0
@@ -245,6 +245,7 @@* - add FUSE_URING_BUFPOOL flag * - add fuse_uring_cmd_req init struct * - add buf_id field to fuse_uring_ent_in_out struct +* - add FUSE_URING_REGISTERED_BUFFERS flag */ #ifndef _LINUX_FUSE_H
@@ -1307,6 +1308,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req flags */ #define FUSE_URING_BUFPOOL (1 << 0) +#define FUSE_URING_REGISTERED_BUFFERS (1 << 1)/** * In the 80B command area of the SQE.
Signature · 1 line
2.52.0
Re: [PATCH v3 08/10] fuse: support registered buffers as buffer pools in io-uring
Quoted 27 lines · Joanne Koong
Allow servers to use an io-uring registered buffer as the buffer pool. When the server registers buffers with io-uring, the pages backing the buffers are pinned upfront. This eliminates the overhead of pinning/unpinning user pages and translating virtual addresses per i/o request. This also allows servers to use the same registered buffers for subsequent backing store I/O (eg read_fixed/write_fixed), keeping data in the same pinned pages without additional pinning or mapping overhead required.
Registered buffers require the buffer pool (
FUSE_URING_BUFPOOL) to be enabled. To use registered buffers, the server needs to register the payload buffers through io-uring and during registration, set theFUSE_URING_REGISTERED_BUFFERSinit flag (alongside theFUSE_URING_BUFPOOLflag) in the registration sqe. For all sqes the server passes in (including the registration ones), it needs to follow standard protocol for using registered buffers (eg setsqe->uring_cmd_flagsto includeIORING_URING_CMD_FIXED, and pass in the index where the registered buffers reside tosqe->buf_index).In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).
Quoted 1 line
Signed-off-by Joanne Koong <[email protected]>
Looks good to me.
Quoted 287 lines
--- fs/fuse/dev_uring.c | 97 ++++++++++++++++++++++++++------------- fs/fuse/dev_uring_i.h | 2 + include/uapi/linux/fuse.h | 2 + 3 files changed, 70 insertions(+), 31 deletions(-)fs/fuse/dev_uring.c +66
-31@@ -46,6 +46,11 @@ static inline bool bufpool_enabled(struct fuse_ring_queue *queue)return queue->bufpool !=NULL ; } +static inline bool bufpool_registered_buffers (struct fuse_ring_queue *queue) +{ +return queue->bufpool && queue->bufpool->registered_buffers; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {@@ -310,7 +315,8 @@ static int fuse_uring_get_iovec_from_sqe(const struct io_uring_sqe *sqe, }static int fuse_uring_bufpool_setup (struct io_uring_cmd *cmd, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +u64 init_flags) {const struct fuse_uring_cmd_req *cmd_req = io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);@@ -353,6 +359,7 @@ static int fuse_uring_bufpool_setup(struct io_uring_cmd *cmd, pool->nbufs = nr_bufs; pool->tail = nr_bufs; + pool->registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; queue->bufpool = pool;@@ -369,8 +376,10 @@ static bool queue_init_flags_consistent(struct fuse_ring_queue *queue, u64 init_flags) {bool bufpool = init_flags &FUSE_URING_BUFPOOL ; +bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; -return bufpool_enabled (queue) == bufpool; +return bufpool_enabled (queue) == bufpool && +bufpool_registered_buffers (queue) == registered_buffers; } static struct fuse_ring_queue *@@ -408,7 +417,7 @@ fuse_uring_create_queue(struct io_uring_cmd *cmd, struct fuse_ring *ring, queue->fpq.processing = pq;if (use_bufpool) { - err =fuse_uring_bufpool_setup (cmd, queue); + err =fuse_uring_bufpool_setup (cmd, queue, init_flags);if (err) {kfree (pq);kfree (queue);@@ -752,28 +761,41 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,static int setup_fuse_copy_state (struct fuse_copy_state *cs,struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent,int dir, -struct iov_iter *iter) +struct iov_iter *iter, +unsigned int issue_flags) { +bool bufpool =bufpool_enabled (ent->queue); +bool registered_buffers =bufpool_registered_buffers (ent->queue);void __user *payload;size_t len; -int err; +int err =0 ;memset (iter,0 ,sizeof (*iter)); -if (bufpool_enabled (ent->queue)) { - payload = (void __user *)ent->payload_buf.addr; - len = ent->payload_buf.len; +if (registered_buffers) { +if (ent->payload_buf.addr) + err =io_uring_cmd_import_fixed (ent->payload_buf.addr, + ent->payload_buf.len, + dir, iter, ent->cmd, + issue_flags); }else { - payload = ent->payload; - len = ring->max_payload_sz; +if (bufpool) { + payload = (void __user *)ent->payload_buf.addr; + len = ent->payload_buf.len; + }else { + payload = ent->payload; + len = ring->max_payload_sz; + } + +if (payload) + err =import_ubuf (dir, payload, len, iter); } -if (payload) { - err =import_ubuf (dir, payload, len, iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; - } +if (err) { +pr_info_ratelimited ("fuse: Importing iter failed, bufpool=%d, " +"registered_buffers=%d\n" , bufpool, + registered_buffers); +return err; }fuse_copy_init (cs, dir ==ITER_DEST , iter);@@ -786,7 +808,8 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req, -struct fuse_ring_ent *ent) +struct fuse_ring_ent *ent, +unsigned int issue_flags) {struct fuse_copy_state cs;struct fuse_args *args = req->args;@@ -799,7 +822,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_SOURCE , &iter, + issue_flags);if (err)return err;@@ -812,7 +836,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,* Copy data from the req to the ring buffer */ static int fuse_uring_args_to_ring (struct fuse_ring *ring,struct fuse_req *req, -struct fuse_ring_ent *ent) +struct fuse_ring_ent *ent, +unsigned int issue_flags) {struct fuse_copy_state cs;struct fuse_args *args = req->args;@@ -828,7 +853,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,if (bufpool_enabled (ent->queue)) ent_in_out.buf_id = ent->payload_buf.id; - err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter); + err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter, + issue_flags);if (err)return err;@@ -863,7 +889,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, }static int fuse_uring_copy_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;@@ -881,7 +908,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,return err;/* copy the request */ - err =fuse_uring_args_to_ring (ring, req, ent); + err =fuse_uring_args_to_ring (ring, req, ent, issue_flags);if (unlikely (err)) {pr_info_ratelimited ("Copy to ring failed: %d\n" , err);return err;@@ -979,11 +1006,12 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent, }static int fuse_uring_prepare_send (struct fuse_ring_ent *ent, -struct fuse_req *req) +struct fuse_req *req, +unsigned int issue_flags) {int err; - err =fuse_uring_copy_to_ring (ent, req); + err =fuse_uring_copy_to_ring (ent, req, issue_flags);if (!err)set_bit (FR_SENT , &req->flags);else @@ -1092,7 +1120,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, goto out; } - err =fuse_uring_copy_from_ring (ring, req, ent); + err =fuse_uring_copy_from_ring (ring, req, ent, issue_flags); out:fuse_uring_req_end (ent, req, err); }@@ -1104,7 +1132,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,* Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue) +struct fuse_ring_queue *queue, +unsigned int issue_flags) {int err;struct fuse_req *req;@@ -1116,7 +1145,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_prepare_send (ent, req); + err =fuse_uring_prepare_send (ent, req, issue_flags);if (err) goto retry; }@@ -1231,7 +1260,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* no-op and the next request will be serviced when a buffer becomes * available. */ -if (fuse_uring_get_next_fuse_req (ent, queue)) +if (fuse_uring_get_next_fuse_req (ent, queue, issue_flags))fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }@@ -1343,7 +1372,12 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,static bool init_flags_valid (u64 init_flags) { -u64 valid_flags =FUSE_URING_BUFPOOL ; +u64 valid_flags =FUSE_URING_BUFPOOL |FUSE_URING_REGISTERED_BUFFERS ; +bool bufpool = init_flags &FUSE_URING_BUFPOOL ; +bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; + +if (registered_buffers && !bufpool) +return false;return !(init_flags & ~valid_flags); }@@ -1487,9 +1521,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)int err;if (!tw.cancel) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); + err =fuse_uring_prepare_send (ent, ent->fuse_req, issue_flags);if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) +if (!fuse_uring_get_next_fuse_req (ent, queue, + issue_flags))return ; err =0 ; }fs/fuse/dev_uring_i.h +2
-0@@ -44,6 +44,8 @@ struct fuse_bufpool_buf {/* implemented as a buffer ring */ struct fuse_bufpool { +bool registered_buffers; +/* metadata tracking state of the bufpool */ unsigned int nbufs;unsigned int head;include/uapi/linux/fuse.h +2
-0@@ -245,6 +245,7 @@* - add FUSE_URING_BUFPOOL flag * - add fuse_uring_cmd_req init struct * - add buf_id field to fuse_uring_ent_in_out struct +* - add FUSE_URING_REGISTERED_BUFFERS flag */ #ifndef _LINUX_FUSE_H@@ -1307,6 +1308,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req flags */ #define FUSE_URING_BUFPOOL (1 << 0) +#define FUSE_URING_REGISTERED_BUFFERS (1 << 1)/** * In the 80B command area of the SQE.
Joanne Koong
maintainerPATCH v3 9/10 fuse: add zero-copy over io-uring
In reply to Joanne KoongImplement zero-copy for fuse io-uring, eliminating memory copies between userspace, the kernel, and the fuse server for read/write operations. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. When a fuse request arrives, the kernel registers the relevant pages into a sparse slot in the server's io_uring registered buffer table. The server can then operate on these pages directly using io-uring fixed buffer operations (eg read_fixed/write_fixed) and the kernel unregisters these pages when the request completes. Non-page-backed args (eg op out headers) will go through the payload buffer as normal.
This requires CAP_SYS_ADMIN and FUSE_URING_REGISTERED_BUFFERS (which requires using bufpools). Gating on registered buffers keeps the configuration space small and avoids partially-optimized modes that are unlikely to be useful in practice. This is gated behind CAP_SYS_ADMIN because zero-copy allows the server direct access to the client's underlying pages, rather than operating on an intermediary buffer that the contents of the client's pages were copied into or on page cache folios.
The request flow for the zero-copy write path (client writes data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister sparse buffer] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args. For these requests, the page-backed args are zero-copied while the non-page-backed args are copied to the buffer selected from the buffer ring:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folioThe throughput improvement from zero-copy depends on how much of the per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED operation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additional io_uring operation may negate the savings from eliminating the memcpy.
In benchmarks using passthrough_hp on a high-performance NVMe-backed system, zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s).
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1
--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 147 +++++++++++++++++++++++++++++++++++--- fs/fuse/dev_uring_i.h | 6 ++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 12 ++++ 5 files changed, 160 insertions(+), 13 deletions(-)
fs/fuse/dev.c +5 -2
@@ -1348,10 +1348,13 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; err =fuse_copy_folios (cs, arg->size, zeroing); -else + }else { err =fuse_copy_one (cs, arg->value, arg->size); + } }return err; }
fs/fuse/dev_uring.c +136 -11
280 lines not on this page · raw message
fs/fuse/dev_uring_i.h +6 -0
@@ -45,6 +45,8 @@ struct fuse_bufpool_buf {/* implemented as a buffer ring */ struct fuse_bufpool {bool registered_buffers; +/* this is only allowed on privileged servers */ +bool use_zero_copy;/* metadata tracking state of the bufpool */ unsigned int nbufs;
@@ -69,6 +71,10 @@ struct fuse_ring_ent { }; }; +/* true if the request's pages are being zero-copied */ +bool zero_copied; +unsigned int zero_copy_index; +/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
fs/fuse/fuse_dev_i.h +1 -0
@@ -325,6 +325,7 @@ struct fuse_copy_state { bool write:1; bool move_folios:1; bool is_uring:1; + bool skip_folio_copy:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;
include/uapi/linux/fuse.h +12 -0
@@ -246,6 +246,7 @@* - add fuse_uring_cmd_req init struct * - add buf_id field to fuse_uring_ent_in_out struct * - add FUSE_URING_REGISTERED_BUFFERS flag +* - add FUSE_URING_ZERO_COPY flag */ #ifndef _LINUX_FUSE_H
@@ -1257,6 +1258,13 @@ struct fuse_supp_groups {#define FUSE_URING_IN_OUT_HEADER_SZ 128#define FUSE_URING_OP_IN_OUT_SZ 128 +/** +* fuse_uring_ent_in_out flags +* +* FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied +*/ +#define FUSE_URING_ENT_ZERO_COPY (1 << 0) +/* Used as part of the fuse_uring_req_header */ struct fuse_uring_ent_in_out {uint64_t flags;
@@ -1309,6 +1317,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req flags */ #define FUSE_URING_BUFPOOL (1 << 0)#define FUSE_URING_REGISTERED_BUFFERS (1 << 1) +#define FUSE_URING_ZERO_COPY (1 << 2)/** * In the 80B command area of the SQE.
@@ -1329,6 +1338,9 @@ struct fuse_uring_cmd_req {/* FUSE_URING_BUFPOOL: buffer pool configuration */ uint32_t buf_size;uint16_t nr_bufs; + +/* FUSE_URING_ZERO_COPY: zero copy configuration */ +uint16_t zero_copy_index; } init; }; };
Signature · 1 line
2.52.0
Re: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 120 lines · Joanne Koong
Implement zero-copy for fuse io-uring, eliminating memory copies between userspace, the kernel, and the fuse server for read/write operations. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. When a fuse request arrives, the kernel registers the relevant pages into a sparse slot in the server's
io_uringregistered buffer table. The server can then operate on these pages directly using io-uring fixed buffer operations (eg read_fixed/write_fixed) and the kernel unregisters these pages when the request completes. Non-page-backed args (eg op out headers) will go through the payload buffer as normal.This requires
CAP_SYS_ADMINandFUSE_URING_REGISTERED_BUFFERS(which requires using bufpools). Gating on registered buffers keeps the configuration space small and avoids partially-optimized modes that are unlikely to be useful in practice. This is gated behindCAP_SYS_ADMINbecause zero-copy allows the server direct access to the client's underlying pages, rather than operating on an intermediary buffer that the contents of the client's pages were copied into or on page cache folios.The request flow for the zero-copy write path (client writes data, server reads it) is as follows:
=======================================================================| Kernel | FUSE server | | | "write(fd, buf, 1MB)" | | | | >sys_write() | | >fuse_file_write_iter() | | >fuse_send_one() | | [req->args->in_pages = true] | | [folios hold client write data] | | | | >fuse_uring_copy_to_ring() | | >copy_header_to_ring(IN_OUT) | | [memcpy fuse_in_header] | | >copy_header_to_ring(OP) | | [memcpy write_in header] | | | | >fuse_uring_args_to_ring() | | >setup_fuse_copy_state() | | [skip_folio_copy = true] | | | | >fuse_uring_set_up_zero_copy() | | [folio_get for each client folio] | | [build bio_vec array from folios] | | >io_buffer_register_bvec() | | [register pages at ent->zero_copy_index] | | [ent->zero_copied = true] | | | | >fuse_copy_args() | | [skip_folio_copy => return 0 | | for page arg, skip data copy] | | | | >copy_header_to_ring(RING_ENT) | | [memcpy ent_in_out] | | >io_uring_cmd_done() | | | | | [CQE received] | | | | [issue io_uring READ at | | ent->zero_copy_index] | | [reads directly from | |client's pages (ZERO_COPY)] | | | | [write data to backing | | store] | | [submit COMMIT AND FETCH] | | | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | >fuse_uring_copy_from_ring() | | >fuse_uring_req_end() | | >io_buffer_unregister(ent->zero_copy_index) | | [unregister sparse buffer] | | >fuse_zero_copy_release() | | [folio_put for each folio] | | [ent->zero_copied = false] | | >fuse_request_end() | | [wake up client] |The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args. For these requests, the page-backed args are zero-copied while the non-page-backed args are copied to the buffer selected from the buffer ring:
zero-copy: pages registered viaio_buffer_register_bvec()
non-page-backed: copied to payload buffer viafuse_copy_args()For a request whose payload is zero-copied, the
registration/unregistration path looks like:register: fuse_uring_set_up_zero_copy() folio_get() for each folio io_buffer_register_bvec(ent->zero_copy_index) unregister: fuse_uring_req_end() io_buffer_unregister(ent->zero_copy_index) -> fuse_zero_copy_release() callback folio_put() for each folioThe throughput improvement from zero-copy depends on how much of the per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s).
Quoted 6 lines
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>
Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight, io_buffer_unregister() is never called. The buf_table slot remains occupied and folios stay pinned until the io_uring ring is destroyed.
So perhaps we need to add the corresponding cleanup in fuse_uring_entry_teardown(). Otherwise looks good to me:
Cheers,
Baokun
Quoted 411 lines
--- fs/fuse/dev.c | 7 +- fs/fuse/dev_uring.c | 147 +++++++++++++++++++++++++++++++++++--- fs/fuse/dev_uring_i.h | 6 ++ fs/fuse/fuse_dev_i.h | 1 + include/uapi/linux/fuse.h | 12 ++++ 5 files changed, 160 insertions(+), 13 deletions(-)fs/fuse/dev.c +5
-2@@ -1348,10 +1348,13 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,for (i =0 ; !err && i < numargs; i++) {struct fuse_arg *arg = &args[i]; -if (i == numargs -1 && argpages) +if (i == numargs -1 && argpages) { +if (cs->skip_folio_copy) +return 0 ; err =fuse_copy_folios (cs, arg->size, zeroing); -else + }else { err =fuse_copy_one (cs, arg->value, arg->size); + } }return err; }fs/fuse/dev_uring.c +136
-11@@ -30,6 +30,11 @@ struct fuse_uring_pdu {struct fuse_ring_ent *ent; }; +struct fuse_zero_copy_bvs { +unsigned int nr_bvs; +struct bio_vec bvs[]; +}; +static const struct fuse_iqueue_ops fuse_io_uring_ops;enum fuse_uring_header_type {@@ -51,6 +56,11 @@ static inline bool bufpool_registered_buffers(struct fuse_ring_queue *queue)return queue->bufpool && queue->bufpool->registered_buffers; } +static inline bool bufpool_zero_copy (struct fuse_ring_queue *queue) +{ +return queue->bufpool && queue->bufpool->use_zero_copy; +} +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {@@ -96,8 +106,18 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) } } +static bool can_zero_copy_req (struct fuse_ring_ent *ent,struct fuse_req *req) +{ +struct fuse_args *args = req->args; + +if (!bufpool_zero_copy (ent->queue)) +return false; + +return args->in_pages || args->out_pages; +} +static void fuse_uring_req_end (struct fuse_ring_ent *ent,struct fuse_req *req, -int error) +int error,unsigned int issue_flags) {struct fuse_ring_queue *queue = ent->queue;struct fuse_ring *ring = queue->ring;@@ -116,6 +136,11 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,spin_unlock (&queue->lock); +if (ent->zero_copied) { +io_buffer_unregister (ent->cmd, ent->zero_copy_index, issue_flags); + ent->zero_copied = false; + } +if (error) req->out.h.error = error;@@ -325,6 +350,7 @@ static int fuse_uring_bufpool_setup(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];uintptr_t payload_addr;size_t payload_size, ring_size; +bool zero_copy = init_flags &FUSE_URING_ZERO_COPY ;struct fuse_bufpool *pool;unsigned int i;int err;@@ -339,7 +365,7 @@ static int fuse_uring_bufpool_setup(struct io_uring_cmd *cmd, payload_addr = (uintptr_t )iov[FUSE_URING_IOV_PAYLOAD ].iov_base; payload_size = iov[FUSE_URING_IOV_PAYLOAD ].iov_len; -if (payload_size < (uint64_t )buf_size * nr_bufs) +if (!zero_copy && (payload_size < (uint64_t )buf_size * nr_bufs))return -EINVAL ;/* create the pool, which is managed as a ring buffer */ @@ -360,6 +386,7 @@ static int fuse_uring_bufpool_setup(struct io_uring_cmd *cmd, pool->nbufs = nr_bufs; pool->tail = nr_bufs; pool->registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; + pool->use_zero_copy = zero_copy; queue->bufpool = pool;@@ -377,9 +404,11 @@ static bool queue_init_flags_consistent(struct fuse_ring_queue *queue, {bool bufpool = init_flags &FUSE_URING_BUFPOOL ;bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; +bool zero_copy = init_flags &FUSE_URING_ZERO_COPY ;return bufpool_enabled (queue) == bufpool && -bufpool_registered_buffers (queue) == registered_buffers; +bufpool_registered_buffers (queue) == registered_buffers && +bufpool_zero_copy (queue) == zero_copy; } static struct fuse_ring_queue *@@ -800,6 +829,9 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,fuse_copy_init (cs, dir ==ITER_DEST , iter); +if (ent->zero_copied) + cs->skip_folio_copy = true; + cs->is_uring = true; cs->req = req;@@ -832,6 +864,64 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,return err; } +static void fuse_zero_copy_release (void *priv) +{ +struct fuse_zero_copy_bvs *zc_bvs = priv; +unsigned int i; + +for (i =0 ; i < zc_bvs->nr_bvs; i++) +folio_put (page_folio (zc_bvs->bvs[i].bv_page)); + +kfree (zc_bvs); +} + +static int fuse_uring_set_up_zero_copy (struct fuse_ring_ent *ent, +struct fuse_req *req, +unsigned int issue_flags) +{ +struct fuse_args_pages *ap; +int err, i, ddir =0 ; +struct fuse_zero_copy_bvs *zc_bvs; +struct bio_vec *bvs; + +/* out_pages indicates a read, in_pages indicates a write */ +if (req->args->out_pages) + ddir |=IO_BUF_DEST ; +if (req->args->in_pages) + ddir |=IO_BUF_SOURCE ; + +WARN_ON_ONCE (!ddir); + + ap =container_of (req->args,typeof (*ap), args); + + zc_bvs =kmalloc (struct_size (zc_bvs, bvs, ap->num_folios), +GFP_KERNEL_ACCOUNT ); +if (!zc_bvs) +return -ENOMEM ; + + zc_bvs->nr_bvs = ap->num_folios; + bvs = zc_bvs->bvs; +for (i =0 ; i < ap->num_folios; i++) { + bvs[i].bv_page =folio_page (ap->folios[i],0 ); + bvs[i].bv_offset = ap->descs[i].offset; + bvs[i].bv_len = ap->descs[i].length; +folio_get (ap->folios[i]); + } + + err =io_buffer_register_bvec (ent->cmd, bvs, ap->num_folios, + fuse_zero_copy_release, zc_bvs, + ddir, ent->zero_copy_index, + issue_flags); +if (err) { +fuse_zero_copy_release (zc_bvs); +return err; + } + + ent->zero_copied = true; + +return 0 ; +} +/* * Copy data from the req to the ring buffer */ @@ -850,8 +940,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; -if (bufpool_enabled (ent->queue)) +if (bufpool_enabled (ent->queue)) { ent_in_out.buf_id = ent->payload_buf.id; +if (can_zero_copy_req (ent, req)) { + ent_in_out.flags |=FUSE_URING_ENT_ZERO_COPY ; + err =fuse_uring_set_up_zero_copy (ent, req, issue_flags); +if (err) +return err; + } + } err =setup_fuse_copy_state (&cs, ring, req, ent,ITER_DEST , &iter, issue_flags);@@ -884,6 +981,10 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; +if (cs.skip_folio_copy && args->in_pages) + ent_in_out.payload_sz += + args->in_args[args->in_numargs -1 ].size; +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , &ent_in_out,sizeof (ent_in_out)); }@@ -919,11 +1020,25 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } -static bool fuse_uring_req_has_payload (struct fuse_req *req) +static bool fuse_uring_req_has_copyable_payload (struct fuse_ring_ent *ent, +struct fuse_req *req) {struct fuse_args *args = req->args; -return args->in_numargs >1 || args->out_numargs; +if (!can_zero_copy_req (ent, req)) +return args->in_numargs >1 || args->out_numargs; + +/* +* the asymmetry between in_numargs > 2 and out_numargs > 1 is because +* the per-op header is extracted before fuse_copy_args() for inargs but +* not for outargs +*/ +if ((args->in_numargs >1 ) && (!args->in_pages || args->in_numargs >2 )) +return true; +if (args->out_numargs && (!args->out_pages || args->out_numargs >1 )) +return true; + +return false; }static int fuse_uring_select_buffer (struct fuse_ring_ent *ent)@@ -981,7 +1096,7 @@ static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,return 0 ; buffer_selected = !!ent->payload_buf.addr; - has_payload =fuse_uring_req_has_payload (req); + has_payload =fuse_uring_req_has_copyable_payload (ent, req);if (has_payload && !buffer_selected)return fuse_uring_select_buffer (ent);@@ -999,7 +1114,7 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,return 0 ;/* no payload to copy, can skip selecting a buffer */ -if (!fuse_uring_req_has_payload (req)) +if (!fuse_uring_req_has_copyable_payload (ent, req))return 0 ;return fuse_uring_select_buffer (ent);@@ -1015,7 +1130,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,if (!err)set_bit (FR_SENT , &req->flags);else -fuse_uring_req_end (ent, req, err); +fuse_uring_req_end (ent, req, err, issue_flags);return err; }@@ -1122,7 +1237,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, err =fuse_uring_copy_from_ring (ring, req, ent, issue_flags); out: -fuse_uring_req_end (ent, req, err); +fuse_uring_req_end (ent, req, err, issue_flags); }/* @@ -1324,6 +1439,8 @@ static struct fuse_ring_ent * fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, struct fuse_ring_queue *queue) { + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);struct fuse_ring *ring = queue->ring;struct fuse_ring_ent *ent;struct iovec iov[FUSE_URING_IOV_SEGS ];@@ -1366,19 +1483,27 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->payload = payload->iov_base; } +if (bufpool_zero_copy (queue)) + ent->zero_copy_index =READ_ONCE (cmd_req->init.zero_copy_index); +atomic_inc (&ring->queue_refs);return ent; }static bool init_flags_valid (u64 init_flags) { -u64 valid_flags =FUSE_URING_BUFPOOL |FUSE_URING_REGISTERED_BUFFERS ; +u64 valid_flags =FUSE_URING_BUFPOOL |FUSE_URING_REGISTERED_BUFFERS + |FUSE_URING_ZERO_COPY ;bool bufpool = init_flags &FUSE_URING_BUFPOOL ;bool registered_buffers = init_flags &FUSE_URING_REGISTERED_BUFFERS ; +bool zero_copy = init_flags &FUSE_URING_ZERO_COPY ;if (registered_buffers && !bufpool)return false; +if (zero_copy && (!registered_buffers || !capable (CAP_SYS_ADMIN ))) +return false; +return !(init_flags & ~valid_flags); }fs/fuse/dev_uring_i.h +6
-0@@ -45,6 +45,8 @@ struct fuse_bufpool_buf {/* implemented as a buffer ring */ struct fuse_bufpool {bool registered_buffers; +/* this is only allowed on privileged servers */ +bool use_zero_copy;/* metadata tracking state of the bufpool */ unsigned int nbufs;@@ -69,6 +71,10 @@ struct fuse_ring_ent { }; }; +/* true if the request's pages are being zero-copied */ +bool zero_copied; +unsigned int zero_copy_index; +/* the ring queue that owns the request */ struct fuse_ring_queue *queue;fs/fuse/fuse_dev_i.h +1
-0@@ -325,6 +325,7 @@ struct fuse_copy_state { bool write:1; bool move_folios:1; bool is_uring:1; + bool skip_folio_copy:1;struct {unsigned int copied_sz;/* copied size into the user buffer */ } ring;include/uapi/linux/fuse.h +12
-0@@ -246,6 +246,7 @@* - add fuse_uring_cmd_req init struct * - add buf_id field to fuse_uring_ent_in_out struct * - add FUSE_URING_REGISTERED_BUFFERS flag +* - add FUSE_URING_ZERO_COPY flag */ #ifndef _LINUX_FUSE_H@@ -1257,6 +1258,13 @@ struct fuse_supp_groups {#define FUSE_URING_IN_OUT_HEADER_SZ 128#define FUSE_URING_OP_IN_OUT_SZ 128 +/** +* fuse_uring_ent_in_out flags +* +* FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied +*/ +#define FUSE_URING_ENT_ZERO_COPY (1 << 0) +/* Used as part of the fuse_uring_req_header */ struct fuse_uring_ent_in_out {uint64_t flags;@@ -1309,6 +1317,7 @@ enum fuse_uring_cmd {/* fuse_uring_cmd_req flags */ #define FUSE_URING_BUFPOOL (1 << 0)#define FUSE_URING_REGISTERED_BUFFERS (1 << 1) +#define FUSE_URING_ZERO_COPY (1 << 2)/** * In the 80B command area of the SQE. @@ -1329,6 +1338,9 @@ struct fuse_uring_cmd_req {/* FUSE_URING_BUFPOOL: buffer pool configuration */ uint32_t buf_size;uint16_t nr_bufs; + +/* FUSE_URING_ZERO_COPY: zero copy configuration */ +uint16_t zero_copy_index; } init; }; };
Joanne Koong
maintainerRe: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 131 lines · Baokun Li
Quoted 120 lines · Joanne Koong
Implement zero-copy for fuse io-uring, eliminating memory copies between userspace, the kernel, and the fuse server for read/write operations. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. When a fuse request arrives, the kernel registers the relevant pages into a sparse slot in the server's
io_uringregistered buffer table. The server can then operate on these pages directly using io-uring fixed buffer operations (eg read_fixed/write_fixed) and the kernel unregisters these pages when the request completes. Non-page-backed args (eg op out headers) will go through the payload buffer as normal.This requires
CAP_SYS_ADMINandFUSE_URING_REGISTERED_BUFFERS(which requires using bufpools). Gating on registered buffers keeps the configuration space small and avoids partially-optimized modes that are unlikely to be useful in practice. This is gated behindCAP_SYS_ADMINbecause zero-copy allows the server direct access to the client's underlying pages, rather than operating on an intermediary buffer that the contents of the client's pages were copied into or on page cache folios.The request flow for the zero-copy write path (client writes data, server reads it) is as follows:
=======================================================================| Kernel | FUSE server | | | "write(fd, buf, 1MB)" | | | | >sys_write() | | >fuse_file_write_iter() | | >fuse_send_one() | | [req->args->in_pages = true] | | [folios hold client write data] | | | | >fuse_uring_copy_to_ring() | | >copy_header_to_ring(IN_OUT) | | [memcpy fuse_in_header] | | >copy_header_to_ring(OP) | | [memcpy write_in header] | | | | >fuse_uring_args_to_ring() | | >setup_fuse_copy_state() | | [skip_folio_copy = true] | | | | >fuse_uring_set_up_zero_copy() | | [folio_get for each client folio] | | [build bio_vec array from folios] | | >io_buffer_register_bvec() | | [register pages at ent->zero_copy_index] | | [ent->zero_copied = true] | | | | >fuse_copy_args() | | [skip_folio_copy => return 0 | | for page arg, skip data copy] | | | | >copy_header_to_ring(RING_ENT) | | [memcpy ent_in_out] | | >io_uring_cmd_done() | | | | | [CQE received] | | | | [issue io_uring READ at | | ent->zero_copy_index] | | [reads directly from | |client's pages (ZERO_COPY)] | | | | [write data to backing | | store] | | [submit COMMIT AND FETCH] | | | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | >fuse_uring_copy_from_ring() | | >fuse_uring_req_end() | | >io_buffer_unregister(ent->zero_copy_index) | | [unregister sparse buffer] | | >fuse_zero_copy_release() | | [folio_put for each folio] | | [ent->zero_copied = false] | | >fuse_request_end() | | [wake up client] |The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args. For these requests, the page-backed args are zero-copied while the non-page-backed args are copied to the buffer selected from the buffer ring:
zero-copy: pages registered viaio_buffer_register_bvec()
non-page-backed: copied to payload buffer viafuse_copy_args()For a request whose payload is zero-copied, the
registration/unregistration path looks like:register: fuse_uring_set_up_zero_copy() folio_get() for each folio io_buffer_register_bvec(ent->zero_copy_index) unregister: fuse_uring_req_end() io_buffer_unregister(ent->zero_copy_index) -> fuse_zero_copy_release() callback folio_put() for each folioThe throughput improvement from zero-copy depends on how much of the per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s).Quoted 6 lines
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>Hi Joanne,
Quoted 8 lines
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:Reviewed-by Baokun Li <[email protected]>
Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in fuse_uring_req_end() when a request is ended, but the abort code calls fuse_uring_stop_fuse_req_end(), not fuse_uring_req_end(). I'll add this in.
Thanks,
Joanne
Quoted 4 lines
Cheers,
Baokun
Joanne Koong
maintainerRe: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 140 lines · Joanne Koong
Quoted 138 lines · Baokun Li
Quoted 120 lines · Joanne Koong
Implement zero-copy for fuse io-uring, eliminating memory copies between userspace, the kernel, and the fuse server for read/write operations. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. When a fuse request arrives, the kernel registers the relevant pages into a sparse slot in the server's
io_uringregistered buffer table. The server can then operate on these pages directly using io-uring fixed buffer operations (eg read_fixed/write_fixed) and the kernel unregisters these pages when the request completes. Non-page-backed args (eg op out headers) will go through the payload buffer as normal.This requires
CAP_SYS_ADMINandFUSE_URING_REGISTERED_BUFFERS(which requires using bufpools). Gating on registered buffers keeps the configuration space small and avoids partially-optimized modes that are unlikely to be useful in practice. This is gated behindCAP_SYS_ADMINbecause zero-copy allows the server direct access to the client's underlying pages, rather than operating on an intermediary buffer that the contents of the client's pages were copied into or on page cache folios.The request flow for the zero-copy write path (client writes data, server reads it) is as follows:
=======================================================================| Kernel | FUSE server | | | "write(fd, buf, 1MB)" | | | | >sys_write() | | >fuse_file_write_iter() | | >fuse_send_one() | | [req->args->in_pages = true] | | [folios hold client write data] | | | | >fuse_uring_copy_to_ring() | | >copy_header_to_ring(IN_OUT) | | [memcpy fuse_in_header] | | >copy_header_to_ring(OP) | | [memcpy write_in header] | | | | >fuse_uring_args_to_ring() | | >setup_fuse_copy_state() | | [skip_folio_copy = true] | | | | >fuse_uring_set_up_zero_copy() | | [folio_get for each client folio] | | [build bio_vec array from folios] | | >io_buffer_register_bvec() | | [register pages at ent->zero_copy_index] | | [ent->zero_copied = true] | | | | >fuse_copy_args() | | [skip_folio_copy => return 0 | | for page arg, skip data copy] | | | | >copy_header_to_ring(RING_ENT) | | [memcpy ent_in_out] | | >io_uring_cmd_done() | | | | | [CQE received] | | | | [issue io_uring READ at | | ent->zero_copy_index] | | [reads directly from | |client's pages (ZERO_COPY)] | | | | [write data to backing | | store] | | [submit COMMIT AND FETCH] | | | >fuse_uring_commit_fetch() | | >fuse_uring_commit() | | >fuse_uring_copy_from_ring() | | >fuse_uring_req_end() | | >io_buffer_unregister(ent->zero_copy_index) | | [unregister sparse buffer] | | >fuse_zero_copy_release() | | [folio_put for each folio] | | [ent->zero_copied = false] | | >fuse_request_end() | | [wake up client] |The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args. For these requests, the page-backed args are zero-copied while the non-page-backed args are copied to the buffer selected from the buffer ring:
zero-copy: pages registered viaio_buffer_register_bvec()
non-page-backed: copied to payload buffer viafuse_copy_args()For a request whose payload is zero-copied, the
registration/unregistration path looks like:register: fuse_uring_set_up_zero_copy() folio_get() for each folio io_buffer_register_bvec(ent->zero_copy_index) unregister: fuse_uring_req_end() io_buffer_unregister(ent->zero_copy_index) -> fuse_zero_copy_release() callback folio_put() for each folioThe throughput improvement from zero-copy depends on how much of the per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s).Quoted 6 lines
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:
Quoted 8 lines
Quoted 1 line
Reviewed-by Baokun Li <[email protected]>Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in
fuse_uring_req_end()when a request is ended, but the abort code callsfuse_uring_stop_fuse_req_end(), notfuse_uring_req_end(). I'll add this in.
Revisiting this... I had this cleanup in the series I sent out last December [1] but in order to accommodate unregistration on teardown during the abort race, it requires nontrivial changes to io-uring, as it requires operating on a ring ctx handle but it's tricky to ascertain if the handle is valid or not. I had discussed it with Jens at LPC last year, and we came to the conclusion that it's not worth introducing the complexity to accommodate this specific scenario and it's fine to leave the buffer table slot occupied with pinned folios until the io-uring ring is destroyed.
Thanks,
Joanne
[1] https://lore.kernel.org/io-uring/[email protected]/
Quoted 8 lines
Thanks,
JoanneQuoted 4 lines
Cheers,
Baokun
Re: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 16 lines · Joanne Koong
Quoted 9 lines
Quoted 9 lines
Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:Quoted 7 lines
Quoted 1 line
Reviewed-by Baokun Li <[email protected]>Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in
fuse_uring_req_end()when a request is ended, but the abort code callsfuse_uring_stop_fuse_req_end(), notfuse_uring_req_end(). I'll add this in.
Quoted 9 lines
Revisiting this... I had this cleanup in the series I sent out last December [1] but in order to accommodate unregistration on teardown during the abort race, it requires nontrivial changes to io-uring, as it requires operating on a ring ctx handle but it's tricky to ascertain if the handle is valid or not. I had discussed it with Jens at LPC last year, and we came to the conclusion that it's not worth introducing the complexity to accommodate this specific scenario and it's fine to leave the buffer table slot occupied with pinned folios until the io-uring ring is destroyed.
Hi Joanne,
Thanks for explaining the background.
I think we can handle this with a simple change in fuse_uring_entry_teardown(), without any io-uring side modifications:
fs/fuse/dev_uring.c +0 -0
@@ -514,8 +514,14 @@ static void fuse_uring_entry_teardown(struct
fuse_ring_ent *ent)
ent->state = FRRS_RELEASED;
spin_unlock(&queue->lock);
- if (cmd)
+ if (cmd) {
+ if (ent->zero_copied) {
+ io_buffer_unregister(cmd, ent->zero_copy_index, + IO_URING_F_UNLOCKED); + ent->zero_copied = false;
+ }
io_uring_cmd_done(cmd, -ENOTCONN, IO_URING_F_UNLOCKED); + }
if (req) fuse_uring_stop_fuse_req_end(req);
Here's why I believe this is safe:
1) No race with IO_URING_F_CANCEL. The cancel path (fuse_uring_cancel) only takes cmd from entries in FRRS_AVAILABLE state:
spin_lock(&queue->lock);
if (ent->state == FRRS_AVAILABLE) {
ent->cmd = NULL;
...
}
spin_unlock(&queue->lock);
When zero_copied == true, the entry is processing a request and is NOT in FRRS_AVAILABLE state. So cancel will never steal cmd from a zero-copied entry. Teardown is guaranteed to find cmd != NULL under queue->lock for this case.
2) Ring ctx is guaranteed alive. Since io_uring_cmd_done() has not been called yet, the io_kiocb is still inflight and holds a reference to the ring ctx. io_buffer_unregister() can safely operate on it.
3) The scenario motivating this fix: a multi-mount fuse daemon (e.g., virtiofsd serving multiple VMs) gets "umount -f" on one mount. The connection aborts but the daemon and its io_uring ring stay alive indefinitely. Without this cleanup, the pinned folios from the aborted mount remain in the buffer table slots until the daemon process eventually exits — freed only through:
close(ring_fd) / process exit
→ io_ring_ctx_free
→ io_sqe_buffers_unregister
→ io_rsrc_data_free
→ io_put_rsrc_node
→ io_free_rsrc_node
→ io_buffer_unmap
→ imu->release(imu->priv)
For a long-lived daemon, this could be days or weeks.
Thanks,
Baokun
Joanne Koong
maintainerRe: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 38 lines · Baokun Li
Quoted 16 lines · Joanne Koong
Quoted 9 lines
Quoted 9 lines
Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:Quoted 7 lines
Quoted 1 line
Reviewed-by Baokun Li <[email protected]>Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in
fuse_uring_req_end()when a request is ended, but the abort code callsfuse_uring_stop_fuse_req_end(), notfuse_uring_req_end(). I'll add this in.Quoted 9 lines
Revisiting this... I had this cleanup in the series I sent out last December [1] but in order to accommodate unregistration on teardown during the abort race, it requires nontrivial changes to io-uring, as it requires operating on a ring ctx handle but it's tricky to ascertain if the handle is valid or not. I had discussed it with Jens at LPC last year, and we came to the conclusion that it's not worth introducing the complexity to accommodate this specific scenario and it's fine to leave the buffer table slot occupied with pinned folios until the io-uring ring is destroyed.
Hi Joanne,
Thanks for explaining the background.
I think we can handle this with a simple change in
fuse_uring_entry_teardown(), without any io-uring side modifications:
fs/fuse/dev_uring.c static void fuse_uring_entry_teardown(struct · line 522
Quoted 10 lines
fs/fuse/dev_uring.c +5
-1@@ -514,8 +514,14 @@ static void fuse_uring_entry_teardown(struct fuse_ring_ent *ent) ent->state = FRRS_RELEASED;spin_unlock (&queue->lock); -if (cmd) +if (cmd) { +if (ent->zero_copied) { +io_buffer_unregister (cmd, ent->zero_copy_index, +IO_URING_F_UNLOCKED ); + ent->zero_copied = false;
Hi Baokun,
Unfortunately, I don't think this fires. For ents in userspace, there is no cmd associated with the ent. cmds are only valid for an ent when the kernel hasn't yet "returned" the cmd to userspace (this is enforced in fuse_uring_send() where ent->cmd gets set to NULL). For the scenario you're describing (a zero-copy request in-flight in userspace when the abort hits), the ent is in FRRS_USERSPACE with ent->cmd == NULL, which means the if (cmd) guard above will skip this logic.
Thanks,
Joanne
Quoted 7 lines
fs/fuse/dev_uring.c +2
-0+ }io_uring_cmd_done (cmd, -ENOTCONN ,IO_URING_F_UNLOCKED ); + }if (req)fuse_uring_stop_fuse_req_end(req);Quoted 1 line
Re: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 49 lines · Joanne Koong
Quoted 36 lines · Baokun Li
Quoted 16 lines · Joanne Koong
Quoted 9 lines
Quoted 9 lines
Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:Quoted 7 lines
Quoted 1 line
Reviewed-by Baokun Li <[email protected]>Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in
fuse_uring_req_end()when a request is ended, but the abort code callsfuse_uring_stop_fuse_req_end(), notfuse_uring_req_end(). I'll add this in.Quoted 9 lines
Revisiting this... I had this cleanup in the series I sent out last December [1] but in order to accommodate unregistration on teardown during the abort race, it requires nontrivial changes to io-uring, as it requires operating on a ring ctx handle but it's tricky to ascertain if the handle is valid or not. I had discussed it with Jens at LPC last year, and we came to the conclusion that it's not worth introducing the complexity to accommodate this specific scenario and it's fine to leave the buffer table slot occupied with pinned folios until the io-uring ring is destroyed.
Hi Joanne,
Thanks for explaining the background.
I think we can handle this with a simple change in
fuse_uring_entry_teardown(), without any io-uring side modifications:fs/fuse/dev_uring.c static void fuse_uring_entry_teardown(struct · line 522
Quoted 10 lines
fs/fuse/dev_uring.c +5
-1@@ -514,8 +514,14 @@ static void fuse_uring_entry_teardown(struct fuse_ring_ent *ent) ent->state = FRRS_RELEASED;spin_unlock (&queue->lock); -if (cmd) +if (cmd) { +if (ent->zero_copied) { +io_buffer_unregister (cmd, ent->zero_copy_index, +IO_URING_F_UNLOCKED ); + ent->zero_copied = false;Hi Baokun,
Quoted 10 lines
Unfortunately, I don't think this fires. For ents in userspace, there is no cmd associated with the ent. cmds are only valid for an ent when the kernel hasn't yet "returned" the cmd to userspace (this is enforced in
fuse_uring_send()whereent->cmdgets set to NULL). For the scenario you're describing (a zero-copy request in-flight in userspace when the abort hits), the ent is inFRRS_USERSPACEwithent->cmd== NULL, which means the if (cmd) guard above will skip this logic.
Hi Joanne,
You're right, I missed that. fuse_uring_send() completes the cmd and sets ent->cmd = NULL before the entry enters FRRS_USERSPACE. So when abort hits a zero-copied entry, cmd is already gone — the if (cmd) guard skips the unregister entirely.
The timeline is:
1. Request dispatched → io_buffer_register_bvec(), zero_copied = true
2. fuse_uring_send() → ent->cmd = NULL, io_uring_cmd_done()
3. Server doing zero-copy IO, ent in FRRS_USERSPACE, no cmd
4. umount -f → teardown → cmd == NULL → unregister never fires
I now understand the problem you described — to unregister without cmd, we'd need an independent handle to the ring ctx, and ensuring its validity is where the complexity lies. I agree that it's not worth introducing that complexity for this scenario, and leaving the buffer table slot occupied until the ring is destroyed is a reasonable tradeoff.
Thanks for taking the time to walk me through this.
Thanks,
Baokun
Joanne Koong
maintainerRe: [PATCH v3 09/10] fuse: add zero-copy over io-uring
Quoted 75 lines · Baokun Li
Quoted 49 lines · Joanne Koong
Quoted 36 lines · Baokun Li
Quoted 16 lines · Joanne Koong
Quoted 9 lines
Quoted 9 lines
Hi Joanne,
If a connection abort happens while a zero-copy request is in-flight,
io_buffer_unregister()is never called. Thebuf_tableslot remains occupied and folios stay pinned until theio_uringring is destroyed.So perhaps we need to add the corresponding cleanup in
fuse_uring_entry_teardown(). Otherwise looks good to me:Quoted 7 lines
Quoted 1 line
Reviewed-by Baokun Li <[email protected]>Hi Baokun,
Thanks for reviewing this series. That's a good point - the io buffer is unregistered in
fuse_uring_req_end()when a request is ended, but the abort code callsfuse_uring_stop_fuse_req_end(), notfuse_uring_req_end(). I'll add this in.Quoted 9 lines
Revisiting this... I had this cleanup in the series I sent out last December [1] but in order to accommodate unregistration on teardown during the abort race, it requires nontrivial changes to io-uring, as it requires operating on a ring ctx handle but it's tricky to ascertain if the handle is valid or not. I had discussed it with Jens at LPC last year, and we came to the conclusion that it's not worth introducing the complexity to accommodate this specific scenario and it's fine to leave the buffer table slot occupied with pinned folios until the io-uring ring is destroyed.
Hi Joanne,
Thanks for explaining the background.
I think we can handle this with a simple change in
fuse_uring_entry_teardown(), without any io-uring side modifications:fs/fuse/dev_uring.c static void fuse_uring_entry_teardown(struct · line 522
Quoted 10 lines
fs/fuse/dev_uring.c +5
-1@@ -514,8 +514,14 @@ static void fuse_uring_entry_teardown(struct fuse_ring_ent *ent) ent->state = FRRS_RELEASED;spin_unlock (&queue->lock); -if (cmd) +if (cmd) { +if (ent->zero_copied) { +io_buffer_unregister (cmd, ent->zero_copy_index, +IO_URING_F_UNLOCKED ); + ent->zero_copied = false;Hi Baokun,
Quoted 10 lines
Unfortunately, I don't think this fires. For ents in userspace, there is no cmd associated with the ent. cmds are only valid for an ent when the kernel hasn't yet "returned" the cmd to userspace (this is enforced in
fuse_uring_send()whereent->cmdgets set to NULL). For the scenario you're describing (a zero-copy request in-flight in userspace when the abort hits), the ent is inFRRS_USERSPACEwithent->cmd== NULL, which means the if (cmd) guard above will skip this logic.Hi Joanne,
You're right, I missed that.
fuse_uring_send()completes the cmd and setsent->cmd= NULL before the entry entersFRRS_USERSPACE. So when abort hits a zero-copied entry, cmd is already gone — the if (cmd) guard skips the unregister entirely.The timeline is:
1. Request dispatched →
io_buffer_register_bvec(),zero_copied= true
2.fuse_uring_send()→ent->cmd= NULL,io_uring_cmd_done()
3. Server doing zero-copy IO, ent inFRRS_USERSPACE, no cmd
4. umount -f → teardown → cmd == NULL → unregister never fires
Quoted 9 lines
I now understand the problem you described — to unregister without cmd, we'd need an independent handle to the ring ctx, and ensuring its validity is where the complexity lies. I agree that it's not worth introducing that complexity for this scenario, and leaving the buffer table slot occupied until the ring is destroyed is a reasonable tradeoff.
Thanks for taking the time to walk me through this.
No worries at all, I think this is very subtle so I will add something about this to the commit message in v4 to make this more clear.
Thanks,
Joanne
Quoted 4 lines
Thanks,
Baokun
Joanne Koong
maintainerPATCH v3 10/10 docs: fuse: add io-uring buffer pool and zero-copy documentation
In reply to Joanne KoongAdd documentation for fuse over io-uring usage of buffer pools and zero-copy.
--- .../filesystems/fuse/fuse-io-uring.rst | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+)
Documentation/filesystems/fuse/fuse-io-uring.rst +194 -0
@@ -95,5 +95,199 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | +Buffer pools +============ +Buffer pools decouple the 1:1 relationship between ents and payload buffers, +which allow for more optimal usage of memory. Entries only 'reserve' a buffer +when actively processing a request that requires a payload buffer. This +decoupling also allows for optimizations such as incremental buffer +consumption which will allow non-overlapping regions of a single buffer to be +used simultaneously across multiple ents/requests, and the possibility of +multiple buffer sizes to be supported per ring in the future. +To use this, the server must set the FUSE_URING_BUFPOOL flag during +registration and provide the payload buffer pool through the iovec payload in +in the first sqe. The server needs to pass init.buf_size and init.nr_bufs +describing the buffer pool configuration in the struct fuse_uring_cmd_req it +passes to the kernel. Buffer pool usage is set on a per-ring basis. Please +note that all subsequent registration SQEs for the same ring must pass in +consistent flags. + +Internally, fuse manages the buffer pool through a buffer ring. At a +high-level, this is how it works: + +* The first REGISTER SQE for a queue creates the queue and sets up the + buffer pool. The server passes in the allocated buffer pool in the first + registration sqe in iovec[1]. +* When a client request arrives, if the request has copyable data, the kernel + selects a buffer from the ring, copies headers and payload data, and + completes the sqe. +* The buf_id of the selected payload buffer is communicated to the server + via the fuse_uring_ent_in_out header. The server uses this to locate the + payload buffer corresponding to the request. +* The server processes the request and sends a COMMIT_AND_FETCH SQE with + the reply. The kernel processes the reply and recycles the buffer. + +Visually, this looks like:: + + Payload buffer pool: + +-----------+-----------+-----------+-----+ + | buf 0 | buf 1 | buf 2 | ... | + | (buf_size)| (buf_size)| (buf_size)| | + +-----------+-----------+-----------+-----+ + selected on demand, recycled after each request + +Buffer pool request flow +------------------------:: + +| Kernel | FUSE daemon +| | +| [client request arrives] | +| >fuse_uring_send() | +| [select payload buf from pool] | +| >fuse_uring_select_buffer() | +| >copy_header_to_ring() | +| [copy payload to selected buf] | +| >fuse_uring_copy_to_ring() | +| [set buf_id in ent_in_out header] | +| >io_uring_cmd_done() | +| | [CQE received] +| | [read header] +| | [read payload from buf_id] +| | [process request] +| | [write reply to header] +| | [write reply payload to buf] +| | >io_uring_submit() +| | COMMIT_AND_FETCH +| >fuse_uring_commit_fetch() | +| >fuse_uring_commit() | +| [copy reply from ring] | +| >fuse_uring_recycle_buffer() | +| >fuse_uring_get_next_fuse_req() | + +Registered buffers +================== + +Servers can pass in an io-uring registered buffer as the buffer pool. When the +server registers buffers with io-uring, the pages backing the buffers are +pinned upfront. This eliminates the overhead of pinning/unpinning user pages +and translating virtual addresses per i/o request. This also allows servers to +use the same registered buffers for subsequent backing store I/O (eg +read_fixed/write_fixed), keeping data in the same pinned pages without +additional pinning or mapping overhead required. + +Registered buffers require the buffer pool (FUSE_URING_BUFPOOL) to be +enabled. To use registered buffers, the server needs to register the +payload buffers through io-uring and during registration, set the +FUSE_URING_REGISTERED_BUFFERS init flag (alongside the +FUSE_URING_BUFPOOL flag) in the registration sqe. For all sqes the +server passes in (including the registration ones), it needs to follow +standard protocol for using registered buffers (eg set +sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED, and pass in the +index where the registered buffers reside to sqe->buf_index). For an example +of how to set this up, please refer to the libfuse/lib/fuse_uring.c file. + +Zero-copy +========= + +Fuse io-uring zero-copy allows the server to directly read from / write to +the client's pages, bypassing any intermediary buffer copies. This requires +the FUSE_URING_ZERO_COPY flag and requires the server to use registered +buffers. This requires CAP_SYS_ADMIN privileges as zero-copy allows the server +direct access to the client's underlying pages, rather than operating on an +intermediary buffer that the contents of client's pages were copied into or on +page cache folios. + +When a fuse request arrives, the kernel registers the relevant pages (pinned +user pages for direct i/o or underlying page cache folios for buffered i/o) +into a sparse slot in the server's io_uring registered buffer table. The +server can then operate on these pages directly using io-uring fixed buffer +operations (eg read_fixed / write_fixed) and the kernel unregisters these +pages when the request completes. Non-page-backed args (eg op out headers) +will go through the payload buffer as normal. + +The request flow for the zero-copy write path (client writes data, server +reads it) is as follows: + +Zero-copy write +---------------:: +| Kernel | FUSE server +| | +| "write(fd, buf, 1MB)" | +| | +| >sys_write() | +| >fuse_file_write_iter() | +| >fuse_send_one() | +| [req->args->in_pages = true] | +| [folios hold client write data] | +| | +| >fuse_uring_copy_to_ring() | +| >copy_header_to_ring(IN_OUT) | +| [memcpy fuse_in_header] | +| >copy_header_to_ring(OP) | +| [memcpy write_in header] | +| | +| >fuse_uring_args_to_ring() | +| >setup_fuse_copy_state() | +| [skip_folio_copy = true] | +| | +| >fuse_uring_set_up_zero_copy() | +| [folio_get for each client folio] | +| [build bio_vec array from folios] | +| >io_buffer_register_bvec() | +| [register pages at | +| ent->zero_copy_index] | +| [ent->zero_copied = true] | +| | +| >fuse_copy_args() | +| [skip_folio_copy => return 0 | +| for page arg, skip data copy] | +| | +| >copy_header_to_ring(RING_ENT) | +| [memcpy ent_in_out] | +| >io_uring_cmd_done() | +| | +| | [CQE received] +| | +| | [issue io_uring READ at +| | ent->zero_copy_index] +| | [reads directly from +| | client's pages (ZERO_COPY)] +| | +| | [write data to backing +| | store] +| | [submit COMMIT AND FETCH] +| | +| >fuse_uring_commit_fetch() | +| >fuse_uring_commit() | +| >fuse_uring_copy_from_ring() | +| >fuse_uring_req_end() | +| >io_buffer_unregister(ent->zero_copy_index) | +| [unregister sparse buffer] | +| >fuse_zero_copy_release() | +| [folio_put for each folio] | +| [ent->zero_copied = false] | +| >fuse_request_end() | +| [wake up client] | + +The zero-copy read path is analogous. + +Some requests may have both page-backed args and non-page-backed args. +For these requests, the page-backed args are zero-copied while the +non-page-backed args are copied to the buffer selected from the buffer +pool: + zero-copy: pages registered via io_buffer_register_bvec() + non-page-backed: copied to payload buffer via fuse_copy_args() + +For a request whose payload is zero-copied, the registration/unregistration +path looks like: + +register: fuse_uring_set_up_zero_copy() + folio_get() for each folio + io_buffer_register_bvec(ent->zero_copy_index) + +unregister: fuse_uring_req_end() + io_buffer_unregister(ent->zero_copy_index) + -> fuse_zero_copy_release() callback + folio_put() for each folio
Signature · 1 line
2.52.0
Re: [PATCH v3 10/10] docs: fuse: add io-uring buffer pool and zero-copy documentation
Quoted 5 lines · Joanne Koong
Add documentation for fuse over io-uring usage of buffer pools and zero-copy.
Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>
Looks good to me, just one nit below:
Quoted 16 lines
--- .../filesystems/fuse/fuse-io-uring.rst | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+)Documentation/filesystems/fuse/fuse-io-uring.rst +3
-0@@ -95,5 +95,199 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | +Buffer pools +============ +Buffer pools decouple the 1:1 relationship between ents and payload buffers,
Documentation/filesystems/fuse/fuse-io-uring.rst Sending requests with CQEs · line 111
Quoted 10 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +9
-0+which allow for more optimal usage of memory. Entries only 'reserve' a buffer +when actively processing a request that requires a payload buffer. This +decoupling also allows for optimizations such as incremental buffer +consumption which will allow non-overlapping regions of a single buffer to be +used simultaneously across multiple ents/requests, and the possibility of +multiple buffer sizes to be supported per ring in the future. +To use this, the server must set the FUSE_URING_BUFPOOL flag during +registration and provide the payload buffer pool through the iovec payload in +in the first sqe. The server needs to pass init.buf_size and init.nr_bufs
Nit: "the iovec payload in in the first sqe" has a duplicated "in".
Quoted 182 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +182
-0+describing the buffer pool configuration in the struct fuse_uring_cmd_req it +passes to the kernel. Buffer pool usage is set on a per-ring basis. Please +note that all subsequent registration SQEs for the same ring must pass in +consistent flags. + +Internally, fuse manages the buffer pool through a buffer ring. At a +high-level, this is how it works: + +* The first REGISTER SQE for a queue creates the queue and sets up the + buffer pool. The server passes in the allocated buffer pool in the first + registration sqe in iovec[1]. +* When a client request arrives, if the request has copyable data, the kernel + selects a buffer from the ring, copies headers and payload data, and + completes the sqe. +* The buf_id of the selected payload buffer is communicated to the server + via the fuse_uring_ent_in_out header. The server uses this to locate the + payload buffer corresponding to the request. +* The server processes the request and sends a COMMIT_AND_FETCH SQE with + the reply. The kernel processes the reply and recycles the buffer. + +Visually, this looks like:: + + Payload buffer pool: + +-----------+-----------+-----------+-----+ + | buf 0 | buf 1 | buf 2 | ... | + | (buf_size)| (buf_size)| (buf_size)| | + +-----------+-----------+-----------+-----+ + selected on demand, recycled after each request + +Buffer pool request flow +------------------------:: + +| Kernel | FUSE daemon +| | +| [client request arrives] | +| >fuse_uring_send() | +| [select payload buf from pool] | +| >fuse_uring_select_buffer() | +| >copy_header_to_ring() | +| [copy payload to selected buf] | +| >fuse_uring_copy_to_ring() | +| [set buf_id in ent_in_out header] | +| >io_uring_cmd_done() | +| | [CQE received] +| | [read header] +| | [read payload from buf_id] +| | [process request] +| | [write reply to header] +| | [write reply payload to buf] +| | >io_uring_submit() +| | COMMIT_AND_FETCH +| >fuse_uring_commit_fetch() | +| >fuse_uring_commit() | +| [copy reply from ring] | +| >fuse_uring_recycle_buffer() | +| >fuse_uring_get_next_fuse_req() | + +Registered buffers +================== + +Servers can pass in an io-uring registered buffer as the buffer pool. When the +server registers buffers with io-uring, the pages backing the buffers are +pinned upfront. This eliminates the overhead of pinning/unpinning user pages +and translating virtual addresses per i/o request. This also allows servers to +use the same registered buffers for subsequent backing store I/O (eg +read_fixed/write_fixed), keeping data in the same pinned pages without +additional pinning or mapping overhead required. + +Registered buffers require the buffer pool (FUSE_URING_BUFPOOL) to be +enabled. To use registered buffers, the server needs to register the +payload buffers through io-uring and during registration, set the +FUSE_URING_REGISTERED_BUFFERS init flag (alongside the +FUSE_URING_BUFPOOL flag) in the registration sqe. For all sqes the +server passes in (including the registration ones), it needs to follow +standard protocol for using registered buffers (eg set +sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED, and pass in the +index where the registered buffers reside to sqe->buf_index). For an example +of how to set this up, please refer to the libfuse/lib/fuse_uring.c file. + +Zero-copy +========= + +Fuse io-uring zero-copy allows the server to directly read from / write to +the client's pages, bypassing any intermediary buffer copies. This requires +the FUSE_URING_ZERO_COPY flag and requires the server to use registered +buffers. This requires CAP_SYS_ADMIN privileges as zero-copy allows the server +direct access to the client's underlying pages, rather than operating on an +intermediary buffer that the contents of client's pages were copied into or on +page cache folios. + +When a fuse request arrives, the kernel registers the relevant pages (pinned +user pages for direct i/o or underlying page cache folios for buffered i/o) +into a sparse slot in the server's io_uring registered buffer table. The +server can then operate on these pages directly using io-uring fixed buffer +operations (eg read_fixed / write_fixed) and the kernel unregisters these +pages when the request completes. Non-page-backed args (eg op out headers) +will go through the payload buffer as normal. + +The request flow for the zero-copy write path (client writes data, server +reads it) is as follows: + +Zero-copy write +---------------:: +| Kernel | FUSE server +| | +| "write(fd, buf, 1MB)" | +| | +| >sys_write() | +| >fuse_file_write_iter() | +| >fuse_send_one() | +| [req->args->in_pages = true] | +| [folios hold client write data] | +| | +| >fuse_uring_copy_to_ring() | +| >copy_header_to_ring(IN_OUT) | +| [memcpy fuse_in_header] | +| >copy_header_to_ring(OP) | +| [memcpy write_in header] | +| | +| >fuse_uring_args_to_ring() | +| >setup_fuse_copy_state() | +| [skip_folio_copy = true] | +| | +| >fuse_uring_set_up_zero_copy() | +| [folio_get for each client folio] | +| [build bio_vec array from folios] | +| >io_buffer_register_bvec() | +| [register pages at | +| ent->zero_copy_index] | +| [ent->zero_copied = true] | +| | +| >fuse_copy_args() | +| [skip_folio_copy => return 0 | +| for page arg, skip data copy] | +| | +| >copy_header_to_ring(RING_ENT) | +| [memcpy ent_in_out] | +| >io_uring_cmd_done() | +| | +| | [CQE received] +| | +| | [issue io_uring READ at +| | ent->zero_copy_index] +| | [reads directly from +| | client's pages (ZERO_COPY)] +| | +| | [write data to backing +| | store] +| | [submit COMMIT AND FETCH] +| | +| >fuse_uring_commit_fetch() | +| >fuse_uring_commit() | +| >fuse_uring_copy_from_ring() | +| >fuse_uring_req_end() | +| >io_buffer_unregister(ent->zero_copy_index) | +| [unregister sparse buffer] | +| >fuse_zero_copy_release() | +| [folio_put for each folio] | +| [ent->zero_copied = false] | +| >fuse_request_end() | +| [wake up client] | + +The zero-copy read path is analogous. + +Some requests may have both page-backed args and non-page-backed args. +For these requests, the page-backed args are zero-copied while the +non-page-backed args are copied to the buffer selected from the buffer +pool: + zero-copy: pages registered via io_buffer_register_bvec() + non-page-backed: copied to payload buffer via fuse_copy_args() + +For a request whose payload is zero-copied, the registration/unregistration +path looks like: + +register: fuse_uring_set_up_zero_copy() + folio_get() for each folio + io_buffer_register_bvec(ent->zero_copy_index) + +unregister: fuse_uring_req_end() + io_buffer_unregister(ent->zero_copy_index) + -> fuse_zero_copy_release() callback + folio_put() for each folio
Joanne Koong
maintainerRe: [PATCH v3 10/10] docs: fuse: add io-uring buffer pool and zero-copy documentation
Quoted 35 lines · Baokun Li
Quoted 5 lines · Joanne Koong
Add documentation for fuse over io-uring usage of buffer pools and zero-copy.
Reviewed-by Jeff Layton <[email protected]>Signed-off-by Joanne Koong <[email protected]>Looks good to me, just one nit below:
Reviewed-by Baokun Li <[email protected]>Quoted 23 lines
--- .../filesystems/fuse/fuse-io-uring.rst | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+)Documentation/filesystems/fuse/fuse-io-uring.rst +9
-0@@ -95,5 +95,199 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | +Buffer pools +============ +Buffer pools decouple the 1:1 relationship between ents and payload buffers, +which allow for more optimal usage of memory. Entries only 'reserve' a buffer +when actively processing a request that requires a payload buffer. This +decoupling also allows for optimizations such as incremental buffer +consumption which will allow non-overlapping regions of a single buffer to be +used simultaneously across multiple ents/requests, and the possibility of +multiple buffer sizes to be supported per ring in the future.
Quoted 5 lines
Documentation/filesystems/fuse/fuse-io-uring.rst Sending requests with CQEs · line 111
Quoted 3 lines
Documentation/filesystems/fuse/fuse-io-uring.rst +3
-0+To use this, the server must set the FUSE_URING_BUFPOOL flag during +registration and provide the payload buffer pool through the iovec payload in +in the first sqe. The server needs to pass init.buf_size and init.nr_bufsNit: "the iovec payload in in the first sqe" has a duplicated "in".
I will fix this up for v4. Thanks for taking a look at this.
Joanne Koong
maintainerQuoted 10 lines · Joanne Koong
Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers infrastructure instead of doing pinning logic in fuse. Get rid of header pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different naming
Just to reiterate, the logic is the same between this version and the previous versions. I just renamed it to "buffer pool" to decouple the implementation detail from the concept since it's uapi facing. The buffer pool is managed as a buffer ring internally.
Bernd, it would be much appreciated if you could take a look at this series when you have the time. I'm targeting the June merge window and the io-uring dependent changes have already been queued up in Jens's tree since the last merge window.
Thanks,
Joanne
Bernd Schubert
maintainerHi Joanne,
Quoted 11 lines · Joanne Koong
Quoted 10 lines · Joanne Koong
Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers infrastructure instead of doing pinning logic in fuse. Get rid of header pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different naming
short question, I didn't check the code is header pinning still *possible*? The problem if we do not pin headers is that io_uring_cmd_complete_in_task() cannot be avoided, as the application cannot write into fuse-server buffers and headers without pre-pinned pages. And io_uring_cmd_complete_in_task() basically means a context switch for all sync requests.
Quoted 10 lines
Just to reiterate, the logic is the same between this version and the previous versions. I just renamed it to "buffer pool" to decouple the implementation detail from the concept since it's uapi facing. The buffer pool is managed as a buffer ring internally.
Bernd, it would be much appreciated if you could take a look at this series when you have the time. I'm targeting the June merge window and the io-uring dependent changes have already been queued up in Jens's tree since the last merge window.
I will try and right is probably a good time, almost through with libfuse sync-init (remaining are bugs), Darricks series is also almost ready. Although I don't promise this weekend - reserved for Darrick and other libfuse changes.
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that could be pinned that way without loosing one page per entry)? For zero-copy we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
Thanks,
Bernd
Thanks,
Bernd
Joanne Koong
maintainerQuoted 16 lines · Bernd Schubert
Hi Joanne,
Quoted 11 lines · Joanne Koong
Quoted 10 lines · Joanne Koong
Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers infrastructure instead of doing pinning logic in fuse. Get rid of header pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different naming
Quoted 6 lines
short question, I didn't check the code is header pinning still *possible*? The problem if we do not pin headers is that
io_uring_cmd_complete_in_task()cannot be avoided, as the application cannot write into fuse-server buffers and headers without pre-pinned pages. Andio_uring_cmd_complete_in_task()basically means a context switch for all sync requests.
Hi Bernd,
Header pinning was dropped from this patchset. I have a separate branch that has header pinning alongside the changes for avoiding the fuse io_uring_cmd_complete_in_task() deferral. I moved the header pinning changes out of this patchset as it doesn't tie in with getting perf improvements from using registered buffers, unlike payload buffer pinning.
Quoted 7 lines
Quoted 7 lines
Just to reiterate, the logic is the same between this version and the previous versions. I just renamed it to "buffer pool" to decouple the implementation detail from the concept since it's uapi facing. The buffer pool is managed as a buffer ring internally.
Quoted 6 lines
Quoted 4 lines
Bernd, it would be much appreciated if you could take a look at this series when you have the time. I'm targeting the June merge window and the io-uring dependent changes have already been queued up in Jens's tree since the last merge window.
I will try and right is probably a good time, almost through with
Thanks!
Quoted 8 lines
libfuse sync-init (remaining are bugs), Darricks series is also almost ready. Although I don't promise this weekend - reserved for Darrick and other libfuse changes.
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that could
I think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the CAP_SYS_ADMIN requirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.
I hope I'm interpreting your question correctly - if not, please let me know!
Thanks,
Joanne
Quoted 7 lines
Thanks,
BerndThanks,
Bernd
Bernd Schubert
maintainerQuoted 72 lines · Joanne Koong
Quoted 16 lines · Bernd Schubert
Hi Joanne,
Quoted 11 lines · Joanne Koong
Quoted 10 lines · Joanne Koong
Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/[email protected]/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers infrastructure instead of doing pinning logic in fuse. Get rid of header pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different namingQuoted 6 lines
short question, I didn't check the code is header pinning still *possible*? The problem if we do not pin headers is that
io_uring_cmd_complete_in_task()cannot be avoided, as the application cannot write into fuse-server buffers and headers without pre-pinned pages. Andio_uring_cmd_complete_in_task()basically means a context switch for all sync requests.Hi Bernd,
Header pinning was dropped from this patchset. I have a separate branch that has header pinning alongside the changes for avoiding the fuse
io_uring_cmd_complete_in_task()deferral. I moved the header pinning changes out of this patchset as it doesn't tie in with getting perf improvements from using registered buffers, unlike payload buffer pinning.Quoted 7 lines
Quoted 7 lines
Just to reiterate, the logic is the same between this version and the previous versions. I just renamed it to "buffer pool" to decouple the implementation detail from the concept since it's uapi facing. The buffer pool is managed as a buffer ring internally.
Quoted 6 lines
Quoted 4 lines
Bernd, it would be much appreciated if you could take a look at this series when you have the time. I'm targeting the June merge window and the io-uring dependent changes have already been queued up in Jens's tree since the last merge window.
I will try and right is probably a good time, almost through with
Thanks!
Quoted 8 lines
libfuse sync-init (remaining are bugs), Darricks series is also almost ready. Although I don't promise this weekend - reserved for Darrick and other libfuse changes.
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
Quoted 7 lines
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
At least I do not see the use case for ring entries without buffers. And that is exactly where my question about keeping the possibility for dynamic ring entry registration comes from. If ring entries register the buffers themselves, every ring could be set up with a single ring entry first and only once ring entries start to get used, libfuse could register more based on demand.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 31 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
Quoted 5 lines · Bernd Schubert
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
Quoted 7 lines
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.
Quoted 2 lines
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
Entries may be used for servicing requests even if there are less buffers than the number of entries. Most requests don't use the full payload buffer size, so for example with incremental buffer consumption, this would allow multiple entries to use non-overlapping regions of the same buffer concurrently. As well, if we want to support multiple buffer sizes per ring, there shouldn't be a 1:1 mapping between ents and buffers. There's also some requests that don't need to use payload buffers at all (eg flush, fsync, rmdir, etc). As well for registered buffers / buffer pinning, it is more optimal for the buffer pool to be allocated contiguously and pinned contiguously rather than passed in one by one as a 1:1 entry:buffer mapping.
Thanks,
Joanne
Quoted 10 lines
At least I do not see the use case for ring entries without buffers. And that is exactly where my question about keeping the possibility for dynamic ring entry registration comes from. If ring entries register the buffers themselves, every ring could be set up with a single ring entry first and only once ring entries start to get used, libfuse could register more based on demand.
Thanks,
Bernd
Bernd Schubert
maintainerQuoted 32 lines · Joanne Koong
Quoted 31 lines · Bernd Schubert
Quoted 28 lines · Joanne Koong
Quoted 5 lines · Bernd Schubert
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.
Quoted 6 lines
Quoted 2 lines
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
Entries may be used for servicing requests even if there are less buffers than the number of entries. Most requests don't use the full payload buffer size, so for example with incremental buffer
I think that is an argument for multi-size bufs per ring.
Quoted 4 lines
consumption, this would allow multiple entries to use non-overlapping regions of the same buffer concurrently. As well, if we want to support multiple buffer sizes per ring, there shouldn't be a 1:1 mapping between ents and buffers. There's also some requests that
I can counter argue that fuse-server can just send ring entries with a different payload sizes - fuse-server could dynamically create as many ring entries of the right size as needed.
Quoted 1 line
don't need to use payload buffers at all (eg flush, fsync, rmdir,
Could be handled by multi buf sizes as well.
Quoted 4 lines
etc). As well for registered buffers / buffer pinning, it is more optimal for the buffer pool to be allocated contiguously and pinned contiguously rather than passed in one by one as a 1:1 entry:buffer mapping.
I don't think how the buffer is passed to fuse-client/kernel matters for (userspace) allocation. I had written before that libfuse's buffer allocation is not ideal, I had it continuous in initial development, then run into bug and for simplicity changed to per entry allocation. Switching back to larger allocations is simple.
Regarding page mapping/pinning, why does it matter if the buffer in kernel is known as per entry or per allocation (assuming payload buffers are page aligned)?
I think the more important part for buf rings are pinned headers. For 4K pages the loss is maybe still acceptable and the remaining size could be used for small requests and I guess would have a perf benefit. For 64K pages the loss is quite a lot.
Maybe let's summarize
Bufrings
- Fixed size, allocated at startup, cannot be easily increased or decreased
- Allows 0 size payloads (FUSE requests that only need the header)
Entries with payload
- Allows more flexible setup - fuse server _could_ (not implemented as of now) create entries based on incoming IO.
Neither provides real multi-size buffers per ring.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 35 lines · Bernd Schubert
Quoted 32 lines · Joanne Koong
Quoted 31 lines · Bernd Schubert
Quoted 28 lines · Joanne Koong
Quoted 5 lines · Bernd Schubert
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
Entries may be used for servicing requests even if there are less buffers than the number of entries. Most requests don't use the full payload buffer size, so for example with incremental buffer
I think that is an argument for multi-size bufs per ring.
Which is what the buffer pool provides (decoupling buffers from entries) if we choose to later add that in the future. With 1:1 entry:buffer mapping, each entry requires a buffer that supports the max payload size at registration time.
Quoted 9 lines
Quoted 5 lines
consumption, this would allow multiple entries to use non-overlapping regions of the same buffer concurrently. As well, if we want to support multiple buffer sizes per ring, there shouldn't be a 1:1 mapping between ents and buffers. There's also some requests that
I can counter argue that fuse-server can just send ring entries with a different payload sizes - fuse-server could dynamically create as many ring entries of the right size as needed.
I don't think this is a good idea. The server would have to guesstimate this as it doesn't know the request size until it sees the request. The server could speculatively pre-register ring ents ahead of time but there's no guarantee that would be the right size for the ring entry either. And once added, if the workload pattern changes, then the ent and allocated buffer may not be useful anymore. This also adds (imo needless) complexity to the kernel side. Additionally, the dynamic ring ents would in the ideal case have payload buffers pinned, which there is not an non-messy way to do (for buffer pools, the pool buffer pinning goes through io-uring registered buffers infrastructure prior to being passed to fuse at registration time). Imo the better way if you wanted to dynamically add ents would be to add a new buffer pool to the ring.
Quoted 4 lines
Quoted 2 lines
don't need to use payload buffers at all (eg flush, fsync, rmdir,
Could be handled by multi buf sizes as well.
These requests don't require payload buffers.
This series already handles this case (eg not assigning a buffer from the pool to the request).
Quoted 6 lines
Quoted 5 lines
etc). As well for registered buffers / buffer pinning, it is more optimal for the buffer pool to be allocated contiguously and pinned contiguously rather than passed in one by one as a 1:1 entry:buffer mapping.
Quoted 8 lines
I don't think how the buffer is passed to fuse-client/kernel matters for (userspace) allocation. I had written before that libfuse's buffer allocation is not ideal, I had it continuous in initial development, then run into bug and for simplicity changed to per entry allocation. Switching back to larger allocations is simple.
Regarding page mapping/pinning, why does it matter if the buffer in kernel is known as per entry or per allocation (assuming payload buffers are page aligned)?
It does matter how the buffers are passed to fuse for page pinning. Please take a look at patch 8 ([PATCH v3 08/10] fuse: support registered buffers as buffer pools in io-uring) [1]. The buffer pool is contiguously allocated and registered as an io-uring registered buffer, and then passed to fuse and by that point, the buffers have already been pinned by io-uring.
Even irrespective of page pinning, it would still be good to pass the buffers contiguously up front as a pool instead of per-ent, as the purpose of this series is to decouple the 1:1 entry:buffer mapping. Eg if the request an ent is servicing does not require buffers, a buffer should not be reserved for that ent, amongst other benefits of a decoupled mapping.
[1] https://lore.kernel.org/fuse-devel/[email protected]/
Quoted 5 lines
I think the more important part for buf rings are pinned headers. For 4K pages the loss is maybe still acceptable and the remaining size could be used for small requests and I guess would have a perf benefit. For 64K pages the loss is quite a lot.
That is orthogonal to this series. This series focuses on payload buffers, and header pinning is not part of this series as bufpools/bufrings do not need pinned headers, as it makes no difference for performance. As mentioned in the previous message, when I was working on a separate series for removing the dispatch overhead, I implemented header pinning as part of that, as that is when header pinning becomes actually useful.
Quoted 8 lines
Maybe let's summarize
Bufrings
- Fixed size, allocated at startup, cannot be easily increased or decreased
- Allows 0 size payloads (FUSE requests that only need the header)
Quoted 6 lines
Entries with payload
- Allows more flexible setup - fuse server _could_ (not implemented as of now) create entries based on incoming IO.Neither provides real multi-size buffers per ring.
I think perhaps there is a misunderstanding of the patches in this series.
This series implements buffer pools, which decouple the 1:1 mapping between ents and buffers that currently exists. Right now, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size.
This series allows servers to pass in a buffer pool where the buffers are not statically mapped to an ent at registration time. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It is the necessary stepping stone for future optimizations such as incremental buffer consumption (allowing multiple entries to use non-overlapping regions of the same buffer concurrently) and supporting multi-size buffers per ring.
Thanks,
Joanne
Quoted 5 lines
Thanks,
Bernd
Bernd Schubert
maintainerQuoted 43 lines · Joanne Koong
Quoted 42 lines · Bernd Schubert
Quoted 38 lines · Joanne Koong
Quoted 31 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
Quoted 5 lines · Bernd Schubert
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
Quoted 7 lines
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.Quoted 2 lines
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
Entries may be used for servicing requests even if there are less buffers than the number of entries. Most requests don't use the full payload buffer size, so for example with incremental buffer
Quoted 6 lines
Quoted 1 line
I think that is an argument for multi-size bufs per ring.
Which is what the buffer pool provides (decoupling buffers from entries) if we choose to later add that in the future. With 1:1 entry:buffer mapping, each entry requires a buffer that supports the max payload size at registration time.
Let's discuss further below, I would really like to see a design discussion how we make that generic for different IO sizes.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
consumption, this would allow multiple entries to use non-overlapping regions of the same buffer concurrently. As well, if we want to support multiple buffer sizes per ring, there shouldn't be a 1:1 mapping between ents and buffers. There's also some requests that
Quoted 7 lines
Quoted 3 lines
I can counter argue that fuse-server can just send ring entries with a different payload sizes - fuse-server could dynamically create as many ring entries of the right size as needed.
I don't think this is a good idea. The server would have to guesstimate this as it doesn't know the request size until it sees the request. The server could speculatively pre-register ring ents ahead
How expensive (in terms of memory) usage it is to set up a few small buffers? Compared to one large buffer?
Quoted 10 lines
of time but there's no guarantee that would be the right size for the ring entry either. And once added, if the workload pattern changes, then the ent and allocated buffer may not be useful anymore. This also adds (imo needless) complexity to the kernel side. Additionally, the dynamic ring ents would in the ideal case have payload buffers pinned, which there is not an non-messy way to do (for buffer pools, the pool buffer pinning goes through io-uring registered buffers infrastructure prior to being passed to fuse at registration time). Imo the better way if you wanted to dynamically add ents would be to add a new buffer pool to the ring.
I think the whole point here is that a few small payload sizes are negligible to one single max payload.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
don't need to use payload buffers at all (eg flush, fsync, rmdir,
Could be handled by multi buf sizes as well.
These requests don't require payload buffers.
This series already handles this case (eg not assigning a buffer from the pool to the request).
It handles exactly one additional case - zero payload size.
Quoted 38 lines
Quoted 7 lines
Quoted 6 lines
etc). As well for registered buffers / buffer pinning, it is more optimal for the buffer pool to be allocated contiguously and pinned contiguously rather than passed in one by one as a 1:1 entry:buffer mapping.
Quoted 8 lines
I don't think how the buffer is passed to fuse-client/kernel matters for (userspace) allocation. I had written before that libfuse's buffer allocation is not ideal, I had it continuous in initial development, then run into bug and for simplicity changed to per entry allocation. Switching back to larger allocations is simple.
Regarding page mapping/pinning, why does it matter if the buffer in kernel is known as per entry or per allocation (assuming payload buffers are page aligned)?It does matter how the buffers are passed to fuse for page pinning. Please take a look at patch 8 ([PATCH v3 08/10] fuse: support registered buffers as buffer pools in io-uring) [1]. The buffer pool is contiguously allocated and registered as an io-uring registered buffer, and then passed to fuse and by that point, the buffers have already been pinned by io-uring.
Even irrespective of page pinning, it would still be good to pass the buffers contiguously up front as a pool instead of per-ent, as the purpose of this series is to decouple the 1:1 entry:buffer mapping. Eg if the request an ent is servicing does not require buffers, a buffer should not be reserved for that ent, amongst other benefits of a decoupled mapping.
[1] https://lore.kernel.org/fuse-devel/[email protected]/
Quoted 5 lines
I think the more important part for buf rings are pinned headers. For 4K pages the loss is maybe still acceptable and the remaining size could be used for small requests and I guess would have a perf benefit. For 64K pages the loss is quite a lot.
Quoted 7 lines
That is orthogonal to this series. This series focuses on payload buffers, and header pinning is not part of this series as bufpools/bufrings do not need pinned headers, as it makes no difference for performance. As mentioned in the previous message, when I was working on a separate series for removing the dispatch overhead, I implemented header pinning as part of that, as that is when header pinning becomes actually useful.
Yes, header pinning is orthogonal, with the exception that ring entries of different payload sizes provide the same feature as buffer pools and with that buffer pools would not be a requirement anymore.
Quoted 24 lines
Quoted 9 lines
Maybe let's summarize
Bufrings
- Fixed size, allocated at startup, cannot be easily increased or decreased
- Allows 0 size payloads (FUSE requests that only need the header)Quoted 6 lines
Entries with payload
- Allows more flexible setup - fuse server _could_ (not implemented as of now) create entries based on incoming IO.Neither provides real multi-size buffers per ring.
I think perhaps there is a misunderstanding of the patches in this series.
This series implements buffer pools, which decouple the 1:1 mapping between ents and buffers that currently exists. Right now, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size.
Quoted 7 lines
This series allows servers to pass in a buffer pool where the buffers are not statically mapped to an ent at registration time. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It is the necessary stepping stone for future optimizations such as incremental buffer consumption (allowing multiple entries to use non-overlapping regions of the same buffer concurrently) and supporting multi-size buffers per ring.
I just don't see "reduces the memory usage" - there are few op code that do not need payload buffers. Do you see that exactly these op code disturb your workflow and you do need additional entries/buffers for those? I do see a lot of lookup/revalidate/getxattr, but not the op codes without payload. Equally I see IO that does not max payload for a lot of requests types - anything namespace related and reads/writes below max payload size.
In the end, what I'm struggling with is that the "zero payload" is just one special case of different IO sizes and the entire code and flow is changed for this one special case, without a discussion forehand how to handle the other cases. And that one special case comes up rarely in my own testing and use cases.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 67 lines · Bernd Schubert
Quoted 43 lines · Joanne Koong
Quoted 42 lines · Bernd Schubert
Quoted 38 lines · Joanne Koong
Quoted 31 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
Quoted 5 lines · Bernd Schubert
One piece I had been pondering a bit about, though did not dive into the code yet.
What is if payload would be registered per ring entry (as without your changes) and just the header would become a buffer ring (so that couldI think payload buffers must be decoupled from ring entries if we want to reduce memory usage in the non-zero-copy path.
Quoted 1 line
be pinned that way without loosing one page per entry)? For zero-copy
For the header pinning changes in my branch, the headers are one contiguous region (not per-entry), which circumvents having to allocate one page per entry.
Quoted 4 lines
we could register entries without a payload? Advantage here would be that that header buffer would determine max number of entries, only little memory would be needed for over-subscribed entries, but entries themselves could be registered dynamically on demand.
Quoted 7 lines
For zero-copy, entries still may have payloads (though the buffers will definitely not need to be as big as the max payload size!), either for the non-zero-copyable args or if the request's args aren't page-backed. Unfortunately I think many servers may not be able to use zero-copy due to the
CAP_SYS_ADMINrequirements, so I think it'd be ideal to minimize the memory consumption needed for the non-zero-copy path, through the buffer pool for the payload buffers.Quoted 2 lines
Well, my problem is still why would we need to register ring entries if they don't have buffers - why not just register less ring entries then.
Entries may be used for servicing requests even if there are less buffers than the number of entries. Most requests don't use the full payload buffer size, so for example with incremental buffer
Quoted 6 lines
Quoted 1 line
I think that is an argument for multi-size bufs per ring.
Which is what the buffer pool provides (decoupling buffers from entries) if we choose to later add that in the future. With 1:1 entry:buffer mapping, each entry requires a buffer that supports the max payload size at registration time.
Let's discuss further below, I would really like to see a design discussion how we make that generic for different IO sizes.
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
consumption, this would allow multiple entries to use non-overlapping regions of the same buffer concurrently. As well, if we want to support multiple buffer sizes per ring, there shouldn't be a 1:1 mapping between ents and buffers. There's also some requests that
I can counter argue that fuse-server can just send ring entries with a different payload sizes - fuse-server could dynamically create as many ring entries of the right size as needed.
Quoted 6 lines
Quoted 3 lines
I don't think this is a good idea. The server would have to guesstimate this as it doesn't know the request size until it sees the request. The server could speculatively pre-register ring ents ahead
How expensive (in terms of memory) usage it is to set up a few small buffers? Compared to one large buffer?
It's not just about the memory savings. Fundamentally, I believe it's the wrong abstraction to statically couple ents to buffers 1:1 as a fixed mapping. Entries and buffers serve different purposes. Entries handle request flow, buffers hold data. Statically coupling them 1:1 is inflexible, introduces unnecessary complexity, and imo has no added benefit. This series decouples these two concepts. It allows dynamically adding ents in the future as *generic* ents not tied to a particular buffer size (eg the added ent can be used to service *any* request size, not just the size of the buffer it's tied to). With tying ents to buffers, the server is stuck with a static distribution that can't adapt to workload changes.
Quoted 73 lines
Quoted 11 lines
of time but there's no guarantee that would be the right size for the ring entry either. And once added, if the workload pattern changes, then the ent and allocated buffer may not be useful anymore. This also adds (imo needless) complexity to the kernel side. Additionally, the dynamic ring ents would in the ideal case have payload buffers pinned, which there is not an non-messy way to do (for buffer pools, the pool buffer pinning goes through io-uring registered buffers infrastructure prior to being passed to fuse at registration time). Imo the better way if you wanted to dynamically add ents would be to add a new buffer pool to the ring.
I think the whole point here is that a few small payload sizes are negligible to one single max payload.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
don't need to use payload buffers at all (eg flush, fsync, rmdir,
Could be handled by multi buf sizes as well.
These requests don't require payload buffers.
This series already handles this case (eg not assigning a buffer from the pool to the request).
It handles exactly one additional case - zero payload size.
Quoted 37 lines
Quoted 6 lines
Quoted 5 lines
etc). As well for registered buffers / buffer pinning, it is more optimal for the buffer pool to be allocated contiguously and pinned contiguously rather than passed in one by one as a 1:1 entry:buffer mapping.
Quoted 8 lines
I don't think how the buffer is passed to fuse-client/kernel matters for (userspace) allocation. I had written before that libfuse's buffer allocation is not ideal, I had it continuous in initial development, then run into bug and for simplicity changed to per entry allocation. Switching back to larger allocations is simple.
Regarding page mapping/pinning, why does it matter if the buffer in kernel is known as per entry or per allocation (assuming payload buffers are page aligned)?It does matter how the buffers are passed to fuse for page pinning. Please take a look at patch 8 ([PATCH v3 08/10] fuse: support registered buffers as buffer pools in io-uring) [1]. The buffer pool is contiguously allocated and registered as an io-uring registered buffer, and then passed to fuse and by that point, the buffers have already been pinned by io-uring.
Even irrespective of page pinning, it would still be good to pass the buffers contiguously up front as a pool instead of per-ent, as the purpose of this series is to decouple the 1:1 entry:buffer mapping. Eg if the request an ent is servicing does not require buffers, a buffer should not be reserved for that ent, amongst other benefits of a decoupled mapping.
[1] https://lore.kernel.org/fuse-devel/[email protected]/
Quoted 5 lines
I think the more important part for buf rings are pinned headers. For 4K pages the loss is maybe still acceptable and the remaining size could be used for small requests and I guess would have a perf benefit. For 64K pages the loss is quite a lot.
Quoted 7 lines
That is orthogonal to this series. This series focuses on payload buffers, and header pinning is not part of this series as bufpools/bufrings do not need pinned headers, as it makes no difference for performance. As mentioned in the previous message, when I was working on a separate series for removing the dispatch overhead, I implemented header pinning as part of that, as that is when header pinning becomes actually useful.
Quoted 3 lines
Yes, header pinning is orthogonal, with the exception that ring entries of different payload sizes provide the same feature as buffer pools and with that buffer pools would not be a requirement anymore.
Buffer pools / buffer rings make entries generic (can service any size), whereas per-entry different sizes locks each entry to a specific size. I don't think ents and buffers should be statically coupled 1:1.
Quoted 38 lines
Quoted 22 lines
Quoted 7 lines
Maybe let's summarize
Bufrings
- Fixed size, allocated at startup, cannot be easily increased or decreased
- Allows 0 size payloads (FUSE requests that only need the header)Quoted 6 lines
Entries with payload
- Allows more flexible setup - fuse server _could_ (not implemented as of now) create entries based on incoming IO.Neither provides real multi-size buffers per ring.
I think perhaps there is a misunderstanding of the patches in this series.
This series implements buffer pools, which decouple the 1:1 mapping between ents and buffers that currently exists. Right now, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size.
Quoted 7 lines
This series allows servers to pass in a buffer pool where the buffers are not statically mapped to an ent at registration time. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It is the necessary stepping stone for future optimizations such as incremental buffer consumption (allowing multiple entries to use non-overlapping regions of the same buffer concurrently) and supporting multi-size buffers per ring.
I just don't see "reduces the memory usage" - there are few op code that do not need payload buffers. Do you see that exactly these op code disturb your workflow and you do need additional entries/buffers for those? I do see a lot of lookup/revalidate/getxattr, but not the op codes without payload. Equally I see IO that does not max payload for a lot of requests types - anything namespace related and reads/writes below max payload size.
Quoted 5 lines
In the end, what I'm struggling with is that the "zero payload" is just one special case of different IO sizes and the entire code and flow is changed for this one special case, without a discussion forehand how to handle the other cases. And that one special case comes up rarely in my own testing and use cases.
I think we're talking past each other. Buffer pools aren't about "zero payload" as a special case - they're about decoupling buffers from entries in general. Zero-payload requests are one beneficiary, but the design enables incremental consumption and multi-size buffers as future work precisely because of this decoupling.
Thanks,
Joanne
Quoted 4 lines
Thanks,
Bernd
Joanne Koong
maintainerQuoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.
Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/
Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.
Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.
Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.
Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
Miklos, what are your thoughts?
Thanks,
Joanne
Bernd Schubert
maintainerQuoted 116 lines · Joanne Koong
Quoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.
Quoted 2 lines
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
I think for B "at setup time" is not entirely correct - buffers in the current model buffers and entries can be registered at run time.
I have been thinking about approach C, which I think is in between A and B. I my mind approach C does the registration per entry, but payload buffers get added to a buf array, if a payload is set.
Max array size would be negotiated as in your approach or maybe better with a config SQE. Different IO size could achieved by multiple buf arrays.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 127 lines · Bernd Schubert
Quoted 116 lines · Joanne Koong
Quoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.Quoted 2 lines
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
I think for B "at setup time" is not entirely correct - buffers in the current model buffers and entries can be registered at run time.
Quoted 5 lines
I have been thinking about approach C, which I think is in between A and B. I my mind approach C does the registration per entry, but payload buffers get added to a buf array, if a payload is set.
Max array size would be negotiated as in your approach or maybe better with a config SQE. Different IO size could achieved by multiple buf arrays.
Ok, so it looks like we agree that entries should be decoupled from buffers. Now it looks like the remaining disagreement is how the buffers get passed in.
I still think it's simpler and cleaner to pass in the buffers as a contiguous pool upfront instead of passing in each buffer individually.
Miklos, what are your thoughts?
Thanks,
Joanne
Quoted 5 lines
Thanks,
Bernd
Bernd Schubert
maintainerQuoted 134 lines · Joanne Koong
Quoted 127 lines · Bernd Schubert
Quoted 116 lines · Joanne Koong
Quoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.Quoted 2 lines
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
I think for B "at setup time" is not entirely correct - buffers in the current model buffers and entries can be registered at run time.
Quoted 5 lines
I have been thinking about approach C, which I think is in between A and B. I my mind approach C does the registration per entry, but payload buffers get added to a buf array, if a payload is set.
Max array size would be negotiated as in your approach or maybe better with a config SQE. Different IO size could achieved by multiple buf arrays.
Quoted 3 lines
Ok, so it looks like we agree that entries should be decoupled from buffers. Now it looks like the remaining disagreement is how the buffers get passed in.
Well let's say, I don't see what speaks against decoupling it. The memory saving effect is due to the special form of different request sizes (0 and max payload).
Quoted 4 lines
I still think it's simpler and cleaner to pass in the buffers as a contiguous pool upfront instead of passing in each buffer individually.
My issue with this approach is still the same, incrementally registering memory when one needs it is hard. Example, I have a few external SSDs that are connected most of the time, but I only need them every few days. Why would I need to spend memory for the fuse transport all the time? The ideal situation would be: libfuse detects heavy usage and registers more entries and buffers.
Current fuse kernel supports it. I just didn't get to it to implement dynamic registration in libfuse yet - a matter of time. Missing is some form of kernel unregister, although easy to add, if unregister is treated as request OP.
With a contiguous buffer pool I wouldn't know how to register on demand and unregistering is even harder.
Also, if we still have races during register phase, doing it on demand wouldn't change anything in the current scheme - either the race is there or it is not, because current scheme is dynamic registration from kernel point of view.
No we need to keep that old scheme one way or the other, but we could deprecate one form and avoid adding new features (like different buffer sizes). However, if buffer pools have a limited usability, once needs to do those features for both approaches.
Quoted 2 lines
Miklos, what are your thoughts?
Just a short reminder that it was actually Miklos to request the dynamic registration way. Initially it was all static, taken from the ublk tag model. Making it dynamic added some headache, but in the end I also noticed that it can be useful.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 160 lines · Bernd Schubert
Quoted 134 lines · Joanne Koong
Quoted 127 lines · Bernd Schubert
Quoted 116 lines · Joanne Koong
Quoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.Quoted 2 lines
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
I think for B "at setup time" is not entirely correct - buffers in the current model buffers and entries can be registered at run time.
Quoted 5 lines
I have been thinking about approach C, which I think is in between A and B. I my mind approach C does the registration per entry, but payload buffers get added to a buf array, if a payload is set.
Max array size would be negotiated as in your approach or maybe better with a config SQE. Different IO size could achieved by multiple buf arrays.Quoted 3 lines
Ok, so it looks like we agree that entries should be decoupled from buffers. Now it looks like the remaining disagreement is how the buffers get passed in.
Well let's say, I don't see what speaks against decoupling it. The memory saving effect is due to the special form of different request sizes (0 and max payload).
Quoted 4 lines
I still think it's simpler and cleaner to pass in the buffers as a contiguous pool upfront instead of passing in each buffer individually.
My issue with this approach is still the same, incrementally registering memory when one needs it is hard. Example, I have a few external SSDs that are connected most of the time, but I only need them every few days. Why would I need to spend memory for the fuse transport all the time? The ideal situation would be: libfuse detects heavy usage and registers more entries and buffers.
Current fuse kernel supports it. I just didn't get to it to implement dynamic registration in libfuse yet - a matter of time. Missing is some form of kernel unregister, although easy to add, if unregister is treated as request OP.
Quoted 2 lines
With a contiguous buffer pool I wouldn't know how to register on demand and unregistering is even harder.
For the scenario you mentioned above, you could register/unregister a buffer pool within one queue or alternatively, register/unregister a new queue, depending on the workload. I think the details can be worked out once we agree on the core design, but I don't think it would be more complicated with contiguous buffer pools.
Quoted 13 lines
Also, if we still have races during register phase, doing it on demand wouldn't change anything in the current scheme - either the race is there or it is not, because current scheme is dynamic registration from kernel point of view.
No we need to keep that old scheme one way or the other, but we could deprecate one form and avoid adding new features (like different buffer sizes). However, if buffer pools have a limited usability, once needs to do those features for both approaches.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Miklos, what are your thoughts?
Just a short reminder that it was actually Miklos to request the dynamic registration way. Initially it was all static, taken from the ublk tag model. Making it dynamic added some headache, but in the end I also noticed that it can be useful.
I wasn't there when that conversation happened a few years ago, so I'm not sure if Miklos's stance still pertains here where ents are decoupled from buffers. I think we need Miklos to clarify this for us.
Thanks,
Joanne
Quoted 6 lines
Thanks,
Bernd
Joanne Koong
maintainerQuoted 191 lines · Joanne Koong
Quoted 160 lines · Bernd Schubert
Quoted 134 lines · Joanne Koong
Quoted 127 lines · Bernd Schubert
Quoted 116 lines · Joanne Koong
Quoted 63 lines · Joanne Koong
This series adds buffer pools (implemented internally as a buffer ring) and zero-copy to fuse io-uring.
Prior to this series, ents and buffers are tightly coupled where each entry has its own dedicated payload buffer, requiring N buffers for N entries where each buffer must be large enough to accomodate the maximum payload size. This is suboptimal as most request types require vastly less bytes than the maximum payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (internally managed as a buffer ring) that the kernel will use for servicing ents/requests. This decoupling reduces the memory usage requirements needed to use fuse-io-uring. It allows for future optimizations such as incremental buffer consumption and supporting multiple buffer sizes per ring.
This series also allows servers to use io-uring registered buffers as the buffer pool which eliminates the overhead of pinning/unpinning user pages and translating virtual addreses per i/o request.
This series additionally adds zero copy to fuse io-uring. The server can directly access client pages or page cache folios without copying data through an intermediary buffer. This requires
CAP_SYS_ADMINprivileges and using registered buffers.This series has a dependency on io-uring registered bvec buffers changes in [1].
The throughput improvements from registered buffers and zero-copy depends on how much of the server's per-request latency is spent on data copying vs backing I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of overall latency. Please also note that for the server to read/write into the zero-copied pages, the read/write must go through io-uring as an
IORING_OP_READ_FIXED/IORING_OP_WRITE_FIXEDoperation. If the server's backing I/O is instantaneous (eg served from cache), the overhead of the additionalio_uringoperation may negate the savings from eliminating the memcpy.In benchmarks using
passthrough_hpon a high-performance NVMe-backed system, using registered buffers showed around a 10% throughput improvement for direct randreads (~2150 MiB/s to ~2400 MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to ~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to ~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500 MiB/s to ~2670 MiB/s).Zero-copy showed around a 35% throughput improvement for direct randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see enough of a clear improvement for writes due to write latency being I/O dominated.
The benchmarks were run using:
fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1The libfuse changes can be found in [2]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough -oio_uring_zero_copy-o io_uring_q_depth=8 Once this series is merged, the libfuse changes will be tidied up and submitted upstream.Quoted 6 lines
Thanks,
Joanne[1] https://lore.kernel.org/io-uring/[email protected]/T/#t
[2] https://github.com/joannekoong/libfuse/commits/zero_copy_v3/Miklos,
This series has stalled on an architectural question and I'd appreciate your input on the direction.
Background:
In fuse io-uring, entries are reusable communication slots between the kernel and the daemon. When a request needs to be sent from the kernel to the daemon, it is assigned to an available entry. The entry carries the request to userspace, and the response comes back through the same entry. The number of entries determines how many requests can be serviced concurrently. Currently, when the daemon registers an entry during fuse io-uring setup, it passes in a payload buffer (sized to the maximum payload, typically 1 MiB) alongside it. The buffer is tied to the entry for the lifetime of the connection.Problem that needs to be solved:
Most request types (lookup, getattr, open, etc) require far less than the maximum payload size, and some (flush, fsync, etc) require no payload buffer at all. Every entry reserves a full 1 MiB buffer regardless of actual need. On systems with many entries, this memory cost scales linearly - for example, 64 entries across 8 queues requires 512 MiB of payload buffers alone.Approach A:
Decouple entries from payload buffers. Entries become generic - any entry can use any buffer from the pool. At setup time, the daemon passes in a contiguous pool of memory, and the kernel dynamically selects and recycles buffers from this pool as entries are assigned requests. Entries servicing requests with no payloads consume no buffer, and the pool can have fewer buffers than entries. This supports adding multi-size buffers and incremental buffer consumption (multiple entries using non-overlapping regions of the same buffer concurrently) as optimizations in the future.Approach B:
Keep the 1:1 entry-buffer coupling, but allow entries to be registered with different payload buffer sizes at setup time. The daemon would register entries with smaller buffers for metadata-heavy workloads and larger buffers for data-heavy ones. This keeps the existing model of passing in a payload buffer during entry registration at setup time and requires no uapi additions. Each entry registration adds buffer capacity whereas with buffer pools, buffer capacity would have to be added as a separate pool.Quoted 2 lines
I want to make sure I'm representing both approaches fairly - Bernd, please correct anything I've gotten wrong.
I think for B "at setup time" is not entirely correct - buffers in the current model buffers and entries can be registered at run time.
Quoted 5 lines
I have been thinking about approach C, which I think is in between A and B. I my mind approach C does the registration per entry, but payload buffers get added to a buf array, if a payload is set.
Max array size would be negotiated as in your approach or maybe better with a config SQE. Different IO size could achieved by multiple buf arrays.Quoted 3 lines
Ok, so it looks like we agree that entries should be decoupled from buffers. Now it looks like the remaining disagreement is how the buffers get passed in.
Well let's say, I don't see what speaks against decoupling it. The memory saving effect is due to the special form of different request sizes (0 and max payload).
Quoted 4 lines
I still think it's simpler and cleaner to pass in the buffers as a contiguous pool upfront instead of passing in each buffer individually.
My issue with this approach is still the same, incrementally registering memory when one needs it is hard. Example, I have a few external SSDs that are connected most of the time, but I only need them every few days. Why would I need to spend memory for the fuse transport all the time? The ideal situation would be: libfuse detects heavy usage and registers more entries and buffers.
Current fuse kernel supports it. I just didn't get to it to implement dynamic registration in libfuse yet - a matter of time. Missing is some form of kernel unregister, although easy to add, if unregister is treated as request OP.Quoted 2 lines
With a contiguous buffer pool I wouldn't know how to register on demand and unregistering is even harder.
For the scenario you mentioned above, you could register/unregister a buffer pool within one queue or alternatively, register/unregister a new queue, depending on the workload. I think the details can be worked out once we agree on the core design, but I don't think it would be more complicated with contiguous buffer pools.
Quoted 13 lines
Also, if we still have races during register phase, doing it on demand wouldn't change anything in the current scheme - either the race is there or it is not, because current scheme is dynamic registration from kernel point of view.
No we need to keep that old scheme one way or the other, but we could deprecate one form and avoid adding new features (like different buffer sizes). However, if buffer pools have a limited usability, once needs to do those features for both approaches.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Miklos, what are your thoughts?
Just a short reminder that it was actually Miklos to request the dynamic registration way. Initially it was all static, taken from the ublk tag model. Making it dynamic added some headache, but in the end I also noticed that it can be useful.
Quoted 7 lines
I wasn't there when that conversation happened a few years ago, so I'm not sure if Miklos's stance still pertains here where ents are decoupled from buffers. I think we need Miklos to clarify this for us.
Thanks,
Joanne
Miklos,
With entries decoupled from buffers, the remaining question is how buffers should be passed in:
Contiguously: daemon passes in a contiguous pool upfront
Per-buffer: daemon passes buffers in individually
In my opinion, passing in the buffers contiguously as a pool leads to simpler and cleaner logic at the fuse, io-uring, and userspace layers. If a daemon wants to dynamically add/remove buffer capacity in the future, they can add/remove another buffer pool or another ring, depending on their workload.
From Bernd's perspective, passing in each buffer individually allows daemons who want to dynamically add/remove buffer capacity in the future to add/remove capacity one buffer at a time based on load instead of needing to commit upfront to adding/removing a pool.
Could you please share your preference? This series is targeting the quickly approaching 7.2 merge window.
Thanks,
Joanne
Quoted 6 lines
Quoted 6 lines
Thanks,
Bernd
Miklos Szeredi
maintainerQuoted 6 lines · Joanne Koong
In my opinion, passing in the buffers contiguously as a pool leads to simpler and cleaner logic at the fuse, io-uring, and userspace layers. If a daemon wants to dynamically add/remove buffer capacity in the future, they can add/remove another buffer pool or another ring, depending on their workload.
Quoted 7 lines
From Bernd's perspective, passing in each buffer individually allows daemons who want to dynamically add/remove buffer capacity in the future to add/remove capacity one buffer at a time based on load instead of needing to commit upfront to adding/removing a pool.
Could you please share your preference? This series is targeting the quickly approaching 7.2 merge window.
I simply lack context to make a good decision right now. I can dive into this next week...
Thanks,
Miklos
Miklos Szeredi
maintainerQuoted 10 lines · Joanne Koong
In my opinion, passing in the buffers contiguously as a pool leads to simpler and cleaner logic at the fuse, io-uring, and userspace layers. If a daemon wants to dynamically add/remove buffer capacity in the future, they can add/remove another buffer pool or another ring, depending on their workload.
From Bernd's perspective, passing in each buffer individually allows daemons who want to dynamically add/remove buffer capacity in the future to add/remove capacity one buffer at a time based on load instead of needing to commit upfront to adding/removing a pool.
Isn't registering a single buffer a special case of registering a buffer pool?
If so, the question becomes how much complexity does the buffer pool add compared to the single buffer case.
Thanks,
Miklos
Joanne Koong
maintainerQuoted 9 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
In my opinion, passing in the buffers contiguously as a pool leads to simpler and cleaner logic at the fuse, io-uring, and userspace layers. If a daemon wants to dynamically add/remove buffer capacity in the future, they can add/remove another buffer pool or another ring, depending on their workload.
Quoted 6 lines
Quoted 4 lines
From Bernd's perspective, passing in each buffer individually allows daemons who want to dynamically add/remove buffer capacity in the future to add/remove capacity one buffer at a time based on load instead of needing to commit upfront to adding/removing a pool.
Isn't registering a single buffer a special case of registering a buffer pool?
Yes.
Quoted 3 lines
If so, the question becomes how much complexity does the buffer pool add compared to the single buffer case.
Internally fuse would manage the buffers as a pool regardless of whether they're passed in at registration time individually as single buffers or as a contiguous pool. This is so that entries (reusable communication slots between kernel and server) can be dynamically assigned a buffer depending on the payload size of a request, instead of being permanently tied to a fixed-size buffer (more details about this are in this thread [1]).
A contiguous pool makes things less complex but individual buffers can't be removed from it. With the single buffer approach, the buffers don't have to be contiguous, which means they can be unregistered and freed individually.
I think a contiguous pool makes more sense for a few reasons:
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
b) it makes things simpler at the native io-uring layer. With io-uring fixed buffers (which bypass per-i/o pinning overhead - benchmarks showed up to ~10% throughput improvement for fuse reads), userspace can register the contiguous pool into one slot. With single buffers, each buffer would need its own slot. With a single contiguous pool, fuse can reuse the existing fixed-buffer import mechanism with no new io-uring apis.
c) the userspace api is more ergonomic. Instead of registering X buffers individually where each buffer has to be passed in alongside an sqe, you can just use one sqe to register all X buffers at once.
This is my perspective on it and I'm trying to be unbiased but obviously there's probably some bias seeping in. If there's anything you want to add to this Bernd, please do.
Thanks,
Joanne
Miklos Szeredi
maintainerQuoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.
Am I missing something?
Thanks,
Miklos
Joanne Koong
maintainerQuoted 14 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Quoted 5 lines
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.Am I missing something?
The interface is a superset. Internally the difference would be X bufpools with a single buffer vs 1 bufpool with X single buffers. There would be more overhead for X bufpools (eg allocating X fuse_bufpools structs, looking across more bufpools to find a free buffer). I don't think it would add much additional complexity, given that multi-bufpool support would also be needed if we wanted to accommodate different buffer sizes in the same ring.
Does that answer your question or am I misreading what you're asking?
Thanks,
Joanne
Quoted 3 lines
Thanks,
Miklos
Miklos Szeredi
maintainerQuoted 20 lines · Joanne Koong
Quoted 18 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.
Quoted 7 lines
Quoted 1 line
Am I missing something?
The interface is a superset. Internally the difference would be X bufpools with a single buffer vs 1 bufpool with X single buffers. There would be more overhead for X bufpools (eg allocating X
fuse_bufpoolsstructs, looking across more bufpools to find a free buffer).
I don't think this is an issue. Allocation should be as simple as keeping per-ring freelist for each buffer size that would have all free buffers across an arbitrary number of buf pools.
I don't think it would add much additional complexity, given
Quoted 2 lines
that multi-bufpool support would also be needed if we wanted to accommodate different buffer sizes in the same ring.
To conclude: bufpools allow allocating/freeing multiple buffers together. The API is a superset, since allocating/freeing a single
buffer is still possible. Additional complexity or performance for
the bufpool layer should not be significant.
Is that correct?
If yes, I'd go with bufpools.
Thanks,
Miklos
Joanne Koong
maintainerQuoted 38 lines · Miklos Szeredi
Quoted 20 lines · Joanne Koong
Quoted 18 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.Quoted 7 lines
Quoted 1 line
Am I missing something?
The interface is a superset. Internally the difference would be X bufpools with a single buffer vs 1 bufpool with X single buffers. There would be more overhead for X bufpools (eg allocating X
fuse_bufpoolsstructs, looking across more bufpools to find a free buffer).I don't think this is an issue. Allocation should be as simple as keeping per-ring freelist for each buffer size that would have all free buffers across an arbitrary number of buf pools.
I don't think it would add much additional complexity, given
Quoted 2 lines
that multi-bufpool support would also be needed if we wanted to accommodate different buffer sizes in the same ring.
Quoted 6 lines
To conclude: bufpools allow allocating/freeing multiple buffers together. The API is a superset, since allocating/freeing a single
buffer is still possible. Additional complexity or performance for
the bufpool layer should not be significant.Is that correct?
That's my understanding of it as well.
Quoted 2 lines
If yes, I'd go with bufpools.
I'll send out a v4 that adds Baokun's Reviewed-bys, fixes the documentation typo, and adds a note to the zero-copy commit message about abort teardown.
Thanks,
Joanne
Bernd Schubert
maintainerQuoted 21 lines · Joanne Koong
Quoted 14 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Quoted 5 lines
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.Am I missing something?
Quoted 7 lines
The interface is a superset. Internally the difference would be X bufpools with a single buffer vs 1 bufpool with X single buffers. There would be more overhead for X bufpools (eg allocating X
fuse_bufpoolsstructs, looking across more bufpools to find a free buffer). I don't think it would add much additional complexity, given that multi-bufpool support would also be needed if we wanted to accommodate different buffer sizes in the same ring.
Really sorry for late replies.
Couldn't we make a config SQE define a max array size and then every array entry would be something like
struct buf_entry {
char *payload;
int buf_pool_idx; /* which buf pool */
int idx; /* index within the buf pool */
}
And then in fuse_uring_ent_in_out you would need to set buf_pool_idx and buf_idx
Avoids which buf pool still has free entries, but buf pools get added to that array, which then would have its own head and tail. Obviously you would need to do another pointer lookup to get the actual payload.
Obviously has one more pointer than just a single buf ring.
Thanks,
Bernd
Joanne Koong
maintainerQuoted 32 lines · Bernd Schubert
Quoted 21 lines · Joanne Koong
Quoted 14 lines · Miklos Szeredi
Quoted 6 lines · Joanne Koong
a) I don't see a compelling use case for needing to dynamically add/remove individual buffers from an existing pool. If the server needs more or less capacity, adding/removing a whole pool or queue seems more practical. If we do eventually need single-buffer additions to an existing pool, that can still be layered on to contiguous pools (but removing buffers can't).
If server knows that it will want to remove individual buffers, it will just have to register them as separate pools (containing a single buffer), right?
Quoted 5 lines
Buffer pools are a more generic concept, hence they come with some
extra complexity and overhead. But the interface should be a
superset of what individual buffers provide.Am I missing something?
Quoted 7 lines
The interface is a superset. Internally the difference would be X bufpools with a single buffer vs 1 bufpool with X single buffers. There would be more overhead for X bufpools (eg allocating X
fuse_bufpoolsstructs, looking across more bufpools to find a free buffer). I don't think it would add much additional complexity, given that multi-bufpool support would also be needed if we wanted to accommodate different buffer sizes in the same ring.
Quoted 4 lines
Really sorry for late replies.
Couldn't we make a config SQE define a max array size and then every array entry would be something like
I think the max-size array works but if it's backed by a linked list instead of a fixed array, bufpool add/remove can be more dynamic (eg no fixed ceiling). I think this is the freelist concept Miklos was talking about. I don't think the implementation details really matter right now though - all the kernel-internal stuff could be changed and figured out after this series.
The uapi changes this series adds are forward-compatible with multi-bufpools. It adds a uint16_t buf_id to fuse_uring_ent_in_out, and there's already a uint16_t padding field right after it that a pool_id can occupy later.
Thanks,
Joanne
Quoted 21 lines
struct
buf_entry{
char *payload;
intbuf_pool_idx; /* which buf pool */
int idx; /* index within the buf pool */
}And then in
fuse_uring_ent_in_outyou would need to setbuf_pool_idxandbuf_idxAvoids which buf pool still has free entries, but buf pools get added to that array, which then would have its own head and tail. Obviously you would need to do another pointer lookup to get the actual payload.
Obviously has one more pointer than just a single buf ring.
Thanks,
Bernd