Skip to contentSkip to conversation

fuse: support io-uring registered buffers

PATCH In review Joanne Koong 5 participants 40 messages Oct 27, 2025 – Nov 25, 2025

Subsystems (4)
lore
Share & tools
lore

Joanne Koong

author maintainer
#0 lore raw

v2 cover fuse: support io-uring registered buffers

This patchset adds fuse support for io-uring registered buffers. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

The main logic for fuse registered buffers is in the last patch (patch 8/8). Patch 1/8 adds an io_uring api for fetching the registered buffer and patches (2-7)/8 refactors the fuse io_uring code, which additionally will make adding in the logic for registered buffers neater.

The libfuse changes can be found in this branch:
https://github.com/joannekoong/libfuse/tree/registered_buffers. The libfuse implementation first tries registered buffers during registration and if this fails, will retry with non-registered buffers. This prevents having to add a new init flag (but does have the downside of printing dmesg errors for the failed registrations when trying the registered buffers). If using registered buffers and the daemon for whatever reason unregisters the buffers midway through, then this will sever server-kernel communication. Libfuse will never do this. Libfuse will only unregister the buffers when the entire session is being destroyed.

Benchmarks will be run and posted.

Thanks,
Joanne

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Add io_uring_cmd_import_fixed_full() patch
* Construct iter using io_uring_cmd_import_fixed_full() per cmd instead of recyling iters.
* Kmap the header instead of using bvec iter for iterating/copying. This makes the code easier to read.

Joanne Koong (8):
io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: add user_ prefix to userspace headers and payload fields
fuse: refactor setting up copy state for payload copying
fuse: support io-uring registered buffers

 fs/fuse/dev_uring.c          | 366 +++++++++++++++++++++++++----------
 fs/fuse/dev_uring_i.h        |  27 ++-
 include/linux/io_uring/cmd.h |   3 +
 io_uring/rsrc.c              |  14 ++
 io_uring/rsrc.h              |   2 +
 io_uring/uring_cmd.c         |  13 ++
 6 files changed, 316 insertions(+), 109 deletions(-)
Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH v2 1/8 io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

---
 include/linux/io_uring/cmd.h |  3 +++
 io_uring/rsrc.c              | 14 ++++++++++++++
 io_uring/rsrc.h              |  2 ++
 io_uring/uring_cmd.c         | 13 +++++++++++++
 4 files changed, 32 insertions(+)
include/linux/io_uring/cmd.h +3 -0
index 7509025b4071..8c11d9a92733 100644
@@ -43,6 +43,9 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
 			      struct iov_iter *iter,
 			      struct io_uring_cmd *ioucmd,
 			      unsigned int issue_flags);
+int io_uring_cmd_import_fixed_full(int rw, struct iov_iter *iter,
+				   struct io_uring_cmd *ioucmd,
+				   unsigned int issue_flags);
 int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
 				  const struct iovec __user *uvec,
 				  size_t uvec_segs,
io_uring/rsrc.c +14 -0
index d787c16dc1c3..2c3d8489ae52 100644
@@ -1147,6 +1147,20 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
 	return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
 }
 
+int io_import_reg_buf_full(struct io_kiocb *req, struct iov_iter *iter,
+			   int ddir, unsigned issue_flags)
+{
+	struct io_rsrc_node *node;
+	struct io_mapped_ubuf *imu;
+
+	node = io_find_buf_node(req, issue_flags);
+	if (!node)
+		return -EFAULT;
+
+	imu = node->buf;
+	return io_import_fixed(ddir, iter, imu, imu->ubuf, imu->len);
+}
+
 /* Lock two rings at once. The rings must be different! */
 static void lock_two_rings(struct io_ring_ctx *ctx1, struct io_ring_ctx *ctx2)
 {
io_uring/rsrc.h +2 -0
index a3ca6ba66596..4e01eb0f277e 100644
@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
 int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
 			u64 buf_addr, size_t len, int ddir,
 			unsigned issue_flags);
+int io_import_reg_buf_full(struct io_kiocb *req, struct iov_iter *iter,
+			   int ddir, unsigned issue_flags);
 int io_import_reg_vec(int ddir, struct iov_iter *iter,
 			struct io_kiocb *req, struct iou_vec *vec,
 			unsigned nr_iovs, unsigned issue_flags);
io_uring/uring_cmd.c +13 -0
index d1e3ba62ee8e..07730ced9449 100644
@@ -292,6 +292,19 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
 }
 EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed);
 
+int io_uring_cmd_import_fixed_full(int rw, struct iov_iter *iter,
+				   struct io_uring_cmd *ioucmd,
+				   unsigned int issue_flags)
+{
+	struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
+
+	if (WARN_ON_ONCE(!(ioucmd->flags & IORING_URING_CMD_FIXED)))
+		return -EINVAL;
+
+	return io_import_reg_buf_full(req, iter, rw, issue_flags);
+}
+EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed_full);
+
 int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
 				  const struct iovec __user *uvec,
 				  size_t uvec_segs,
Signature · 1 line
2.47.3
Reviewed-by on v2 1/8
#2 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 37 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

---
 include/linux/io_uring/cmd.h |  3 +++
 io_uring/rsrc.c              | 14 ++++++++++++++
 io_uring/rsrc.h              |  2 ++
 io_uring/uring_cmd.c         | 13 +++++++++++++
 4 files changed, 32 insertions(+)
include/linux/io_uring/cmd.h +3 -0
index 7509025b4071..8c11d9a92733 100644
@@ -43,6 +43,9 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
                              struct iov_iter *iter,
                              struct io_uring_cmd *ioucmd,
                              unsigned int issue_flags);
+int io_uring_cmd_import_fixed_full(int rw, struct iov_iter *iter,
+                                  struct io_uring_cmd *ioucmd,
+                                  unsigned int issue_flags);
 int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
                                  const struct iovec __user *uvec,
                                  size_t uvec_segs,
io_uring/rsrc.c +2 -0
index d787c16dc1c3..2c3d8489ae52 100644
@@ -1147,6 +1147,20 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
        return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
 }
 
+int io_import_reg_buf_full(struct io_kiocb *req, struct iov_iter *iter,
+                          int ddir, unsigned issue_flags)

io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1161

Quoted 10 lines
io_uring/rsrc.c +10 -0
+{
+       struct io_rsrc_node *node;
+       struct io_mapped_ubuf *imu;
+
+       node = io_find_buf_node(req, issue_flags);
+       if (!node)
+               return -EFAULT;
+
+       imu = node->buf;
+       return io_import_fixed(ddir, iter, imu, imu->ubuf, imu->len);

It's probably possible to avoid the logic in io_import_fixed() for checking the user address range against the registered buffer's range and offsetting the iov_iter, but that could always be done as future optimization work.

Quoted 45 lines
io_uring/rsrc.c +2 -0
+}
+
 /* Lock two rings at once. The rings must be different! */
 static void lock_two_rings(struct io_ring_ctx *ctx1, struct io_ring_ctx *ctx2)
 {
io_uring/rsrc.h +2 -0
index a3ca6ba66596..4e01eb0f277e 100644
@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
 int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
                        u64 buf_addr, size_t len, int ddir,
                        unsigned issue_flags);
+int io_import_reg_buf_full(struct io_kiocb *req, struct iov_iter *iter,
+                          int ddir, unsigned issue_flags);
 int io_import_reg_vec(int ddir, struct iov_iter *iter,
                        struct io_kiocb *req, struct iou_vec *vec,
                        unsigned nr_iovs, unsigned issue_flags);
io_uring/uring_cmd.c +13 -1
index d1e3ba62ee8e..07730ced9449 100644
@@ -292,6 +292,19 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
 }
 EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed);
 
+int io_uring_cmd_import_fixed_full(int rw, struct iov_iter *iter,
+                                  struct io_uring_cmd *ioucmd,
+                                  unsigned int issue_flags)
+{
+       struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
+
+       if (WARN_ON_ONCE(!(ioucmd->flags & IORING_URING_CMD_FIXED)))
+               return -EINVAL;
+
+       return io_import_reg_buf_full(req, iter, rw, issue_flags);
+}
+EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed_full);
+
 int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
                                  const struct iovec __user *uvec,
                                  size_t uvec_segs,
--

2.47.3

#3 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

In reply to Joanne Koong
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Signature · 1 line
Pavel Begunkov
author maintainer
#4 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.

Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return -EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.

Thanks,
Joanne

Quoted 3 lines

--
Pavel Begunkov

#5 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 6 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.

I don't think it would be a huge issue to let userspace allocate a large buffer and to distribute it among requests - there is nothing in the kernel side to be done for that?
(I think I had even done that for the 1st io-uring patches and removed it because there were other issues and I wanted to keep the initial code simple).

Thanks,
Bernd

#6 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 31 lines · Bernd Schubert
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 6 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.

Quoted 3 lines

I don't think it would be a huge issue to let userspace allocate a large buffer and to distribute it among requests - there is nothing in the kernel side to be done for that?

You can, but unless I missed something with this patchset you'd need to register it N times, which is not terrible but have memory overaccounting problems and feels less flexible.

Quoted 3 lines

(I think I had even done that for the 1st io-uring patches and removed it because there were other issues and I wanted to keep the initial code simple).

Signature · 1 line
Pavel Begunkov
#7 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

In reply to Joanne Koong
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

In either case, I don't have a problem with this patch, just found it a bit off.

Quoted 3 lines

My worry with this is that it would lead to
suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit

It wouldn't affect locality of the user buffers, that depends on the user space implementation. Are you sharing an io_uring instance between threads?

Quoted 14 lines

simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.

Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return -EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.

The devil is in details, i.e. synchronisation. Taking a long term node reference might be fine. Does this change the uapi for this patchset? If not, I'd do it as a follow up. It also sounds like you can apply this optimisation regardless of whether you take a full registered buffer or go with sub ranges.

Signature · 1 line
Pavel Begunkov
#8 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

Quoted 2 lines

In either case, I don't have a problem with this patch, just found it a bit off.

Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.

The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.

Thanks,
Bernd

author maintainer
#9 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 46 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

Quoted 2 lines

In either case, I don't have a problem with this patch, just found it a bit off.

Quoted 8 lines

Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.

The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.

With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to io_uring_prep_rw() and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).

Thanks,
Joanne

Quoted 2 lines

Thanks,
Bernd

#10 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 56 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

Quoted 2 lines

In either case, I don't have a problem with this patch, just found it a bit off.

Quoted 8 lines

Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.

The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.

Quoted 5 lines

With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to io_uring_prep_rw() and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).

Ah that is good to know, maybe worth to be mentioned to the commit message.

Btw, I will start to work on libfuse around next week to add another io-uring interface, so that the application can own the ring and let libfuse submit and fetch from it. I.e. that way the same ring can be used for libfuse and application IO.

Thanks,
Bernd

author maintainer
#11 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

Quoted 56 lines · Bernd Schubert
Quoted 54 lines · Joanne Koong
Quoted 52 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

Quoted 2 lines

In either case, I don't have a problem with this patch, just found it a bit off.

Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.

The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.

Quoted 7 lines
Quoted 5 lines

With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to io_uring_prep_rw() and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).

Ah that is good to know, maybe worth to be mentioned to the commit message.

Will do. I will add this to the commit message.

Quoted 5 lines

Btw, I will start to work on libfuse around next week to add another io-uring interface, so that the application can own the ring and let libfuse submit and fetch from it. I.e. that way the same ring can be used for libfuse and application IO.

Sounds great! Looking forward to the changes.

Thanks,
Joanne

Quoted 3 lines

Thanks,
Bernd

author maintainer
#12 lore raw

Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()

In reply to Pavel Begunkov
Quoted 28 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong

Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.

Quoted 6 lines

Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.

Hi Pavel,

Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).

Quoted 2 lines

Do you think it is better to allocate one large buffer and have the requests use subchunks?

Quoted 11 lines

I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.

Thanks for your thoughts, I'll think about this some more.

Quoted 11 lines

In either case, I don't have a problem with this patch, just found it a bit off.

Quoted 3 lines

My worry with this is that it would lead to
suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit

It wouldn't affect locality of the user buffers, that depends on the user space implementation. Are you sharing an io_uring instance between threads?

For request offloading, the different threads would share the io uring instance. When the main thread that does the io_uring_wait_cqe() receives a cqe, it'll dispatch the cqe to a worker thread to fulfill while it then looks at the next cqe to send off to the next worker thread and so on.

If there's one registered buffer that maps to the one big buffer allocated by the server for the ring, then each worker thread might be executing on different cpus when it accesses that buffer, which seems like that could lead to cache line bouncing. Or at least that's the scenario I was thinking of for suboptimal cache locality with the one big buffer. But I do like how a big buffer would save on the memory overhead that would be internally incurred in iouring for tracking every registered buffer.

Quoted 15 lines
Quoted 4 lines

simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.

Quoted 11 lines

Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return -EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.

Quoted 5 lines

The devil is in details, i.e. synchronisation. Taking a long term node reference might be fine. Does this change the uapi for this patchset? If not, I'd do it as a follow up. It also sounds like you can apply this optimisation regardless of whether you take a full registered buffer or go with sub ranges.

Thanks, this is helpful.

Quoted 4 lines

--
Pavel Begunkov

author maintainer
#13 lore raw

PATCH v2 2/8 fuse: refactor io-uring logic for getting next fuse request

In reply to Joanne Koong

Simplify the logic for getting the next fuse request.

---
 fs/fuse/dev_uring.c | 78 ++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
index f6b12aebb8bb..415924b346c0 100644
@@ -710,34 +710,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 	return err;
 }
 
-/*
- * Write data to the ring buffer and send the request to userspace,
- * userspace will read it
- * This is comparable with classical read(/dev/fuse)
- */
-static int fuse_uring_send_next_to_ring(struct fuse_ring_ent *ent,
-					struct fuse_req *req,
-					unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-	int err;
-	struct io_uring_cmd *cmd;
-
-	err = fuse_uring_prepare_send(ent, req);
-	if (err)
-		return err;
-
-	spin_lock(&queue->lock);
-	cmd = ent->cmd;
-	ent->cmd = NULL;
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, 0, issue_flags);
-	return 0;
-}
-
 /*
  * Make a ring entry available for fuse_req assignment
  */
@@ -834,11 +806,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 }
 
 /*
- * Get the next fuse req and send it
+ * Get the next fuse req.
+ *
+ * Returns true if the next fuse request has been assigned to the ent.
+ * Else, there is no next fuse request and this returns false.
  */
-static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
-				     struct fuse_ring_queue *queue,
-				     unsigned int issue_flags)
+static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
+					 struct fuse_ring_queue *queue)
 {
 	int err;
 	struct fuse_req *req;
@@ -850,10 +824,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
 	spin_unlock(&queue->lock);
 
 	if (req) {
-		err = fuse_uring_send_next_to_ring(ent, req, issue_flags);
+		err = fuse_uring_prepare_send(ent, req);
 		if (err)
 			goto retry;
 	}
+
+	return req != NULL;
 }
 
 static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
@@ -871,6 +847,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
 	return 0;
 }
 
+static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
+			    ssize_t ret, unsigned int issue_flags)
+{
+	struct fuse_ring_queue *queue = ent->queue;
+
+	spin_lock(&queue->lock);
+	ent->state = FRRS_USERSPACE;
+	list_move_tail(&ent->list, &queue->ent_in_userspace);
+	ent->cmd = NULL;
+	spin_unlock(&queue->lock);
+
+	io_uring_cmd_done(cmd, ret, issue_flags);
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -942,7 +932,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	fuse_uring_next_fuse_req(ent, queue, issue_flags);
+	if (fuse_uring_get_next_fuse_req(ent, queue))
+		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
 
@@ -1190,20 +1181,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
 	return -EIOCBQUEUED;
 }
 
-static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
-			    ssize_t ret, unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-
-	spin_lock(&queue->lock);
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	ent->cmd = NULL;
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, ret, issue_flags);
-}
-
 /*
  * This prepares and sends the ring request in fuse-uring task context.
  * User buffers are not mapped yet - the application does not have permission
@@ -1219,8 +1196,9 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 	if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
 		err = fuse_uring_prepare_send(ent, ent->fuse_req);
 		if (err) {
-			fuse_uring_next_fuse_req(ent, queue, issue_flags);
-			return;
+			if (!fuse_uring_get_next_fuse_req(ent, queue))
+				return;
+			err = 0;
 		}
 	} else {
 		err = -ECANCELED;
Signature · 1 line
2.47.3
Reviewed-by on v2 2/8
#14 lore raw

Re: [PATCH v2 2/8] fuse: refactor io-uring logic for getting next fuse request

Quoted 132 lines · Joanne Koong

Simplify the logic for getting the next fuse request.

---
 fs/fuse/dev_uring.c | 78 ++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +25 -48
index f6b12aebb8bb..415924b346c0 100644
@@ -710,34 +710,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 	return err;
 }
 
-/*
- * Write data to the ring buffer and send the request to userspace,
- * userspace will read it
- * This is comparable with classical read(/dev/fuse)
- */
-static int fuse_uring_send_next_to_ring(struct fuse_ring_ent *ent,
-					struct fuse_req *req,
-					unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-	int err;
-	struct io_uring_cmd *cmd;
-
-	err = fuse_uring_prepare_send(ent, req);
-	if (err)
-		return err;
-
-	spin_lock(&queue->lock);
-	cmd = ent->cmd;
-	ent->cmd = NULL;
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, 0, issue_flags);
-	return 0;
-}
-
 /*
  * Make a ring entry available for fuse_req assignment
  */
@@ -834,11 +806,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 }
 
 /*
- * Get the next fuse req and send it
+ * Get the next fuse req.
+ *
+ * Returns true if the next fuse request has been assigned to the ent.
+ * Else, there is no next fuse request and this returns false.
  */
-static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
-				     struct fuse_ring_queue *queue,
-				     unsigned int issue_flags)
+static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
+					 struct fuse_ring_queue *queue)
 {
 	int err;
 	struct fuse_req *req;
@@ -850,10 +824,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,
 	spin_unlock(&queue->lock);
 
 	if (req) {
-		err = fuse_uring_send_next_to_ring(ent, req, issue_flags);
+		err = fuse_uring_prepare_send(ent, req);
 		if (err)
 			goto retry;
 	}
+
+	return req != NULL;
 }
 
 static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
@@ -871,6 +847,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)
 	return 0;
 }
 
+static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
+			    ssize_t ret, unsigned int issue_flags)
+{
+	struct fuse_ring_queue *queue = ent->queue;
+
+	spin_lock(&queue->lock);
+	ent->state = FRRS_USERSPACE;
+	list_move_tail(&ent->list, &queue->ent_in_userspace);
+	ent->cmd = NULL;
+	spin_unlock(&queue->lock);
+
+	io_uring_cmd_done(cmd, ret, issue_flags);
+}
+
 /* FUSE_URING_CMD_COMMIT_AND_FETCH handler */
 static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 				   struct fuse_conn *fc)
@@ -942,7 +932,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	fuse_uring_next_fuse_req(ent, queue, issue_flags);
+	if (fuse_uring_get_next_fuse_req(ent, queue))
+		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
 
@@ -1190,20 +1181,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
 	return -EIOCBQUEUED;
 }
 
-static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd,
-			    ssize_t ret, unsigned int issue_flags)
-{
-	struct fuse_ring_queue *queue = ent->queue;
-
-	spin_lock(&queue->lock);
-	ent->state = FRRS_USERSPACE;
-	list_move_tail(&ent->list, &queue->ent_in_userspace);
-	ent->cmd = NULL;
-	spin_unlock(&queue->lock);
-
-	io_uring_cmd_done(cmd, ret, issue_flags);
-}
-
 /*
  * This prepares and sends the ring request in fuse-uring task context.
  * User buffers are not mapped yet - the application does not have permission
@@ -1219,8 +1196,9 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 	if (!(issue_flags & IO_URING_F_TASK_DEAD)) {

fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1204

Quoted 10 lines
fs/fuse/dev_uring.c +3 -2
 		err = fuse_uring_prepare_send(ent, ent->fuse_req);
 		if (err) {
-			fuse_uring_next_fuse_req(ent, queue, issue_flags);
-			return;
+			if (!fuse_uring_get_next_fuse_req(ent, queue))
+				return;
+			err = 0;
 		}
 	} else {
 		err = -ECANCELED;

LGTM

(I had already start to go into this direction
https://lore.kernel.org/all/[email protected]/)

author maintainer
#15 lore raw

PATCH v2 3/8 fuse: refactor io-uring header copying to ring

In reply to Joanne Koong

Move header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------
 1 file changed, 20 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +20 -18
index 415924b346c0..e94af90d4d46 100644
@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static int copy_header_to_ring(void __user *ring, const void *header,
+			       size_t header_size)
+{
+	if (copy_to_user(ring, header, header_size)) {
+		pr_info_ratelimited("Copying header to ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_to_user(&ent->headers->op_in, in_args->value,
-					   in_args->size);
-			if (err) {
-				pr_info_ratelimited(
-					"Copying the header failed.\n");
-				return -EFAULT;
-			}
+			err = copy_header_to_ring(&ent->headers->op_in,
+						  in_args->value,
+						  in_args->size);
+			if (err)
+				return err;
 		}
 		in_args++;
 		num_args--;
@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	err = copy_to_user(&ent->headers->ring_ent_in_out, &ent_in_out,
-			   sizeof(ent_in_out));
-	return err ? -EFAULT : 0;
+	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+				   sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	err = copy_to_user(&ent->headers->in_out, &req->in.h,
-			   sizeof(req->in.h));
-	if (err) {
-		err = -EFAULT;
-		return err;
-	}
-
-	return 0;
+	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+				   sizeof(req->in.h));
 }
 
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.47.3
#16 lore raw

Re: [PATCH v2 3/8] fuse: refactor io-uring header copying to ring

Quoted 66 lines · Joanne Koong

Move header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------
 1 file changed, 20 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +18 -11
index 415924b346c0..e94af90d4d46 100644
@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static int copy_header_to_ring(void __user *ring, const void *header,
+			       size_t header_size)
+{
+	if (copy_to_user(ring, header, header_size)) {
+		pr_info_ratelimited("Copying header to ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_to_user(&ent->headers->op_in, in_args->value,
-					   in_args->size);
-			if (err) {
-				pr_info_ratelimited(
-					"Copying the header failed.\n");
-				return -EFAULT;
-			}
+			err = copy_header_to_ring(&ent->headers->op_in,
+						  in_args->value,
+						  in_args->size);
+			if (err)
+				return err;
 		}
 		in_args++;
 		num_args--;
@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	err = copy_to_user(&ent->headers->ring_ent_in_out, &ent_in_out,
-			   sizeof(ent_in_out));
-	return err ? -EFAULT : 0;
+	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+				   sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	err = copy_to_user(&ent->headers->in_out, &req->in.h,

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 699

Quoted 10 lines
fs/fuse/dev_uring.c +2 -7
-			   sizeof(req->in.h));
-	if (err) {
-		err = -EFAULT;
-		return err;
-	}
-
-	return 0;
+	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+				   sizeof(req->in.h));
 }

This will give Miklos a bit headache, because of a merge conflict with

https://lore.kernel.org/r/[email protected]

Any chance you could rebase your series on this patch?

Thanks,
Bernd

Quoted 2 lines
fs/fuse/dev_uring.c +0 -0
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
author maintainer
#17 lore raw

Re: [PATCH v2 3/8] fuse: refactor io-uring header copying to ring

Quoted 81 lines · Bernd Schubert
Quoted 66 lines · Joanne Koong

Move header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------
 1 file changed, 20 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +18 -11
index 415924b346c0..e94af90d4d46 100644
@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
      return err;
 }
 
+static int copy_header_to_ring(void __user *ring, const void *header,
+                            size_t header_size)
+{
+     if (copy_to_user(ring, header, header_size)) {
+             pr_info_ratelimited("Copying header to ring failed.\n");
+             return -EFAULT;
+     }
+
+     return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
                                   struct fuse_req *req,
                                   struct fuse_ring_ent *ent)
@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
               * Some op code have that as zero size.
               */
              if (args->in_args[0].size > 0) {
-                     err = copy_to_user(&ent->headers->op_in, in_args->value,
-                                        in_args->size);
-                     if (err) {
-                             pr_info_ratelimited(
-                                     "Copying the header failed.\n");
-                             return -EFAULT;
-                     }
+                     err = copy_header_to_ring(&ent->headers->op_in,
+                                               in_args->value,
+                                               in_args->size);
+                     if (err)
+                             return err;
              }
              in_args++;
              num_args--;
@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
      }
 
      ent_in_out.payload_sz = cs.ring.copied_sz;
-     err = copy_to_user(&ent->headers->ring_ent_in_out, &ent_in_out,
-                        sizeof(ent_in_out));
-     return err ? -EFAULT : 0;
+     return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+                                sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
      }
 
      /* copy fuse_in_header */
-     err = copy_to_user(&ent->headers->in_out, &req->in.h,

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 699

Quoted 10 lines
fs/fuse/dev_uring.c +2 -7
-                        sizeof(req->in.h));
-     if (err) {
-             err = -EFAULT;
-             return err;
-     }
-
-     return 0;
+     return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+                                sizeof(req->in.h));
 }

This will give Miklos a bit headache, because of a merge conflict with

Quoted 3 lines

https://lore.kernel.org/r/[email protected]

Any chance you could rebase your series on this patch?

Definitely, I will do that for v3. I saw that you submitted a bunch of io-uring patches earlier last week alongside this one - I haven't gotten around to reviewing those but I will do so.

Thanks,
Joanne

Quoted 8 lines

Thanks,
Bernd

Quoted 2 lines
fs/fuse/dev_uring.c +0 -0
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
author maintainer
#18 lore raw

PATCH v2 4/8 fuse: refactor io-uring header copying from ring

In reply to Joanne Koong

Move header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.

---
 fs/fuse/dev_uring.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +17 -6
index e94af90d4d46..faa7217e85c4 100644
@@ -585,6 +585,17 @@ static int copy_header_to_ring(void __user *ring, const void *header,
 	return 0;
 }
 
+static int copy_header_from_ring(void *header, const void __user *ring,
+				 size_t header_size)
+{
+	if (copy_from_user(header, ring, header_size)) {
+		pr_info_ratelimited("Copying header from ring failed.\n");
+		return -EFAULT;
+	}
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -595,10 +606,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_from_user(&ring_in_out, &ent->headers->ring_ent_in_out,
-			     sizeof(ring_in_out));
+	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
+				    sizeof(ring_in_out));
 	if (err)
-		return -EFAULT;
+		return err;
 
 	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
 			  &iter);
@@ -789,10 +800,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 
-	err = copy_from_user(&req->out.h, &ent->headers->in_out,
-			     sizeof(req->out.h));
+	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+				    sizeof(req->out.h));
 	if (err) {
-		req->out.h.error = -EFAULT;
+		req->out.h.error = err;
 		goto out;
 	}
 
Signature · 1 line
2.47.3
author maintainer
#19 lore raw

PATCH v2 5/8 fuse: use enum types for header copying

In reply to Joanne Koong

Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.

---
 fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 45 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +45 -10
index faa7217e85c4..d96368e93e8d 100644
@@ -31,6 +31,12 @@ struct fuse_uring_pdu {
 
 static const struct fuse_iqueue_ops fuse_io_uring_ops;
 
+enum fuse_uring_header_type {
+	FUSE_URING_HEADER_IN_OUT,
+	FUSE_URING_HEADER_OP,
+	FUSE_URING_HEADER_RING_ENT,
+};
+
 static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
 				   struct fuse_ring_ent *ring_ent)
 {
@@ -574,9 +580,31 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
-static int copy_header_to_ring(void __user *ring, const void *header,
-			       size_t header_size)
+static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
+					 enum fuse_uring_header_type type)
+{
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		return &ent->headers->in_out;
+	case FUSE_URING_HEADER_OP:
+		return &ent->headers->op_in;
+	case FUSE_URING_HEADER_RING_ENT:
+		return &ent->headers->ring_ent_in_out;
+	}
+
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
+static int copy_header_to_ring(struct fuse_ring_ent *ent,
+			       enum fuse_uring_header_type type,
+			       const void *header, size_t header_size)
 {
+	void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_to_user(ring, header, header_size)) {
 		pr_info_ratelimited("Copying header to ring failed.\n");
 		return -EFAULT;
@@ -585,9 +613,15 @@ static int copy_header_to_ring(void __user *ring, const void *header,
 	return 0;
 }
 
-static int copy_header_from_ring(void *header, const void __user *ring,
-				 size_t header_size)
+static int copy_header_from_ring(struct fuse_ring_ent *ent,
+				 enum fuse_uring_header_type type,
+				 void *header, size_t header_size)
 {
+	const void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_from_user(header, ring, header_size)) {
 		pr_info_ratelimited("Copying header from ring failed.\n");
 		return -EFAULT;
@@ -606,8 +640,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
-				    sizeof(ring_in_out));
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				    &ring_in_out, sizeof(ring_in_out));
 	if (err)
 		return err;
 
@@ -656,7 +690,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_header_to_ring(&ent->headers->op_in,
+			err = copy_header_to_ring(ent, FUSE_URING_HEADER_OP,
 						  in_args->value,
 						  in_args->size);
 			if (err)
@@ -675,7 +709,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				   &ent_in_out,
 				   sizeof(ent_in_out));
 }
 
@@ -705,7 +740,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,
 				   sizeof(req->in.h));
 }
 
@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 
-	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
 				    sizeof(req->out.h));
 	if (err) {
 		req->out.h.error = err;
Signature · 1 line
2.47.3
maintainer Reviewed-by on v2 5/8
#20 lore raw

Re: [PATCH v2 5/8] fuse: use enum types for header copying

Quoted 14 lines · Joanne Koong

Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.

---
 fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 45 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +0 -0
index faa7217e85c4..d96368e93e8d 100644

fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35

Quoted 5 lines
fs/fuse/dev_uring.c +2 -0
@@ -31,6 +31,12 @@ struct fuse_uring_pdu {
 
 static const struct fuse_iqueue_ops fuse_io_uring_ops;
 
+enum fuse_uring_header_type {
+	FUSE_URING_HEADER_IN_OUT,

In post review of my own names, headers->in_out is rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_ FUSE_URING_HEADER_MSG_IN_OUT?

Quoted 99 lines
fs/fuse/dev_uring.c +42 -9
+	FUSE_URING_HEADER_OP,
+	FUSE_URING_HEADER_RING_ENT,
+};
+
 static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
 				   struct fuse_ring_ent *ring_ent)
 {
@@ -574,9 +580,31 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
-static int copy_header_to_ring(void __user *ring, const void *header,
-			       size_t header_size)
+static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
+					 enum fuse_uring_header_type type)
+{
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		return &ent->headers->in_out;
+	case FUSE_URING_HEADER_OP:
+		return &ent->headers->op_in;
+	case FUSE_URING_HEADER_RING_ENT:
+		return &ent->headers->ring_ent_in_out;
+	}
+
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
+static int copy_header_to_ring(struct fuse_ring_ent *ent,
+			       enum fuse_uring_header_type type,
+			       const void *header, size_t header_size)
 {
+	void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_to_user(ring, header, header_size)) {
 		pr_info_ratelimited("Copying header to ring failed.\n");
 		return -EFAULT;
@@ -585,9 +613,15 @@ static int copy_header_to_ring(void __user *ring, const void *header,
 	return 0;
 }
 
-static int copy_header_from_ring(void *header, const void __user *ring,
-				 size_t header_size)
+static int copy_header_from_ring(struct fuse_ring_ent *ent,
+				 enum fuse_uring_header_type type,
+				 void *header, size_t header_size)
 {
+	const void __user *ring = get_user_ring_header(ent, type);
+
+	if (!ring)
+		return -EINVAL;
+
 	if (copy_from_user(header, ring, header_size)) {
 		pr_info_ratelimited("Copying header from ring failed.\n");
 		return -EFAULT;
@@ -606,8 +640,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	int err;
 	struct fuse_uring_ent_in_out ring_in_out;
 
-	err = copy_header_from_ring(&ring_in_out, &ent->headers->ring_ent_in_out,
-				    sizeof(ring_in_out));
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				    &ring_in_out, sizeof(ring_in_out));
 	if (err)
 		return err;
 
@@ -656,7 +690,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		 * Some op code have that as zero size.
 		 */
 		if (args->in_args[0].size > 0) {
-			err = copy_header_to_ring(&ent->headers->op_in,
+			err = copy_header_to_ring(ent, FUSE_URING_HEADER_OP,
 						  in_args->value,
 						  in_args->size);
 			if (err)
@@ -675,7 +709,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 	}
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
-	return copy_header_to_ring(&ent->headers->ring_ent_in_out, &ent_in_out,
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
+				   &ent_in_out,
 				   sizeof(ent_in_out));
 }
 
@@ -705,7 +740,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 	}
 
 	/* copy fuse_in_header */
-	return copy_header_to_ring(&ent->headers->in_out, &req->in.h,
+	return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,
 				   sizeof(req->in.h));
 }
 

fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841

Quoted 8 lines
fs/fuse/dev_uring.c +1 -1
@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
 	struct fuse_conn *fc = ring->fc;
 	ssize_t err = 0;
 
-	err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+	err = copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
 				    sizeof(req->out.h));
 	if (err) {
 		req->out.h.error = err;
author maintainer
#21 lore raw

Re: [PATCH v2 5/8] fuse: use enum types for header copying

Quoted 17 lines · Bernd Schubert
Quoted 15 lines · Joanne Koong

Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.

---
 fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 45 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +0 -0
index faa7217e85c4..d96368e93e8d 100644
@@ -31,6 +31,12 @@ struct fuse_uring_pdu {
 
Quoted 8 lines

fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35

Quoted 4 lines
fs/fuse/dev_uring.c +2 -0
 static const struct fuse_iqueue_ops fuse_io_uring_ops;
 
+enum fuse_uring_header_type {
+     FUSE_URING_HEADER_IN_OUT,

In post review of my own names, headers->in_out is rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_ FUSE_URING_HEADER_MSG_IN_OUT?

Ahh I personally find "msg" a bit more confusing because "message" makes me think it refers just to the payload since the whole thing is usually called the request. So if we had to rename it, maybe FUSE_URING_HEADER_REQ_IN_OUT? Though I do like your original naming of it, FUSE_URING_HEADER_IN_OUT since FUSE_URING_FUSE_HEADER_IN_OUT sounds a little redundant.

I'll add some comments on top of this too, eg "/*struct fuse_in_header / struct_fuse_out_header */, to clarify.

Quoted 8 lines
Quoted 8 lines
fs/fuse/dev_uring.c +3 -0
+     FUSE_URING_HEADER_OP,
+     FUSE_URING_HEADER_RING_ENT,
+};
@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
      struct fuse_conn *fc = ring->fc;
      ssize_t err = 0;
 
Quoted 8 lines

fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841

Quoted 5 lines
fs/fuse/dev_uring.c +1 -1
-     err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+     err = copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
                                  sizeof(req->out.h));
      if (err) {
              req->out.h.error = err;

Thanks for reviewing the patches!

maintainer
#22 lore raw

Re: [PATCH v2 5/8] fuse: use enum types for header copying

Quoted 27 lines · Joanne Koong
Quoted 17 lines · Bernd Schubert
Quoted 15 lines · Joanne Koong

Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.

---
 fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 45 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +0 -0
index faa7217e85c4..d96368e93e8d 100644
@@ -31,6 +31,12 @@ struct fuse_uring_pdu {
 
Quoted 8 lines

fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35

Quoted 4 lines
fs/fuse/dev_uring.c +2 -0
 static const struct fuse_iqueue_ops fuse_io_uring_ops;
 
+enum fuse_uring_header_type {
+     FUSE_URING_HEADER_IN_OUT,

In post review of my own names, headers->in_out is rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_ FUSE_URING_HEADER_MSG_IN_OUT?

Quoted 6 lines

Ahh I personally find "msg" a bit more confusing because "message" makes me think it refers just to the payload since the whole thing is usually called the request. So if we had to rename it, maybe FUSE_URING_HEADER_REQ_IN_OUT? Though I do like your original naming of it, FUSE_URING_HEADER_IN_OUT since FUSE_URING_FUSE_HEADER_IN_OUT sounds a little redundant.

FUSE_URING_HEADER_REQ_IN_OUT sounds nice to me. Renaming was just a suggestion. Let's keep the current name if you prefer that.

Quoted 3 lines

I'll add some comments on top of this too, eg "/*struct fuse_in_header / struct_fuse_out_header */, to clarify.

Comments are always helpful :)

Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
fs/fuse/dev_uring.c +3 -0
+     FUSE_URING_HEADER_OP,
+     FUSE_URING_HEADER_RING_ENT,
+};
@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
      struct fuse_conn *fc = ring->fc;
      ssize_t err = 0;
 
Quoted 8 lines

fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841

Quoted 5 lines
fs/fuse/dev_uring.c +1 -1
-     err = copy_header_from_ring(&req->out.h, &ent->headers->in_out,
+     err = copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
                                  sizeof(req->out.h));
      if (err) {
              req->out.h.error = err;

Thanks for reviewing the patches!

author maintainer
#23 lore raw

PATCH v2 6/8 fuse: add user_ prefix to userspace headers and payload fields

In reply to Joanne Koong

Rename the headers and payload fields to user_headers and user_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.

---
 fs/fuse/dev_uring.c   | 17 ++++++++---------
 fs/fuse/dev_uring_i.h |  4 ++--
 2 files changed, 10 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +8 -9
index d96368e93e8d..c814b571494f 100644
@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 {
 	switch (type) {
 	case FUSE_URING_HEADER_IN_OUT:
-		return &ent->headers->in_out;
+		return &ent->user_headers->in_out;
 	case FUSE_URING_HEADER_OP:
-		return &ent->headers->op_in;
+		return &ent->user_headers->op_in;
 	case FUSE_URING_HEADER_RING_ENT:
-		return &ent->headers->ring_ent_in_out;
+		return &ent->user_headers->ring_ent_in_out;
 	}
 
 	WARN_ON_ONCE(1);
@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	if (err)
 		return err;
 
-	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
+	err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
 			  &iter);
 	if (err)
 		return err;
@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		.commit_id = req->in.h.unique,
 	};
 
-	err = import_ubuf(ITER_DEST, ent->payload, ring->max_payload_sz, &iter);
+	err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
 	if (err) {
 		pr_info_ratelimited("fuse: Import of user buffer failed\n");
 		return err;
@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
 	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
-				   &ent_in_out,
-				   sizeof(ent_in_out));
+				   &ent_in_out, sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	INIT_LIST_HEAD(&ent->list);
 
 	ent->queue = queue;
-	ent->headers = iov[0].iov_base;
-	ent->payload = iov[1].iov_base;
+	ent->user_headers = iov[0].iov_base;
+	ent->user_payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
fs/fuse/dev_uring_i.h +2 -2
index 51a563922ce1..381fd0b8156a 100644
@@ -39,8 +39,8 @@ enum fuse_ring_req_state {
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
 	/* userspace buffer */
-	struct fuse_uring_req_header __user *headers;
-	void __user *payload;
+	struct fuse_uring_req_header __user *user_headers;
+	void __user *user_payload;
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;
Signature · 1 line
2.47.3
#24 lore raw

Re: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields

Quoted 50 lines · Joanne Koong

Rename the headers and payload fields to user_headers and user_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.

---
 fs/fuse/dev_uring.c   | 17 ++++++++---------
 fs/fuse/dev_uring_i.h |  4 ++--
 2 files changed, 10 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +5 -5
index d96368e93e8d..c814b571494f 100644
@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 {
        switch (type) {
        case FUSE_URING_HEADER_IN_OUT:
-               return &ent->headers->in_out;
+               return &ent->user_headers->in_out;
        case FUSE_URING_HEADER_OP:
-               return &ent->headers->op_in;
+               return &ent->user_headers->op_in;
        case FUSE_URING_HEADER_RING_ENT:
-               return &ent->headers->ring_ent_in_out;
+               return &ent->user_headers->ring_ent_in_out;
        }
 
        WARN_ON_ONCE(1);
@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
        if (err)
                return err;
 
-       err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
+       err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
                          &iter);
        if (err)
                return err;
@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
                .commit_id = req->in.h.unique,
        };
 
-       err = import_ubuf(ITER_DEST, ent->payload, ring->max_payload_sz, &iter);
+       err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
        if (err) {
                pr_info_ratelimited("fuse: Import of user buffer failed\n");
                return err;

fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 713

Quoted 6 lines
fs/fuse/dev_uring.c +1 -2
@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 
        ent_in_out.payload_sz = cs.ring.copied_sz;
        return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
-                                  &ent_in_out,
-                                  sizeof(ent_in_out));
+                                  &ent_in_out, sizeof(ent_in_out));

nit: looks like an unnecessary formatting change here. Either drop it or move it to the earlier commit that added these lines?

Best,
Caleb

Quoted 32 lines
fs/fuse/dev_uring.c +2 -2
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
        INIT_LIST_HEAD(&ent->list);
 
        ent->queue = queue;
-       ent->headers = iov[0].iov_base;
-       ent->payload = iov[1].iov_base;
+       ent->user_headers = iov[0].iov_base;
+       ent->user_payload = iov[1].iov_base;
 
        atomic_inc(&ring->queue_refs);
        return ent;
fs/fuse/dev_uring_i.h +2 -3
index 51a563922ce1..381fd0b8156a 100644
@@ -39,8 +39,8 @@ enum fuse_ring_req_state {
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
        /* userspace buffer */
-       struct fuse_uring_req_header __user *headers;
-       void __user *payload;
+       struct fuse_uring_req_header __user *user_headers;
+       void __user *user_payload;
 
        /* the ring queue that owns the request */
        struct fuse_ring_queue *queue;
--

2.47.3

author maintainer
#25 lore raw

Re: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields

Quoted 53 lines · Caleb Sander Mateos
Quoted 51 lines · Joanne Koong

Rename the headers and payload fields to user_headers and user_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.

---
 fs/fuse/dev_uring.c   | 17 ++++++++---------
 fs/fuse/dev_uring_i.h |  4 ++--
 2 files changed, 10 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +5 -5
index d96368e93e8d..c814b571494f 100644
@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 {
        switch (type) {
        case FUSE_URING_HEADER_IN_OUT:
-               return &ent->headers->in_out;
+               return &ent->user_headers->in_out;
        case FUSE_URING_HEADER_OP:
-               return &ent->headers->op_in;
+               return &ent->user_headers->op_in;
        case FUSE_URING_HEADER_RING_ENT:
-               return &ent->headers->ring_ent_in_out;
+               return &ent->user_headers->ring_ent_in_out;
        }
 
        WARN_ON_ONCE(1);
@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
        if (err)
                return err;
 
-       err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
+       err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
                          &iter);
        if (err)
                return err;
@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
                .commit_id = req->in.h.unique,
        };
 
-       err = import_ubuf(ITER_DEST, ent->payload, ring->max_payload_sz, &iter);
+       err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
        if (err) {
                pr_info_ratelimited("fuse: Import of user buffer failed\n");
                return err;
@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 
Quoted 8 lines

fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 713

Quoted 5 lines
fs/fuse/dev_uring.c +1 -2
        ent_in_out.payload_sz = cs.ring.copied_sz;
        return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
-                                  &ent_in_out,
-                                  sizeof(ent_in_out));
+                                  &ent_in_out, sizeof(ent_in_out));

nit: looks like an unnecessary formatting change here. Either drop it or move it to the earlier commit that added these lines?

I will move this to the earlier commit that added this.

Thanks,
Joanne

Quoted 36 lines

Best,
Caleb

Quoted 32 lines
fs/fuse/dev_uring.c +2 -2
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
        INIT_LIST_HEAD(&ent->list);
 
        ent->queue = queue;
-       ent->headers = iov[0].iov_base;
-       ent->payload = iov[1].iov_base;
+       ent->user_headers = iov[0].iov_base;
+       ent->user_payload = iov[1].iov_base;
 
        atomic_inc(&ring->queue_refs);
        return ent;
fs/fuse/dev_uring_i.h +2 -3
index 51a563922ce1..381fd0b8156a 100644
@@ -39,8 +39,8 @@ enum fuse_ring_req_state {
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
        /* userspace buffer */
-       struct fuse_uring_req_header __user *headers;
-       void __user *payload;
+       struct fuse_uring_req_header __user *user_headers;
+       void __user *user_payload;
 
        /* the ring queue that owns the request */
        struct fuse_ring_queue *queue;
--

2.47.3

maintainer
#26 lore raw

Re: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields

In reply to Joanne Koong
Quoted 74 lines · Joanne Koong

Rename the headers and payload fields to user_headers and user_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.

---
 fs/fuse/dev_uring.c   | 17 ++++++++---------
 fs/fuse/dev_uring_i.h |  4 ++--
 2 files changed, 10 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +8 -9
index d96368e93e8d..c814b571494f 100644
@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 {
 	switch (type) {
 	case FUSE_URING_HEADER_IN_OUT:
-		return &ent->headers->in_out;
+		return &ent->user_headers->in_out;
 	case FUSE_URING_HEADER_OP:
-		return &ent->headers->op_in;
+		return &ent->user_headers->op_in;
 	case FUSE_URING_HEADER_RING_ENT:
-		return &ent->headers->ring_ent_in_out;
+		return &ent->user_headers->ring_ent_in_out;
 	}
 
 	WARN_ON_ONCE(1);
@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	if (err)
 		return err;
 
-	err = import_ubuf(ITER_SOURCE, ent->payload, ring->max_payload_sz,
+	err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
 			  &iter);
 	if (err)
 		return err;
@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		.commit_id = req->in.h.unique,
 	};
 
-	err = import_ubuf(ITER_DEST, ent->payload, ring->max_payload_sz, &iter);
+	err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
 	if (err) {
 		pr_info_ratelimited("fuse: Import of user buffer failed\n");
 		return err;
@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 
 	ent_in_out.payload_sz = cs.ring.copied_sz;
 	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
-				   &ent_in_out,
-				   sizeof(ent_in_out));
+				   &ent_in_out, sizeof(ent_in_out));
 }
 
 static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	INIT_LIST_HEAD(&ent->list);
 
 	ent->queue = queue;
-	ent->headers = iov[0].iov_base;
-	ent->payload = iov[1].iov_base;
+	ent->user_headers = iov[0].iov_base;
+	ent->user_payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
fs/fuse/dev_uring_i.h +0 -0
index 51a563922ce1..381fd0b8156a 100644

fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 46

Quoted 10 lines
fs/fuse/dev_uring_i.h +2 -2
@@ -39,8 +39,8 @@ enum fuse_ring_req_state {
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
 	/* userspace buffer */
-	struct fuse_uring_req_header __user *headers;
-	void __user *payload;
+	struct fuse_uring_req_header __user *user_headers;
+	void __user *user_payload;
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;
author maintainer
#27 lore raw

PATCH v2 7/8 fuse: refactor setting up copy state for payload copying

In reply to Joanne Koong

Add a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.

---
 fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +25 -14
index c814b571494f..c6b22b14b354 100644
@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
 	return 0;
 }
 
+static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
+				 struct fuse_ring_ent* ent, int rw,
+				 struct iov_iter *iter,
+				 struct fuse_copy_state *cs)
+{
+	int err;
+
+	err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+			  iter);
+	if (err) {
+		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+		return err;
+	}
+
+	fuse_copy_init(cs, rw == ITER_DEST, iter);
+
+	cs->is_uring = true;
+	cs->req = req;
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -645,15 +667,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	if (err)
 		return err;
 
-	err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
-			  &iter);
+	err = setup_fuse_copy_state(ring, req, ent, ITER_SOURCE, &iter, &cs);
 	if (err)
 		return err;
 
-	fuse_copy_init(&cs, false, &iter);
-	cs.is_uring = true;
-	cs.req = req;
-
 	return fuse_copy_out_args(&cs, args, ring_in_out.payload_sz);
 }
 
@@ -674,15 +691,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		.commit_id = req->in.h.unique,
 	};
 
-	err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+	err = setup_fuse_copy_state(ring, req, ent, ITER_DEST, &iter, &cs);
+	if (err)
 		return err;
-	}
-
-	fuse_copy_init(&cs, true, &iter);
-	cs.is_uring = true;
-	cs.req = req;
 
 	if (num_args > 0) {
 		/*
Signature · 1 line
2.47.3
Reviewed-by on v2 7/8
#28 lore raw

Re: [PATCH v2 7/8] fuse: refactor setting up copy state for payload copying

Quoted 13 lines · Joanne Koong

Add a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.

---
 fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +0 -0
index c814b571494f..c6b22b14b354 100644

fs/fuse/dev_uring.c static int copy_header_from_ring(struct fuse_ring_ent *ent, · line 636

Quoted 7 lines
fs/fuse/dev_uring.c +4 -0
@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
 	return 0;
 }
 
+static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
+				 struct fuse_ring_ent* ent, int rw,
+				 struct iov_iter *iter,
+				 struct fuse_copy_state *cs)

Maybe 'struct fuse_copy_state *cs' as first argument? Just a suggestion and would good if it wouldn't end up in the middle when there are more arguments added at some point.

Quoted 46 lines
fs/fuse/dev_uring.c +20 -9
+{
+	int err;
+
+	err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+			  iter);
+	if (err) {
+		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+		return err;
+	}
+
+	fuse_copy_init(cs, rw == ITER_DEST, iter);
+
+	cs->is_uring = true;
+	cs->req = req;
+
+	return 0;
+}
+
 static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 				     struct fuse_req *req,
 				     struct fuse_ring_ent *ent)
@@ -645,15 +667,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
 	if (err)
 		return err;
 
-	err = import_ubuf(ITER_SOURCE, ent->user_payload, ring->max_payload_sz,
-			  &iter);
+	err = setup_fuse_copy_state(ring, req, ent, ITER_SOURCE, &iter, &cs);
 	if (err)
 		return err;
 
-	fuse_copy_init(&cs, false, &iter);
-	cs.is_uring = true;
-	cs.req = req;
-
 	return fuse_copy_out_args(&cs, args, ring_in_out.payload_sz);
 }
 
@@ -674,15 +691,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
 		.commit_id = req->in.h.unique,
 	};
 
-	err = import_ubuf(ITER_DEST, ent->user_payload, ring->max_payload_sz, &iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
+	err = setup_fuse_copy_state(ring, req, ent, ITER_DEST, &iter, &cs);

fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 699

Quoted 10 lines
fs/fuse/dev_uring.c +1 -5
+	if (err)
 		return err;
-	}
-
-	fuse_copy_init(&cs, true, &iter);
-	cs.is_uring = true;
-	cs.req = req;
 
 	if (num_args > 0) {
 		/*
author maintainer
#29 lore raw

Re: [PATCH v2 7/8] fuse: refactor setting up copy state for payload copying

Quoted 18 lines · Bernd Schubert
Quoted 16 lines · Joanne Koong

Add a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.

---
 fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +0 -0
index c814b571494f..c6b22b14b354 100644
@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
      return 0;
 }
 
Quoted 9 lines

fs/fuse/dev_uring.c static int copy_header_from_ring(struct fuse_ring_ent *ent, · line 636

Quoted 4 lines
fs/fuse/dev_uring.c +4 -0
+static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
+                              struct fuse_ring_ent* ent, int rw,
+                              struct iov_iter *iter,
+                              struct fuse_copy_state *cs)

Maybe 'struct fuse_copy_state *cs' as first argument? Just a suggestion and would good if it wouldn't end up in the middle when there are more arguments added at some point.

I like your suggestion a lot, I think that makes it clearer to read too :) I'll reorder this to be the first arg. Thanks!

author maintainer
#30 lore raw

PATCH v2 8/8 fuse: support io-uring registered buffers

In reply to Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +159 -41
index c6b22b14b354..f501bc81f331 100644
@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static void *get_kernel_ring_header(struct fuse_ring_ent *ent,
+				    enum fuse_uring_header_type type)
+{
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		return &ent->headers->in_out;
+	case FUSE_URING_HEADER_OP:
+		return &ent->headers->op_in;
+	case FUSE_URING_HEADER_RING_ENT:
+		return &ent->headers->ring_ent_in_out;
+	}
+
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 					 enum fuse_uring_header_type type)
 {
@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,
 			       enum fuse_uring_header_type type,
 			       const void *header, size_t header_size)
 {
-	void __user *ring = get_user_ring_header(ent, type);
+	if (ent->fixed_buffer) {
+		void *ring = get_kernel_ring_header(ent, type);
 
-	if (!ring)
-		return -EINVAL;
+		if (!ring)
+			return -EINVAL;
+		memcpy(ring, header, header_size);
+	} else {
+		void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_to_user(ring, header, header_size)) {
-		pr_info_ratelimited("Copying header to ring failed.\n");
-		return -EFAULT;
+		if (!ring)
+			return -EINVAL;
+		if (copy_to_user(ring, header, header_size)) {
+			pr_info_ratelimited("Copying header to ring failed.\n");
+			return -EFAULT;
+		}
 	}
-
 	return 0;
 }
 
@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
 				 enum fuse_uring_header_type type,
 				 void *header, size_t header_size)
 {
-	const void __user *ring = get_user_ring_header(ent, type);
+	if (ent->fixed_buffer) {
+		const void *ring = get_kernel_ring_header(ent, type);
 
-	if (!ring)
-		return -EINVAL;
+		if (!ring)
+			return -EINVAL;
+		memcpy(header, ring, header_size);
+	} else {
+		const void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_from_user(header, ring, header_size)) {
-		pr_info_ratelimited("Copying header from ring failed.\n");
-		return -EFAULT;
+		if (!ring)
+			return -EINVAL;
+		if (copy_from_user(header, ring, header_size)) {
+			pr_info_ratelimited("Copying header from ring failed.\n");
+			return -EFAULT;
+		}
 	}
 
 	return 0;
@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
 {
 	int err;
 
-	err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
-			  iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
-		return err;
+	if (ent->fixed_buffer) {
+		*iter = ent->payload_iter;
+	} else {
+		err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+				  iter);
+		if (err) {
+			pr_info_ratelimited("fuse: Import of user buffer failed\n");
+			return err;
+		}
 	}
 
 	fuse_copy_init(cs, rw == ITER_DEST, iter);
@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 				   sizeof(req->in.h));
 }
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+	size_t header_size = sizeof(struct fuse_uring_req_header);
+	struct iov_iter iter;
+	struct page *header_page;
+	size_t count, start;
+	ssize_t copied;
+	int err;
+
+	if (!ent->fixed_buffer)
+		return 0;
+
+	err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);
+	if (err)
+		return err;
+
+	count = iov_iter_count(&iter);
+	if (count < header_size || count & (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	/* Adjust the payload iter to protect the header from any overwrites */
+	ent->payload_iter = iter;
+	iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+	/* Set up the headers */
+	iov_iter_advance(&iter, count - header_size);
+	copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);
+	if (copied < header_size)
+		return -EFAULT;
+	ent->headers = kmap_local_page(header_page) + start;
+
+	/*
+	 * We can release the acquired reference on the header page immediately
+	 * since the page is pinned and io_uring_cmd_import_fixed_full()
+	 * prevents it from being unpinned while we are using it.
+	 */
+	put_page(header_page);
+
+	return 0;
+}
+
+static void fuse_uring_unmap_buffer(struct fuse_ring_ent *ent)
+{
+	if (ent->fixed_buffer)
+		kunmap_local(ent->headers);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 				   struct fuse_req *req)
 {
@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	unsigned int qid = READ_ONCE(cmd_req->qid);
 	struct fuse_pqueue *fpq;
 	struct fuse_req *req;
+	bool next_req;
 
 	err = -ENOTCONN;
 	if (!ring)
@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
 	/* without the queue lock, as other locks are taken */
 	fuse_uring_prepare_cancel(cmd, issue_flags, ent);
+
+	err = fuse_uring_map_buffer(ent);
+	if (err) {
+		fuse_uring_req_end(ent, req, err);
+		return err;
+	}
+
 	fuse_uring_commit(ent, req, issue_flags);
 
 	/*
@@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	if (fuse_uring_get_next_fuse_req(ent, queue))
+	next_req = fuse_uring_get_next_fuse_req(ent, queue);
+	fuse_uring_unmap_buffer(ent);
+	if (next_req)
 		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	struct iovec iov[FUSE_URING_IOV_SEGS];
 	int err;
 
+	err = -ENOMEM;
+	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+	if (!ent)
+		return ERR_PTR(err);
+
+	INIT_LIST_HEAD(&ent->list);
+
+	ent->queue = queue;
+
+	if (READ_ONCE(cmd->sqe->uring_cmd_flags) & IORING_URING_CMD_FIXED) {
+		ent->fixed_buffer = true;
+		atomic_inc(&ring->queue_refs);
+		return ent;
+	}
+
 	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
 	if (err) {
 		pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
 				    err);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	err = -EINVAL;
 	if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
 		pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	payload_size = iov[1].iov_len;
 	if (payload_size < ring->max_payload_sz) {
 		pr_info_ratelimited("Invalid req payload len %zu\n",
 				    payload_size);
-		return ERR_PTR(err);
+		goto error;
 	}
-
-	err = -ENOMEM;
-	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-	if (!ent)
-		return ERR_PTR(err);
-
-	INIT_LIST_HEAD(&ent->list);
-
-	ent->queue = queue;
 	ent->user_headers = iov[0].iov_base;
 	ent->user_payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
+
+error:
+	kfree(ent);
+	return ERR_PTR(err);
 }
 
 /*
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
 	struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
 	struct fuse_ring_queue *queue = ent->queue;
+	bool send_ent = true;
 	int err;
 
-	if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-		err = fuse_uring_prepare_send(ent, ent->fuse_req);
-		if (err) {
-			if (!fuse_uring_get_next_fuse_req(ent, queue))
-				return;
-			err = 0;
-		}
-	} else {
-		err = -ECANCELED;
+	if (issue_flags & IO_URING_F_TASK_DEAD) {
+		fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+		return;
+	}
+
+	err = fuse_uring_map_buffer(ent);
+	if (err) {
+		fuse_uring_req_end(ent, ent->fuse_req, err);
+		return;
+	}
+
+	err = fuse_uring_prepare_send(ent, ent->fuse_req);
+	if (err) {
+		send_ent = fuse_uring_get_next_fuse_req(ent, queue);
+		err = 0;
 	}
+	fuse_uring_unmap_buffer(ent);
 
-	fuse_uring_send(ent, cmd, err, issue_flags);
+	if (send_ent)
+		fuse_uring_send(ent, cmd, err, issue_flags);
 }
 
 static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring)
fs/fuse/dev_uring_i.h +24 -3
index 381fd0b8156a..fe14acccd6a6 100644
@@ -7,6 +7,7 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +39,29 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-	/* userspace buffer */
-	struct fuse_uring_req_header __user *user_headers;
-	void __user *user_payload;
+	/*
+	 * If true, the buffer was pre-registered by the daemon and the
+	 * pages backing it are pinned in kernel memory. The fixed buffer layout
+	 * is: [payload][header at end]. Use payload_iter and headers for
+	 * copying to/from the ring.
+	 *
+	 * Otherwise, use user_headers and user_payload which point to userspace
+	 * addresses representing the ring memory.
+	 */
+	bool fixed_buffer;
+
+	union {
+		/* fixed_buffer == false */
+		struct {
+			struct fuse_uring_req_header __user *user_headers;
+			void __user *user_payload;
+		};
+		/* fixed_buffer == true */
+		struct {
+			struct fuse_uring_req_header *headers;
+			struct iov_iter payload_iter;
+		};
+	};
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;
Signature · 1 line
2.47.3
#31 lore raw
Quoted 226 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +118 -19
index c6b22b14b354..f501bc81f331 100644
@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
        return err;
 }
 
+static void *get_kernel_ring_header(struct fuse_ring_ent *ent,
+                                   enum fuse_uring_header_type type)
+{
+       switch (type) {
+       case FUSE_URING_HEADER_IN_OUT:
+               return &ent->headers->in_out;
+       case FUSE_URING_HEADER_OP:
+               return &ent->headers->op_in;
+       case FUSE_URING_HEADER_RING_ENT:
+               return &ent->headers->ring_ent_in_out;
+       }
+
+       WARN_ON_ONCE(1);
+       return NULL;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
                                         enum fuse_uring_header_type type)
 {
@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,
                               enum fuse_uring_header_type type,
                               const void *header, size_t header_size)
 {
-       void __user *ring = get_user_ring_header(ent, type);
+       if (ent->fixed_buffer) {
+               void *ring = get_kernel_ring_header(ent, type);
 
-       if (!ring)
-               return -EINVAL;
+               if (!ring)
+                       return -EINVAL;
+               memcpy(ring, header, header_size);
+       } else {
+               void __user *ring = get_user_ring_header(ent, type);
 
-       if (copy_to_user(ring, header, header_size)) {
-               pr_info_ratelimited("Copying header to ring failed.\n");
-               return -EFAULT;
+               if (!ring)
+                       return -EINVAL;
+               if (copy_to_user(ring, header, header_size)) {
+                       pr_info_ratelimited("Copying header to ring failed.\n");
+                       return -EFAULT;
+               }
        }
-
        return 0;
 }
 
@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
                                 enum fuse_uring_header_type type,
                                 void *header, size_t header_size)
 {
-       const void __user *ring = get_user_ring_header(ent, type);
+       if (ent->fixed_buffer) {
+               const void *ring = get_kernel_ring_header(ent, type);
 
-       if (!ring)
-               return -EINVAL;
+               if (!ring)
+                       return -EINVAL;
+               memcpy(header, ring, header_size);
+       } else {
+               const void __user *ring = get_user_ring_header(ent, type);
 
-       if (copy_from_user(header, ring, header_size)) {
-               pr_info_ratelimited("Copying header from ring failed.\n");
-               return -EFAULT;
+               if (!ring)
+                       return -EINVAL;
+               if (copy_from_user(header, ring, header_size)) {
+                       pr_info_ratelimited("Copying header from ring failed.\n");
+                       return -EFAULT;
+               }
        }
 
        return 0;
@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
 {
        int err;
 
-       err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
-                         iter);
-       if (err) {
-               pr_info_ratelimited("fuse: Import of user buffer failed\n");
-               return err;
+       if (ent->fixed_buffer) {
+               *iter = ent->payload_iter;
+       } else {
+               err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+                                 iter);
+               if (err) {
+                       pr_info_ratelimited("fuse: Import of user buffer failed\n");
+                       return err;
+               }
        }
 
        fuse_copy_init(cs, rw == ITER_DEST, iter);
@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
                                   sizeof(req->in.h));
 }
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+       size_t header_size = sizeof(struct fuse_uring_req_header);
+       struct iov_iter iter;
+       struct page *header_page;
+       size_t count, start;
+       ssize_t copied;
+       int err;
+
+       if (!ent->fixed_buffer)
+               return 0;
+
+       err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);
+       if (err)
+               return err;
+
+       count = iov_iter_count(&iter);
+       if (count < header_size || count & (PAGE_SIZE - 1))
+               return -EINVAL;
+
+       /* Adjust the payload iter to protect the header from any overwrites */
+       ent->payload_iter = iter;
+       iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+       /* Set up the headers */
+       iov_iter_advance(&iter, count - header_size);
+       copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);
+       if (copied < header_size)
+               return -EFAULT;
+       ent->headers = kmap_local_page(header_page) + start;
+
+       /*
+        * We can release the acquired reference on the header page immediately
+        * since the page is pinned and io_uring_cmd_import_fixed_full()
+        * prevents it from being unpinned while we are using it.
+        */
+       put_page(header_page);
+
+       return 0;
+}
+
+static void fuse_uring_unmap_buffer(struct fuse_ring_ent *ent)
+{
+       if (ent->fixed_buffer)
+               kunmap_local(ent->headers);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
                                   struct fuse_req *req)
 {
@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
        unsigned int qid = READ_ONCE(cmd_req->qid);
        struct fuse_pqueue *fpq;
        struct fuse_req *req;
+       bool next_req;
 
        err = -ENOTCONN;
        if (!ring)
@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
        /* without the queue lock, as other locks are taken */
        fuse_uring_prepare_cancel(cmd, issue_flags, ent);
+
+       err = fuse_uring_map_buffer(ent);
+       if (err) {
+               fuse_uring_req_end(ent, req, err);
+               return err;
+       }
+
        fuse_uring_commit(ent, req, issue_flags);
 
        /*
@@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
         * and fetching is done in one step vs legacy fuse, which has separated
         * read (fetch request) and write (commit result).
         */
-       if (fuse_uring_get_next_fuse_req(ent, queue))
+       next_req = fuse_uring_get_next_fuse_req(ent, queue);
+       fuse_uring_unmap_buffer(ent);
+       if (next_req)
                fuse_uring_send(ent, cmd, 0, issue_flags);
        return 0;
 }
@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
        struct iovec iov[FUSE_URING_IOV_SEGS];
        int err;
 

fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1197

Quoted 10 lines
fs/fuse/dev_uring.c +10 -0
+       err = -ENOMEM;
+       ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+       if (!ent)
+               return ERR_PTR(err);
+
+       INIT_LIST_HEAD(&ent->list);
+
+       ent->queue = queue;
+
+       if (READ_ONCE(cmd->sqe->uring_cmd_flags) & IORING_URING_CMD_FIXED) {

Just use cmd->flags. That avoids having to deal with any possibility of userspace changing sqe-> uring_cmd_flags between the multiple loads of it.

Quoted 107 lines
fs/fuse/dev_uring.c +31 -22
+               ent->fixed_buffer = true;
+               atomic_inc(&ring->queue_refs);
+               return ent;
+       }
+
        err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
        if (err) {
                pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
                                    err);
-               return ERR_PTR(err);
+               goto error;
        }
 
        err = -EINVAL;
        if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
                pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-               return ERR_PTR(err);
+               goto error;
        }
 
        payload_size = iov[1].iov_len;
        if (payload_size < ring->max_payload_sz) {
                pr_info_ratelimited("Invalid req payload len %zu\n",
                                    payload_size);
-               return ERR_PTR(err);
+               goto error;
        }
-
-       err = -ENOMEM;
-       ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-       if (!ent)
-               return ERR_PTR(err);
-
-       INIT_LIST_HEAD(&ent->list);
-
-       ent->queue = queue;
        ent->user_headers = iov[0].iov_base;
        ent->user_payload = iov[1].iov_base;
 
        atomic_inc(&ring->queue_refs);
        return ent;
+
+error:
+       kfree(ent);
+       return ERR_PTR(err);
 }
 
 /*
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
        struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
        struct fuse_ring_queue *queue = ent->queue;
+       bool send_ent = true;
        int err;
 
-       if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-               err = fuse_uring_prepare_send(ent, ent->fuse_req);
-               if (err) {
-                       if (!fuse_uring_get_next_fuse_req(ent, queue))
-                               return;
-                       err = 0;
-               }
-       } else {
-               err = -ECANCELED;
+       if (issue_flags & IO_URING_F_TASK_DEAD) {
+               fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+               return;
+       }
+
+       err = fuse_uring_map_buffer(ent);
+       if (err) {
+               fuse_uring_req_end(ent, ent->fuse_req, err);
+               return;
+       }
+
+       err = fuse_uring_prepare_send(ent, ent->fuse_req);
+       if (err) {
+               send_ent = fuse_uring_get_next_fuse_req(ent, queue);
+               err = 0;
        }
+       fuse_uring_unmap_buffer(ent);
 
-       fuse_uring_send(ent, cmd, err, issue_flags);
+       if (send_ent)
+               fuse_uring_send(ent, cmd, err, issue_flags);
 }
 
 static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring)
fs/fuse/dev_uring_i.h +1 -3
index 381fd0b8156a..fe14acccd6a6 100644
@@ -7,6 +7,7 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +39,29 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-       /* userspace buffer */
-       struct fuse_uring_req_header __user *user_headers;
-       void __user *user_payload;

fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 51

Quoted 10 lines
fs/fuse/dev_uring_i.h +10 -0
+       /*
+        * If true, the buffer was pre-registered by the daemon and the
+        * pages backing it are pinned in kernel memory. The fixed buffer layout
+        * is: [payload][header at end]. Use payload_iter and headers for
+        * copying to/from the ring.
+        *
+        * Otherwise, use user_headers and user_payload which point to userspace
+        * addresses representing the ring memory.
+        */
+       bool fixed_buffer;

Could use cmd->flags instead of adding this field. It's an extra indirection vs. space tradeoff, I guess.

Best,
Caleb

Quoted 19 lines
fs/fuse/dev_uring_i.h +13 -1
+
+       union {
+               /* fixed_buffer == false */
+               struct {
+                       struct fuse_uring_req_header __user *user_headers;
+                       void __user *user_payload;
+               };
+               /* fixed_buffer == true */
+               struct {
+                       struct fuse_uring_req_header *headers;
+                       struct iov_iter payload_iter;
+               };
+       };
 
        /* the ring queue that owns the request */
        struct fuse_ring_queue *queue;
--

2.47.3

author maintainer
#32 lore raw
Quoted 228 lines · Caleb Sander Mateos
Quoted 226 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +118 -19
index c6b22b14b354..f501bc81f331 100644
@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
        return err;
 }
 
+static void *get_kernel_ring_header(struct fuse_ring_ent *ent,
+                                   enum fuse_uring_header_type type)
+{
+       switch (type) {
+       case FUSE_URING_HEADER_IN_OUT:
+               return &ent->headers->in_out;
+       case FUSE_URING_HEADER_OP:
+               return &ent->headers->op_in;
+       case FUSE_URING_HEADER_RING_ENT:
+               return &ent->headers->ring_ent_in_out;
+       }
+
+       WARN_ON_ONCE(1);
+       return NULL;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
                                         enum fuse_uring_header_type type)
 {
@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,
                               enum fuse_uring_header_type type,
                               const void *header, size_t header_size)
 {
-       void __user *ring = get_user_ring_header(ent, type);
+       if (ent->fixed_buffer) {
+               void *ring = get_kernel_ring_header(ent, type);
 
-       if (!ring)
-               return -EINVAL;
+               if (!ring)
+                       return -EINVAL;
+               memcpy(ring, header, header_size);
+       } else {
+               void __user *ring = get_user_ring_header(ent, type);
 
-       if (copy_to_user(ring, header, header_size)) {
-               pr_info_ratelimited("Copying header to ring failed.\n");
-               return -EFAULT;
+               if (!ring)
+                       return -EINVAL;
+               if (copy_to_user(ring, header, header_size)) {
+                       pr_info_ratelimited("Copying header to ring failed.\n");
+                       return -EFAULT;
+               }
        }
-
        return 0;
 }
 
@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
                                 enum fuse_uring_header_type type,
                                 void *header, size_t header_size)
 {
-       const void __user *ring = get_user_ring_header(ent, type);
+       if (ent->fixed_buffer) {
+               const void *ring = get_kernel_ring_header(ent, type);
 
-       if (!ring)
-               return -EINVAL;
+               if (!ring)
+                       return -EINVAL;
+               memcpy(header, ring, header_size);
+       } else {
+               const void __user *ring = get_user_ring_header(ent, type);
 
-       if (copy_from_user(header, ring, header_size)) {
-               pr_info_ratelimited("Copying header from ring failed.\n");
-               return -EFAULT;
+               if (!ring)
+                       return -EINVAL;
+               if (copy_from_user(header, ring, header_size)) {
+                       pr_info_ratelimited("Copying header from ring failed.\n");
+                       return -EFAULT;
+               }
        }
 
        return 0;
@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
 {
        int err;
 
-       err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
-                         iter);
-       if (err) {
-               pr_info_ratelimited("fuse: Import of user buffer failed\n");
-               return err;
+       if (ent->fixed_buffer) {
+               *iter = ent->payload_iter;
+       } else {
+               err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+                                 iter);
+               if (err) {
+                       pr_info_ratelimited("fuse: Import of user buffer failed\n");
+                       return err;
+               }
        }
 
        fuse_copy_init(cs, rw == ITER_DEST, iter);
@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
                                   sizeof(req->in.h));
 }
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+       size_t header_size = sizeof(struct fuse_uring_req_header);
+       struct iov_iter iter;
+       struct page *header_page;
+       size_t count, start;
+       ssize_t copied;
+       int err;
+
+       if (!ent->fixed_buffer)
+               return 0;
+
+       err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);
+       if (err)
+               return err;
+
+       count = iov_iter_count(&iter);
+       if (count < header_size || count & (PAGE_SIZE - 1))
+               return -EINVAL;
+
+       /* Adjust the payload iter to protect the header from any overwrites */
+       ent->payload_iter = iter;
+       iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+       /* Set up the headers */
+       iov_iter_advance(&iter, count - header_size);
+       copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);
+       if (copied < header_size)
+               return -EFAULT;
+       ent->headers = kmap_local_page(header_page) + start;
+
+       /*
+        * We can release the acquired reference on the header page immediately
+        * since the page is pinned and io_uring_cmd_import_fixed_full()
+        * prevents it from being unpinned while we are using it.
+        */
+       put_page(header_page);
+
+       return 0;
+}
+
+static void fuse_uring_unmap_buffer(struct fuse_ring_ent *ent)
+{
+       if (ent->fixed_buffer)
+               kunmap_local(ent->headers);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
                                   struct fuse_req *req)
 {
@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
        unsigned int qid = READ_ONCE(cmd_req->qid);
        struct fuse_pqueue *fpq;
        struct fuse_req *req;
+       bool next_req;
 
        err = -ENOTCONN;
        if (!ring)
@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
        /* without the queue lock, as other locks are taken */
        fuse_uring_prepare_cancel(cmd, issue_flags, ent);
+
+       err = fuse_uring_map_buffer(ent);
+       if (err) {
+               fuse_uring_req_end(ent, req, err);
+               return err;
+       }
+
        fuse_uring_commit(ent, req, issue_flags);
 
        /*
@@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
         * and fetching is done in one step vs legacy fuse, which has separated
         * read (fetch request) and write (commit result).
         */
-       if (fuse_uring_get_next_fuse_req(ent, queue))
+       next_req = fuse_uring_get_next_fuse_req(ent, queue);
+       fuse_uring_unmap_buffer(ent);
+       if (next_req)
                fuse_uring_send(ent, cmd, 0, issue_flags);
        return 0;
 }
@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
        struct iovec iov[FUSE_URING_IOV_SEGS];
        int err;
 
Quoted 14 lines

fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1197

Quoted 10 lines
fs/fuse/dev_uring.c +10 -0
+       err = -ENOMEM;
+       ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+       if (!ent)
+               return ERR_PTR(err);
+
+       INIT_LIST_HEAD(&ent->list);
+
+       ent->queue = queue;
+
+       if (READ_ONCE(cmd->sqe->uring_cmd_flags) & IORING_URING_CMD_FIXED) {

Just use cmd->flags. That avoids having to deal with any possibility of userspace changing sqe-> uring_cmd_flags between the multiple loads of it.

Awesome, I'll switch this to just use cmd->flags.

Thank you for looking at the patches.

Quoted 144 lines
Quoted 108 lines
fs/fuse/dev_uring.c +31 -22
+               ent->fixed_buffer = true;
+               atomic_inc(&ring->queue_refs);
+               return ent;
+       }
+
        err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
        if (err) {
                pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
                                    err);
-               return ERR_PTR(err);
+               goto error;
        }
 
        err = -EINVAL;
        if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
                pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-               return ERR_PTR(err);
+               goto error;
        }
 
        payload_size = iov[1].iov_len;
        if (payload_size < ring->max_payload_sz) {
                pr_info_ratelimited("Invalid req payload len %zu\n",
                                    payload_size);
-               return ERR_PTR(err);
+               goto error;
        }
-
-       err = -ENOMEM;
-       ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-       if (!ent)
-               return ERR_PTR(err);
-
-       INIT_LIST_HEAD(&ent->list);
-
-       ent->queue = queue;
        ent->user_headers = iov[0].iov_base;
        ent->user_payload = iov[1].iov_base;
 
        atomic_inc(&ring->queue_refs);
        return ent;
+
+error:
+       kfree(ent);
+       return ERR_PTR(err);
 }
 
 /*
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
        struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
        struct fuse_ring_queue *queue = ent->queue;
+       bool send_ent = true;
        int err;
 
-       if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-               err = fuse_uring_prepare_send(ent, ent->fuse_req);
-               if (err) {
-                       if (!fuse_uring_get_next_fuse_req(ent, queue))
-                               return;
-                       err = 0;
-               }
-       } else {
-               err = -ECANCELED;
+       if (issue_flags & IO_URING_F_TASK_DEAD) {
+               fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+               return;
+       }
+
+       err = fuse_uring_map_buffer(ent);
+       if (err) {
+               fuse_uring_req_end(ent, ent->fuse_req, err);
+               return;
+       }
+
+       err = fuse_uring_prepare_send(ent, ent->fuse_req);
+       if (err) {
+               send_ent = fuse_uring_get_next_fuse_req(ent, queue);
+               err = 0;
        }
+       fuse_uring_unmap_buffer(ent);
 
-       fuse_uring_send(ent, cmd, err, issue_flags);
+       if (send_ent)
+               fuse_uring_send(ent, cmd, err, issue_flags);
 }
 
 static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring)
fs/fuse/dev_uring_i.h +1 -3
index 381fd0b8156a..fe14acccd6a6 100644
@@ -7,6 +7,7 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +39,29 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-       /* userspace buffer */
-       struct fuse_uring_req_header __user *user_headers;
-       void __user *user_payload;

fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 51

Quoted 10 lines
fs/fuse/dev_uring_i.h +10 -0
+       /*
+        * If true, the buffer was pre-registered by the daemon and the
+        * pages backing it are pinned in kernel memory. The fixed buffer layout
+        * is: [payload][header at end]. Use payload_iter and headers for
+        * copying to/from the ring.
+        *
+        * Otherwise, use user_headers and user_payload which point to userspace
+        * addresses representing the ring memory.
+        */
+       bool fixed_buffer;

Could use cmd->flags instead of adding this field. It's an extra indirection vs. space tradeoff, I guess.

Best,
Caleb

Quoted 19 lines
fs/fuse/dev_uring_i.h +13 -1
+
+       union {
+               /* fixed_buffer == false */
+               struct {
+                       struct fuse_uring_req_header __user *user_headers;
+                       void __user *user_payload;
+               };
+               /* fixed_buffer == true */
+               struct {
+                       struct fuse_uring_req_header *headers;
+                       struct iov_iter payload_iter;
+               };
+       };
 
        /* the ring queue that owns the request */
        struct fuse_ring_queue *queue;
--

2.47.3

maintainer
#33 lore raw
In reply to Joanne Koong
Quoted 141 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +63 -18
index c6b22b14b354..f501bc81f331 100644
@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,
 	return err;
 }
 
+static void *get_kernel_ring_header(struct fuse_ring_ent *ent,
+				    enum fuse_uring_header_type type)
+{
+	switch (type) {
+	case FUSE_URING_HEADER_IN_OUT:
+		return &ent->headers->in_out;
+	case FUSE_URING_HEADER_OP:
+		return &ent->headers->op_in;
+	case FUSE_URING_HEADER_RING_ENT:
+		return &ent->headers->ring_ent_in_out;
+	}
+
+	WARN_ON_ONCE(1);
+	return NULL;
+}
+
 static void __user *get_user_ring_header(struct fuse_ring_ent *ent,
 					 enum fuse_uring_header_type type)
 {
@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,
 			       enum fuse_uring_header_type type,
 			       const void *header, size_t header_size)
 {
-	void __user *ring = get_user_ring_header(ent, type);
+	if (ent->fixed_buffer) {
+		void *ring = get_kernel_ring_header(ent, type);
 
-	if (!ring)
-		return -EINVAL;
+		if (!ring)
+			return -EINVAL;
+		memcpy(ring, header, header_size);
+	} else {
+		void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_to_user(ring, header, header_size)) {
-		pr_info_ratelimited("Copying header to ring failed.\n");
-		return -EFAULT;
+		if (!ring)
+			return -EINVAL;
+		if (copy_to_user(ring, header, header_size)) {
+			pr_info_ratelimited("Copying header to ring failed.\n");
+			return -EFAULT;
+		}
 	}
-
 	return 0;
 }
 
@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
 				 enum fuse_uring_header_type type,
 				 void *header, size_t header_size)
 {
-	const void __user *ring = get_user_ring_header(ent, type);
+	if (ent->fixed_buffer) {
+		const void *ring = get_kernel_ring_header(ent, type);
 
-	if (!ring)
-		return -EINVAL;
+		if (!ring)
+			return -EINVAL;
+		memcpy(header, ring, header_size);
+	} else {
+		const void __user *ring = get_user_ring_header(ent, type);
 
-	if (copy_from_user(header, ring, header_size)) {
-		pr_info_ratelimited("Copying header from ring failed.\n");
-		return -EFAULT;
+		if (!ring)
+			return -EINVAL;
+		if (copy_from_user(header, ring, header_size)) {
+			pr_info_ratelimited("Copying header from ring failed.\n");
+			return -EFAULT;
+		}
 	}
 
 	return 0;
@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req,
 {
 	int err;
 
-	err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
-			  iter);
-	if (err) {
-		pr_info_ratelimited("fuse: Import of user buffer failed\n");
-		return err;
+	if (ent->fixed_buffer) {
+		*iter = ent->payload_iter;
+	} else {
+		err = import_ubuf(rw, ent->user_payload, ring->max_payload_sz,
+				  iter);
+		if (err) {
+			pr_info_ratelimited("fuse: Import of user buffer failed\n");
+			return err;
+		}
 	}
 
 	fuse_copy_init(cs, rw == ITER_DEST, iter);
@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
 				   sizeof(req->in.h));
 }
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+	size_t header_size = sizeof(struct fuse_uring_req_header);

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 811

Quoted 10 lines
fs/fuse/dev_uring.c +10 -0
+	struct iov_iter iter;
+	struct page *header_page;
+	size_t count, start;
+	ssize_t copied;
+	int err;
+
+	if (!ent->fixed_buffer)
+		return 0;
+
+	err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817

Quoted 6 lines
fs/fuse/dev_uring.c +6 -0
+	if (err)
+		return err;
+
+	count = iov_iter_count(&iter);
+	if (count < header_size || count & (PAGE_SIZE - 1))
+		return -EINVAL;

|| !PAGE_ALIGNED(count)) ?

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825

Quoted 8 lines
fs/fuse/dev_uring.c +8 -0
+
+	/* Adjust the payload iter to protect the header from any overwrites */
+	ent->payload_iter = iter;
+	iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+	/* Set up the headers */
+	iov_iter_advance(&iter, count - header_size);
+	copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);

The iter is later used for the payload, but I miss a reset? iov_iter_revert()?

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828

Quoted 3 lines
fs/fuse/dev_uring.c +3 -0
+	if (copied < header_size)
+		return -EFAULT;
+	ent->headers = kmap_local_page(header_page) + start;

My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?

I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like

256 x 4K
8 x 128K
4 x 1M

I think there are currently two ways to do that

1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.

With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.

Quoted 130 lines
fs/fuse/dev_uring.c +51 -21
+
+	/*
+	 * We can release the acquired reference on the header page immediately
+	 * since the page is pinned and io_uring_cmd_import_fixed_full()
+	 * prevents it from being unpinned while we are using it.
+	 */
+	put_page(header_page);
+
+	return 0;
+}
+
+static void fuse_uring_unmap_buffer(struct fuse_ring_ent *ent)
+{
+	if (ent->fixed_buffer)
+		kunmap_local(ent->headers);
+}
+
 static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
 				   struct fuse_req *req)
 {
@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	unsigned int qid = READ_ONCE(cmd_req->qid);
 	struct fuse_pqueue *fpq;
 	struct fuse_req *req;
+	bool next_req;
 
 	err = -ENOTCONN;
 	if (!ring)
@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 
 	/* without the queue lock, as other locks are taken */
 	fuse_uring_prepare_cancel(cmd, issue_flags, ent);
+
+	err = fuse_uring_map_buffer(ent);
+	if (err) {
+		fuse_uring_req_end(ent, req, err);
+		return err;
+	}
+
 	fuse_uring_commit(ent, req, issue_flags);
 
 	/*
@@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 	 * and fetching is done in one step vs legacy fuse, which has separated
 	 * read (fetch request) and write (commit result).
 	 */
-	if (fuse_uring_get_next_fuse_req(ent, queue))
+	next_req = fuse_uring_get_next_fuse_req(ent, queue);
+	fuse_uring_unmap_buffer(ent);
+	if (next_req)
 		fuse_uring_send(ent, cmd, 0, issue_flags);
 	return 0;
 }
@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
 	struct iovec iov[FUSE_URING_IOV_SEGS];
 	int err;
 
+	err = -ENOMEM;
+	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
+	if (!ent)
+		return ERR_PTR(err);
+
+	INIT_LIST_HEAD(&ent->list);
+
+	ent->queue = queue;
+
+	if (READ_ONCE(cmd->sqe->uring_cmd_flags) & IORING_URING_CMD_FIXED) {
+		ent->fixed_buffer = true;
+		atomic_inc(&ring->queue_refs);
+		return ent;
+	}
+
 	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
 	if (err) {
 		pr_info_ratelimited("Failed to get iovec from sqe, err=%d\n",
 				    err);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	err = -EINVAL;
 	if (iov[0].iov_len < sizeof(struct fuse_uring_req_header)) {
 		pr_info_ratelimited("Invalid header len %zu\n", iov[0].iov_len);
-		return ERR_PTR(err);
+		goto error;
 	}
 
 	payload_size = iov[1].iov_len;
 	if (payload_size < ring->max_payload_sz) {
 		pr_info_ratelimited("Invalid req payload len %zu\n",
 				    payload_size);
-		return ERR_PTR(err);
+		goto error;
 	}
-
-	err = -ENOMEM;
-	ent = kzalloc(sizeof(*ent), GFP_KERNEL_ACCOUNT);
-	if (!ent)
-		return ERR_PTR(err);
-
-	INIT_LIST_HEAD(&ent->list);
-
-	ent->queue = queue;
 	ent->user_headers = iov[0].iov_base;
 	ent->user_payload = iov[1].iov_base;
 
 	atomic_inc(&ring->queue_refs);
 	return ent;
+
+error:
+	kfree(ent);
+	return ERR_PTR(err);
 }
 
 /*
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
 	struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
 	struct fuse_ring_queue *queue = ent->queue;
+	bool send_ent = true;
 	int err;
 
-	if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-		err = fuse_uring_prepare_send(ent, ent->fuse_req);
-		if (err) {
-			if (!fuse_uring_get_next_fuse_req(ent, queue))
-				return;
-			err = 0;
-		}
-	} else {

fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372

Quoted 10 lines
fs/fuse/dev_uring.c +9 -1
-		err = -ECANCELED;
+	if (issue_flags & IO_URING_F_TASK_DEAD) {
+		fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+		return;
+	}
+
+	err = fuse_uring_map_buffer(ent);
+	if (err) {
+		fuse_uring_req_end(ent, ent->fuse_req, err);
+		return;

I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.

Quoted 50 lines
fs/fuse/dev_uring.c +9 -1
+	}
+
+	err = fuse_uring_prepare_send(ent, ent->fuse_req);
+	if (err) {
+		send_ent = fuse_uring_get_next_fuse_req(ent, queue);
+		err = 0;
 	}
+	fuse_uring_unmap_buffer(ent);
 
-	fuse_uring_send(ent, cmd, err, issue_flags);
+	if (send_ent)
+		fuse_uring_send(ent, cmd, err, issue_flags);
 }
 
 static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring)
fs/fuse/dev_uring_i.h +17 -3
index 381fd0b8156a..fe14acccd6a6 100644
@@ -7,6 +7,7 @@
 #ifndef _FS_FUSE_DEV_URING_I_H
 #define _FS_FUSE_DEV_URING_I_H
 
+#include <linux/uio.h>
 #include "fuse_i.h"
 
 #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +39,29 @@ enum fuse_ring_req_state {
 
 /** A fuse ring entry, part of the ring queue */
 struct fuse_ring_ent {
-	/* userspace buffer */
-	struct fuse_uring_req_header __user *user_headers;
-	void __user *user_payload;
+	/*
+	 * If true, the buffer was pre-registered by the daemon and the
+	 * pages backing it are pinned in kernel memory. The fixed buffer layout
+	 * is: [payload][header at end]. Use payload_iter and headers for
+	 * copying to/from the ring.
+	 *
+	 * Otherwise, use user_headers and user_payload which point to userspace
+	 * addresses representing the ring memory.
+	 */
+	bool fixed_buffer;
+
+	union {
+		/* fixed_buffer == false */
+		struct {
+			struct fuse_uring_req_header __user *user_headers;
+			void __user *user_payload;

fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 67

Quoted 10 lines
fs/fuse/dev_uring_i.h +7 -0
+		};
+		/* fixed_buffer == true */
+		struct {
+			struct fuse_uring_req_header *headers;
+			struct iov_iter payload_iter;
+		};
+	};
 
 	/* the ring queue that owns the request */
 	struct fuse_ring_queue *queue;

Thanks,
Bernd

author maintainer
#34 lore raw
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +20 -0
index c6b22b14b354..f501bc81f331 100644
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+     size_t header_size = sizeof(struct fuse_uring_req_header);
+     struct iov_iter iter;
+     struct page *header_page;
+     size_t count, start;
+     ssize_t copied;
+     int err;
+
+     if (!ent->fixed_buffer)
+             return 0;
Quoted 8 lines

fs/fuse/dev_uring.c

Quoted 2 lines
fs/fuse/dev_uring.c +2 -0
+
+     err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Quoted 9 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817

Quoted 7 lines
fs/fuse/dev_uring.c +6 -0
+     if (err)
+             return err;
+
+     count = iov_iter_count(&iter);
+     if (count < header_size || count & (PAGE_SIZE - 1))
+             return -EINVAL;

|| !PAGE_ALIGNED(count)) ?

Nice, I didn't realize this macro existed. Thanks.

Quoted 11 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825

Quoted 9 lines
fs/fuse/dev_uring.c +8 -0
+
+     /* Adjust the payload iter to protect the header from any overwrites */
+     ent->payload_iter = iter;
+     iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+     /* Set up the headers */
+     iov_iter_advance(&iter, count - header_size);
+     copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);

The iter is later used for the payload, but I miss a reset? iov_iter_revert()?

This iter is separate from the payload iter and doesn't affect the payload iter's values because the "ent->payload_iter = iter;" assignment above shallow copies that out first.

Quoted 5 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828

Quoted 4 lines
fs/fuse/dev_uring.c +3 -0
+     if (copied < header_size)
+             return -EFAULT;
+     ent->headers = kmap_local_page(header_page) + start;
Quoted 7 lines

My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?

I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.

Quoted 7 lines

I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like

256 x 4K
8 x 128K
4 x 1M

I think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.

Thanks,
joanne

Quoted 33 lines

I think there are currently two ways to do that

1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.

With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.

Quoted 23 lines

/*

fs/fuse/dev_uring.c +6 -9
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
      struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
      struct fuse_ring_queue *queue = ent->queue;
+     bool send_ent = true;
      int err;
 
-     if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-             err = fuse_uring_prepare_send(ent, ent->fuse_req);
-             if (err) {
-                     if (!fuse_uring_get_next_fuse_req(ent, queue))
-                             return;
-                     err = 0;
-             }
-     } else {
-             err = -ECANCELED;
+     if (issue_flags & IO_URING_F_TASK_DEAD) {
+             fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+             return;
+     }
+
Quoted 8 lines

fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372

Quoted 4 lines
fs/fuse/dev_uring.c +4 -0
+     err = fuse_uring_map_buffer(ent);
+     if (err) {
+             fuse_uring_req_end(ent, ent->fuse_req, err);
+             return;

I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.

This approach makes sense to me and makes things a bit simpler. I'll add this to v3.

Thanks,
Joanne

maintainer
#35 lore raw
Quoted 56 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +20 -0
index c6b22b14b354..f501bc81f331 100644
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+     size_t header_size = sizeof(struct fuse_uring_req_header);
+     struct iov_iter iter;
+     struct page *header_page;
+     size_t count, start;
+     ssize_t copied;
+     int err;
+
+     if (!ent->fixed_buffer)
+             return 0;
Quoted 8 lines

fs/fuse/dev_uring.c

Quoted 2 lines
fs/fuse/dev_uring.c +2 -0
+
+     err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

Quoted 8 lines

I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Hmm, I had seen this discussion, but I don't find anything about preventing unregistration?

Thanks,
Bernd

#36 lore raw
Quoted 66 lines · Bernd Schubert
Quoted 63 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +20 -0
index c6b22b14b354..f501bc81f331 100644
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+     size_t header_size = sizeof(struct fuse_uring_req_header);
+     struct iov_iter iter;
+     struct page *header_page;
+     size_t count, start;
+     ssize_t copied;
+     int err;
+
+     if (!ent->fixed_buffer)
+             return 0;
Quoted 8 lines

fs/fuse/dev_uring.c

Quoted 2 lines
fs/fuse/dev_uring.c +2 -0
+
+     err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.

Quoted 4 lines
Quoted 1 line

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Hmm, I had seen this discussion, but I don't find anything about preventing unregistration?

Ah found it, sorry for the noise.

maintainer
#37 lore raw
In reply to Joanne Koong
Quoted 169 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +20 -0
index c6b22b14b354..f501bc81f331 100644
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+     size_t header_size = sizeof(struct fuse_uring_req_header);
+     struct iov_iter iter;
+     struct page *header_page;
+     size_t count, start;
+     ssize_t copied;
+     int err;
+
+     if (!ent->fixed_buffer)
+             return 0;
Quoted 8 lines

fs/fuse/dev_uring.c

Quoted 2 lines
fs/fuse/dev_uring.c +2 -0
+
+     err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Quoted 9 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817

Quoted 7 lines
fs/fuse/dev_uring.c +6 -0
+     if (err)
+             return err;
+
+     count = iov_iter_count(&iter);
+     if (count < header_size || count & (PAGE_SIZE - 1))
+             return -EINVAL;

|| !PAGE_ALIGNED(count)) ?

Nice, I didn't realize this macro existed. Thanks.

Quoted 11 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825

Quoted 9 lines
fs/fuse/dev_uring.c +8 -0
+
+     /* Adjust the payload iter to protect the header from any overwrites */
+     ent->payload_iter = iter;
+     iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+     /* Set up the headers */
+     iov_iter_advance(&iter, count - header_size);
+     copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);

The iter is later used for the payload, but I miss a reset? iov_iter_revert()?

This iter is separate from the payload iter and doesn't affect the payload iter's values because the "ent->payload_iter = iter;" assignment above shallow copies that out first.

Quoted 5 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828

Quoted 4 lines
fs/fuse/dev_uring.c +3 -0
+     if (copied < header_size)
+             return -EFAULT;
+     ent->headers = kmap_local_page(header_page) + start;
Quoted 7 lines

My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?

I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.

Quoted 7 lines

I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like

256 x 4K
8 x 128K
4 x 1M

I think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.

Thanks,
joanne

Quoted 33 lines

I think there are currently two ways to do that

1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.

With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.

Quoted 23 lines

/*

fs/fuse/dev_uring.c +6 -9
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
      struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
      struct fuse_ring_queue *queue = ent->queue;
+     bool send_ent = true;
      int err;
 
-     if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-             err = fuse_uring_prepare_send(ent, ent->fuse_req);
-             if (err) {
-                     if (!fuse_uring_get_next_fuse_req(ent, queue))
-                             return;
-                     err = 0;
-             }
-     } else {
-             err = -ECANCELED;
+     if (issue_flags & IO_URING_F_TASK_DEAD) {
+             fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+             return;
+     }
+
Quoted 8 lines

fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372

Quoted 4 lines
fs/fuse/dev_uring.c +4 -0
+     err = fuse_uring_map_buffer(ent);
+     if (err) {
+             fuse_uring_req_end(ent, ent->fuse_req, err);
+             return;

I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.

Quoted 2 lines

This approach makes sense to me and makes things a bit simpler. I'll add this to v3.

This is a just heads up, while I'm testing sync FUSE_INIT I'm running in several issues. Part of it needs to be a re-send on -EGAIN and -EINTR in libfuse - had slipped through so far, but another is related to our page pinning. I just thought I can easy abort the connection when that fails, but that results in a double lock, because so far we assumed IO_URING_F_UNLOCKED in teardown context.
I'm going to submit a patch to teardown in uring task context, so that we are sure about the flags.

I.e. I think my suggestion to abort here would run into the same issue as below.

[10375.669761] fuse: FUSE_IO_URING_CMD_REGISTER returned -4
[10375.670632] fuse: FUSE_IO_URING_CMD_REGISTER failed err=-4
[10375.671922] ============================================
[10375.672754] WARNING: possible recursive locking detected
[10375.673577] 6.8.12+ #6 Tainted: G        W  O      
[10375.674377] --------------------------------------------
[10375.675208] fuse-ring-0/7658 is trying to acquire lock:
[10375.676022] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: io_uring_cmd_done+0x14f/0x210
[10375.677335] 
               but task is already holding lock:
[10375.678292] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.679678] 
               other info that might help us debug this:
[10375.680722]  Possible unsafe locking scenario:
[10375.681692]        CPU0
[10375.682156]        ----
[10375.682633]   lock(&ctx->uring_lock);
[10375.683260]   lock(&ctx->uring_lock);
[10375.683879] 
                *** DEADLOCK ***
[10375.684916]  May be due to missing lock nesting notation
[10375.685994] 1 lock held by fuse-ring-0/7658:
[10375.686699]  #0: ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.688141] 
               stack backtrace:
[10375.688923] CPU: 0 PID: 7658 Comm: fuse-ring-0 Tainted: G        W  O       6.8.12+ #6
[10375.690150] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[10375.691534] Call Trace:
[10375.692019]  <TASK>
[10375.692445]  dump_stack_lvl+0x66/0x90
[10375.695219]  __lock_acquire+0x13b8/0x25c0
[10375.695901]  lock_acquire+0xb7/0x290
[10375.696507]  ? io_uring_cmd_done+0x14f/0x210
[10375.697226]  ? __lock_acquire+0x459/0x25c0
[10375.697913]  __mutex_lock+0x7d/0xae0
[10375.698530]  ? io_uring_cmd_done+0x14f/0x210
[10375.699251]  ? io_uring_cmd_done+0x14f/0x210
[10375.699967]  ? lock_acquire+0xb7/0x290
[10375.700606]  ? fuse_uring_stop_list_entries+0x1c2/0x280 [fuse]
[10375.701508]  ? io_uring_cmd_done+0x14f/0x210
[10375.702210]  ? lock_release+0x24b/0x390
[10375.702862]  io_uring_cmd_done+0x14f/0x210
[10375.703546]  fuse_uring_stop_list_entries+0x24f/0x280 [fuse]
[10375.704439]  fuse_uring_stop_queues+0xf7/0x240 [fuse]
[10375.705251]  fuse_abort_conn+0x3e6/0x3f0 [fuse]
[10375.705998]  fuse_uring_cmd+0x8de/0xf40 [fuse]
[10375.706742]  ? lock_acquired+0xb1/0x320
[10375.707415]  io_uring_cmd+0x6b/0x170
[10375.708080]  io_issue_sqe+0x4e/0x460
[10375.708705]  io_submit_sqes+0x22e/0x710
[10375.709356]  __x64_sys_io_uring_enter+0x696/0xbf0
[10375.710109]  do_syscall_64+0x6a/0x130
[10375.710743]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[10375.711541] RIP: 0033:0x7f31a073db95
[10375.712160] Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e2 02 74 c2 f0 48 83 0c 24
[10375.714772] RSP: 002b:00007f319a4f4c88 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
[10375.715946] RAX: ffffffffffffffda RBX: 00007f31992f5060 RCX: 00007f31a073db95
[10375.717000] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000006
[10375.718056] RBP: 00007f319a4f4d60 R08: 0000000000000000 R09: 0000000000000008
[10375.719113] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f31992f5000
[10375.720162] R13: 00000fe63325ea00 R14: 0000000000000000 R15: 00007f319a4f4cd0
[10375.721223]  </TASK>

Thanks,
Bernd

author maintainer
#38 lore raw
Quoted 247 lines · Bernd Schubert
Quoted 169 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong

Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.

To be backwards compatible, fuse uring still needs to support non-registered buffers as well.

---
 fs/fuse/dev_uring.c   | 200 +++++++++++++++++++++++++++++++++---------
 fs/fuse/dev_uring_i.h |  27 +++++-
 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +20 -0
index c6b22b14b354..f501bc81f331 100644
 
+/*
+ * Prepare fixed buffer for access. Sets up the payload iter and kmaps the
+ * header.
+ *
+ * Callers must call fuse_uring_unmap_buffer() in the same scope to release the
+ * header mapping.
+ *
+ * For non-fixed buffers, this is a no-op.
+ */
+static int fuse_uring_map_buffer(struct fuse_ring_ent *ent)
+{
+     size_t header_size = sizeof(struct fuse_uring_req_header);
+     struct iov_iter iter;
+     struct page *header_page;
+     size_t count, start;
+     ssize_t copied;
+     int err;
+
+     if (!ent->fixed_buffer)
+             return 0;
Quoted 8 lines

fs/fuse/dev_uring.c

Quoted 2 lines
fs/fuse/dev_uring.c +2 -0
+
+     err = io_uring_cmd_import_fixed_full(ITER_DEST, &iter, ent->cmd, 0);

This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.

I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]/

Quoted 9 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817

Quoted 7 lines
fs/fuse/dev_uring.c +6 -0
+     if (err)
+             return err;
+
+     count = iov_iter_count(&iter);
+     if (count < header_size || count & (PAGE_SIZE - 1))
+             return -EINVAL;

|| !PAGE_ALIGNED(count)) ?

Nice, I didn't realize this macro existed. Thanks.

Quoted 11 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825

Quoted 9 lines
fs/fuse/dev_uring.c +8 -0
+
+     /* Adjust the payload iter to protect the header from any overwrites */
+     ent->payload_iter = iter;
+     iov_iter_truncate(&ent->payload_iter, count - header_size);
+
+     /* Set up the headers */
+     iov_iter_advance(&iter, count - header_size);
+     copied = iov_iter_get_pages2(&iter, &header_page, header_size, 1, &start);

The iter is later used for the payload, but I miss a reset? iov_iter_revert()?

This iter is separate from the payload iter and doesn't affect the payload iter's values because the "ent->payload_iter = iter;" assignment above shallow copies that out first.

Quoted 5 lines

fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828

Quoted 4 lines
fs/fuse/dev_uring.c +3 -0
+     if (copied < header_size)
+             return -EFAULT;
+     ent->headers = kmap_local_page(header_page) + start;
Quoted 7 lines

My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?

I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.

Quoted 7 lines

I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like

256 x 4K
8 x 128K
4 x 1M

I think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.

Thanks,
joanne

Quoted 33 lines

I think there are currently two ways to do that

1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.

With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.

Quoted 23 lines

/*

fs/fuse/dev_uring.c +6 -9
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,
 {
      struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd);
      struct fuse_ring_queue *queue = ent->queue;
+     bool send_ent = true;
      int err;
 
-     if (!(issue_flags & IO_URING_F_TASK_DEAD)) {
-             err = fuse_uring_prepare_send(ent, ent->fuse_req);
-             if (err) {
-                     if (!fuse_uring_get_next_fuse_req(ent, queue))
-                             return;
-                     err = 0;
-             }
-     } else {
-             err = -ECANCELED;
+     if (issue_flags & IO_URING_F_TASK_DEAD) {
+             fuse_uring_send(ent, cmd, -ECANCELED, issue_flags);
+             return;
+     }
+
Quoted 8 lines

fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372

Quoted 4 lines
fs/fuse/dev_uring.c +4 -0
+     err = fuse_uring_map_buffer(ent);
+     if (err) {
+             fuse_uring_req_end(ent, ent->fuse_req, err);
+             return;

I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.

Quoted 2 lines

This approach makes sense to me and makes things a bit simpler. I'll add this to v3.

This is a just heads up, while I'm testing sync FUSE_INIT I'm running in several issues. Part of it needs to be a re-send on -EGAIN and -EINTR in libfuse - had slipped through so far, but another is related to our page pinning. I just thought I can easy abort the connection when that fails, but that results in a double lock, because so far we assumed IO_URING_F_UNLOCKED in teardown context.
I'm going to submit a patch to teardown in uring task context, so that we are sure about the flags.

I.e. I think my suggestion to abort here would run into the same issue as below.

[10375.669761] fuse: FUSE_IO_URING_CMD_REGISTER returned -4
[10375.670632] fuse: FUSE_IO_URING_CMD_REGISTER failed err=-4
[10375.671922] ============================================
[10375.672754] WARNING: possible recursive locking detected
[10375.673577] 6.8.12+ #6 Tainted: G        W  O
[10375.674377] --------------------------------------------
[10375.675208] fuse-ring-0/7658 is trying to acquire lock:
[10375.676022] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: io_uring_cmd_done+0x14f/0x210
[10375.677335]
               but task is already holding lock:
[10375.678292] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.679678]
               other info that might help us debug this:
[10375.680722]  Possible unsafe locking scenario:
[10375.681692]        CPU0
[10375.682156]        ----
[10375.682633]   lock(&ctx->uring_lock);
[10375.683260]   lock(&ctx->uring_lock);
[10375.683879]
                *** DEADLOCK ***
[10375.684916]  May be due to missing lock nesting notation
[10375.685994] 1 lock held by fuse-ring-0/7658:
[10375.686699]  #0: ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.688141]
               stack backtrace:
[10375.688923] CPU: 0 PID: 7658 Comm: fuse-ring-0 Tainted: G        W  O       6.8.12+ #6
[10375.690150] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[10375.691534] Call Trace:
[10375.692019]  <TASK>
[10375.692445]  dump_stack_lvl+0x66/0x90
[10375.695219]  __lock_acquire+0x13b8/0x25c0
[10375.695901]  lock_acquire+0xb7/0x290
[10375.696507]  ? io_uring_cmd_done+0x14f/0x210
[10375.697226]  ? __lock_acquire+0x459/0x25c0
[10375.697913]  __mutex_lock+0x7d/0xae0
[10375.698530]  ? io_uring_cmd_done+0x14f/0x210
[10375.699251]  ? io_uring_cmd_done+0x14f/0x210
[10375.699967]  ? lock_acquire+0xb7/0x290
[10375.700606]  ? fuse_uring_stop_list_entries+0x1c2/0x280 [fuse]
[10375.701508]  ? io_uring_cmd_done+0x14f/0x210
[10375.702210]  ? lock_release+0x24b/0x390
[10375.702862]  io_uring_cmd_done+0x14f/0x210
[10375.703546]  fuse_uring_stop_list_entries+0x24f/0x280 [fuse]
[10375.704439]  fuse_uring_stop_queues+0xf7/0x240 [fuse]
[10375.705251]  fuse_abort_conn+0x3e6/0x3f0 [fuse]
[10375.705998]  fuse_uring_cmd+0x8de/0xf40 [fuse]
[10375.706742]  ? lock_acquired+0xb1/0x320
[10375.707415]  io_uring_cmd+0x6b/0x170
[10375.708080]  io_issue_sqe+0x4e/0x460
[10375.708705]  io_submit_sqes+0x22e/0x710
[10375.709356]  __x64_sys_io_uring_enter+0x696/0xbf0
[10375.710109]  do_syscall_64+0x6a/0x130
[10375.710743]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[10375.711541] RIP: 0033:0x7f31a073db95
Quoted 9 lines
[10375.712160] Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e2 02 74 c2 f0 48 83 0c 24
[10375.714772] RSP: 002b:00007f319a4f4c88 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
[10375.715946] RAX: ffffffffffffffda RBX: 00007f31992f5060 RCX: 00007f31a073db95
[10375.717000] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000006
[10375.718056] RBP: 00007f319a4f4d60 R08: 0000000000000000 R09: 0000000000000008
[10375.719113] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f31992f5000
[10375.720162] R13: 00000fe63325ea00 R14: 0000000000000000 R15: 00007f319a4f4cd0
[10375.721223]  </TASK>

Thanks for the update, Bernd!

Quoted 4 lines

Thanks,
Bernd

author maintainer
#39 lore raw
In reply to Joanne Koong
Quoted 5 lines · Joanne Koong

This patchset adds fuse support for io-uring registered buffers. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.

Registered buffers helps with the I/O overhead, but there's still significant memory waste within each buffer. Each entry in a queue allocates a dedicated buffer of at least 1 MB (for some performant servers like passthrough_hp, each entry's buffer is 4 MB, which adds up (eg with the libfuse default queue depth of 8 on a 64-core machine, that's 2 GB of buffers per fuse connection)) but most of this space goes unused. In practice, entries on a queue will rarely consume their full buffer capacity simultaneously.

Instead of using registered buffers, I think it's better if we go with a kernel managed ring buffer. It'll have the same advantages in reducing I/O overhead as registered buffers but also give us optionality later on to support IORING_CQE_F_BUF_MORE for incremental buffer consumption or add multiple different-sized ring buffer pools (eg for large payloads vs small payloads, which is also something we have to differentiate between for zero-copy), whereas we would have no way of supporting that with registered buffers.

For v3, I'm going to go this direction.

Thanks,
Joanne

Quoted 51 lines

The main logic for fuse registered buffers is in the last patch (patch 8/8). Patch 1/8 adds an io_uring api for fetching the registered buffer and patches (2-7)/8 refactors the fuse io_uring code, which additionally will make adding in the logic for registered buffers neater.

The libfuse changes can be found in this branch:
https://github.com/joannekoong/libfuse/tree/registered_buffers. The libfuse implementation first tries registered buffers during registration and if this fails, will retry with non-registered buffers. This prevents having to add a new init flag (but does have the downside of printing dmesg errors for the failed registrations when trying the registered buffers). If using registered buffers and the daemon for whatever reason unregisters the buffers midway through, then this will sever server-kernel communication. Libfuse will never do this. Libfuse will only unregister the buffers when the entire session is being destroyed.

Benchmarks will be run and posted.

Thanks,
Joanne

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Add io_uring_cmd_import_fixed_full() patch
* Construct iter using io_uring_cmd_import_fixed_full() per cmd instead of recyling iters.
* Kmap the header instead of using bvec iter for iterating/copying. This makes the code easier to read.

Joanne Koong (8):
io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: add user_ prefix to userspace headers and payload fields
fuse: refactor setting up copy state for payload copying
fuse: support io-uring registered buffers

 fs/fuse/dev_uring.c          | 366 +++++++++++++++++++++++++----------
 fs/fuse/dev_uring_i.h        |  27 ++-
 include/linux/io_uring/cmd.h |   3 +
 io_uring/rsrc.c              |  14 ++
 io_uring/rsrc.h              |   2 +
 io_uring/uring_cmd.c         |  13 ++
 6 files changed, 316 insertions(+), 109 deletions(-)

--
2.47.3