fuse: support io-uring registered buffers
Subsystems (4)
Joanne Koong
maintainerv2 cover fuse: support io-uring registered buffers
This patchset adds fuse support for io-uring registered buffers. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
The main logic for fuse registered buffers is in the last patch (patch 8/8). Patch 1/8 adds an io_uring api for fetching the registered buffer and patches (2-7)/8 refactors the fuse io_uring code, which additionally will make adding in the logic for registered buffers neater.
The libfuse changes can be found in this branch:
https://github.com/joannekoong/libfuse/tree/registered_buffers. The libfuse implementation first tries registered buffers during registration and if this fails, will retry with non-registered buffers. This prevents having to add a new init flag (but does have the downside of printing dmesg errors for the failed registrations when trying the registered buffers). If using registered buffers and the daemon for whatever reason unregisters the buffers midway through, then this will sever server-kernel communication. Libfuse will never do this. Libfuse will only unregister the buffers when the entire session is being destroyed.
Benchmarks will be run and posted.
Thanks,
Joanne
v1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Add io_uring_cmd_import_fixed_full() patch
* Construct iter using io_uring_cmd_import_fixed_full() per cmd instead of recyling iters.
* Kmap the header instead of using bvec iter for iterating/copying. This makes the code easier to read.
Joanne Koong (8):
io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: add user_ prefix to userspace headers and payload fields
fuse: refactor setting up copy state for payload copying
fuse: support io-uring registered buffers
fs/fuse/dev_uring.c | 366 +++++++++++++++++++++++++---------- fs/fuse/dev_uring_i.h | 27 ++- include/linux/io_uring/cmd.h | 3 + io_uring/rsrc.c | 14 ++ io_uring/rsrc.h | 2 + io_uring/uring_cmd.c | 13 ++ 6 files changed, 316 insertions(+), 109 deletions(-)
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v2 1/8 io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Add an API for fetching the registered buffer associated with a io_uring cmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.
--- include/linux/io_uring/cmd.h | 3 +++ io_uring/rsrc.c | 14 ++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 13 +++++++++++++ 4 files changed, 32 insertions(+)
include/linux/io_uring/cmd.h +3 -0
@@ -43,6 +43,9 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,struct iov_iter *iter, struct io_uring_cmd *ioucmd, unsigned int issue_flags); +int io_uring_cmd_import_fixed_full (int rw,struct iov_iter *iter, +struct io_uring_cmd *ioucmd, +unsigned int issue_flags); int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, const struct iovec __user *uvec, size_t uvec_segs,
io_uring/rsrc.c +14 -0
@@ -1147,6 +1147,20 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_import_reg_buf_full (struct io_kiocb *req,struct iov_iter *iter, +int ddir,unsigned issue_flags) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + + node =io_find_buf_node (req, issue_flags); +if (!node) +return -EFAULT ; + + imu = node->buf; +return io_import_fixed (ddir, iter, imu, imu->ubuf, imu->len); +} +/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2) {
io_uring/rsrc.h +2 -0
@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_import_reg_buf_full (struct io_kiocb *req,struct iov_iter *iter, +int ddir,unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);
io_uring/uring_cmd.c +13 -0
@@ -292,6 +292,19 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed); +int io_uring_cmd_import_fixed_full (int rw,struct iov_iter *iter, +struct io_uring_cmd *ioucmd, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +if (WARN_ON_ONCE (!(ioucmd->flags &IORING_URING_CMD_FIXED ))) +return -EINVAL ; + +return io_import_reg_buf_full (req, iter, rw, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_full); + int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, const struct iovec __user *uvec, size_t uvec_segs,
Signature · 1 line
2.47.3
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 37 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Signed-off-by Joanne Koong <[email protected]>--- include/linux/io_uring/cmd.h | 3 +++ io_uring/rsrc.c | 14 ++++++++++++++ io_uring/rsrc.h | 2 ++ io_uring/uring_cmd.c | 13 +++++++++++++ 4 files changed, 32 insertions(+)include/linux/io_uring/cmd.h +3
-0@@ -43,6 +43,9 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,struct iov_iter *iter, struct io_uring_cmd *ioucmd, unsigned int issue_flags); +int io_uring_cmd_import_fixed_full (int rw,struct iov_iter *iter, +struct io_uring_cmd *ioucmd, +unsigned int issue_flags); int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, const struct iovec __user *uvec, size_t uvec_segs,io_uring/rsrc.c +2
-0@@ -1147,6 +1147,20 @@ int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } +int io_import_reg_buf_full (struct io_kiocb *req,struct iov_iter *iter, +int ddir,unsigned issue_flags)
io_uring/rsrc.c int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, · line 1161
Quoted 10 lines
io_uring/rsrc.c +10
-0+{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; + + node =io_find_buf_node (req, issue_flags); +if (!node) +return -EFAULT ; + + imu = node->buf; +return io_import_fixed (ddir, iter, imu, imu->ubuf, imu->len);
It's probably possible to avoid the logic in io_import_fixed() for checking the user address range against the registered buffer's range and offsetting the iov_iter, but that could always be done as future optimization work.
Quoted 45 lines
io_uring/rsrc.c +2
-0+} +/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2) {io_uring/rsrc.h +2
-0@@ -64,6 +64,8 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter,u64 buf_addr,size_t len,int ddir,unsigned issue_flags); +int io_import_reg_buf_full (struct io_kiocb *req,struct iov_iter *iter, +int ddir,unsigned issue_flags);int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);io_uring/uring_cmd.c +13
-1@@ -292,6 +292,19 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, }EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed); +int io_uring_cmd_import_fixed_full (int rw,struct iov_iter *iter, +struct io_uring_cmd *ioucmd, +unsigned int issue_flags) +{ +struct io_kiocb *req =cmd_to_io_kiocb (ioucmd); + +if (WARN_ON_ONCE (!(ioucmd->flags &IORING_URING_CMD_FIXED ))) +return -EINVAL ; + +return io_import_reg_buf_full (req, iter, rw, issue_flags); +} +EXPORT_SYMBOL_GPL (io_uring_cmd_import_fixed_full); + int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, const struct iovec __user *uvec, size_t uvec_segs, --2.47.3
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
In reply to Joanne KoongQuoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Signature · 1 line
Pavel Begunkov
Joanne Koong
maintainerRe: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.
Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.
Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return -EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.
Thanks,
Joanne
Quoted 3 lines
--
Pavel Begunkov
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 6 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.
I don't think it would be a huge issue to let userspace allocate a large buffer and to distribute it among requests - there is nothing in the kernel side to be done for that?
(I think I had even done that for the 1st io-uring patches and removed it because there were other issues and I wanted to keep the initial code simple).
Thanks,
Bernd
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 31 lines · Bernd Schubert
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 6 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks? My worry with this is that it would lead to suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.
Quoted 3 lines
I don't think it would be a huge issue to let userspace allocate a large buffer and to distribute it among requests - there is nothing in the kernel side to be done for that?
You can, but unless I missed something with this patchset you'd need to register it N times, which is not terrible but have memory overaccounting problems and feels less flexible.
Quoted 3 lines
(I think I had even done that for the 1st io-uring patches and removed it because there were other issues and I wanted to keep the initial code simple).
Signature · 1 line
Pavel Begunkov
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
In reply to Joanne KoongQuoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need? io_uring caps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.
In either case, I don't have a problem with this patch, just found it a bit off.
Quoted 3 lines
My worry with this is that it would lead to
suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bit
It wouldn't affect locality of the user buffers, that depends on the user space implementation. Are you sharing an io_uring instance between threads?
Quoted 14 lines
simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.
Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return
-EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.
The devil is in details, i.e. synchronisation. Taking a long term node reference might be fine. Does this change the uapi for this patchset? If not, I'd do it as a follow up. It also sounds like you can apply this optimisation regardless of whether you take a full registered buffer or go with sub ranges.
Signature · 1 line
Pavel Begunkov
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need?
io_uringcaps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.
Quoted 2 lines
In either case, I don't have a problem with this patch, just found it a bit off.
Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.
The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 46 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need?
io_uringcaps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.Quoted 2 lines
In either case, I don't have a problem with this patch, just found it a bit off.
Quoted 8 lines
Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.
The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.
With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to io_uring_prep_rw() and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).
Thanks,
Joanne
Quoted 2 lines
Thanks,
Bernd
Re: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 56 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need?
io_uringcaps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.Quoted 2 lines
In either case, I don't have a problem with this patch, just found it a bit off.
Quoted 8 lines
Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.
The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.
Quoted 5 lines
With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to
io_uring_prep_rw()and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).
Ah that is good to know, maybe worth to be mentioned to the commit message.
Btw, I will start to work on libfuse around next week to add another io-uring interface, so that the application can own the ring and let libfuse submit and fetch from it. I.e. that way the same ring can be used for libfuse and application IO.
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
Quoted 56 lines · Bernd Schubert
Quoted 54 lines · Joanne Koong
Quoted 52 lines · Bernd Schubert
Quoted 39 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need?
io_uringcaps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.Quoted 2 lines
In either case, I don't have a problem with this patch, just found it a bit off.
Maybe we could address that later on, so far I don't like the idea of a single buffer size for all ring entries. Maybe it would make sense to introduce buffer pools of different sizes and let ring entries use a needed buffer size dynamically.
The part I'm still not too happy about is the need for fuse server changes - my alternative patch didn't need that at all.
Quoted 7 lines
Quoted 5 lines
With pinning through io-uring registered buffers, this lets us also automatically use pinned pages for writing it out (eg if we're writing it out to local disk, we can pass that sqe directly to
io_uring_prep_rw()and since it's marked as a registered buffer in io uring, it'll skip that pinning/translation overhead).Ah that is good to know, maybe worth to be mentioned to the commit message.
Will do. I will add this to the commit message.
Quoted 5 lines
Btw, I will start to work on libfuse around next week to add another io-uring interface, so that the application can own the ring and let libfuse submit and fetch from it. I.e. that way the same ring can be used for libfuse and application IO.
Sounds great! Looking forward to the changes.
Thanks,
Joanne
Quoted 3 lines
Thanks,
Bernd
Joanne Koong
maintainerRe: [PATCH v2 1/8] io_uring/uring_cmd: add io_uring_cmd_import_fixed_full()
In reply to Pavel BegunkovQuoted 28 lines · Pavel Begunkov
Quoted 23 lines · Joanne Koong
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Joanne Koong
Add an API for fetching the registered buffer associated with a
io_uringcmd. This is useful for callers who need access to the buffer but do not have prior knowledge of the buffer's user address or length.Quoted 6 lines
Joanne, is it needed because you don't want to pass {offset,size} via fuse uapi? It's often more convenient to allocate and register one large buffer and let requests to use subchunks. Shouldn't be different for performance, but e.g. if you try to overlay it onto huge pages it'll be severely overaccounted.
Hi Pavel,
Yes, I was thinking this would be a simpler interface than the userspace caller having to pass in the uaddr and size on every request. Right now the way it is structured is that userspace allocates a buffer per request, then registers all those buffers. On the kernel side when it fetches the buffer, it'll always fetch the whole buffer (eg offset is 0 and size is the full size).
Quoted 2 lines
Do you think it is better to allocate one large buffer and have the requests use subchunks?
Quoted 11 lines
I think so, but that's general advice, I don't know the fuse implementation details, and it's not a strong opinion. It'll be great if you take a look at what other server implementations might want and do, and if whether this approach is flexible enough, and how amendable it is if you change it later on. E.g. how many registered buffers it might need?
io_uringcaps it at some 1000s. How large buffers are? Each separate buffer has memory footprint. And because of the same footprint there might be cache misses as well if there are too many. Can you always predict the max number of buffers to avoid resizing the table? Do you ever want to use huge pages while being restricted by mlock limits? And so on.
Thanks for your thoughts, I'll think about this some more.
Quoted 11 lines
In either case, I don't have a problem with this patch, just found it a bit off.
Quoted 3 lines
My worry with this is that it would lead to
suboptimal cache locality when servers offload handling requests to separate thread workers. From a code perspective it seems a bitIt wouldn't affect locality of the user buffers, that depends on the user space implementation. Are you sharing an
io_uringinstance between threads?
For request offloading, the different threads would share the io uring instance. When the main thread that does the io_uring_wait_cqe() receives a cqe, it'll dispatch the cqe to a worker thread to fulfill while it then looks at the next cqe to send off to the next worker thread and so on.
If there's one registered buffer that maps to the one big buffer allocated by the server for the ring, then each worker thread might be executing on different cpus when it accesses that buffer, which seems like that could lead to cache line bouncing. Or at least that's the scenario I was thinking of for suboptimal cache locality with the one big buffer. But I do like how a big buffer would save on the memory overhead that would be internally incurred in iouring for tracking every registered buffer.
Quoted 15 lines
Quoted 4 lines
simpler to have each request have its own buffer, but it wouldn't be much more complicated to have it all be part of one large buffer.
Quoted 11 lines
Right now, we are fetching the bvec iter every time there's a request because of the possibility that the buffer might have been unregistered (libfuse will not do this, but some other rogue userspace program could). If we added a flag to tell io uring that attempts at unregistration should return
-EBUSY, then we could just fetch the bvec iter once and use that for the lifetime of the server connection instead of having to fetch it every request, and then when the connection is aborted, we could unset the flag so that userspace can then successfully unregister their buffers. Do you think this is a good idea to have in io-uring? If this is fine to add then I'll add this to v3.
Quoted 5 lines
The devil is in details, i.e. synchronisation. Taking a long term node reference might be fine. Does this change the uapi for this patchset? If not, I'd do it as a follow up. It also sounds like you can apply this optimisation regardless of whether you take a full registered buffer or go with sub ranges.
Thanks, this is helpful.
Quoted 4 lines
--
Pavel Begunkov
Joanne Koong
maintainerPATCH v2 2/8 fuse: refactor io-uring logic for getting next fuse request
In reply to Joanne KoongSimplify the logic for getting the next fuse request.
--- fs/fuse/dev_uring.c | 78 ++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 50 deletions(-)
fs/fuse/dev_uring.c +28 -50
@@ -710,34 +710,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,return err; } -/* -* Write data to the ring buffer and send the request to userspace, -* userspace will read it -* This is comparable with classical read(/dev/fuse) -*/ -static int fuse_uring_send_next_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req, -unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; -int err; -struct io_uring_cmd *cmd; - - err =fuse_uring_prepare_send (ent, req); -if (err) -return err; - -spin_lock (&queue->lock); - cmd = ent->cmd; - ent->cmd =NULL ; - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd,0 , issue_flags); -return 0 ; -} -/* * Make a ring entry available for fuse_req assignment */
@@ -834,11 +806,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, }/* -* Get the next fuse req and send it +* Get the next fuse req. +* +* Returns true if the next fuse request has been assigned to the ent. +* Else, there is no next fuse request and this returns false. */ -static void fuse_uring_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue, -unsigned int issue_flags) +static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, +struct fuse_ring_queue *queue) {int err;struct fuse_req *req;
@@ -850,10 +824,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_send_next_to_ring (ent, req, issue_flags); + err =fuse_uring_prepare_send (ent, req);if (err) goto retry; } + +return req !=NULL ; }static int fuse_ring_ent_set_commit (struct fuse_ring_ent *ent)
@@ -871,6 +847,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)return 0 ; } +static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, +ssize_t ret,unsigned int issue_flags) +{ +struct fuse_ring_queue *queue = ent->queue; + +spin_lock (&queue->lock); + ent->state =FRRS_USERSPACE ; +list_move_tail (&ent->list, &queue->ent_in_userspace); + ent->cmd =NULL ; +spin_unlock (&queue->lock); + +io_uring_cmd_done (cmd, ret, issue_flags); +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_conn *fc)
@@ -942,7 +932,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -fuse_uring_next_fuse_req (ent, queue, issue_flags); +if (fuse_uring_get_next_fuse_req (ent, queue)) +fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1190,20 +1181,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)return -EIOCBQUEUED ; } -static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, -ssize_t ret,unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; - -spin_lock (&queue->lock); - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); - ent->cmd =NULL ; -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd, ret, issue_flags); -} -/* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission
@@ -1219,8 +1196,9 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,if (!(issue_flags &IO_URING_F_TASK_DEAD )) { err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -fuse_uring_next_fuse_req (ent, queue, issue_flags); -return ; +if (!fuse_uring_get_next_fuse_req (ent, queue)) +return ; + err =0 ; } }else { err = -ECANCELED ;
Signature · 1 line
2.47.3
Re: [PATCH v2 2/8] fuse: refactor io-uring logic for getting next fuse request
Quoted 132 lines · Joanne Koong
Simplify the logic for getting the next fuse request.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 78 ++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 50 deletions(-)fs/fuse/dev_uring.c +25
-48@@ -710,34 +710,6 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,return err; } -/* -* Write data to the ring buffer and send the request to userspace, -* userspace will read it -* This is comparable with classical read(/dev/fuse) -*/ -static int fuse_uring_send_next_to_ring (struct fuse_ring_ent *ent, -struct fuse_req *req, -unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; -int err; -struct io_uring_cmd *cmd; - - err =fuse_uring_prepare_send (ent, req); -if (err) -return err; - -spin_lock (&queue->lock); - cmd = ent->cmd; - ent->cmd =NULL ; - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd,0 , issue_flags); -return 0 ; -} -/* * Make a ring entry available for fuse_req assignment */ @@ -834,11 +806,13 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, }/* -* Get the next fuse req and send it +* Get the next fuse req. +* +* Returns true if the next fuse request has been assigned to the ent. +* Else, there is no next fuse request and this returns false. */ -static void fuse_uring_next_fuse_req (struct fuse_ring_ent *ent, -struct fuse_ring_queue *queue, -unsigned int issue_flags) +static bool fuse_uring_get_next_fuse_req (struct fuse_ring_ent *ent, +struct fuse_ring_queue *queue) {int err;struct fuse_req *req;@@ -850,10 +824,12 @@ static void fuse_uring_next_fuse_req(struct fuse_ring_ent *ent,spin_unlock (&queue->lock);if (req) { - err =fuse_uring_send_next_to_ring (ent, req, issue_flags); + err =fuse_uring_prepare_send (ent, req);if (err) goto retry; } + +return req !=NULL ; }static int fuse_ring_ent_set_commit (struct fuse_ring_ent *ent)@@ -871,6 +847,20 @@ static int fuse_ring_ent_set_commit(struct fuse_ring_ent *ent)return 0 ; } +static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, +ssize_t ret,unsigned int issue_flags) +{ +struct fuse_ring_queue *queue = ent->queue; + +spin_lock (&queue->lock); + ent->state =FRRS_USERSPACE ; +list_move_tail (&ent->list, &queue->ent_in_userspace); + ent->cmd =NULL ; +spin_unlock (&queue->lock); + +io_uring_cmd_done (cmd, ret, issue_flags); +} +/* FUSE_URING_CMD_COMMIT_AND_FETCH handler */ static int fuse_uring_commit_fetch (struct io_uring_cmd *cmd,int issue_flags,struct fuse_conn *fc)@@ -942,7 +932,8 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -fuse_uring_next_fuse_req (ent, queue, issue_flags); +if (fuse_uring_get_next_fuse_req (ent, queue)) +fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }@@ -1190,20 +1181,6 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)return -EIOCBQUEUED ; } -static void fuse_uring_send (struct fuse_ring_ent *ent,struct io_uring_cmd *cmd, -ssize_t ret,unsigned int issue_flags) -{ -struct fuse_ring_queue *queue = ent->queue; - -spin_lock (&queue->lock); - ent->state =FRRS_USERSPACE ; -list_move_tail (&ent->list, &queue->ent_in_userspace); - ent->cmd =NULL ; -spin_unlock (&queue->lock); - -io_uring_cmd_done (cmd, ret, issue_flags); -} -/* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission @@ -1219,8 +1196,9 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd,if (!(issue_flags &IO_URING_F_TASK_DEAD )) {
fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1204
Quoted 10 lines
fs/fuse/dev_uring.c +3
-2err =fuse_uring_prepare_send (ent, ent->fuse_req);if (err) { -fuse_uring_next_fuse_req (ent, queue, issue_flags); -return ; +if (!fuse_uring_get_next_fuse_req (ent, queue)) +return ; + err =0 ; } }else { err = -ECANCELED ;
LGTM
(I had already start to go into this direction
https://lore.kernel.org/all/[email protected]/)
Joanne Koong
maintainerPATCH v2 3/8 fuse: refactor io-uring header copying to ring
In reply to Joanne KoongMove header copying to ring logic into a new copy_header_to_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)
fs/fuse/dev_uring.c +20 -18
@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int copy_header_to_ring (void __user *ring,const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;
@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err =copy_to_user (&ent->headers->in_out, &req->in.h, -sizeof (req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Signature · 1 line
2.47.3
Re: [PATCH v2 3/8] fuse: refactor io-uring header copying to ring
Quoted 66 lines · Joanne Koong
Move header copying to ring logic into a new
copy_header_to_ring()function. This consolidates error handling.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)fs/fuse/dev_uring.c +18
-11@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int copy_header_to_ring (void __user *ring,const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err = copy_to_user(&ent->headers->in_out, &req->in.h,
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 699
Quoted 10 lines
fs/fuse/dev_uring.c +2
-7- sizeof(req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); }
This will give Miklos a bit headache, because of a merge conflict with
https://lore.kernel.org/r/[email protected]
Any chance you could rebase your series on this patch?
Thanks,
Bernd
Quoted 2 lines
fs/fuse/dev_uring.c +0
-0static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Joanne Koong
maintainerRe: [PATCH v2 3/8] fuse: refactor io-uring header copying to ring
Quoted 81 lines · Bernd Schubert
Quoted 66 lines · Joanne Koong
Move header copying to ring logic into a new
copy_header_to_ring()function. This consolidates error handling.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)fs/fuse/dev_uring.c +18
-11@@ -574,6 +574,17 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static int copy_header_to_ring (void __user *ring,const void *header, +size_t header_size) +{ +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)@@ -634,13 +645,11 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_to_user (&ent->headers->op_in, in_args->value, - in_args->size); -if (err) { -pr_info_ratelimited ( -"Copying the header failed.\n" ); -return -EFAULT ; - } + err =copy_header_to_ring (&ent->headers->op_in, + in_args->value, + in_args->size); +if (err) +return err; } in_args++; num_args--;@@ -655,9 +664,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; - err =copy_to_user (&ent->headers->ring_ent_in_out, &ent_in_out, -sizeof (ent_in_out)); -return err ? -EFAULT :0 ; +return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -686,14 +694,8 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ - err = copy_to_user(&ent->headers->in_out, &req->in.h,fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 699
Quoted 10 lines
fs/fuse/dev_uring.c +2
-7- sizeof(req->in.h)); -if (err) { - err = -EFAULT ; -return err; - } - -return 0 ; +return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +sizeof (req->in.h)); }This will give Miklos a bit headache, because of a merge conflict with
Quoted 3 lines
https://lore.kernel.org/r/[email protected]
Any chance you could rebase your series on this patch?
Definitely, I will do that for v3. I saw that you submitted a bunch of io-uring patches earlier last week alongside this one - I haven't gotten around to reviewing those but I will do so.
Thanks,
Joanne
Quoted 8 lines
Thanks,
BerndQuoted 2 lines
fs/fuse/dev_uring.c +0
-0static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
Joanne Koong
maintainerPATCH v2 4/8 fuse: refactor io-uring header copying from ring
In reply to Joanne KoongMove header copying from ring logic into a new copy_header_from_ring() function. This consolidates error handling.
--- fs/fuse/dev_uring.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
fs/fuse/dev_uring.c +17 -6
@@ -585,6 +585,17 @@ static int copy_header_to_ring(void __user *ring, const void *header,return 0 ; } +static int copy_header_from_ring (void *header,const void __user *ring, +size_t header_size) +{ +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -595,10 +606,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_from_user (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, +sizeof (ring_in_out));if (err) -return -EFAULT ; +return err; err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, &iter);
@@ -789,10 +800,10 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_from_user (&req->out.h, &ent->headers->in_out, -sizeof (req->out.h)); + err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, +sizeof (req->out.h));if (err) { - req->out.h.error = -EFAULT ; + req->out.h.error = err; goto out; }
Signature · 1 line
2.47.3
Joanne Koong
maintainerPATCH v2 5/8 fuse: use enum types for header copying
In reply to Joanne KoongUse enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.
--- fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-)
fs/fuse/dev_uring.c +45 -10
@@ -31,6 +31,12 @@ struct fuse_uring_pdu {static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +FUSE_URING_HEADER_IN_OUT , +FUSE_URING_HEADER_OP , +FUSE_URING_HEADER_RING_ENT , +}; +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {
@@ -574,9 +580,31 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } -static int copy_header_to_ring (void __user *ring,const void *header, -size_t header_size) +static void __user *get_user_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} + +static int copy_header_to_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +const void *header,size_t header_size) { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) {pr_info_ratelimited ("Copying header to ring failed.\n" );return -EFAULT ;
@@ -585,9 +613,15 @@ static int copy_header_to_ring(void __user *ring, const void *header,return 0 ; } -static int copy_header_from_ring (void *header,const void __user *ring, -size_t header_size) +static int copy_header_from_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +void *header,size_t header_size) { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) {pr_info_ratelimited ("Copying header from ring failed.\n" );return -EFAULT ;
@@ -606,8 +640,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ring_in_out,sizeof (ring_in_out));if (err)return err;
@@ -656,7 +690,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_header_to_ring (&ent->headers->op_in, + err =copy_header_to_ring (ent,FUSE_URING_HEADER_OP , in_args->value, in_args->size);if (err)
@@ -675,7 +709,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; -return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ent_in_out,sizeof (ent_in_out)); }
@@ -705,7 +740,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ -return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +return copy_header_to_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->in.h,sizeof (req->in.h)); }
@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;
Signature · 1 line
2.47.3
Re: [PATCH v2 5/8] fuse: use enum types for header copying
Quoted 14 lines · Joanne Koong
Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-)fs/fuse/dev_uring.c +0
-0
fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35
Quoted 5 lines
fs/fuse/dev_uring.c +2
-0@@ -31,6 +31,12 @@ struct fuse_uring_pdu {static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +FUSE_URING_HEADER_IN_OUT ,
In post review of my own names, headers->in_out is rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_ FUSE_URING_HEADER_MSG_IN_OUT?
Quoted 99 lines
fs/fuse/dev_uring.c +42
-9+FUSE_URING_HEADER_OP , + FUSE_URING_HEADER_RING_ENT, +}; +static void uring_cmd_set_ring_ent (struct io_uring_cmd *cmd,struct fuse_ring_ent *ring_ent) {@@ -574,9 +580,31 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } -static int copy_header_to_ring (void __user *ring,const void *header, -size_t header_size) +static void __user *get_user_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} + +static int copy_header_to_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +const void *header,size_t header_size) { +void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) {pr_info_ratelimited ("Copying header to ring failed.\n" );return -EFAULT ;@@ -585,9 +613,15 @@ static int copy_header_to_ring(void __user *ring, const void *header,return 0 ; } -static int copy_header_from_ring (void *header,const void __user *ring, -size_t header_size) +static int copy_header_from_ring (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type, +void *header,size_t header_size) { +const void __user *ring =get_user_ring_header (ent, type); + +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) {pr_info_ratelimited ("Copying header from ring failed.\n" );return -EFAULT ;@@ -606,8 +640,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,int err;struct fuse_uring_ent_in_out ring_in_out; - err =copy_header_from_ring (&ring_in_out, &ent->headers->ring_ent_in_out, -sizeof (ring_in_out)); + err =copy_header_from_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ring_in_out,sizeof (ring_in_out));if (err)return err;@@ -656,7 +690,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,* Some op code have that as zero size. */ if (args->in_args[0 ].size >0 ) { - err =copy_header_to_ring (&ent->headers->op_in, + err =copy_header_to_ring (ent,FUSE_URING_HEADER_OP , in_args->value, in_args->size);if (err)@@ -675,7 +709,8 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; -return copy_header_to_ring (&ent->headers->ring_ent_in_out, &ent_in_out, +return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , + &ent_in_out,sizeof (ent_in_out)); }@@ -705,7 +740,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, }/* copy fuse_in_header */ -return copy_header_to_ring (&ent->headers->in_out, &req->in.h, +return copy_header_to_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->in.h,sizeof (req->in.h)); }
fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841
Quoted 8 lines
fs/fuse/dev_uring.c +1
-1@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ; - err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;
Joanne Koong
maintainerRe: [PATCH v2 5/8] fuse: use enum types for header copying
Quoted 17 lines · Bernd Schubert
Quoted 15 lines · Joanne Koong
Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-)fs/fuse/dev_uring.c +0
-0@@ -31,6 +31,12 @@ struct fuse_uring_pdu {
Quoted 8 lines
fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35
Quoted 4 lines
fs/fuse/dev_uring.c +2
-0static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +FUSE_URING_HEADER_IN_OUT ,In post review of my own names,
headers->in_outis rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_FUSE_URING_HEADER_MSG_IN_OUT?
Ahh I personally find "msg" a bit more confusing because "message" makes me think it refers just to the payload since the whole thing is usually called the request. So if we had to rename it, maybe FUSE_URING_HEADER_REQ_IN_OUT? Though I do like your original naming of it, FUSE_URING_HEADER_IN_OUT since FUSE_URING_FUSE_HEADER_IN_OUT sounds a little redundant.
I'll add some comments on top of this too, eg "/*struct fuse_in_header / struct_fuse_out_header */, to clarify.
Quoted 8 lines
Quoted 8 lines
fs/fuse/dev_uring.c +3
-0+FUSE_URING_HEADER_OP , + FUSE_URING_HEADER_RING_ENT, +};@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ;
Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841
Quoted 5 lines
fs/fuse/dev_uring.c +1
-1- err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;Reviewed-by Bernd Schubert <[email protected]>
Thanks for reviewing the patches!
Bernd Schubert
maintainerRe: [PATCH v2 5/8] fuse: use enum types for header copying
Quoted 27 lines · Joanne Koong
Quoted 17 lines · Bernd Schubert
Quoted 15 lines · Joanne Koong
Use enum types to identify which part of the header needs to be copied. This improves the interface and will simplify both kernel-space and user-space header addresses when fixed buffer support is added.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-)fs/fuse/dev_uring.c +0
-0@@ -31,6 +31,12 @@ struct fuse_uring_pdu {Quoted 8 lines
fs/fuse/dev_uring.c struct fuse_uring_pdu { · line 35
Quoted 4 lines
fs/fuse/dev_uring.c +2
-0static const struct fuse_iqueue_ops fuse_io_uring_ops; +enum fuse_uring_header_type { +FUSE_URING_HEADER_IN_OUT ,In post review of my own names,
headers->in_outis rather hard to understand, I would have probably chosen "msg_in_out" now. With that _maybe_FUSE_URING_HEADER_MSG_IN_OUT?
Quoted 6 lines
Ahh I personally find "msg" a bit more confusing because "message" makes me think it refers just to the payload since the whole thing is usually called the request. So if we had to rename it, maybe
FUSE_URING_HEADER_REQ_IN_OUT? Though I do like your original naming of it,FUSE_URING_HEADER_IN_OUTsinceFUSE_URING_FUSE_HEADER_IN_OUTsounds a little redundant.
FUSE_URING_HEADER_REQ_IN_OUT sounds nice to me. Renaming was just a suggestion. Let's keep the current name if you prefer that.
Quoted 3 lines
I'll add some comments on top of this too, eg "/*struct
fuse_in_header/struct_fuse_out_header*/, to clarify.
Comments are always helpful :)
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
fs/fuse/dev_uring.c +3
-0+FUSE_URING_HEADER_OP , + FUSE_URING_HEADER_RING_ENT, +};@@ -800,7 +835,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,struct fuse_conn *fc = ring->fc;ssize_t err =0 ;Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, · line 841
Quoted 5 lines
fs/fuse/dev_uring.c +1
-1- err =copy_header_from_ring (&req->out.h, &ent->headers->in_out, + err =copy_header_from_ring (ent,FUSE_URING_HEADER_IN_OUT , &req->out.h,sizeof (req->out.h));if (err) { req->out.h.error = err;Reviewed-by Bernd Schubert <[email protected]>Thanks for reviewing the patches!
Joanne Koong
maintainerPATCH v2 6/8 fuse: add user_ prefix to userspace headers and payload fields
In reply to Joanne KoongRename the headers and payload fields to user_headers and user_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.
--- fs/fuse/dev_uring.c | 17 ++++++++--------- fs/fuse/dev_uring_i.h | 4 ++-- 2 files changed, 10 insertions(+), 11 deletions(-)
fs/fuse/dev_uring.c +8 -9
@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent, {switch (type) {case FUSE_URING_HEADER_IN_OUT : -return &ent->headers->in_out; +return &ent->user_headers->in_out;case FUSE_URING_HEADER_OP : -return &ent->headers->op_in; +return &ent->user_headers->op_in;case FUSE_URING_HEADER_RING_ENT : -return &ent->headers->ring_ent_in_out; +return &ent->user_headers->ring_ent_in_out; }WARN_ON_ONCE (1 );
@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, + err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, &iter);if (err)return err;
@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); + err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter);if (err) {pr_info_ratelimited ("fuse: Import of user buffer failed\n" );return err;
@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, ent_in_out.payload_sz = cs.ring.copied_sz;return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , - &ent_in_out, -sizeof (ent_in_out)); + &ent_in_out,sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,INIT_LIST_HEAD (&ent->list); ent->queue = queue; - ent->headers = iov[0 ].iov_base; - ent->payload = iov[1 ].iov_base; + ent->user_headers = iov[0 ].iov_base; + ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent;
fs/fuse/dev_uring_i.h +2 -2
@@ -39,8 +39,8 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent {/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload;/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
Signature · 1 line
2.47.3
Re: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields
Quoted 50 lines · Joanne Koong
Rename the headers and payload fields to
user_headersanduser_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 17 ++++++++--------- fs/fuse/dev_uring_i.h | 4 ++-- 2 files changed, 10 insertions(+), 11 deletions(-)fs/fuse/dev_uring.c +5
-5@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent, {switch (type) {case FUSE_URING_HEADER_IN_OUT : -return &ent->headers->in_out; +return &ent->user_headers->in_out;case FUSE_URING_HEADER_OP : -return &ent->headers->op_in; +return &ent->user_headers->op_in;case FUSE_URING_HEADER_RING_ENT : -return &ent->headers->ring_ent_in_out; +return &ent->user_headers->ring_ent_in_out; }WARN_ON_ONCE (1 );@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, + err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, &iter);if (err)return err;@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); + err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter);if (err) {pr_info_ratelimited ("fuse: Import of user buffer failed\n" );return err;
fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 713
Quoted 6 lines
fs/fuse/dev_uring.c +1
-2@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, ent_in_out.payload_sz = cs.ring.copied_sz;return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , - &ent_in_out, -sizeof (ent_in_out)); + &ent_in_out,sizeof (ent_in_out));
nit: looks like an unnecessary formatting change here. Either drop it or move it to the earlier commit that added these lines?
Best,
Caleb
Quoted 32 lines
fs/fuse/dev_uring.c +2
-2} static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,INIT_LIST_HEAD (&ent->list); ent->queue = queue; - ent->headers = iov[0 ].iov_base; - ent->payload = iov[1 ].iov_base; + ent->user_headers = iov[0 ].iov_base; + ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent;fs/fuse/dev_uring_i.h +2
-3@@ -39,8 +39,8 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent {/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload;/* the ring queue that owns the request */ struct fuse_ring_queue *queue; --2.47.3
Joanne Koong
maintainerRe: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields
Quoted 53 lines · Caleb Sander Mateos
Quoted 51 lines · Joanne Koong
Rename the headers and payload fields to
user_headersanduser_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 17 ++++++++--------- fs/fuse/dev_uring_i.h | 4 ++-- 2 files changed, 10 insertions(+), 11 deletions(-)fs/fuse/dev_uring.c +5
-5@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent, {switch (type) {case FUSE_URING_HEADER_IN_OUT : -return &ent->headers->in_out; +return &ent->user_headers->in_out;case FUSE_URING_HEADER_OP : -return &ent->headers->op_in; +return &ent->user_headers->op_in;case FUSE_URING_HEADER_RING_ENT : -return &ent->headers->ring_ent_in_out; +return &ent->user_headers->ring_ent_in_out; }WARN_ON_ONCE (1 );@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, + err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, &iter);if (err)return err;@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); + err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter);if (err) {pr_info_ratelimited ("fuse: Import of user buffer failed\n" );return err;@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
Quoted 8 lines
fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 713
Quoted 5 lines
fs/fuse/dev_uring.c +1
-2ent_in_out.payload_sz = cs.ring.copied_sz;return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , - &ent_in_out, -sizeof (ent_in_out)); + &ent_in_out,sizeof (ent_in_out));nit: looks like an unnecessary formatting change here. Either drop it or move it to the earlier commit that added these lines?
I will move this to the earlier commit that added this.
Thanks,
Joanne
Quoted 36 lines
Best,
CalebQuoted 32 lines
fs/fuse/dev_uring.c +2
-2} static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,INIT_LIST_HEAD (&ent->list); ent->queue = queue; - ent->headers = iov[0 ].iov_base; - ent->payload = iov[1 ].iov_base; + ent->user_headers = iov[0 ].iov_base; + ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent;fs/fuse/dev_uring_i.h +2
-3@@ -39,8 +39,8 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent {/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload;/* the ring queue that owns the request */ struct fuse_ring_queue *queue; --2.47.3
Bernd Schubert
maintainerRe: [PATCH v2 6/8] fuse: add user_ prefix to userspace headers and payload fields
In reply to Joanne KoongQuoted 74 lines · Joanne Koong
Rename the headers and payload fields to
user_headersanduser_payload. This makes it explicit that these pointers reference userspace addresses and prepares for upcoming fixed buffer support, where there will be separate fields for kernel-space pointers to the payload and headers.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 17 ++++++++--------- fs/fuse/dev_uring_i.h | 4 ++-- 2 files changed, 10 insertions(+), 11 deletions(-)fs/fuse/dev_uring.c +8
-9@@ -585,11 +585,11 @@ static void __user *get_user_ring_header(struct fuse_ring_ent *ent, {switch (type) {case FUSE_URING_HEADER_IN_OUT : -return &ent->headers->in_out; +return &ent->user_headers->in_out;case FUSE_URING_HEADER_OP : -return &ent->headers->op_in; +return &ent->user_headers->op_in;case FUSE_URING_HEADER_RING_ENT : -return &ent->headers->ring_ent_in_out; +return &ent->user_headers->ring_ent_in_out; }WARN_ON_ONCE (1 );@@ -645,7 +645,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->payload, ring->max_payload_sz, + err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, &iter);if (err)return err;@@ -674,7 +674,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->payload, ring->max_payload_sz, &iter); + err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter);if (err) {pr_info_ratelimited ("fuse: Import of user buffer failed\n" );return err;@@ -710,8 +710,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, ent_in_out.payload_sz = cs.ring.copied_sz;return copy_header_to_ring (ent,FUSE_URING_HEADER_RING_ENT , - &ent_in_out, -sizeof (ent_in_out)); + &ent_in_out,sizeof (ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,@@ -1104,8 +1103,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,INIT_LIST_HEAD (&ent->list); ent->queue = queue; - ent->headers = iov[0 ].iov_base; - ent->payload = iov[1 ].iov_base; + ent->user_headers = iov[0 ].iov_base; + ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent;fs/fuse/dev_uring_i.h +0
-0
fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 46
Quoted 10 lines
fs/fuse/dev_uring_i.h +2
-2@@ -39,8 +39,8 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent {/* userspace buffer */ -struct fuse_uring_req_header __user *headers; -void __user *payload; +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload;/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
Joanne Koong
maintainerPATCH v2 7/8 fuse: refactor setting up copy state for payload copying
In reply to Joanne KoongAdd a new helper function setup_fuse_copy_state() to contain the logic for setting up the copy state for payload copying.
--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-)
fs/fuse/dev_uring.c +25 -14
@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; } +static int setup_fuse_copy_state (struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent * ent,int rw, +struct iov_iter *iter, +struct fuse_copy_state *cs) +{ +int err; + + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } + +fuse_copy_init (cs, rw ==ITER_DEST , iter); + + cs->is_uring = true; + cs->req = req; + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)
@@ -645,15 +667,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, - &iter); + err =setup_fuse_copy_state (ring, req, ent,ITER_SOURCE , &iter, &cs);if (err)return err; -fuse_copy_init (&cs, false, &iter); - cs.is_uring = true; - cs.req = req; -return fuse_copy_out_args (&cs, args, ring_in_out.payload_sz); }
@@ -674,15 +691,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); + err =setup_fuse_copy_state (ring, req, ent,ITER_DEST , &iter, &cs); +if (err)return err; - } - -fuse_copy_init (&cs, true, &iter); - cs.is_uring = true; - cs.req = req;if (num_args >0 ) {/*
Signature · 1 line
2.47.3
Re: [PATCH v2 7/8] fuse: refactor setting up copy state for payload copying
Quoted 13 lines · Joanne Koong
Add a new helper function
setup_fuse_copy_state()to contain the logic for setting up the copy state for payload copying.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-)fs/fuse/dev_uring.c +0
-0
fs/fuse/dev_uring.c static int copy_header_from_ring(struct fuse_ring_ent *ent, · line 636
Quoted 7 lines
fs/fuse/dev_uring.c +4
-0@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; } +static int setup_fuse_copy_state (struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent * ent,int rw, +struct iov_iter *iter, +struct fuse_copy_state *cs)
Maybe 'struct fuse_copy_state *cs' as first argument? Just a suggestion and would good if it wouldn't end up in the middle when there are more arguments added at some point.
Quoted 46 lines
fs/fuse/dev_uring.c +20
-9+{ +int err; + + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } + +fuse_copy_init (cs, rw ==ITER_DEST , iter); + + cs->is_uring = true; + cs->req = req; + +return 0 ; +} +static int fuse_uring_copy_from_ring (struct fuse_ring *ring,struct fuse_req *req,struct fuse_ring_ent *ent)@@ -645,15 +667,10 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,if (err)return err; - err =import_ubuf (ITER_SOURCE , ent->user_payload, ring->max_payload_sz, - &iter); + err =setup_fuse_copy_state (ring, req, ent,ITER_SOURCE , &iter, &cs);if (err)return err; -fuse_copy_init (&cs, false, &iter); - cs.is_uring = true; - cs.req = req; -return fuse_copy_out_args (&cs, args, ring_in_out.payload_sz); }@@ -674,15 +691,9 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err =import_ubuf (ITER_DEST , ent->user_payload, ring->max_payload_sz, &iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); + err =setup_fuse_copy_state (ring, req, ent,ITER_DEST , &iter, &cs);
fs/fuse/dev_uring.c static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, · line 699
Quoted 10 lines
fs/fuse/dev_uring.c +1
-5+if (err)return err; - } - -fuse_copy_init (&cs, true, &iter); - cs.is_uring = true; - cs.req = req;if (num_args >0 ) {/*
Joanne Koong
maintainerRe: [PATCH v2 7/8] fuse: refactor setting up copy state for payload copying
Quoted 18 lines · Bernd Schubert
Quoted 16 lines · Joanne Koong
Add a new helper function
setup_fuse_copy_state()to contain the logic for setting up the copy state for payload copying.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-)fs/fuse/dev_uring.c +0
-0@@ -630,6 +630,28 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,return 0 ; }
Quoted 9 lines
fs/fuse/dev_uring.c static int copy_header_from_ring(struct fuse_ring_ent *ent, · line 636
Quoted 4 lines
fs/fuse/dev_uring.c +4
-0+static int setup_fuse_copy_state (struct fuse_ring *ring,struct fuse_req *req, +struct fuse_ring_ent * ent,int rw, +struct iov_iter *iter, +struct fuse_copy_state *cs)Maybe 'struct
fuse_copy_state*cs' as first argument? Just a suggestion and would good if it wouldn't end up in the middle when there are more arguments added at some point.
I like your suggestion a lot, I think that makes it clearer to read too :) I'll reorder this to be the first arg. Thanks!
Joanne Koong
maintainerPATCH v2 8/8 fuse: support io-uring registered buffers
In reply to Joanne KoongAdd support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the fuse_uring_req_header is offset from the end of the buffer.
To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)
fs/fuse/dev_uring.c +159 -41
@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static void *get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {
@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (ring, header, header_size); + } else { +void __user *ring =get_user_ring_header (ent, type); -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } } -return 0 ; }
@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, void *header, size_t header_size) { - const void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + const void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (header, ring, header_size); + } else { +const void __user *ring =get_user_ring_header (ent, type); -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } }return 0 ;
@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req, {int err; - err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, - iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (ent->fixed_buffer) { + *iter = ent->payload_iter; + }else { + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, rw ==ITER_DEST , iter);
@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ; + + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 ); +if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ; + +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start); +if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start; + +/* +* We can release the acquired reference on the header page immediately +* since the page is pinned and io_uring_cmd_import_fixed_full() +* prevents it from being unpinned while we are using it. +*/ +put_page (header_page); + +return 0 ; +} + +static void fuse_uring_unmap_buffer (struct fuse_ring_ent *ent) +{ +if (ent->fixed_buffer) +kunmap_local (ent->headers); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {
@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,unsigned int qid =READ_ONCE (cmd_req->qid);struct fuse_pqueue *fpq;struct fuse_req *req; +bool next_req; err = -ENOTCONN ;if (!ring)
@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, req, err); +return err; + } +fuse_uring_commit (ent, req, issue_flags);/*
@@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) + next_req =fuse_uring_get_next_fuse_req (ent, queue); +fuse_uring_unmap_buffer (ent); +if (next_req)fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }
@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err; + err = -ENOMEM ; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (err); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + +if (READ_ONCE (cmd->sqe->uring_cmd_flags) &IORING_URING_CMD_FIXED ) { + ent->fixed_buffer = true; +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->user_headers = iov[0 ].iov_base; ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/*
@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ; + } + + err =fuse_uring_prepare_send (ent, ent->fuse_req); +if (err) { + send_ent =fuse_uring_get_next_fuse_req (ent, queue); + err =0 ; } +fuse_uring_unmap_buffer (ent); -fuse_uring_send (ent, cmd, err, issue_flags); +if (send_ent) +fuse_uring_send (ent, cmd, err, issue_flags); }static struct fuse_ring_queue *fuse_uring_task_to_queue (struct fuse_ring *ring)
fs/fuse/dev_uring_i.h +24 -3
@@ -7,6 +7,7 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> #include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING
@@ -38,9 +39,29 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *user_headers; -void __user *user_payload; +/* +* If true, the buffer was pre-registered by the daemon and the +* pages backing it are pinned in kernel memory. The fixed buffer layout +* is: [payload][header at end]. Use payload_iter and headers for +* copying to/from the ring. +* +* Otherwise, use user_headers and user_payload which point to userspace +* addresses representing the ring memory. +*/ +bool fixed_buffer; + +union { +/* fixed_buffer == false */ +struct { +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload; + }; +/* fixed_buffer == true */ +struct { +struct fuse_uring_req_header *headers; +struct iov_iter payload_iter; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
Signature · 1 line
2.47.3
Quoted 226 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +118
-19@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static void *get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (ring, header, header_size); + } else { +void __user *ring =get_user_ring_header (ent, type); -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } } -return 0 ; }@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, void *header, size_t header_size) { - const void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + const void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (header, ring, header_size); + } else { +const void __user *ring =get_user_ring_header (ent, type); -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } }return 0 ;@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req, {int err; - err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, - iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (ent->fixed_buffer) { + *iter = ent->payload_iter; + }else { + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, rw ==ITER_DEST , iter);@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ; + + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 ); +if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ; + +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start); +if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start; + +/* +* We can release the acquired reference on the header page immediately +* since the page is pinned and io_uring_cmd_import_fixed_full() +* prevents it from being unpinned while we are using it. +*/ +put_page (header_page); + +return 0 ; +} + +static void fuse_uring_unmap_buffer (struct fuse_ring_ent *ent) +{ +if (ent->fixed_buffer) +kunmap_local (ent->headers); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,unsigned int qid =READ_ONCE (cmd_req->qid);struct fuse_pqueue *fpq;struct fuse_req *req; +bool next_req; err = -ENOTCONN ;if (!ring)@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, req, err); +return err; + } +fuse_uring_commit (ent, req, issue_flags);/* @@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) + next_req =fuse_uring_get_next_fuse_req (ent, queue); +fuse_uring_unmap_buffer (ent); +if (next_req)fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err;
fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1197
Quoted 10 lines
fs/fuse/dev_uring.c +10
-0+ err = -ENOMEM ; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (err); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + +if (READ_ONCE (cmd->sqe->uring_cmd_flags) &IORING_URING_CMD_FIXED ) {
Just use cmd->flags. That avoids having to deal with any possibility of userspace changing sqe-> uring_cmd_flags between the multiple loads of it.
Quoted 107 lines
fs/fuse/dev_uring.c +31
-22+ ent->fixed_buffer = true; +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->user_headers = iov[0 ].iov_base; ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/* @@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ; + } + + err =fuse_uring_prepare_send (ent, ent->fuse_req); +if (err) { + send_ent =fuse_uring_get_next_fuse_req (ent, queue); + err =0 ; } +fuse_uring_unmap_buffer (ent); -fuse_uring_send (ent, cmd, err, issue_flags); +if (send_ent) +fuse_uring_send (ent, cmd, err, issue_flags); }static struct fuse_ring_queue *fuse_uring_task_to_queue (struct fuse_ring *ring)fs/fuse/dev_uring_i.h +1
-3@@ -7,6 +7,7 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> #include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING @@ -38,9 +39,29 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *user_headers; -void __user *user_payload;
fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 51
Quoted 10 lines
fs/fuse/dev_uring_i.h +10
-0+/* +* If true, the buffer was pre-registered by the daemon and the +* pages backing it are pinned in kernel memory. The fixed buffer layout +* is: [payload][header at end]. Use payload_iter and headers for +* copying to/from the ring. +* +* Otherwise, use user_headers and user_payload which point to userspace +* addresses representing the ring memory. +*/ +bool fixed_buffer;
Could use cmd->flags instead of adding this field. It's an extra indirection vs. space tradeoff, I guess.
Best,
Caleb
Quoted 19 lines
fs/fuse/dev_uring_i.h +13
-1+ +union { +/* fixed_buffer == false */ +struct { +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload; + }; +/* fixed_buffer == true */ +struct { +struct fuse_uring_req_header *headers; +struct iov_iter payload_iter; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue; --2.47.3
Joanne Koong
maintainerQuoted 228 lines · Caleb Sander Mateos
Quoted 226 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +118
-19@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static void *get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (ring, header, header_size); + } else { +void __user *ring =get_user_ring_header (ent, type); -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } } -return 0 ; }@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, void *header, size_t header_size) { - const void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + const void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (header, ring, header_size); + } else { +const void __user *ring =get_user_ring_header (ent, type); -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } }return 0 ;@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req, {int err; - err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, - iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (ent->fixed_buffer) { + *iter = ent->payload_iter; + }else { + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, rw ==ITER_DEST , iter);@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ; + + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 ); +if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ; + +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start); +if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start; + +/* +* We can release the acquired reference on the header page immediately +* since the page is pinned and io_uring_cmd_import_fixed_full() +* prevents it from being unpinned while we are using it. +*/ +put_page (header_page); + +return 0 ; +} + +static void fuse_uring_unmap_buffer (struct fuse_ring_ent *ent) +{ +if (ent->fixed_buffer) +kunmap_local (ent->headers); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,unsigned int qid =READ_ONCE (cmd_req->qid);struct fuse_pqueue *fpq;struct fuse_req *req; +bool next_req; err = -ENOTCONN ;if (!ring)@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, req, err); +return err; + } +fuse_uring_commit (ent, req, issue_flags);/* @@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) + next_req =fuse_uring_get_next_fuse_req (ent, queue); +fuse_uring_unmap_buffer (ent); +if (next_req)fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err;
Quoted 14 lines
fs/fuse/dev_uring.c fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · line 1197
Quoted 10 lines
fs/fuse/dev_uring.c +10
-0+ err = -ENOMEM ; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (err); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + +if (READ_ONCE (cmd->sqe->uring_cmd_flags) &IORING_URING_CMD_FIXED ) {Just use
cmd->flags. That avoids having to deal with any possibility of userspace changing sqe->uring_cmd_flagsbetween the multiple loads of it.
Awesome, I'll switch this to just use cmd->flags.
Thank you for looking at the patches.
Quoted 144 lines
Quoted 108 lines
fs/fuse/dev_uring.c +31
-22+ ent->fixed_buffer = true; +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->user_headers = iov[0 ].iov_base; ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/* @@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ; + } + + err =fuse_uring_prepare_send (ent, ent->fuse_req); +if (err) { + send_ent =fuse_uring_get_next_fuse_req (ent, queue); + err =0 ; } +fuse_uring_unmap_buffer (ent); -fuse_uring_send (ent, cmd, err, issue_flags); +if (send_ent) +fuse_uring_send (ent, cmd, err, issue_flags); }static struct fuse_ring_queue *fuse_uring_task_to_queue (struct fuse_ring *ring)fs/fuse/dev_uring_i.h +1
-3@@ -7,6 +7,7 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> #include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING @@ -38,9 +39,29 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *user_headers; -void __user *user_payload;fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 51
Quoted 10 lines
fs/fuse/dev_uring_i.h +10
-0+/* +* If true, the buffer was pre-registered by the daemon and the +* pages backing it are pinned in kernel memory. The fixed buffer layout +* is: [payload][header at end]. Use payload_iter and headers for +* copying to/from the ring. +* +* Otherwise, use user_headers and user_payload which point to userspace +* addresses representing the ring memory. +*/ +bool fixed_buffer;Could use
cmd->flagsinstead of adding this field. It's an extra indirection vs. space tradeoff, I guess.Best,
CalebQuoted 19 lines
fs/fuse/dev_uring_i.h +13
-1+ +union { +/* fixed_buffer == false */ +struct { +struct fuse_uring_req_header __user *user_headers; +void __user *user_payload; + }; +/* fixed_buffer == true */ +struct { +struct fuse_uring_req_header *headers; +struct iov_iter payload_iter; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue; --2.47.3
Bernd Schubert
maintainerQuoted 141 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +63
-18@@ -580,6 +580,22 @@ static int fuse_uring_out_header_has_err(struct fuse_out_header *oh,return err; } +static void *get_kernel_ring_header (struct fuse_ring_ent *ent, +enum fuse_uring_header_type type) +{ +switch (type) { +case FUSE_URING_HEADER_IN_OUT : +return &ent->headers->in_out; +case FUSE_URING_HEADER_OP : +return &ent->headers->op_in; +case FUSE_URING_HEADER_RING_ENT : +return &ent->headers->ring_ent_in_out; + } + +WARN_ON_ONCE (1 ); +return NULL ; +} +static void __user *get_user_ring_header (struct fuse_ring_ent *ent,enum fuse_uring_header_type type) {@@ -600,16 +616,22 @@ static int copy_header_to_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, const void *header, size_t header_size) { - void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (ring, header, header_size); + } else { +void __user *ring =get_user_ring_header (ent, type); -if (copy_to_user (ring, header, header_size)) { -pr_info_ratelimited ("Copying header to ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_to_user (ring, header, header_size)) { +pr_info_ratelimited ("Copying header to ring failed.\n" ); +return -EFAULT ; + } } -return 0 ; }@@ -617,14 +639,21 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,enum fuse_uring_header_type type, void *header, size_t header_size) { - const void __user *ring = get_user_ring_header(ent, type); + if (ent->fixed_buffer) { + const void *ring = get_kernel_ring_header(ent, type); -if (!ring) -return -EINVAL ; +if (!ring) +return -EINVAL ; +memcpy (header, ring, header_size); + } else { +const void __user *ring =get_user_ring_header (ent, type); -if (copy_from_user (header, ring, header_size)) { -pr_info_ratelimited ("Copying header from ring failed.\n" ); -return -EFAULT ; +if (!ring) +return -EINVAL ; +if (copy_from_user (header, ring, header_size)) { +pr_info_ratelimited ("Copying header from ring failed.\n" ); +return -EFAULT ; + } }return 0 ;@@ -637,11 +666,15 @@ static int setup_fuse_copy_state(struct fuse_ring *ring, struct fuse_req *req, {int err; - err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, - iter); -if (err) { -pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); -return err; +if (ent->fixed_buffer) { + *iter = ent->payload_iter; + }else { + err =import_ubuf (rw, ent->user_payload, ring->max_payload_sz, + iter); +if (err) { +pr_info_ratelimited ("fuse: Import of user buffer failed\n" ); +return err; + } }fuse_copy_init (cs, rw ==ITER_DEST , iter);@@ -754,6 +787,62 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, sizeof(req->in.h)); } +/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header );
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 811
Quoted 10 lines
fs/fuse/dev_uring.c +10
-0+struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ; + + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );
This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817
Quoted 6 lines
fs/fuse/dev_uring.c +6
-0+if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ;
|| !PAGE_ALIGNED(count)) ?
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825
Quoted 8 lines
fs/fuse/dev_uring.c +8
-0+ +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start);
The iter is later used for the payload, but I miss a reset? iov_iter_revert()?
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828
Quoted 3 lines
fs/fuse/dev_uring.c +3
-0+if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start;
My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?
I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like
256 x 4K
8 x 128K
4 x 1M
I think there are currently two ways to do that
1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.
With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.
Quoted 130 lines
fs/fuse/dev_uring.c +51
-21+ +/* +* We can release the acquired reference on the header page immediately +* since the page is pinned and io_uring_cmd_import_fixed_full() +* prevents it from being unpinned while we are using it. +*/ +put_page (header_page); + +return 0 ; +} + +static void fuse_uring_unmap_buffer (struct fuse_ring_ent *ent) +{ +if (ent->fixed_buffer) +kunmap_local (ent->headers); +} +static int fuse_uring_prepare_send (struct fuse_ring_ent *ent,struct fuse_req *req) {@@ -932,6 +1021,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,unsigned int qid =READ_ONCE (cmd_req->qid);struct fuse_pqueue *fpq;struct fuse_req *req; +bool next_req; err = -ENOTCONN ;if (!ring)@@ -982,6 +1072,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,/* without the queue lock, as other locks are taken */ fuse_uring_prepare_cancel (cmd, issue_flags, ent); + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, req, err); +return err; + } +fuse_uring_commit (ent, req, issue_flags);/* @@ -990,7 +1087,9 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,* and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). */ -if (fuse_uring_get_next_fuse_req (ent, queue)) + next_req =fuse_uring_get_next_fuse_req (ent, queue); +fuse_uring_unmap_buffer (ent); +if (next_req)fuse_uring_send (ent, cmd,0 , issue_flags);return 0 ; }@@ -1086,39 +1185,49 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,struct iovec iov[FUSE_URING_IOV_SEGS ];int err; + err = -ENOMEM ; + ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); +if (!ent) +return ERR_PTR (err); + +INIT_LIST_HEAD (&ent->list); + + ent->queue = queue; + +if (READ_ONCE (cmd->sqe->uring_cmd_flags) &IORING_URING_CMD_FIXED ) { + ent->fixed_buffer = true; +atomic_inc (&ring->queue_refs); +return ent; + } + err =fuse_uring_get_iovec_from_sqe (cmd->sqe, iov);if (err) {pr_info_ratelimited ("Failed to get iovec from sqe, err=%d\n" , err); -return ERR_PTR (err); + goto error; } err = -EINVAL ;if (iov[0 ].iov_len <sizeof (struct fuse_uring_req_header )) {pr_info_ratelimited ("Invalid header len %zu\n" , iov[0 ].iov_len); -return ERR_PTR (err); + goto error; } payload_size = iov[1 ].iov_len;if (payload_size < ring->max_payload_sz) {pr_info_ratelimited ("Invalid req payload len %zu\n" , payload_size); -return ERR_PTR (err); + goto error; } - - err = -ENOMEM ; - ent =kzalloc (sizeof (*ent),GFP_KERNEL_ACCOUNT ); -if (!ent) -return ERR_PTR (err); - -INIT_LIST_HEAD (&ent->list); - - ent->queue = queue; ent->user_headers = iov[0 ].iov_base; ent->user_payload = iov[1 ].iov_base;atomic_inc (&ring->queue_refs);return ent; + +error: +kfree (ent); +return ERR_PTR (err); }/* @@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else {
fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372
Quoted 10 lines
fs/fuse/dev_uring.c +9
-1- err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } + + err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ;
I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.
Quoted 50 lines
fs/fuse/dev_uring.c +9
-1+ } + + err =fuse_uring_prepare_send (ent, ent->fuse_req); +if (err) { + send_ent =fuse_uring_get_next_fuse_req (ent, queue); + err =0 ; } +fuse_uring_unmap_buffer (ent); -fuse_uring_send (ent, cmd, err, issue_flags); +if (send_ent) +fuse_uring_send (ent, cmd, err, issue_flags); }static struct fuse_ring_queue *fuse_uring_task_to_queue (struct fuse_ring *ring)fs/fuse/dev_uring_i.h +17
-3@@ -7,6 +7,7 @@#ifndef _FS_FUSE_DEV_URING_I_H#define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> #include "fuse_i.h" #ifdef CONFIG_FUSE_IO_URING @@ -38,9 +39,29 @@ enum fuse_ring_req_state {/** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { -/* userspace buffer */ -struct fuse_uring_req_header __user *user_headers; -void __user *user_payload; + /* + * If true, the buffer was pre-registered by the daemon and the + * pages backing it are pinned in kernel memory. The fixed buffer layout + * is: [payload][header at end]. Use payload_iter and headers for + * copying to/from the ring. + * + * Otherwise, use user_headers and user_payload which point to userspace + * addresses representing the ring memory. + */ + bool fixed_buffer; + + union { + /* fixed_buffer == false */ + struct { + struct fuse_uring_req_header __user *user_headers; + void __user *user_payload;
fs/fuse/dev_uring_i.h enum fuse_ring_req_state { · line 67
Quoted 10 lines
fs/fuse/dev_uring_i.h +7
-0+ }; +/* fixed_buffer == true */ +struct { +struct fuse_uring_req_header *headers; +struct iov_iter payload_iter; + }; + };/* the ring queue that owns the request */ struct fuse_ring_queue *queue;
Thanks,
Bernd
Joanne Koong
maintainerQuoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +20
-0+/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ;
Quoted 8 lines
fs/fuse/dev_uring.c
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+ + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.
I think the best approach is to get rid of the call entirely by returning -EBUSY to the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Quoted 9 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817
Quoted 7 lines
fs/fuse/dev_uring.c +6
-0+if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ;|| !PAGE_ALIGNED(count)) ?
Nice, I didn't realize this macro existed. Thanks.
Quoted 11 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825
Quoted 9 lines
fs/fuse/dev_uring.c +8
-0+ +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start);The iter is later used for the payload, but I miss a reset?
iov_iter_revert()?
This iter is separate from the payload iter and doesn't affect the payload iter's values because the "ent->payload_iter = iter;" assignment above shallow copies that out first.
Quoted 5 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828
Quoted 4 lines
fs/fuse/dev_uring.c +3
-0+if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start;
Quoted 7 lines
My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?
I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.
Quoted 7 lines
I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like
256 x 4K
8 x 128K
4 x 1M
I think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.
Thanks,
joanne
Quoted 33 lines
I think there are currently two ways to do that
1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.
Quoted 23 lines
/*
fs/fuse/dev_uring.c +6
-9@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } +
Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372
Quoted 4 lines
fs/fuse/dev_uring.c +4
-0+ err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ;I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.
This approach makes sense to me and makes things a bit simpler. I'll add this to v3.
Thanks,
Joanne
Bernd Schubert
maintainerQuoted 56 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +20
-0+/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ;Quoted 8 lines
fs/fuse/dev_uring.c
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+ + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.
Quoted 8 lines
I think the best approach is to get rid of the call entirely by returning
-EBUSYto the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Hmm, I had seen this discussion, but I don't find anything about preventing unregistration?
Thanks,
Bernd
Quoted 66 lines · Bernd Schubert
Quoted 63 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +20
-0+/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ;Quoted 8 lines
fs/fuse/dev_uring.c
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+ + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.I think the best approach is to get rid of the call entirely by returning
-EBUSYto the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.
Quoted 4 lines
Quoted 1 line
[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Hmm, I had seen this discussion, but I don't find anything about preventing unregistration?
Ah found it, sorry for the noise.
Bernd Schubert
maintainerQuoted 169 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +20
-0+/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ;Quoted 8 lines
fs/fuse/dev_uring.c
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+ + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.I think the best approach is to get rid of the call entirely by returning
-EBUSYto the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Quoted 9 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817
Quoted 7 lines
fs/fuse/dev_uring.c +6
-0+if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ;|| !PAGE_ALIGNED(count)) ?
Nice, I didn't realize this macro existed. Thanks.
Quoted 11 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825
Quoted 9 lines
fs/fuse/dev_uring.c +8
-0+ +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start);The iter is later used for the payload, but I miss a reset?
iov_iter_revert()?This iter is separate from the payload iter and doesn't affect the payload iter's values because the "
ent->payload_iter= iter;" assignment above shallow copies that out first.Quoted 5 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828
Quoted 4 lines
fs/fuse/dev_uring.c +3
-0+if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start;Quoted 7 lines
My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.
Quoted 7 lines
I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like
256 x 4K
8 x 128K
4 x 1MI think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.
Thanks,
joanneQuoted 33 lines
I think there are currently two ways to do that
1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.
Quoted 23 lines
/*
fs/fuse/dev_uring.c +6
-9@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } +Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372
Quoted 4 lines
fs/fuse/dev_uring.c +4
-0+ err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ;I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.
Quoted 2 lines
This approach makes sense to me and makes things a bit simpler. I'll add this to v3.
This is a just heads up, while I'm testing sync FUSE_INIT I'm running in several issues. Part of it needs to be a re-send on -EGAIN and -EINTR in libfuse - had slipped through so far, but another is related to our page pinning. I just thought I can easy abort the connection when that fails, but that results in a double lock, because so far we assumed IO_URING_F_UNLOCKED in teardown context.
I'm going to submit a patch to teardown in uring task context, so that we are sure about the flags.
I.e. I think my suggestion to abort here would run into the same issue as below.
[10375.669761] fuse: FUSE_IO_URING_CMD_REGISTER returned -4 [10375.670632] fuse: FUSE_IO_URING_CMD_REGISTER failed err=-4
[10375.671922] ============================================
[10375.672754] WARNING: possible recursive locking detected
[10375.673577] 6.8.12+ #6 Tainted: G W O
[10375.674377] --------------------------------------------
[10375.675208] fuse-ring-0/7658 is trying to acquire lock:
[10375.676022] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: io_uring_cmd_done+0x14f/0x210
[10375.677335]
but task is already holding lock:
[10375.678292] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.679678]
other info that might help us debug this:
[10375.680722] Possible unsafe locking scenario:[10375.681692] CPU0
[10375.682156] ----
[10375.682633] lock(&ctx->uring_lock);
[10375.683260] lock(&ctx->uring_lock);
[10375.683879]
*** DEADLOCK ***[10375.684916] May be due to missing lock nesting notation
[10375.685994] 1 lock held by fuse-ring-0/7658:
[10375.686699] #0: ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0
[10375.688141]
stack backtrace:
[10375.688923] CPU: 0 PID: 7658 Comm: fuse-ring-0 Tainted: G W O 6.8.12+ #6
[10375.690150] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[10375.691534] Call Trace:
[10375.692019] <TASK>
[10375.692445] dump_stack_lvl+0x66/0x90
[10375.695219] __lock_acquire+0x13b8/0x25c0
[10375.695901] lock_acquire+0xb7/0x290
[10375.696507] ? io_uring_cmd_done+0x14f/0x210
[10375.697226] ? __lock_acquire+0x459/0x25c0
[10375.697913] __mutex_lock+0x7d/0xae0
[10375.698530] ? io_uring_cmd_done+0x14f/0x210
[10375.699251] ? io_uring_cmd_done+0x14f/0x210
[10375.699967] ? lock_acquire+0xb7/0x290
[10375.700606] ? fuse_uring_stop_list_entries+0x1c2/0x280 [fuse]
[10375.701508] ? io_uring_cmd_done+0x14f/0x210
[10375.702210] ? lock_release+0x24b/0x390
[10375.702862] io_uring_cmd_done+0x14f/0x210
[10375.703546] fuse_uring_stop_list_entries+0x24f/0x280 [fuse]
[10375.704439] fuse_uring_stop_queues+0xf7/0x240 [fuse]
[10375.705251] fuse_abort_conn+0x3e6/0x3f0 [fuse]
[10375.705998] fuse_uring_cmd+0x8de/0xf40 [fuse]
[10375.706742] ? lock_acquired+0xb1/0x320
[10375.707415] io_uring_cmd+0x6b/0x170
[10375.708080] io_issue_sqe+0x4e/0x460
[10375.708705] io_submit_sqes+0x22e/0x710
[10375.709356] __x64_sys_io_uring_enter+0x696/0xbf0
[10375.710109] do_syscall_64+0x6a/0x130
[10375.710743] entry_SYSCALL_64_after_hwframe+0x4b/0x53
[10375.711541] RIP: 0033:0x7f31a073db95
[10375.712160] Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e2 02 74 c2 f0 48 83 0c 24
[10375.714772] RSP: 002b:00007f319a4f4c88 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
[10375.715946] RAX: ffffffffffffffda RBX: 00007f31992f5060 RCX: 00007f31a073db95
[10375.717000] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000006
[10375.718056] RBP: 00007f319a4f4d60 R08: 0000000000000000 R09: 0000000000000008
[10375.719113] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f31992f5000
[10375.720162] R13: 00000fe63325ea00 R14: 0000000000000000 R15: 00007f319a4f4cd0
[10375.721223] </TASK>Thanks,
Bernd
Joanne Koong
maintainerQuoted 247 lines · Bernd Schubert
Quoted 169 lines · Joanne Koong
Quoted 46 lines · Bernd Schubert
Quoted 44 lines · Joanne Koong
Add support for io-uring registered buffers for fuse daemons communicating through the io-uring interface. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
To support page-aligned payloads, the buffer is structured such that the payload is at the front of the buffer and the
fuse_uring_req_headeris offset from the end of the buffer.To be backwards compatible, fuse uring still needs to support non-registered buffers as well.
Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev_uring.c | 200 +++++++++++++++++++++++++++++++++--------- fs/fuse/dev_uring_i.h | 27 +++++- 2 files changed, 183 insertions(+), 44 deletions(-)fs/fuse/dev_uring.c +20
-0+/* +* Prepare fixed buffer for access. Sets up the payload iter and kmaps the +* header. +* +* Callers must call fuse_uring_unmap_buffer() in the same scope to release the +* header mapping. +* +* For non-fixed buffers, this is a no-op. +*/ +static int fuse_uring_map_buffer (struct fuse_ring_ent *ent) +{ +size_t header_size =sizeof (struct fuse_uring_req_header ); +struct iov_iter iter; +struct page *header_page; +size_t count, start; +ssize_t copied; +int err; + +if (!ent->fixed_buffer) +return 0 ;Quoted 8 lines
fs/fuse/dev_uring.c
Quoted 2 lines
fs/fuse/dev_uring.c +2
-0+ + err =io_uring_cmd_import_fixed_full (ITER_DEST , &iter, ent->cmd,0 );This seems to be a rather expensive call, especially as it gets called twice (during submit and fetch).
Wouldn't be there be a possibility to check if the user buffer changed and then keep the existing iter? I think Caleb had a similar idea in patch 1/8.I think the best approach is to get rid of the call entirely by returning
-EBUSYto the server if it tries unregistering the buffers while a connection is still alive. Then we would just have to set this up once at registration time, and use that for the lifetime of the connection. The discussion about this with Pavel is in [1] - I'm planning to do this as a separate follow-up.[1] https://lore.kernel.org/linux-fsdevel/[email protected]/
Quoted 9 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 817
Quoted 7 lines
fs/fuse/dev_uring.c +6
-0+if (err) +return err; + + count =iov_iter_count (&iter); +if (count < header_size || count & (PAGE_SIZE -1 )) +return -EINVAL ;|| !PAGE_ALIGNED(count)) ?
Nice, I didn't realize this macro existed. Thanks.
Quoted 11 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 825
Quoted 9 lines
fs/fuse/dev_uring.c +8
-0+ +/* Adjust the payload iter to protect the header from any overwrites */ + ent->payload_iter = iter; +iov_iter_truncate (&ent->payload_iter, count - header_size); + +/* Set up the headers */ +iov_iter_advance (&iter, count - header_size); + copied =iov_iter_get_pages2 (&iter, &header_page, header_size,1 , &start);The iter is later used for the payload, but I miss a reset?
iov_iter_revert()?This iter is separate from the payload iter and doesn't affect the payload iter's values because the "
ent->payload_iter= iter;" assignment above shallow copies that out first.Quoted 5 lines
fs/fuse/dev_uring.c static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · line 828
Quoted 4 lines
fs/fuse/dev_uring.c +3
-0+if (copied < header_size) +return -EFAULT ; + ent->headers =kmap_local_page (header_page) + start;Quoted 7 lines
My plan for the alternative pinning patch (with io-uring) was to let the header be shared by multiple entries. Current libfuse master handles a fixed page size buffer for the payload (prepared page pinning - I didn't expect I was blocked for 9 months on other work), missing is to share it between ring entries.
I think this wouldn't work with registered buffer approach - it always needs one full page?I've been working on the patches for zero-copy and that has required the design for registered buffers in this patch to change, namely that the payload and the headers must be separated out. For v3, I have them separate now.
Quoted 7 lines
I would also like to discuss dynamic multiple payload sizes per queue. For example to have something like
256 x 4K
8 x 128K
4 x 1MI think zero-copy might obviate the need for this. The way I have it right now, it uses sparse buffers for payloads, which prevents the server from needing to allocate the 1M buffer per ent. I'm hoping to send out the patches for this as part of v3 at the end of next week or next next week.
Thanks,
joanneQuoted 33 lines
I think there are currently two ways to do that
1) Sort entries into pools
2) Sort buffers into pools and let entries use these. Here the header would be fixed and payload would come from a pool.With the appraoch to have payload and header in one buffer we couldn't use 2). Using 1) should be fine, though.
Quoted 23 lines
/*
fs/fuse/dev_uring.c +6
-9@@ -1249,20 +1358,29 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, {struct fuse_ring_ent *ent =uring_cmd_to_ring_ent (cmd);struct fuse_ring_queue *queue = ent->queue; +bool send_ent = true;int err; -if (!(issue_flags &IO_URING_F_TASK_DEAD )) { - err =fuse_uring_prepare_send (ent, ent->fuse_req); -if (err) { -if (!fuse_uring_get_next_fuse_req (ent, queue)) -return ; - err =0 ; - } - }else { - err = -ECANCELED ; +if (issue_flags &IO_URING_F_TASK_DEAD ) { +fuse_uring_send (ent, cmd, -ECANCELED , issue_flags); +return ; + } +Quoted 8 lines
fs/fuse/dev_uring.c static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, · line 1372
Quoted 4 lines
fs/fuse/dev_uring.c +4
-0+ err =fuse_uring_map_buffer (ent); +if (err) { +fuse_uring_req_end (ent, ent->fuse_req, err); +return ;I think this needs to abort the connection now. There could be multiple commands on the queue and they would be stuck now and there is no notification to fuse server either.
Quoted 2 lines
This approach makes sense to me and makes things a bit simpler. I'll add this to v3.
This is a just heads up, while I'm testing sync
FUSE_INITI'm running in several issues. Part of it needs to be a re-send on-EGAINand-EINTRin libfuse - had slipped through so far, but another is related to our page pinning. I just thought I can easy abort the connection when that fails, but that results in a double lock, because so far we assumedIO_URING_F_UNLOCKEDin teardown context.
I'm going to submit a patch to teardown in uring task context, so that we are sure about the flags.I.e. I think my suggestion to abort here would run into the same issue as below.
[10375.669761] fuse: FUSE_IO_URING_CMD_REGISTER returned -4 [10375.670632] fuse: FUSE_IO_URING_CMD_REGISTER failed err=-4[10375.671922] ============================================ [10375.672754] WARNING: possible recursive locking detected [10375.673577] 6.8.12+ #6 Tainted: G W O [10375.674377] -------------------------------------------- [10375.675208] fuse-ring-0/7658 is trying to acquire lock: [10375.676022] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: io_uring_cmd_done+0x14f/0x210 [10375.677335] but task is already holding lock: [10375.678292] ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0 [10375.679678] other info that might help us debug this: [10375.680722] Possible unsafe locking scenario:[10375.681692] CPU0 [10375.682156] ---- [10375.682633] lock(&ctx->uring_lock); [10375.683260] lock(&ctx->uring_lock); [10375.683879] *** DEADLOCK ***[10375.684916] May be due to missing lock nesting notation[10375.685994] 1 lock held by fuse-ring-0/7658: [10375.686699] #0: ffff8881040910b0 (&ctx->uring_lock){+.+.}-{4:4}, at: __x64_sys_io_uring_enter+0x68a/0xbf0 [10375.688141] stack backtrace: [10375.688923] CPU: 0 PID: 7658 Comm: fuse-ring-0 Tainted: G W O 6.8.12+ #6 [10375.690150] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [10375.691534] Call Trace: [10375.692019] <TASK> [10375.692445] dump_stack_lvl+0x66/0x90 [10375.695219] __lock_acquire+0x13b8/0x25c0 [10375.695901] lock_acquire+0xb7/0x290 [10375.696507] ? io_uring_cmd_done+0x14f/0x210 [10375.697226] ? __lock_acquire+0x459/0x25c0 [10375.697913] __mutex_lock+0x7d/0xae0 [10375.698530] ? io_uring_cmd_done+0x14f/0x210 [10375.699251] ? io_uring_cmd_done+0x14f/0x210 [10375.699967] ? lock_acquire+0xb7/0x290 [10375.700606] ? fuse_uring_stop_list_entries+0x1c2/0x280 [fuse] [10375.701508] ? io_uring_cmd_done+0x14f/0x210 [10375.702210] ? lock_release+0x24b/0x390 [10375.702862] io_uring_cmd_done+0x14f/0x210 [10375.703546] fuse_uring_stop_list_entries+0x24f/0x280 [fuse] [10375.704439] fuse_uring_stop_queues+0xf7/0x240 [fuse] [10375.705251] fuse_abort_conn+0x3e6/0x3f0 [fuse] [10375.705998] fuse_uring_cmd+0x8de/0xf40 [fuse] [10375.706742] ? lock_acquired+0xb1/0x320 [10375.707415] io_uring_cmd+0x6b/0x170 [10375.708080] io_issue_sqe+0x4e/0x460 [10375.708705] io_submit_sqes+0x22e/0x710 [10375.709356] __x64_sys_io_uring_enter+0x696/0xbf0 [10375.710109] do_syscall_64+0x6a/0x130 [10375.710743] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [10375.711541] RIP: 0033:0x7f31a073db95
Quoted 9 lines
[10375.712160] Code: 00 00 00 44 89 d0 41 b9 08 00 00 00 83 c8 10 f6 87 d0 00 00 00 01 8b bf cc 00 00 00 44 0f 45 d0 45 31 c0 b8 aa 01 00 00 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 41 83 e2 02 74 c2 f0 48 83 0c 24 [10375.714772] RSP: 002b:00007f319a4f4c88 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa [10375.715946] RAX: ffffffffffffffda RBX: 00007f31992f5060 RCX: 00007f31a073db95 [10375.717000] RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000006 [10375.718056] RBP: 00007f319a4f4d60 R08: 0000000000000000 R09: 0000000000000008 [10375.719113] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f31992f5000 [10375.720162] R13: 00000fe63325ea00 R14: 0000000000000000 R15: 00007f319a4f4cd0 [10375.721223] </TASK>
Thanks for the update, Bernd!
Quoted 4 lines
Thanks,
Bernd
Joanne Koong
maintainerQuoted 5 lines · Joanne Koong
This patchset adds fuse support for io-uring registered buffers. Daemons may register buffers ahead of time, which will eliminate the overhead of pinning/unpinning user pages and translating virtual addresses for every server-kernel interaction.
Registered buffers helps with the I/O overhead, but there's still significant memory waste within each buffer. Each entry in a queue allocates a dedicated buffer of at least 1 MB (for some performant servers like passthrough_hp, each entry's buffer is 4 MB, which adds up (eg with the libfuse default queue depth of 8 on a 64-core machine, that's 2 GB of buffers per fuse connection)) but most of this space goes unused. In practice, entries on a queue will rarely consume their full buffer capacity simultaneously.
Instead of using registered buffers, I think it's better if we go with a kernel managed ring buffer. It'll have the same advantages in reducing I/O overhead as registered buffers but also give us optionality later on to support IORING_CQE_F_BUF_MORE for incremental buffer consumption or add multiple different-sized ring buffer pools (eg for large payloads vs small payloads, which is also something we have to differentiate between for zero-copy), whereas we would have no way of supporting that with registered buffers.
For v3, I'm going to go this direction.
Thanks,
Joanne
Quoted 51 lines
The main logic for fuse registered buffers is in the last patch (patch 8/8). Patch 1/8 adds an
io_uringapi for fetching the registered buffer and patches (2-7)/8 refactors the fuseio_uringcode, which additionally will make adding in the logic for registered buffers neater.The libfuse changes can be found in this branch:
https://github.com/joannekoong/libfuse/tree/registered_buffers. The libfuse implementation first tries registered buffers during registration and if this fails, will retry with non-registered buffers. This prevents having to add a new init flag (but does have the downside of printing dmesg errors for the failed registrations when trying the registered buffers). If using registered buffers and the daemon for whatever reason unregisters the buffers midway through, then this will sever server-kernel communication. Libfuse will never do this. Libfuse will only unregister the buffers when the entire session is being destroyed.Benchmarks will be run and posted.
Thanks,
Joannev1: https://lore.kernel.org/linux-fsdevel/[email protected]/
v1 -> v2:
* Addio_uring_cmd_import_fixed_full()patch
* Construct iter usingio_uring_cmd_import_fixed_full()per cmd instead of recyling iters.
* Kmap the header instead of using bvec iter for iterating/copying. This makes the code easier to read.Joanne Koong (8):
io_uring/uring_cmd: addio_uring_cmd_import_fixed_full()
fuse: refactor io-uring logic for getting next fuse request
fuse: refactor io-uring header copying to ring
fuse: refactor io-uring header copying from ring
fuse: use enum types for header copying
fuse: adduser_prefix to userspace headers and payload fields
fuse: refactor setting up copy state for payload copying
fuse: support io-uring registered buffersfs/fuse/dev_uring.c | 366 +++++++++++++++++++++++++---------- fs/fuse/dev_uring_i.h | 27 ++- include/linux/io_uring/cmd.h | 3 + io_uring/rsrc.c | 14 ++ io_uring/rsrc.h | 2 + io_uring/uring_cmd.c | 13 ++ 6 files changed, 316 insertions(+), 109 deletions(-)--
2.47.3