Rewrite rsrc node handling
Jens Axboe
maintainerv2 cover Rewrite rsrc node handling
Hi,
v2 of this RFC, now a bit more polished and split up.
For a full explanation of the goal of the series, see patch #6 and #10. tldr is that our currently rsrc node handling can block freeing of resources for an indeterminite amount of time, which is very unfortunate for potentially long lived request. For example, networked workloads and using fixed files, where a previously long lived socket has the full resource tables of the entire ring pinned. That can lead to files being held open for a very long time, where they should be freed+closed instead.
This series handles the resource nodes separately, so a request pins just the resources it needs, and only for the duration of that request. In doing so, it also unifies how these resources are tracked. As it stands, the current kernel duplicates state across user_bufs and buf_data, and ditto for the file_table and file_data. Not only is some of it duplicated (like the node arrays), it also needs to alloc and copy the tags that are potentially associated with the resource. With the unification, state is only in one spot for each type of resource, and tags are handled at registration time rather than needing to be retained for the duration of the resource. As with cleaning up of structures, it also shrinks io_ring_ctx by 64b (should be more, it adds holes too in spots), and the actual resource node goes from needing 48b and 16b of put info, to 32b.
Tests out well here, both liburing test suite but also application testing. Most notably, the infamous test case that held all 10k sockets open during new opens and updates now only has the few open you'd expect. And it removes a net of about 280 lines of core io_uring code. In my opinion, it's also easier to follow.
Can also be found here:
https://git.kernel.dk/cgit/linux/log/?h=io_uring-rsrc
Changes since v1:
- Rebase on -rc5 + pending io_uring work. Each step now works, as it should, and passes testing.
- Add and use node lookup helper consistently
- Add a few patches killing mostly useless helpers
- Split out patches from the main bigger patches
- Fix an assumption in test/rsrc_tags.c that prevented it from working with the per-node refs
- Add NOP patch that enables testing of both registered files and buffers
- Remove 'index' struct io_rsrc_node, it was unused now. That shrinks the node size fo 32b, fitting two in a cacheline.
include/linux/io_uring_types.h | 25 +- include/uapi/linux/io_uring.h | 3 + io_uring/cancel.c | 8 +- io_uring/fdinfo.c | 14 +- io_uring/filetable.c | 79 ++--- io_uring/filetable.h | 37 +-- io_uring/io_uring.c | 51 +-- io_uring/msg_ring.c | 33 +- io_uring/net.c | 16 +- io_uring/nop.c | 47 ++- io_uring/notif.c | 3 +- io_uring/opdef.c | 2 + io_uring/register.c | 3 +- io_uring/rsrc.c | 587 +++++++++++---------------------- io_uring/rsrc.h | 96 +++--- io_uring/rw.c | 15 +- io_uring/splice.c | 42 ++- io_uring/splice.h | 1 + io_uring/uring_cmd.c | 20 +- 19 files changed, 437 insertions(+), 645 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v2 1/13 io_uring/nop: add support for testing registered files and buffers
Useful for testing performance/efficiency impact of registered files and buffers, vs (particularly) non-registered files.
--- include/uapi/linux/io_uring.h | 3 +++ io_uring/nop.c | 49 +++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 5 deletions(-)
include/uapi/linux/io_uring.h +3 -0
@@ -416,6 +416,9 @@ enum io_uring_msg_ring_flags {* IORING_NOP_INJECT_RESULT Inject result from sqe->result */ #define IORING_NOP_INJECT_RESULT (1U << 0) +#define IORING_NOP_FILE (1U << 1) +#define IORING_NOP_FIXED_FILE (1U << 2) +#define IORING_NOP_FIXED_BUFFER (1U << 3)/* * IO completion data structure (Completion Queue Entry)
io_uring/nop.c +44 -5
79 lines not on this page · raw message
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 2/13 io_uring/rsrc: move struct io_fixed_file to rsrc.h header
In reply to Jens AxboeThere's no need for this internal structure to be visible, move it to the private rsrc.h header instead.
--- include/linux/io_uring_types.h | 5 ----- io_uring/filetable.h | 1 + io_uring/rsrc.h | 5 +++++ 3 files changed, 6 insertions(+), 5 deletions(-)
include/linux/io_uring_types.h +0 -5
@@ -55,11 +55,6 @@ struct io_wq_work {int cancel_seq; }; -struct io_fixed_file { -/* file * with additional FFS_* flags */ -unsigned long file_ptr; -}; -struct io_file_table {struct io_fixed_file *files;unsigned long *bitmap;
io_uring/filetable.h +1 -0
@@ -4,6 +4,7 @@#include <linux/file.h> #include <linux/io_uring_types.h> +#include "rsrc.h" bool io_alloc_file_tables (struct io_file_table *table,unsigned nr_files);void io_free_file_tables (struct io_file_table *table);
io_uring/rsrc.h +5 -0
@@ -40,6 +40,11 @@ struct io_rsrc_node {struct io_rsrc_put item; }; +struct io_fixed_file { +/* file * with additional FFS_* flags */ +unsigned long file_ptr; +}; +struct io_mapped_ubuf {u64 ubuf;unsigned int len;
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 3/13 io_uring: specify freeptr usage for SLAB_TYPESAFE_BY_RCU io_kiocb cache
In reply to Jens AxboeDoesn't matter right now as there's still some bytes left for it, but let's prepare for the io_kiocb potentially growing and add a specific freeptr offset for it.
--- io_uring/io_uring.c | 2 ++ 1 file changed, 2 insertions(+)
io_uring/io_uring.c +2 -0
@@ -3846,6 +3846,8 @@ static int __init io_uring_init(void)struct kmem_cache_args kmem_args = { .useroffset = offsetof(struct io_kiocb, cmd.data), .usersize = sizeof_field(struct io_kiocb, cmd.data), + .freeptr_offset = offsetof(struct io_kiocb , work), + .use_freeptr_offset = true, };#define __BUILD_BUG_VERIFY_OFFSET_SIZE (stype, eoffset, esize, ename) do { \
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 4/13 io_uring/splice: open code 2nd direct file assignment
In reply to Jens AxboeIn preparation for not pinning the whole registered file table, open code the second potential direct file assignment. This will be handled by appropriate helpers in the future, for now just do it manually.
--- io_uring/opdef.c | 2 ++ io_uring/splice.c | 44 ++++++++++++++++++++++++++++++++++++-------- io_uring/splice.h | 1 + 3 files changed, 39 insertions(+), 8 deletions(-)
io_uring/opdef.c +2 -0
@@ -641,6 +641,7 @@ const struct io_cold_def io_cold_defs[] = { }, [IORING_OP_SPLICE] = { .name = "SPLICE", + .cleanup = io_splice_cleanup, }, [IORING_OP_PROVIDE_BUFFERS] = { .name = "PROVIDE_BUFFERS",
@@ -650,6 +651,7 @@ const struct io_cold_def io_cold_defs[] = { }, [IORING_OP_TEE] = { .name = "TEE", + .cleanup = io_splice_cleanup, }, [IORING_OP_SHUTDOWN] = { .name = "SHUTDOWN",
io_uring/splice.c +36 -8
74 lines not on this page · raw message
io_uring/splice.h +1 -0
@@ -3,5 +3,6 @@int io_tee_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_tee (struct io_kiocb *req,unsigned int issue_flags); +void io_splice_cleanup (struct io_kiocb *req);int io_splice_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_splice (struct io_kiocb *req,unsigned int issue_flags);
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 5/13 io_uring/rsrc: kill io_charge_rsrc_node()
In reply to Jens AxboeIt's only used from __io_req_set_rsrc_node(), and it takes both the ctx and node itself, while never using the ctx. Just open-code the basic refs++ in __io_req_set_rsrc_node() instead.
--- io_uring/rsrc.h | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
io_uring/rsrc.h +1 -7
@@ -97,18 +97,12 @@ static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_nodeio_rsrc_node_ref_zero (node); } -static inline void io_charge_rsrc_node (struct io_ring_ctx *ctx, -struct io_rsrc_node *node) -{ - node->refs++; -} -static inline void __io_req_set_rsrc_node (struct io_kiocb *req,struct io_ring_ctx *ctx) {lockdep_assert_held (&ctx->uring_lock); req->rsrc_node = ctx->rsrc_node; -io_charge_rsrc_node (ctx, ctx->rsrc_node); + ctx->rsrc_node->refs++; } static inline void io_req_set_rsrc_node(struct io_kiocb *req,
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 6/13 io_uring/rsrc: get rid of per-ring io_rsrc_node list
In reply to Jens AxboeWork in progress, but get rid of the per-ring serialization of resource nodes, like registered buffers and files. Main issue here is that one node can otherwise hold up a bunch of other nodes from getting freed, which is especially a problem for file resource nodes and networked workloads where some descriptors may not see activity in a long time.
As an example, instantiate an io_uring ring fd and create a sparse registered file table. Even 2 will do. Then create a socket and register it as fixed file 0, F0. The number of open files in the app is now 5, with 0/1/2 being the usual stdin/out/err, 3 being the ring fd, and 4 being the socket. Register this socket (eg "the listener") in slot 0 of the registered file table. Now add an operation on the socket that uses slot 0. Finally, loop N times, where each loop creates a new socket, registers said socket as a file, then unregisters the socket, and finally closes the socket. This is roughly similar to what a basic accept loop would look like.
At the end of this loop, it's not unreasonable to expect that there would still be 5 open files. Each socket created and registered in the loop is also unregistered and closed. But since the listener socket registered first still has references to its resource node due to still being active, each subsequent socket unregistration is stuck behind it for reclaim. Hence 5 + N files are still open at that point, where N is awaiting the final put held up by the listener socket.
Rewrite the io_rsrc_node handling to NOT rely on serialization. Struct io_kiocb now gets explicit resource nodes assigned, with each holding a reference to the parent node. A parent node is either of type FILE or BUFFER, which are the two types of nodes that exist. A request can have two nodes assigned, if it's using both registered files and buffers. Since request issue and task_work completion is both under the ring private lock, no atomics are needed to handle these references. It's a simple unlocked inc/dec. As before, the registered buffer or file table each hold a reference as well to the registered nodes. Final put of the node will remove the node and free the underlying resource, eg unmap the buffer or put the file.
Outside of removing the stall in resource reclaim described above, it has the following advantages:
1) It's a lot simpler than the previous scheme, and easier to follow. No need to specific quiesce handling anymore.
2) There are no resource node allocations in the fast path, all of that happens at resource registration time.
3) The structs related to resource handling can all get simplified quite a bit, like io_rsrc_node and io_rsrc_data. io_rsrc_put can go away completely.
4) Handling of resource tags is much simpler, and doesn't require persistent storage as it can simply get assigned up front at registration time. Just copy them in one-by-one at registration time and assign to the resource node.
The only real downside is that a request is now explicitly limited to pinning 2 resources, one file and one buffer, where before just assigning a resource node to a request would pin all of them. The upside is that it's easier to follow now, as an individual resource is explicitly referenced and assigned to the request.
With this in place, the above mentioned example will be using exactly 5 files at the end of the loop, not N.
Needs to get broken up a bit and there are certainly rough edges, but that's why it's a work in progress... But it does remove a ton more code than it adds, and passes the liburing tests.
--- include/linux/io_uring_types.h | 10 +- io_uring/fdinfo.c | 2 +- io_uring/filetable.c | 52 ++-- io_uring/filetable.h | 25 +- io_uring/io_uring.c | 38 +-- io_uring/net.c | 11 +- io_uring/nop.c | 6 +- io_uring/notif.c | 3 +- io_uring/rsrc.c | 488 +++++++++++++-------------------- io_uring/rsrc.h | 75 ++--- io_uring/rw.c | 8 +- io_uring/splice.c | 16 +- io_uring/uring_cmd.c | 12 +- 13 files changed, 280 insertions(+), 466 deletions(-)
include/linux/io_uring_types.h +3 -7
@@ -56,7 +56,7 @@ struct io_wq_work { };struct io_file_table { -struct io_fixed_file *files; +struct io_rsrc_node **nodes;unsigned long *bitmap;unsigned int alloc_hint; };
@@ -264,7 +264,6 @@ struct io_ring_ctx {* Fixed resources fast path, should be accessed only under * uring_lock, and updated through io_uring_register(2) */ -struct io_rsrc_node *rsrc_node;atomic_t cancel_seq;/*
@@ -277,7 +276,7 @@ struct io_ring_ctx {struct io_wq_work_list iopoll_list;struct io_file_table file_table; -struct io_mapped_ubuf **user_bufs; +struct io_rsrc_node **user_bufs;unsigned nr_user_files;unsigned nr_user_bufs;
@@ -372,10 +371,7 @@ struct io_ring_ctx {struct io_rsrc_data *buf_data;/* protected by ->uring_lock */ -struct list_head rsrc_ref_list;struct io_alloc_cache rsrc_node_cache; -struct wait_queue_head rsrc_quiesce_wq; -unsigned rsrc_quiesce;u32 pers_next;struct xarray personalities;
@@ -642,7 +638,7 @@ struct io_kiocb {__poll_t apoll_events; }; -struct io_rsrc_node *rsrc_node; +struct io_rsrc_node *rsrc_nodes[2 ];atomic_t refs;bool cancel_seq_set;
io_uring/fdinfo.c +1 -1
@@ -176,7 +176,7 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) }seq_printf (m,"UserBufs:\t%u\n" , ctx->nr_user_bufs);for (i =0 ; has_lock && i < ctx->nr_user_bufs; i++) { -struct io_mapped_ubuf *buf = ctx->user_bufs[i]; +struct io_mapped_ubuf *buf = ctx->user_bufs[i]->buf;seq_printf (m,"%5u: 0x%llx/%u\n" , i, buf->ubuf, buf->len); }
io_uring/filetable.c +19 -33
97 lines not on this page · raw message
io_uring/filetable.h +12 -13
@@ -34,36 +34,35 @@ static inline void io_file_bitmap_set(struct io_file_table *table, int bit) table->alloc_hint = bit +1 ; } -static inline struct io_fixed_file * -io_fixed_file_slot (struct io_file_table *table,unsigned i) -{ -return &table->files[i]; -} -#define FFS_NOWAIT 0x1UL#define FFS_ISREG 0x2UL#define FFS_MASK ~(FFS_NOWAIT|FFS_ISREG) -static inline unsigned int io_slot_flags (struct io_fixed_file *slot) +static inline unsigned int io_slot_flags (struct io_rsrc_node *node) { -return (slot->file_ptr & ~FFS_MASK ) <<REQ_F_SUPPORT_NOWAIT_BIT ; + +return (node->file_ptr & ~FFS_MASK ) <<REQ_F_SUPPORT_NOWAIT_BIT ; } -static inline struct file *io_slot_file (struct io_fixed_file *slot) +static inline struct file *io_slot_file (struct io_rsrc_node *node) { -return (struct file *)(slot->file_ptr &FFS_MASK ); +return (struct file *)(node->file_ptr &FFS_MASK ); }static inline struct file *io_file_from_index (struct io_file_table *table,int index) { -return io_slot_file (io_fixed_file_slot (table, index)); +struct io_rsrc_node *node = table->nodes[index]; + +if (node) +return io_slot_file (node); +return NULL ; } -static inline void io_fixed_file_set (struct io_fixed_file *file_slot, +static inline void io_fixed_file_set (struct io_rsrc_node *node,struct file *file) { - file_slot->file_ptr = (unsigned long )file | + node->file_ptr = (unsigned long )file | (io_file_get_flags (file) >>REQ_F_SUPPORT_NOWAIT_BIT ); }
io_uring/io_uring.c +12 -26
@@ -333,7 +333,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)mutex_init (&ctx->uring_lock);init_waitqueue_head (&ctx->cq_wait);init_waitqueue_head (&ctx->poll_wq); -init_waitqueue_head (&ctx->rsrc_quiesce_wq);spin_lock_init (&ctx->completion_lock);spin_lock_init (&ctx->timeout_lock);INIT_WQ_LIST (&ctx->iopoll_list);
@@ -341,7 +340,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_LIST_HEAD (&ctx->defer_list);INIT_LIST_HEAD (&ctx->timeout_list);INIT_LIST_HEAD (&ctx->ltimeout_list); -INIT_LIST_HEAD (&ctx->rsrc_ref_list);init_llist_head (&ctx->work_llist);INIT_LIST_HEAD (&ctx->tctx_list); ctx->submit_state.free_list.next =NULL ;
@@ -1415,7 +1413,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,io_clean_op (req); }io_put_file (req); -io_put_rsrc_node (ctx, req->rsrc_node); +io_req_put_rsrc_nodes (req);io_put_task (req->task); node = req->comp_list.next;
@@ -1878,7 +1876,7 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, unsigned int issue_flags) {struct io_ring_ctx *ctx = req->ctx; -struct io_fixed_file *slot; +struct io_rsrc_node *node;struct file *file =NULL ;io_ring_submit_lock (ctx, issue_flags);
@@ -1886,11 +1884,12 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd,if (unlikely ((unsigned int )fd >= ctx->nr_user_files)) goto out; fd =array_index_nospec (fd, ctx->nr_user_files); - slot =io_fixed_file_slot (&ctx->file_table, fd); -if (!req->rsrc_node) -__io_req_set_rsrc_node (req, ctx); - req->flags |=io_slot_flags (slot); - file =io_slot_file (slot); + node = ctx->file_table.nodes[fd]; +if (node) { +io_req_assign_rsrc_node (req, node); + req->flags |=io_slot_flags (node); + file =io_slot_file (node); + } out:io_ring_submit_unlock (ctx, issue_flags);return file;
@@ -2036,7 +2035,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t ) sqe_flags; req->cqe.user_data =READ_ONCE (sqe->user_data); req->file =NULL ; - req->rsrc_node =NULL ; + req->rsrc_nodes[IORING_RSRC_FILE ] =NULL ; + req->rsrc_nodes[IORING_RSRC_BUFFER ] =NULL ; req->task = current; req->cancel_seq_set = false;
@@ -2718,15 +2718,10 @@ static void io_req_caches_free(struct io_ring_ctx *ctx)static __cold void io_ring_ctx_free (struct io_ring_ctx *ctx) {io_sq_thread_finish (ctx); -/* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ -if (WARN_ON_ONCE (!list_empty (&ctx->rsrc_ref_list))) -return ;mutex_lock (&ctx->uring_lock); -if (ctx->buf_data) -__io_sqe_buffers_unregister (ctx); -if (ctx->file_data) -__io_sqe_files_unregister (ctx); +io_sqe_buffers_unregister (ctx); +io_sqe_files_unregister (ctx);io_cqring_overflow_kill (ctx);io_eventfd_unregister (ctx);io_alloc_cache_free (&ctx->apoll_cache, kfree);
@@ -2743,11 +2738,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)if (ctx->submitter_task)put_task_struct (ctx->submitter_task); -/* there are no registered resources left, nobody uses it */ -if (ctx->rsrc_node) -io_rsrc_node_destroy (ctx, ctx->rsrc_node); - -WARN_ON_ONCE (!list_empty (&ctx->rsrc_ref_list));WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list));io_alloc_cache_free (&ctx->rsrc_node_cache, kfree);
@@ -3729,10 +3719,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,if (ret) goto err; - ret =io_rsrc_init (ctx); -if (ret) - goto err; - p->features =IORING_FEAT_SINGLE_MMAP |IORING_FEAT_NODROP |IORING_FEAT_SUBMIT_STABLE |IORING_FEAT_RW_CUR_POS | IORING_FEAT_CUR_PERSONALITY | IORING_FEAT_FAST_POLL |
io_uring/net.c +6 -5
@@ -1342,15 +1342,15 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags)if (sr->flags &IORING_RECVSEND_FIXED_BUF ) {struct io_ring_ctx *ctx = req->ctx; -struct io_mapped_ubuf *imu; +struct io_rsrc_node *node;int idx; ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags);if (sr->buf_index < ctx->nr_user_bufs) { idx =array_index_nospec (sr->buf_index, ctx->nr_user_bufs); - imu =READ_ONCE (ctx->user_bufs[idx]); -io_req_set_rsrc_node (sr->notif, ctx); + node = ctx->user_bufs[idx]; +io_req_assign_rsrc_node (sr->notif, node); ret =0 ; }io_ring_submit_unlock (ctx, issue_flags);
@@ -1358,8 +1358,9 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags)if (unlikely (ret))return ret; - ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, imu, - (u64 )(uintptr_t )sr->buf, sr->len); + ret =io_import_fixed (ITER_SOURCE , &kmsg->msg.msg_iter, + node->buf, (u64 )(uintptr_t )sr->buf, + sr->len);if (unlikely (ret))return ret; kmsg->msg.sg_from_iter = io_sg_from_iter;
io_uring/nop.c +3 -3
@@ -61,15 +61,15 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) }if (nop->flags &IORING_NOP_FIXED_BUFFER ) {struct io_ring_ctx *ctx = req->ctx; -struct io_mapped_ubuf *imu; +struct io_rsrc_node *node;int idx; ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags);if (nop->buffer < ctx->nr_user_bufs) { idx =array_index_nospec (nop->buffer, ctx->nr_user_bufs); - imu =READ_ONCE (ctx->user_bufs[idx]); -io_req_set_rsrc_node (req, ctx); + node =READ_ONCE (ctx->user_bufs[idx]); +io_req_assign_rsrc_node (req, node); ret =0 ; }io_ring_submit_unlock (ctx, issue_flags);
io_uring/notif.c +2 -1
@@ -117,7 +117,8 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->file =NULL ; notif->task = current;io_get_task_refs (1 ); - notif->rsrc_node =NULL ; + notif->rsrc_nodes[IORING_RSRC_FILE ] =NULL ; + notif->rsrc_nodes[IORING_RSRC_BUFFER ] =NULL ; nd =io_notif_to_data (notif); nd->zc_report = false;
io_uring/rsrc.c +184 -304
726 lines not on this page · raw message
io_uring/rsrc.h +21 -54
121 lines not on this page · raw message
io_uring/rw.c +4 -4
@@ -330,7 +330,7 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);struct io_ring_ctx *ctx = req->ctx; -struct io_mapped_ubuf *imu; +struct io_rsrc_node *node;struct io_async_rw *io;u16 index;int ret;
@@ -342,11 +342,11 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqeif (unlikely (req->buf_index >= ctx->nr_user_bufs))return -EFAULT ; index =array_index_nospec (req->buf_index, ctx->nr_user_bufs); - imu = ctx->user_bufs[index]; -io_req_set_rsrc_node (req, ctx); + node = ctx->user_bufs[index]; +io_req_assign_rsrc_node (req, node); io = req->async_data; - ret =io_import_fixed (ddir, &io->iter, imu, rw->addr, rw->len); + ret =io_import_fixed (ddir, &io->iter, node->buf, rw->addr, rw->len);iov_iter_save_state (&io->iter, &io->iter_state);return ret; }
io_uring/splice.c +9 -7
@@ -51,7 +51,7 @@ void io_splice_cleanup(struct io_kiocb *req) { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); -io_put_rsrc_node (req->ctx, sp->rsrc_node); +io_put_rsrc_node (sp->rsrc_node); } static struct file *io_splice_get_file(struct io_kiocb *req,
@@ -59,7 +59,7 @@ static struct file *io_splice_get_file(struct io_kiocb *req, { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice);struct io_ring_ctx *ctx = req->ctx; -struct io_fixed_file *slot; +struct io_rsrc_node *node;struct file *file =NULL ;if (!(sp->flags &SPLICE_F_FD_IN_FIXED ))
@@ -69,11 +69,13 @@ static struct file *io_splice_get_file(struct io_kiocb *req,if (unlikely (sp->splice_fd_in >= ctx->nr_user_files)) goto out; sp->splice_fd_in =array_index_nospec (sp->splice_fd_in, ctx->nr_user_files); - slot = &ctx->file_table.files[sp->splice_fd_in]; -if (!req->rsrc_node) -__io_req_set_rsrc_node (req, ctx); - file =io_slot_file (slot); - req->flags |=REQ_F_NEED_CLEANUP ; + node = ctx->file_table.nodes[sp->splice_fd_in]; +if (node) { + node->refs++; + sp->rsrc_node = node; + file =io_slot_file (node); + req->flags |=REQ_F_NEED_CLEANUP ; + } out:io_ring_submit_unlock (ctx, issue_flags);return file;
io_uring/uring_cmd.c +4 -8
@@ -220,7 +220,7 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)* being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ -io_req_set_rsrc_node (req, ctx); +io_req_assign_rsrc_node (req, ctx->user_bufs[req->buf_index]); } ioucmd->cmd_op =READ_ONCE (sqe->cmd_op);
@@ -276,15 +276,11 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) { struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); -struct io_ring_ctx *ctx = req->ctx; +struct io_rsrc_node *node = req->rsrc_nodes[IORING_RSRC_BUFFER ];/* Must have had rsrc_node assigned at prep time */ -if (req->rsrc_node) { -struct io_mapped_ubuf *imu; - - imu =READ_ONCE (ctx->user_bufs[req->buf_index]); -return io_import_fixed (rw, iter, imu, ubuf, len); - } +if (node) +return io_import_fixed (rw, iter, node->buf, ubuf, len);return -EFAULT ; }
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 7/13 io_uring/rsrc: get rid of io_rsrc_node allocation cache
In reply to Jens AxboeIt's not going to be needed in the fast path going forward, so kill it off.
--- include/linux/io_uring_types.h | 3 --- io_uring/io_uring.c | 6 +----- io_uring/rsrc.c | 22 ++++++++-------------- 3 files changed, 9 insertions(+), 22 deletions(-)
include/linux/io_uring_types.h +0 -3
@@ -370,9 +370,6 @@ struct io_ring_ctx {struct io_rsrc_data *file_data;struct io_rsrc_data *buf_data; -/* protected by ->uring_lock */ -struct io_alloc_cache rsrc_node_cache; -u32 pers_next;struct xarray personalities;
io_uring/io_uring.c +1 -5
@@ -312,9 +312,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_LIST_HEAD (&ctx->sqd_list);INIT_LIST_HEAD (&ctx->cq_overflow_list);INIT_LIST_HEAD (&ctx->io_buffers_cache); - ret =io_alloc_cache_init (&ctx->rsrc_node_cache,IO_NODE_ALLOC_CACHE_MAX , -sizeof (struct io_rsrc_node )); - ret |=io_alloc_cache_init (&ctx->apoll_cache,IO_POLL_ALLOC_CACHE_MAX , + ret =io_alloc_cache_init (&ctx->apoll_cache,IO_POLL_ALLOC_CACHE_MAX ,sizeof (struct async_poll )); ret |=io_alloc_cache_init (&ctx->netmsg_cache,IO_ALLOC_CACHE_MAX ,sizeof (struct io_async_msghdr ));
@@ -358,7 +356,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) free_ref:percpu_ref_exit (&ctx->refs); err: -io_alloc_cache_free (&ctx->rsrc_node_cache, kfree);io_alloc_cache_free (&ctx->apoll_cache, kfree);io_alloc_cache_free (&ctx->netmsg_cache, io_netmsg_cache_free);io_alloc_cache_free (&ctx->rw_cache, io_rw_cache_free);
@@ -2740,7 +2737,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list)); -io_alloc_cache_free (&ctx->rsrc_node_cache, kfree);if (ctx->mm_account) {mmdrop (ctx->mm_account); ctx->mm_account =NULL ;
io_uring/rsrc.c +8 -14
@@ -13,7 +13,6 @@#include <uapi/linux/io_uring.h> #include "io_uring.h" -#include "alloc_cache.h" #include "openclose.h" #include "rsrc.h" #include "memmap.h"
@@ -132,18 +131,14 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, {struct io_rsrc_node *node; - node =io_alloc_cache_get (&ctx->rsrc_node_cache); -if (!node) { - node =kzalloc (sizeof (*node),GFP_KERNEL ); -if (!node) -return NULL ; + node =kzalloc (sizeof (*node),GFP_KERNEL ); +if (node) { + node->ctx = ctx; + node->refs =1 ; + node->type = type; + node->tag =0 ; + node->rsrc =NULL ; } - - node->ctx = ctx; - node->refs =1 ; - node->type = type; - node->tag =0 ; - node->rsrc =NULL ;return node; }
@@ -492,8 +487,7 @@ void io_free_rsrc_node(struct io_rsrc_node *node)break ; } -if (!io_alloc_cache_put (&ctx->rsrc_node_cache, node)) -kfree (node); +kfree (node); }static void __io_sqe_files_unregister (struct io_ring_ctx *ctx)
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 8/13 io_uring/rsrc: add an empty io_rsrc_node for sparse buffer entries
In reply to Jens AxboeRather than allocate an io_rsrc_node for an empty/sparse buffer entry, add a const entry that can be used for that. This just needs checking for writing the tag, and the put check needs to check for that sparse node rather than NULL for validity.
This avoids allocating rsrc nodes for sparse buffer entries.
--- io_uring/io_uring.c | 4 ++-- io_uring/notif.c | 4 ++-- io_uring/rsrc.c | 52 +++++++++++++++++++++++++++------------------ io_uring/rsrc.h | 11 +++++++--- io_uring/splice.c | 2 +- 5 files changed, 44 insertions(+), 29 deletions(-)
io_uring/io_uring.c +2 -2
@@ -2032,8 +2032,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t ) sqe_flags; req->cqe.user_data =READ_ONCE (sqe->user_data); req->file =NULL ; - req->rsrc_nodes[IORING_RSRC_FILE ] =NULL ; - req->rsrc_nodes[IORING_RSRC_BUFFER ] =NULL ; + req->rsrc_nodes[IORING_RSRC_FILE ] = rsrc_empty_node; + req->rsrc_nodes[IORING_RSRC_BUFFER ] = rsrc_empty_node; req->task = current; req->cancel_seq_set = false;
io_uring/notif.c +2 -2
@@ -117,8 +117,8 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->file =NULL ; notif->task = current;io_get_task_refs (1 ); - notif->rsrc_nodes[IORING_RSRC_FILE ] =NULL ; - notif->rsrc_nodes[IORING_RSRC_BUFFER ] =NULL ; + notif->rsrc_nodes[IORING_RSRC_FILE ] = rsrc_empty_node; + notif->rsrc_nodes[IORING_RSRC_BUFFER ] = rsrc_empty_node; nd =io_notif_to_data (notif); nd->zc_report = false;
io_uring/rsrc.c +31 -21
109 lines not on this page · raw message
io_uring/rsrc.h +8 -3
@@ -74,9 +74,12 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,int io_register_rsrc (struct io_ring_ctx *ctx,void __user *arg,unsigned int size,unsigned int type); +extern const struct io_rsrc_node empty_node; +#define rsrc_empty_node (struct io_rsrc_node *) &empty_node +static inline void io_put_rsrc_node (struct io_rsrc_node *node) { -if (node && !--node->refs) +if (node != &empty_node && !--node->refs)io_free_rsrc_node (node); }
@@ -89,8 +92,10 @@ static inline void io_req_put_rsrc_nodes(struct io_kiocb *req)static inline void io_req_assign_rsrc_node (struct io_kiocb *req,struct io_rsrc_node *node) { - node->refs++; - req->rsrc_nodes[node->type] = node; +if (node != rsrc_empty_node) { + node->refs++; + req->rsrc_nodes[node->type] = node; + } }int io_files_update (struct io_kiocb *req,unsigned int issue_flags);
io_uring/splice.c +1 -1
@@ -35,7 +35,7 @@ static int __io_splice_prep(struct io_kiocb *req,if (unlikely (sp->flags & ~valid_flags))return -EINVAL ; sp->splice_fd_in =READ_ONCE (sqe->splice_fd_in); - sp->rsrc_node =NULL ; + sp->rsrc_node = rsrc_empty_node; req->flags |=REQ_F_FORCE_ASYNC ;return 0 ; }
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 9/13 io_uring: only initialize io_kiocb rsrc_nodes when needed
In reply to Jens AxboeAdd the empty node initializing to the preinit part of the io_kiocb allocation, and reset them if they have been used.
--- io_uring/io_uring.c | 4 ++-- io_uring/rsrc.h | 10 ++++++++-- 2 files changed, 10 insertions(+), 4 deletions(-)
io_uring/io_uring.c +2 -2
@@ -947,6 +947,8 @@ void io_req_defer_failed(struct io_kiocb *req, s32 res)static void io_preinit_req (struct io_kiocb *req,struct io_ring_ctx *ctx) { req->ctx = ctx; + req->rsrc_nodes[IORING_RSRC_FILE ] = rsrc_empty_node; + req->rsrc_nodes[IORING_RSRC_BUFFER ] = rsrc_empty_node; req->link =NULL ; req->async_data =NULL ;/* not necessary, but safer to zero */
@@ -2032,8 +2034,6 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t ) sqe_flags; req->cqe.user_data =READ_ONCE (sqe->user_data); req->file =NULL ; - req->rsrc_nodes[IORING_RSRC_FILE ] = rsrc_empty_node; - req->rsrc_nodes[IORING_RSRC_BUFFER ] = rsrc_empty_node; req->task = current; req->cancel_seq_set = false;
io_uring/rsrc.h +8 -2
@@ -85,8 +85,14 @@ static inline void io_put_rsrc_node(struct io_rsrc_node *node)static inline void io_req_put_rsrc_nodes (struct io_kiocb *req) { -io_put_rsrc_node (req->rsrc_nodes[IORING_RSRC_FILE ]); -io_put_rsrc_node (req->rsrc_nodes[IORING_RSRC_BUFFER ]); +if (req->rsrc_nodes[IORING_RSRC_FILE ] != rsrc_empty_node) { +io_put_rsrc_node (req->rsrc_nodes[IORING_RSRC_FILE ]); + req->rsrc_nodes[IORING_RSRC_FILE ] = rsrc_empty_node; + } +if (req->rsrc_nodes[IORING_RSRC_BUFFER ] != rsrc_empty_node) { +io_put_rsrc_node (req->rsrc_nodes[IORING_RSRC_BUFFER ]); + req->rsrc_nodes[IORING_RSRC_BUFFER ] = rsrc_empty_node; + } } static inline void io_req_assign_rsrc_node(struct io_kiocb *req,
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 10/13 io_uring/rsrc: unify file and buffer resource tables
In reply to Jens AxboeFor files, there's nr_user_files/file_table/file_data, and buffers have nr_user_bufs/user_bufs/buf_data. There's no reason why file_table and file_data can't be the same thing, and ditto for the buffer side. That gets rid of more io_ring_ctx state that's in two spots rather than just being in one spot, as it should be. Put all the registered file data in one locations, and ditto on the buffer front.
This also avoids having both io_rsrc_data->nodes being an allocated array, and ->user_bufs[] or ->file_table.nodes. There's no reason to have this information duplicated. Keep it in one spot, io_rsrc_data, along with how many resources are available.
--- include/linux/io_uring_types.h | 15 ++- io_uring/cancel.c | 4 +- io_uring/fdinfo.c | 10 +- io_uring/filetable.c | 47 ++++---- io_uring/filetable.h | 2 +- io_uring/io_uring.c | 7 +- io_uring/msg_ring.c | 4 +- io_uring/net.c | 6 +- io_uring/nop.c | 6 +- io_uring/register.c | 3 +- io_uring/rsrc.c | 212 ++++++++++----------------------- io_uring/rsrc.h | 17 ++- io_uring/rw.c | 6 +- io_uring/splice.c | 6 +- io_uring/uring_cmd.c | 6 +- 15 files changed, 131 insertions(+), 220 deletions(-)
include/linux/io_uring_types.h +7 -8
@@ -55,8 +55,13 @@ struct io_wq_work {int cancel_seq; }; +struct io_rsrc_data { +unsigned int nr; +struct io_rsrc_node **nodes; +}; +struct io_file_table { -struct io_rsrc_node **nodes; +struct io_rsrc_data data;unsigned long *bitmap;unsigned int alloc_hint; };
@@ -276,9 +281,7 @@ struct io_ring_ctx {struct io_wq_work_list iopoll_list;struct io_file_table file_table; -struct io_rsrc_node **user_bufs; -unsigned nr_user_files; -unsigned nr_user_bufs; +struct io_rsrc_data buf_table;struct io_submit_state submit_state;
@@ -366,10 +369,6 @@ struct io_ring_ctx {struct wait_queue_head poll_wq;struct io_restriction restrictions; -/* slow path rsrc auxilary data, used by update/register */ -struct io_rsrc_data *file_data; -struct io_rsrc_data *buf_data; -u32 pers_next;struct xarray personalities;
io_uring/cancel.c +2 -2
@@ -240,9 +240,9 @@ static int __io_sync_cancel(struct io_uring_task *tctx,/* fixed must be grabbed every time since we drop the uring_lock */ if ((cd->flags &IORING_ASYNC_CANCEL_FD ) && (cd->flags &IORING_ASYNC_CANCEL_FD_FIXED )) { -if (unlikely (fd >= ctx->nr_user_files)) +if (unlikely (fd >= ctx->file_table.data.nr))return -EBADF ; - fd =array_index_nospec (fd, ctx->nr_user_files); + fd =array_index_nospec (fd, ctx->file_table.data.nr); cd->file =io_file_from_index (&ctx->file_table, fd);if (!cd->file)return -EBADF ;
io_uring/fdinfo.c +5 -5
@@ -165,8 +165,8 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file)seq_printf (m,"SqThreadCpu:\t%d\n" , sq_cpu);seq_printf (m,"SqTotalTime:\t%llu\n" , sq_total_time);seq_printf (m,"SqWorkTime:\t%llu\n" , sq_work_time); -seq_printf (m,"UserFiles:\t%u\n" , ctx->nr_user_files); -for (i =0 ; has_lock && i < ctx->nr_user_files; i++) { +seq_printf (m,"UserFiles:\t%u\n" , ctx->file_table.data.nr); +for (i =0 ; has_lock && i < ctx->file_table.data.nr; i++) {struct file *f =io_file_from_index (&ctx->file_table, i);if (f)
@@ -174,9 +174,9 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file)else seq_printf (m,"%5u: <none>\n" , i); } -seq_printf (m,"UserBufs:\t%u\n" , ctx->nr_user_bufs); -for (i =0 ; has_lock && i < ctx->nr_user_bufs; i++) { -struct io_mapped_ubuf *buf = ctx->user_bufs[i]->buf; +seq_printf (m,"UserBufs:\t%u\n" , ctx->buf_table.nr); +for (i =0 ; has_lock && i < ctx->buf_table.nr; i++) { +struct io_mapped_ubuf *buf = ctx->buf_table.nodes[i]->buf;seq_printf (m,"%5u: 0x%llx/%u\n" , i, buf->ubuf, buf->len); }
io_uring/filetable.c +21 -26
91 lines not on this page · raw message
io_uring/filetable.h +1 -1
@@ -52,7 +52,7 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node)static inline struct file *io_file_from_index (struct io_file_table *table,int index) { -struct io_rsrc_node *node = table->nodes[index]; +struct io_rsrc_node *node = table->data.nodes[index];if (node)return io_slot_file (node);
io_uring/io_uring.c +1 -6
@@ -1879,17 +1879,12 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd,struct file *file =NULL ;io_ring_submit_lock (ctx, issue_flags); - -if (unlikely ((unsigned int )fd >= ctx->nr_user_files)) - goto out; - fd =array_index_nospec (fd, ctx->nr_user_files); - node = ctx->file_table.nodes[fd]; + node =io_rsrc_node_lookup (&ctx->file_table.data, fd);if (node) {io_req_assign_rsrc_node (req, node); req->flags |=io_slot_flags (node); file =io_slot_file (node); } -out:io_ring_submit_unlock (ctx, issue_flags);return file; }
io_uring/msg_ring.c +2 -2
@@ -180,8 +180,8 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flint idx = msg->src_fd;io_ring_submit_lock (ctx, issue_flags); -if (likely (idx < ctx->nr_user_files)) { - idx =array_index_nospec (idx, ctx->nr_user_files); +if (likely (idx < ctx->file_table.data.nr)) { + idx =array_index_nospec (idx, ctx->file_table.data.nr); file =io_file_from_index (&ctx->file_table, idx);if (file)get_file (file);
io_uring/net.c +3 -3
@@ -1347,9 +1347,9 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags); -if (sr->buf_index < ctx->nr_user_bufs) { - idx =array_index_nospec (sr->buf_index, ctx->nr_user_bufs); - node = ctx->user_bufs[idx]; +if (sr->buf_index < ctx->buf_table.nr) { + idx =array_index_nospec (sr->buf_index, ctx->buf_table.nr); + node = ctx->buf_table.nodes[idx];io_req_assign_rsrc_node (sr->notif, node); ret =0 ; }
io_uring/nop.c +3 -3
@@ -66,9 +66,9 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags); -if (nop->buffer < ctx->nr_user_bufs) { - idx =array_index_nospec (nop->buffer, ctx->nr_user_bufs); - node =READ_ONCE (ctx->user_bufs[idx]); +if (nop->buffer < ctx->buf_table.nr) { + idx =array_index_nospec (nop->buffer, ctx->buf_table.nr); + node =READ_ONCE (ctx->buf_table.nodes[idx]);io_req_assign_rsrc_node (req, node); ret =0 ; }
io_uring/register.c +2 -1
@@ -937,7 +937,8 @@ SYSCALL_DEFINE4(io_uring_register, unsigned int, fd, unsigned int, opcode,mutex_lock (&ctx->uring_lock); ret =__io_uring_register (ctx, opcode, arg, nr_args);mutex_unlock (&ctx->uring_lock); -trace_io_uring_register (ctx, opcode, ctx->nr_user_files, ctx->nr_user_bufs, ret); +trace_io_uring_register (ctx, opcode, ctx->file_table.data.nr, + ctx->buf_table.nr, ret);if (!use_registered_ring)fput (file);return ret;
io_uring/rsrc.c +65 -147
417 lines not on this page · raw message
io_uring/rsrc.h +12 -5
@@ -14,11 +14,6 @@ enum { IORING_RSRC_INVALID = 2, }; -struct io_rsrc_data { -unsigned int nr; -struct io_rsrc_node **nodes; -}; -struct io_rsrc_node {struct io_ring_ctx *ctx;int refs;
@@ -54,6 +49,8 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, struct io_rsrc_data *data, int index, int type);void io_free_rsrc_node (struct io_rsrc_node *node); +void io_rsrc_data_free (struct io_rsrc_data *data); +int io_rsrc_data_alloc (struct io_rsrc_data *data,unsigned nr); int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu,
@@ -77,6 +74,16 @@ int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg,extern const struct io_rsrc_node empty_node;#define rsrc_empty_node (struct io_rsrc_node *) &empty_node +static inline struct io_rsrc_node *io_rsrc_node_lookup (struct io_rsrc_data *data, +int index) +{ +if (index < data->nr) { + index =array_index_nospec (index, data->nr); +return data->nodes[index]; + } +return NULL ; +} +static inline void io_put_rsrc_node (struct io_rsrc_node *node) {if (node != &empty_node && !--node->refs)
io_uring/rw.c +3 -3
@@ -339,10 +339,10 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqeif (unlikely (ret))return ret; -if (unlikely (req->buf_index >= ctx->nr_user_bufs)) +if (unlikely (req->buf_index >= ctx->buf_table.nr))return -EFAULT ; - index =array_index_nospec (req->buf_index, ctx->nr_user_bufs); - node = ctx->user_bufs[index]; + index =array_index_nospec (req->buf_index, ctx->buf_table.nr); + node = ctx->buf_table.nodes[index];io_req_assign_rsrc_node (req, node); io = req->async_data;
io_uring/splice.c +1 -5
@@ -66,17 +66,13 @@ static struct file *io_splice_get_file(struct io_kiocb *req,return io_file_get_normal (req, sp->splice_fd_in);io_ring_submit_lock (ctx, issue_flags); -if (unlikely (sp->splice_fd_in >= ctx->nr_user_files)) - goto out; - sp->splice_fd_in =array_index_nospec (sp->splice_fd_in, ctx->nr_user_files); - node = ctx->file_table.nodes[sp->splice_fd_in]; + node =io_rsrc_node_lookup (&ctx->file_table.data, sp->splice_fd_in);if (node) { node->refs++; sp->rsrc_node = node; file =io_slot_file (node); req->flags |=REQ_F_NEED_CLEANUP ; } -out:io_ring_submit_unlock (ctx, issue_flags);return file; }
io_uring/uring_cmd.c +3 -3
@@ -212,15 +212,15 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)u16 index; index =READ_ONCE (sqe->buf_index); -if (unlikely (index >= ctx->nr_user_bufs)) +if (unlikely (index >= ctx->buf_table.nr))return -EFAULT ; - req->buf_index =array_index_nospec (index, ctx->nr_user_bufs); + req->buf_index =array_index_nospec (index, ctx->buf_table.nr);/* * Pi node upfront, prior to io_uring_cmd_import_fixed() * being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ -io_req_assign_rsrc_node (req, ctx->user_bufs[req->buf_index]); +io_req_assign_rsrc_node (req, ctx->buf_table.nodes[req->buf_index]); } ioucmd->cmd_op =READ_ONCE (sqe->cmd_op);
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 11/13 io_uring/rsrc: use io_rsrc_node_lookup() consistently
In reply to Jens AxboeVarious spots open code this helper, use the provided one. The helper now returns the adjusted, if needed, index into the array, for the cases where the caller needs that.
--- io_uring/cancel.c | 8 +++++--- io_uring/filetable.c | 24 +++++++++++++----------- io_uring/filetable.h | 9 +++++---- io_uring/io_uring.c | 2 +- io_uring/msg_ring.c | 33 ++++++++++++++++----------------- io_uring/net.c | 7 +++---- io_uring/nop.c | 6 ++---- io_uring/rsrc.c | 15 +++++++++------ io_uring/rsrc.h | 8 ++++---- io_uring/rw.c | 9 ++++----- io_uring/splice.c | 2 +- io_uring/uring_cmd.c | 10 +++++----- 12 files changed, 68 insertions(+), 65 deletions(-)
io_uring/cancel.c +5 -3
@@ -240,10 +240,12 @@ static int __io_sync_cancel(struct io_uring_task *tctx,/* fixed must be grabbed every time since we drop the uring_lock */ if ((cd->flags &IORING_ASYNC_CANCEL_FD ) && (cd->flags &IORING_ASYNC_CANCEL_FD_FIXED )) { -if (unlikely (fd >= ctx->file_table.data.nr)) +struct io_rsrc_node *node; + + node =io_rsrc_node_lookup (&ctx->file_table.data, &fd); +if (unlikely (!node))return -EBADF ; - fd =array_index_nospec (fd, ctx->file_table.data.nr); - cd->file =io_file_from_index (&ctx->file_table, fd); + cd->file =io_slot_file (node);if (!cd->file)return -EBADF ; }
io_uring/filetable.c +13 -11
@@ -55,10 +55,10 @@ void io_free_file_tables(struct io_file_table *table) }static int io_install_fixed_file (struct io_ring_ctx *ctx,struct file *file, -u32 slot_index) +int slot_index) __must_hold(&req->ctx->uring_lock) { -struct io_rsrc_node *node; +struct io_rsrc_node *node, *old_node;if (io_is_uring_fops (file))return -EBADF ;
@@ -72,9 +72,9 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file,if (IS_ERR (node))return -ENOMEM ; - slot_index =array_index_nospec (slot_index, ctx->file_table.data.nr); -if (ctx->file_table.data.nodes[slot_index]) -io_put_rsrc_node (ctx->file_table.data.nodes[slot_index]); + old_node =io_rsrc_node_lookup (&ctx->file_table.data, &slot_index); +if (old_node) +io_put_rsrc_node (old_node);else io_file_bitmap_set (&ctx->file_table, slot_index);
@@ -84,7 +84,7 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, }int __io_fixed_fd_install (struct io_ring_ctx *ctx,struct file *file, -unsigned int file_slot) +int file_slot) {bool alloc_slot = file_slot ==IORING_FILE_INDEX_ALLOC ;int ret;
@@ -108,7 +108,7 @@ int __io_fixed_fd_install(struct io_ring_ctx *ctx, struct file *file,* fput() is called correspondingly. */ int io_fixed_fd_install (struct io_kiocb *req,unsigned int issue_flags, -struct file *file,unsigned int file_slot) +struct file *file,int file_slot) {struct io_ring_ctx *ctx = req->ctx;int ret;
@@ -122,17 +122,19 @@ int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags,return ret; } -int io_fixed_fd_remove (struct io_ring_ctx *ctx,unsigned int offset) +int io_fixed_fd_remove (struct io_ring_ctx *ctx,int offset) { +struct io_rsrc_node *node; +if (unlikely (!ctx->file_table.data.nr))return -ENXIO ;if (offset >= ctx->file_table.data.nr)return -EINVAL ; - offset =array_index_nospec (offset, ctx->file_table.data.nr); -if (!ctx->file_table.data.nodes[offset]) + node =io_rsrc_node_lookup (&ctx->file_table.data, &offset); +if (!node)return -EBADF ; -io_put_rsrc_node (ctx->file_table.data.nodes[offset]); +io_put_rsrc_node (node); ctx->file_table.data.nodes[offset] =NULL ;io_file_bitmap_clear (&ctx->file_table, offset);return 0 ;
io_uring/filetable.h +5 -4
@@ -10,10 +10,10 @@ bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files);void io_free_file_tables (struct io_file_table *table);int io_fixed_fd_install (struct io_kiocb *req,unsigned int issue_flags, -struct file *file,unsigned int file_slot); +struct file *file,int file_slot);int __io_fixed_fd_install (struct io_ring_ctx *ctx,struct file *file, -unsigned int file_slot); -int io_fixed_fd_remove (struct io_ring_ctx *ctx,unsigned int offset); +int file_slot); +int io_fixed_fd_remove (struct io_ring_ctx *ctx,int offset);int io_register_file_alloc_range (struct io_ring_ctx *ctx,struct io_uring_file_index_range __user *arg);
@@ -52,8 +52,9 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node)static inline struct file *io_file_from_index (struct io_file_table *table,int index) { -struct io_rsrc_node *node = table->data.nodes[index]; +struct io_rsrc_node *node; + node =io_rsrc_node_lookup (&table->data, &index);if (node)return io_slot_file (node);return NULL ;
io_uring/io_uring.c +1 -1
@@ -1879,7 +1879,7 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd,struct file *file =NULL ;io_ring_submit_lock (ctx, issue_flags); - node =io_rsrc_node_lookup (&ctx->file_table.data, fd); + node =io_rsrc_node_lookup (&ctx->file_table.data, &fd);if (node) {io_req_assign_rsrc_node (req, node); req->flags |=io_slot_flags (node);
io_uring/msg_ring.c +16 -17
@@ -25,7 +25,7 @@ struct io_msg {u64 user_data;u32 len;u32 cmd; -u32 src_fd; +int src_fd;union {u32 dst_fd;u32 cqe_flags;
@@ -172,22 +172,24 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags)return __io_msg_ring_data (target_ctx, msg, issue_flags); } -static struct file *io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags) +static int io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);struct io_ring_ctx *ctx = req->ctx; -struct file *file =NULL ; -int idx = msg->src_fd; +struct io_rsrc_node *node; +int ret = -EBADF ;io_ring_submit_lock (ctx, issue_flags); -if (likely (idx < ctx->file_table.data.nr)) { - idx =array_index_nospec (idx, ctx->file_table.data.nr); - file =io_file_from_index (&ctx->file_table, idx); -if (file) -get_file (file); + node =io_rsrc_node_lookup (&ctx->file_table.data, &msg->src_fd); +if (node) { + msg->src_file =io_slot_file (node); +if (msg->src_file) +get_file (msg->src_file); + req->flags |=REQ_F_NEED_CLEANUP ; + ret =0 ; }io_ring_submit_unlock (ctx, issue_flags); -return file; +return ret; }static int io_msg_install_complete (struct io_kiocb *req,unsigned int issue_flags)
@@ -256,7 +258,6 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);struct io_ring_ctx *ctx = req->ctx; -struct file *src_file = msg->src_file;if (msg->len)return -EINVAL ;
@@ -264,12 +265,10 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)return -EINVAL ;if (target_ctx->flags &IORING_SETUP_R_DISABLED )return -EBADFD ; -if (!src_file) { - src_file =io_msg_grab_file (req, issue_flags); -if (!src_file) -return -EBADF ; - msg->src_file = src_file; - req->flags |=REQ_F_NEED_CLEANUP ; +if (!msg->src_file) { +int ret =io_msg_grab_file (req, issue_flags); +if (unlikely (ret)) +return ret; }if (io_msg_need_remote (target_ctx))
io_uring/net.c +3 -4
@@ -1343,13 +1343,12 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags)if (sr->flags &IORING_RECVSEND_FIXED_BUF ) {struct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node; -int idx; +int idx = sr->buf_index; ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags); -if (sr->buf_index < ctx->buf_table.nr) { - idx =array_index_nospec (sr->buf_index, ctx->buf_table.nr); - node = ctx->buf_table.nodes[idx]; + node =io_rsrc_node_lookup (&ctx->buf_table, &idx); +if (node) {io_req_assign_rsrc_node (sr->notif, node); ret =0 ; }
io_uring/nop.c +2 -4
@@ -62,13 +62,11 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags)if (nop->flags &IORING_NOP_FIXED_BUFFER ) {struct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node; -int idx; ret = -EFAULT ;io_ring_submit_lock (ctx, issue_flags); -if (nop->buffer < ctx->buf_table.nr) { - idx =array_index_nospec (nop->buffer, ctx->buf_table.nr); - node =READ_ONCE (ctx->buf_table.nodes[idx]); + node =io_rsrc_node_lookup (&ctx->buf_table, &nop->buffer); +if (node) {io_req_assign_rsrc_node (req, node); ret =0 ; }
io_uring/rsrc.c +9 -6
@@ -186,6 +186,7 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,return -EINVAL ;for (done =0 ; done < nr_args; done++) { +struct io_rsrc_node *node;u64 tag =0 ; if ((tags && copy_from_user(&tag, &tags[done], sizeof(tag))) ||
@@ -200,9 +201,10 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,if (fd ==IORING_REGISTER_FILES_SKIP )continue ; - i =array_index_nospec (up->offset + done, ctx->file_table.data.nr); -if (ctx->file_table.data.nodes[i]) { -io_put_rsrc_node (ctx->file_table.data.nodes[i]); + i = up->offset + done; + node =io_rsrc_node_lookup (&ctx->file_table.data, &i); +if (node) { +io_put_rsrc_node (node); ctx->file_table.data.nodes[i] =NULL ;io_file_bitmap_clear (&ctx->file_table, i); }
@@ -965,13 +967,14 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx goto out_unlock;for (i =0 ; i < nbufs; i++) { -struct io_rsrc_node *src_node = src_ctx->buf_table.nodes[i]; -struct io_rsrc_node *dst_node; +struct io_rsrc_node *dst_node, *src_node; +int index = i; + src_node =io_rsrc_node_lookup (&src_ctx->buf_table, &index);if (src_node == rsrc_empty_node) { dst_node = rsrc_empty_node; }else { - dst_node =io_rsrc_node_alloc (ctx, &data, i,IORING_RSRC_BUFFER ); + dst_node =io_rsrc_node_alloc (ctx, &data, index,IORING_RSRC_BUFFER );if (!dst_node) goto out_put_free;
io_uring/rsrc.h +4 -4
@@ -75,11 +75,11 @@ extern const struct io_rsrc_node empty_node;#define rsrc_empty_node (struct io_rsrc_node *) &empty_nodestatic inline struct io_rsrc_node *io_rsrc_node_lookup (struct io_rsrc_data *data, -int index) +int *index) { -if (index < data->nr) { - index =array_index_nospec (index, data->nr); -return data->nodes[index]; +if (*index < data->nr) { + *index =array_index_nospec (*index, data->nr); +return data->nodes[*index]; }return NULL ; }
io_uring/rw.c +4 -5
@@ -332,17 +332,16 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqestruct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node;struct io_async_rw *io; -u16 index; -int ret; +int index, ret; ret =io_prep_rw (req, sqe, ddir, false);if (unlikely (ret))return ret; -if (unlikely (req->buf_index >= ctx->buf_table.nr)) + index = req->buf_index; + node =io_rsrc_node_lookup (&ctx->buf_table, &index); +if (!node)return -EFAULT ; - index =array_index_nospec (req->buf_index, ctx->buf_table.nr); - node = ctx->buf_table.nodes[index];io_req_assign_rsrc_node (req, node); io = req->async_data;
io_uring/splice.c +1 -1
@@ -66,7 +66,7 @@ static struct file *io_splice_get_file(struct io_kiocb *req,return io_file_get_normal (req, sp->splice_fd_in);io_ring_submit_lock (ctx, issue_flags); - node =io_rsrc_node_lookup (&ctx->file_table.data, sp->splice_fd_in); + node =io_rsrc_node_lookup (&ctx->file_table.data, &sp->splice_fd_in);if (node) { node->refs++; sp->rsrc_node = node;
io_uring/uring_cmd.c +5 -5
@@ -209,18 +209,18 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (ioucmd->flags &IORING_URING_CMD_FIXED ) {struct io_ring_ctx *ctx = req->ctx; -u16 index; +int index = req->buf_index; +struct io_rsrc_node *node; - index =READ_ONCE (sqe->buf_index); -if (unlikely (index >= ctx->buf_table.nr)) + node =io_rsrc_node_lookup (&ctx->buf_table, &index); +if (unlikely (!node))return -EFAULT ; - req->buf_index =array_index_nospec (index, ctx->buf_table.nr);/* * Pi node upfront, prior to io_uring_cmd_import_fixed() * being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ -io_req_assign_rsrc_node (req, ctx->buf_table.nodes[req->buf_index]); +io_req_assign_rsrc_node (req, node); } ioucmd->cmd_op =READ_ONCE (sqe->cmd_op);
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 12/13 io_uring/filetable: remove io_file_from_index() helper
In reply to Jens AxboeIt's only used in fdinfo, nothing really gained from having this helper.
--- io_uring/fdinfo.c | 4 +++- io_uring/filetable.h | 11 ----------- 2 files changed, 3 insertions(+), 12 deletions(-)
io_uring/fdinfo.c +3 -1
@@ -167,8 +167,10 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file)seq_printf (m,"SqWorkTime:\t%llu\n" , sq_work_time);seq_printf (m,"UserFiles:\t%u\n" , ctx->file_table.data.nr);for (i =0 ; has_lock && i < ctx->file_table.data.nr; i++) { -struct file *f =io_file_from_index (&ctx->file_table, i); +struct file *f =NULL ; +if (ctx->file_table.data.nodes[i]) + f =io_slot_file (ctx->file_table.data.nodes[i]);if (f)seq_printf (m,"%5u: %s\n" , i,file_dentry (f)->d_iname);else
io_uring/filetable.h +0 -11
@@ -49,17 +49,6 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node)return (struct file *)(node->file_ptr &FFS_MASK ); } -static inline struct file *io_file_from_index (struct io_file_table *table, -int index) -{ -struct io_rsrc_node *node; - - node =io_rsrc_node_lookup (&table->data, &index); -if (node) -return io_slot_file (node); -return NULL ; -} -static inline void io_fixed_file_set (struct io_rsrc_node *node,struct file *file) {
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH v2 13/13 io_uring/filetable: kill io_reset_alloc_hint() helper
In reply to Jens AxboeIt's only used internally, and in one spot, just open-code ti.
--- io_uring/filetable.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
io_uring/filetable.h +1 -6
@@ -56,17 +56,12 @@ static inline void io_fixed_file_set(struct io_rsrc_node *node, (io_file_get_flags(file) >> REQ_F_SUPPORT_NOWAIT_BIT); } -static inline void io_reset_alloc_hint (struct io_ring_ctx *ctx) -{ - ctx->file_table.alloc_hint = ctx->file_alloc_start; -} -static inline void io_file_table_set_alloc_range (struct io_ring_ctx *ctx,unsigned off,unsigned len) { ctx->file_alloc_start = off; ctx->file_alloc_end = off + len; -io_reset_alloc_hint (ctx); + ctx->file_table.alloc_hint = ctx->file_alloc_start; }#endif
Signature · 1 line
2.45.2