CQ locking optimisation
v2 cover CQ locking optimisation
Optimise CQ locking for event posting depending on a number of ring setup flags. QD1 nop benchmark showed 12.067 -> 12.565 MIOPS increase, which more than 8.5% of the io_uring kernel overhead (taking into account that the syscall overhead is over 50%) or 4.12% of the total performance. Naturally, it's not only about QD1, applications can submit a bunch of requests but their completions will may arrive randomly hurting batching and so performance (or latency).
The downside is that we have to punt all io-wq completions to the original task. The performance win should diminish with better completion batching, but it should be worth it for as it also helps tw, which in reality often don't complete too many requests.
The feature depends on DEFER_TASKRUN but can be relaxed to SINGLE_ISSUER
v2: some general msg_ring fixes (patche 1,2)
fix exiting ring potentially modifying CQ in parallel (8/12) use task_work instead of overflowing msg_ring CQEs, which could've
messed with CQE ordering (9-11)
Pavel Begunkov (12):
io_uring: dont remove file from msg_ring reqs
io_uring: improve io_double_lock_ctx fail handling
io_uring: skip overflow CQE posting for dying ring
io_uring: don't check overflow flush failures
io_uring: complete all requests in task context
io_uring: force multishot CQEs into task context
io_uring: use tw for putting rsrc
io_uring: never run tw and fallback in parallel
io_uring: get rid of double locking
io_uring: extract a io_msg_install_complete helper
io_uring: do msg_ring in target task via tw
io_uring: skip spinlocking for ->task_complete
include/linux/io_uring.h | 2 + include/linux/io_uring_types.h | 3 + io_uring/io_uring.c | 165 ++++++++++++++++++++++----------- io_uring/io_uring.h | 12 ++- io_uring/msg_ring.c | 163 ++++++++++++++++++++++---------- io_uring/msg_ring.h | 1 + io_uring/net.c | 21 +++++ io_uring/opdef.c | 8 ++ io_uring/opdef.h | 2 + io_uring/rsrc.c | 19 +++- io_uring/rsrc.h | 1 + 11 files changed, 291 insertions(+), 106 deletions(-)
Signature · 1 line
2.38.1
PATCH v2 1/12 io_uring: dont remove file from msg_ring reqs
We should not be messing with req->file outside of core paths. Clearing it makes msg_ring non reentrant, i.e. luckily io_msg_send_fd() fails the request on failed io_double_lock_ctx() but clearly was originally intended to do retries instead.
--- io_uring/io_uring.c | 2 +- io_uring/msg_ring.c | 4 ---- io_uring/opdef.c | 7 +++++++ io_uring/opdef.h | 2 ++ 4 files changed, 10 insertions(+), 5 deletions(-)
io_uring/io_uring.c +1 -1
@@ -1810,7 +1810,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)return ret;/* If the op doesn't have a file, we're not polling for it */ -if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && req->file) +if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && def->iopoll_queue)io_iopoll_req_issued (req, issue_flags);return 0 ;
io_uring/msg_ring.c +0 -4
@@ -167,9 +167,5 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags)if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 ); -/* put file to avoid an attempt to IOPOLL the req */ -if (!(req->flags &REQ_F_FIXED_FILE )) -io_put_file (req->file); - req->file =NULL ;return IOU_OK ; }
io_uring/opdef.c +7 -0
@@ -63,6 +63,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "READV", .prep = io_prep_rw,
@@ -80,6 +81,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "WRITEV", .prep = io_prep_rw,
@@ -103,6 +105,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "READ_FIXED", .prep = io_prep_rw,
@@ -118,6 +121,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "WRITE_FIXED", .prep = io_prep_rw,
@@ -277,6 +281,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "READ", .prep = io_prep_rw,
@@ -292,6 +297,7 @@ const struct io_op_def io_op_defs[] = { .audit_skip = 1, .ioprio = 1, .iopoll = 1, + .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .name = "WRITE", .prep = io_prep_rw,
@@ -481,6 +487,7 @@ const struct io_op_def io_op_defs[] = { .plug = 1, .name = "URING_CMD", .iopoll = 1, + .iopoll_queue = 1, .async_size = uring_cmd_pdu_size(1), .prep = io_uring_cmd_prep, .issue = io_uring_cmd,
io_uring/opdef.h +2 -0
@@ -25,6 +25,8 @@ struct io_op_def { unsigned ioprio : 1;/* supports iopoll */ unsigned iopoll : 1; +/* have to be put into the iopoll list */ + unsigned iopoll_queue : 1;/* opcode specific path will handle ->async_data allocation if needed */ unsigned manual_alloc : 1;/* size of async data needed, if any */
Signature · 1 line
2.38.1
PATCH v2 2/12 io_uring: improve io_double_lock_ctx fail handling
In reply to Pavel Begunkovmsg_ring will fail the request if it can't lock rings, instead punt it to io-wq as was originally intended.
--- io_uring/msg_ring.c | 2 ++ 1 file changed, 2 insertions(+)
io_uring/msg_ring.c +2 -0
@@ -164,6 +164,8 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags) } done: +if (ret == -EAGAIN ) +return -EAGAIN ;if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 );
Signature · 1 line
2.38.1
PATCH v2 3/12 io_uring: skip overflow CQE posting for dying ring
In reply to Pavel BegunkovAfter io_ring_ctx_wait_and_kill() is called there should be no users poking into rings and so there is no need to post CQEs. So, instead of trying to post overflowed CQEs into the CQ, drop them. Also, do it in io_ring_exit_work() in a loop to reduce the number of contexts it can be executed from and even when it struggles to quiesce the ring we won't be leaving memory allocated for longer than needed.
--- io_uring/io_uring.c | 45 +++++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-)
io_uring/io_uring.c +31 -14
@@ -611,12 +611,30 @@ void io_cq_unlock_post(struct io_ring_ctx *ctx) }/* Returns true if there are no backlogged entries after the flush */ -static bool __io_cqring_overflow_flush (struct io_ring_ctx *ctx,bool force) +static void io_cqring_overflow_kill (struct io_ring_ctx *ctx) +{ +struct io_overflow_cqe *ocqe; +LIST_HEAD (list); + +io_cq_lock (ctx); +list_splice_init (&ctx->cq_overflow_list, &list); +clear_bit (IO_CHECK_CQ_OVERFLOW_BIT , &ctx->check_cq); +io_cq_unlock (ctx); + +while (!list_empty (&list)) { + ocqe = list_first_entry(&list, struct io_overflow_cqe, list); +list_del (&ocqe->list); +kfree (ocqe); + } +} + +/* Returns true if there are no backlogged entries after the flush */ +static bool __io_cqring_overflow_flush (struct io_ring_ctx *ctx) {bool all_flushed;size_t cqe_size =sizeof (struct io_uring_cqe ); -if (!force &&__io_cqring_events (ctx) == ctx->cq_entries) +if (__io_cqring_events (ctx) == ctx->cq_entries)return false;if (ctx->flags &IORING_SETUP_CQE32 )
@@ -627,15 +645,11 @@ static bool __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool force)struct io_uring_cqe *cqe =io_get_cqe_overflow (ctx, true);struct io_overflow_cqe *ocqe; -if (!cqe && !force) +if (!cqe)break ; ocqe =list_first_entry (&ctx->cq_overflow_list, struct io_overflow_cqe, list); -if (cqe) -memcpy (cqe, &ocqe->cqe, cqe_size); -else -io_account_cq_overflow (ctx); - +memcpy (cqe, &ocqe->cqe, cqe_size);list_del (&ocqe->list);kfree (ocqe); }
@@ -658,7 +672,7 @@ static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx)/* iopoll syncs against uring_lock, not completion_lock */ if (ctx->flags &IORING_SETUP_IOPOLL )mutex_lock (&ctx->uring_lock); - ret =__io_cqring_overflow_flush (ctx, false); + ret =__io_cqring_overflow_flush (ctx);if (ctx->flags &IORING_SETUP_IOPOLL )mutex_unlock (&ctx->uring_lock); }
@@ -1478,7 +1492,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) check_cq =READ_ONCE (ctx->check_cq);if (unlikely (check_cq)) {if (check_cq &BIT (IO_CHECK_CQ_OVERFLOW_BIT )) -__io_cqring_overflow_flush (ctx, false); +__io_cqring_overflow_flush (ctx);/* * Similarly do not spin if we have not informed the user of any * dropped CQE.
@@ -2646,8 +2660,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)__io_sqe_buffers_unregister (ctx);if (ctx->file_data)__io_sqe_files_unregister (ctx); -if (ctx->rings) -__io_cqring_overflow_flush (ctx, true); +io_cqring_overflow_kill (ctx);io_eventfd_unregister (ctx);io_alloc_cache_free (&ctx->apoll_cache, io_apoll_cache_free);io_alloc_cache_free (&ctx->netmsg_cache, io_netmsg_cache_free);
@@ -2788,6 +2801,12 @@ static __cold void io_ring_exit_work(struct work_struct *work)* as nobody else will be looking for them. */ do { + if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) { + mutex_lock(&ctx->uring_lock); + io_cqring_overflow_kill(ctx); + mutex_unlock(&ctx->uring_lock); + } + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) io_move_task_work_from_local(ctx);
@@ -2853,8 +2872,6 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)mutex_lock (&ctx->uring_lock);percpu_ref_kill (&ctx->refs); -if (ctx->rings) -__io_cqring_overflow_flush (ctx, true);xa_for_each (&ctx->personalities, index, creds)io_unregister_personality (ctx, index);if (ctx->rings)
Signature · 1 line
2.38.1
PATCH v2 4/12 io_uring: don't check overflow flush failures
In reply to Pavel BegunkovThe only way to fail overflowed CQEs flush is for CQ to be fully packed. There is one place checking for flush failures, i.e. io_cqring_wait(), but we limit the number to be waited for by the CQ size, so getting a failure automatically means that we're done with waiting.
Don't check for failures, rarely but they might spuriously fail CQ waiting with -EBUSY.
--- io_uring/io_uring.c | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-)
io_uring/io_uring.c +6 -18
@@ -629,13 +629,12 @@ static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) }/* Returns true if there are no backlogged entries after the flush */ -static bool __io_cqring_overflow_flush (struct io_ring_ctx *ctx) +static void __io_cqring_overflow_flush (struct io_ring_ctx *ctx) { -bool all_flushed;size_t cqe_size =sizeof (struct io_uring_cqe );if (__io_cqring_events (ctx) == ctx->cq_entries) -return false; +return ;if (ctx->flags &IORING_SETUP_CQE32 ) cqe_size <<=1 ;
@@ -654,30 +653,23 @@ static bool __io_cqring_overflow_flush(struct io_ring_ctx *ctx)kfree (ocqe); } - all_flushed =list_empty (&ctx->cq_overflow_list); -if (all_flushed) { +if (list_empty (&ctx->cq_overflow_list)) {clear_bit (IO_CHECK_CQ_OVERFLOW_BIT , &ctx->check_cq);atomic_andnot (IORING_SQ_CQ_OVERFLOW , &ctx->rings->sq_flags); } -io_cq_unlock_post (ctx); -return all_flushed; } -static bool io_cqring_overflow_flush (struct io_ring_ctx *ctx) +static void io_cqring_overflow_flush (struct io_ring_ctx *ctx) { -bool ret = true; -if (test_bit (IO_CHECK_CQ_OVERFLOW_BIT , &ctx->check_cq)) {/* iopoll syncs against uring_lock, not completion_lock */ if (ctx->flags &IORING_SETUP_IOPOLL )mutex_lock (&ctx->uring_lock); - ret =__io_cqring_overflow_flush (ctx); +__io_cqring_overflow_flush (ctx);if (ctx->flags &IORING_SETUP_IOPOLL )mutex_unlock (&ctx->uring_lock); } - -return ret; }void __io_put_task (struct task_struct *task,int nr)
@@ -2505,11 +2497,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,trace_io_uring_cqring_wait (ctx, min_events); do { - /* if we can't even flush overflow, don't wait for more */ - if (!io_cqring_overflow_flush(ctx)) { - ret = -EBUSY; - break; - } + io_cqring_overflow_flush(ctx); prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, timeout);
Signature · 1 line
2.38.1
PATCH v2 5/12 io_uring: complete all requests in task context
In reply to Pavel BegunkovThis patch adds ctx->task_complete flag. If set, we'll complete all requests in the context of the original task. Note, this extends to completion CQE posting only but not io_kiocb cleanup / free, e.g. io-wq may free the requests in the free calllback. This flag will be used later for optimisations purposes.
--- include/linux/io_uring.h | 2 ++ include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 14 +++++++++++--- 3 files changed, 15 insertions(+), 3 deletions(-)
include/linux/io_uring.h +2 -0
@@ -11,6 +11,8 @@ enum io_uring_cmd_flags {IO_URING_F_UNLOCKED =2 ,/* the request is executed from poll, it should not be freed */ IO_URING_F_MULTISHOT =4 , +/* executed by io-wq */ +IO_URING_F_IOWQ =8 ,/* int's last bit, sign checks are usually faster than a bit test */ IO_URING_F_NONBLOCK = INT_MIN,
include/linux/io_uring_types.h +2 -0
@@ -208,6 +208,8 @@ struct io_ring_ctx { unsigned int drain_disabled: 1; unsigned int has_evfd: 1; unsigned int syscall_iopoll: 1; +/* all CQEs should be posted only by the submitter task */ + unsigned int task_complete: 1; }____cacheline_aligned_in_smp ;/* submission data */
io_uring/io_uring.c +11 -3
@@ -932,8 +932,11 @@ static void __io_req_complete_post(struct io_kiocb *req)void io_req_complete_post (struct io_kiocb *req,unsigned issue_flags) { -if (!(issue_flags &IO_URING_F_UNLOCKED ) || - !(req->ctx->flags &IORING_SETUP_IOPOLL )) { +if (req->ctx->task_complete && (issue_flags &IO_URING_F_IOWQ )) { + req->io_task_work.func = io_req_task_complete; +io_req_task_work_add (req); + }else if (!(issue_flags &IO_URING_F_UNLOCKED ) || + !(req->ctx->flags &IORING_SETUP_IOPOLL )) {__io_req_complete_post (req); }else {struct io_ring_ctx *ctx = req->ctx;
@@ -1841,7 +1844,7 @@ void io_wq_submit_work(struct io_wq_work *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work);const struct io_op_def *def = &io_op_defs[req->opcode]; -unsigned int issue_flags =IO_URING_F_UNLOCKED ; +unsigned int issue_flags =IO_URING_F_UNLOCKED |IO_URING_F_IOWQ ;bool needs_poll = false;int ret =0 , err = -ECANCELED ;
@@ -3501,6 +3504,11 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,if (!ctx)return -ENOMEM ; +if ((ctx->flags &IORING_SETUP_DEFER_TASKRUN ) && + !(ctx->flags &IORING_SETUP_IOPOLL ) && + !(ctx->flags &IORING_SETUP_SQPOLL )) + ctx->task_complete = true; +/* * When SETUP_IOPOLL and SETUP_SQPOLL are both enabled, user * space applications don't need to do io completion events
Signature · 1 line
2.38.1
PATCH v2 6/12 io_uring: force multishot CQEs into task context
In reply to Pavel BegunkovMultishot are posting CQEs outside of the normal request completion path, which is usually done from within a task work handler. However, it might be not the case when it's yet to be polled but has been punted to io-wq. Make it abide ->task_complete and push it to the polling path when executed by io-wq.
--- io_uring/net.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
io_uring/net.c +21 -0
@@ -67,6 +67,19 @@ struct io_sr_msg {struct io_kiocb *notif; }; +static inline bool io_check_multishot (struct io_kiocb *req, +unsigned int issue_flags) +{ +/* +* When ->locked_cq is set we only allow to post CQEs from the original +* task context. Usual request completions will be handled in other +* generic paths but multipoll may decide to post extra cqes. +*/ +return !(issue_flags &IO_URING_F_IOWQ ) || + !(issue_flags &IO_URING_F_MULTISHOT ) || + !req->ctx->task_complete; +} +int io_shutdown_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) { struct io_shutdown *shutdown = io_kiocb_to_cmd(req, struct io_shutdown);
@@ -730,6 +743,9 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST))return io_setup_async_msg (req, kmsg, issue_flags); +if (!io_check_multishot (req, issue_flags)) +return io_setup_async_msg (req, kmsg, issue_flags); + retry_multishot:if (io_do_buffer_select (req)) {void __user *buf;
@@ -829,6 +845,9 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST))return -EAGAIN ; +if (!io_check_multishot (req, issue_flags)) +return -EAGAIN ; + sock =sock_from_file (req->file);if (unlikely (!sock))return -ENOTSOCK ;
@@ -1280,6 +1299,8 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags)struct file *file;int ret, fd; +if (!io_check_multishot (req, issue_flags)) +return -EAGAIN ; retry:if (!fixed) { fd =__get_unused_fd_flags (accept->flags, accept->nofile);
Signature · 1 line
2.38.1
PATCH v2 7/12 io_uring: use tw for putting rsrc
In reply to Pavel BegunkovUse task_work for completing rsrc removals, it'll be needed later for spinlock optimisations.
--- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 1 + io_uring/rsrc.c | 19 +++++++++++++++++-- io_uring/rsrc.h | 1 + 4 files changed, 20 insertions(+), 2 deletions(-)
include/linux/io_uring_types.h +1 -0
@@ -328,6 +328,7 @@ struct io_ring_ctx {struct io_rsrc_data *buf_data;struct delayed_work rsrc_put_work; +struct callback_head rsrc_put_tw;struct llist_head rsrc_put_llist;struct list_head rsrc_ref_list;spinlock_t rsrc_ref_lock;
io_uring/io_uring.c +1 -0
@@ -326,6 +326,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)spin_lock_init (&ctx->rsrc_ref_lock);INIT_LIST_HEAD (&ctx->rsrc_ref_list);INIT_DELAYED_WORK (&ctx->rsrc_put_work, io_rsrc_put_work); +init_task_work (&ctx->rsrc_put_tw, io_rsrc_put_tw);init_llist_head (&ctx->rsrc_put_llist);init_llist_head (&ctx->work_llist);INIT_LIST_HEAD (&ctx->tctx_list);
io_uring/rsrc.c +17 -2
@@ -204,6 +204,14 @@ void io_rsrc_put_work(struct work_struct *work) } } +void io_rsrc_put_tw (struct callback_head *cb) +{ + struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, + rsrc_put_tw); + +io_rsrc_put_work (&ctx->rsrc_put_work.work); +} +void io_wait_rsrc_data (struct io_rsrc_data *data) {if (data && !atomic_dec_and_test (&data->refs))
@@ -242,8 +250,15 @@ static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref) }spin_unlock_irqrestore (&ctx->rsrc_ref_lock, flags); -if (first_add) -mod_delayed_work (system_wq, &ctx->rsrc_put_work, delay); +if (!first_add) +return ; + +if (ctx->submitter_task) { +if (!task_work_add (ctx->submitter_task, &ctx->rsrc_put_tw, + ctx->notify_method)) +return ; + } +mod_delayed_work (system_wq, &ctx->rsrc_put_work, delay); }static struct io_rsrc_node *io_rsrc_node_alloc (void )
io_uring/rsrc.h +1 -0
@@ -53,6 +53,7 @@ struct io_mapped_ubuf {struct bio_vec bvec[]; }; +void io_rsrc_put_tw (struct callback_head *cb);void io_rsrc_put_work (struct work_struct *work);void io_rsrc_refs_refill (struct io_ring_ctx *ctx);void io_wait_rsrc_data (struct io_rsrc_data *data);
Signature · 1 line
2.38.1
PATCH v2 8/12 io_uring: never run tw and fallback in parallel
In reply to Pavel BegunkovOnce we fallback a tw we want all requests to that task to be given to the fallback wq so we dont run it in parallel with the last, i.e. post PF_EXITING, tw run of the task.
--- io_uring/io_uring.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
io_uring/io_uring.c +10 -2
@@ -149,6 +149,7 @@ static void io_clean_op(struct io_kiocb *req);static void io_queue_sqe (struct io_kiocb *req);static void io_move_task_work_from_local (struct io_ring_ctx *ctx);static void __io_submit_flush_completions (struct io_ring_ctx *ctx); +static __cold void io_fallback_tw (struct io_uring_task *tctx);static struct kmem_cache *req_cachep;
@@ -1160,10 +1161,17 @@ void tctx_task_work(struct callback_head *cb) struct io_uring_task *tctx = container_of(cb, struct io_uring_task, task_work);struct llist_node fake = {}; -struct llist_node *node =io_llist_xchg (&tctx->task_list, &fake); +struct llist_node *node;unsigned int loops =1 ; -unsigned int count =handle_tw_list (node, &ctx, &uring_locked,NULL ); +unsigned int count; + +if (unlikely (current->flags &PF_EXITING )) { +io_fallback_tw (tctx); +return ; + } + node =io_llist_xchg (&tctx->task_list, &fake); + count =handle_tw_list (node, &ctx, &uring_locked,NULL ); node =io_llist_cmpxchg (&tctx->task_list, &fake,NULL );while (node != &fake) { loops++;
Signature · 1 line
2.38.1
PATCH v2 9/12 io_uring: get rid of double locking
In reply to Pavel BegunkovWe don't need to take both uring_locks at once, msg_ring can be split in two parts, first getting a file from the filetable of the first ring and then installing it into the second one.
--- io_uring/msg_ring.c | 85 ++++++++++++++++++++++++++------------------- io_uring/msg_ring.h | 1 + io_uring/opdef.c | 1 + 3 files changed, 51 insertions(+), 36 deletions(-)
io_uring/msg_ring.c +49 -36
@@ -15,6 +15,7 @@struct io_msg {struct file *file; +struct file *src_file;u64 user_data;u32 len;u32 cmd;
@@ -23,6 +24,17 @@ struct io_msg {u32 flags; }; +void io_msg_ring_cleanup (struct io_kiocb *req) +{ + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); + +if (WARN_ON_ONCE (!msg->src_file)) +return ; + +fput (msg->src_file); + msg->src_file =NULL ; +} +static int io_msg_ring_data (struct io_kiocb *req) {struct io_ring_ctx *target_ctx = req->file->private_data;
@@ -37,17 +49,13 @@ static int io_msg_ring_data(struct io_kiocb *req)return -EOVERFLOW ; } -static void io_double_unlock_ctx (struct io_ring_ctx *ctx, -struct io_ring_ctx *octx, +static void io_double_unlock_ctx (struct io_ring_ctx *octx,unsigned int issue_flags) { -if (issue_flags &IO_URING_F_UNLOCKED ) -mutex_unlock (&ctx->uring_lock);mutex_unlock (&octx->uring_lock); } -static int io_double_lock_ctx (struct io_ring_ctx *ctx, -struct io_ring_ctx *octx, +static int io_double_lock_ctx (struct io_ring_ctx *octx,unsigned int issue_flags) {/*
@@ -60,17 +68,28 @@ static int io_double_lock_ctx(struct io_ring_ctx *ctx,return -EAGAIN ;return 0 ; } +mutex_lock (&octx->uring_lock); +return 0 ; +} -/* Always grab smallest value ctx first. We know ctx != octx. */ -if (ctx < octx) { -mutex_lock (&ctx->uring_lock); -mutex_lock (&octx->uring_lock); - }else { -mutex_lock (&octx->uring_lock); -mutex_lock (&ctx->uring_lock); +static struct file *io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags) +{ + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +struct io_ring_ctx *ctx = req->ctx; +struct file *file =NULL ; +unsigned long file_ptr; +int idx = msg->src_fd; + +io_ring_submit_lock (ctx, issue_flags); +if (likely (idx < ctx->nr_user_files)) { + idx =array_index_nospec (idx, ctx->nr_user_files); + file_ptr =io_fixed_file_slot (&ctx->file_table, idx)->file_ptr; + file = (struct file *) (file_ptr &FFS_MASK ); +if (file) +get_file (file); } - -return 0 ; +io_ring_submit_unlock (ctx, issue_flags); +return file; }static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags)
@@ -78,34 +97,27 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);struct io_ring_ctx *ctx = req->ctx; -unsigned long file_ptr; -struct file *src_file; +struct file *src_file = msg->src_file;int ret;if (target_ctx == ctx)return -EINVAL ; +if (!src_file) { + src_file =io_msg_grab_file (req, issue_flags); +if (!src_file) +return -EBADF ; + msg->src_file = src_file; + req->flags |=REQ_F_NEED_CLEANUP ; + } - ret =io_double_lock_ctx (ctx, target_ctx, issue_flags); -if (unlikely (ret)) -return ret; - - ret = -EBADF ; -if (unlikely (msg->src_fd >= ctx->nr_user_files)) - goto out_unlock; - - msg->src_fd =array_index_nospec (msg->src_fd, ctx->nr_user_files); - file_ptr =io_fixed_file_slot (&ctx->file_table, msg->src_fd)->file_ptr; -if (!file_ptr) - goto out_unlock; - - src_file = (struct file *) (file_ptr &FFS_MASK ); -get_file (src_file); +if (unlikely (io_double_lock_ctx (target_ctx, issue_flags))) +return -EAGAIN ; ret =__io_fixed_fd_install (target_ctx, src_file, msg->dst_fd); -if (ret <0 ) { -fput (src_file); +if (ret <0 ) goto out_unlock; - } + msg->src_file =NULL ; + req->flags &= ~REQ_F_NEED_CLEANUP ;if (msg->flags &IORING_MSG_RING_CQE_SKIP ) goto out_unlock;
@@ -119,7 +131,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)if (!io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) ret = -EOVERFLOW ; out_unlock: -io_double_unlock_ctx (ctx, target_ctx, issue_flags); +io_double_unlock_ctx (target_ctx, issue_flags);return ret; }
@@ -130,6 +142,7 @@ int io_msg_ring_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)if (unlikely (sqe->buf_index || sqe->personality))return -EINVAL ; + msg->src_file =NULL ; msg->user_data =READ_ONCE (sqe->off); msg->len =READ_ONCE (sqe->len); msg->cmd =READ_ONCE (sqe->addr);
io_uring/msg_ring.h +1 -0
@@ -2,3 +2,4 @@int io_msg_ring_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_msg_ring (struct io_kiocb *req,unsigned int issue_flags); +void io_msg_ring_cleanup (struct io_kiocb *req);
io_uring/opdef.c +1 -0
@@ -445,6 +445,7 @@ const struct io_op_def io_op_defs[] = { .name = "MSG_RING", .prep = io_msg_ring_prep, .issue = io_msg_ring, + .cleanup = io_msg_ring_cleanup, }, [IORING_OP_FSETXATTR] = { .needs_file = 1,
Signature · 1 line
2.38.1
PATCH v2 10/12 io_uring: extract a io_msg_install_complete helper
In reply to Pavel BegunkovExtract a helper called io_msg_install_complete() from io_msg_send_fd(), will be used later.
--- io_uring/msg_ring.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-)
io_uring/msg_ring.c +21 -13
@@ -92,36 +92,25 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flreturn file; } -static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags) +static int io_msg_install_complete (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); -struct io_ring_ctx *ctx = req->ctx;struct file *src_file = msg->src_file;int ret; -if (target_ctx == ctx) -return -EINVAL ; -if (!src_file) { - src_file =io_msg_grab_file (req, issue_flags); -if (!src_file) -return -EBADF ; - msg->src_file = src_file; - req->flags |=REQ_F_NEED_CLEANUP ; - } -if (unlikely (io_double_lock_ctx (target_ctx, issue_flags)))return -EAGAIN ; ret =__io_fixed_fd_install (target_ctx, src_file, msg->dst_fd);if (ret <0 ) goto out_unlock; + msg->src_file =NULL ; req->flags &= ~REQ_F_NEED_CLEANUP ;if (msg->flags &IORING_MSG_RING_CQE_SKIP ) goto out_unlock; -/* * If this fails, the target still received the file descriptor but * wasn't notified of the fact. This means that if this request
@@ -135,6 +124,25 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)return ret; } +static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_ring_ctx *target_ctx = req->file->private_data; + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +struct io_ring_ctx *ctx = req->ctx; +struct file *src_file = msg->src_file; + +if (target_ctx == ctx) +return -EINVAL ; +if (!src_file) { + src_file =io_msg_grab_file (req, issue_flags); +if (!src_file) +return -EBADF ; + msg->src_file = src_file; + req->flags |=REQ_F_NEED_CLEANUP ; + } +return io_msg_install_complete (req, issue_flags); +} +int io_msg_ring_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg);
Signature · 1 line
2.38.1
PATCH v2 11/12 io_uring: do msg_ring in target task via tw
In reply to Pavel BegunkovWhile executing in a context of one io_uring instance msg_ring manipulates another ring. We're trying to keep CQEs posting contained in the context of the ring-owner task, use task_work to send the request to the target ring's task when we're modifying its CQ or trying to install a file. Note, we can't safely use io_uring task_work infra and have to use task_work directly.
--- io_uring/msg_ring.c | 56 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 53 insertions(+), 3 deletions(-)
io_uring/msg_ring.c +53 -3
@@ -16,6 +16,7 @@struct io_msg {struct file *file;struct file *src_file; +struct callback_head tw;u64 user_data;u32 len;u32 cmd;
@@ -35,6 +36,23 @@ void io_msg_ring_cleanup(struct io_kiocb *req) msg->src_file =NULL ; } +static void io_msg_tw_complete (struct callback_head *head) +{ + struct io_msg *msg = container_of(head, struct io_msg, tw); +struct io_kiocb *req =cmd_to_io_kiocb (msg); +struct io_ring_ctx *target_ctx = req->file->private_data; +int ret =0 ; + +if (current->flags &PF_EXITING ) + ret = -EOWNERDEAD ; +else if (!io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 )) + ret = -EOVERFLOW ; + +if (ret <0 ) +req_set_fail (req); +io_req_queue_tw_complete (req, ret); +} +static int io_msg_ring_data (struct io_kiocb *req) {struct io_ring_ctx *target_ctx = req->file->private_data;
@@ -43,6 +61,15 @@ static int io_msg_ring_data(struct io_kiocb *req)if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ; +if (target_ctx->task_complete && current != target_ctx->submitter_task) { +init_task_work (&msg->tw, io_msg_tw_complete); +if (task_work_add (target_ctx->submitter_task, &msg->tw, +TWA_SIGNAL )) +return -EOWNERDEAD ; + +return IOU_ISSUE_SKIP_COMPLETE ; + } +if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len,0 ))return 0 ;
@@ -124,6 +151,19 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flagreturn ret; } +static void io_msg_tw_fd_complete (struct callback_head *head) +{ + struct io_msg *msg = container_of(head, struct io_msg, tw); +struct io_kiocb *req =cmd_to_io_kiocb (msg); +int ret = -EOWNERDEAD ; + +if (!(current->flags &PF_EXITING )) + ret =io_msg_install_complete (req,IO_URING_F_UNLOCKED ); +if (ret <0 ) +req_set_fail (req); +io_req_queue_tw_complete (req, ret); +} +static int io_msg_send_fd (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data;
@@ -140,6 +180,15 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) msg->src_file = src_file; req->flags |=REQ_F_NEED_CLEANUP ; } + +if (target_ctx->task_complete && current != target_ctx->submitter_task) { +init_task_work (&msg->tw, io_msg_tw_fd_complete); +if (task_work_add (target_ctx->submitter_task, &msg->tw, +TWA_SIGNAL )) +return -EOWNERDEAD ; + +return IOU_ISSUE_SKIP_COMPLETE ; + }return io_msg_install_complete (req, issue_flags); }
@@ -185,10 +234,11 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags) } done: -if (ret == -EAGAIN ) -return -EAGAIN ; -if (ret <0 ) +if (ret <0 ) { +if (ret == -EAGAIN || ret ==IOU_ISSUE_SKIP_COMPLETE ) +return ret;req_set_fail (req); + }io_req_set_res (req, ret,0 );return IOU_OK ; }
Signature · 1 line
2.38.1
PATCH v2 12/12 io_uring: skip spinlocking for ->task_complete
In reply to Pavel Begunkov->task_complete was added to serialised CQE posting by doing it from the task context only (or fallback wq when the task is dead), and now we can use that to avoid taking ->completion_lock while filling CQ entries. The patch skips spinlocking only in two spots, __io_submit_flush_completions() and flushing in io_aux_cqe, it's safer and covers all cases we care about. Extra care is taken to force taking the lock while queueing overflow entries.
It fundamentally relies on SINGLE_ISSUER to have only one task posting events. It also need to take into account overflowed CQEs, flushing of which happens in the cq wait path, and so this implementation also needs DEFER_TASKRUN to limit waiters. For the same reason we disable it for SQPOLL, and for IOPOLL as it won't benefit from it in any case. DEFER_TASKRUN, SQPOLL and IOPOLL requirement may be relaxed in the future.
--- io_uring/io_uring.c | 71 +++++++++++++++++++++++++++++++++------------ io_uring/io_uring.h | 12 ++++++-- 2 files changed, 62 insertions(+), 21 deletions(-)
io_uring/io_uring.c +52 -19
@@ -595,13 +595,25 @@ static inline void io_cq_unlock(struct io_ring_ctx *ctx)spin_unlock (&ctx->completion_lock); } +static inline void __io_cq_lock (struct io_ring_ctx *ctx) + __acquires(ctx->completion_lock) +{ +if (!ctx->task_complete) +spin_lock (&ctx->completion_lock); +} + +static inline void __io_cq_unlock (struct io_ring_ctx *ctx) +{ +if (!ctx->task_complete) +spin_unlock (&ctx->completion_lock); +} +/* keep it inlined for io_submit_flush_completions() */ -static inline void io_cq_unlock_post_inline (struct io_ring_ctx *ctx) +static inline void __io_cq_unlock_post (struct io_ring_ctx *ctx) __releases(ctx->completion_lock) {io_commit_cqring (ctx); -spin_unlock (&ctx->completion_lock); - +__io_cq_unlock (ctx);io_commit_cqring_flush (ctx);io_cqring_wake (ctx); }
@@ -609,7 +621,10 @@ static inline void io_cq_unlock_post_inline(struct io_ring_ctx *ctx)void io_cq_unlock_post (struct io_ring_ctx *ctx) __releases(ctx->completion_lock) { -io_cq_unlock_post_inline (ctx); +io_commit_cqring (ctx); +spin_unlock (&ctx->completion_lock); +io_commit_cqring_flush (ctx); +io_cqring_wake (ctx); }/* Returns true if there are no backlogged entries after the flush */
@@ -796,12 +811,13 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow)return &rings->cqes[off]; } -static bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res,u32 cflags, -bool allow_overflow) +static bool io_fill_cqe_aux (struct io_ring_ctx *ctx,u64 user_data,s32 res, +u32 cflags) {struct io_uring_cqe *cqe; -lockdep_assert_held (&ctx->completion_lock); +if (!ctx->task_complete) +lockdep_assert_held (&ctx->completion_lock); ctx->cq_extra++;
@@ -824,10 +840,6 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 }return true; } - -if (allow_overflow) -return io_cqring_event_overflow (ctx, user_data, res, cflags,0 ,0 ); -return false; }
@@ -841,7 +853,17 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx)for (i =0 ; i < state->cqes_count; i++) {struct io_uring_cqe *cqe = &state->cqes[i]; -io_fill_cqe_aux (ctx, cqe->user_data, cqe->res, cqe->flags, true); +if (!io_fill_cqe_aux (ctx, cqe->user_data, cqe->res, cqe->flags)) { +if (ctx->task_complete) { +spin_lock (&ctx->completion_lock); +io_cqring_event_overflow (ctx, cqe->user_data, + cqe->res, cqe->flags,0 ,0 ); +spin_unlock (&ctx->completion_lock); + }else { +io_cqring_event_overflow (ctx, cqe->user_data, + cqe->res, cqe->flags,0 ,0 ); + } + } } state->cqes_count =0 ; }
@@ -852,7 +874,10 @@ static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, ubool filled;io_cq_lock (ctx); - filled =io_fill_cqe_aux (ctx, user_data, res, cflags, allow_overflow); + filled =io_fill_cqe_aux (ctx, user_data, res, cflags); +if (!filled && allow_overflow) + filled =io_cqring_event_overflow (ctx, user_data, res, cflags,0 ,0 ); +io_cq_unlock_post (ctx);return filled; }
@@ -876,10 +901,10 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32lockdep_assert_held (&ctx->uring_lock);if (ctx->submit_state.cqes_count == length) { -io_cq_lock (ctx); +__io_cq_lock (ctx);__io_flush_post_cqes (ctx);/* no need to flush - flush is deferred */ -io_cq_unlock (ctx); +__io_cq_unlock_post (ctx); }/* For defered completions this is not as strict as it is otherwise,
@@ -1414,7 +1439,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)struct io_wq_work_node *node, *prev;struct io_submit_state *state = &ctx->submit_state; -io_cq_lock (ctx); +__io_cq_lock (ctx);/* must come first to preserve CQE ordering in failure cases */ if (state->cqes_count)__io_flush_post_cqes (ctx);
@@ -1422,10 +1447,18 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); -if (!(req->flags &REQ_F_CQE_SKIP )) -io_fill_cqe_req (ctx, req); +if (!(req->flags &REQ_F_CQE_SKIP ) && +unlikely (!__io_fill_cqe_req (ctx, req))) { +if (ctx->task_complete) { +spin_lock (&ctx->completion_lock); +io_req_cqe_overflow (req); +spin_unlock (&ctx->completion_lock); + }else { +io_req_cqe_overflow (req); + } + } } -io_cq_unlock_post_inline (ctx); +__io_cq_unlock_post (ctx);if (!wq_list_empty (&ctx->submit_state.compl_reqs)) {io_free_batch_list (ctx, state->compl_reqs.first);
io_uring/io_uring.h +10 -2
@@ -110,7 +110,7 @@ static inline struct io_uring_cqe *io_get_cqe(struct io_ring_ctx *ctx)return io_get_cqe_overflow (ctx, false); } -static inline bool io_fill_cqe_req (struct io_ring_ctx *ctx, +static inline bool __io_fill_cqe_req (struct io_ring_ctx *ctx,struct io_kiocb *req) {struct io_uring_cqe *cqe;
@@ -122,7 +122,7 @@ static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx,*/ cqe =io_get_cqe (ctx);if (unlikely (!cqe)) -return io_req_cqe_overflow (req); +return false; trace_io_uring_complete(req->ctx, req, req->cqe.user_data, req->cqe.res, req->cqe.flags,
@@ -145,6 +145,14 @@ static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx,return true; } +static inline bool io_fill_cqe_req (struct io_ring_ctx *ctx, +struct io_kiocb *req) +{ +if (likely (__io_fill_cqe_req (ctx, req))) +return true; +return io_req_cqe_overflow (req); +} +static inline void req_set_fail (struct io_kiocb *req) { req->flags |=REQ_F_FAIL ;
Signature · 1 line
2.38.1
Jens Axboe
maintainerRe: [PATCH for-next v2 01/12] io_uring: dont remove file from msg_ring reqs
In reply to Pavel BegunkovQuoted 4 lines · Pavel Begunkov
We should not be messing with
req->fileoutside of core paths. Clearing it makesmsg_ringnon reentrant, i.e. luckilyio_msg_send_fd()fails the request on failedio_double_lock_ctx()but clearly was originally intended to do retries instead.
That's basically what I had in my patch, except I just went for the negated one instead to cut down on churn. Why not just do that?
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH for-next v2 11/12] io_uring: do msg_ring in target task via tw
In reply to Pavel Begunkovio_uring/msg_ring.c static int io_msg_ring_data(struct io_kiocb *req) · line 72
Quoted 13 lines · Pavel Begunkov
io_uring/msg_ring.c +9
-0@@ -43,6 +61,15 @@ static int io_msg_ring_data(struct io_kiocb *req)if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ; +if (target_ctx->task_complete && current != target_ctx->submitter_task) { +init_task_work (&msg->tw, io_msg_tw_complete); +if (task_work_add (target_ctx->submitter_task, &msg->tw, +TWA_SIGNAL )) +return -EOWNERDEAD ; + +return IOU_ISSUE_SKIP_COMPLETE ; + } +
We should probably be able to get by with TWA_SIGNAL_NO_IPI here, no?
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH for-next v2 11/12] io_uring: do msg_ring in target task via tw
Quoted 5 lines · Jens Axboe
Quoted 4 lines · Pavel Begunkov
io_uring/msg_ring.c +0
-0@@ -43,6 +61,15 @@ static int io_msg_ring_data(struct io_kiocb *req)if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ;
Quoted 11 lines
io_uring/msg_ring.c static int io_msg_ring_data(struct io_kiocb *req) · line 72
Quoted 9 lines
io_uring/msg_ring.c +9
-0+if (target_ctx->task_complete && current != target_ctx->submitter_task) { +init_task_work (&msg->tw, io_msg_tw_complete); +if (task_work_add (target_ctx->submitter_task, &msg->tw, +TWA_SIGNAL )) +return -EOWNERDEAD ; + +return IOU_ISSUE_SKIP_COMPLETE ; + } +We should probably be able to get by with
TWA_SIGNAL_NO_IPIhere, no?
Considering we didn't even wake before, I'd say that's a solid yes.
Signature · 1 line
Jens Axboe
Re: [PATCH for-next v2 01/12] io_uring: dont remove file from msg_ring reqs
In reply to Jens AxboeQuoted 8 lines · Jens Axboe
Quoted 4 lines · Pavel Begunkov
We should not be messing with
req->fileoutside of core paths. Clearing it makesmsg_ringnon reentrant, i.e. luckilyio_msg_send_fd()fails the request on failedio_double_lock_ctx()but clearly was originally intended to do retries instead.That's basically what I had in my patch, except I just went for the negated one instead to cut down on churn. Why not just do that?
I just already had this patch so left it as is, but if I have to find a reason it would be: 1) considering that the req->file check is already an exception to the rule, the negative would be an exception to the exception, and 2) it removes that extra req->file check.
Signature · 1 line
Pavel Begunkov
Re: [PATCH for-next v2 11/12] io_uring: do msg_ring in target task via tw
In reply to Jens AxboeQuoted 18 lines · Jens Axboe
Quoted 5 lines · Jens Axboe
Quoted 4 lines · Pavel Begunkov
io_uring/msg_ring.c +0
-0@@ -43,6 +61,15 @@ static int io_msg_ring_data(struct io_kiocb *req)if (msg->src_fd || msg->dst_fd || msg->flags)return -EINVAL ;Quoted 11 lines
io_uring/msg_ring.c static int io_msg_ring_data(struct io_kiocb *req) · line 72
Quoted 9 lines
io_uring/msg_ring.c +9
-0+if (target_ctx->task_complete && current != target_ctx->submitter_task) { +init_task_work (&msg->tw, io_msg_tw_complete); +if (task_work_add (target_ctx->submitter_task, &msg->tw, +TWA_SIGNAL )) +return -EOWNERDEAD ; + +return IOU_ISSUE_SKIP_COMPLETE ; + } +We should probably be able to get by with
TWA_SIGNAL_NO_IPIhere, no?
Quoted 1 line
Considering we didn't even wake before, I'd say that's a solid yes.
I'm not so sure. It'll work, but a naive approach would also lack IORING_SETUP_TASKRUN_FLAG and so mess with latencies when it's not desirable.
option 1)
method = TWA_SIGNAL;
if (flags & IORING_SETUP_TASKRUN_FLAG)
method = NO_IPI;
option 2)
task_work_add(NO_IPI); atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);
Might be better to have one of those.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH for-next v2 11/12] io_uring: do msg_ring in target task via tw
Quoted 30 lines · Pavel Begunkov
Quoted 17 lines · Jens Axboe
Quoted 15 lines · Jens Axboe
io_uring/msg_ring.c static int io_msg_ring_data(struct io_kiocb *req) · line 71
Quoted 12 lines · Pavel Begunkov
io_uring/msg_ring.c +8
-0@@ -43,6 +61,15 @@ static int io_msg_ring_data(struct io_kiocb *req) if (msg->src_fd || msg->dst_fd || msg->flags) return -EINVAL; + if (target_ctx->task_complete && current != target_ctx->submitter_task) { + init_task_work(&msg->tw, io_msg_tw_complete); + if (task_work_add(target_ctx->submitter_task, &msg->tw, + TWA_SIGNAL)) + return -EOWNERDEAD; + + return IOU_ISSUE_SKIP_COMPLETE; + } +We should probably be able to get by with
TWA_SIGNAL_NO_IPIhere, no?Quoted 1 line
Considering we didn't even wake before, I'd say that's a solid yes.
I'm not so sure. It'll work, but a naive approach would also lack
IORING_SETUP_TASKRUN_FLAGand so mess with latencies when it's not desirable.option 1)
method =
TWA_SIGNAL;
if (flags &IORING_SETUP_TASKRUN_FLAG)
method =NO_IPI;
Quoted 7 lines
option 2)
task_work_add(NO_IPI);atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);Might be better to have one of those.
I like option 2, which should be fine.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH for-next v2 01/12] io_uring: dont remove file from msg_ring reqs
In reply to Pavel BegunkovQuoted 7 lines · Pavel Begunkov
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Pavel Begunkov
We should not be messing with
req->fileoutside of core paths. Clearing it makesmsg_ringnon reentrant, i.e. luckilyio_msg_send_fd()fails the request on failedio_double_lock_ctx()but clearly was originally intended to do retries instead.
Quoted 7 lines
Quoted 2 lines
That's basically what I had in my patch, except I just went for the negated one instead to cut down on churn. Why not just do that?
I just already had this patch so left it as is, but if I have to find a reason it would be: 1) considering that the
req->filecheck is already an exception to the rule, the negative would be an exception to the exception, and 2) it removes that extrareq->filecheck.
Let's just leave it as-is, was only mostly concerned with potential backport pain.
Signature · 1 line
Jens Axboe