for-next optimisations
cover for-next optimisations
There are two main lines intervened. The first one is pt.2 of ctx field shuffling for better caching. There is a couple of things left on that front.
The second is optimising (assumably) rarely used offset-based timeouts and draining. There is a downside (see 12/12), which will be fixed later. In plans to queue a task_work clearing drain_used (under uring_lock) from io_queue_deferred() once all drainee are gone.
nops(batch=32):
15.9 MIOPS vs 17.3 MIOPS
nullblk (irqmode=2 completion_nsec=0 submit_queues=16), no merges, no stat
1002 KIOPS vs 1050 KIOPSThough the second test is very slow comparing to what I've seen before, so might be not represantative.
Pavel Begunkov (12):
io_uring: keep SQ pointers in a single cacheline
io_uring: move ctx->flags from SQ cacheline
io_uring: shuffle more fields into SQ ctx section
io_uring: refactor io_get_sqe()
io_uring: don't cache number of dropped SQEs
io_uring: optimise completion timeout flushing
io_uring: small io_submit_sqe() optimisation
io_uring: clean up check_overflow flag
io_uring: wait heads renaming
io_uring: move uring_lock location
io_uring: refactor io_req_defer()
io_uring: optimise non-drain path
fs/io_uring.c | 226 +++++++++++++++++++++++++------------------------- 1 file changed, 111 insertions(+), 115 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/12 io_uring: keep SQ pointers in a single cacheline
sq_array and sq_sqes are always used together, however they are in different cachelines, where the borderline is right before cq_overflow_list is rather rarely touched. Move the fields together so it loads only one cacheline.
--- fs/io_uring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/io_uring.c +1 -2
@@ -364,6 +364,7 @@ struct io_ring_ctx {* array. */ u32 *sq_array; +struct io_uring_sqe *sq_sqes;unsigned cached_sq_head;unsigned sq_entries;unsigned sq_thread_idle;
@@ -373,8 +374,6 @@ struct io_ring_ctx {struct list_head defer_list;struct list_head timeout_list;struct list_head cq_overflow_list; - -struct io_uring_sqe *sq_sqes; }____cacheline_aligned_in_smp ;struct {
Signature · 1 line
2.31.1
PATCH 2/12 io_uring: move ctx->flags from SQ cacheline
In reply to Pavel Begunkovctx->flags are heavily used by both, completion and submission sides, so move it out from the ctx fields related to submissions. Instead, place it together with ctx->refs, because it's already cacheline-aligned and so pads lots of space, and both almost never change. Also, in most occasions they are accessed together as refs are taken at submission time and put back during completion.
Do same with ctx->rings, where the pointer itself is never modified apart from ring init/free.
Note: in percpu mode, struct percpu_ref doesn't modify the struct itself but takes indirection with ref->percpu_count_ptr.
--- fs/io_uring.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
fs/io_uring.c +4 -4
@@ -341,17 +341,19 @@ struct io_submit_state { };struct io_ring_ctx { +/* const or read-mostly hot data */ struct {struct percpu_ref refs; - }____cacheline_aligned_in_smp ; -struct { +struct io_rings *rings;unsigned int flags;unsigned int compat:1 ;unsigned int drain_next:1 ;unsigned int eventfd_async:1 ;unsigned int restricted:1 ; + }____cacheline_aligned_in_smp ; +struct {/* * Ring buffer of indices into array of io_uring_sqe, which is * mmapped by the application using the IORING_OFF_SQES offset.
@@ -386,8 +388,6 @@ struct io_ring_ctx {struct list_head locked_free_list;unsigned int locked_free_nr; -struct io_rings *rings; -const struct cred *sq_creds;/* cred used for __io_sq_thread() */ struct io_sq_data *sq_data;/* if using sq thread polling */
Signature · 1 line
2.31.1
PATCH 3/12 io_uring: shuffle more fields into SQ ctx section
In reply to Pavel BegunkovSince moving locked_free_* out of struct io_submit_state ctx->submit_state is accessed on submission side only, so move it into the submission section. Same goes for rsrc table pointers/nodes/etc., they must be taken and checked during submission because sync'ed by uring_lock, so move them there as well.
--- fs/io_uring.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
fs/io_uring.c +17 -18
@@ -353,6 +353,7 @@ struct io_ring_ctx { unsigned int restricted: 1; }____cacheline_aligned_in_smp ; +/* submission data */ struct {/* * Ring buffer of indices into array of io_uring_sqe, which is
@@ -369,13 +370,27 @@ struct io_ring_ctx {struct io_uring_sqe *sq_sqes;unsigned cached_sq_head;unsigned sq_entries; -unsigned sq_thread_idle;unsigned cached_sq_dropped;unsigned long sq_check_overflow; -struct list_head defer_list; + +/* +* Fixed resources fast path, should be accessed only under +* uring_lock, and updated through io_uring_register(2) +*/ +struct io_rsrc_node *rsrc_node; +struct io_file_table file_table; +unsigned nr_user_files; +unsigned nr_user_bufs; +struct io_mapped_ubuf **user_bufs; + +struct io_submit_state submit_state;struct list_head timeout_list;struct list_head cq_overflow_list; +struct xarray io_buffers; +struct xarray personalities; +u32 pers_next; +unsigned sq_thread_idle; }____cacheline_aligned_in_smp ;struct {
@@ -383,7 +398,6 @@ struct io_ring_ctx {wait_queue_head_t wait; }____cacheline_aligned_in_smp ; -struct io_submit_state submit_state;/* IRQ completion list, under ->completion_lock */ struct list_head locked_free_list;unsigned int locked_free_nr;
@@ -394,21 +408,6 @@ struct io_ring_ctx {struct wait_queue_head sqo_sq_wait;struct list_head sqd_list; -/* -* Fixed resources fast path, should be accessed only under uring_lock, -* and updated through io_uring_register(2) -*/ -struct io_rsrc_node *rsrc_node; - -struct io_file_table file_table; -unsigned nr_user_files; -unsigned nr_user_bufs; -struct io_mapped_ubuf **user_bufs; - -struct xarray io_buffers; -struct xarray personalities; -u32 pers_next; -struct {unsigned cached_cq_tail;unsigned cq_entries;
Signature · 1 line
2.31.1
PATCH 4/12 io_uring: refactor io_get_sqe()
In reply to Pavel BegunkovThe line of io_get_sqe() evaluating @head consists of too many operations including READ_ONCE(), it's not convenient for probing. Refactor it also improving readability.
--- fs/io_uring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/io_uring.c +2 -2
@@ -6685,8 +6685,8 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)*/ static const struct io_uring_sqe *io_get_sqe (struct io_ring_ctx *ctx) { -u32 *sq_array = ctx->sq_array;unsigned head, mask = ctx->sq_entries -1 ; +unsigned sq_idx = ctx->cached_sq_head++ & mask;/* * The cached sq head (or cq tail) serves two purposes:
@@ -6696,7 +6696,7 @@ static const struct io_uring_sqe *io_get_sqe(struct io_ring_ctx *ctx)* 2) allows the kernel side to track the head on its own, even * though the application is the one updating it. */ - head =READ_ONCE (sq_array[ctx->cached_sq_head++ & mask]); + head =READ_ONCE (ctx->sq_array[sq_idx]);if (likely (head < ctx->sq_entries))return &ctx->sq_sqes[head];
Signature · 1 line
2.31.1
PATCH 5/12 io_uring: don't cache number of dropped SQEs
In reply to Pavel BegunkovKill ->cached_sq_dropped and wire DRAIN sequence number correction via ->cq_extra, which is there exactly for that purpose. User visible dropped counter will be populated by incrementing it instead of keeping a copy, similarly as it was done not so long ago with cq_overflow.
--- fs/io_uring.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
fs/io_uring.c +5 -7
@@ -370,7 +370,6 @@ struct io_ring_ctx {struct io_uring_sqe *sq_sqes;unsigned cached_sq_head;unsigned sq_entries; -unsigned cached_sq_dropped;unsigned long sq_check_overflow;struct list_head defer_list;
@@ -5994,13 +5993,11 @@ static u32 io_get_sequence(struct io_kiocb *req) {struct io_kiocb *pos;struct io_ring_ctx *ctx = req->ctx; -u32 total_submitted, nr_reqs =0 ; +u32 nr_reqs =0 ;io_for_each_link (pos, req) nr_reqs++; - - total_submitted = ctx->cached_sq_head - ctx->cached_sq_dropped; -return total_submitted - nr_reqs; +return ctx->cached_sq_head - nr_reqs; }static int io_req_defer (struct io_kiocb *req)
@@ -6701,8 +6698,9 @@ static const struct io_uring_sqe *io_get_sqe(struct io_ring_ctx *ctx)return &ctx->sq_sqes[head];/* drop invalid entries */ - ctx->cached_sq_dropped++; -WRITE_ONCE (ctx->rings->sq_dropped, ctx->cached_sq_dropped); + ctx->cq_extra--; +WRITE_ONCE (ctx->rings->sq_dropped, +READ_ONCE (ctx->rings->sq_dropped) +1 );return NULL ; }
Signature · 1 line
2.31.1
PATCH 6/12 io_uring: optimise completion timeout flushing
In reply to Pavel Begunkovio_commit_cqring() might be very hot and we definitely don't want to touch ->timeout_list there, because 1) it's shared with the submission side so might lead to cache bouncing and 2) may need to load an extra cache line, especially for IRQ completions.
We're interested in it at the completion side only when there are offset-mode timeouts, which are not so popular. Replace list_empty(->timeout_list) hot path check with a new one-way flag, which is set when we prepare the first offset-mode timeout.
note: the flag sits in the same line as briefly used after ->rings
--- fs/io_uring.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
fs/io_uring.c +6 -4
@@ -351,6 +351,7 @@ struct io_ring_ctx { unsigned int drain_next: 1; unsigned int eventfd_async: 1; unsigned int restricted: 1; + unsigned int off_timeout_used: 1; }____cacheline_aligned_in_smp ;/* submission data */
@@ -1318,12 +1319,12 @@ static void io_flush_timeouts(struct io_ring_ctx *ctx) {u32 seq; -if (list_empty (&ctx->timeout_list)) +if (likely (!ctx->off_timeout_used))return ; seq = ctx->cached_cq_tail -atomic_read (&ctx->cq_timeouts); - do { +while (!list_empty (&ctx->timeout_list)) {u32 events_needed, events_got;struct io_kiocb *req =list_first_entry (&ctx->timeout_list, struct io_kiocb, timeout.list);
@@ -1345,8 +1346,7 @@ static void io_flush_timeouts(struct io_ring_ctx *ctx)list_del_init (&req->timeout.list);io_kill_timeout (req,0 ); - }while (!list_empty (&ctx->timeout_list)); - + } ctx->cq_last_tm_flush = seq; }
@@ -5651,6 +5651,8 @@ static int io_timeout_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe,return -EINVAL ; req->timeout.off = off; +if (unlikely (off && !req->ctx->off_timeout_used)) + req->ctx->off_timeout_used = true;if (!req->async_data &&io_alloc_async_data (req))return -ENOMEM ;
Signature · 1 line
2.31.1
PATCH 7/12 io_uring: small io_submit_sqe() optimisation
In reply to Pavel Begunkovsubmit_state.link is used only to assemble a link and not used for actual submission, so clear it before io_queue_sqe() in io_submit_sqe(), awhile it's hot and in caches and queueing doesn't spoil it. May also potentially help compiler with spilling or to do other optimisations.
--- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +1 -1
@@ -6616,8 +6616,8 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,/* last request of a link, enqueue the link */ if (!(req->flags & (REQ_F_LINK |REQ_F_HARDLINK ))) { -io_queue_sqe (head); link->head =NULL ; +io_queue_sqe (head); } }else {if (unlikely (ctx->drain_next)) {
Signature · 1 line
2.31.1
PATCH 8/12 io_uring: clean up check_overflow flag
In reply to Pavel BegunkovThere are no users of ->sq_check_overflow, only ->cq_check_overflow is used. Combine it and move out of completion related part of struct io_ring_ctx.
A not so obvious benefit of it is fitting all completion side fields into a single cacheline. It was taking 2 lines before with 56B padding, and io_cqring_ev_posted*() were still touching both of them.
--- fs/io_uring.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)
fs/io_uring.c +9 -11
@@ -371,7 +371,6 @@ struct io_ring_ctx {struct io_uring_sqe *sq_sqes;unsigned cached_sq_head;unsigned sq_entries; -unsigned long sq_check_overflow;struct list_head defer_list;/*
@@ -408,13 +407,14 @@ struct io_ring_ctx {struct wait_queue_head sqo_sq_wait;struct list_head sqd_list; +unsigned long check_cq_overflow; +struct {unsigned cached_cq_tail;unsigned cq_entries;atomic_t cq_timeouts;unsigned cq_last_tm_flush;unsigned cq_extra; -unsigned long cq_check_overflow;struct wait_queue_head cq_wait;struct fasync_struct *cq_fasync;struct eventfd_ctx *cq_ev_fd;
@@ -1464,8 +1464,7 @@ static bool __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool force) all_flushed =list_empty (&ctx->cq_overflow_list);if (all_flushed) { -clear_bit (0 , &ctx->sq_check_overflow); -clear_bit (0 , &ctx->cq_check_overflow); +clear_bit (0 , &ctx->check_cq_overflow); ctx->rings->sq_flags &= ~IORING_SQ_CQ_OVERFLOW ; }
@@ -1481,7 +1480,7 @@ static bool io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool force) {bool ret = true; -if (test_bit (0 , &ctx->cq_check_overflow)) { +if (test_bit (0 , &ctx->check_cq_overflow)) {/* iopoll syncs against uring_lock, not completion_lock */ if (ctx->flags &IORING_SETUP_IOPOLL )mutex_lock (&ctx->uring_lock);
@@ -1544,8 +1543,7 @@ static bool io_cqring_event_overflow(struct io_ring_ctx *ctx, u64 user_data,return false; }if (list_empty (&ctx->cq_overflow_list)) { -set_bit (0 , &ctx->sq_check_overflow); -set_bit (0 , &ctx->cq_check_overflow); +set_bit (0 , &ctx->check_cq_overflow); ctx->rings->sq_flags |=IORING_SQ_CQ_OVERFLOW ; } ocqe->cqe.user_data = user_data;
@@ -2391,7 +2389,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)* If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ -if (test_bit (0 , &ctx->cq_check_overflow)) +if (test_bit (0 , &ctx->check_cq_overflow))__io_cqring_overflow_flush (ctx, false);if (io_cqring_events (ctx)) goto out;
@@ -6965,7 +6963,7 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode,* Cannot safely flush overflowed CQEs from here, ensure we wake up * the task, and the next invocation will do it. */ -if (io_should_wake (iowq) ||test_bit (0 , &iowq->ctx->cq_check_overflow)) +if (io_should_wake (iowq) ||test_bit (0 , &iowq->ctx->check_cq_overflow))return autoremove_wake_function (curr, mode, wake_flags, key);return -1 ; }
@@ -6993,7 +6991,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,if (ret ||io_should_wake (iowq))return ret;/* let the caller flush overflows, retry */ -if (test_bit (0 , &ctx->cq_check_overflow)) +if (test_bit (0 , &ctx->check_cq_overflow))return 1 ; *timeout =schedule_timeout (*timeout);
@@ -8702,7 +8700,7 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait)* Users may get EPOLLIN meanwhile seeing nothing in cqring, this * pushs them to do the flush. */ -if (io_cqring_events (ctx) ||test_bit (0 , &ctx->cq_check_overflow)) +if (io_cqring_events (ctx) ||test_bit (0 , &ctx->check_cq_overflow)) mask |=EPOLLIN |EPOLLRDNORM ;return mask;
Signature · 1 line
2.31.1
PATCH 9/12 io_uring: wait heads renaming
In reply to Pavel BegunkovWe use several wait_queue_head's for different purposes, but namings are confusing. First rename ctx->cq_wait into ctx->poll_wait, because this one is used for polling an io_uring instance. Then rename ctx->wait into ctx->cq_wait, which is responsible for CQE waiting.
--- fs/io_uring.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
fs/io_uring.c +15 -15
@@ -394,7 +394,7 @@ struct io_ring_ctx {struct {struct mutex uring_lock; -wait_queue_head_t wait; +wait_queue_head_t cq_wait; }____cacheline_aligned_in_smp ;/* IRQ completion list, under ->completion_lock */
@@ -415,7 +415,7 @@ struct io_ring_ctx {atomic_t cq_timeouts;unsigned cq_last_tm_flush;unsigned cq_extra; -struct wait_queue_head cq_wait; +struct wait_queue_head poll_wait;struct fasync_struct *cq_fasync;struct eventfd_ctx *cq_ev_fd; }____cacheline_aligned_in_smp ;
@@ -1178,13 +1178,13 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) ctx->flags = p->flags;init_waitqueue_head (&ctx->sqo_sq_wait);INIT_LIST_HEAD (&ctx->sqd_list); -init_waitqueue_head (&ctx->cq_wait); +init_waitqueue_head (&ctx->poll_wait);INIT_LIST_HEAD (&ctx->cq_overflow_list);init_completion (&ctx->ref_comp);xa_init_flags (&ctx->io_buffers,XA_FLAGS_ALLOC1 );xa_init_flags (&ctx->personalities,XA_FLAGS_ALLOC1 );mutex_init (&ctx->uring_lock); -init_waitqueue_head (&ctx->wait); +init_waitqueue_head (&ctx->cq_wait);spin_lock_init (&ctx->completion_lock);INIT_LIST_HEAD (&ctx->iopoll_list);INIT_LIST_HEAD (&ctx->defer_list);
@@ -1404,14 +1404,14 @@ static void io_cqring_ev_posted(struct io_ring_ctx *ctx)/* see waitqueue_active() comment */ smp_mb (); -if (waitqueue_active (&ctx->wait)) -wake_up (&ctx->wait); +if (waitqueue_active (&ctx->cq_wait)) +wake_up (&ctx->cq_wait);if (ctx->sq_data &&waitqueue_active (&ctx->sq_data->wait))wake_up (&ctx->sq_data->wait);if (io_should_trigger_evfd (ctx))eventfd_signal (ctx->cq_ev_fd,1 ); -if (waitqueue_active (&ctx->cq_wait)) { -wake_up_interruptible (&ctx->cq_wait); +if (waitqueue_active (&ctx->poll_wait)) { +wake_up_interruptible (&ctx->poll_wait);kill_fasync (&ctx->cq_fasync,SIGIO ,POLL_IN ); } }
@@ -1422,13 +1422,13 @@ static void io_cqring_ev_posted_iopoll(struct io_ring_ctx *ctx)smp_mb ();if (ctx->flags &IORING_SETUP_SQPOLL ) { -if (waitqueue_active (&ctx->wait)) -wake_up (&ctx->wait); +if (waitqueue_active (&ctx->cq_wait)) +wake_up (&ctx->cq_wait); }if (io_should_trigger_evfd (ctx))eventfd_signal (ctx->cq_ev_fd,1 ); -if (waitqueue_active (&ctx->cq_wait)) { -wake_up_interruptible (&ctx->cq_wait); +if (waitqueue_active (&ctx->poll_wait)) { +wake_up_interruptible (&ctx->poll_wait);kill_fasync (&ctx->cq_fasync,SIGIO ,POLL_IN ); } }
@@ -7056,10 +7056,10 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = -EBUSY ;break ; } -prepare_to_wait_exclusive (&ctx->wait, &iowq.wq, +prepare_to_wait_exclusive (&ctx->cq_wait, &iowq.wq,TASK_INTERRUPTIBLE ); ret =io_cqring_wait_schedule (ctx, &iowq, &timeout); -finish_wait (&ctx->wait, &iowq.wq); +finish_wait (&ctx->cq_wait, &iowq.wq);cond_resched (); }while (ret >0 );
@@ -8678,7 +8678,7 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait)struct io_ring_ctx *ctx = file->private_data;__poll_t mask =0 ; -poll_wait (file, &ctx->cq_wait, wait); +poll_wait (file, &ctx->poll_wait, wait);/* * synchronizes with barrier from wq_has_sleeper call in * io_commit_cqring
Signature · 1 line
2.31.1
PATCH 10/12 io_uring: move uring_lock location
In reply to Pavel Begunkov->uring_lock is prevalently used for submission, even though it protects many other things like iopoll, registeration, selected bufs, and more. And it's placed together with ->cq_wait poked on completion and CQ waiting sides. Move them apart, ->uring_lock goes to the submission data, and cq_wait to completion related chunk. The last one requires some reshuffling so everything needed by io_cqring_ev_posted*() is in one cacheline.
--- fs/io_uring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
fs/io_uring.c +7 -9
@@ -356,6 +356,8 @@ struct io_ring_ctx {/* submission data */ struct { +struct mutex uring_lock; +/* * Ring buffer of indices into array of io_uring_sqe, which is * mmapped by the application using the IORING_OFF_SQES offset.
@@ -392,11 +394,6 @@ struct io_ring_ctx {unsigned sq_thread_idle; }____cacheline_aligned_in_smp ; -struct { -struct mutex uring_lock; -wait_queue_head_t cq_wait; - }____cacheline_aligned_in_smp ; -/* IRQ completion list, under ->completion_lock */ struct list_head locked_free_list;unsigned int locked_free_nr;
@@ -412,12 +409,13 @@ struct io_ring_ctx {struct {unsigned cached_cq_tail;unsigned cq_entries; -atomic_t cq_timeouts; -unsigned cq_last_tm_flush; -unsigned cq_extra; +struct eventfd_ctx *cq_ev_fd;struct wait_queue_head poll_wait; +struct wait_queue_head cq_wait; +unsigned cq_extra; +atomic_t cq_timeouts;struct fasync_struct *cq_fasync; -struct eventfd_ctx *cq_ev_fd; +unsigned cq_last_tm_flush; }____cacheline_aligned_in_smp ;struct {
Signature · 1 line
2.31.1
PATCH 11/12 io_uring: refactor io_req_defer()
In reply to Pavel BegunkovRename io_req_defer() into io_drain_req() and refactor it uncoupling it from io_queue_sqe() error handling and preparing for coming optimisations. Also, prioritise non IOSQE_ASYNC path.
--- fs/io_uring.c | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-)
fs/io_uring.c +19 -20
@@ -5998,7 +5998,7 @@ static u32 io_get_sequence(struct io_kiocb *req)return ctx->cached_sq_head - nr_reqs; } -static int io_req_defer (struct io_kiocb *req) +static bool io_drain_req (struct io_kiocb *req) {struct io_ring_ctx *ctx = req->ctx;struct io_defer_entry *de;
@@ -6008,27 +6008,29 @@ static int io_req_defer(struct io_kiocb *req)/* Still need defer if there is pending req in defer list. */ if (likely (list_empty_careful (&ctx->defer_list) && !(req->flags &REQ_F_IO_DRAIN ))) -return 0 ; +return false; seq =io_get_sequence (req);/* Still a chance to pass the sequence check */ if (!req_need_defer (req, seq) &&list_empty_careful (&ctx->defer_list)) -return 0 ; +return false; ret =io_req_prep_async (req);if (ret)return ret;io_prep_async_link (req); de =kmalloc (sizeof (*de),GFP_KERNEL ); -if (!de) -return -ENOMEM ; +if (!de) { +io_req_complete_failed (req, ret); +return true; + }spin_lock_irq (&ctx->completion_lock);if (!req_need_defer (req, seq) &&list_empty (&ctx->defer_list)) {spin_unlock_irq (&ctx->completion_lock);kfree (de);io_queue_async_work (req); -return -EIOCBQUEUED ; +return true; }trace_io_uring_defer (ctx, req, req->user_data);
@@ -6036,7 +6038,7 @@ static int io_req_defer(struct io_kiocb *req) de->seq = seq;list_add_tail (&de->list, &ctx->defer_list);spin_unlock_irq (&ctx->completion_lock); -return -EIOCBQUEUED ; +return true; }static void io_clean_op (struct io_kiocb *req)
@@ -6447,21 +6449,18 @@ static void __io_queue_sqe(struct io_kiocb *req)static void io_queue_sqe (struct io_kiocb *req) { -int ret; +if (io_drain_req (req)) +return ; - ret =io_req_defer (req); -if (ret) { -if (ret != -EIOCBQUEUED ) { -fail_req: -io_req_complete_failed (req, ret); - } - }else if (req->flags &REQ_F_FORCE_ASYNC ) { - ret =io_req_prep_async (req); -if (unlikely (ret)) - goto fail_req; -io_queue_async_work (req); - }else { +if (likely (!(req->flags &REQ_F_FORCE_ASYNC ))) {__io_queue_sqe (req); + }else { +int ret =io_req_prep_async (req); + +if (unlikely (ret)) +io_req_complete_failed (req, ret); +else +io_queue_async_work (req); } }
Signature · 1 line
2.31.1
PATCH 12/12 io_uring: optimise non-drain path
In reply to Pavel BegunkovReplace drain checks with one-way flag set upon seeing the first IOSQE_IO_DRAIN request. There are several places where it cuts cycles well:
1) It's much faster than the fast check with two conditions in io_drain_req() including pretty complex list_empty_careful().
2) We can mark io_queue_sqe() inline now, that's a huge win.
3) It replaces timeout and drain checks in io_commit_cqring() with a single flags test. Also great not touching ->defer_list there without a reason so limiting cache bouncing.
It adds a small amount of overhead to drain path, but it's negligible. The main nuisance is that once it meets any DRAIN request in io_uring instance lifetime it will _always_ go through a slower path, so drain-less and offset-mode timeout less applications are preferable. The overhead in that case would be not big, but it's worth to bear in mind.
--- fs/io_uring.c | 57 +++++++++++++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 27 deletions(-)
fs/io_uring.c +30 -27
@@ -352,6 +352,7 @@ struct io_ring_ctx { unsigned int eventfd_async: 1; unsigned int restricted: 1; unsigned int off_timeout_used: 1; + unsigned int drain_used: 1; }____cacheline_aligned_in_smp ;/* submission data */
@@ -1299,9 +1300,9 @@ static void io_kill_timeout(struct io_kiocb *req, int status) } } -static void __io_queue_deferred (struct io_ring_ctx *ctx) +static void io_queue_deferred (struct io_ring_ctx *ctx) { - do { +while (!list_empty (&ctx->defer_list)) {struct io_defer_entry *de =list_first_entry (&ctx->defer_list, struct io_defer_entry, list);
@@ -1310,17 +1311,12 @@ static void __io_queue_deferred(struct io_ring_ctx *ctx)list_del_init (&de->list);io_req_task_queue (de->req);kfree (de); - }while (!list_empty (&ctx->defer_list)); + } }static void io_flush_timeouts (struct io_ring_ctx *ctx) { -u32 seq; - -if (likely (!ctx->off_timeout_used)) -return ; - - seq = ctx->cached_cq_tail -atomic_read (&ctx->cq_timeouts); +u32 seq = ctx->cached_cq_tail -atomic_read (&ctx->cq_timeouts);while (!list_empty (&ctx->timeout_list)) {u32 events_needed, events_got;
@@ -1350,13 +1346,14 @@ static void io_flush_timeouts(struct io_ring_ctx *ctx)static void io_commit_cqring (struct io_ring_ctx *ctx) { -io_flush_timeouts (ctx); - +if (unlikely (ctx->off_timeout_used || ctx->drain_used)) { +if (ctx->off_timeout_used) +io_flush_timeouts (ctx); +if (ctx->drain_used) +io_queue_deferred (ctx); + }/* order cqe stores with ring update */ smp_store_release (&ctx->rings->cq.tail, ctx->cached_cq_tail); - -if (unlikely (!list_empty (&ctx->defer_list))) -__io_queue_deferred (ctx); }static inline bool io_sqring_full (struct io_ring_ctx *ctx)
@@ -6447,9 +6444,9 @@ static void __io_queue_sqe(struct io_kiocb *req)io_queue_linked_timeout (linked_timeout); } -static void io_queue_sqe (struct io_kiocb *req) +static inline void io_queue_sqe (struct io_kiocb *req) { -if (io_drain_req (req)) +if (unlikely (req->ctx->drain_used) &&io_drain_req (req))return ;if (likely (!(req->flags &REQ_F_FORCE_ASYNC ))) {
@@ -6573,6 +6570,23 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,io_req_complete_failed (req, ret);return ret; } + +if (unlikely (req->flags &REQ_F_IO_DRAIN )) { + ctx->drain_used = true; + +/* +* Taking sequential execution of a link, draining both sides +* of the link also fullfils IOSQE_IO_DRAIN semantics for all +* requests in the link. So, it drains the head and the +* next after the link request. The last one is done via +* drain_next flag to persist the effect across calls. +*/ +if (link->head) { + link->head->flags |=REQ_F_IO_DRAIN ; + ctx->drain_next =1 ; + } + } + ret =io_req_prep (req, sqe);if (unlikely (ret)) goto fail_req;
@@ -6591,17 +6605,6 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,if (link->head) {struct io_kiocb *head = link->head; -/* -* Taking sequential execution of a link, draining both sides -* of the link also fullfils IOSQE_IO_DRAIN semantics for all -* requests in the link. So, it drains the head and the -* next after the link request. The last one is done via -* drain_next flag to persist the effect across calls. -*/ -if (req->flags &REQ_F_IO_DRAIN ) { - head->flags |=REQ_F_IO_DRAIN ; - ctx->drain_next =1 ; - } ret =io_req_prep_async (req);if (unlikely (ret)) goto fail_req;
Signature · 1 line
2.31.1
Quoted 8 lines · Pavel Begunkov
There are two main lines intervened. The first one is pt.2 of ctx field shuffling for better caching. There is a couple of things left on that front.
The second is optimising (assumably) rarely used offset-based timeouts and draining. There is a downside (see 12/12), which will be fixed later. In plans to queue a
task_workclearingdrain_used(underuring_lock) fromio_queue_deferred()once all drainee are gone.
There is a much better way to disable it back: in io_drain_req(). The series is good to go, I'll patch on top later.
Quoted 25 lines
nops(batch=32):
15.9 MIOPS vs 17.3 MIOPS nullblk (irqmode=2 completion_nsec=0 submit_queues=16), no merges, no stat 1002 KIOPS vs 1050 KIOPSThough the second test is very slow comparing to what I've seen before, so might be not represantative.
Pavel Begunkov (12):
io_uring: keep SQ pointers in a single cacheline
io_uring: movectx->flagsfrom SQ cacheline
io_uring: shuffle more fields into SQ ctx section
io_uring: refactorio_get_sqe()
io_uring: don't cache number of dropped SQEs
io_uring: optimise completion timeout flushing
io_uring: smallio_submit_sqe()optimisation
io_uring: clean upcheck_overflowflag
io_uring: wait heads renaming
io_uring: moveuring_locklocation
io_uring: refactorio_req_defer()
io_uring: optimise non-drain pathfs/io_uring.c | 226 +++++++++++++++++++++++++------------------------- 1 file changed, 111 insertions(+), 115 deletions(-)
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 9 lines · Pavel Begunkov
There are two main lines intervened. The first one is pt.2 of ctx field shuffling for better caching. There is a couple of things left on that front.
The second is optimising (assumably) rarely used offset-based timeouts and draining. There is a downside (see 12/12), which will be fixed later. In plans to queue a
task_workclearingdrain_used(underuring_lock) fromio_queue_deferred()once all drainee are gone.
Quoted 7 lines
nops(batch=32):
15.9 MIOPS vs 17.3 MIOPS nullblk (irqmode=2 completion_nsec=0 submit_queues=16), no merges, no stat 1002 KIOPS vs 1050 KIOPSThough the second test is very slow comparing to what I've seen before, so might be not represantative.
Applied, thanks. I'll run this through my testing, too.
Signature · 1 line
Jens Axboe