random for-next patches
cover random for-next patches
1/5 returns back an old lost optimisation
Others are small cleanups
Pavel Begunkov (5):
io_uring: return back links tw run optimisation
io_uring: don't export io_put_task()
io_uring: simplify fallback execution
io_uring: optimise ctx flags layout
io_uring: refactor __io_req_complete_post
include/linux/io_uring_types.h | 6 +++--- io_uring/io_uring.c | 34 ++++++++++++++++++++++------------ io_uring/io_uring.h | 10 ---------- 3 files changed, 25 insertions(+), 25 deletions(-)
Signature · 1 line
2.38.1
PATCH 3/5 io_uring: simplify fallback execution
Lock the ring with uring_lock in io_fallback_req_func(), which should make it a bit safer and easier. With that we also don't need refs pinning as io_ring_exit_work() will wait until uring_lock is freed.
--- io_uring/io_uring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
io_uring/io_uring.c +6 -8
@@ -245,17 +245,15 @@ static __cold void io_fallback_req_func(struct work_struct *work) fallback_work.work);struct llist_node *node =llist_del_all (&ctx->fallback_llist);struct io_kiocb *req, *tmp; -bool locked = false; +bool locked = true; -percpu_ref_get (&ctx->refs); +mutex_lock (&ctx->uring_lock);llist_for_each_entry_safe (req, tmp, node, io_task_work.node) req->io_task_work.func (req, &locked); - -if (locked) { -io_submit_flush_completions (ctx); -mutex_unlock (&ctx->uring_lock); - } -percpu_ref_put (&ctx->refs); +if (WARN_ON_ONCE (!locked)) +return ; +io_submit_flush_completions (ctx); +mutex_unlock (&ctx->uring_lock); }static int io_alloc_hash_table (struct io_hash_table *table,unsigned bits)
Signature · 1 line
2.38.1
PATCH 5/5 io_uring: refactor __io_req_complete_post
In reply to Pavel BegunkovKeep parts of __io_req_complete_post() relying on req->flags together so the value can be cached.
--- io_uring/io_uring.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
io_uring/io_uring.c +2 -2
@@ -974,14 +974,14 @@ static void __io_req_complete_post(struct io_kiocb *req) req->link =NULL ; } } +io_put_kbuf_comp (req); +io_dismantle_req (req);io_req_put_rsrc (req);/* * Selected buffer deallocation in io_clean_op() assumes that * we don't hold ->completion_lock. Clean them here to avoid * deadlocks. */ -io_put_kbuf_comp (req); -io_dismantle_req (req);io_put_task (req->task,1 );wq_list_add_head (&req->comp_list, &ctx->locked_free_list); ctx->locked_free_nr++;
Signature · 1 line
2.38.1
PATCH 2/5 io_uring: don't export io_put_task()
In reply to Pavel Begunkovio_put_task() is only used in uring.c so enclose it there together with __io_put_task().
--- io_uring/io_uring.c | 11 ++++++++++- io_uring/io_uring.h | 10 ---------- 2 files changed, 10 insertions(+), 11 deletions(-)
io_uring/io_uring.c +10 -1
@@ -715,7 +715,7 @@ static void io_cqring_overflow_flush(struct io_ring_ctx *ctx)io_cqring_do_overflow_flush (ctx); } -void __io_put_task (struct task_struct *task,int nr) +static void __io_put_task (struct task_struct *task,int nr) {struct io_uring_task *tctx = task->io_uring;
@@ -725,6 +725,15 @@ void __io_put_task(struct task_struct *task, int nr)put_task_struct_many (task, nr); } +/* must to be called somewhat shortly after putting a request */ +static inline void io_put_task (struct task_struct *task,int nr) +{ +if (likely (task == current)) + task->io_uring->cached_refs += nr; +else +__io_put_task (task, nr); +} +void io_task_refs_refill (struct io_uring_task *tctx) {unsigned int refill = -tctx->cached_refs +IO_TCTX_REFS_CACHE_NR ;
io_uring/io_uring.h +0 -10
@@ -70,7 +70,6 @@ void io_wq_submit_work(struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req); -void __io_put_task (struct task_struct *task,int nr);void io_task_refs_refill (struct io_uring_task *tctx);bool __io_alloc_req_refill (struct io_ring_ctx *ctx);
@@ -319,15 +318,6 @@ static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx)__io_commit_cqring_flush (ctx); } -/* must to be called somewhat shortly after putting a request */ -static inline void io_put_task (struct task_struct *task,int nr) -{ -if (likely (task == current)) - task->io_uring->cached_refs += nr; -else -__io_put_task (task, nr); -} -static inline void io_get_task_refs (int nr) {struct io_uring_task *tctx = current->io_uring;
Signature · 1 line
2.38.1
PATCH 4/5 io_uring: optimise ctx flags layout
In reply to Pavel BegunkovThere may be different cost for reeading just one byte or more, so it's benificial to keep ctx flag bits that we access together in a single byte. That affected code generation of __io_cq_unlock_post_flush() and removed one memory load.
--- include/linux/io_uring_types.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/io_uring_types.h +3 -3
@@ -196,17 +196,17 @@ struct io_ring_ctx {/* const or read-mostly hot data */ struct {unsigned int flags; -unsigned int compat:1 ;unsigned int drain_next:1 ;unsigned int restricted:1 ;unsigned int off_timeout_used:1 ;unsigned int drain_active:1 ; -unsigned int drain_disabled:1 ;unsigned int has_evfd:1 ; -unsigned int syscall_iopoll:1 ;/* all CQEs should be posted only by the submitter task */ unsigned int task_complete:1 ; +unsigned int syscall_iopoll:1 ;unsigned int poll_activated:1 ; +unsigned int drain_disabled:1 ; +unsigned int compat:1 ;enum task_work_notify_mode notify_method;struct io_rings *rings;
Signature · 1 line
2.38.1
PATCH 1/5 io_uring: return back links tw run optimisation
In reply to Pavel Begunkovio_submit_flush_completions() may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.
--- io_uring/io_uring.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
io_uring/io_uring.c +4 -1
@@ -1344,8 +1344,11 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked)if (!llist_empty (&ctx->work_llist)) goto again; -if (*locked) +if (*locked) {io_submit_flush_completions (ctx); +if (!llist_empty (&ctx->work_llist)) + goto again; + }trace_io_uring_local_work_run (ctx, ret, loops);return ret; }
Signature · 1 line
2.38.1
Jens Axboe
maintainerRe: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.
Did you check when it got lost? Would be nice to add a Fixes link?
Signature · 1 line
Jens Axboe
Re: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
Looks like the optimisation was there for normal task_work, then disappeared in f88262e60bb9c ("io_uring: lockless task list"). DEFERRED_TASKRUN came later and this patch handles exclusively deferred tw. I probably need to send a patch for normal tw as well.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
Quoted 9 lines · Pavel Begunkov
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
I'm not either as it isn't fully descriptive, but it is better than not having that reference imho.
Quoted 4 lines
Looks like the optimisation was there for normal
task_work, then disappeared inf88262e60bb9c("io_uring: lockless task list").DEFERRED_TASKRUNcame later and this patch handles exclusively deferred tw. I probably need to send a patch for normal tw as well.
So maybe just use that commit? I can make a note in the message on how it relates.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Pavel Begunkov
1/5 returns back an old lost optimisation
Others are small cleanupsPavel Begunkov (5):
io_uring: return back links tw run optimisation
io_uring: don't exportio_put_task()
io_uring: simplify fallback execution
io_uring: optimise ctx flags layout
io_uring: refactor__io_req_complete_post[...]
Applied, thanks!
[1/5] io_uring: return back links tw run optimisation
commit: b48f4ef033089cf03c28bb09ae054dbfdf11635a
[2/5] io_uring: don't export io_put_task()
commit: 41cc377f69cc1702d989c33eccbacd845d463c72
[3/5] io_uring: simplify fallback execution
commit: ae96a39a7537ab49b9fb497e7c5e860ffc6fde72
[4/5] io_uring: optimise ctx flags layout
commit: 4a26869e3c95ee20d03b178e413a619928a84d26
[5/5] io_uring: refactor __io_req_complete_post
commit: 2c5c148670c650381bce849e164757ab6a2729beBest regards,
Signature · 1 line
Jens Axboe
Re: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
Quoted 6 lines · Pavel Begunkov
Quoted 5 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
I'm not either as it isn't fully descriptive, but it is better than not having that reference imho.
Agree, but it's also not great that it might be tried to be backported. Maybe adding a link would be nicer?
Quoted 7 lines
Quoted 4 lines
Looks like the optimisation was there for normal
task_work, then disappeared inf88262e60bb9c("io_uring: lockless task list").DEFERRED_TASKRUNcame later and this patch handles exclusively deferred tw. I probably need to send a patch for normal tw as well.So maybe just use that commit? I can make a note in the message on how it relates.
Signature · 1 line
Pavel Begunkov
Re: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
In reply to Jens AxboeQuoted 14 lines · Jens Axboe
Quoted 9 lines · Pavel Begunkov
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
I'm not either as it isn't fully descriptive, but it is better than not having that reference imho.
Quoted 7 lines
Quoted 4 lines
Looks like the optimisation was there for normal
task_work, then disappeared inf88262e60bb9c("io_uring: lockless task list").DEFERRED_TASKRUNcame later and this patch handles exclusively deferred tw. I probably need to send a patch for normal tw as well.So maybe just use that commit? I can make a note in the message on how it relates.
Yes please, thanks
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
In reply to Pavel BegunkovQuoted 12 lines · Pavel Begunkov
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Pavel Begunkov
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
Quoted 7 lines
Quoted 2 lines
I'm not either as it isn't fully descriptive, but it is better than not having that reference imho.
Agree, but it's also not great that it might be tried to be backported. Maybe adding a link would be nicer?
Only the auto-select bot would pick it, but I'm guessing it'll fail and that'll be the end of that. Normal stable additions need a cc stable as well, the fixes is not enough to trigger that.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerRe: [PATCH for-next 1/5] io_uring: return back links tw run optimisation
In reply to Pavel BegunkovQuoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 9 lines · Pavel Begunkov
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Pavel Begunkov
io_submit_flush_completions()may queue new requests for tw execution, especially true for linked requests. Recheck the tw list for emptiness after flushing completions.Did you check when it got lost? Would be nice to add a Fixes link?
fwiw, not fan of putting a "Fixes" tag on sth that is not a fix.
I'm not either as it isn't fully descriptive, but it is better than not having that reference imho.
Quoted 4 lines
Looks like the optimisation was there for normal
task_work, then disappeared inf88262e60bb9c("io_uring: lockless task list").DEFERRED_TASKRUNcame later and this patch handles exclusively deferred tw. I probably need to send a patch for normal tw as well.
Quoted 4 lines
Quoted 2 lines
So maybe just use that commit? I can make a note in the message on how it relates.
Yes please, thanks
Done:
Signature · 1 line
Jens Axboe