Skip to content

rework and optimise submission+completion paths

Pavel Begunkov io-uring 2 versions In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest24 patches 7 repliesIn review
  2. v1 RFC23 patches 0 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/24io_uring: mark having different creds unlikelyfs/io_uring.c+1 -10 replies
2/24io_uring: force_nonspinfs/io_uring.c+6 -60 replies
3/24io_uring: make io_do_iopoll return number of reqsfs/io_uring.c+17 -170 replies
4/24io_uring: use slist for completion batchingfs/io_uring.c+25 -273 replies
5/24io_uring: remove allocation cache arrayfs/io_uring.c+17 -430 replies
6/24io-wq: add io_wq_work_node based stackfs/io-wq.h+50 -70 replies
7/24io_uring: replace list with stack for req cachesfs/io_uring.c+24 -270 replies
8/24io_uring: split iopoll loopfs/io_uring.c+19 -130 replies
9/24io_uring: use single linked list for iopollfs/io-wq.h fs/io_uring.c+30 -260 replies
10/24io_uring: add a helper for batch freefs/io_uring.c+21 -132 replies
11/24io_uring: convert iopoll_completed to store_releasefs/io_uring.c+8 -110 replies
12/24io_uring: optimise batch completionfs/io_uring.c+10 -281 reply
13/24io_uring: inline completion batching helpersfs/io_uring.c+22 -440 replies
14/24io_uring: don't pass tail into io_free_batch_listfs/io_uring.c+4 -80 replies
15/24io_uring: don't pass state to io_submit_state_endfs/io_uring.c+5 -30 replies
16/24io_uring: deduplicate io_queue_sqe() call sitesfs/io_uring.c+9 -110 replies
17/24io_uring: remove drain_active check from hot pathfs/io_uring.c+29 -240 replies
18/24io_uring: split slow path from io_queue_sqefs/io_uring.c+11 -40 replies
19/24io_uring: inline hot path of __io_queue_sqe()fs/io_uring.c+28 -200 replies
20/24io_uring: reshuffle queue_sqe completion handlingfs/io_uring.c+2 -30 replies
21/24io_uring: restructure submit sqes to_submit checksfs/io_uring.c+6 -30 replies
22/24io_uring: kill off ->inflight_entry fieldfs/io_uring.c+3 -60 replies
23/24io_uring: comment why inline complete calls io_clean_op()fs/io_uring.c+1 -00 replies
24/24io_uring: disable draining earlierfs/io_uring.c+4 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/24] rework and optimise submission+completion paths

24 MIOPS vs 31.5, or ~30% win for fio/t/io_uring nops batching=32 Jens mentioned that with his standard test against Optane it gave yet another +3% to throughput.

1-14 are about optimising the completion path:
- replaces lists with single linked lists
- kills 64 * 8B of caches in ctx
- adds some shuffling of iopoll bits
- list splice instead of per-req list_add in one place
- inlines io_req_free_batch() and other helpers

15-22: inlines __io_queue_sqe() so all the submission path up to io_issue_sqe() is inlined + little tweaks

v2: rebase for-5.16/io_uring

multicqe_drain was hanging because it's a bit buggy, i.e. doesn't consider that requests may get punted, but still add 24th patch to avoid it.

Pavel Begunkov (24):
io_uring: mark having different creds unlikely
io_uring: force_nonspin
io_uring: make io_do_iopoll return number of reqs
io_uring: use slist for completion batching
io_uring: remove allocation cache array
io-wq: add io_wq_work_node based stack
io_uring: replace list with stack for req caches
io_uring: split iopoll loop
io_uring: use single linked list for iopoll
io_uring: add a helper for batch free
io_uring: convert iopoll_completed to store_release
io_uring: optimise batch completion
io_uring: inline completion batching helpers
io_uring: don't pass tail into io_free_batch_list
io_uring: don't pass state to io_submit_state_end
io_uring: deduplicate io_queue_sqe() call sites
io_uring: remove drain_active check from hot path
io_uring: split slow path from io_queue_sqe
io_uring: inline hot path of __io_queue_sqe()
io_uring: reshuffle queue_sqe completion handling
io_uring: restructure submit sqes to_submit checks
io_uring: kill off ->inflight_entry field
io_uring: comment why inline complete calls io_clean_op()
io_uring: disable draining earlier

 fs/io-wq.h    |  60 +++++-
 fs/io_uring.c | 508 +++++++++++++++++++++++---------------------------
 2 files changed, 287 insertions(+), 281 deletions(-)
Signature · 1 line
2.33.0