Skip to content

CQ locking optimisation

Pavel Begunkov io-uring 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest12 patches 7 repliesIn review
  2. v17 patches 9 repliesApplied

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12io_uring: dont remove file from msg_ring reqsio_uring/io_uring.c io_uring/msg_ring.c io_uring/opdef.c +1 more+10 -53 replies
2/12io_uring: improve io_double_lock_ctx fail handlingio_uring/msg_ring.c+2 -00 replies
3/12io_uring: skip overflow CQE posting for dying ringio_uring/io_uring.c+31 -140 replies
4/12io_uring: don't check overflow flush failuresio_uring/io_uring.c+6 -180 replies
5/12io_uring: complete all requests in task contextinclude/linux/io_uring.h include/linux/io_uring_types.h io_uring/io_uring.c+15 -30 replies
6/12io_uring: force multishot CQEs into task contextio_uring/net.c+21 -00 replies
7/12io_uring: use tw for putting rsrcinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/rsrc.c +1 more+20 -20 replies
8/12io_uring: never run tw and fallback in parallelio_uring/io_uring.c+10 -20 replies
9/12io_uring: get rid of double lockingio_uring/msg_ring.c io_uring/msg_ring.h io_uring/opdef.c+51 -360 replies
10/12io_uring: extract a io_msg_install_complete helperio_uring/msg_ring.c+21 -130 replies
11/12io_uring: do msg_ring in target task via twio_uring/msg_ring.c+53 -34 replies
12/12io_uring: skip spinlocking for ->task_completeio_uring/io_uring.c io_uring/io_uring.h+62 -210 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH for-next v2 00/12] CQ locking optimisation

Optimise CQ locking for event posting depending on a number of ring setup flags. QD1 nop benchmark showed 12.067 -> 12.565 MIOPS increase, which more than 8.5% of the io_uring kernel overhead (taking into account that the syscall overhead is over 50%) or 4.12% of the total performance. Naturally, it's not only about QD1, applications can submit a bunch of requests but their completions will may arrive randomly hurting batching and so performance (or latency).

The downside is that we have to punt all io-wq completions to the original task. The performance win should diminish with better completion batching, but it should be worth it for as it also helps tw, which in reality often don't complete too many requests.

The feature depends on DEFER_TASKRUN but can be relaxed to SINGLE_ISSUER

v2: some general msg_ring fixes (patche 1,2)
fix exiting ring potentially modifying CQ in parallel (8/12) use task_work instead of overflowing msg_ring CQEs, which could've
messed with CQE ordering (9-11)

Pavel Begunkov (12):
io_uring: dont remove file from msg_ring reqs
io_uring: improve io_double_lock_ctx fail handling
io_uring: skip overflow CQE posting for dying ring
io_uring: don't check overflow flush failures
io_uring: complete all requests in task context
io_uring: force multishot CQEs into task context
io_uring: use tw for putting rsrc
io_uring: never run tw and fallback in parallel
io_uring: get rid of double locking
io_uring: extract a io_msg_install_complete helper
io_uring: do msg_ring in target task via tw
io_uring: skip spinlocking for ->task_complete

 include/linux/io_uring.h       |   2 +
 include/linux/io_uring_types.h |   3 +
 io_uring/io_uring.c            | 165 ++++++++++++++++++++++-----------
 io_uring/io_uring.h            |  12 ++-
 io_uring/msg_ring.c            | 163 ++++++++++++++++++++++----------
 io_uring/msg_ring.h            |   1 +
 io_uring/net.c                 |  21 +++++
 io_uring/opdef.c               |   8 ++
 io_uring/opdef.h               |   2 +
 io_uring/rsrc.c                |  19 +++-
 io_uring/rsrc.h                |   1 +
 11 files changed, 291 insertions(+), 106 deletions(-)
Signature · 1 line
2.38.1