Skip to content

caching and SQ/CQ optimisations

Pavel Begunkov io-uring 2 versions – Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest15 patches 1 replyApplied
  2. v1 RFC16 patches 6 repliesSuperseded

Changes in v2 from the cover letter

removed static_key, it'll be submitted later after it rolls out well minor description changes

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/15io_uring: improve cqe !tracing hot pathio_uring/io_uring.h+5 -40 replies
2/15io_uring: cqe init hardeningio_uring/io_uring.c+1 -10 replies
3/15io_uring: simplify big_cqe handlinginclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h +1 more+14 -300 replies
4/15io_uring: refactor __io_get_cqe()io_uring/io_uring.c io_uring/io_uring.h+16 -200 replies
5/15io_uring: optimise extra io_get_cqe null checkio_uring/io_uring.c io_uring/io_uring.h+12 -150 replies
6/15io_uring: reorder cqring_flush and wakeupsio_uring/io_uring.c io_uring/rw.c+4 -120 replies
7/15io_uring: merge iopoll and normal completion pathsinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h +1 more+19 -260 replies
8/15io_uring: force inline io_fill_cqe_reqio_uring/io_uring.h+2 -10 replies
9/15io_uring: compact SQ/CQ heads/tailsinclude/linux/io_uring_types.h+2 -20 replies
10/15io_uring: add option to remove SQ indirectioninclude/uapi/linux/io_uring.h io_uring/io_uring.c+37 -200 replies
11/15io_uring: move non aligned field to the endinclude/linux/io_uring_types.h+18 -180 replies
12/15io_uring: banish non-hot data to end of io_ring_ctxinclude/linux/io_uring_types.h+19 -180 replies
13/15io_uring: separate task_work/waiting cache lineinclude/linux/io_uring_types.h+12 -70 replies
14/15io_uring: move multishot cqe cache in ctxinclude/linux/io_uring_types.h io_uring/io_uring.c+5 -40 replies
15/15io_uring: move iopoll ctx fields aroundinclude/linux/io_uring_types.h+11 -140 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/15] caching and SQ/CQ optimisations

Patch 1-5 optimise io_fill_cqe_req

Patch 6-7 combine iopoll and normal completion paths

Patch 8 inlines io_fill_cqe_req.

Patch 9 should improve CPU caching of SQ/CQ pointers

Patch 10 removes conditionally SQ indirection (->sq_array). Assuming we'll make it a default in liburing, Patch 10 optimises it with static_key.

Patch 11-15 shuffle io_ring_ctx fields.

Testing with t/io_uring nops only for now

                QD2     QD4     QD8     QD16    QD32
baseline:       17.3    26.6    36.4    43.7    49.4
Patches 1-15:   17.8    27.4    37.9    45.8    51.2
Patches 1-16:   17.9    28.2    39.3    47.8    54

v2:
removed static_key, it'll be submitted later after it rolls out well minor description changes

Pavel Begunkov (15):
io_uring: improve cqe !tracing hot path
io_uring: cqe init hardening
io_uring: simplify big_cqe handling
io_uring: refactor __io_get_cqe()
io_uring: optimise extra io_get_cqe null check
io_uring: reorder cqring_flush and wakeups
io_uring: merge iopoll and normal completion paths
io_uring: force inline io_fill_cqe_req
io_uring: compact SQ/CQ heads/tails
io_uring: add option to remove SQ indirection
io_uring: move non aligned field to the end
io_uring: banish non-hot data to end of io_ring_ctx
io_uring: separate task_work/waiting cache line
io_uring: move multishot cqe cache in ctx
io_uring: move iopoll ctx fields around

 include/linux/io_uring_types.h | 129 +++++++++++++++++----------------
 include/uapi/linux/io_uring.h  |   5 ++
 io_uring/io_uring.c            | 120 +++++++++++++++---------------
 io_uring/io_uring.h            |  58 +++++++--------
 io_uring/rw.c                  |  24 ++----
 io_uring/uring_cmd.c           |   5 +-
 6 files changed, 163 insertions(+), 178 deletions(-)
Signature · 1 line
2.41.0