Skip to content

CQ waiting / task_work optimisations

Pavel Begunkov io-uring 1 version Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first

  1. v3 Latest11 patches 1 replyApplied

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/11io_uring: move submitter_task out of cold cachelineinclude/linux/io_uring_types.h+5 -50 replies
2/11io_uring: refactor io_wake_functionio_uring/io_uring.c+2 -40 replies
3/11io_uring: don't set TASK_RUNNING in local tw runnerio_uring/io_uring.c+2 -30 replies
4/11io_uring: mark io_run_local_work staticio_uring/io_uring.c io_uring/io_uring.h+1 -20 replies
5/11io_uring: move io_run_local_work_lockedio_uring/io_uring.c io_uring/io_uring.h+17 -180 replies
6/11io_uring: separate wq for ring pollinginclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h+12 -20 replies
7/11io_uring: add lazy poll_wq activationinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h+64 -50 replies
8/11io_uring: wake up optimisationsio_uring/io_uring.c+20 -10 replies
9/11io_uring: waitqueue-less cq waitingio_uring/io_uring.c+12 -70 replies
10/11io_uring: add io_req_local_work_add wake fast pathinclude/linux/io_uring_types.h io_uring/io_uring.c+7 -10 replies
11/11io_uring: optimise deferred tw executionio_uring/io_uring.c+7 -170 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 00/11] CQ waiting / task_work optimisations

For DEFER_TASKRUN rings replace CQ waitqueues with a custom implementation based on the fact that only one task may be waiting for completions. Also, improve deferred task running by removing one atomic in patch 11

Benchmarking QD1 with simulated tw arrival right after we start waiting:
7.5 MIOPS -> 9.3 (+23%), where half of CPU cycles goes to syscall overhead.

v2: remove merged cleanups and add new ones
add 11/11 removing one extra atomic
a small sync adjustment in 10/10
add extra comments

Pavel Begunkov (11):
io_uring: move submitter_task out of cold cacheline
io_uring: refactor io_wake_function
io_uring: don't set TASK_RUNNING in local tw runner
io_uring: mark io_run_local_work static
io_uring: move io_run_local_work_locked
io_uring: separate wq for ring polling
io_uring: add lazy poll_wq activation
io_uring: wake up optimisations
io_uring: waitqueue-less cq waiting
io_uring: add io_req_local_work_add wake fast path
io_uring: optimise deferred tw execution

 include/linux/io_uring_types.h |  15 +--
 io_uring/io_uring.c            | 161 ++++++++++++++++++++++++++-------
 io_uring/io_uring.h            |  28 ++----
 3 files changed, 144 insertions(+), 60 deletions(-)
Signature · 1 line
2.38.1