Skip to content

RFC CQ waiting and wake up optimisations

Pavel Begunkov io-uring 1 version Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 RFC Latest13 patches 8 repliesApplied

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/13io_uring: rearrange defer list checksio_uring/io_uring.c io_uring/io_uring.h+1 -40 replies
2/13io_uring: don't iterate cq wait fast pathio_uring/io_uring.c+8 -100 replies
3/13io_uring: kill io_run_task_work_ctxio_uring/io_uring.c io_uring/io_uring.h+5 -210 replies
4/13io_uring: move defer tw task checksio_uring/io_uring.c io_uring/io_uring.h+11 -90 replies
5/13io_uring: parse check_cq out of wq waitingio_uring/io_uring.c+18 -140 replies
6/13io_uring: mimimise io_cqring_wait_scheduleio_uring/io_uring.c+23 -160 replies
7/13io_uring: simplify io_has_workio_uring/io_uring.c+1 -20 replies
8/13io_uring: set TASK_RUNNING right after scheduleio_uring/io_uring.c+2 -30 replies
9/13io_uring: separate wq for ring pollinginclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h+12 -16 replies
10/13io_uring: add lazy poll_wq activationinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h+45 -40 replies
11/13io_uring: wake up optimisationsio_uring/io_uring.c+20 -10 replies
12/13io_uring: waitqueue-less cq waitingio_uring/io_uring.c+16 -50 replies
13/13io_uring: add io_req_local_work_add wake fast pathinclude/linux/io_uring_types.h io_uring/io_uring.c+6 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [RFC v2 00/13] CQ waiting and wake up optimisations

The series replaces waitqueues for CQ waiting with a custom waiting loop and adds a couple more perf tweak around it. Benchmarking is done for QD1 with simulated tw arrival right after we start waiting, it gets us from 7.5 MIOPS to 9.2, which is +22%, or double the number for the in-kernel io_uring overhead (i.e. without syscall and userspace). That matches profiles, wake_up() _without_ wake_up_state() was taking 12-14% and prepare_to_wait_exclusive() was around 4-6%.

Another 15% reported in the v1 are not there as it got optimised in the meanwhile by 52ea806ad9834 ("io_uring: finish waiting before flushing overflow entries"). So, comparing to a couple of weeks ago the perf of this test case should've jumped more than 30% end-to-end. (Again, spend only half of cycles in io_uring kernel code).

1-8 are preparation patches, they might be taken right away. The rest needs more comments and maybe a little brushing.

Pavel Begunkov (13):
io_uring: rearrange defer list checks
io_uring: don't iterate cq wait fast path
io_uring: kill io_run_task_work_ctx
io_uring: move defer tw task checks
io_uring: parse check_cq out of wq waiting
io_uring: mimimise io_cqring_wait_schedule
io_uring: simplify io_has_work
io_uring: set TASK_RUNNING right after schedule
io_uring: separate wq for ring polling
io_uring: add lazy poll_wq activation
io_uring: wake up optimisations
io_uring: waitqueue-less cq waiting
io_uring: add io_req_local_work_add wake fast path

 include/linux/io_uring_types.h |   4 +
 io_uring/io_uring.c            | 194 +++++++++++++++++++++++----------
 io_uring/io_uring.h            |  35 +++---
 3 files changed, 155 insertions(+), 78 deletions(-)
Signature · 1 line
2.38.1