Skip to content

optimise resheduling due to deferred tw

Pavel Begunkov io-uring lkml 1 version Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first

  1. v2 Latest8 patches 2 repliesApplied

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/8io_uring: move pinning out of io_req_local_work_addio_uring/io_uring.c+4 -70 replies
2/8io_uring: optimie local tw add ctx pinningio_uring/io_uring.c+6 -20 replies
3/8io_uring: refactor __io_cq_unlock_post_flush()io_uring/io_uring.c io_uring/io_uring.h+4 -131 reply
4/8io_uring: add tw add flagsio_uring/io_uring.c io_uring/io_uring.h+11 -50 replies
5/8io_uring: inline llist_add()io_uring/io_uring.c+8 -10 replies
6/8io_uring: reduce scheduling due to twinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/io_uring.h +3 more+61 -250 replies
7/8io_uring: refactor __io_cq_unlock_post_flush()io_uring/io_uring.c+12 -100 replies
8/8io_uring: optimise io_req_local_work_addio_uring/io_uring.c+16 -80 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/8] optimise resheduling due to deferred tw

io_uring extensively uses task_work, but when a task is waiting every new queued task_work batch will try to wake it up and so cause lots of scheduling activity. This series optimises it, specifically applied for rw completions and send-zc notifications for now, and will helpful for further optimisations.

Quick testing shows similar to v1 results, numbers from v1:
For my zc net test once in a while waiting for a portion of buffers I've got 10x descrease in the number of context switches and 2x improvement in CPU util (17% vs 8%). In profiles, io_cqring_work() got down from 40-50% of CPU to ~13%.

There is also an improvement on the softirq side for io_uring notifications as io_req_local_work_add() doesn't trigger wake_up() as often. System wide profiles show reduction of cycles taken by io_req_local_work_add() from 3% -> 0.5%, which is mostly not reflected in the numbers above as it was firing off of a different CPU.

v2: Remove atomics decrements by the queueing side and instead carry
all the info in requests. It's definitely simpler and removes extra atomics, the downside is touching the previous request, which might be not cached.

Add a couple of patches from backlog optimising and cleaning io_req_local_work_add().

Pavel Begunkov (8):
io_uring: move pinning out of io_req_local_work_add
io_uring: optimie local tw add ctx pinning
io_uring: refactor __io_cq_unlock_post_flush()
io_uring: add tw add flags
io_uring: inline llist_add()
io_uring: reduce scheduling due to tw
io_uring: refactor __io_cq_unlock_post_flush()
io_uring: optimise io_req_local_work_add

 include/linux/io_uring_types.h |   3 +-
 io_uring/io_uring.c            | 131 ++++++++++++++++++++++-----------
 io_uring/io_uring.h            |  29 +++++---
 io_uring/notif.c               |   2 +-
 io_uring/notif.h               |   2 +-
 io_uring/rw.c                  |   2 +-
 6 files changed, 110 insertions(+), 59 deletions(-)
Signature · 1 line
2.40.0