Skip to content

Allow batching of inline completions

Jens Axboe io-uring 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest5 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5io_uring: provide generic io_req_complete() helperfs/io_uring.c+43 -630 replies
2/5io_uring: add 'io_comp_state' to struct io_submit_statefs/io_uring.c+16 -20 replies
3/5io_uring: pass down completion state on the issue sidefs/io_uring.c+50 -170 replies
4/5io_uring: pass in completion state to appropriate issue side handlersfs/io_uring.c+86 -670 replies
5/5io_uring: enable READ/WRITE to use deferred completionsfs/io_uring.c+23 -150 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCHSET 0/5] Allow batching of inline completions

As Xuan Zhuo <[email protected]> reported here:

https://lore.kernel.org/io-uring/[email protected]/T/#me32d6897f976e8268284ff5cbdb3696010c2b7ba

we can do a bit better when dealing with inline completions from the submission path. This patchset cleans up the standard completion logic, then builds on top of that to allow collecting completions done at submission time. This allows io_uring to amortize the cost of needing to grab the completion lock, and updating the CQ ring as well.

On a silly t/io_uring NOP test on my laptop, this brings about a 20% increase in performance. Xuan Zhuo reports that it changes his SQPOLL based UDP processing (running at 800K PPS) profile from:

17.97% [kernel] [k] copy_user_generic_unrolled
13.92% [kernel] [k] io_commit_cqring
11.04% [kernel] [k] __io_cqring_fill_event
10.33% [kernel] [k] udp_recvmsg
5.94% [kernel] [k] skb_release_data
4.31% [kernel] [k] udp_rmem_release
2.68% [kernel] [k] __check_object_size
2.24% [kernel] [k] __slab_free
2.22% [kernel] [k] _raw_spin_lock_bh
2.21% [kernel] [k] kmem_cache_free
2.13% [kernel] [k] free_pcppages_bulk
1.83% [kernel] [k] io_submit_sqes
1.38% [kernel] [k] page_frag_free
1.31% [kernel] [k] inet_recvmsg

to

19.99% [kernel] [k] copy_user_generic_unrolled
11.63% [kernel] [k] skb_release_data
9.36% [kernel] [k] udp_rmem_release
8.64% [kernel] [k] udp_recvmsg
6.21% [kernel] [k] __slab_free
4.39% [kernel] [k] __check_object_size
3.64% [kernel] [k] free_pcppages_bulk
2.41% [kernel] [k] kmem_cache_free
2.00% [kernel] [k] io_submit_sqes
1.95% [kernel] [k] page_frag_free
1.54% [kernel] [k] io_put_req
[...]
0.07% [kernel] [k] io_commit_cqring
0.44% [kernel] [k] __io_cqring_fill_event

which looks much nicer.

Patches are against my for-5.9/io_uring branch.

Signature · 1 line
Jens Axboe