RFC io_uring/rsrc: reduce node allocation cost on sparse file table installs
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest3 patches 5 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cacheinclude/ io_uring/ io_uring/ +3 more | +26 | 2 replies | |
| 2/3 | io_uring/rsrc: bulk refill the node cache on allocation missio_uring/ io_uring/ | +27 | 0 replies | |
| 3/3 | io_uring/rsrc: prefill the node cache when a file table is registered emptyio_uring/ io_uring/ io_uring/ | +54 | 3 replies |
Reviews trailers per patch
None of the 3 patches has a Reviewed-by or Acked-by.
| Patch | Chengfeng Lin |
|---|---|
| 1/3 | T-b |
| 2/3 | T-b |
| 3/3 | T-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [RFC PATCH 0/3] io_uring/rsrc: reduce node allocation cost on sparse file table installs
Chengfeng Lin reported an 11.6% per-install slowdown in MSG_RING SEND_FD fixed-file installation, bisected to 7029acd8a950 ("io_uring/rsrc: get rid of per-ring io_rsrc_node list"):
https://lore.kernel.org/io-uring/CANGjgdmt0FQ=offsdfn+wEaDxbOFoAa6bi92X_vEo4S6aCZ56A@mail.gmail.com/
That commit is not being questioned here. It removed a real serialisation cost. The side effect is that each fixed-file install now allocates its own io_rsrc_node, and on a first fill of a sparse table every one of those is an allocator miss: io_alloc_cache_init() only allocates the pointer array, io_alloc_cache_put() on the free path is the only thing that populates it, and io_reset_rsrc_node() returns early on a NULL slot so nothing is freed during a first fill. With IO_ALLOC_CACHE_MAX at 128, warming the existing cache cannot cover a 4,096-slot fill.
We spent some time isolating where the per-install cost actually lives. All timing below is Chengfeng's, on a bare-metal i7-12700KF with a pinned core, performance governor, turbo off and fresh boots per point; the evidence tree with reproducers and full logs is linked at the end.
hypothesis result
---------------------------------- ---------------------------------
slab merging defeats locality slab_nomerge: no change
SLAB_ACCOUNT on the new cache +8.5% cost; removed (memcg
accounting the old path never did)
allocator call overhead bulk refill, 32x fewer calls:
+0.007%, path verified by probe fresh slab page creation slab primed, 0 new pages: +0.9%
per-object SLUB allocation path prefill removes it: -9.5%
So the cost is the per-object allocation itself, not calls or pages, and the only way to take it off the install path is to not allocate there.
This series does that in three steps:
0001 dedicated kmem_cache for io_rsrc_node. Neutral on its own; exists
so 0002/0003 can use kmem_cache_alloc_bulk().
0002 bulk refill of the per-ring cache on a miss. Also neutral on the
reported workload, for the reason above; kept for the machinery.
0003 when a sparse table of N slots is registered, grow the per-ring
cache to min(N, 4096) and bulk-fill it.
Measured on the actual series (v6.18-rc4, 6146a0f1dfae):
workload unpatched 0001+2 +0003
---------------------------------- --------- -------- --------
reported 4,096-slot first fill, 119.423 118.151 107.731 -9.8%
ns/install
same-ring remove+refill, 4,096 712.451 710.384 578.365 -18.8%
files, us
one-shot register through fill, 518.387 520.663 526.873 +1.6%
4,096 slots, usregister 4096 / install 64, us 20.361 21.193 84.407 worse
The second row was not predicted: the enlarged cache retains nodes released by FILES_UPDATE, so steady-state churn stops allocating entirely.
The last two rows are the cost, stated plainly. Prefill moves the allocation work to registration rather than removing it, so a program that registers once and fills once sees no total saving, and a program that registers many slots and installs few pays for nodes it never uses (up to 4096 x 32 bytes plus a 32 KiB pointer array, freed at ring teardown).
Whether that trade is acceptable as default behaviour is the question this RFC raises. An alternative would be an opt-in registration flag, so a program that intends to fill the table can say so and nobody else pays. I am happy to rework 0003 into that shape if it is the preferred one.
Testing: builds and boots on io_uring-6.18; the file table, rsrc and msg_ring liburing tests pass on the patched kernel. The full runtests suite was not run to completion on my build VM (the networking tests take its interface down); the targeted set was. Chengfeng ran the alloc_cache code through allocation-failure, limit and cleanup cases under ASan/UBSan/LeakSanitizer, and verified the 8,192-slot case prefills 4,096 and falls through to bulk for the remainder.
Evidence, reproducers and diagnostic patches:
https://github.com/lcf0399/linux-regression-evidence/tree/af2bf8eaae547d940c64ae8dcd1803baf31f2139/io-uring-msg-ring-send-fd-install
Uzair Beg (3):
io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cache io_uring/rsrc: bulk refill the node cache on allocation miss io_uring/rsrc: prefill the node cache when a file table is registered
empty
include/linux/io_uring_types.h | 1 + io_uring/alloc_cache.c | 75 ++++++++++++++++++++++++++++++++-- io_uring/alloc_cache.h | 11 ++++- io_uring/io_uring.c | 5 +++ io_uring/io_uring.h | 1 + io_uring/rsrc.c | 6 +++ 6 files changed, 94 insertions(+), 5 deletions(-)
Signature · 1 line
2.43.0