Skip to content

RFC io_uring/rsrc: reduce node allocation cost on sparse file table installs

Uzair Beg io-uring lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest3 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cacheinclude/linux/io_uring_types.h io_uring/alloc_cache.c io_uring/alloc_cache.h +3 more+26 -52 replies
2/3io_uring/rsrc: bulk refill the node cache on allocation missio_uring/alloc_cache.c io_uring/alloc_cache.h+27 -30 replies
3/3io_uring/rsrc: prefill the node cache when a file table is registered emptyio_uring/alloc_cache.c io_uring/alloc_cache.h io_uring/rsrc.c+54 -103 replies

Reviews trailers per patch

None of the 3 patches has a Reviewed-by or Acked-by.

PatchChengfeng Lin
1/3T-b
2/3T-b
3/3T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 0/3] io_uring/rsrc: reduce node allocation cost on sparse file table installs

Chengfeng Lin reported an 11.6% per-install slowdown in MSG_RING SEND_FD fixed-file installation, bisected to 7029acd8a950 ("io_uring/rsrc: get rid of per-ring io_rsrc_node list"):

https://lore.kernel.org/io-uring/CANGjgdmt0FQ=offsdfn+wEaDxbOFoAa6bi92X_vEo4S6aCZ56A@mail.gmail.com/

That commit is not being questioned here. It removed a real serialisation cost. The side effect is that each fixed-file install now allocates its own io_rsrc_node, and on a first fill of a sparse table every one of those is an allocator miss: io_alloc_cache_init() only allocates the pointer array, io_alloc_cache_put() on the free path is the only thing that populates it, and io_reset_rsrc_node() returns early on a NULL slot so nothing is freed during a first fill. With IO_ALLOC_CACHE_MAX at 128, warming the existing cache cannot cover a 4,096-slot fill.

We spent some time isolating where the per-install cost actually lives. All timing below is Chengfeng's, on a bare-metal i7-12700KF with a pinned core, performance governor, turbo off and fresh boots per point; the evidence tree with reproducers and full logs is linked at the end.

  hypothesis                          result
  ----------------------------------  ---------------------------------
  slab merging defeats locality       slab_nomerge: no change
  SLAB_ACCOUNT on the new cache       +8.5% cost; removed (memcg
                                      accounting the old path never did)
  allocator call overhead             bulk refill, 32x fewer calls:
                                      +0.007%, path verified by probe

fresh slab page creation slab primed, 0 new pages: +0.9%
per-object SLUB allocation path prefill removes it: -9.5%

So the cost is the per-object allocation itself, not calls or pages, and the only way to take it off the install path is to not allocate there.

This series does that in three steps:

0001 dedicated kmem_cache for io_rsrc_node. Neutral on its own; exists

        so 0002/0003 can use kmem_cache_alloc_bulk().

0002 bulk refill of the per-ring cache on a miss. Also neutral on the
reported workload, for the reason above; kept for the machinery.
0003 when a sparse table of N slots is registered, grow the per-ring
cache to min(N, 4096) and bulk-fill it.

Measured on the actual series (v6.18-rc4, 6146a0f1dfae):

  workload                            unpatched   0001+2    +0003
  ----------------------------------  ---------  --------  --------
  reported 4,096-slot first fill,     119.423    118.151   107.731   -9.8%
    ns/install
  same-ring remove+refill, 4,096      712.451    710.384   578.365   -18.8%
    files, us
  one-shot register through fill,     518.387    520.663   526.873   +1.6%
    4,096 slots, us

register 4096 / install 64, us 20.361 21.193 84.407 worse

The second row was not predicted: the enlarged cache retains nodes released by FILES_UPDATE, so steady-state churn stops allocating entirely.

The last two rows are the cost, stated plainly. Prefill moves the allocation work to registration rather than removing it, so a program that registers once and fills once sees no total saving, and a program that registers many slots and installs few pays for nodes it never uses (up to 4096 x 32 bytes plus a 32 KiB pointer array, freed at ring teardown).

Whether that trade is acceptable as default behaviour is the question this RFC raises. An alternative would be an opt-in registration flag, so a program that intends to fill the table can say so and nobody else pays. I am happy to rework 0003 into that shape if it is the preferred one.

Testing: builds and boots on io_uring-6.18; the file table, rsrc and msg_ring liburing tests pass on the patched kernel. The full runtests suite was not run to completion on my build VM (the networking tests take its interface down); the targeted set was. Chengfeng ran the alloc_cache code through allocation-failure, limit and cleanup cases under ASan/UBSan/LeakSanitizer, and verified the 8,192-slot case prefills 4,096 and falls through to bulk for the remainder.

Evidence, reproducers and diagnostic patches:
https://github.com/lcf0399/linux-regression-evidence/tree/af2bf8eaae547d940c64ae8dcd1803baf31f2139/io-uring-msg-ring-send-fd-install

Uzair Beg (3):
io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cache io_uring/rsrc: bulk refill the node cache on allocation miss io_uring/rsrc: prefill the node cache when a file table is registered
empty

 include/linux/io_uring_types.h |  1 +
 io_uring/alloc_cache.c         | 75 ++++++++++++++++++++++++++++++++--
 io_uring/alloc_cache.h         | 11 ++++-
 io_uring/io_uring.c            |  5 +++
 io_uring/io_uring.h            |  1 +
 io_uring/rsrc.c                |  6 +++
 6 files changed, 94 insertions(+), 5 deletions(-)
Signature · 1 line
2.43.0