svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER
Subsystems (4)
Chuck Lever
maintainerPATCH 1/1 svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER
From: Chuck Lever <[email protected]>
Jonathan Flynn reports that commit 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink buffers") regresses NFS/RDMA WRITE throughput from 73.9 GiB/s to 30.3 GiB/s on a 128-core single-NUMA-node server driving dual 400Gb/s links with 640 nfsd threads. In the regressed configuration, server CPU utilization rises from 8.5% to 76%, and 73% of all server CPU cycles are spent in native_queued_spin_lock_slowpath.
The contended lock is zone->lock. The page allocator serves allocations only up to PAGE_ALLOC_COSTLY_ORDER (3) from its per-CPU page lists; SVC_RDMA_CONTIG_MAX_ORDER is 4, so every contiguous sink buffer allocation falls through to rmqueue_buddy() and acquires the zone lock. The workload above issues roughly half a million order-4 allocations per second, all serialized on the single zone lock of the one NUMA node. Replacing the GFP mask with GFP_NOWAIT did not change the profile because direct reclaim never
ran: the cycles are spent acquiring the lock, not reclaiming memory.
Cap the allocation order at PAGE_ALLOC_COSTLY_ORDER so contiguous sink buffer allocations remain eligible for the per-CPU page lists, where zone lock acquisition is amortized across pcp batch refills. An order-3 chunk still replaces eight per-page bvecs with one.
--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_rw.c +5 -4
@@ -746,11 +746,12 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, }/* -* Cap contiguous RDMA Read sink allocations at order-4. Higher orders risk -* allocation failure under GFP_NOWAIT, which would negate the benefit of -* the contiguous fast path. +* Cap contiguous RDMA Read sink allocations at PAGE_ALLOC_COSTLY_ORDER. +* The page allocator serves allocations at or below that order from +* its per-CPU page lists; above it, every allocation acquires the +* zone lock, which serializes all nfsd threads. */ -#define SVC_RDMA_CONTIG_MAX_ORDER 4 +#define SVC_RDMA_CONTIG_MAX_ORDER PAGE_ALLOC_COSTLY_ORDER/** * svc_rdma_alloc_read_pages - Allocate physically contiguous pages
Signature · 1 line
2.54.0
I tested the PAGE_ALLOC_COSTLY_ORDER change on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previous GFP_NOWAIT test, measuring 25.4 GiB/s.
Current results are:
Original regressed build: ~30.3 GiB/sGFP_NOWAIT build: ~31.0 GiB/sPAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/s
I added the results to the shared bundle. (including flamegraph)
The GFP_NOWAIT and the Original Commit flamegraphs are nearly identical. The dominant stack being:svc_recv()
-> svc_rdma_build_read_segment_contig()
-> alloc_pages_noprof()
-> get_page_from_freelist()
-> rmqueue_buddy()
The PAGE_ALLOC_COSTLY_ORDER flamegraph is different. Time spent under alloc_pages_noprof() is reduced, but the reduction does not translate into improved throughput.
The following percentages were observed:
Original GFP_NOWAITCOSTLY_ORDERsvc_recv() 76.09% 75.99%
78.44%alloc_pages_noprof() 58.07% 57.99% 40.29%folios_put_refs() 7.15% 7.19%
16.06%svc_rdma_read_complete() 7.18% 7.21% 16.08%
In other words, the PAGE_ALLOC_COSTLY_ORDER change reduces time spent in the allocation path, but a larger fraction of CPU time then appears under svc_rdma_read_complete() and folios_put_refs(), while overall throughput decreases further.
-Jon
Quoted 10 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Friday, June 5, 2026 9:57 PM
To: Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>; Jonathan Flynn <[email protected]>
Subject: [PATCH] svcrdma: Cap Read sink allocations atPAGE_ALLOC_COSTLY_ORDER
Quoted 3 lines
From: Chuck Lever <[email protected]>
Jonathan Flynn reports that commit
18755b8c2f24("svcrdma: Use
contiguous
Quoted 2 lines
pages for RDMA Read sink buffers") regresses NFS/RDMA WRITE throughput from 73.9 GiB/s to 30.3 GiB/s on a 128-core single-NUMA-node server
driving
Quoted 1 line
dual 400Gb/s links with 640 nfsd threads. In the regressed
configuration,
Quoted 1 line
server CPU utilization rises from 8.5% to 76%, and 73% of all server CPU
cycles
Quoted 3 lines
are spent in
native_queued_spin_lock_slowpath.The contended lock is
zone->lock. The page allocator serves allocations
only
Quoted 3 lines
up to
PAGE_ALLOC_COSTLY_ORDER(3) from its per-CPU page lists;SVC_RDMA_CONTIG_MAX_ORDERis 4, so every contiguous sink buffer allocation falls through tormqueue_buddy()and acquires the zone lock.
The
Quoted 1 line
workload above issues roughly half a million order-4 allocations per
second,
Quoted 1 line
all serialized on the single zone lock of the one NUMA node. Replacing
the
Quoted 6 lines
GFP mask with
GFP_NOWAITdid not change the profile because direct reclaim never
ran: the cycles are spent acquiring the lock, not reclaiming memory.Cap the allocation order at
PAGE_ALLOC_COSTLY_ORDERso contiguous sink buffer allocations remain eligible for the per-CPU page lists, where
zone lock
Quoted 1 line
acquisition is amortized across pcp batch refills. An order-3 chunk
still
Quoted 16 lines
replaces eight per-page bvecs with one.
Reported-by Jonathan Flynn <[email protected]>Fixes 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sinkbuffers")
Signed-off-by Chuck Lever <[email protected]>--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)net/sunrpc/xprtrdma/svc_rdma_rw.c +0
-0
net/sunrpc/xprtrdma/svc_rdma_rw.c int svc_rdma_prepare_reply_chunk(struct · line 749
Quoted 4 lines
net/sunrpc/xprtrdma/svc_rdma_rw.c +0
-1@@ -746,11 +746,12 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, }/* -* Cap contiguous RDMA Read sink allocations at order-4. Higher orders
risk
net/sunrpc/xprtrdma/svc_rdma_rw.c int svc_rdma_prepare_reply_chunk(struct · line 750
Quoted 1 line
net/sunrpc/xprtrdma/svc_rdma_rw.c +0
-1-* allocation failure under GFP_NOWAIT, which would negate the benefit
of
Quoted 14 lines
net/sunrpc/xprtrdma/svc_rdma_rw.c +1
-1-* the contiguous fast path. +* Cap contiguous RDMA Read sink allocations at
PAGE_ALLOC_COSTLY_ORDER.
+ * The page allocator serves allocations at or below that order from
+ * its per-CPU page lists; above it, every allocation acquires the
+ * zone lock, which serializes all nfsd threads.
*/net/sunrpc/xprtrdma/svc_rdma_rw.c +1
-2-#define SVC_RDMA_CONTIG_MAX_ORDER 4 +#define SVC_RDMA_CONTIG_MAX_ORDER PAGE_ALLOC_COSTLY_ORDER/** * svc_rdma_alloc_read_pages - Allocate physically contiguous pages -- 2.54.0
Chuck Lever
maintainerQuoted 34 lines · Jonathan Flynn
I tested the
PAGE_ALLOC_COSTLY_ORDERchange on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previousGFP_NOWAITtest, measuring 25.4 GiB/s.Current results are:
Original regressed build: ~30.3 GiB/sGFP_NOWAITbuild: ~31.0 GiB/sPAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/sI added the results to the shared bundle. (including flamegraph)
The
GFP_NOWAITand the Original Commit flamegraphs are nearly identical. The dominant stack being:svc_recv()
->svc_rdma_build_read_segment_contig()
->alloc_pages_noprof()
->get_page_from_freelist()
->rmqueue_buddy()The
PAGE_ALLOC_COSTLY_ORDERflamegraph is different. Time spent underalloc_pages_noprof()is reduced, but the reduction does not translate into improved throughput.The following percentages were observed:
OriginalGFP_NOWAITCOSTLY_ORDERsvc_recv()76.09% 75.99%
78.44%alloc_pages_noprof()58.07% 57.99% 40.29%folios_put_refs()7.15% 7.19%
16.06%svc_rdma_read_complete()7.18% 7.21% 16.08%
Quoted 4 lines
In other words, the
PAGE_ALLOC_COSTLY_ORDERchange reduces time spent in the allocation path, but a larger fraction of CPU time then appears undersvc_rdma_read_complete()andfolios_put_refs(), while overall throughput decreases further.
The two failed fixes demonstrate that the current folio allocator is not up to the task -- the problem appears to be on the release side, where the individual pages have to be merged back into an order-4 compound page. I don't yet see a straightforward way to make it work.
Since we're right up against v7.1-rc7, I've added a patch to nfsd-next to revert 18755b8c2f24 -- it will get pulled back into 7.1.y as soon as the v7.2 merge window closes in three weeks.
Signature · 1 line
Chuck Lever
Quoted 12 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Saturday, June 6, 2026 9:18 PM
To: Jonathan Flynn <[email protected]>; Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>
Subject: Re: [PATCH] svcrdma: Cap Read sink allocations atPAGE_ALLOC_COSTLY_ORDER
Quoted 4 lines
Quoted 3 lines · Jonathan Flynn
I tested the
PAGE_ALLOC_COSTLY_ORDERchange on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previousGFP_NOWAITtest, measuring 25.4
GiB/s.
Quoted 7 lines
Quoted 7 lines
Current results are:
Original regressed build: ~30.3 GiB/sGFP_NOWAITbuild: ~31.0 GiB/sPAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/s
Quoted 17 lines
Quoted 3 lines
I added the results to the shared bundle. (including flamegraph)
The
GFP_NOWAITand the Original Commit flamegraphs are nearlyidentical.
Quoted 13 lines
The dominant stack being:
svc_recv()
->svc_rdma_build_read_segment_contig()
->alloc_pages_noprof()
->get_page_from_freelist()
->rmqueue_buddy()The
PAGE_ALLOC_COSTLY_ORDERflamegraph is different. Time spent underalloc_pages_noprof()is reduced, but the reduction does not translate into improved throughput.The following percentages were observed:
Original
GFP_NOWAIT
Quoted 4 lines
Quoted 4 lines
COSTLY_ORDERsvc_recv()76.09% 75.99%
78.44%alloc_pages_noprof()58.07% 57.99%
40.29%
Quoted 4 lines
Quoted 4 lines
folios_put_refs()7.15% 7.19%
16.06%svc_rdma_read_complete()7.18% 7.21% 16.08%
Quoted 8 lines
Quoted 1 line
In other words, the
PAGE_ALLOC_COSTLY_ORDERchange reduces timespent
Quoted 4 lines
in the allocation path, but a larger fraction of CPU time then appears under
svc_rdma_read_complete()andfolios_put_refs(), while overall throughput decreases further.The two failed fixes demonstrate that the current folio allocator is not
up to
Quoted 5 lines
the task -- the problem appears to be on the release side, where the individual pages have to be merged back into an order-4 compound page. I don't yet see a straightforward way to make it work.
Since we're right up against v7.1-rc7, I've added a patch to nfsd-next
to revert
Quoted 1 line
18755b8c2f24-- it will get pulled back into 7.1.y as soon as the v7.2
merge
Quoted 5 lines
window closes in three weeks.
--
Chuck Lever
This sounds good.
Thank you for taking the time to investigate it and for working through the test results with us.
If you continue exploring this area in the future and still see promise in the contiguous allocation approach, I'd be happy to help test additional changes as time permits.
-Jon