Skip to contentSkip to conversation

svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER

PATCH In review Chuck Lever 2 participants 4 messages Jun 6 – Jun 7

Subsystems (4)
lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

PATCH 1/1 svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER

From: Chuck Lever <[email protected]>

Jonathan Flynn reports that commit 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink buffers") regresses NFS/RDMA WRITE throughput from 73.9 GiB/s to 30.3 GiB/s on a 128-core single-NUMA-node server driving dual 400Gb/s links with 640 nfsd threads. In the regressed configuration, server CPU utilization rises from 8.5% to 76%, and 73% of all server CPU cycles are spent in native_queued_spin_lock_slowpath.

The contended lock is zone->lock. The page allocator serves allocations only up to PAGE_ALLOC_COSTLY_ORDER (3) from its per-CPU page lists; SVC_RDMA_CONTIG_MAX_ORDER is 4, so every contiguous sink buffer allocation falls through to rmqueue_buddy() and acquires the zone lock. The workload above issues roughly half a million order-4 allocations per second, all serialized on the single zone lock of the one NUMA node. Replacing the GFP mask with GFP_NOWAIT did not change the profile because direct reclaim never
ran: the cycles are spent acquiring the lock, not reclaiming memory.

Cap the allocation order at PAGE_ALLOC_COSTLY_ORDER so contiguous sink buffer allocations remain eligible for the per-CPU page lists, where zone lock acquisition is amortized across pcp batch refills. An order-3 chunk still replaces eight per-page bvecs with one.

Fixes 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink buffers")
---
 net/sunrpc/xprtrdma/svc_rdma_rw.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_rw.c +5 -4
index efde26cac961..4546e594f2d7 100644
@@ -746,11 +746,12 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,
 }
 
 /*
- * Cap contiguous RDMA Read sink allocations at order-4. Higher orders risk
- * allocation failure under GFP_NOWAIT, which would negate the benefit of
- * the contiguous fast path.
+ * Cap contiguous RDMA Read sink allocations at PAGE_ALLOC_COSTLY_ORDER.
+ * The page allocator serves allocations at or below that order from
+ * its per-CPU page lists; above it, every allocation acquires the
+ * zone lock, which serializes all nfsd threads.
  */
-#define SVC_RDMA_CONTIG_MAX_ORDER	4
+#define SVC_RDMA_CONTIG_MAX_ORDER	PAGE_ALLOC_COSTLY_ORDER
 
 /**
  * svc_rdma_alloc_read_pages - Allocate physically contiguous pages
Signature · 1 line
2.54.0
#1 lore raw

I tested the PAGE_ALLOC_COSTLY_ORDER change on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previous GFP_NOWAIT test, measuring 25.4 GiB/s.

Current results are:
Original regressed build: ~30.3 GiB/s
GFP_NOWAIT build: ~31.0 GiB/s
PAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/s

I added the results to the shared bundle. (including flamegraph)

The GFP_NOWAIT and the Original Commit flamegraphs are nearly identical. The dominant stack being:
svc_recv()
-> svc_rdma_build_read_segment_contig()
-> alloc_pages_noprof()
-> get_page_from_freelist()
-> rmqueue_buddy()

The PAGE_ALLOC_COSTLY_ORDER flamegraph is different. Time spent under alloc_pages_noprof() is reduced, but the reduction does not translate into improved throughput.

The following percentages were observed:
Original GFP_NOWAIT
COSTLY_ORDER
svc_recv() 76.09% 75.99%
78.44%
alloc_pages_noprof() 58.07% 57.99% 40.29%
folios_put_refs() 7.15% 7.19%
16.06%
svc_rdma_read_complete() 7.18% 7.21% 16.08%

In other words, the PAGE_ALLOC_COSTLY_ORDER change reduces time spent in the allocation path, but a larger fraction of CPU time then appears under svc_rdma_read_complete() and folios_put_refs(), while overall throughput decreases further.

-Jon

Quoted 10 lines

-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Friday, June 5, 2026 9:57 PM
To: Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>; Jonathan Flynn <[email protected]>
Subject: [PATCH] svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER

Quoted 3 lines

From: Chuck Lever <[email protected]>

Jonathan Flynn reports that commit 18755b8c2f24 ("svcrdma: Use

contiguous

Quoted 2 lines

pages for RDMA Read sink buffers") regresses NFS/RDMA WRITE throughput from 73.9 GiB/s to 30.3 GiB/s on a 128-core single-NUMA-node server

driving

Quoted 1 line

dual 400Gb/s links with 640 nfsd threads. In the regressed

configuration,

Quoted 1 line

server CPU utilization rises from 8.5% to 76%, and 73% of all server CPU

cycles

Quoted 3 lines

are spent in native_queued_spin_lock_slowpath.

The contended lock is zone->lock. The page allocator serves allocations

only

Quoted 3 lines

up to PAGE_ALLOC_COSTLY_ORDER (3) from its per-CPU page lists; SVC_RDMA_CONTIG_MAX_ORDER is 4, so every contiguous sink buffer allocation falls through to rmqueue_buddy() and acquires the zone lock.

The

Quoted 1 line

workload above issues roughly half a million order-4 allocations per

second,

Quoted 1 line

all serialized on the single zone lock of the one NUMA node. Replacing

the

Quoted 6 lines

GFP mask with GFP_NOWAIT did not change the profile because direct reclaim never
ran: the cycles are spent acquiring the lock, not reclaiming memory.

Cap the allocation order at PAGE_ALLOC_COSTLY_ORDER so contiguous sink buffer allocations remain eligible for the per-CPU page lists, where

zone lock

Quoted 1 line

acquisition is amortized across pcp batch refills. An order-3 chunk

still

Quoted 16 lines

replaces eight per-page bvecs with one.

Fixes 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink

buffers")

---
 net/sunrpc/xprtrdma/svc_rdma_rw.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_rw.c +0 -0
index efde26cac961..4546e594f2d7 100644

net/sunrpc/xprtrdma/svc_rdma_rw.c int svc_rdma_prepare_reply_chunk(struct · line 749

Quoted 4 lines
net/sunrpc/xprtrdma/svc_rdma_rw.c +0 -1
@@ -746,11 +746,12 @@ int svc_rdma_prepare_reply_chunk(struct
 svcxprt_rdma *rdma,  }
 
 /*
- * Cap contiguous RDMA Read sink allocations at order-4. Higher orders

risk

net/sunrpc/xprtrdma/svc_rdma_rw.c int svc_rdma_prepare_reply_chunk(struct · line 750

Quoted 1 line
net/sunrpc/xprtrdma/svc_rdma_rw.c +0 -1
- * allocation failure under GFP_NOWAIT, which would negate the benefit

of

Quoted 14 lines
net/sunrpc/xprtrdma/svc_rdma_rw.c +1 -1
- * the contiguous fast path.
+ * Cap contiguous RDMA Read sink allocations at

PAGE_ALLOC_COSTLY_ORDER.
+ * The page allocator serves allocations at or below that order from
+ * its per-CPU page lists; above it, every allocation acquires the
+ * zone lock, which serializes all nfsd threads.
*/

net/sunrpc/xprtrdma/svc_rdma_rw.c +1 -2
-#define SVC_RDMA_CONTIG_MAX_ORDER	4
+#define SVC_RDMA_CONTIG_MAX_ORDER	PAGE_ALLOC_COSTLY_ORDER
 
 /**
  * svc_rdma_alloc_read_pages - Allocate physically contiguous pages
--

2.54.0

author maintainer
#2 lore raw
Quoted 34 lines · Jonathan Flynn

I tested the PAGE_ALLOC_COSTLY_ORDER change on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previous GFP_NOWAIT test, measuring 25.4 GiB/s.

Current results are:
Original regressed build: ~30.3 GiB/s
GFP_NOWAIT build: ~31.0 GiB/s
PAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/s

I added the results to the shared bundle. (including flamegraph)

The GFP_NOWAIT and the Original Commit flamegraphs are nearly identical. The dominant stack being:
svc_recv()
-> svc_rdma_build_read_segment_contig()
-> alloc_pages_noprof()
-> get_page_from_freelist()
-> rmqueue_buddy()

The PAGE_ALLOC_COSTLY_ORDER flamegraph is different. Time spent under alloc_pages_noprof() is reduced, but the reduction does not translate into improved throughput.

The following percentages were observed:
Original GFP_NOWAIT
COSTLY_ORDER
svc_recv() 76.09% 75.99%
78.44%
alloc_pages_noprof() 58.07% 57.99% 40.29%
folios_put_refs() 7.15% 7.19%
16.06%
svc_rdma_read_complete() 7.18% 7.21% 16.08%

Quoted 4 lines

In other words, the PAGE_ALLOC_COSTLY_ORDER change reduces time spent in the allocation path, but a larger fraction of CPU time then appears under svc_rdma_read_complete() and folios_put_refs(), while overall throughput decreases further.

The two failed fixes demonstrate that the current folio allocator is not up to the task -- the problem appears to be on the release side, where the individual pages have to be merged back into an order-4 compound page. I don't yet see a straightforward way to make it work.

Since we're right up against v7.1-rc7, I've added a patch to nfsd-next to revert 18755b8c2f24 -- it will get pulled back into 7.1.y as soon as the v7.2 merge window closes in three weeks.

Signature · 1 line
Chuck Lever
#3 lore raw
Quoted 12 lines

-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Saturday, June 6, 2026 9:18 PM
To: Jonathan Flynn <[email protected]>; Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>
Subject: Re: [PATCH] svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER

Quoted 4 lines
Quoted 3 lines · Jonathan Flynn

I tested the PAGE_ALLOC_COSTLY_ORDER change on the same setup. Unfortunately, it did not improve the regression. Throughput was slightly worse than the previous GFP_NOWAIT test, measuring 25.4

GiB/s.

Quoted 7 lines
Quoted 7 lines

Current results are:
Original regressed build: ~30.3 GiB/s
GFP_NOWAIT build: ~31.0 GiB/s
PAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
Commit reverted: ~73.9 GiB/s

Quoted 17 lines
Quoted 3 lines

I added the results to the shared bundle. (including flamegraph)

The GFP_NOWAIT and the Original Commit flamegraphs are nearly

identical.

Quoted 13 lines

The dominant stack being:
svc_recv()
-> svc_rdma_build_read_segment_contig()
-> alloc_pages_noprof()
-> get_page_from_freelist()
-> rmqueue_buddy()

The PAGE_ALLOC_COSTLY_ORDER flamegraph is different. Time spent under alloc_pages_noprof() is reduced, but the reduction does not translate into improved throughput.

The following percentages were observed:
Original

GFP_NOWAIT

Quoted 4 lines
Quoted 4 lines

COSTLY_ORDER
svc_recv() 76.09% 75.99%
78.44%
alloc_pages_noprof() 58.07% 57.99%

40.29%

Quoted 4 lines
Quoted 4 lines

folios_put_refs() 7.15% 7.19%
16.06%
svc_rdma_read_complete() 7.18% 7.21% 16.08%

Quoted 8 lines
Quoted 1 line

In other words, the PAGE_ALLOC_COSTLY_ORDER change reduces time

spent

Quoted 4 lines

in the allocation path, but a larger fraction of CPU time then appears under
svc_rdma_read_complete() and folios_put_refs(), while overall throughput decreases further.

The two failed fixes demonstrate that the current folio allocator is not

up to

Quoted 5 lines

the task -- the problem appears to be on the release side, where the individual pages have to be merged back into an order-4 compound page. I don't yet see a straightforward way to make it work.

Since we're right up against v7.1-rc7, I've added a patch to nfsd-next

to revert

Quoted 1 line

18755b8c2f24 -- it will get pulled back into 7.1.y as soon as the v7.2

merge

Quoted 5 lines

window closes in three weeks.

--
Chuck Lever

This sounds good.

Thank you for taking the time to investigate it and for working through the test results with us.

If you continue exploring this area in the future and still see promise in the contiguous allocation approach, I'd be happy to help test additional changes as time permits.

-Jon