svcrdma: Avoid direct reclaim when allocating Read sink buffers
Subsystems (4)
Chuck Lever
maintainerPATCH 1/1 svcrdma: Avoid direct reclaim when allocating Read sink buffers
From: Chuck Lever <[email protected]>
svc_rdma_alloc_read_pages() passes __GFP_NORETRY, which limits the allocator to a single round of direct reclaim and asynchronous compaction per attempt. Under memory pressure or fragmentation that round can take a long time, and the fallback loop repeats it at each order, multiplying the stall while the RPC waits for its Read sink buffer.
The contiguous allocation is opportunistic: when it fails, Read sink buffers come from the pages already in rq_pages[]. Direct reclaim effort buys little here. Allocate with GFP_NOWAIT instead, which omits __GFP_DIRECT_RECLAIM so the allocator takes pages only from the free lists and returns NULL immediately when none are available. GFP_NOWAIT retains __GFP_KSWAPD_RECLAIM, so a failed attempt still wakes kswapd to replenish higher-order pages in the background, and it already includes __GFP_NOWARN. __GFP_NORETRY has no effect once direct reclaim is off. skb_page_frag_refill() takes the same approach for its opportunistic high-order allocation.
--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
Given the perf symbol resolution inaccuracies I can't swear this will fix the issue, but here's a stab at it.
net/sunrpc/xprtrdma/svc_rdma_rw.c +9 -9
@@ -746,10 +746,9 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, }/* -* Cap contiguous RDMA Read sink allocations at order-4. -* Higher orders risk allocation failure under -* __GFP_NORETRY, which would negate the benefit of the -* contiguous fast path. +* Cap contiguous RDMA Read sink allocations at order-4. Higher orders risk +* allocation failure under GFP_NOWAIT, which would negate the benefit of +* the contiguous fast path. */ #define SVC_RDMA_CONTIG_MAX_ORDER 4
@@ -758,9 +757,11 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,* @nr_pages: number of pages needed * @order: on success, set to the allocation order * -* Attempts a higher-order allocation, falling back to smaller orders. -* The returned pages are split immediately so each sub-page has its -* own refcount and can be freed independently. +* Attempts a higher-order allocation, falling back to smaller orders. The +* allocation is opportunistic: it takes pages only from the free lists, +* without direct reclaim, so it fails fast under memory pressure. The +* returned pages are split immediately so each sub-page has its own +* refcount and can be freed independently. * * Returns a pointer to the first page on success, or NULL if even * order-1 allocation fails.
@@ -775,8 +776,7 @@ svc_rdma_alloc_read_pages(unsigned int nr_pages, unsigned int *order) SVC_RDMA_CONTIG_MAX_ORDER);while (o >=1 ) { - page =alloc_pages (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN, - o); + page =alloc_pages (GFP_NOWAIT , o);if (page) {split_page (page, o); *order = o;
Signature · 1 line
2.54.0
Thanks Chuck, give me a few to test it out!
Quoted 10 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Friday, June 5, 2026 4:31 PM
To: Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>; Jonathan Flynn <[email protected]>
Subject: [PATCH] svcrdma: Avoid direct reclaim when allocating Read sink buffers
Quoted 4 lines
From: Chuck Lever <[email protected]>
svc_rdma_alloc_read_pages()passes__GFP_NORETRY, which limits the allocator to a single round of direct reclaim and asynchronous
compaction per
Quoted 1 line
attempt. Under memory pressure or fragmentation that round can take a
long
Quoted 1 line
time, and the fallback loop repeats it at each order, multiplying the
stall while
Quoted 3 lines
the RPC waits for its Read sink buffer.
The contiguous allocation is opportunistic: when it fails, Read sink
buffers
Quoted 1 line
come from the pages already in rq_pages[]. Direct reclaim effort buys
little
Quoted 2 lines
here. Allocate with
GFP_NOWAITinstead, which omits__GFP_DIRECT_RECLAIMso the allocator takes pages only from the free
lists
Quoted 2 lines
and returns NULL immediately when none are available.
GFP_NOWAITretains__GFP_KSWAPD_RECLAIM, so a failed attempt still wakes kswapd to
replenish
Quoted 3 lines
higher-order pages in the background, and it already includes
__GFP_NOWARN.__GFP_NORETRYhas no effect once direct reclaim is off.skb_page_frag_refill()takes the same approach for its opportunistic
high-
Quoted 4 lines
order allocation.
Reported-by Jonathan Flynn <[email protected]>Fixes 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink
Quoted 8 lines
buffers")
Signed-off-by Chuck Lever <[email protected]>--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)Given the perf symbol resolution inaccuracies I can't swear this will
fix the
Quoted 56 lines
issue, but here's a stab at it.
net/sunrpc/xprtrdma/svc_rdma_rw.c +13
-9@@ -746,10 +746,9 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, }/* -* Cap contiguous RDMA Read sink allocations at order-4. -* Higher orders risk allocation failure under -* __GFP_NORETRY, which would negate the benefit of the -* contiguous fast path. +* Cap contiguous RDMA Read sink allocations at order-4. Higher orders +risk +* allocation failure under GFP_NOWAIT, which would negate the benefit +of +* the contiguous fast path. */ #define SVC_RDMA_CONTIG_MAX_ORDER 4@@ -758,9 +757,11 @@ int svc_rdma_prepare_reply_chunk(structsvcxprt_rdma *rdma, * @nr_pages: number of pages needed * @order: on success, set to the allocation order * - * Attempts a higher-order allocation, falling back to smaller orders. - * The returned pages are split immediately so each sub-page has its - * own refcount and can be freed independently. + * Attempts a higher-order allocation, falling back to smaller orders. + The + * allocation is opportunistic: it takes pages only from the free + lists, + * without direct reclaim, so it fails fast under memory pressure. The + * returned pages are split immediately so each sub-page has its own + * refcount and can be freed independently. * * Returns a pointer to the first page on success, or NULL if even * order-1 allocation fails.@@ -775,8 +776,7 @@ svc_rdma_alloc_read_pages(unsigned int nr_pages, unsigned int *order) SVC_RDMA_CONTIG_MAX_ORDER);while (o >=1 ) { - page =alloc_pages (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN, - o); + page =alloc_pages (GFP_NOWAIT , o);if (page) {net/sunrpc/xprtrdma/svc_rdma_rw.c +0
-1split_page (page, o); *order = o; --2.54.0
Quoted 10 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Friday, June 5, 2026 4:31 PM
To: Mike Snitzer <[email protected]>
Cc: [email protected]; [email protected]; Chuck Lever <[email protected]>; Jonathan Flynn <[email protected]>
Subject: [PATCH] svcrdma: Avoid direct reclaim when allocating Read sink buffers
Quoted 4 lines
From: Chuck Lever <[email protected]>
svc_rdma_alloc_read_pages()passes__GFP_NORETRY, which limits the allocator to a single round of direct reclaim and asynchronous
compaction per
Quoted 1 line
attempt. Under memory pressure or fragmentation that round can take a
long
Quoted 1 line
time, and the fallback loop repeats it at each order, multiplying the
stall while
Quoted 3 lines
the RPC waits for its Read sink buffer.
The contiguous allocation is opportunistic: when it fails, Read sink
buffers
Quoted 1 line
come from the pages already in rq_pages[]. Direct reclaim effort buys
little
Quoted 2 lines
here. Allocate with
GFP_NOWAITinstead, which omits__GFP_DIRECT_RECLAIMso the allocator takes pages only from the free
lists
Quoted 2 lines
and returns NULL immediately when none are available.
GFP_NOWAITretains__GFP_KSWAPD_RECLAIM, so a failed attempt still wakes kswapd to
replenish
Quoted 3 lines
higher-order pages in the background, and it already includes
__GFP_NOWARN.__GFP_NORETRYhas no effect once direct reclaim is off.skb_page_frag_refill()takes the same approach for its opportunistic
high-
Quoted 4 lines
order allocation.
Reported-by Jonathan Flynn <[email protected]>Fixes 18755b8c2f24 ("svcrdma: Use contiguous pages for RDMA Read sink
Quoted 8 lines
buffers")
Signed-off-by Chuck Lever <[email protected]>--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)Given the perf symbol resolution inaccuracies I can't swear this will
fix the
Quoted 55 lines
issue, but here's a stab at it.
net/sunrpc/xprtrdma/svc_rdma_rw.c +13
-9@@ -746,10 +746,9 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, }/* -* Cap contiguous RDMA Read sink allocations at order-4. -* Higher orders risk allocation failure under -* __GFP_NORETRY, which would negate the benefit of the -* contiguous fast path. +* Cap contiguous RDMA Read sink allocations at order-4. Higher orders +risk +* allocation failure under GFP_NOWAIT, which would negate the benefit +of +* the contiguous fast path. */ #define SVC_RDMA_CONTIG_MAX_ORDER 4@@ -758,9 +757,11 @@ int svc_rdma_prepare_reply_chunk(structsvcxprt_rdma *rdma, * @nr_pages: number of pages needed * @order: on success, set to the allocation order * - * Attempts a higher-order allocation, falling back to smaller orders. - * The returned pages are split immediately so each sub-page has its - * own refcount and can be freed independently. + * Attempts a higher-order allocation, falling back to smaller orders. + The + * allocation is opportunistic: it takes pages only from the free + lists, + * without direct reclaim, so it fails fast under memory pressure. The + * returned pages are split immediately so each sub-page has its own + * refcount and can be freed independently. * * Returns a pointer to the first page on success, or NULL if even * order-1 allocation fails.@@ -775,8 +776,7 @@ svc_rdma_alloc_read_pages(unsigned int nr_pages, unsigned int *order) SVC_RDMA_CONTIG_MAX_ORDER);while (o >=1 ) { - page =alloc_pages (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN, - o); + page =alloc_pages (GFP_NOWAIT , o);if (page) {net/sunrpc/xprtrdma/svc_rdma_rw.c +0
-1split_page (page, o); *order = o; --
Quoted 1 line
2.54.0
Unfortunately, the GFP_NOWAIT change did not materially affect either throughput or the perf profile. The allocator-heavy stack rooted at svc_rdma_build_read_segment_contig() remains dominant, with alloc_pages_noprof() and rmqueue_buddy() continuing to account for a significant portion of the samples, similar to the original regressed build.
I have added a gfp-nowait directory to the OneDrive link referenced in my previous email. It contains the fio results, perf reports, and a flamegraph for the GFP_NOWAIT test.
I have also added a flamegraph to:
rpcrdma-regression/regressed/phase2/server
for the original regressed configuration.
-Jon