Skip to contentSkip to conversation

mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

PATCH Applied Salvatore Dipietro 4 participants 6 messages Oct 1 – Nov 5

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v6 1/1 mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the !can_direct_reclaim bailout, which since commit 4c0ed883e051 ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. The exclusion is not restricted to costly orders: any __GFP_NORETRY caller has said it can cope with failure, so failing is preferable to fragmenting for all of them.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -

  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%
---
 mm/page_alloc.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +18 -3
index 12fac9084c48..64d09964ca77 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

Signature · 6 lines
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
maintainer
#1 lore raw
Quoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Quoted 9 lines

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

Thanks, I'll update mm-hotfixes to this version.

Here's what changed:

mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations +3 -0
@@ -4930,8 +4930,11 @@ retry:
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

_

#2 lore raw
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Quoted 9 lines
Quoted 9 lines

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

Since it expands the impact of a pre-existing bug of nonblocking allocation eating reserves [1] it shouldn't proceed to mainline without a fix for that (Johannes posted it there but the review discussion got stalled).

Quoted 1 line

Thanks, I'll update mm-hotfixes to this version.

I've also argued for not handling it as a hotfix [2]

[1] https://lore.kernel.org/all/[email protected]/

[2] https://lore.kernel.org/all/[email protected]/

Quoted 18 lines

Here's what changed:

mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations +3 -0
@@ -4930,8 +4930,11 @@ retry:
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

_

maintainer
#3 lore raw
Quoted 25 lines · On Fri, 2 Oct 2026 10:23:47 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Quoted 9 lines
Quoted 9 lines

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

Since it expands the impact of a pre-existing bug of nonblocking allocation eating reserves [1] it shouldn't proceed to mainline without a fix for that (Johannes posted it there but the review discussion got stalled).

Quoted 8 lines
Quoted 1 line

Thanks, I'll update mm-hotfixes to this version.

I've also argued for not handling it as a hotfix [2]

[1] https://lore.kernel.org/all/[email protected]/

[2] https://lore.kernel.org/all/[email protected]/

Thanks. I moved this to tail of mm-unstable, removed its cc:stable and added a note indicating that changes are expected.

Tested-by on v6 1/1
#4 lore raw
In reply to Salvatore Dipietro
Quoted 118 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the !can_direct_reclaim bailout, which since commit 4c0ed883e051 ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. The exclusion is not restricted to costly orders: any __GFP_NORETRY caller has said it can cope with failure, so failing is preferable to fragmenting for all of them.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -

  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%
---
 mm/page_alloc.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +8 -3
index 12fac9084c48..64d09964ca77 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4819

Quoted 10 lines
mm/page_alloc.c +7 -0
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

This would trigger a warning if __GFP_NOFAIL and __GFP_NORETRY were ever used together, though that does not make sense and I haven't found a case where it is used in practice.

Aside from either warning for the specific case of these flags being used together or adding a comment right here to that effect, I don't think any other change is needed, though.

Another comment about the general approach: TTM pool page allocator does use __GFP_NORETRY and __GFP_DIRECT_RECLAIM for high orders and when we previously suggested to drop __GFP_DIRECT_RECLAIM to avoid compaction, it was frowned upon. In the end, a lot of other changes were made and for the relevant cases, this does not seem to change its current behavior, amongst other things, because it uses __GFP_THISNODE.

I did run some TTM GTT allocations with some fragmentation workload and I haven't seen much change in the latency for these high order allocations, so:

Quoted 24 lines
mm/page_alloc.c +3 -0
@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

Signature · 6 lines
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
#5 lore raw
Quoted 125 lines · Thadeu Lima de Souza Cascardo
Quoted 124 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the !can_direct_reclaim bailout, which since commit 4c0ed883e051 ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. The exclusion is not restricted to costly orders: any __GFP_NORETRY caller has said it can cope with failure, so failing is preferable to fragmenting for all of them.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -

  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%
---
 mm/page_alloc.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +14 -3
index 12fac9084c48..64d09964ca77 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
Quoted 8 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4819

Quoted 4 lines
mm/page_alloc.c +1 -0
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

This would trigger a warning if __GFP_NOFAIL and __GFP_NORETRY were ever used together, though that does not make sense and I haven't found a case where it is used in practice.

Right. But always there is some risk someone does that unintentionally by trying to turn an allocation with incoming gfp flags (including unexpected __GFP_NOFAIL) to an opportunistic one first by appending __GFP_NORETRY, etc.

Quoted 3 lines

Aside from either warning for the specific case of these flags being used together or adding a comment right here to that effect, I don't think any other change is needed, though.

Indeed. I thought we did have such warnings. Also a combination with __GFP_RETRY_MAYFAIL would make no sense. So we should perhaps add them.

Quoted 7 lines

Another comment about the general approach: TTM pool page allocator does use __GFP_NORETRY and __GFP_DIRECT_RECLAIM for high orders and when we previously suggested to drop __GFP_DIRECT_RECLAIM to avoid compaction, it was frowned upon. In the end, a lot of other changes were made and for the relevant cases, this does not seem to change its current behavior, amongst other things, because it uses __GFP_THISNODE.

Quoted 5 lines

I did run some TTM GTT allocations with some fragmentation workload and I haven't seen much change in the latency for these high order allocations, so:

Thanks!

Quoted 24 lines
Quoted 24 lines
mm/page_alloc.c +3 -0
@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935

Signature · 6 lines
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico