Skip to contentSkip to conversation

mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

PATCH Applied Salvatore Dipietro 8 participants 41 messages Sep 4 – Oct 29

Subsystems (4)

v4 of 6 Applied 1/1 Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

---
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
mm/page_alloc.c +15 -3
index 12fac9084c48..542c2ec31061 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
Signature · 6 lines
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
maintainer Acked-by on v4 1/1
#1 lore raw
Quoted 54 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc Andrew Morton <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Quoted 8 lines

I guess this will have to do until unlikely(we figure out a better API)...

Quoted 47 lines

---
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
mm/page_alloc.c +15 -3
index 12fac9084c48..542c2ec31061 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
reviewer Acked-by
#2 lore raw
Quoted 64 lines · Vlastimil Babka (SUSE)
Quoted 54 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc Andrew Morton <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Quoted 8 lines
Quoted 1 line

I guess this will have to do until unlikely(we figure out a better API)...

We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on various policies we currently have. Even better if callers can do that instead.

Quoted 35 lines
Quoted 32 lines

---
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
mm/page_alloc.c +3 -3
index 12fac9084c48..542c2ec31061 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
Quoted 4 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4808

Quoted 4 lines
mm/page_alloc.c +4 -0
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */

Should this also be documented in gfp_types.h? It currently only says,

"__GFP_NORETRY: The VM implementation will try only very lightweight memory direct reclaim to get some memory under memory pressure (thus it can sleep)."

Otherwise,

Quoted 11 lines
Quoted 11 lines
mm/page_alloc.c +8 -0
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
Signature · 2 lines
Best Regards,
Yan, Zi
#3 lore raw
Quoted 65 lines · Zi Yan
Quoted 64 lines · Vlastimil Babka (SUSE)
Quoted 62 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Quoted 4 lines
Quoted 1 line

I guess this will have to do until unlikely(we figure out a better API)...

We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on various policies we currently have.

Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

Quoted 2 lines

Even better if callers can do that
instead.

Not sure about burdening the callers (e.g. "every filesystem should do X" etc), unless they are some intermediate wrappers within mm itself. For example, THP (but only anonymous?) is now handled in such a special way, it could be a candidate. But I also have a vague feeling this was already done in the past. Worth investigating perhaps.

Quoted 65 lines
Quoted 35 lines
Quoted 32 lines

---
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
mm/page_alloc.c +3 -3
index 12fac9084c48..542c2ec31061 100644
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
Quoted 4 lines

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4808

Quoted 4 lines
mm/page_alloc.c +4 -0
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */

Should this also be documented in gfp_types.h? It currently only says,

"__GFP_NORETRY: The VM implementation will try only very lightweight memory direct reclaim to get some memory under memory pressure (thus it can sleep)."

Otherwise,

Quoted 11 lines
Quoted 11 lines
mm/page_alloc.c +8 -0
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
reviewer
#4 lore raw
Quoted 66 lines · Vlastimil Babka (SUSE)
Quoted 65 lines · Zi Yan
Quoted 64 lines · Vlastimil Babka (SUSE)
Quoted 62 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I guess this will have to do until unlikely(we figure out a better API)...

We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on various policies we currently have.

Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

But alloc_flags are not exposed outside of MM, so alloc_page*() users are still limited by the existing GFP_* flags. Are you suggesting alloc_flags could be an addtional input for alloc_page*()?

Quoted 6 lines
Quoted 3 lines

Even better if callers can do that
instead.

Not sure about burdening the callers (e.g. "every filesystem should do X" etc), unless they are some intermediate wrappers within mm itself.

Not all callers. I assume gfp_policy() will only be used by advance users who want specific page allocation behavior without knowing the details of the page allocator or changing how the page allocator behave.

Quoted 4 lines

For example, THP (but only anonymous?) is now handled in such a special way, it could be a candidate. But I also have a vague feeling this was already done in the past. Worth investigating perhaps.

The number of wrappers can grow if different users have different needs. Some might not want direct reclaim, some might not want direct compaction, some might not want kswapd, and so on. I am not sure we want to add wrappers for each.

For this patch, the caller can use gfp_policy() to strip out __GFP_DIRECT_RECLAIM based on the conditions and pass the adjusted gfp to the page allocator.

Signature · 2 lines
Best Regards,
Yan, Zi
#5 lore raw
Quoted 82 lines · Zi Yan
Quoted 66 lines · Vlastimil Babka (SUSE)
Quoted 65 lines · Zi Yan
Quoted 64 lines · Vlastimil Babka (SUSE)
Quoted 62 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I guess this will have to do until unlikely(we figure out a better API)...

We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on various policies we currently have.

Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

But alloc_flags are not exposed outside of MM, so alloc_page*() users are still limited by the existing GFP_* flags. Are you suggesting alloc_flags could be an addtional input for alloc_page*()?

Quoted 4 lines
Quoted 3 lines

Even better if callers can do that
instead.

Quoted 6 lines
Quoted 2 lines

Not sure about burdening the callers (e.g. "every filesystem should do X" etc), unless they are some intermediate wrappers within mm itself.

Not all callers. I assume gfp_policy() will only be used by advance users who want specific page allocation behavior without knowing the details of the page allocator or changing how the page allocator behave.

"without knowing the details"

Quoted 9 lines
Quoted 4 lines

For example, THP (but only anonymous?) is now handled in such a special way, it could be a candidate. But I also have a vague feeling this was already done in the past. Worth investigating perhaps.

The number of wrappers can grow if different users have different needs. Some might not want direct reclaim, some might not want direct compaction, some might not want kswapd, and so on. I am not sure we want to add wrappers for each.

Yet all these are the details?

Quoted 3 lines

For this patch, the caller can use gfp_policy() to strip out __GFP_DIRECT_RECLAIM based on the conditions and pass the adjusted gfp to the page allocator.

I guess I'd need to see a concrete example to grasp this fully.

reviewer Reviewed-by on v4 1/1
#6 lore raw
In reply to Salvatore Dipietro
Quoted 54 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc Andrew Morton <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Johannes Weiner <[email protected]>
Quoted 8 lines

Looks good to me, thanks for keeping at it.

maintainer
#7 lore raw
In reply to Salvatore Dipietro
Quoted 6 lines · On Fri, 4 Sep 2026 11:56:28 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Is there anything particularly unusual about this test case?

Quoted 5 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Quoted 4 lines

That's three years old. v6.10.

This is not good. Can you think of an exceptional reason why this took so long to surface, or is it simply that our testing isn't good?

And thanks for fixing it.

AI review asked a few serious-looking questions:

	https://sashiko.dev/#/patchset/[email protected]
#8 lore raw
Quoted 24 lines · Andrew Morton
Quoted 6 lines · On Fri, 4 Sep 2026 11:56:28 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Is there anything particularly unusual about this test case?

Quoted 5 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Quoted 4 lines
Quoted 4 lines

That's three years old. v6.10.

This is not good. Can you think of an exceptional reason why this took so long to surface, or is it simply that our testing isn't good?

Neither. We've know about these excessive high-order allocation-> direct reclaim compaction costs for a lot longer than this, and have quite a number of filesystem based workloads that have been easily able to reproduce it on demand. XFS first worked around direct compaction overhead back around ~5.17/18.

So why has it taken so long for this to be (kinda) fixed in the memory reclaim code?

It is not for lack of awareness or trying.

I've responded to about half a dozen "direct compaction is expensive" bug reports in the past couple of years where I've said "this needs to be fixed in the reclaim code, not at the individual call sites". Not just fs stuff, but graphics and other subsystems, too.

However, convincing the MM manintainers to do anything w.r.t. changing memory reclaim takes forever, and requires excessive amounts of persistence before anything happens despite clear evidence of the issue and the performance numbers showing how costly direct compaction actually is.

A directly relatable example of this is kvmalloc() semantics. We needed kvmalloc() use fail fast semantics for the high order kmalloc() because it is much less costly to fall back to vmalloc() than do direct compaction....

This was what the XFS changes in 5.17 worked around, and that fix didn't get pulled into the kvmalloc() implementation until 6.15 with commit 46459154f997 ("mm: kvmalloc: make kmalloc fast path real fast path").

IOWs, it often takes years from "bad behaviour known" to "fixed in MM infrastructure", and most of that time is spent trying to convince the MM maintainers that the (relatively simple) fix needs to be done in the MM code itself...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#9 lore raw
In reply to Andrew Morton
Quoted 1 line · On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <[email protected]>…

Is there anything particularly unusual about this test case?

It is a stock pgbench simple-update PostgreSQL workload on a large instance (96 vCPUs), using standard PostgreSQL settings and with no huge pages assigned to the database. We deliberately overprovision the pgbench clients: 1024 clients over 96 threads. That keeps enough writers in the buffered write path concurrently to hit the costly-order allocation failure path continuously. The memory fragmentation comes from page tables: PostgreSQL spawns a new process per client, and those page tables consume ~40% of memory, which significantly limits the page cache and the free memory available.

Quoted 7 lines
Quoted 5 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered. Here is the summary, same host and workload throughout, average of 3 runs:

Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
  v7.3-rc1 + v4                                 155,409        +161.6%

The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version, so not directly comparable, but it shows the performance this workload used to get on this host.

A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 - iomap_write_iter() has been rewritten since - so the reverted row is a one-line behavioural revert, forcing the write loop back to copying at most PAGE_SIZE per iteration:

  -	size_t chunk = mapping_max_folio_size(mapping);
  +	size_t chunk = PAGE_SIZE;

which is what the pre-5d8edfb900d5 loop computed. That makes iomap_get_folio() pass no order hint, so the path issues only order-0 allocations.

Quoted 2 lines

AI review asked a few serious-looking questions:

	https://sashiko.dev/#/patchset/[email protected]

Thanks for pointing that out. To address them, we can have something like the patch below. Performance results are still similar to v4. Happy to submit a formal v5 patch with it if you would like.

Config Avg TPS % vs baseline
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1 + v4                                 155,409        +161.6%
  v7.3-rc1 + proposed patch                     161,994        +172.7%
mm/page_alloc.c +14 -2
index 12fac9084c48..be8b0d72a2db 100644
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);
 	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
 	bool nofail = gfp_mask & __GFP_NOFAIL;
-	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

maintainer
#10 lore raw
Quoted 5 lines · On Thu, 10 Sep 2026 11:46:02 +0000 Salvatore Dipietro <[email protected]> wrote:
Quoted 1 line · On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <[email protected]>…

Is there anything particularly unusual about this test case?

Quoted 9 lines

It is a stock pgbench simple-update PostgreSQL workload on a large instance (96 vCPUs), using standard PostgreSQL settings and with no huge pages assigned to the database. We deliberately overprovision the pgbench clients: 1024 clients over 96 threads. That keeps enough writers in the buffered write path concurrently to hit the costly-order allocation failure path continuously. The memory fragmentation comes from page tables: PostgreSQL spawns a new process per client, and those page tables consume ~40% of memory, which significantly limits the page cache and the free memory available.

OK, thanks.

Quoted 10 lines
Quoted 8 lines
Quoted 6 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered.

Great. That's worth mentioning in the changelog.

Quoted 7 lines
Quoted 3 lines

AI review asked a few serious-looking questions:

	https://sashiko.dev/#/patchset/[email protected]

Thanks for pointing that out. To address them, we can have something like the patch below. Performance results are still similar to v4. Happy to submit a formal v5 patch with it if you would like.

Yes please, a v5 would be good.

#11 lore raw
Quoted 1 line · On Thu, 10 Sep 2026 15:00:00 -0700 Andrew Morton <[email protected]>…

Yes please, a v5 would be good.

posted v5 version in
https://lore.kernel.org/linux-mm/[email protected]/T/#u

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#12 lore raw
In reply to Salvatore Dipietro
Quoted 50 lines · Salvatore Dipietro
Quoted 1 line · On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <[email protected]>…

Is there anything particularly unusual about this test case?

It is a stock pgbench simple-update PostgreSQL workload on a large instance (96 vCPUs), using standard PostgreSQL settings and with no huge pages assigned to the database. We deliberately overprovision the pgbench clients: 1024 clients over 96 threads. That keeps enough writers in the buffered write path concurrently to hit the costly-order allocation failure path continuously. The memory fragmentation comes from page tables: PostgreSQL spawns a new process per client, and those page tables consume ~40% of memory, which significantly limits the page cache and the free memory available.

Quoted 7 lines
Quoted 5 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered. Here is the summary, same host and workload throughout, average of 3 runs:

Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
  v7.3-rc1 + v4                                 155,409        +161.6%

The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version, so not directly comparable, but it shows the performance this workload used to get on this host.

A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 - iomap_write_iter() has been rewritten since - so the reverted row is a one-line behavioural revert, forcing the write loop back to copying at most PAGE_SIZE per iteration:

  -	size_t chunk = mapping_max_folio_size(mapping);
  +	size_t chunk = PAGE_SIZE;

which is what the pre-5d8edfb900d5 loop computed. That makes iomap_get_folio() pass no order hint, so the path issues only order-0 allocations.

Quoted 6 lines
Quoted 2 lines

AI review asked a few serious-looking questions:

	https://sashiko.dev/#/patchset/[email protected]

Thanks for pointing that out. To address them, we can have something like the patch below. Performance results are still similar to v4. Happy to submit a formal v5 patch with it if you would like.

The feedback looks misleading to me.

ALLOC_NON_BLOCKING itself is fine, it usually only means something in conjunction with other flags, like __GFP_HIGH. The access to MIGRATE_HIGHATOMIC that it points out is in itself too generous. This seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT into the highatomic reserves as well, for example.

The rt issue seems made up too. We're talking about costly_order && __GFP_NORETRY allocations. Why would they need access to reserves?

Quoted 18 lines

Config Avg TPS % vs baseline
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1 + v4                                 155,409        +161.6%
  v7.3-rc1 + proposed patch                     161,994        +172.7%
mm/page_alloc.c +4 -1
index 12fac9084c48..be8b0d72a2db 100644
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt

mm/page_alloc.c static inline struct page * · line 4800

Quoted 10 lines
mm/page_alloc.c +10 -0
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);

IMO this is worse because now there is a disconnect between these flags and the gfp mask again. This sets us up for subtle bugs.

I much preferred v4.

#13 lore raw
Quoted 51 lines · Johannes Weiner
Quoted 50 lines · Salvatore Dipietro
Quoted 1 line · On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <[email protected]>…

Is there anything particularly unusual about this test case?

It is a stock pgbench simple-update PostgreSQL workload on a large instance (96 vCPUs), using standard PostgreSQL settings and with no huge pages assigned to the database. We deliberately overprovision the pgbench clients: 1024 clients over 96 threads. That keeps enough writers in the buffered write path concurrently to hit the costly-order allocation failure path continuously. The memory fragmentation comes from page tables: PostgreSQL spawns a new process per client, and those page tables consume ~40% of memory, which significantly limits the page cache and the free memory available.

Quoted 7 lines
Quoted 5 lines

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered. Here is the summary, same host and workload throughout, average of 3 runs:

Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
  v7.3-rc1 + v4                                 155,409        +161.6%

The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version, so not directly comparable, but it shows the performance this workload used to get on this host.

A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 - iomap_write_iter() has been rewritten since - so the reverted row is a one-line behavioural revert, forcing the write loop back to copying at most PAGE_SIZE per iteration:

  -	size_t chunk = mapping_max_folio_size(mapping);
  +	size_t chunk = PAGE_SIZE;

which is what the pre-5d8edfb900d5 loop computed. That makes iomap_get_folio() pass no order hint, so the path issues only order-0 allocations.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

AI review asked a few serious-looking questions:

	https://sashiko.dev/#/patchset/[email protected]

Thanks for pointing that out. To address them, we can have something like the patch below. Performance results are still similar to v4. Happy to submit a formal v5 patch with it if you would like.

The feedback looks misleading to me.

Agreed.

Quoted 2 lines

ALLOC_NON_BLOCKING itself is fine, it usually only means something in conjunction with other flags, like __GFP_HIGH.

+1

Quoted 4 lines

The access to
MIGRATE_HIGHATOMIC that it points out is in itself too generous. This seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT into the highatomic reserves as well, for example.

I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?

Quoted 2 lines

The rt issue seems made up too. We're talking about costly_order && __GFP_NORETRY allocations. Why would they need access to reserves?

Yep.

And the concern about warning is also moot as __GFP_NORETRY is still needed and its users virtually always add __GFP_NOWARN as well.

Quoted 32 lines
Quoted 18 lines

Config Avg TPS % vs baseline
v7.3-rc1 baseline (no patch) 59,408 -

  v7.3-rc1 + v4                                 155,409        +161.6%
  v7.3-rc1 + proposed patch                     161,994        +172.7%
mm/page_alloc.c +4 -1
index 12fac9084c48..be8b0d72a2db 100644
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt

mm/page_alloc.c static inline struct page * · line 4800

Quoted 10 lines
mm/page_alloc.c +10 -0
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);

IMO this is worse because now there is a disconnect between these flags and the gfp mask again. This sets us up for subtle bugs.

Quoted 1 line

I much preferred v4.

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

#14 lore raw
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 4 lines · Johannes Weiner

The access to
MIGRATE_HIGHATOMIC that it points out is in itself too generous. This seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT into the highatomic reserves as well, for example.

I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?

rmqueue_buddy() has this:

			/*
			 * If the allocation fails, allow OOM handling and
			 * order-0 (atomic) allocs access to HIGHATOMIC
			 * reserves as failing now is worse than failing a
			 * high-order atomic allocation in the future.
			 */
			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

It says "atomic", but it's checking only ALLOC_NON_BLOCK, which is broader than GFP_ATOMIC. alloc_flags_nonblocking();

	if (gfp_mask & __GFP_DIRECT_RECLAIM)
		return 0;

	if (gfp_mask & __GFP_NOMEMALLOC)
		return 0;

	alloc_flags |= ALLOC_NON_BLOCK;

	if (order > 0 && (gfp_mask & __GFP_HIGH))
		alloc_flags |= ALLOC_HIGHATOMIC;

So this can apply to random !direct_reclaim requests, no?

I have to correct myself on GFP_TRANSHUGE_LIGHT because it happens to include __GFP_NOMEMALLOC, and so won't actually get ALLOC_NON_BLOCK.

But what about random GFP_NOWAIT and & ~__GFP_DIRECT_RECLAIM sites? Those explicitly don't get watermark exemptions already, and weren't the intent of the rmqueue_buddy() exemption above.

Something like this?

mm/page_alloc.c +2 -1
index 12fac9084c48..c79cc7aa4dff 100644
@@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page && ((alloc_flags & ALLOC_OOM) ||
+				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
mm/page_alloc.h +3 -0
index b9259deddb59..11714ddca254 100644
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flag combination from GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*
Quoted 2 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

maintainer
#15 lore raw
Quoted 5 lines · On Wed, 16 Sep 2026 11:58:29 -0400 Johannes Weiner <[email protected]> wrote:
Quoted 2 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

So I'm hearing that I should drop v5 and revert to v4?

maintainer Acked-by Vlastimil Babka (SUSE) Acked-by Zi Yan Reviewed-by Johannes Weiner Reviewed-by Christoph Hellwig
#16 lore raw
Quoted 10 lines · On Wed, 16 Sep 2026 15:34:26 -0700 Andrew Morton <[email protected]>…
Quoted 5 lines · On Wed, 16 Sep 2026 11:58:29 -0400 Johannes Weiner <[email protected]> wrote:
Quoted 2 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

So I'm hearing that I should drop v5 and revert to v4?

v4:

From: Salvatore Dipietro <[email protected]>
Subject: mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations
Date: Fri, 4 Sep 2026 11:56:28 +0000

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 59,408 -
With this patch 155,409 +161.6%

---

 mm/page_alloc.c |   18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-compaction-for-costly-__gfp_noretry-allocations +15 -3
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, u
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

_

#17 lore raw
Quoted 29 lines · Andrew Morton

From: Salvatore Dipietro <[email protected]>
Subject: mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations
Date: Fri, 4 Sep 2026 11:56:28 +0000

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Quoted 4 lines

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

This patch is still piling hack on hack. We haven't made a serious effort to understand what's going on, we're just adjusting flags until things stop sucking.

What we know:
- Doing compaction every time (which is what we currently do) performs badly
- Doing compaction once at the requested large size and then not again until the smallest size also performs badly. So it's not that we're doing too much compaction, it's that we're doing compaction at all.

But we don't know why compaction is performing badly. For example, we could specify MIGRATE_ASYNC or MIGRATE_SYNC_LIGHT if __GFP_NORETRY is set. Just as a reminder, here's how __GFP_NORETRY is documented:

* %__GFP_NORETRY: The VM implementation will try only very lightweight
* memory direct reclaim to get some memory under memory pressure (thus
* it can sleep). It will avoid disruptive actions like OOM killer. The
* caller must handle the failure which is quite likely to happen under
* heavy memory pressure. The flag is suitable when failure can easily be
* handled at small cost, such as reduced throughput.

So the callers aren't doing anything unreasonable when they say __GFP_NORETRY. It's the page allocator (and apparently the compaction side of it) that's not living up to the documented contract.

So that's one approach which has not, as far as I can tell, been investigated.

The other thing that icks me about this patch is how complex the condition is:

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, u · line 4811

Quoted 7 lines
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-compaction-for-costly-__gfp_noretry-allocations +7 -0
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;

So basically we're saying that __GFP_NORETRY means don't do compaction unless __GFP_THISNODE is set, which is just special pleading. Surely the right answer is to remove __GFP_NORETRY from the one caller which needs __GFP_THISNODE?

A slightly unrelated critique of this patch is that it's far too complicated for what it does. We could achieve the same thing by doing:

static inline bool gfp_compaction_allowed(gfp_t gfp_mask) {
- return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO);
+ return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO) &&
+ (!(gfp_mask & __GFP_NORETRY) || (gfp_mask & __GFP_THISNODE));
}

Something I've been noodling on the past day or so is cleaning up the GFP flags for "how hard to reclaim". We currently have five possibilities encoded in four bits:

1. No direct reclaim (__GFP_DIRECT_RECLAIM clear)
2. Light reclaim (__GFP_DIRECT_RECLAIM | __GFP_NORETRY)
3. Normal reclaim (__GFP_DIRECT_RECLAIM)
4. Extra reclaim (__GFP_DIRECT_RECLAIM | ___GFP_RETRY_MAYFAIL)
5. Reclaim forever (__GFP_DIRECT_RECLAIM | __GFP_NOFAIL)

Clearly we can save ourselves a GFP flag bit by encoding those five options into three bits. Some of the code that manipulates "how hard to reclaim" will need to be adjusted, but it shouldn't be that many places to change.

If we do that, we can insert more options into the mix if they're really needed. Like we could have:

0 - No direct reclaim
1 - Light reclaim, no compaction
2 - Light reclaim with compaction
3 - Normal reclaim
4 - Extra reclaim
5 - Reclaim forever

and we'd still have two extra states in case we need to add more flavours.

#18 lore raw
Quoted 39 lines · Matthew Wilcox
Quoted 29 lines · Andrew Morton

From: Salvatore Dipietro <[email protected]>
Subject: mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations
Date: Fri, 4 Sep 2026 11:56:28 +0000

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Quoted 4 lines

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

This patch is still piling hack on hack. We haven't made a serious effort to understand what's going on, we're just adjusting flags until things stop sucking.

Quoted 7 lines

What we know:
- Doing compaction every time (which is what we currently do) performs badly
- Doing compaction once at the requested large size and then not again until the smallest size also performs badly. So it's not that we're doing too much compaction, it's that we're doing compaction at all.

The very first version [1] mentions the associated drain_all_pages(), interestingly.

Quoted 3 lines

But we don't know why compaction is performing badly. For example, we could specify MIGRATE_ASYNC or MIGRATE_SYNC_LIGHT if __GFP_NORETRY is set. Just as a reminder, here's how __GFP_NORETRY is documented:

That already happens (for costly orders) in __alloc_pages_slowpath() where it sets INIT_COMPACT_PRIORITY.

Quoted 10 lines

* %__GFP_NORETRY: The VM implementation will try only very lightweight
* memory direct reclaim to get some memory under memory pressure (thus
* it can sleep). It will avoid disruptive actions like OOM killer. The
* caller must handle the failure which is quite likely to happen under
* heavy memory pressure. The flag is suitable when failure can easily be
* handled at small cost, such as reduced throughput.

So the callers aren't doing anything unreasonable when they say __GFP_NORETRY. It's the page allocator (and apparently the compaction side of it) that's not living up to the documented contract.

Note that all these different versions of this patch mention compaction, but implement a removal of __GFP_DIRECT_RECLAIM, thus suppressing both reclaim and compaction, in response to __GFP_NORETRY. Thus turning this effectively to a GFP_NOWAIT allocation. So that's also a violation of the contract, no?

In [1] thread and also later on v2 thread by you [2] it was proposed to make the caller itself do this downgrade to GFP_NOWAIT. But Christoph and Dave didn't like that approach (that was conditional to costly order, I don't think yours had a response), which is understandable for the costly-order case, but for your case it makes sense that if you don't want any reclaim+compaction, you tell the allocator GFP_NOWAIT and not __GFP_NORETRY.

Quoted 49 lines

So that's one approach which has not, as far as I can tell, been investigated.

The other thing that icks me about this patch is how complex the condition is:

mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, u · line 4811

Quoted 7 lines
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-compaction-for-costly-__gfp_noretry-allocations +7 -0
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;

So basically we're saying that __GFP_NORETRY means don't do compaction unless __GFP_THISNODE is set, which is just special pleading. Surely the right answer is to remove __GFP_NORETRY from the one caller which needs __GFP_THISNODE?

A slightly unrelated critique of this patch is that it's far too complicated for what it does. We could achieve the same thing by doing:

static inline bool gfp_compaction_allowed(gfp_t gfp_mask) {

(quoted diff) +2 -1
-       return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO);
+       return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO) &&
+               (!(gfp_mask & __GFP_NORETRY) || (gfp_mask & __GFP_THISNODE));
 }

Something I've been noodling on the past day or so is cleaning up the GFP flags for "how hard to reclaim". We currently have five possibilities encoded in four bits:

1. No direct reclaim (__GFP_DIRECT_RECLAIM clear)
2. Light reclaim (__GFP_DIRECT_RECLAIM | __GFP_NORETRY)
3. Normal reclaim (__GFP_DIRECT_RECLAIM)
4. Extra reclaim (__GFP_DIRECT_RECLAIM | ___GFP_RETRY_MAYFAIL)
5. Reclaim forever (__GFP_DIRECT_RECLAIM | __GFP_NOFAIL)

Clearly we can save ourselves a GFP flag bit by encoding those five options into three bits. Some of the code that manipulates "how hard to reclaim" will need to be adjusted, but it shouldn't be that many places to change.

If we do that, we can insert more options into the mix if they're really needed. Like we could have:

Quoted 6 lines

0 - No direct reclaim
1 - Light reclaim, no compaction
2 - Light reclaim with compaction
3 - Normal reclaim
4 - Extra reclaim
5 - Reclaim forever

There's also still (for better or worse) the THP special case of light compaction with no reclaim.
And the long-standing principle (maybe cargo cult? who knows) that for non-costly-order allocations (with fallback) the tradeoff of e.g. 2 is more favorable than 0. But we don't want the callers to have to change flags depending on the order. So it would have to be a distinct option?

Quoted 2 lines

and we'd still have two extra states in case we need to add more flavours.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

#19 lore raw
Quoted 3 lines · Matthew Wilcox

This patch is still piling hack on hack. We haven't made a serious effort to understand what's going on, we're just adjusting flags until things stop sucking.

I have tested two new kernel patches on v7.3-rc1, each isolating one behaviour (diffs at the bottom), and ran the same pgbench simple-update workload (1024 clients / 96 threads / 1200s, 3 iterations each). Baseline is the unpatched regressed kernel; target is ~135k, the pre-5d8edfb900d5 number.

Config Avg TPS % vs baseline
v7.3-rc1 baseline (no patch) 59,408 -
(a) post-reclaim drain step skipped 70,615 +18.9%
(b) compaction disabled only 107,210 +80.5%
(c) v4 (full non-blocking, reference) 154,835 +160.6%

We can notice that:

1. The drain_all_pages() IPI is not the driver. (a) skips the whole post-reclaim drain step for costly __GFP_NORETRY -- that is unreserve_highatomic_pageblock(), drain_all_pages() and the retry together -- and the entire step is worth only ~11k of the ~96k gap. Whatever the split between the three, the cross-CPU drain cannot account for the bulk of it.

2. Turning compaction off is not enough. Your gfp_compaction_allowed() one-liner (b) gets about half the gap, and it declines across iterations (135k -> 98k -> 89k). It drops compaction for __GFP_NORETRY at every order that can use it, __GFP_THISNODE excepted, not only at the costly order v4 gates on; the decline is consistent with the zone no longer being repaired. (c) only makes the costly attempt non-blocking and leaves kswapd/kcompactd working, and it does not show the decline.

3. We have collected metrics to understand how often the allocator stalls and normalised to a million page writebacks (nr_written), over the same 3 x 1200s iterations as above:

compact_ allocstall_ pgscan_
stall movable direct
v7.3-rc1 baseline 513 277 90,230
(a) post-reclaim drain skipped 912 472 129,249
(b) compaction disabled 0 778 44,737
(c) v4 (full non-blocking) 5 39 4,407

The baseline enters stall compaction ~100x more often per page written than (c) does. (a) makes all three metrics worse. (b) removes direct compaction entirely (0 stalls) but does not remove the work: it enters direct reclaim 2.8x more often than the baseline (778 vs 277 allocstall_movable), and still does half the baseline's direct scanning (44,737 vs 90,230 pgscan_direct). (c) cuts both instead -- reclaim stalls 7x lower and direct scanning 20x lower (39 and 4,407). That is why (b) recovers only half the gap -- the stall moves from compaction into reclaim instead of going away.

If you agree this is the right direction, I am happy to send a v6 with (c)'s behaviour plus the defrag_mode fix discussed in the v5 thread.

Thanks,
Salvatore

---
For reproducibility, here is the exact diff behind each measured row above (all against v7.3-rc1).

(a) post-reclaim drain step skipped -- 70,615 tps:

  diff --git a/mm/page_alloc.c b/mm/page_alloc.c
  --- a/mm/page_alloc.c
  +++ b/mm/page_alloc.c
  @@ -4487,7 +4487,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
   	 * pages are pinned on the per-cpu lists or in high alloc reserves.
   	 * Shrink them and try again
   	 */
  -	if (!page && !drained) {
  +	if (!page && !drained &&
  +	    !(order > PAGE_ALLOC_COSTLY_ORDER && (gfp_mask & __GFP_NORETRY))) {
   		unreserve_highatomic_pageblock(ac, false);
   		drain_all_pages(NULL);
   		drained = true;

(b) compaction disabled only -- 107,210 tps:

  diff --git a/include/linux/gfp.h b/include/linux/gfp.h
  --- a/include/linux/gfp.h
  +++ b/include/linux/gfp.h
  @@ -380,7 +380,8 @@ static inline bool gfp_has_io_fs(gfp_t gfp)
    */
   static inline bool gfp_compaction_allowed(gfp_t gfp_mask)
   {
  -	return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO);
  +	return IS_ENABLED(CONFIG_COMPACTION) && (gfp_mask & __GFP_IO) &&
  +		(!(gfp_mask & __GFP_NORETRY) || (gfp_mask & __GFP_THISNODE));
   }

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#20 lore raw
In reply to Johannes Weiner
Quoted 12 lines · Johannes Weiner
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 4 lines · Johannes Weiner

The access to
MIGRATE_HIGHATOMIC that it points out is in itself too generous. This seems like a real but separate bug. It will allow GFP_TRANSHUGE_LIGHT into the highatomic reserves as well, for example.

I don't follow this part. For ALLOC_HIGHATOMIC you need __GFP_HIGH in alloc_flags_nonblocking(). So GFP_TRANSHUGE_LIGHT won't get the access, no?

rmqueue_buddy() has this:

Quoted 8 lines
			/*
			 * If the allocation fails, allow OOM handling and
			 * order-0 (atomic) allocs access to HIGHATOMIC
			 * reserves as failing now is worse than failing a
			 * high-order atomic allocation in the future.
			 */
			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Ah right, comes from Matt's 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")

Quoted 14 lines

It says "atomic", but it's checking only ALLOC_NON_BLOCK, which is broader than GFP_ATOMIC. alloc_flags_nonblocking();

	if (gfp_mask & __GFP_DIRECT_RECLAIM)
		return 0;

	if (gfp_mask & __GFP_NOMEMALLOC)
		return 0;

	alloc_flags |= ALLOC_NON_BLOCK;

	if (order > 0 && (gfp_mask & __GFP_HIGH))
		alloc_flags |= ALLOC_HIGHATOMIC;
Quoted 1 line

So this can apply to random !direct_reclaim requests, no?

Yes, and I agree it shouldn't.

Quoted 2 lines

I have to correct myself on GFP_TRANSHUGE_LIGHT because it happens to include __GFP_NOMEMALLOC, and so won't actually get ALLOC_NON_BLOCK.

At least there's that.

Quoted 25 lines

But what about random GFP_NOWAIT and & ~__GFP_DIRECT_RECLAIM sites? Those explicitly don't get watermark exemptions already, and weren't the intent of the rmqueue_buddy() exemption above.

Something like this?

mm/page_alloc.c +2 -1
index 12fac9084c48..c79cc7aa4dff 100644
@@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page && ((alloc_flags & ALLOC_OOM) ||
+				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
mm/page_alloc.h +0 -0
index b9259deddb59..11714ddca254 100644

mm/page_alloc.h line 68

Quoted 9 lines
mm/page_alloc.h +3 -0
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flag combination from GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*

Should work.

Quoted 6 lines
Quoted 3 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

But Salvatore's change (v4, not v5 IIUC) will make it worse because __GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts with fallback will start eating the highatomic reserves too?

In that case the fix for this should probably be part of the same PR to Linus and be also stable with Fixes: 281dd25c1a01

maintainer
#21 lore raw
Quoted 6 lines · On Fri, 18 Sep 2026 09:05:40 +0200 "Vlastimil Babka (SUSE)"…
Quoted 5 lines
Quoted 2 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

Quoted 6 lines

But Salvatore's change (v4, not v5 IIUC) will make it worse because __GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts with fallback will start eating the highatomic reserves too?

In that case the fix for this should probably be part of the same PR to Linus and be also stable with Fixes: 281dd25c1a01

Oh. I thought this was settled.

I've removed "mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations" v4 from mm-hotfixes-stable and re-added it to mm-hotfixes-unstable and I'll await developments.

#22 lore raw
Quoted 12 lines · Andrew Morton
Quoted 10 lines · On Fri, 18 Sep 2026 09:05:40 +0200 "Vlastimil Babka (SUSE)"…
Quoted 5 lines
Quoted 2 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

But Salvatore's change (v4, not v5 IIUC) will make it worse because __GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts with fallback will start eating the highatomic reserves too?

Quoted 8 lines
Quoted 2 lines

In that case the fix for this should probably be part of the same PR to Linus and be also stable with Fixes: 281dd25c1a01

Oh. I thought this was settled.

I've removed "mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations" v4 from mm-hotfixes-stable and re-added it to mm-hotfixes-unstable and I'll await developments.

Since it's fixing a bug from 6.6 it would actually IMHO be better to target 7.4 and not treat it as an urgent 7.3 fix.

#23 lore raw
In reply to Vlastimil Babka (SUSE)
Quoted 39 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · Johannes Weiner
mm/page_alloc.c +2 -1
index 12fac9084c48..c79cc7aa4dff 100644
@@ -3246,7 +3246,8 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page && ((alloc_flags & ALLOC_OOM) ||
+				      ((alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC)))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
mm/page_alloc.h +0 -0
index b9259deddb59..11714ddca254 100644

mm/page_alloc.h line 68

Quoted 9 lines
mm/page_alloc.h +3 -0
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flag combination from GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*

Should work.

Quoted 6 lines
Quoted 3 lines

Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko concerns can be dismissed and then indeed v4 is the better version.

It looks like a real issue to me, but one that already exists independent of Salvatore's change.

Quoted 6 lines

But Salvatore's change (v4, not v5 IIUC) will make it worse because __GFP_DIRECT_RECLAIM + __GFP_NORETRY costly order opportunistic attempts with fallback will start eating the highatomic reserves too?

In that case the fix for this should probably be part of the same PR to Linus and be also stable with Fixes: 281dd25c1a01

Yes, agreed on both.

Sorry for the delay - while poking around I found another oddity in how the reserves are accessed.

I'll reply with two patches that I think we want to go in as fixes, whether Salvatore's patch goes ahead or not.

#24 lore raw

PATCH 1/2 mm: page_alloc: do not give all non-blocking requests reserve access

281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves") accidentally gave all non-blocking allocation requests access to highatomic reserves, which includes GFP_NOWAIT and other sites clearing __GFP_DIRECT_RECLAIM.

While this improved atomic request success rate, it's unintentionally broad and can actually worsen highatomic requests by squandering the reserves on requests that don't need it.

There are concurrent efforts to clear __GFP_DIRECT_RECLAIM altogether for costly order __GFP_NORETRY requests, which would have then also fall into this exemption and deplete reserves even faster.

What GFP_ATOMIC has over the others is __GFP_HIGH, which alloc_flags_slowpath() translates to ALLOC_MIN_RESERVE.

Narrow the exemption to contexts that have both ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE.

Fixes 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves")
---
 mm/page_alloc.c | 4 +++-
 mm/page_alloc.h | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
mm/page_alloc.c +3 -1
index 12fac9084c48..7b46d0ac3a56 100644
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
 
 			if (!page) {
mm/page_alloc.h +3 -0
index b9259deddb59..ad89f83d1dab 100644
@@ -60,6 +60,9 @@
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
+/* Flags that mean GFP_ATOMIC */
+#define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
+
 /*
  * Structure for holding the mostly immutable allocation parameters passed
  * between functions involved in allocations, including the alloc_pages*
Signature · 1 line
2.55.0
#25 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3252

Quoted 10 lines · Johannes Weiner

+++ b/mm/page_alloc.c

mm/page_alloc.c +3 -1
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

static inline bool may_access_reserves(unsigned int alloc_flags)

{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}

[...]

- if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+ if (!page && may_access_reserves(alloc_flags))
page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

#26 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 15 lines · Matthew Wilcox

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3252

Quoted 10 lines · Johannes Weiner

+++ b/mm/page_alloc.c

mm/page_alloc.c +3 -1
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

static inline bool may_access_reserves(unsigned int alloc_flags)

Quoted 8 lines
{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}

I tend to be hesitant with single-use abstractions, but no objection if people think this is better.

#27 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 15 lines · Johannes Weiner
Quoted 14 lines · Matthew Wilcox

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3252

Quoted 10 lines · Johannes Weiner

+++ b/mm/page_alloc.c

mm/page_alloc.c +3 -1
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

Quoted 9 lines
Quoted 9 lines

static inline bool may_access_reserves(unsigned int alloc_flags)

{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}

I think it should be named e.g. may_access_highatomic_reserves() as may_access_reserves() is rather generic and would seem to imply an ALLOC_RESERVES match (see 2/2).

Quoted 3 lines

I tend to be hesitant with single-use abstractions, but no objection if people think this is better.

True but single-use ALLOC_MASK_ATOMIC is also not that great, and the usage makes the code hard to decipher. And see my reply to 2/2.

#28 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 28 lines · Vlastimil Babka (SUSE)
Quoted 15 lines · Johannes Weiner
Quoted 14 lines · Matthew Wilcox

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3252

Quoted 10 lines · Johannes Weiner

+++ b/mm/page_alloc.c

mm/page_alloc.c +3 -1
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

Quoted 9 lines
Quoted 9 lines

static inline bool may_access_reserves(unsigned int alloc_flags)

{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}
Quoted 3 lines

I think it should be named e.g. may_access_highatomic_reserves() as may_access_reserves() is rather generic and would seem to imply an ALLOC_RESERVES match (see 2/2).

Note that it doesn't actually check ALLOC_HIGHATOMIC itself. It's just the hail-mary AFTER trying the primary migratetype. So the name still doesn't look right, and rmqueue_buddy() reads kind of awkardly:

if (alloc_flags & ALLOC_HIGHATOMIC)

	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);

	/* Allow OOM and order-0 atomic */
	if (!page && may_access_highatomic_reserve())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}

It would have to be may_access_highatomic_reserve_as_last_resort() or something?

Hm, this is a mess. Looking closer, I think there is more breakage, name aside.

You suggest in 2/2 to use the same helper in unusable_free. But I think that's broken. My 2/2 does not look like the full fix either:

The fundamental problem is including the highatomics reserves in the watermark check for any allocations that first prefer a different migratetype - "regular memory". Any time we do this, we allow those allocations to draw down regular memory to 0 based on the presence of the highatomic reserves. And when reclaim, swap etc. come along there is nothing left for them. ALLOC_NO_WATERMARKS e.g. permits ignoring the wmarks but doesn't grant access to highatomic *freelists*.

So I think there are two choices:

(1) Let *everything* with some sort of reserve access fall back to highatomic, or

(2) Only allow ALLOC_HIGHATOMIC to include highatomic reserves in their watermarks check. With limited opportunistic fallback to the highatomics *freelists*, like the order-0 atomics above.

My intuition is that (1) might weaken the highatomic reserves to the point of uselessness, and we should probably go with (2):

watermarks:
if (alloc_flags & ALLOC_HIGHATOMIC)

	unusable_free += zone->nr_free_highatomic

freelists:
if (alloc_flags & ALLOC_HIGHATOMIC)

	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);
	/* Opportunistic fallback for order-0 atomics */
	if (!page && opportunistic_highatomic_fallback())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}
Quoted 5 lines
Quoted 2 lines

I tend to be hesitant with single-use abstractions, but no objection if people think this is better.

True but single-use ALLOC_MASK_ATOMIC is also not that great, and the usage makes the code hard to decipher. And see my reply to 2/2.

There is one small upside, which is that it pairs with the ALLOC_HIGHATOMIC check that precedes it. The comment says "order-0 atomics" get a hail mary, but there is no order check. That order-0 comes out of the sequence of events here: we first check highatomic, which is order > 0 && atomic. If that, and the native type, fail, we do the hail mary for atomic, which must be by definition order-0.

If you abstract that privilege into a generic "can access highatomic reserves" without an order check, it tempts refactors that cause bugs like the above.

#29 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 56 lines · Johannes Weiner
Quoted 28 lines · Vlastimil Babka (SUSE)
Quoted 15 lines · Johannes Weiner
Quoted 14 lines · Matthew Wilcox

mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3252

Quoted 10 lines · Johannes Weiner

+++ b/mm/page_alloc.c

mm/page_alloc.c +3 -1
@@ -3246,7 +3246,9 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
 			 * reserves as failing now is worse than failing a
 			 * high-order atomic allocation in the future.
 			 */
-			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK)))
+			if (!page &&
+			    ((alloc_flags & ALLOC_OOM) ||
+			     (alloc_flags & ALLOC_MASK_ATOMIC) == ALLOC_MASK_ATOMIC))
 				page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);

Would this be slightly neater?

Quoted 9 lines
Quoted 9 lines

static inline bool may_access_reserves(unsigned int alloc_flags)

{
	if (alloc_flags & ALLOC_OOM)
		return true;
	if (alloc_flags & (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)) ==
	    (ALLOC_NON_BLOCK | ALLOC_MIN_RESERVE)
		return true;
	return false;
}
Quoted 3 lines

I think it should be named e.g. may_access_highatomic_reserves() as may_access_reserves() is rather generic and would seem to imply an ALLOC_RESERVES match (see 2/2).

Note that it doesn't actually check ALLOC_HIGHATOMIC itself. It's just the hail-mary AFTER trying the primary migratetype. So the name still doesn't look right, and rmqueue_buddy() reads kind of awkardly:

if (alloc_flags & ALLOC_HIGHATOMIC)

	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);

	/* Allow OOM and order-0 atomic */
	if (!page && may_access_highatomic_reserve())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}

It would have to be may_access_highatomic_reserve_as_last_resort() or something?

Hm, this is a mess. Looking closer, I think there is more breakage, name aside.

You suggest in 2/2 to use the same helper in unusable_free. But I think that's broken. My 2/2 does not look like the full fix either:

Quoted 7 lines

The fundamental problem is including the highatomics reserves in the watermark check for any allocations that first prefer a different migratetype - "regular memory". Any time we do this, we allow those allocations to draw down regular memory to 0 based on the presence of the highatomic reserves. And when reclaim, swap etc. come along there is nothing left for them. ALLOC_NO_WATERMARKS e.g. permits ignoring the wmarks but doesn't grant access to highatomic *freelists*.

Hmm yes, we'd have to be nuanced about which freelists we allow allocating from depending on their particular watermarks. I.e. if we passed the watermark checks only thanks to zone->nr_free_highatomic, we would have to try those first. I assume it would be complicated and perhaps with perf impact. But maybe if we managed to keep this complexity outside of fastpath attempts?

The CMA handling had a similar issue and now we have that balancing heuristic since 16867664936e ("mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations")

Quoted 11 lines

So I think there are two choices:

(1) Let *everything* with some sort of reserve access fall back to highatomic, or

(2) Only allow ALLOC_HIGHATOMIC to include highatomic reserves in their watermarks check. With limited opportunistic fallback to the highatomics *freelists*, like the order-0 atomics above.

My intuition is that (1) might weaken the highatomic reserves to the point of uselessness, and we should probably go with (2):

I think the goal of commit 281dd25c1a01 ("mm/page_alloc: let GFP_ATOMIC order-0 allocs access highatomic reserves") would be defeated with (2). The order-0 atomic fallback to MIGRATE_HIGHATOMIC freelists would be there, but the watermark check would never let such allocation through anymore, if there wasn't a free page on the regular freelists? So it would become (some races aside maybe) a dead code?

I guess your current patches are a compromise that can work, as patch 2/2 improves the current worse-than-(1) state to exclude the plain non-block allocations.

Quoted 27 lines

watermarks:
if (alloc_flags & ALLOC_HIGHATOMIC)

	unusable_free += zone->nr_free_highatomic

freelists:
if (alloc_flags & ALLOC_HIGHATOMIC)

	page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
if (!page) {
	page = __rmqueue(..., migratetype, ...);
	/* Opportunistic fallback for order-0 atomics */
	if (!page && opportunistic_highatomic_fallback())
		page = __rmqueue_smallest(..., MIGRATE_HIGHATOMIC);
}
Quoted 5 lines
Quoted 2 lines

I tend to be hesitant with single-use abstractions, but no objection if people think this is better.

True but single-use ALLOC_MASK_ATOMIC is also not that great, and the usage makes the code hard to decipher. And see my reply to 2/2.

There is one small upside, which is that it pairs with the ALLOC_HIGHATOMIC check that precedes it. The comment says "order-0 atomics" get a hail mary, but there is no order check. That order-0 comes out of the sequence of events here: we first check highatomic, which is order > 0 && atomic. If that, and the native type, fail, we do the hail mary for atomic, which must be by definition order-0.

Quoted 3 lines

If you abstract that privilege into a generic "can access highatomic reserves" without an order check, it tempts refactors that cause bugs like the above.

Ack.

#30 lore raw

PATCH 2/2 mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

In reply to Johannes Weiner

1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") stopped handing out reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the flag was left in ALLOC_RESERVES, which produces something odd:

With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't subtract the free highatomic pages for them. So in the slowpath, they get to consume regular blocks below the min watermark by the number of free highatomic pages. The highatomic reserve is capped at 1% of the zone, which on any decently sized machine is a multiple of the min
watermark: GFP_NOWAIT can drain regular memory to zero.

The user-visible result is brutal hiccups during bursts of GFP_NOWAIT allocations under memory pressure. On a 32G box with an anonymous working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT burst drove regular free memory in the 28G Normal zone (min=60M) to 28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap table, reclaim scanned 13M pages to reclaim 200k, page faults stalled for tens to hundreds of milliseconds. The machine survives it, but not by design: direct reclaimers eventually fail and start unreserving highatomic blocks, until the allocation succeeds or the reserve is gone and the OOM killer runs. That reserve exists for high-order atomic allocations; here it is destroyed to bail out a GFP_NOWAIT consumer that was never entitled to the memory.

Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT burst is stopped short at the min watermark. No direct reclaim, no stalls, no failed allocations, and the highatomic reserve stays intact for the requests it exists for.

__zone_watermark_ok() is unaffected, since everything it keys on ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't block; the reserve math belongs with ALLOC_MIN_RESERVE.

Fixes 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
Assisted-by LLM
---
 mm/page_alloc.h | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
mm/page_alloc.h +5 -6
index ad89f83d1dab..c8af79decbd0 100644
@@ -32,12 +32,11 @@
 #define ALLOC_OOM		ALLOC_NO_WATERMARKS
 #endif
 
-#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
-				       * to 25% of the min watermark or
-				       * 62.5% if __GFP_HIGH is set.
-				       */
+#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
 #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
-				       * of the min watermark.
+				       * of the min watermark, or 62.5% if
+				       * the caller cannot block either
+				       * (ALLOC_NON_BLOCK).
 				       */
 #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
 #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
@@ -58,7 +57,7 @@
 #define ALLOC_NO_CODETAG       0x1000
 
 /* Flags that allow allocations below the min watermark. */
-#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
+#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
 /* Flags that mean GFP_ATOMIC */
 #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
Signature · 1 line
2.55.0
#31 lore raw

Re: [PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

Quoted 13 lines · Johannes Weiner

1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") stopped handing out reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the flag was left in ALLOC_RESERVES, which produces something odd:

With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't subtract the free highatomic pages for them. So in the slowpath, they get to consume regular blocks below the min watermark by the number of free highatomic pages. The highatomic reserve is capped at 1% of the zone, which on any decently sized machine is a multiple of the min
watermark: GFP_NOWAIT can drain regular memory to zero.

Quoted 12 lines

The user-visible result is brutal hiccups during bursts of GFP_NOWAIT allocations under memory pressure. On a 32G box with an anonymous working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT burst drove regular free memory in the 28G Normal zone (min=60M) to 28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap table, reclaim scanned 13M pages to reclaim 200k, page faults stalled for tens to hundreds of milliseconds. The machine survives it, but not by design: direct reclaimers eventually fail and start unreserving highatomic blocks, until the allocation succeeds or the reserve is gone and the OOM killer runs. That reserve exists for high-order atomic allocations; here it is destroyed to bail out a GFP_NOWAIT consumer that was never entitled to the memory.

This suggests to me that a LLM review of 1/2 spotted this issue and also (or you) constructed a test doing the GFP_NOWAIT bursts to confirm the impact, but it has not been observed in production? But if it was, can we make it clear?

Quoted 10 lines

Remove ALLOC_NON_BLOCK from ALLOC_RESERVES. With that, the GFP_NOWAIT burst is stopped short at the min watermark. No direct reclaim, no stalls, no failed allocations, and the highatomic reserve stays intact for the requests it exists for.

__zone_watermark_ok() is unaffected, since everything it keys on ALLOC_NON_BLOCK is already nested under ALLOC_MIN_RESERVE. Update the flag comments accordingly: ALLOC_NON_BLOCK just means the caller can't block; the reserve math belongs with ALLOC_MIN_RESERVE.

Quoted 4 lines
Fixes 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves")
Assisted-by LLM

The change itself is fine and we don't need ALLOC_NON_BLOCK in ALLOC_RESERVES. But I wonder if we should also make the __zone_watermark_unusable_free() check more precise, by using may_access_highatomic_reserves() there instead of ALLOC_RESERVES? Which would mean that the function should however also evaluate ALLOC_HIGHATOMIC, and restrict the other checks to order=0, to be usable from both callers.

Quoted 34 lines
---
 mm/page_alloc.h | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
mm/page_alloc.h +5 -6
index ad89f83d1dab..c8af79decbd0 100644
@@ -32,12 +32,11 @@
 #define ALLOC_OOM		ALLOC_NO_WATERMARKS
 #endif
 
-#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. Allow access
-				       * to 25% of the min watermark or
-				       * 62.5% if __GFP_HIGH is set.
-				       */
+#define ALLOC_NON_BLOCK		 0x10 /* Caller cannot block. */
 #define ALLOC_MIN_RESERVE	 0x20 /* __GFP_HIGH set. Allow access to 50%
-				       * of the min watermark.
+				       * of the min watermark, or 62.5% if
+				       * the caller cannot block either
+				       * (ALLOC_NON_BLOCK).
 				       */
 #define ALLOC_CPUSET		 0x40 /* check for correct cpuset */
 #define ALLOC_CMA		 0x80 /* allow allocations from CMA areas */
@@ -58,7 +57,7 @@
 #define ALLOC_NO_CODETAG       0x1000
 
 /* Flags that allow allocations below the min watermark. */
-#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
+#define ALLOC_RESERVES (ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
 /* Flags that mean GFP_ATOMIC */
 #define ALLOC_MASK_ATOMIC (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE)
#32 lore raw

Re: [PATCH 2/2] mm: page_alloc: remove ALLOC_NON_BLOCK from ALLOC_RESERVES

Quoted 27 lines · Vlastimil Babka (SUSE)
Quoted 13 lines · Johannes Weiner

1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") stopped handing out reserve access for ALLOC_NON_BLOCK on its own: the extra 25% below the min watermark is now only granted on top of ALLOC_MIN_RESERVE. But the flag was left in ALLOC_RESERVES, which produces something odd:

With the ALLOC_RESERVES match, __zone_watermark_unusable_free() doesn't subtract the free highatomic pages for them. So in the slowpath, they get to consume regular blocks below the min watermark by the number of free highatomic pages. The highatomic reserve is capped at 1% of the zone, which on any decently sized machine is a multiple of the min
watermark: GFP_NOWAIT can drain regular memory to zero.

Quoted 12 lines

The user-visible result is brutal hiccups during bursts of GFP_NOWAIT allocations under memory pressure. On a 32G box with an anonymous working set, swap, and a filled 290M highatomic reserve, a GFP_NOWAIT burst drove regular free memory in the 28G Normal zone (min=60M) to 28M, 0.8M and 0.6M in three runs. Swapout failed to allocate its swap table, reclaim scanned 13M pages to reclaim 200k, page faults stalled for tens to hundreds of milliseconds. The machine survives it, but not by design: direct reclaimers eventually fail and start unreserving highatomic blocks, until the allocation succeeds or the reserve is gone and the OOM killer runs. That reserve exists for high-order atomic allocations; here it is destroyed to bail out a GFP_NOWAIT consumer that was never entitled to the memory.

Quoted 4 lines

This suggests to me that a LLM review of 1/2 spotted this issue and also (or you) constructed a test doing the GFP_NOWAIT bursts to confirm the impact, but it has not been observed in production? But if it was, can we make it clear?

Yes, and yes. The test is manufactured, I just pushed it to stick to a realistic sequence of events. But it wasn't observed in the wild.

Quoted 7 lines

The change itself is fine and we don't need ALLOC_NON_BLOCK in ALLOC_RESERVES. But I wonder if we should also make the __zone_watermark_unusable_free() check more precise, by using may_access_highatomic_reserves() there instead of ALLOC_RESERVES? Which would mean that the function should however also evaluate ALLOC_HIGHATOMIC, and restrict the other checks to order=0, to be usable from both callers.

I responded to this in 1/2 because it had more context.

#33 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

In reply to Johannes Weiner

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#34 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 7 lines · Salvatore Dipietro

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Quoted 7 lines

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

It would be really important to get some fix for this into 7.3. We can still perfect it later.

maintainer Acked-by Vlastimil Babka (SUSE) Acked-by Zi Yan Reviewed-by Johannes Weiner Reviewed-by Christoph Hellwig
#35 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 13 lines · On Wed, 28 Oct 2026 15:41:48 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 12 lines · Salvatore Dipietro

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

Quoted 5 lines
Quoted 2 lines

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

It would be really important to get some fix for this into 7.3. We can still perfect it later.

What is "this"? You're referring to the performance regression which Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations" addresses?

In
https://lore.kernel.org/all/[email protected]/
vbabka indicated that Fixes: 281dd25c1a01 is needed on that patch. The patch I'm sitting on (below) has Fixes: 5d8edfb900d5. Please clarify.

In
https://lore.kernel.org/all/[email protected]/
vbabka argued that Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations" should be a 7.4 thing.

So it's all clear as mud. Can people please clarify?

I have hannes's series "mm: page_alloc: fix reserve access for non-blocking requests, v2" in mm-hotfixes, planned for 7.4-rcX.

Should I move Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations, v6" into mm-hotfixes also, and restore its cc:stable?

My vote: "yes" - that's a pretty obnoxious perf regression.

From: Salvatore Dipietro <[email protected]>
Subject: mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations
Date: Thu, 1 Oct 2026 08:21:51 +0000

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. Reclaim and compaction move to the background instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the !can_direct_reclaim bailout, which since commit 4c0ed883e051 ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. The exclusion is not restricted to costly orders: any __GFP_NORETRY caller has said it can cope with failure, so failing is preferable to fragmenting for all of them.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -

  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%

---

 mm/page_alloc.c |   21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations +18 -3
@@ -4773,10 +4773,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4791,6 +4791,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, u
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * We don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
@@ -4907,8 +4919,11 @@ retry:
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

_

#36 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 21 lines · Andrew Morton
Quoted 13 lines · On Wed, 28 Oct 2026 15:41:48 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 12 lines · Salvatore Dipietro

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

Quoted 5 lines
Quoted 2 lines

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

It would be really important to get some fix for this into 7.3. We can still perfect it later.

Quoted 7 lines

What is "this"? You're referring to the performance regression which Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations" addresses?

In
https://lore.kernel.org/all/[email protected]/
vbabka indicated that Fixes: 281dd25c1a01 is needed on that patch. The

"that patch" being Johannes's and the v2 you picked up has it

Quoted 1 line

patch I'm sitting on (below) has Fixes: 5d8edfb900d5. Please clarify.

It's correct.

Quoted 7 lines

In
https://lore.kernel.org/all/[email protected]/
vbabka argued that Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations" should be a 7.4 thing.

So it's all clear as mud. Can people please clarify?

Quoted 8 lines

I have hannes's series "mm: page_alloc: fix reserve access for non-blocking requests, v2" in mm-hotfixes, planned for 7.4-rcX.

Should I move Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations, v6" into mm-hotfixes also, and restore its cc:stable?

My vote: "yes" - that's a pretty obnoxious perf regression.

Yes please.

#37 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

In reply to Andrew Morton
Quoted 6 lines · Andrew Morton
Quoted 2 lines

It would be really important to get some fix for this into 7.3. We can still perfect it later.

What is "this"? You're referring to the performance regression which Salvatore's "mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations" addresses?

As in I don't really care how the xfs/iomap performance issue gets fixed, I just want it fixed yesterday.

#38 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

In reply to Salvatore Dipietro
Quoted 12 lines · Salvatore Dipietro

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

Quoted 4 lines

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

[1] https://lore.kernel.org/linux-mm/[email protected]/

I'll work on this today, will have patches before EOD. Sorry for dropping this.

#39 lore raw

Re: [PATCH 1/2] mm: page_alloc: do not give all non-blocking requests reserve access

Quoted 13 lines · Johannes Weiner
Quoted 12 lines · Salvatore Dipietro

Hi Johannes,

Vlastimil asked that v6 of my page_alloc patch [1] not go to mainline before these two patches land, so Andrew parked v6 at the tail of mm-unstable and dropped its Cc: stable until then.

Are you still planning to work on 1/2 and 2/2, and if so, do you have a rough ETA? Or would you rather post them as a separate series as they are? If anything is needed on my side to move them forward, I'm happy to help.

Quoted 7 lines
Quoted 4 lines

For what it's worth, I ran both stacked under v6 on v7.3-rc1 and saw no performance difference on my benchmark with or without them.

[1] https://lore.kernel.org/linux-mm/[email protected]/

I'll work on this today, will have patches before EOD. Sorry for dropping this.

https://lore.kernel.org/all/[email protected]/