Skip to contentSkip to conversation

mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

PATCH Superseded Salvatore Dipietro 5 participants 10 messages Sep 11 – Oct 1

Subsystems (4)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, suppress direct reclaim for the whole slowpath by computing can_direct_reclaim (and hence can_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. __GFP_NOFAIL is exempted as well, so a must-not-fail allocation is never made to fail.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%

The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.

---
v5: Derive the decision into a local flag instead of mutating gfp_mask
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
mm/page_alloc.c +14 -2
index 12fac9084c48..be8b0d72a2db 100644
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);
 	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
 	bool nofail = gfp_mask & __GFP_NOFAIL;
-	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
Signature · 6 lines
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
reviewer Acked-by on v5 1/1
#1 lore raw
Quoted 72 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, suppress direct reclaim for the whole slowpath by computing can_direct_reclaim (and hence can_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. __GFP_NOFAIL is exempted as well, so a must-not-fail allocation is never made to fail.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%

The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.

Quoted 8 lines

---
v5: Derive the decision into a local flag instead of mutating gfp_mask
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

LGTM.

Best Regards,
Yan, Zi

maintainer
#2 lore raw
In reply to Salvatore Dipietro
Quoted 44 lines · On Fri, 11 Sep 2026 14:21:02 +0000 Salvatore Dipietro <[email protected]> wrote:

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, suppress direct reclaim for the whole slowpath by computing can_direct_reclaim (and hence can_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. __GFP_NOFAIL is exempted as well, so a must-not-fail allocation is never made to fail.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%

Quoted 6 lines

The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.

Thanks, I'll update mm.git's mm-hotfixes-unstable branch with this.

Quoted 1 line

v5: Derive the decision into a local flag instead of mutating gfp_mask

The acks from vbabka, hannes and hch were dropped. Fair enough - that's always a hard call.

Sashiko is worried:

	https://sashiko.dev/#/patchset/[email protected]

That's different from Sashiko's v4 complaints:

	https://sashiko.dev/#/patchset/[email protected]

does any of this look real?

#3 lore raw
Quoted 1 line · On Fri, 11 Sep 2026 09:32:00 -0700 Andrew Morton <[email protected]>…

does any of this look real?

The mechanism seems real, but it only shows up with vm.defrag_mode=1, which is off by default.

With defrag_mode=1, a costly __GFP_NORETRY allocation that used to fail cleanly can now succeed by fragmenting another migratetype's pageblock, which seems like the wrong trade for a caller whose premise is a cheap lower-order fallback. Johannes, do we want to keep it strict? If so, we could add a condition like below. It is equally applicable to v4 and v5. Happy to send it separately or in v6.

	/*
	 * Reclaim/compaction cannot run, so defrag_mode's strategy
	 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
	 * fallbacks rather than failing the allocation outright.
	 * Not for costly __GFP_NORETRY: those have a cheap lower
	 * order fallback, so failing beats fragmenting.
	 */
	if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
	    !(costly_order && (gfp_mask & __GFP_NORETRY)) &&
	    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
		alloc_flags &= ~ALLOC_NOFRAGMENT;
		goto retry;
	}

Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#4 lore raw
Quoted 5 lines · Salvatore Dipietro
Quoted 1 line · On Fri, 11 Sep 2026 09:32:00 -0700 Andrew Morton <[email protected]>…

does any of this look real?

Quoted 7 lines

The mechanism seems real, but it only shows up with vm.defrag_mode=1, which is off by default.

With defrag_mode=1, a costly __GFP_NORETRY allocation that used to fail cleanly can now succeed by fragmenting another migratetype's pageblock, which seems like the wrong trade for a caller whose premise is a cheap lower-order fallback. Johannes, do we want to keep it strict?

Yes, avoiding the fallback is a good idea. The user already communicated they can handle failure and have fallback options.

Quoted 12 lines

If so, we could add a condition like below. It is equally applicable to v4 and v5. Happy to send it separately or in v6.

/*
* Reclaim/compaction cannot run, so defrag_mode's strategy
* of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
* fallbacks rather than failing the allocation outright.
* Not for costly __GFP_NORETRY: those have a cheap lower
* order fallback, so failing beats fragmenting. */
if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
!(costly_order && (gfp_mask & __GFP_NORETRY)) &&

It would be possible to remove the costly_order gate here and just bail on __GFP_NORETRY. But no strong feelings.

#5 lore raw
Quoted 2 lines · Johannes Weiner

It would be possible to remove the costly_order gate here and just bail on __GFP_NORETRY. But no strong feelings.

Will do, I'll drop the costly_order gate and bail on __GFP_NORETRY.

Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#6 lore raw
Quoted 10 lines · Salvatore Dipietro
Quoted 2 lines · Johannes Weiner

It would be possible to remove the costly_order gate here and just bail on __GFP_NORETRY. But no strong feelings.

Will do, I'll drop the costly_order gate and bail on __GFP_NORETRY.

Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.

v4 is in mm-hotfixes-stable now, so an incremental fix will be easier now I think.

Quoted 10 lines

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

maintainer
#7 lore raw
Quoted 6 lines · On Fri, 18 Sep 2026 08:40:05 +0200 "Vlastimil Babka (SUSE)"…
Quoted 5 lines · Salvatore Dipietro
Quoted 2 lines · Johannes Weiner

It would be possible to remove the costly_order gate here and just bail on __GFP_NORETRY. But no strong feelings.

Quoted 9 lines
Quoted 5 lines

Will do, I'll drop the costly_order gate and bail on __GFP_NORETRY.

Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.

v4 is in mm-hotfixes-stable now, so an incremental fix will be easier now I think.

I just demoted it into mm-hotfixes-unstable.

So a clean v6 would be best now, please.

#8 lore raw
Quoted 1 line · On Fri, 18 Sep 2026 14:35:00 -0700 Andrew Morton <[email protected]>…

So a clean v6 would be best now, please.

Done, v6 is here:

https://lore.kernel.org/linux-mm/[email protected]/T/#u

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

#9 lore raw
In reply to Salvatore Dipietro
Quoted 72 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, suppress direct reclaim for the whole slowpath by computing can_direct_reclaim (and hence can_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.

Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. __GFP_NOFAIL is exempted as well, so a must-not-fail allocation is never made to fail.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%

The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.

Quoted 2 lines

---
v5: Derive the decision into a local flag instead of mutating gfp_mask

I commented on the v5 variant here:

https://lore.kernel.org/all/[email protected]/

I think v4 was better.