mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.45x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, skip direct compaction but wake kcompactd on the preferred node so that background compaction can defragment memory for future allocations, and return failure immediately so the caller can fall back.

This keeps compaction working for long-term system health while removing it from the latency-critical direct allocation path.

Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)

             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    next-20260707
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

Config Avg TPS % vs Baseline
baseline (no patch) 70,389.24 -
With this patch 154,977.02 +120.17%

Link https://lore.kernel.org/all/[email protected]/T/#t [v1] open here

Link https://lore.kernel.org/linux-mm/[email protected]/T/#u [v2] open here

Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")

Cc [email protected]

Signed-off-by Salvatore Dipietro <[email protected]>

---
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

mm/page_alloc.c +2 -0

index a63733dac659..2d02703d8f0f 100644
@@ -4883,6 +4883,26 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	/* If allocation has taken excessively long, warn about it */
 	check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
 
+	/*
+	 * Costly allocations with __GFP_NORETRY are opportunistic - Don't

mm/page_alloc.c +3 -0

+		goto nopage;
+	}
+
 	/* Try direct reclaim and then allocating */
 	if (!compact_first) {
 		page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,

Signature · 1 line

2.47.3

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path.

Hi Johannes, Matthew,

Thank you both for the alternative proposals. I've tested both approaches on the same test environment used for v3.

Results (4 runs each):

Config Avg TPS % vs Baseline
--------------------------------------------------------
baseline (no patch) 70,735 -
Johannes' approach 156,908 +121.8%
Matthew's approach 70,145 -0.8% (within noise)

Johannes' approach (clearing __GFP_DIRECT_RECLAIM early in the slowpath for costly __GFP_NORETRY) delivers the same ~2.2x speedup as v3, as expected - it prevents the entire direct reclaim and compaction machinery from running for these opportunistic allocations.

Matthew's filemap.c approach does not help in this workload. The reason is that the first allocation attempt at max order still carries __GFP_DIRECT_RECLAIM and enters the slowpath with direct compaction enabled. The __GFP_DIRECT_RECLAIM clearing only takes effect for subsequent lower-order attempts in the fallback loop, but the costly compaction has already executed on the first try.

Let me know if you have any other variant you want me to test, or if I should prepare a v4 based on Johannes' suggestion.

This is what I tested for Johannes' approach:

mm/page_alloc.c +17 -3

index a63733dac659..6e960c969e67 100644
@@ -4733,10 +4733,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 					struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;

@@ -4751,6 +4751,20 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY allocations are opportunistic: the caller
+	 * can fall back to smaller orders.  Don't stall on direct reclaim
+	 * or compaction; clearing __GFP_DIRECT_RECLAIM makes the entire
+	 * slowpath treat this as a non-blocking request.  kswapd will wake
+	 * kcompactd as needed for background defragmentation.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

Thanks,
Salvatore

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

AFAICS both your v3 and the implemented Johannes' suggestion effectively make all the compact_first logic dead code - at least for the intended THP use case based on costly_order. It will only continue to do something for the non-movable order>0 case. So that's a significant change which will not show up in your results, but might affect other workloads (I'd expect mainly by having fewer THPs).

Quoted 6 lines · Vlastimil Babka (SUSE)

AFAICS both your v3 and the implemented Johannes' suggestion effectively make all the compact_first logic dead code - at least for the intended THP use case based on costly_order. It will only continue to do something for the non-movable order>0 case. So that's a significant change which will not show up in your results, but might affect other workloads (I'd expect mainly by having fewer THPs).

Right - the change above modifies THP allocation requests as well. So instead of applying it to every costly __GFP_NORETRY allocation, I restrict it to callers that do not ask for __GFP_THISNODE. That leaves the local-node THP attempt untouched: the same bit pair is already tested further down the function to bail out of the compact_first block (page_alloc.c:4899), and alloc_pages_mpol() sets both flags on the local-node-first attempt (mempolicy.c:2476-2478).

I tested this with my PostgreSQL workload on kernel v7.2-rc7 and results look good:

Baseline:       70,735 tps
This version:  144,670 tps  (145,280 / 144,037 / 144,694)  +104.5%

This looks really nice. Can you turn this into a formal patch?

Quoted 3 lines · Salvatore Dipietro

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path.

https://lore.kernel.org/linux-mm/[email protected]/

I just had a go at implementing what I thought might be the right design (having a folio_alloc_orders(min, max, gfp)), but that's not really what __filemap_get_folio_mpol() wants because it needs to integrate the actual adding of folios to the page cache into the retry loop.

So instead, let's try this. The idea is that we want to try direct reclaim _twice_. Once gently (ie with NORETRY specified) when we're trying to allocate the maximum order folio. But now that we've tried that once, there's no point trying direct reclaim for other sizes, we just want to ask the page allocator if it can give us memory of any subsequent size.

Until we come to the minimum order. Then we want to try exactly as hard as we were originally asked to try. So revert to the original gfp flags and don't set the NOWARN or NORETRY flags.

mm/filemap.c +0 -0

index 58eb9d240643..23eecaf9b328 100644

mm/filemap.c +3 -4

@@ -2004,12 +2005,11 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 			order = __ffs(index);
 
 		do {
-			gfp_t alloc_gfp = gfp;
-
 			err = -ENOMEM;
-			if (order > min_order)
-				alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
+			if (order == min_order)
+				alloc_gfp = gfp;
 			folio = filemap_alloc_folio(alloc_gfp, order, policy);
+			alloc_gfp &= ~__GFP_DIRECT_RECLAIM;
 			if (!folio)
 				continue;