mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations
Subsystems (4)
PATCH v5 1/1 mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations
Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.
For costly-order allocations with __GFP_NORETRY, suppress direct reclaim for the whole slowpath by computing can_direct_reclaim (and hence can_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.
Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. __GFP_NOFAIL is exempted as well, so a must-not-fail allocation is never made to fail.
Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)
12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200s
Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%
The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.
---
v5: Derive the decision into a local flag instead of mutating gfp_mask
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
__GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation
mm/page_alloc.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
mm/page_alloc.c +14 -2
@@ -4784,10 +4784,22 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct alloc_context *ac) { - bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; + const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; +/* +* Costly __GFP_NORETRY callers have a cheap fallback to a lower order, +* so don't stall them in direct reclaim or direct compaction. Exempt +* __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct +* compaction) and __GFP_NOFAIL (must not be made to fail). Don't +* clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also +* change the alloc_flags derived by alloc_flags_slowpath(). +*/ +const bool costly_noretry = costly_order && + (gfp_mask & __GFP_NORETRY) && + !(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL)); +bool can_direct_reclaim = !costly_noretry && + (gfp_mask & __GFP_DIRECT_RECLAIM);bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask);bool nofail = gfp_mask & __GFP_NOFAIL; -const bool costly_order = order >PAGE_ALLOC_COSTLY_ORDER ;struct page *page =NULL ;unsigned int alloc_flags;unsigned long did_some_progress;
Signature · 6 lines
2.50.1 AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Quoted 72 lines · Salvatore Dipietro
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.For costly-order allocations with
__GFP_NORETRY, suppress direct reclaim for the whole slowpath by computingcan_direct_reclaim(and hencecan_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and nodrain_all_pages()IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.Allocations that also request
__GFP_THISNODEare exempted. That flag pairing identifies the local-node-first THP attempt issued byalloc_pages_mpol()(mempolicy.c), which relies on direct compaction to form transparent huge pages.__GFP_NOFAILis exempted as well, so a must-not-fail allocation is never made to fail.Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200sResults (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit
5d8edfb900d5("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc David Hildenbrand <[email protected]>Cc Michal Hocko <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dave Chinner <[email protected]>Cc Ritesh Harjani <[email protected]>Cc Zi Yan <[email protected]>Signed-off-by Salvatore Dipietro <[email protected]>
Quoted 8 lines
---
v5: Derive the decision into a local flag instead of mutatinggfp_mask
v4: Clear__GFP_DIRECT_RECLAIMearly in the slowpath and exempt
__GFP_THISNODEso THP attempt keeps using direct compaction
v3: Move tomm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move fromfs/iomap/buffered-io.ctomm/filemap.c
v1: Avoid compaction in iomap folio allocation
LGTM.
Best Regards,
Yan, Zi
Andrew Morton
maintainerQuoted 44 lines · On Fri, 11 Sep 2026 14:21:02 +0000 Salvatore Dipietro <[email protected]> wrote:
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.For costly-order allocations with
__GFP_NORETRY, suppress direct reclaim for the whole slowpath by computingcan_direct_reclaim(and hencecan_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and nodrain_all_pages()IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.Allocations that also request
__GFP_THISNODEare exempted. That flag pairing identifies the local-node-first THP attempt issued byalloc_pages_mpol()(mempolicy.c), which relies on direct compaction to form transparent huge pages.__GFP_NOFAILis exempted as well, so a must-not-fail allocation is never made to fail.Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200sResults (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%
Quoted 6 lines
The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit
5d8edfb900d5("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.
Thanks, I'll update mm.git's mm-hotfixes-unstable branch with this.
Quoted 1 line
v5: Derive the decision into a local flag instead of mutating
gfp_mask
The acks from vbabka, hannes and hch were dropped. Fair enough - that's always a hard call.
Sashiko is worried:
https://sashiko.dev/#/patchset/[email protected]
That's different from Sashiko's v4 complaints:
https://sashiko.dev/#/patchset/[email protected]
does any of this look real?
Quoted 1 line · On Fri, 11 Sep 2026 09:32:00 -0700 Andrew Morton <[email protected]>…
does any of this look real?
The mechanism seems real, but it only shows up with vm.defrag_mode=1, which is off by default.
With defrag_mode=1, a costly __GFP_NORETRY allocation that used to fail cleanly can now succeed by fragmenting another migratetype's pageblock, which seems like the wrong trade for a caller whose premise is a cheap lower-order fallback. Johannes, do we want to keep it strict? If so, we could add a condition like below. It is equally applicable to v4 and v5. Happy to send it separately or in v6.
/*
* Reclaim/compaction cannot run, so defrag_mode's strategy
* of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
* fallbacks rather than failing the allocation outright.
* Not for costly __GFP_NORETRY: those have a cheap lower
* order fallback, so failing beats fragmenting.
*/
if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
!(costly_order && (gfp_mask & __GFP_NORETRY)) &&
(gfp_mask & __GFP_KSWAPD_RECLAIM)) {
alloc_flags &= ~ALLOC_NOFRAGMENT;
goto retry;
}Salvatore
AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Johannes Weiner
reviewerQuoted 5 lines · Salvatore Dipietro
Quoted 1 line · On Fri, 11 Sep 2026 09:32:00 -0700 Andrew Morton <[email protected]>…
does any of this look real?
Quoted 7 lines
The mechanism seems real, but it only shows up with vm.defrag_mode=1, which is off by default.
With defrag_mode=1, a costly
__GFP_NORETRYallocation that used to fail cleanly can now succeed by fragmenting another migratetype's pageblock, which seems like the wrong trade for a caller whose premise is a cheap lower-order fallback. Johannes, do we want to keep it strict?
Yes, avoiding the fallback is a good idea. The user already communicated they can handle failure and have fallback options.
Quoted 12 lines
If so, we could add a condition like below. It is equally applicable to v4 and v5. Happy to send it separately or in v6.
/*
* Reclaim/compaction cannot run, so defrag_mode's strategy
* of enforcingALLOC_NOFRAGMENTcannot be fulfilled. Allow
* fallbacks rather than failing the allocation outright.
* Not for costly__GFP_NORETRY: those have a cheap lower
* order fallback, so failing beats fragmenting. */
if (defrag_mode&& (alloc_flags&ALLOC_NOFRAGMENT) &&
!(costly_order && (gfp_mask&__GFP_NORETRY)) &&
It would be possible to remove the costly_order gate here and just bail on __GFP_NORETRY. But no strong feelings.
Quoted 2 lines · Johannes Weiner
It would be possible to remove the
costly_ordergate here and just bail on__GFP_NORETRY. But no strong feelings.
Will do, I'll drop the costly_order gate and bail on __GFP_NORETRY.
Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.
Thanks,
Salvatore
AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Vlastimil Babka (SUSE)
maintainerQuoted 10 lines · Salvatore Dipietro
Quoted 2 lines · Johannes Weiner
It would be possible to remove the
costly_ordergate here and just bail on__GFP_NORETRY. But no strong feelings.Will do, I'll drop the
costly_ordergate and bail on__GFP_NORETRY.Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.
v4 is in mm-hotfixes-stable now, so an incremental fix will be easier now I think.
Quoted 10 lines
Thanks,
SalvatoreAMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Andrew Morton
maintainerQuoted 6 lines · On Fri, 18 Sep 2026 08:40:05 +0200 "Vlastimil Babka (SUSE)"…
Quoted 5 lines · Salvatore Dipietro
Quoted 2 lines · Johannes Weiner
It would be possible to remove the
costly_ordergate here and just bail on__GFP_NORETRY. But no strong feelings.
Quoted 9 lines
Quoted 5 lines
Will do, I'll drop the
costly_ordergate and bail on__GFP_NORETRY.Andrew, since you dropped v5 and restored v4 in mm-unstable, I'll send a v6 (v4 plus the above change) unless you'd rather have an incremental patch on top of v4. Let me know if anything else is needed on my side.
v4 is in mm-hotfixes-stable now, so an incremental fix will be easier now I think.
I just demoted it into mm-hotfixes-unstable.
So a clean v6 would be best now, please.
Quoted 1 line · On Fri, 18 Sep 2026 14:35:00 -0700 Andrew Morton <[email protected]>…
So a clean v6 would be best now, please.
Done, v6 is here:
https://lore.kernel.org/linux-mm/[email protected]/T/#u
Thanks,
Salvatore
AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Johannes Weiner
reviewerQuoted 72 lines · Salvatore Dipietro
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.For costly-order allocations with
__GFP_NORETRY, suppress direct reclaim for the whole slowpath by computingcan_direct_reclaim(and hencecan_compact) as false. The !can_direct_reclaim check near the top of the slowpath then short-circuits to nopage: no direct reclaim, no direct compaction and nodrain_all_pages()IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path.Allocations that also request
__GFP_THISNODEare exempted. That flag pairing identifies the local-node-first THP attempt issued byalloc_pages_mpol()(mempolicy.c), which relies on direct compaction to form transparent huge pages.__GFP_NOFAILis exempted as well, so a must-not-fail allocation is never made to fail.Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200sResults (average of 3 runs, TPS):
Config Avg TPS % vs baseline
AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a
v7.3-rc1 baseline (no patch) 59,408 -
v7.3-rc1 + this patch 161,994 +172.7%The patch fully recovers the pre-5d8edfb900d5 performance, bringing throughput back above the pre-regression level and well clear of the ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit
5d8edfb900d5("iomap: Copy larger chunks from userspace"), so it is not directly comparable, but it shows the pre-regression level and confirms that the ~59k baseline is the anomaly and not the norm.Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc David Hildenbrand <[email protected]>Cc Michal Hocko <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dave Chinner <[email protected]>Cc Ritesh Harjani <[email protected]>Cc Zi Yan <[email protected]>Signed-off-by Salvatore Dipietro <[email protected]>
Quoted 2 lines
---
v5: Derive the decision into a local flag instead of mutatinggfp_mask
I commented on the v5 variant here:
https://lore.kernel.org/all/[email protected]/
I think v4 was better.