mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations
Subsystems (4)
PATCH v6 1/1 mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations
Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct reclaim, direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.
For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.
Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages.
Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the !can_direct_reclaim bailout, which since commit 4c0ed883e051 ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on. With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude __GFP_NORETRY from the relaxation. The exclusion is not restricted to costly orders: any __GFP_NORETRY caller has said it can cope with failure, so failing is preferable to fragmenting for all of them.
Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)
12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200s
Results (average of 3 runs, TPS):
Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -
v7.3-rc1 + this patch (vm.defrag_mode=0) 157,246 +168.3% v7.3-rc1 baseline (vm.defrag_mode=1) 109,012 - v7.3-rc1 + this patch (vm.defrag_mode=1) 157,778 +44.7%
--- mm/page_alloc.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +18 -3
@@ -4784,10 +4784,10 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct alloc_context *ac) { - bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; -bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); -bool nofail = gfp_mask & __GFP_NOFAIL; const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; +bool can_direct_reclaim; +bool can_compact; +bool nofail;struct page *page =NULL ;unsigned int alloc_flags;unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,bool can_retry_reserves = true;unsigned long alloc_start_time = jiffies; +/* +* Costly __GFP_NORETRY callers have a cheap fallback, so don't stall +* them in reclaim or compaction. __GFP_THISNODE callers are exempt. +*/ +if (costly_order && (gfp_mask & __GFP_NORETRY) && + !(gfp_mask & __GFP_THISNODE)) + gfp_mask &= ~__GFP_DIRECT_RECLAIM; + + can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; + can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); + nofail = gfp_mask & __GFP_NOFAIL; +if (unlikely (nofail)) {/* * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* Reclaim/compaction cannot run, so defrag_mode's strategy * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow * fallbacks rather than failing the allocation outright. +* Not for __GFP_NORETRY: those have a cheap lower order +* fallback, so failing beats fragmenting. */ if (defrag_mode && (alloc_flags &ALLOC_NOFRAGMENT ) && + !(gfp_mask & __GFP_NORETRY) && (gfp_mask & __GFP_KSWAPD_RECLAIM)) { alloc_flags &= ~ALLOC_NOFRAGMENT ; goto retry;
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
Signature · 6 lines
2.50.1 AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Andrew Morton
maintainerQuoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct reclaim, direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
Quoted 9 lines
The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.
Thanks, I'll update mm-hotfixes to this version.
Here's what changed:
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations +3 -0
@@ -4930,8 +4930,11 @@ retry:* Reclaim/compaction cannot run, so defrag_mode's strategy * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow * fallbacks rather than failing the allocation outright. +* Not for __GFP_NORETRY: those have a cheap lower order +* fallback, so failing beats fragmenting. */ if (defrag_mode && (alloc_flags &ALLOC_NOFRAGMENT ) && + !(gfp_mask & __GFP_NORETRY) && (gfp_mask & __GFP_KSWAPD_RECLAIM)) { alloc_flags &= ~ALLOC_NOFRAGMENT ; goto retry;
_
Vlastimil Babka (SUSE)
maintainerQuoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct reclaim, direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
Quoted 9 lines
Quoted 9 lines
The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.
Since it expands the impact of a pre-existing bug of nonblocking allocation eating reserves [1] it shouldn't proceed to mainline without a fix for that (Johannes posted it there but the review discussion got stalled).
Quoted 1 line
Thanks, I'll update mm-hotfixes to this version.
I've also argued for not handling it as a hotfix [2]
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 18 lines
Here's what changed:
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations +3
-0@@ -4930,8 +4930,11 @@ retry:* Reclaim/compaction cannot run, so defrag_mode's strategy * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow * fallbacks rather than failing the allocation outright. +* Not for __GFP_NORETRY: those have a cheap lower order +* fallback, so failing beats fragmenting. */ if (defrag_mode && (alloc_flags &ALLOC_NOFRAGMENT ) && + !(gfp_mask & __GFP_NORETRY) && (gfp_mask & __GFP_KSWAPD_RECLAIM)) { alloc_flags &= ~ALLOC_NOFRAGMENT ; goto retry;_
Andrew Morton
maintainerQuoted 25 lines · On Fri, 2 Oct 2026 10:23:47 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <[email protected]> wrote:
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct reclaim, direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.Quoted 9 lines
Quoted 9 lines
The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.Since it expands the impact of a pre-existing bug of nonblocking allocation eating reserves [1] it shouldn't proceed to mainline without a fix for that (Johannes posted it there but the review discussion got stalled).
Quoted 8 lines
Quoted 1 line
Thanks, I'll update mm-hotfixes to this version.
I've also argued for not handling it as a hotfix [2]
Thanks. I moved this to tail of mm-unstable, removed its cc:stable and added a note indicating that changes are expected.
Quoted 118 lines · Salvatore Dipietro
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct reclaim, direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.For costly-order allocations with
__GFP_NORETRY, clear__GFP_DIRECT_RECLAIMat the very start of the slowpath, beforecan_direct_reclaim,can_compactand the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and nodrain_all_pages()IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.Allocations that also request
__GFP_THISNODEare exempted. That flag pairing identifies the local-node-first THP attempt issued byalloc_pages_mpol()(mempolicy.c), which relies on direct compaction to form transparent huge pages.Clearing
__GFP_DIRECT_RECLAIMmakes these allocations reach the !can_direct_reclaim bailout, which since commit4c0ed883e051("mm/page_alloc: fixdefrag_modefor non-reclaimable allocations") dropsALLOC_NOFRAGMENTand retries instead of failing whendefrag_modeis on. With vm.defrag_mode=1 a__GFP_NORETRYallocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude__GFP_NORETRYfrom the relaxation. The exclusion is not restricted to costly orders: any__GFP_NORETRYcaller has said it can cope with failure, so failing is preferable to fragmenting for all of them.Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200sResults (average of 3 runs, TPS):
Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -v7.3-rc1 + this patch (vm.defrag_mode=0) 157,246 +168.3% v7.3-rc1 baseline (vm.defrag_mode=1) 109,012 - v7.3-rc1 + this patch (vm.defrag_mode=1) 157,778 +44.7%Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc David Hildenbrand <[email protected]>Cc Michal Hocko <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dave Chinner <[email protected]>Cc Ritesh Harjani <[email protected]>Cc Zi Yan <[email protected]>Cc Dmitry Ilvokhin <[email protected]>Acked-by Zi Yan <[email protected]>Reviewed-by Johannes Weiner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Salvatore Dipietro <[email protected]>--- mm/page_alloc.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)mm/page_alloc.c +8
-3@@ -4784,10 +4784,10 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct alloc_context *ac) { - bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; -bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); -bool nofail = gfp_mask & __GFP_NOFAIL; const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; +bool can_direct_reclaim; +bool can_compact; +bool nofail;struct page *page =NULL ;unsigned int alloc_flags;unsigned long did_some_progress;@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,bool can_retry_reserves = true;unsigned long alloc_start_time = jiffies; +/* +* Costly __GFP_NORETRY callers have a cheap fallback, so don't stall +* them in reclaim or compaction. __GFP_THISNODE callers are exempt. +*/ + if (costly_order && (gfp_mask & __GFP_NORETRY) &&
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4819
Quoted 10 lines
mm/page_alloc.c +7
-0+ !(gfp_mask & __GFP_THISNODE)) + gfp_mask &= ~__GFP_DIRECT_RECLAIM; + + can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; + can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); + nofail = gfp_mask & __GFP_NOFAIL; +if (unlikely (nofail)) {/* * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
This would trigger a warning if __GFP_NOFAIL and __GFP_NORETRY were ever used together, though that does not make sense and I haven't found a case where it is used in practice.
Aside from either warning for the specific case of these flags being used together or adding a comment right here to that effect, I don't think any other change is needed, though.
Another comment about the general approach: TTM pool page allocator does use __GFP_NORETRY and __GFP_DIRECT_RECLAIM for high orders and when we previously suggested to drop __GFP_DIRECT_RECLAIM to avoid compaction, it was frowned upon. In the end, a lot of other changes were made and for the relevant cases, this does not seem to change its current behavior, amongst other things, because it uses __GFP_THISNODE.
I did run some TTM GTT allocations with some fragmentation workload and I haven't seen much change in the latency for these high order allocations, so:
Quoted 24 lines
mm/page_alloc.c +3
-0@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* Reclaim/compaction cannot run, so defrag_mode's strategy * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow * fallbacks rather than failing the allocation outright. +* Not for __GFP_NORETRY: those have a cheap lower order +* fallback, so failing beats fragmenting. */ if (defrag_mode && (alloc_flags &ALLOC_NOFRAGMENT ) && + !(gfp_mask & __GFP_NORETRY) && (gfp_mask & __GFP_KSWAPD_RECLAIM)) { alloc_flags &= ~ALLOC_NOFRAGMENT ; goto retry;base-commit:
cee9395acd8043be0644b25c34bfa86623f2b935Signature · 6 lines
2.50.1 AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
Vlastimil Babka (SUSE)
maintainerQuoted 125 lines · Thadeu Lima de Souza Cascardo
Quoted 124 lines · Salvatore Dipietro
Commit
5d8edfb900d5("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters__alloc_pages_slowpath()which runs direct reclaim, direct compaction anddrain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.The root issue is that direct reclaim and direct compaction are too expensive for hot allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol()already marks higher-order allocations with__GFP_NORETRY|__GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still enters the full blocking slowpath for costly orders with__GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order.For costly-order allocations with
__GFP_NORETRY, clear__GFP_DIRECT_RECLAIMat the very start of the slowpath, beforecan_direct_reclaim,can_compactand the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and nodrain_all_pages()IPI across every CPU. Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for defragmentation. The system keeps its long-term health while the latency-critical direct allocation path no longer stalls.Allocations that also request
__GFP_THISNODEare exempted. That flag pairing identifies the local-node-first THP attempt issued byalloc_pages_mpol()(mempolicy.c), which relies on direct compaction to form transparent huge pages.Clearing
__GFP_DIRECT_RECLAIMmakes these allocations reach the !can_direct_reclaim bailout, which since commit4c0ed883e051("mm/page_alloc: fixdefrag_modefor non-reclaimable allocations") dropsALLOC_NOFRAGMENTand retries instead of failing whendefrag_modeis on. With vm.defrag_mode=1 a__GFP_NORETRYallocation could therefore succeed by fragmenting another migratetype's pageblock. That is the wrong trade for a caller that has already communicated it can handle failure and has a cheap lower-order fallback, so exclude__GFP_NORETRYfrom the relaxation. The exclusion is not restricted to costly orders: any__GFP_NORETRYcaller has said it can cope with failure, so failing is preferable to fragmenting for all of them.Test environment:
Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64)12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200sResults (average of 3 runs, TPS):
Config Avg TPS % vs baseline
v7.3-rc1 baseline (vm.defrag_mode=0) 58,602 -v7.3-rc1 + this patch (vm.defrag_mode=0) 157,246 +168.3% v7.3-rc1 baseline (vm.defrag_mode=1) 109,012 - v7.3-rc1 + this patch (vm.defrag_mode=1) 157,778 +44.7%Fixes 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc David Hildenbrand <[email protected]>Cc Michal Hocko <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Dave Chinner <[email protected]>Cc Ritesh Harjani <[email protected]>Cc Zi Yan <[email protected]>Cc Dmitry Ilvokhin <[email protected]>Acked-by Zi Yan <[email protected]>Reviewed-by Johannes Weiner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Salvatore Dipietro <[email protected]>--- mm/page_alloc.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)mm/page_alloc.c +14
-3@@ -4784,10 +4784,10 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct alloc_context *ac) { - bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; -bool can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); -bool nofail = gfp_mask & __GFP_NOFAIL; const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; +bool can_direct_reclaim; +bool can_compact; +bool nofail;struct page *page =NULL ;unsigned int alloc_flags;unsigned long did_some_progress;@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,bool can_retry_reserves = true;unsigned long alloc_start_time = jiffies; +/* +* Costly __GFP_NORETRY callers have a cheap fallback, so don't stall +* them in reclaim or compaction. __GFP_THISNODE callers are exempt. +*/ +if (costly_order && (gfp_mask & __GFP_NORETRY) && + !(gfp_mask & __GFP_THISNODE)) + gfp_mask &= ~__GFP_DIRECT_RECLAIM; + + can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; + can_compact = can_direct_reclaim &&gfp_compaction_allowed (gfp_mask); + nofail = gfp_mask & __GFP_NOFAIL;
Quoted 8 lines
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4819
Quoted 4 lines
mm/page_alloc.c +1
-0+if (unlikely (nofail)) {/* * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM, This would trigger a warning if
__GFP_NOFAILand__GFP_NORETRYwere ever used together, though that does not make sense and I haven't found a case where it is used in practice.
Right. But always there is some risk someone does that unintentionally by trying to turn an allocation with incoming gfp flags (including unexpected __GFP_NOFAIL) to an opportunistic one first by appending __GFP_NORETRY, etc.
Quoted 3 lines
Aside from either warning for the specific case of these flags being used together or adding a comment right here to that effect, I don't think any other change is needed, though.
Indeed. I thought we did have such warnings. Also a combination with __GFP_RETRY_MAYFAIL would make no sense. So we should perhaps add them.
Quoted 7 lines
Another comment about the general approach: TTM pool page allocator does use
__GFP_NORETRYand__GFP_DIRECT_RECLAIMfor high orders and when we previously suggested to drop__GFP_DIRECT_RECLAIMto avoid compaction, it was frowned upon. In the end, a lot of other changes were made and for the relevant cases, this does not seem to change its current behavior, amongst other things, because it uses__GFP_THISNODE.
Quoted 5 lines
I did run some TTM GTT allocations with some fragmentation workload and I haven't seen much change in the latency for these high order allocations, so:
Thanks!
Quoted 24 lines
Quoted 24 lines
mm/page_alloc.c +3
-0@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* Reclaim/compaction cannot run, so defrag_mode's strategy * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow * fallbacks rather than failing the allocation outright. +* Not for __GFP_NORETRY: those have a cheap lower order +* fallback, so failing beats fragmenting. */ if (defrag_mode && (alloc_flags &ALLOC_NOFRAGMENT ) && + !(gfp_mask & __GFP_NORETRY) && (gfp_mask & __GFP_KSWAPD_RECLAIM)) { alloc_flags &= ~ALLOC_NOFRAGMENT ; goto retry;base-commit:
cee9395acd8043be0644b25c34bfa86623f2b935Signature · 6 lines
2.50.1 AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico