mm/page_alloc: drop flag-conversion "optimisation"
Subsystems (2)
PATCH v2 1/1 mm/page_alloc: drop flag-conversion "optimisation"
This code uses flag equivalences to try to optimise conversion from GFP_ to ALLOC_ but there's no clear reason to believe it makes things faster. Even if it gets rid of conditional branches, it just trades them for a data dependency.
CPUs are pretty good at conditional branches. But, in my GCC x86 build it doesn't look like there are any branches anyway, the compiler found some conditional instruction tricks. (Caveat: This was extracted & annotated by Gemini AI, I did not actually read the disasm myself)
Old code:
ae50: 8b 04 24 mov (%rsp),%eax # Load gfp_mask
...
ae5d: 41 89 c4 mov %eax,%r12d
ae64: 41 81 e4 20 08 00 00 and $0x820,%r12d # Mask both flags at once
...
ae6f: 44 89 e1 mov %r12d,%ecx
ae77: 83 c9 40 or $0x40,%ecx # OR with ALLOC_CPUSET (0x40)
ae7a: 89 4c 24 60 mov %ecx,0x60(%rsp) # Store to alloc_flagsNew code:
For __GFP_HIGH ( 0x20 ):
It uses the Carry Flag (via sbb ) to conditionally add 0x20 to the base 0x40 ( ALLOC_CPUSET ) flag:
ae63: 83 e0 20 and $0x20,%eax # Test __GFP_HIGH
...
ae6a: 83 f8 01 cmp $0x1,%eax # Set carry flag if 0
ae6f: 45 19 e4 sbb %r12d,%r12d # %r12d = (gfp & 0x20) ? 0 : -1
ae80: 41 83 e4 e0 and $0xffffffe0,%r12d # %r12d = (gfp & 0x20) ? 0 : -32
ae87: 41 83 c4 60 add $0x60,%r12d # %r12d = (gfp & 0x20) ? 0x60 : 0x40 For __GFP_KSWAPD_RECLAIM ( 0x800 ):
It uses a conditional move ( cmov ) later in the function to set the ALLOC_KSWAPD ( 0x800 ) bit:
ae72: 25 00 08 00 00 and $0x800,%eax # Test __GFP_KSWAPD_RECLAIM
ae77: 89 44 24 30 mov %eax,0x30(%rsp) # Store result ...
af2c: 80 cf 08 or $0x8,%bh # Set ALLOC_KSWAPD (0x800) in temp reg
af2f: 45 85 c9 test %r9d,%r9d # Check if __GFP_KSWAPD_RECLAIM was set
af32: 0f 44 d8 cmove %eax,%ebx # If not, revert to flags without it
Testing with a modified version[0] of lib/free_pages_test.c (adding printks with timing)...
Old results from a Sapphire Rapids consumer CPU:
[ 67.157118] page_alloc_test: Testing with GFP_KERNEL [ 67.157122] page_alloc_test: Starting 1,000,000 allocations... [ 70.704446] page_alloc_test: Completed. Time: 3543002 us (Avg: 3543.00 ns per alloc+free loop) [ 70.704456] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 70.704460] page_alloc_test: Starting 1,000,000 allocations... [ 70.944672] page_alloc_test: Completed. Time: 239980 us (Avg: 239.98 ns per alloc+free loop) [ 70.944675] page_alloc_test: Test completed
New results:
[ 70.079015] page_alloc_test: Testing with GFP_KERNEL [ 70.079020] page_alloc_test: Starting 1,000,000 allocations... [ 73.669396] page_alloc_test: Completed. Time: 3586954 us (Avg: 3586.95 ns per alloc+free loop) [ 73.669402] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 73.669405] page_alloc_test: Starting 1,000,000 allocations... [ 73.905084] page_alloc_test: Completed. Time: 235496 us (Avg: 235.49 ns per alloc+free loop) [ 73.905086] page_alloc_test: Test completed
Seems like a wash.
So, drop the flag value coupling here and let the compiler and CPU do their job. Superscalar CPUs are pretty neat after all.
(Used AI for the disasm but the rest is all manual).
---
Changes in v2:
- Updated alloc_flags_nofragment() too.
- Link to v1: https://lore.kernel.org/r/[email protected]
--- mm/page_alloc.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-)
mm/page_alloc.c +7 -16
@@ -3739,13 +3739,10 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)static inline unsigned int alloc_flags_nofragment (struct zone *zone,gfp_t gfp_mask) { -unsigned int alloc_flags; +unsigned int alloc_flags =0 ; -/* -* __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD -* to save a branch. -*/ - alloc_flags = (__force int ) (gfp_mask & __GFP_KSWAPD_RECLAIM); +if (gfp_mask & __GFP_KSWAPD_RECLAIM) + alloc_flags |=ALLOC_KSWAPD ;if (defrag_mode) { alloc_flags |=ALLOC_NOFRAGMENT ;
@@ -4478,22 +4475,16 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) {unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ; -/* -* __GFP_HIGH is assumed to be the same as ALLOC_MIN_RESERVE -* and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD -* to save two branches. -*/ -BUILD_BUG_ON (__GFP_HIGH != (__force gfp_t )ALLOC_MIN_RESERVE ); -BUILD_BUG_ON (__GFP_KSWAPD_RECLAIM != (__force gfp_t )ALLOC_KSWAPD ); -/* * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ - alloc_flags |= (__force int ) - (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); +if (gfp_mask & __GFP_HIGH) + alloc_flags |=ALLOC_MIN_RESERVE ; +if (gfp_mask & __GFP_KSWAPD_RECLAIM) + alloc_flags |=ALLOC_KSWAPD ;if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {/*
---
base-commit: ca2351ac6da277a470d4fcf122b53267e02b2716
change-id: 20260612-gfp-pessimisation-b258a4bb5ebd
Best regards,
Signature · 1 line
Brendan Jackman <[email protected]>
Brendan Jackman
reviewer Reviewed-by Vlastimil Babka (SUSE) Reviewed-by Zi Yan Reviewed-by Gregory Price Acked-by Johannes Weiner Acked-by Harry Yoo (Oracle) on v2 1/1Quoted 6 lines · Brendan Jackman
Signed-off-by Brendan Jackman <[email protected]>---
Changes in v2:
- Updatedalloc_flags_nofragment()too.
- Link to v1: https://lore.kernel.org/r/[email protected]
Sigh, maybe one day I'll send a patch without immediately following up with a "oops, I forgot to ...". But today is not that day.
Forgot to add these tags from the v1:
Thanks everyone for the prompt reviews.
Brendan Jackman
reviewerQuoted 13 lines · Brendan Jackman
Quoted 7 lines
Signed-off-by Brendan Jackman <[email protected]>---
Changes in v2:
- Updatedalloc_flags_nofragment()too.
- Link to v1: https://lore.kernel.org/r/[email protected]Sigh, maybe one day I'll send a patch without immediately following up with a "oops, I forgot to ...". But today is not that day.
Forgot to add these tags from the v1:
Quoted 7 lines
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Gregory Price <[email protected]>Acked-by Johannes Weiner <[email protected]>Thanks everyone for the prompt reviews.
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
Zi Yan
reviewerQuoted 20 lines · Brendan Jackman
Quoted 19 lines · Brendan Jackman
Quoted 7 lines
Signed-off-by Brendan Jackman <[email protected]>---
Changes in v2:
- Updatedalloc_flags_nofragment()too.
- Link to v1: https://lore.kernel.org/r/[email protected]Sigh, maybe one day I'll send a patch without immediately following up with a "oops, I forgot to ...". But today is not that day.
Forgot to add these tags from the v1:
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Gregory Price <[email protected]>Acked-by Johannes Weiner <[email protected]>
Quoted 6 lines
Quoted 1 line
Thanks everyone for the prompt reviews.
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
It is in the quiet period, no patch will be picked up until -rc1 is out. If yours is not picked up then, feel free to resend it.
Signature · 2 lines
Best Regards, Yan, Zi
Brendan Jackman
reviewerQuoted 23 lines · Zi Yan
Quoted 22 lines · Brendan Jackman
Quoted 20 lines · Brendan Jackman
Quoted 7 lines
Signed-off-by Brendan Jackman <[email protected]>---
Changes in v2:
- Updatedalloc_flags_nofragment()too.
- Link to v1: https://lore.kernel.org/r/[email protected]Sigh, maybe one day I'll send a patch without immediately following up with a "oops, I forgot to ...". But today is not that day.
Forgot to add these tags from the v1:
Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Gregory Price <[email protected]>Acked-by Johannes Weiner <[email protected]>Thanks everyone for the prompt reviews.
Quoted 7 lines
Quoted 4 lines
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
It is in the quiet period, no patch will be picked up until -rc1 is out. If yours is not picked up then, feel free to resend it.
Ah, I thought mm-unstable didn't care about merge windows. It got updated 2 days ago, but now I realise that's probably just coz Andrew was updating mm-stable and then rebased onto it?
Thanks for clarifying.
Quoted 8 lines · Brendan Jackman
Quoted 7 lines · Zi Yan
Quoted 5 lines · Brendan Jackman
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
Quoted 8 lines
Quoted 2 lines
It is in the quiet period, no patch will be picked up until -rc1 is out. If yours is not picked up then, feel free to resend it.
Ah, I thought mm-unstable didn't care about merge windows. It got updated 2 days ago, but now I realise that's probably just coz Andrew was updating mm-stable and then rebased onto it?
Thanks for clarifying.
We're targeting for mm-unstable to only have material that is already upstream at the end of the merge window.
Signature · 3 lines
Cheers, David
Andrew Morton
maintainerQuoted 12 lines · On Fri, 19 Jun 2026 16:53:41 +0200 "David Hildenbrand (Arm)"…
Quoted 11 lines · Brendan Jackman
Quoted 9 lines · Zi Yan
Quoted 5 lines · Brendan Jackman
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
It is in the quiet period, no patch will be picked up until -rc1 is out. If yours is not picked up then, feel free to resend it.
Quoted 8 lines
Quoted 5 lines
Ah, I thought mm-unstable didn't care about merge windows. It got updated 2 days ago, but now I realise that's probably just coz Andrew was updating mm-stable and then rebased onto it?
Thanks for clarifying.
We're targeting for mm-unstable to only have material that is already upstream at the end of the merge window.
What they all said ;)
Officially (and for good reasons) we aren't supposed to add new material to linux-next until -rc1 is released.
mm.git has the mm-new branch which isn't included in linux-next so after MM is all merged up I start working through the backlog which I accumulated during quiet-time and merging some of it into mm-new.
Backlog was ~1700 emails this time. Most of which gets skipped for various reasons,
This particular patch looks nice to me. Code is cleaner and the change reduces page_alloc.o's .text (x86_64 defconfig) from 46571 to 46443, which is a good sign. So I'll add it, along with a "needs more review" note.
Thanks.
Zi Yan
reviewerQuoted 32 lines · Andrew Morton
Quoted 12 lines · On Fri, 19 Jun 2026 16:53:41 +0200 "David Hildenbrand (Arm)"…
Quoted 11 lines · Brendan Jackman
Quoted 9 lines · Zi Yan
Quoted 5 lines · Brendan Jackman
Hi Andrew,
This one didn't make it into any of the latest mm-* branches, is anything blocked here?
It is in the quiet period, no patch will be picked up until -rc1 is out. If yours is not picked up then, feel free to resend it.
Quoted 8 lines
Quoted 5 lines
Ah, I thought mm-unstable didn't care about merge windows. It got updated 2 days ago, but now I realise that's probably just coz Andrew was updating mm-stable and then rebased onto it?
Thanks for clarifying.
We're targeting for mm-unstable to only have material that is already upstream at the end of the merge window.
What they all said ;)
Officially (and for good reasons) we aren't supposed to add new material to linux-next until -rc1 is released.
mm.git has the mm-new branch which isn't included in linux-next so after MM is all merged up I start working through the backlog which I accumulated during quiet-time and merging some of it into mm-new.
Quoted 7 lines
Backlog was ~1700 emails this time. Most of which gets skipped for various reasons,
This particular patch looks nice to me. Code is cleaner and the change reduces page_alloc.o's .text (
x86_64defconfig) from 46571 to 46443, which is a good sign. So I'll add it, along with a "needs more review" note.
It is actually well reviewed[1], but Brendan forgot to add tags in the original patch.
[1] https://lore.kernel.org/all/[email protected]/
Signature · 2 lines
Best Regards, Yan, Zi
Andrew Morton
maintainerQuoted 9 lines · On Thu, 25 Jun 2026 19:02:43 -0400 "Zi Yan" <[email protected]> wrote:
Quoted 4 lines
This particular patch looks nice to me. Code is cleaner and the change reduces page_alloc.o's .text (
x86_64defconfig) from 46571 to 46443, which is a good sign. So I'll add it, along with a "needs more review" note.It is actually well reviewed[1], but Brendan forgot to add tags in the original patch.
Yep, thanks, I saw. I added your r-b: because you anticipated the alloc_flags_nofragment() change.