mm/page_alloc: use existing highatomic reserves on the buddy fastpath
Subsystems (3)
PATCH v2 1/1 mm/page_alloc: use existing highatomic reserves on the buddy fastpath
ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.
However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC reserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.
Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing MIGRATE_HIGHATOMIC reserves on the buddy fastpath. Change the semantics of ALLOC_HIGHATOMIC so that it only allows access to the reserves without permission to grow them. Add a new flag ALLOC_HIGHATOMIC_RESERVE that specifically allows growing the reserves.
A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.
---
v2:
- decouple use semantics from ALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1
v1: https://lore.kernel.org/linux-mm/[email protected]/
mm/internal.h | 1 + mm/page_alloc.c | 30 +++++++++++++++++++++++++----- 2 files changed, 26 insertions(+), 5 deletions(-)
mm/internal.h +1 -0
@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_HIGHATOMIC_RESERVE 0x1000/* Allows growing MIGRATE_HIGHATOMIC reserves */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
mm/page_alloc.c +25 -5
@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ -if (unlikely (alloc_flags &ALLOC_HIGHATOMIC )) +if (unlikely (alloc_flags &ALLOC_HIGHATOMIC_RESERVE ))reserve_highatomic_pageblock (page, order, zone);__count_zid_vm_events (PGALLOC ,page_zonenum (page),1 << order);
@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,* * Instead, direct it towards the reserves by * returning NULL, which will make the caller fall -* back to rmqueue_buddy. This will try to use the -* reserves first and grow them if needed. +* back to rmqueue_buddy. There it will try to use +* the reserves first and grow them if needed and +* permitted by the ALLOC_HIGHATOMIC_RESERVE flag. */ if (alloc_flags &ALLOC_HIGHATOMIC )return NULL ;
@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)return alloc_flags; } +/* +* Let high-priority non-blocking allocations above order-0 and up +* to the costly order try to use existing MIGRATE_HIGHATOMIC +* reserves on the fastpath. +*/ +static inline unsigned int +alloc_flags_highatomic_fastpath (gfp_t gfp_mask,unsigned int order) +{ +if (!order || order >PAGE_ALLOC_COSTLY_ORDER ) +return 0 ; +if (!(gfp_mask & __GFP_HIGH)) +return 0 ; +if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC)) +return 0 ; + +return ALLOC_HIGHATOMIC ; +} +/* Must be called after current_gfp_context() which can change gfp_mask */ static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask,unsigned int alloc_flags)
@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) alloc_flags |=ALLOC_NON_BLOCK ;if (order >0 && (alloc_flags &ALLOC_MIN_RESERVE )) - alloc_flags |=ALLOC_HIGHATOMIC ; + alloc_flags |= (ALLOC_HIGHATOMIC |ALLOC_HIGHATOMIC_RESERVE ); }/*
@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,* Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ - alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); + alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp) | +alloc_flags_highatomic_fastpath (alloc_gfp, order);/* First allocation attempt */ page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac);
Signature · 1 line
2.54.0
Johannes Weiner
reviewerQuoted 14 lines · JP Kobryn
ALLOC_HIGHATOMICcurrently provides both access toMIGRATE_HIGHATOMICfree pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.However, the fastpath can reach
rmqueue_buddy()whileMIGRATE_HIGHATOMICreserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing
MIGRATE_HIGHATOMICreserves on the buddy fastpath. Change the semantics ofALLOC_HIGHATOMICso that it only allows access to the reserves without permission to grow them. Add a new flagALLOC_HIGHATOMIC_RESERVEthat specifically allows growing the reserves.
Quoted 7 lines
A UDP receive workload was run with free
MIGRATE_HIGHATOMICpageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existingMIGRATE_HIGHATOMICpageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.
Thanks for digging deeper into this! That's a great find.
Quoted 32 lines
Signed-off-by JP Kobryn <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>---
v2:
- decouple use semantics fromALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1v1: https://lore.kernel.org/linux-mm/[email protected]/
mm/internal.h | 1 + mm/page_alloc.c | 30 +++++++++++++++++++++++++----- 2 files changed, 26 insertions(+), 5 deletions(-)mm/internal.h +1
-0@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_HIGHATOMIC_RESERVE 0x1000/* Allows growing MIGRATE_HIGHATOMIC reserves */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)mm/page_alloc.c +0
-0
mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3242
Quoted 6 lines
mm/page_alloc.c +1
-1@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ -if (unlikely (alloc_flags &ALLOC_HIGHATOMIC )) +if (unlikely (alloc_flags &ALLOC_HIGHATOMIC_RESERVE ))reserve_highatomic_pageblock (page, order, zone);
You could check ALLOC_WMARK_MIN to determine the slowpath. This way you wouldn't need another alloc flag:
/* Slowpath (precarious) high-atomic allocation. Maybe reserve block */ if (unlikely((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)) == (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN))) reserve_highatomic_pageblock(page, order, zone);
[ we really ought to generalize gfp_has_flags() ... ]
Quoted 17 lines
mm/page_alloc.c +3
-2__count_zid_vm_events (PGALLOC ,page_zonenum (page),1 << order);@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,* * Instead, direct it towards the reserves by * returning NULL, which will make the caller fall -* back to rmqueue_buddy. This will try to use the -* reserves first and grow them if needed. +* back to rmqueue_buddy. There it will try to use +* the reserves first and grow them if needed and +* permitted by the ALLOC_HIGHATOMIC_RESERVE flag. */ if (alloc_flags &ALLOC_HIGHATOMIC )return NULL ;@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)return alloc_flags; }
mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3781
Quoted 10 lines
mm/page_alloc.c +10
-0+/* +* Let high-priority non-blocking allocations above order-0 and up +* to the costly order try to use existing MIGRATE_HIGHATOMIC +* reserves on the fastpath. +*/ +static inline unsigned int +alloc_flags_highatomic_fastpath (gfp_t gfp_mask,unsigned int order) +{ +if (!order || order >PAGE_ALLOC_COSTLY_ORDER ) +return 0 ;
There seems to be a mismatch between this and gfp_to_alloc_flags() (slowpath), where slowpath is still allowed to tap highatomic reserves for costly orders. Is that on purpose?
mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3785
Quoted 4 lines
mm/page_alloc.c +4
-0+if (!(gfp_mask & __GFP_HIGH)) +return 0 ; +if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC)) +return 0 ;
This duplicates gfp_to_alloc_flags() logic which seems fragile. How about the below:
Quoted 17 lines
mm/page_alloc.c +5
-1+ +return ALLOC_HIGHATOMIC ; +} +/* Must be called after current_gfp_context() which can change gfp_mask */ static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask,unsigned int alloc_flags)@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) alloc_flags |=ALLOC_NON_BLOCK ;if (order >0 && (alloc_flags &ALLOC_MIN_RESERVE )) - alloc_flags |=ALLOC_HIGHATOMIC ; + alloc_flags |= (ALLOC_HIGHATOMIC |ALLOC_HIGHATOMIC_RESERVE ); }/*
mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5238
Quoted 6 lines
mm/page_alloc.c +2
-1@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,* Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ - alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); + alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp) | +alloc_flags_highatomic_fastpath (alloc_gfp, order);
alloc_flags |= gfp_to_alloc_flags(gfp, order) & ALLOC_HIGHATOMIC;
Or factor gfp_to_alloc_flags_nonblocking() from gfp_to_alloc_flags() and reuse that here, to save a few cycles in the fast path.
Quoted 4 lines
mm/page_alloc.c +0
-0/* First allocation attempt */ page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac);Signature · 1 line
2.54.0
Quoted 23 lines · Johannes Weiner
Quoted 14 lines · JP Kobryn
ALLOC_HIGHATOMICcurrently provides both access toMIGRATE_HIGHATOMICfree pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath.However, the fastpath can reach
rmqueue_buddy()whileMIGRATE_HIGHATOMICreserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first.Allow high-priority non-blocking allocations above order-0 and up to the costly order to use existing
MIGRATE_HIGHATOMICreserves on the buddy fastpath. Change the semantics ofALLOC_HIGHATOMICso that it only allows access to the reserves without permission to grow them. Add a new flagALLOC_HIGHATOMIC_RESERVEthat specifically allows growing the reserves.Quoted 7 lines
A UDP receive workload was run with free
MIGRATE_HIGHATOMICpageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existingMIGRATE_HIGHATOMICpageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks.
Quoted 1 line
Thanks for digging deeper into this! That's a great find.
Thanks :)
Quoted 43 lines
Quoted 33 lines
Signed-off-by JP Kobryn <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>---
v2:
- decouple use semantics fromALLOC_HIGHATOMIC_RESERVE
- update changelog to reflect above change and reword test paragraph
- adjust comment in PCP path
- rebase onto Linus' tree ~v7.2-rc1v1: https://lore.kernel.org/linux-mm/[email protected]/
mm/internal.h | 1 + mm/page_alloc.c | 30 +++++++++++++++++++++++++----- 2 files changed, 26 insertions(+), 5 deletions(-)mm/internal.h +1
-0@@ -1478,6 +1478,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400/* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#define ALLOC_HIGHATOMIC_RESERVE 0x1000/* Allows growing MIGRATE_HIGHATOMIC reserves */ /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)mm/page_alloc.c +0
-0mm/page_alloc.c struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, · line 3242
Quoted 6 lines
mm/page_alloc.c +1
-1@@ -3238,7 +3238,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ -if (unlikely (alloc_flags &ALLOC_HIGHATOMIC )) +if (unlikely (alloc_flags &ALLOC_HIGHATOMIC_RESERVE ))reserve_highatomic_pageblock (page, order, zone);You could check
ALLOC_WMARK_MINto determine the slowpath. This way you wouldn't need another alloc flag:
Quoted 3 lines
/* Slowpath (precarious) high-atomic allocation. Maybe reserve block */ if (unlikely((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN)) == (ALLOC_HIGHATOMIC|ALLOC_WMARK_MIN))) reserve_highatomic_pageblock(page, order, zone);
Hmm so we just use watermark low vs min to determine fast/slow path. We would need to mask out the watermark bits since ALLOC_WMARK_MIN is zero, but I think the idea can work.
Quoted 20 lines
[ we really ought to generalize
gfp_has_flags()... ]Quoted 17 lines
mm/page_alloc.c +3
-2__count_zid_vm_events (PGALLOC ,page_zonenum (page),1 << order);@@ -3320,8 +3320,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order,* * Instead, direct it towards the reserves by * returning NULL, which will make the caller fall -* back to rmqueue_buddy. This will try to use the -* reserves first and grow them if needed. +* back to rmqueue_buddy. There it will try to use +* the reserves first and grow them if needed and +* permitted by the ALLOC_HIGHATOMIC_RESERVE flag. */ if (alloc_flags &ALLOC_HIGHATOMIC )return NULL ;@@ -3768,6 +3769,24 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)return alloc_flags; }
Quoted 14 lines
mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3781
Quoted 10 lines
mm/page_alloc.c +10
-0+/* +* Let high-priority non-blocking allocations above order-0 and up +* to the costly order try to use existing MIGRATE_HIGHATOMIC +* reserves on the fastpath. +*/ +static inline unsigned int +alloc_flags_highatomic_fastpath (gfp_t gfp_mask,unsigned int order) +{ +if (!order || order >PAGE_ALLOC_COSTLY_ORDER ) +return 0 ;There seems to be a mismatch between this and
gfp_to_alloc_flags()(slowpath), where slowpath is still allowed to tap highatomic reserves for costly orders. Is that on purpose?
Yes, the intention was to improve allocator outcomes for frequent net allocations. So I kept the policy narrow. But I thought about this some more and agree it's worth removing the order cap. With the cap, if a costly+ order atomic allocation reaches the buddy path there may be two non-ideal outcomes: fallback to some other migratetype in the fastpath or enter the slowpath and make use of any available reserves. In the latter case, the slowpath could have been avoided altogether. So I'll make this change in v3.
Quoted 35 lines
mm/page_alloc.c alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) · line 3785
Quoted 5 lines
mm/page_alloc.c +4
-0+if (!(gfp_mask & __GFP_HIGH)) +return 0 ; +if (gfp_mask & (__GFP_DIRECT_RECLAIM | __GFP_NOMEMALLOC)) +return 0 ;This duplicates
gfp_to_alloc_flags()logic which seems fragile. How about the below:Quoted 17 lines
mm/page_alloc.c +5
-1+ +return ALLOC_HIGHATOMIC ; +} +/* Must be called after current_gfp_context() which can change gfp_mask */ static inline unsigned int gfp_to_alloc_flags_cma (gfp_t gfp_mask,unsigned int alloc_flags)@@ -4495,7 +4514,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) alloc_flags |=ALLOC_NON_BLOCK ;if (order >0 && (alloc_flags &ALLOC_MIN_RESERVE )) - alloc_flags |=ALLOC_HIGHATOMIC ; + alloc_flags |= (ALLOC_HIGHATOMIC |ALLOC_HIGHATOMIC_RESERVE ); }/* mm/page_alloc.c struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, · line 5238
Quoted 6 lines
mm/page_alloc.c +2
-1@@ -5215,7 +5234,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,* Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. */ - alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp); + alloc_flags |=alloc_flags_nofragment (zonelist_zone (ac.preferred_zoneref), gfp) | +alloc_flags_highatomic_fastpath (alloc_gfp, order);alloc_flags |= gfp_to_alloc_flags(gfp, order) & ALLOC_HIGHATOMIC;
Quoted 2 lines
Or factor
gfp_to_alloc_flags_nonblocking()fromgfp_to_alloc_flags()and reuse that here, to save a few cycles in the fast path.
This should be possible now that I'll be removing the order cap.
Quoted 5 lines
Quoted 5 lines
mm/page_alloc.c +0
-0/* First allocation attempt */ page =get_page_from_freelist (alloc_gfp, order, alloc_flags, &ac);Signature · 1 line
2.54.0