Discard __GFP_ATOMIC
Subsystems (11)
v3 cover Discard __GFP_ATOMIC
This replaces the "Discard __GFP_ATOMIC v2" series in mm-unstable. There are changelog and patch replacements that make -fix patches impractical.
Changelog since v2
o Non-blocking (GFP_NOWAIT) allocations get no reserve access (mhocko)
o __GFP_NOFAIL before OOM reserve access reduced (mhocko)
o Changelog clarifications (mhocko)
o Note that rt_task treatment to be deleted in changelog (mhocko)
o One ack dropped as the patch changed enough to invalidate it
Changelog since v1
o Split one patch (vbabka)
o Improve OOM reserve handling (vbabka)
o Fix __GFP_RECLAIM vs __GFP_DIRECT_RECLAIM (vbabka)
Neil's patch has been residing in mm-unstable as commit 2fafb4fe8f7a ("mm: discard __GFP_ATOMIC") for a long time and recently brought up again. Most recently, I was worried that __GFP_HIGH allocations could use high-order atomic reserves which is unintentional but there was no response so lets revisit -- this series reworks how min reserves are used, protects highorder reserves and then finishes with Neil's patch with very minor modifications so it fits on top.
There was a review discussion on renaming __GFP_DIRECT_RECLAIM to __GFP_ALLOW_BLOCKING but I didn't think it was that big an issue and is ortogonal to the removal of __GFP_ATOMIC.
There were some concerns about how the gfp flags affect the min reserves but it never reached a solid conclusion so I made my own attempt.
The series tries to iron out some of the details on how reserves are used. ALLOC_HIGH becomes ALLOC_MIN_RESERVE and ALLOC_HARDER becomes ALLOC_NON_BLOCK and documents how the reserves are affected. For example, ALLOC_NON_BLOCK (no direct reclaim) on its own allows 25% of the min reserve. ALLOC_MIN_RESERVE (__GFP_HIGH) allows 50% and both combined allows deeper access again. ALLOC_OOM allows access to 75%.
High-order atomic allocations are explicitly handled with the caveat that no __GFP_ATOMIC flag means that any high-order allocation that specifies GFP_HIGH and cannot enter direct reclaim will be treated as if it was GFP_ATOMIC.
Documentation/mm/balance.rst | 2 +- drivers/iommu/tegra-smmu.c | 4 +- include/linux/gfp_types.h | 12 ++-- include/trace/events/mmflags.h | 1 - lib/test_printf.c | 8 +-- mm/internal.h | 15 ++++- mm/page_alloc.c | 106 ++++++++++++++++++++------------- tools/perf/builtin-kmem.c | 1 - 8 files changed, 88 insertions(+), 61 deletions(-)
Signature · 1 line
2.35.3
PATCH v3 1/6 mm/page_alloc: Rename ALLOC_HIGH to ALLOC_MIN_RESERVE
__GFP_HIGH aliases to ALLOC_HIGH but the name does not really hint what it means. As ALLOC_HIGH is internal to the allocator, rename it to ALLOC_MIN_RESERVE to document that the min reserves can be depleted.
--- mm/internal.h | 4 +++- mm/page_alloc.c | 8 ++++---- 2 files changed, 7 insertions(+), 5 deletions(-)
mm/internal.h +3 -1
@@ -736,7 +736,9 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#endif #define ALLOC_HARDER 0x10/* try to alloc harder */ -#define ALLOC_HIGH 0x20/* __GFP_HIGH set */ +#define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% +* of the min watermark. +*/ #define ALLOC_CPUSET 0x40/* check for correct cpuset */ #define ALLOC_CMA 0x80/* allow allocations from CMA areas */ #ifdef CONFIG_ZONE_DMA32
mm/page_alloc.c +4 -4
@@ -3976,7 +3976,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,/* free_pages may go negative - that's OK */ free_pages -=__zone_watermark_unusable_free (z, order, alloc_flags); -if (alloc_flags &ALLOC_HIGH ) +if (alloc_flags &ALLOC_MIN_RESERVE ) min -= min /2 ;if (unlikely (alloc_harder)) {
@@ -4818,18 +4818,18 @@ gfp_to_alloc_flags(gfp_t gfp_mask)unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ;/* -* __GFP_HIGH is assumed to be the same as ALLOC_HIGH +* __GFP_HIGH is assumed to be the same as ALLOC_MIN_RESERVE * and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD * to save two branches. */ -BUILD_BUG_ON (__GFP_HIGH != (__force gfp_t )ALLOC_HIGH ); +BUILD_BUG_ON (__GFP_HIGH != (__force gfp_t )ALLOC_MIN_RESERVE );BUILD_BUG_ON (__GFP_KSWAPD_RECLAIM != (__force gfp_t )ALLOC_KSWAPD );/* * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_HIGH (__GFP_HIGH). +* set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int ) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM));
Signature · 1 line
2.35.3
PATCH v3 2/6 mm/page_alloc: Treat RT tasks similar to __GFP_HIGH
In reply to Mel GormanRT tasks are allowed to dip below the min reserve but ALLOC_HARDER is typically combined with ALLOC_MIN_RESERVE so RT tasks are a little unusual. While there is some justification for allowing RT tasks access to memory reserves, there is a strong chance that a RT task that is also under memory pressure is at risk of missing deadlines anyway. Relax how much reserves an RT task can access by treating it the same as __GFP_HIGH allocations.
Note that in a future kernel release that the RT special casing will be removed. Hard realtime tasks should be locking down resources in advance and ensuring enough memory is available. Even a soft-realtime task like audio or video live decoding which cannot jitter should be allocating both memory and any disk space required up-front before the recording starts instead of relying on reserves. At best, reserve access will only delay the problem by a very short interval.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -4847,7 +4847,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)*/ alloc_flags &= ~ALLOC_CPUSET ; }else if (unlikely (rt_task (current)) &&in_task ()) - alloc_flags |=ALLOC_HARDER ; + alloc_flags |=ALLOC_MIN_RESERVE ; alloc_flags =gfp_to_alloc_flags_cma (gfp_mask, alloc_flags);
Signature · 1 line
2.35.3
PATCH v3 3/6 mm/page_alloc: Explicitly record high-order atomic allocations in alloc_flags
In reply to Mel GormanA high-order ALLOC_HARDER allocation is assumed to be atomic. While that is accurate, it changes later in the series. In preparation, explicitly record high-order atomic allocations in gfp_to_alloc_flags().
--- mm/internal.h | 1 + mm/page_alloc.c | 29 +++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)
mm/internal.h +1 -0
@@ -746,6 +746,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#else #define ALLOC_NOFRAGMENT 0x0#endif +#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ enum ttu_flags ;
mm/page_alloc.c +23 -6
@@ -3706,10 +3706,20 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* reserved for high-order atomic allocation, so order-0 * request should skip it. */ -if (order >0 && alloc_flags &ALLOC_HARDER ) +if (alloc_flags &ALLOC_HIGHATOMIC ) page =__rmqueue_smallest (zone, order,MIGRATE_HIGHATOMIC );if (!page) { page =__rmqueue (zone, order, migratetype, alloc_flags); + +/* +* If the allocation fails, allow OOM handling access +* to HIGHATOMIC reserves as failing now is worse than +* failing a high-order atomic allocation in the +* future. +*/ +if (!page && (alloc_flags &ALLOC_OOM )) + page =__rmqueue_smallest (zone, order,MIGRATE_HIGHATOMIC ); +if (!page) {spin_unlock_irqrestore (&zone->lock, flags);return NULL ;
@@ -4023,8 +4033,10 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, return true; }#endif -if (alloc_harder && !free_area_empty (area,MIGRATE_HIGHATOMIC )) +if ((alloc_flags & (ALLOC_HIGHATOMIC |ALLOC_OOM )) && + !free_area_empty (area,MIGRATE_HIGHATOMIC )) {return true; + } }return false; }
@@ -4286,7 +4298,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,* If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ -if (unlikely (order && (alloc_flags &ALLOC_HARDER ))) +if (unlikely (alloc_flags &ALLOC_HIGHATOMIC ))reserve_highatomic_pageblock (page, zone, order);return page;
@@ -4813,7 +4825,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, }static inline unsigned int -gfp_to_alloc_flags (gfp_t gfp_mask) +gfp_to_alloc_flags (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ;
@@ -4839,8 +4851,13 @@ gfp_to_alloc_flags(gfp_t gfp_mask)* Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ -if (!(gfp_mask & __GFP_NOMEMALLOC)) +if (!(gfp_mask & __GFP_NOMEMALLOC)) { alloc_flags |=ALLOC_HARDER ; + +if (order >0 ) + alloc_flags |=ALLOC_HIGHATOMIC ; + } +/* * Ignore cpuset mems for GFP_ATOMIC rather than fail, see the * comment for __cpuset_node_allowed().
@@ -5048,7 +5065,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask); + alloc_flags =gfp_to_alloc_flags (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator
Signature · 1 line
2.35.3
Re: [PATCH 3/6] mm/page_alloc: Explicitly record high-order atomic allocations in alloc_flags
Quoted 6 lines · Mel Gorman
A high-order
ALLOC_HARDERallocation is assumed to be atomic. While that is accurate, it changes later in the series. In preparation, explicitly record high-order atomic allocations ingfp_to_alloc_flags().Signed-off-by Mel Gorman <[email protected]>Acked-by Vlastimil Babka <[email protected]>
Quoted 99 lines
--- mm/internal.h | 1 + mm/page_alloc.c | 29 +++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)mm/internal.h +1
-0@@ -746,6 +746,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#else #define ALLOC_NOFRAGMENT 0x0#endif +#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ enum ttu_flags ;mm/page_alloc.c +23
-6@@ -3706,10 +3706,20 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,* reserved for high-order atomic allocation, so order-0 * request should skip it. */ -if (order >0 && alloc_flags &ALLOC_HARDER ) +if (alloc_flags &ALLOC_HIGHATOMIC ) page =__rmqueue_smallest (zone, order,MIGRATE_HIGHATOMIC );if (!page) { page =__rmqueue (zone, order, migratetype, alloc_flags); + +/* +* If the allocation fails, allow OOM handling access +* to HIGHATOMIC reserves as failing now is worse than +* failing a high-order atomic allocation in the +* future. +*/ +if (!page && (alloc_flags &ALLOC_OOM )) + page =__rmqueue_smallest (zone, order,MIGRATE_HIGHATOMIC ); +if (!page) {spin_unlock_irqrestore (&zone->lock, flags);return NULL ;@@ -4023,8 +4033,10 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, return true; }#endif -if (alloc_harder && !free_area_empty (area,MIGRATE_HIGHATOMIC )) +if ((alloc_flags & (ALLOC_HIGHATOMIC |ALLOC_OOM )) && + !free_area_empty (area,MIGRATE_HIGHATOMIC )) {return true; + } }return false; }@@ -4286,7 +4298,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,* If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ -if (unlikely (order && (alloc_flags &ALLOC_HARDER ))) +if (unlikely (alloc_flags &ALLOC_HIGHATOMIC ))reserve_highatomic_pageblock (page, zone, order);return page;@@ -4813,7 +4825,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, }static inline unsigned int -gfp_to_alloc_flags (gfp_t gfp_mask) +gfp_to_alloc_flags (gfp_t gfp_mask,unsigned int order) {unsigned int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ;@@ -4839,8 +4851,13 @@ gfp_to_alloc_flags(gfp_t gfp_mask)* Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ -if (!(gfp_mask & __GFP_NOMEMALLOC)) +if (!(gfp_mask & __GFP_NOMEMALLOC)) { alloc_flags |=ALLOC_HARDER ; + +if (order >0 ) + alloc_flags |=ALLOC_HIGHATOMIC ; + } +/* * Ignore cpuset mems for GFP_ATOMIC rather than fail, see the * comment for __cpuset_node_allowed(). @@ -5048,7 +5065,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =gfp_to_alloc_flags (gfp_mask); + alloc_flags =gfp_to_alloc_flags (gfp_mask, order);/* * We need to recalculate the starting point for the zonelist iterator Signature · 1 line
2.35.3
Signature · 2 lines
Michal Hocko SUSE Labs
PATCH v3 4/6 mm/page_alloc: Explicitly define what alloc flags deplete min reserves
In reply to Mel GormanAs there are more ALLOC_ flags that affect reserves, define what flags affect reserves and clarify the effect of each flag.
--- mm/internal.h | 3 +++ mm/page_alloc.c | 34 ++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-)
mm/internal.h +3 -0
@@ -749,6 +749,9 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_HIGHATOMIC 0x200/* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +/* Flags that allow allocations below the min watermark. */ +#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +enum ttu_flags ;struct tlbflush_unmap_batch ;
mm/page_alloc.c +22 -12
@@ -3949,15 +3949,14 @@ ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE);static inline long __zone_watermark_unusable_free (struct zone *z,unsigned int order,unsigned int alloc_flags) { -const bool alloc_harder = (alloc_flags & (ALLOC_HARDER |ALLOC_OOM ));long unusable_free = (1 << order) -1 ;/* -* If the caller does not have rights to ALLOC_HARDER then subtract -* the high-atomic reserves. This will over-estimate the size of the -* atomic reserve but it avoids a search. +* If the caller does not have rights to reserves below the min +* watermark then subtract the high-atomic reserves. This will +* over-estimate the size of the atomic reserve but it avoids a search. */ -if (likely (!alloc_harder)) +if (likely (!(alloc_flags &ALLOC_RESERVES ))) unusable_free += z->nr_reserved_highatomic;#ifdef CONFIG_CMA
@@ -3981,25 +3980,36 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, {long min = mark;int o; -const bool alloc_harder = (alloc_flags & (ALLOC_HARDER |ALLOC_OOM ));/* free_pages may go negative - that's OK */ free_pages -=__zone_watermark_unusable_free (z, order, alloc_flags); -if (alloc_flags &ALLOC_MIN_RESERVE ) - min -= min /2 ; +if (unlikely (alloc_flags &ALLOC_RESERVES )) { +/* +* __GFP_HIGH allows access to 50% of the min reserve as well +* as OOM. +*/ +if (alloc_flags &ALLOC_MIN_RESERVE ) + min -= min /2 ; -if (unlikely (alloc_harder)) {/* -* OOM victims can try even harder than normal ALLOC_HARDER +* Non-blocking allocations can access some of the reserve +* with more access if also __GFP_HIGH. The reasoning is that +* a non-blocking caller may incur a more severe penalty +* if it cannot get memory quickly, particularly if it's +* also __GFP_HIGH. +*/ +if (alloc_flags &ALLOC_HARDER ) + min -= min /4 ; + +/* +* OOM victims can try even harder than the normal reserve * users on the grounds that it's definitely going to be in * the exit path shortly and free memory. Any allocation it * makes during the free path will be small and short-lived. */ if (alloc_flags &ALLOC_OOM ) min -= min /2 ; -else - min -= min /4 ; }/*
Signature · 1 line
2.35.3
PATCH v3 5/6 mm/page_alloc: Explicitly define how __GFP_HIGH non-blocking allocations accesses reserves
In reply to Mel GormanGFP_ATOMIC allocations get flagged ALLOC_HARDER which is a vague description. In preparation for the removal of GFP_ATOMIC redefine __GFP_ATOMIC to simply mean non-blocking and renaming ALLOC_HARDER to ALLOC_NON_BLOCK accordingly. __GFP_HIGH is required for access to reserves but non-blocking is granted more access. For example, GFP_NOWAIT is non-blocking but has no special access to reserves. A __GFP_NOFAIL blocking allocation is granted access similar to __GFP_HIGH if the only alternative is an OOM kill.
--- mm/internal.h | 7 +++++-- mm/page_alloc.c | 44 ++++++++++++++++++++++++-------------------- 2 files changed, 29 insertions(+), 22 deletions(-)
mm/internal.h +5 -2
@@ -735,7 +735,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_OOM ALLOC_NO_WATERMARKS#endif -#define ALLOC_HARDER 0x10/* try to alloc harder */ +#define ALLOC_NON_BLOCK 0x10/* Caller cannot block. Allow access +* to 25% of the min watermark or +* 62.5% if __GFP_HIGH is set. +*/ #define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% * of the min watermark. */
@@ -750,7 +753,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)enum ttu_flags ;struct tlbflush_unmap_batch ;
mm/page_alloc.c +24 -20
82 lines not on this page · raw message
Signature · 1 line
2.35.3
Re: [PATCH 5/6] mm/page_alloc: Explicitly define how __GFP_HIGH non-blocking allocations accesses reserves
Quoted 10 lines · Mel Gorman
GFP_ATOMICallocations get flaggedALLOC_HARDERwhich is a vague description. In preparation for the removal ofGFP_ATOMICredefine__GFP_ATOMICto simply mean non-blocking and renamingALLOC_HARDERtoALLOC_NON_BLOCKaccordingly.__GFP_HIGHis required for access to reserves but non-blocking is granted more access. For example,GFP_NOWAITis non-blocking but has no special access to reserves. A__GFP_NOFAILblocking allocation is granted access similar to__GFP_HIGHif the only alternative is an OOM kill.Signed-off-by Mel Gorman <[email protected]>
Thanks!
Quoted 121 lines
--- mm/internal.h | 7 +++++-- mm/page_alloc.c | 44 ++++++++++++++++++++++++-------------------- 2 files changed, 29 insertions(+), 22 deletions(-)mm/internal.h +5
-2@@ -735,7 +735,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_OOM ALLOC_NO_WATERMARKS#endif -#define ALLOC_HARDER 0x10/* try to alloc harder */ +#define ALLOC_NON_BLOCK 0x10/* Caller cannot block. Allow access +* to 25% of the min watermark or +* 62.5% if __GFP_HIGH is set. +*/ #define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% * of the min watermark. */ @@ -750,7 +753,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)enum ttu_flags ;struct tlbflush_unmap_batch ;mm/page_alloc.c +24
-20@@ -3989,18 +3989,19 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,* __GFP_HIGH allows access to 50% of the min reserve as well * as OOM. */ -if (alloc_flags &ALLOC_MIN_RESERVE ) +if (alloc_flags &ALLOC_MIN_RESERVE ) { min -= min /2 ; -/* -* Non-blocking allocations can access some of the reserve -* with more access if also __GFP_HIGH. The reasoning is that -* a non-blocking caller may incur a more severe penalty -* if it cannot get memory quickly, particularly if it's -* also __GFP_HIGH. -*/ -if (alloc_flags &ALLOC_HARDER ) - min -= min /4 ; +/* +* Non-blocking allocations (e.g. GFP_ATOMIC) can +* access more reserves than just __GFP_HIGH. Other +* non-blocking allocations requests such as GFP_NOWAIT +* or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get +* access to the min reserve. +*/ +if (alloc_flags &ALLOC_NON_BLOCK ) + min -= min /4 ; + }/* * OOM victims can try even harder than the normal reserve @@ -4851,28 +4852,30 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_MIN_RESERVE(__GFP_HIGH). +* set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int ) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); -if (gfp_mask & __GFP_ATOMIC) { +if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |=ALLOC_HARDER ; + alloc_flags |=ALLOC_NON_BLOCK ;if (order >0 ) alloc_flags |=ALLOC_HIGHATOMIC ; }/* -* Ignore cpuset mems for GFP_ATOMIC rather than fail, see the -* comment for __cpuset_node_allowed(). +* Ignore cpuset mems for non-blocking __GFP_HIGH (probably +* GFP_ATOMIC) rather than fail, see the comment for +* __cpuset_node_allowed(). */ - alloc_flags &= ~ALLOC_CPUSET ; +if (alloc_flags &ALLOC_MIN_RESERVE ) + alloc_flags &= ~ALLOC_CPUSET ; }else if (unlikely (rt_task (current)) &&in_task ()) alloc_flags |=ALLOC_MIN_RESERVE ;@@ -5303,12 +5306,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,WARN_ON_ONCE_GFP (costly_order, gfp_mask);/* -* Help non-failing allocations by giving them access to memory -* reserves but do not use ALLOC_NO_WATERMARKS because this +* Help non-failing allocations by giving some access to memory +* reserves normally used for high priority non-blocking +* allocations but do not use ALLOC_NO_WATERMARKS because this * could deplete whole memory reserves which would just make -* the situation worse +* the situation worse. */ - page =__alloc_pages_cpuset_fallback (gfp_mask, order,ALLOC_HARDER , ac); + page =__alloc_pages_cpuset_fallback (gfp_mask, order,ALLOC_MIN_RESERVE , ac);if (page) goto got_pg;Signature · 1 line
2.35.3
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 5/6] mm/page_alloc: Explicitly define how __GFP_HIGH non-blocking allocations accesses reserves
In reply to Mel GormanQuoted 2 lines · Mel Gorman
GFP_ATOMICallocations get flaggedALLOC_HARDERwhich is a vague description. In preparation for the removal ofGFP_ATOMICredefine
^ __GFP_ATOMC
Quoted 8 lines
__GFP_ATOMICto simply mean non-blocking and renamingALLOC_HARDERtoALLOC_NON_BLOCKaccordingly.__GFP_HIGHis required for access to reserves but non-blocking is granted more access. For example,GFP_NOWAITis non-blocking but has no special access to reserves. A__GFP_NOFAILblocking allocation is granted access similar to__GFP_HIGHif the only alternative is an OOM kill.Signed-off-by Mel Gorman <[email protected]>
Well just for the lore record (too late for git)
Nit below:
Quoted 10 lines
--- mm/internal.h | 7 +++++-- mm/page_alloc.c | 44 ++++++++++++++++++++++++-------------------- 2 files changed, 29 insertions(+), 22 deletions(-)mm/internal.h +0
-0
mm/internal.h unsigned int reclaim_clean_pages_from_list(struct zone *zone, · line 740
Quoted 7 lines
mm/internal.h +3
-1@@ -735,7 +735,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_OOM ALLOC_NO_WATERMARKS#endif -#define ALLOC_HARDER 0x10/* try to alloc harder */ +#define ALLOC_NON_BLOCK 0x10/* Caller cannot block. Allow access +* to 25% of the min watermark or +* 62.5% if __GFP_HIGH is set.
This is now (as of v3) inaccurate (the 25% part), right?
Quoted 102 lines
mm/internal.h +2
-1+*/ #define ALLOC_MIN_RESERVE 0x20/* __GFP_HIGH set. Allow access to 50% * of the min watermark. */ @@ -750,7 +753,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,#define ALLOC_KSWAPD 0x800/* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)enum ttu_flags ;struct tlbflush_unmap_batch ;mm/page_alloc.c +24
-20@@ -3989,18 +3989,19 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,* __GFP_HIGH allows access to 50% of the min reserve as well * as OOM. */ -if (alloc_flags &ALLOC_MIN_RESERVE ) +if (alloc_flags &ALLOC_MIN_RESERVE ) { min -= min /2 ; -/* -* Non-blocking allocations can access some of the reserve -* with more access if also __GFP_HIGH. The reasoning is that -* a non-blocking caller may incur a more severe penalty -* if it cannot get memory quickly, particularly if it's -* also __GFP_HIGH. -*/ -if (alloc_flags &ALLOC_HARDER ) - min -= min /4 ; +/* +* Non-blocking allocations (e.g. GFP_ATOMIC) can +* access more reserves than just __GFP_HIGH. Other +* non-blocking allocations requests such as GFP_NOWAIT +* or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get +* access to the min reserve. +*/ +if (alloc_flags &ALLOC_NON_BLOCK ) + min -= min /4 ; + }/* * OOM victims can try even harder than the normal reserve @@ -4851,28 +4852,30 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order)* The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_MIN_RESERVE(__GFP_HIGH). +* set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int ) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); -if (gfp_mask & __GFP_ATOMIC) { +if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) {/* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |=ALLOC_HARDER ; + alloc_flags |=ALLOC_NON_BLOCK ;if (order >0 ) alloc_flags |=ALLOC_HIGHATOMIC ; }/* -* Ignore cpuset mems for GFP_ATOMIC rather than fail, see the -* comment for __cpuset_node_allowed(). +* Ignore cpuset mems for non-blocking __GFP_HIGH (probably +* GFP_ATOMIC) rather than fail, see the comment for +* __cpuset_node_allowed(). */ - alloc_flags &= ~ALLOC_CPUSET ; +if (alloc_flags &ALLOC_MIN_RESERVE ) + alloc_flags &= ~ALLOC_CPUSET ; }else if (unlikely (rt_task (current)) &&in_task ()) alloc_flags |=ALLOC_MIN_RESERVE ;@@ -5303,12 +5306,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,WARN_ON_ONCE_GFP (costly_order, gfp_mask);/* -* Help non-failing allocations by giving them access to memory -* reserves but do not use ALLOC_NO_WATERMARKS because this +* Help non-failing allocations by giving some access to memory +* reserves normally used for high priority non-blocking +* allocations but do not use ALLOC_NO_WATERMARKS because this * could deplete whole memory reserves which would just make -* the situation worse +* the situation worse. */ - page =__alloc_pages_cpuset_fallback (gfp_mask, order,ALLOC_HARDER , ac); + page =__alloc_pages_cpuset_fallback (gfp_mask, order,ALLOC_MIN_RESERVE , ac);if (page) goto got_pg;
PATCH v3 6/6 mm: discard __GFP_ATOMIC
In reply to Mel GormanFrom: NeilBrown <[email protected]>
__GFP_ATOMIC serves little purpose. Its main effect is to set ALLOC_HARDER which adds a few little boosts to increase the chance of an allocation succeeding, one of which is to lower the water-mark at which it will succeed.
It is *always* paired with __GFP_HIGH which sets ALLOC_HIGH which also adjusts this watermark. It is probable that other users of __GFP_HIGH should benefit from the other little bonuses that __GFP_ATOMIC gets.
__GFP_ATOMIC also gives a warning if used with __GFP_DIRECT_RECLAIM. There is little point to this. We already get a might_sleep() warning if __GFP_DIRECT_RECLAIM is set.
__GFP_ATOMIC allows the "watermark_boost" to be side-stepped. It is probable that testing ALLOC_HARDER is a better fit here.
__GFP_ATOMIC is used by tegra-smmu.c to check if the allocation might sleep. This should test __GFP_DIRECT_RECLAIM instead.
This patch:
- removes __GFP_ATOMIC
- allows __GFP_HIGH allocations to ignore watermark boosting as well as GFP_ATOMIC requests.
- makes other adjustments as suggested by the above.
The net result is not change to GFP_ATOMIC allocations. Other allocations that use __GFP_HIGH will benefit from a few different extra privileges. This affects:
xen, dm, md, ntfs3
the vermillion frame buffer
hibernation
ksm
swap
all of which likely produce more benefit than cost if these selected allocation are more likely to succeed quickly.
--- Documentation/mm/balance.rst | 2 +- drivers/iommu/tegra-smmu.c | 4 ++-- include/linux/gfp_types.h | 12 ++++-------- include/trace/events/mmflags.h | 1 - lib/test_printf.c | 8 ++++---- mm/internal.h | 2 +- mm/page_alloc.c | 13 +++---------- tools/perf/builtin-kmem.c | 1 - 8 files changed, 15 insertions(+), 28 deletions(-)
Documentation/mm/balance.rst +1 -1
@@ -6,7 +6,7 @@ Memory Balancing Started Jan 2000 by Kanoj Sarcar <[email protected]> -Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as +Memory balancing is needed for !__GFP_HIGH and !__GFP_KSWAPD_RECLAIM as well as for non __GFP_IO allocations. The first reason why a caller may avoid reclaim is that the caller can not
drivers/iommu/tegra-smmu.c +2 -2
@@ -671,12 +671,12 @@ static struct page *as_get_pde_page(struct tegra_smmu_as *as,* allocate page in a sleeping context if GFP flags permit. Hence * spinlock needs to be unlocked and re-locked after allocation. */ -if (!(gfp & __GFP_ATOMIC)) +if (gfpflags_allow_blocking (gfp))spin_unlock_irqrestore (&as->lock, *flags); page =alloc_page (gfp | __GFP_DMA | __GFP_ZERO); -if (!(gfp & __GFP_ATOMIC)) +if (gfpflags_allow_blocking (gfp))spin_lock_irqsave (&as->lock, *flags);/*
include/linux/gfp_types.h +4 -8
@@ -31,7 +31,7 @@ typedef unsigned int __bitwise gfp_t;#define ___GFP_IO 0x40u#define ___GFP_FS 0x80u#define ___GFP_ZERO 0x100u -#define ___GFP_ATOMIC 0x200u +/* 0x200u unused */ #define ___GFP_DIRECT_RECLAIM 0x400u#define ___GFP_KSWAPD_RECLAIM 0x800u#define ___GFP_WRITE 0x1000u
@@ -116,11 +116,8 @@ typedef unsigned int __bitwise gfp_t;* * %__GFP_HIGH indicates that the caller is high-priority and that granting * the request is necessary before the system can make forward progress. -* For example, creating an IO context to clean pages. -* -* %__GFP_ATOMIC indicates that the caller cannot reclaim or sleep and is -* high priority. Users are typically interrupt handlers. This may be -* used in conjunction with %__GFP_HIGH +* For example creating an IO context to clean pages and requests +* from atomic context. * * %__GFP_MEMALLOC allows access to all memory. This should only be used when * the caller guarantees the allocation will allow more memory to be freed
@@ -135,7 +132,6 @@ typedef unsigned int __bitwise gfp_t;* %__GFP_NOMEMALLOC is used to explicitly forbid access to emergency reserves. * This takes precedence over the %__GFP_MEMALLOC flag if both are set. */ -#define __GFP_ATOMIC ((__force gfp_t)___GFP_ATOMIC)#define __GFP_HIGH ((__force gfp_t)___GFP_HIGH)#define __GFP_MEMALLOC ((__force gfp_t)___GFP_MEMALLOC)#define __GFP_NOMEMALLOC ((__force gfp_t)___GFP_NOMEMALLOC)
@@ -329,7 +325,7 @@ typedef unsigned int __bitwise gfp_t;* version does not attempt reclaim/compaction at all and is by default used * in page fault path, while the non-light is used by khugepaged. */ -#define GFP_ATOMIC (__GFP_HIGH|__GFP_ATOMIC|__GFP_KSWAPD_RECLAIM) +#define GFP_ATOMIC (__GFP_HIGH|__GFP_KSWAPD_RECLAIM)#define GFP_KERNEL (__GFP_RECLAIM | __GFP_IO | __GFP_FS)#define GFP_KERNEL_ACCOUNT (GFP_KERNEL | __GFP_ACCOUNT)#define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM)
include/trace/events/mmflags.h +0 -1
@@ -31,7 +31,6 @@gfpflag_string (__GFP_HIGHMEM), \gfpflag_string (GFP_DMA32 ), \gfpflag_string (__GFP_HIGH), \ -gfpflag_string (__GFP_ATOMIC), \gfpflag_string (__GFP_IO), \gfpflag_string (__GFP_FS), \ gfpflag_string(__GFP_NOWARN), \
lib/test_printf.c +4 -4
@@ -674,17 +674,17 @@ flags(void) gfp =GFP_ATOMIC |__GFP_DMA;test ("GFP_ATOMIC|GFP_DMA" ,"%pGg" , &gfp); - gfp = __GFP_ATOMIC; -test ("__GFP_ATOMIC" ,"%pGg" , &gfp); + gfp = __GFP_HIGH; +test ("__GFP_HIGH" ,"%pGg" , &gfp);/* Any flags not translated by the table should remain numeric */ gfp = ~__GFP_BITS_MASK;snprintf (cmp_buffer,BUF_SIZE ,"%#lx" , (unsigned long ) gfp);test (cmp_buffer,"%pGg" , &gfp); -snprintf (cmp_buffer,BUF_SIZE ,"__GFP_ATOMIC|%#lx" , +snprintf (cmp_buffer,BUF_SIZE ,"__GFP_HIGH|%#lx" , (unsigned long ) gfp); - gfp |= __GFP_ATOMIC; + gfp |= __GFP_HIGH;test (cmp_buffer,"%pGg" , &gfp);kfree (cmp_buffer);
mm/internal.h +1 -1
@@ -24,7 +24,7 @@ struct folio_batch;#define GFP_RECLAIM_MASK (__GFP_RECLAIM|__GFP_HIGH|__GFP_IO|__GFP_FS|\ __GFP_NOWARN|__GFP_RETRY_MAYFAIL|__GFP_NOFAIL|\ __GFP_NORETRY|__GFP_MEMALLOC|__GFP_NOMEMALLOC|\ - __GFP_ATOMIC|__GFP_NOLOCKDEP) + __GFP_NOLOCKDEP)/* The GFP flags allowed during early boot */ #define GFP_BOOT_MASK (__GFP_BITS_MASK & ~(__GFP_RECLAIM|__GFP_IO|__GFP_FS))
mm/page_alloc.c +3 -10
@@ -4087,13 +4087,14 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order,if (__zone_watermark_ok (z, order, mark, highest_zoneidx, alloc_flags, free_pages))return true; +/* -* Ignore watermark boosting for GFP_ATOMIC order-0 allocations +* Ignore watermark boosting for __GFP_HIGH order-0 allocations * when checking the min watermark. The min watermark is the * point where boosting is ignored so that kswapd is woken up * when below the low watermark. */ -if (unlikely (!order && (gfp_mask & __GFP_ATOMIC) && z->watermark_boost +if (unlikely (!order && (alloc_flags &ALLOC_MIN_RESERVE ) && z->watermark_boost && ((alloc_flags &ALLOC_WMARK_MASK ) ==WMARK_MIN ))) { mark = z->_watermark[WMARK_MIN ]; return __zone_watermark_ok(z, order, mark, highest_zoneidx,
@@ -5058,14 +5059,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,unsigned int zonelist_iter_cookie;int reserve_flags; -/* -* We also sanity check to catch abuse of atomic reserves being used by -* callers that are not in atomic context. -*/ -if (WARN_ON_ONCE ((gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)) == - (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM))) - gfp_mask &= ~__GFP_ATOMIC; - restart: compaction_retries =0 ; no_progress_loops =0 ;
tools/perf/builtin-kmem.c +0 -1
@@ -641,7 +641,6 @@ static const struct { { "__GFP_HIGHMEM", "HM" }, { "GFP_DMA32", "D32" }, { "__GFP_HIGH", "H" }, - { "__GFP_ATOMIC", "_A" }, { "__GFP_IO", "I" }, { "__GFP_FS", "F" }, { "__GFP_NOWARN", "NWR" },
Signature · 1 line
2.35.3