mm: remove min_free_kbytes adjustment for THP
Subsystems (5)
PATCH v3 1/1 mm: remove min_free_kbytes adjustment for THP
When THP is enabled, set_recommended_min_free_kbytes() may raise min_free_kbytes using a heuristic that scales with pageblock_nr_pages. Commit f000565adb77 ("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zone
Even with that cap, min_free_kbytes can reach excessive levels.
The automatic min_free_kbytes increase predates proactive compaction and many subsequent changes to compaction. Given those changes, increasing min_free_kbytes for THP by default is no longer clearly justified.
Remove set_recommended_min_free_kbytes() and all associated recalculation paths. With this policy gone, min_free_kbytes is controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
---
v3:
* Replace the 1 GiB cap with removal of the THP-driven min_free_kbytes increase as suggested by Michal.
* Remove all associated recalculation paths.
* Make min_free_kbytes, user_min_free_kbytes, calculate_min_free_kbytes(), and setup_per_zone_wmarks() static.
* Drop the now-unused declarations and khugepaged's page_alloc.h include.
* Drop the obsolete v2 documentation addition.
v2:
* Use a named constant for the cap.
* Drop explicit linux/sizes.h include.
* Update min_free_kbytes documentation.
https://lore.kernel.org/r/[email protected]/
v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/
RFC v1:
https://lore.kernel.org/r/[email protected]/
--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)
include/linux/khugepaged.h +0 -5
@@ -15,7 +15,6 @@ extern void __khugepaged_enter(struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm);extern void khugepaged_enter_vma (struct vm_area_struct *vma,vm_flags_t vm_flags); -extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void );void collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd);
@@ -47,10 +46,6 @@ static inline void collapse_pte_mapped_thp(struct mm_struct *mm, { } -static inline void khugepaged_min_free_kbytes_update (void ) -{ -} -static inline bool current_is_khugepaged (void ) {return false;
mm/huge_memory.c +0 -14
@@ -499,13 +499,6 @@ static ssize_t enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count; }
@@ -735,13 +728,6 @@ static ssize_t anon_enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count;
mm/internal.h +0 -8
@@ -689,11 +689,6 @@ int user_proactive_reclaim(char *buf,*/ pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address); -/* -* in mm/khugepaged.c -*/ -void set_recommended_min_free_kbytes (void ); -/* * in mm/page_alloc.c */
@@ -701,11 +696,8 @@ void set_recommended_min_free_kbytes(void);extern char *const zone_names[MAX_NR_ZONES ]; -extern int min_free_kbytes;extern int defrag_mode; -void setup_per_zone_wmarks (void ); -void calculate_min_free_kbytes (void );int __meminit init_per_zone_wmark_min (void );extern int __isolate_free_page (struct page *page,unsigned int order);
mm/khugepaged.c +0 -60
81 lines not on this page · raw message
mm/page_alloc.c +4 -7
@@ -50,7 +50,6 @@#include <linux/ftrace.h> #include <linux/lockdep.h> #include <linux/psi.h> -#include <linux/khugepaged.h> #include <linux/delayacct.h> #include <linux/cacheinfo.h> #include <linux/pgalloc_tag.h>
@@ -272,8 +271,8 @@ const char * const migratetype_names[MIGRATE_TYPES] = {#endif }; -int min_free_kbytes =1024 ; -int user_min_free_kbytes =-1 ; +static int min_free_kbytes =1024 ; +static int user_min_free_kbytes =-1 ;static int watermark_boost_factor__read_mostly =15000 ;static int watermark_scale_factor =10 ;int defrag_mode;
@@ -6663,7 +6662,7 @@ static void __setup_per_zone_wmarks(void)* Ensures that the watermark[min,low,high] values for each zone are set * correctly with respect to min_free_kbytes. */ -void setup_per_zone_wmarks (void ) +static void setup_per_zone_wmarks (void ) {struct zone *zone;static DEFINE_SPINLOCK (lock);
@@ -6704,7 +6703,7 @@ void setup_per_zone_wmarks(void)* 8192MB: 11584k * 16384MB: 16384k */ -void calculate_min_free_kbytes (void ) +static void calculate_min_free_kbytes (void ) {unsigned long lowmem_kbytes;int new_min_free_kbytes;
@@ -6732,8 +6731,6 @@ int __meminit init_per_zone_wmark_min(void)setup_min_slab_ratio ();#endif -khugepaged_min_free_kbytes_update (); -return 0 ; }postcore_initcall (init_per_zone_wmark_min)
mm/page_alloc.h +0 -2
@@ -227,8 +227,6 @@ void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order); -extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid,nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \
mm/shmem.c +0 -7
@@ -5782,13 +5782,6 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,int err =start_stop_khugepaged ();if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode hasn't changed -* to preserve the legacy behavior, as this is always called -* inside start_stop_khugepaged(). -*/ -set_recommended_min_free_kbytes (); }return count;
Signature · 1 line
2.45.0
Quoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
I think this is definitely worth try. Let's see if this leads to any sort of regressions. I am aware of setups where people were actively fighting against this increased value so let's see who might just depend on that without knowing.
Quoted 286 lines
---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/RFC v1:
https://lore.kernel.org/r/[email protected]/--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)include/linux/khugepaged.h +0
-5@@ -15,7 +15,6 @@ extern void __khugepaged_enter(struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm);extern void khugepaged_enter_vma (struct vm_area_struct *vma,vm_flags_t vm_flags); -extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void );void collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd);@@ -47,10 +46,6 @@ static inline void collapse_pte_mapped_thp(struct mm_struct *mm, { } -static inline void khugepaged_min_free_kbytes_update (void ) -{ -} -static inline bool current_is_khugepaged (void ) {return false;mm/huge_memory.c +0
-14@@ -499,13 +499,6 @@ static ssize_t enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count; }@@ -735,13 +728,6 @@ static ssize_t anon_enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count;mm/internal.h +0
-8@@ -689,11 +689,6 @@ int user_proactive_reclaim(char *buf,*/ pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address); -/* -* in mm/khugepaged.c -*/ -void set_recommended_min_free_kbytes (void ); -/* * in mm/page_alloc.c */ @@ -701,11 +696,8 @@ void set_recommended_min_free_kbytes(void);extern char *const zone_names[MAX_NR_ZONES ]; -extern int min_free_kbytes;extern int defrag_mode; -void setup_per_zone_wmarks (void ); -void calculate_min_free_kbytes (void );int __meminit init_per_zone_wmark_min (void );extern int __isolate_free_page (struct page *page,unsigned int order);mm/khugepaged.c +0
-60@@ -27,7 +27,6 @@#include <asm/tlb.h> #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" enum scan_result {@@ -3086,57 +3085,6 @@ static int khugepaged(void *none)return 0 ; } -void set_recommended_min_free_kbytes (void ) -{ -struct zone *zone; -int nr_zones =0 ; -unsigned long recommended_min; - -if (!hugepage_enabled ()) { -calculate_min_free_kbytes (); - goto update_wmarks; - } - -for_each_populated_zone (zone) { -/* -* We don't need to worry about fragmentation of -* ZONE_MOVABLE since it only has movable pages. -*/ -if (zone_idx (zone) >gfp_zone (GFP_USER )) -continue ; - - nr_zones++; - } - -/* Ensure 2 pageblocks are free to assist fragmentation avoidance */ - recommended_min = pageblock_nr_pages * nr_zones *2 ; - -/* -* Make sure that on average at least two pageblocks are almost free -* of another type, one for a migratetype to fall back to and a -* second to avoid subsequent fallbacks of other types There are 3 -* MIGRATE_TYPES we care about. -*/ - recommended_min += pageblock_nr_pages * nr_zones * -MIGRATE_PCPTYPES *MIGRATE_PCPTYPES ; - -/* don't ever allow to reserve more than 5% of the lowmem */ - recommended_min =min (recommended_min, - (unsigned long )nr_free_buffer_pages () /20 ); - recommended_min <<= (PAGE_SHIFT -10 ); - -if (recommended_min > min_free_kbytes) { -if (user_min_free_kbytes >=0 ) -pr_info_ratelimited ("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n" , - min_free_kbytes, recommended_min); - - min_free_kbytes = recommended_min; - } - -update_wmarks: -setup_per_zone_wmarks (); -} -int start_stop_khugepaged (void ) {guard (mutex)(&khugepaged_mutex);@@ -3160,17 +3108,9 @@ int start_stop_khugepaged(void)kthread_stop (khugepaged_thread); khugepaged_thread =NULL ; } -set_recommended_min_free_kbytes ();return 0 ; } -void khugepaged_min_free_kbytes_update (void ) -{ -guard (mutex)(&khugepaged_mutex); -if (hugepage_enabled () && khugepaged_thread) -set_recommended_min_free_kbytes (); -} -bool current_is_khugepaged (void ) {return kthread_func (current) == khugepaged;mm/page_alloc.c +4
-7@@ -50,7 +50,6 @@#include <linux/ftrace.h> #include <linux/lockdep.h> #include <linux/psi.h> -#include <linux/khugepaged.h> #include <linux/delayacct.h> #include <linux/cacheinfo.h> #include <linux/pgalloc_tag.h> @@ -272,8 +271,8 @@ const char * const migratetype_names[MIGRATE_TYPES] = {#endif }; -int min_free_kbytes =1024 ; -int user_min_free_kbytes =-1 ; +static int min_free_kbytes =1024 ; +static int user_min_free_kbytes =-1 ;static int watermark_boost_factor__read_mostly =15000 ;static int watermark_scale_factor =10 ;int defrag_mode;@@ -6663,7 +6662,7 @@ static void __setup_per_zone_wmarks(void)* Ensures that the watermark[min,low,high] values for each zone are set * correctly with respect to min_free_kbytes. */ -void setup_per_zone_wmarks (void ) +static void setup_per_zone_wmarks (void ) {struct zone *zone;static DEFINE_SPINLOCK (lock);@@ -6704,7 +6703,7 @@ void setup_per_zone_wmarks(void)* 8192MB: 11584k * 16384MB: 16384k */ -void calculate_min_free_kbytes (void ) +static void calculate_min_free_kbytes (void ) {unsigned long lowmem_kbytes;int new_min_free_kbytes;@@ -6732,8 +6731,6 @@ int __meminit init_per_zone_wmark_min(void)setup_min_slab_ratio ();#endif -khugepaged_min_free_kbytes_update (); -return 0 ; }postcore_initcall (init_per_zone_wmark_min)mm/page_alloc.h +0
-2@@ -227,8 +227,6 @@ void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order); -extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid,nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \mm/shmem.c +0
-7@@ -5782,13 +5782,6 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,int err =start_stop_khugepaged ();if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode hasn't changed -* to preserve the legacy behavior, as this is always called -* inside start_stop_khugepaged(). -*/ -set_recommended_min_free_kbytes (); }return count;Signature · 1 line
2.45.0
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 52 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/
Quoted 10 lines
--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)
The stats look nice. Like you said in the commit message, user can still have the old behavior by increasing vm.min_free_kbytes. Thanks.
Best Regards,
Yan, Zi
Johannes Weiner
reviewerQuoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.
We've had problems with compaction before when min_free_kbytes was too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.
So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zone
I question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive" min_free_kbytes issue as well.
Quoted 4 lines
The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.
That's pretty handwavy. How would these changes specifically eliminate the need for compaction scratch space and allocator fallback options to stave off fragmentation during placement?
Zi Yan
reviewerQuoted 27 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
Quoted 6 lines
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.
But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs. The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
After removing automatic min_free_kbytes boosting, user can still increase it via sysctl to restore the old free memory head room. It is much easier, right?
Quoted 9 lines
Quoted 5 lines
The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.That's pretty handwavy. How would these changes specifically eliminate the need for compaction scratch space and allocator fallback options to stave off fragmentation during placement?
Extra free memory is still necessary. min_free_kbytes can be adjusted at machine boot time to achieve it, right?
Best Regards,
Yan, Zi
Johannes Weiner
reviewerQuoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the min_free_kbytes
issue: Fragmentation events will poison half a gig at once, should_try_claim_block() becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...
Seems to me the excessive min_free_kbytes is just a symptom of a deeper problem.
Quoted 10 lines
After removing automatic
min_free_kbytesboosting, user can still increase it via sysctl to restore the old free memory head room. It is much easier, right?Quoted 6 lines
Quoted 5 lines
The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.
Quoted 6 lines
Quoted 3 lines
That's pretty handwavy. How would these changes specifically eliminate the need for compaction scratch space and allocator fallback options to stave off fragmentation during placement?
Extra free memory is still necessary.
min_free_kbytescan be adjusted at machine boot time to achieve it, right?
That argument cuts both ways, no? ;)
Zi Yan
reviewerQuoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...
I actually wonder why such a big pageblock would still result in a lot of fallbacks. Basically it indicates at some point kernel allocates a lot of unmovable pages that use many 512MB pageblocks and the life time of these unmovable pages are so diverse, leading to all these pageblocks remain unmovable and free pages spread across all these pageblocks. I thought bigger pageblocks can keep unmovable pages constrained within fewer pageblocks, leaving more contiguous free memory.
Do you mind elaborating more on bigger pageblocks cause bigger issues? Maybe it is something more fundamental related to our unmovable page management (which almost does not exit)? It seems to me that the large pageblock amplifies the issues.
Quoted 3 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.
Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive min_free_kbytes to get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it via min_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boosting min_free_kbytes?
Quoted 11 lines
Quoted 11 lines
After removing automatic
min_free_kbytesboosting, user can still increase it via sysctl to restore the old free memory head room. It is much easier, right?Quoted 6 lines
Quoted 5 lines
The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
That's pretty handwavy. How would these changes specifically eliminate the need for compaction scratch space and allocator fallback options to stave off fragmentation during placement?
Extra free memory is still necessary.
min_free_kbytescan be adjusted at machine boot time to achieve it, right?That argument cuts both ways, no? ;)
Yes, but with current khugepaged code, min_free_kbytes is recalculated whenever any of THP, mTHP, and shmem THP enablements is changed. After the change, min_free_kbytes is basically a set and forget thing.
Best Regards,
Yan, Zi
Lorenzo Stoakes (ARM)
reviewerQuoted 100 lines · Zi Yan
Quoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...I actually wonder why such a big pageblock would still result in a lot of fallbacks. Basically it indicates at some point kernel allocates a lot of unmovable pages that use many 512MB pageblocks and the life time of these unmovable pages are so diverse, leading to all these pageblocks remain unmovable and free pages spread across all these pageblocks. I thought bigger pageblocks can keep unmovable pages constrained within fewer pageblocks, leaving more contiguous free memory.
Do you mind elaborating more on bigger pageblocks cause bigger issues? Maybe it is something more fundamental related to our unmovable page management (which almost does not exit)? It seems to me that the large pageblock amplifies the issues.
Quoted 3 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?Quoted 15 lines
Quoted 15 lines
After removing automatic
min_free_kbytesboosting, user can still increase it via sysctl to restore the old free memory head room. It is much easier, right?Quoted 9 lines
Quoted 5 lines
The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.That's pretty handwavy. How would these changes specifically eliminate the need for compaction scratch space and allocator fallback options to stave off fragmentation during placement?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Extra free memory is still necessary.
min_free_kbytescan be adjusted at machine boot time to achieve it, right?That argument cuts both ways, no? ;)
Yes, but with current khugepaged code,
min_free_kbytesis recalculated whenever any of THP, mTHP, and shmem THP enablements is changed. After the change,min_free_kbytesis basically a set and forget thing.
Yeah, overall we have gone back-and-forth on the 512 MiB page block thing, different solutions have been proposed (Nimrod included a list of these in his initial patches), and each one has not ended up suitable because they all seem to hack in things that violate the assumptions built into the pageblock mode.
So I see this as a 2-stage thing - firstly eliminate the reservation issue so people can practically stop having to deal with problems there. The second - look at pageblocks/compaction more carefully.
And as Zi says, if you need a bigger min_free_kbytes, set a bigger min_free_kbytes :)
My suggestion here was the 'so dumb it kinda makes sense' approach of - hey if we want to limit reserve size to X just express it like that, don't play games with page blocks.
But I like this solution a lot more - as Michal points out, with proactive compaction and changes in how that code works, it doesn't really make sense for this 'if THP is enabled set to arbitrary value' stuff.
So I'm absolutely for us just cutting this out now, and that frees people up to solve the bigger issue underlying this that in an mTHP and ma-hoo-sive PMD size world, pageblocks == PMD size makes absolutely no sense any longer.
I think Kiryl's upcoming work to make THP no longer treat PMD as the unit of measure for everything will help, and hopefully if Rik can find a way to put forward his GiB superpageblock stuff in a way that can be iteratively taken upstream that could be some way forwards on compaction.
Quoted 4 lines
Best Regards,
Yan, Zi
--
Cheers, Lorenzo
Johannes Weiner
reviewerQuoted 61 lines · Zi Yan
Quoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...
Quoted 2 lines
I actually wonder why such a big pageblock would still result in a lot of fallbacks.
If you look at try_to_claim_block(), you need half of a block to be free or compatible with the requested migratetype in order to convert it. When memory is full, LRU pages are scattered all over, and compaction is not involved (order-0), this gets more difficult the bigger the block is. You can get into a situation where LRU reclaim will not clear sufficient room for conversion in any given pageblock anymore and you're stuck with the type distribution. A large share of buddy requests then go permanently through the slower fallback path.
Usama knows more about this, but we have seen this on GB300 hosts, and have JUST started to deploy kernels with smaller pageblocks (2M).
Quoted 7 lines
Basically it indicates at some point kernel allocates a lot of unmovable pages that use many 512MB pageblocks and the life time of these unmovable pages are so diverse, leading to all these pageblocks remain unmovable and free pages spread across all these pageblocks. I thought bigger pageblocks can keep unmovable pages constrained within fewer pageblocks, leaving more contiguous free memory.
The idea is that the pageblock maintains contiguity for the largest size you routinely expect to allocate.
The page allocator is very passive right now, and it doesn't work super reliably. But even in the current regime, smaller blocks have a better chance of containment.
For example, when the ever-growing page cache runs out of movable block space, it spills into unmovable free space. When the next unmovable request finds no space, it runs LRU reclaim - which is more likely to free space in one of the many movable blocks. And so the next block is poisoned. Smaller blocks have a better chance of filling up natively, means less pressure to spill into incompatible ones.
And the higher min_free_kbytes, the more likely there are still native options when the zones are down to the watermarks. E.g. better odds there is still unmovable free space, you just need to reclaim some movable/reclaimable space elsewhere to satisfy the watermarks.
I've been working on making this more robust with the huge page allocator / defrag_mode stuff: instead of falling back and poisoning a block, invoke reclaim/compaction to produce a neutral block that can be converted entirely.
It's the same idea as the higher min_free_kbytes and watermark boosting, but it is more targeted at the end result: readily available space in compatible or convertible blocks.
But with that active regime, oversized pageblocks are even worse. You'd pay ongoing compaction work to produce a level of contiguity that you don't actually need.
Quoted 9 lines
Quoted 2 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?
I'm just wondering what the easiest way forward is to fix the ARM 64k page problem.
Yes, optimally, reclaim would work to satisfy compaction space by itself. We've seen it fail at that before, though.
How critical set_recommended_min_free_kbytes() is today is a question that neither of us has a clear answer to. It's from 2011 and a lot has changed. However, knowing Andrea, I'm willing to bet he added this based on seeing a need in testing data. And I would actually expect it to work better now with proactive compaction, since that has a better chance of turning low-order chunks of that volume into pageblocks that can be converted instead of needing a poisoning steal.
It's a change of long-standing behavior for everybody. It has a regression risk and requires careful evaluation and testing.
Meanwhile, adjusting the pageblock size on 64k page arm configs has a much smaller blast radius, appears to be the right move ANYWAY given what pageblocks are for, and makes the min_free_kbytes a non-issue.
Usama Arif
reviewerQuoted 74 lines · Johannes Weiner
Quoted 61 lines · Zi Yan
Quoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...Quoted 2 lines
I actually wonder why such a big pageblock would still result in a lot of fallbacks.
If you look at
try_to_claim_block(), you need half of a block to be free or compatible with the requested migratetype in order to convert it. When memory is full, LRU pages are scattered all over, and compaction is not involved (order-0), this gets more difficult the bigger the block is. You can get into a situation where LRU reclaim will not clear sufficient room for conversion in any given pageblock anymore and you're stuck with the type distribution. A large share of buddy requests then go permanently through the slower fallback path.
Quoted 2 lines
Usama knows more about this, but we have seen this on GB300 hosts, and have JUST started to deploy kernels with smaller pageblocks (2M).
Yes, just to confirm above, unfortunately, there were too many problems we were seeing in Meta production with 512M pageblock size for arm 64K base page size and just had to switch to 2M.
Quoted 71 lines
Quoted 8 lines
Basically it indicates at some point kernel allocates a lot of unmovable pages that use many 512MB pageblocks and the life time of these unmovable pages are so diverse, leading to all these pageblocks remain unmovable and free pages spread across all these pageblocks. I thought bigger pageblocks can keep unmovable pages constrained within fewer pageblocks, leaving more contiguous free memory.
The idea is that the pageblock maintains contiguity for the largest size you routinely expect to allocate.
The page allocator is very passive right now, and it doesn't work super reliably. But even in the current regime, smaller blocks have a better chance of containment.
For example, when the ever-growing page cache runs out of movable block space, it spills into unmovable free space. When the next unmovable request finds no space, it runs LRU reclaim - which is more likely to free space in one of the many movable blocks. And so the next block is poisoned. Smaller blocks have a better chance of filling up natively, means less pressure to spill into incompatible ones.
And the higher
min_free_kbytes, the more likely there are still native options when the zones are down to the watermarks. E.g. better odds there is still unmovable free space, you just need to reclaim some movable/reclaimable space elsewhere to satisfy the watermarks.I've been working on making this more robust with the huge page allocator /
defrag_modestuff: instead of falling back and poisoning a block, invoke reclaim/compaction to produce a neutral block that can be converted entirely.It's the same idea as the higher
min_free_kbytesand watermark boosting, but it is more targeted at the end result: readily available space in compatible or convertible blocks.But with that active regime, oversized pageblocks are even worse. You'd pay ongoing compaction work to produce a level of contiguity that you don't actually need.
Quoted 9 lines
Quoted 2 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?I'm just wondering what the easiest way forward is to fix the ARM 64k page problem.
Yes, optimally, reclaim would work to satisfy compaction space by itself. We've seen it fail at that before, though.
How critical
set_recommended_min_free_kbytes()is today is a question that neither of us has a clear answer to. It's from 2011 and a lot has changed. However, knowing Andrea, I'm willing to bet he added this based on seeing a need in testing data. And I would actually expect it to work better now with proactive compaction, since that has a better chance of turning low-order chunks of that volume into pageblocks that can be converted instead of needing a poisoning steal.It's a change of long-standing behavior for everybody. It has a regression risk and requires careful evaluation and testing.
Meanwhile, adjusting the pageblock size on 64k page arm configs has a much smaller blast radius, appears to be the right move ANYWAY given what pageblocks are for, and makes the
min_free_kbytesa non-issue.
Zi Yan
reviewerQuoted 75 lines · Usama Arif
Quoted 74 lines · Johannes Weiner
Quoted 61 lines · Zi Yan
Quoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...Quoted 2 lines
I actually wonder why such a big pageblock would still result in a lot of fallbacks.
If you look at
try_to_claim_block(), you need half of a block to be free or compatible with the requested migratetype in order to convert it. When memory is full, LRU pages are scattered all over, and compaction is not involved (order-0), this gets more difficult the bigger the block is. You can get into a situation where LRU reclaim will not clear sufficient room for conversion in any given pageblock anymore and you're stuck with the type distribution. A large share of buddy requests then go permanently through the slower fallback path.
Quoted 6 lines
Quoted 2 lines
Usama knows more about this, but we have seen this on GB300 hosts, and have JUST started to deploy kernels with smaller pageblocks (2M).
Yes, just to confirm above, unfortunately, there were too many problems we were seeing in Meta production with 512M pageblock size for arm 64K base page size and just had to switch to 2M.
Do you mind elaborating on the problems? Johannes has given good arguments. I would like to hear more from you, since he said you know more about the issues.
Thanks.
Best Regards,
Yan, Zi
Usama Arif
reviewerQuoted 80 lines · Zi Yan
Quoted 78 lines · Usama Arif
Quoted 76 lines · Johannes Weiner
Quoted 61 lines · Zi Yan
Quoted 51 lines · Johannes Weiner
Quoted 32 lines · Zi Yan
Quoted 30 lines · Johannes Weiner
Quoted 5 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.We've had problems with compaction before when
min_free_kbyteswas too small on large machines. Competing free space scanners do a lot of work only to fight over a very small set of possible target pages.So I'm a bit uneasy that you didn't include any benchmark numbers with this that prove basic functionality on larger hosts isn't regressed.
Quoted 7 lines
The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneI question whether pageblocks need to be 512M on those machines to begin with. After this patch, you're still asking the page allocator to optimize grouping such that 512M pages can be allocated at runtime. Only now you took away part of the mechanism to do so.
If you're using 512M THPs, I would kind of assume it's on machines with a memory size where 5.5G for defrag purposes isn't devastating.
Quoted 6 lines
Quoted 3 lines
And if you're not, it would make more sense to lower the pageblock size to the mTHP size you're actually using. And that would fix the "excessive"
min_free_kbytesissue as well.But lowering pageblock size requires a kernel compilation. That means maintaining two sets of kernels for different needs.
That depends on whether anyone actually wants 512M pageblocks...
Quoted 3 lines
The ultimate solution is to enable better compaction to generate THPs bigger than a pageblock size, like Rik's super-pageblock proposal.
...or whether we can say, at that point, use gigablocks/cma+hugetlb.
And then the static pageblock size for the fallback logic etc. can be a smaller, saner default for everybody.
Quoted 7 lines
Because the point you didn't address: it doesn't make really sense to have 512M pageblocks on smaller machines, beyond the
min_free_kbytes
issue: Fragmentation events will poison half a gig at once,should_try_claim_block()becomes harder which results in less conversions and more allocations falling through to stealing, page isolation is more likely to fail, compaction locks and operates on oversized chunks which is bad for latency and concurrency...Quoted 2 lines
I actually wonder why such a big pageblock would still result in a lot of fallbacks.
If you look at
try_to_claim_block(), you need half of a block to be free or compatible with the requested migratetype in order to convert it. When memory is full, LRU pages are scattered all over, and compaction is not involved (order-0), this gets more difficult the bigger the block is. You can get into a situation where LRU reclaim will not clear sufficient room for conversion in any given pageblock anymore and you're stuck with the type distribution. A large share of buddy requests then go permanently through the slower fallback path.Usama knows more about this, but we have seen this on GB300 hosts, and have JUST started to deploy kernels with smaller pageblocks (2M).
Quoted 7 lines
Quoted 3 lines
Yes, just to confirm above, unfortunately, there were too many problems we were seeing in Meta production with 512M pageblock size for arm 64K base page size and just had to switch to 2M.
Do you mind elaborating on the problems? Johannes has given good arguments. I would like to hear more from you, since he said you know more about the issues.
The issue that finally pushed us toward 2 MiB pageblocks occurred on GPU hosts during AI-training checkpointing. We observed significant latency spikes and workload timeouts.
These hosts had substantial unmovable memory, and order-0 allocations from TCP paths repeatedly fell back to MIGRATE_MOVABLE free lists. With 512 MiB pageblocks, try_to_claim_block() often could not meet the half-pageblock threshold, so allocation fell through to stealing individual pages.
This contaminated very large movable pageblocks with unmovable pages. Compaction then repeatedly scanned those pageblocks but could not evacuate the unmovable allocations. We observed sustained compaction activity during the latency spikes.
Quoted 6 lines
Thanks.
Best Regards,
Yan, Zi
Lorenzo Stoakes (ARM)
reviewerQuoted 7 lines · Johannes Weiner
The idea is that the pageblock maintains contiguity for the largest size you routinely expect to allocate.
The page allocator is very passive right now, and it doesn't work super reliably. But even in the current regime, smaller blocks have a better chance of containment.
Quoted 6 lines
For example, when the ever-growing page cache runs out of movable block space, it spills into unmovable free space. When the next unmovable request finds no space, it runs LRU reclaim - which is more likely to free space in one of the many movable blocks. And so the next block is poisoned. Smaller blocks have a better chance of filling up natively, means less pressure to spill into incompatible ones.
It strikes me that a lot of THP problems are really compaction problems.
Quoted 5 lines
And the higher
min_free_kbytes, the more likely there are still native options when the zones are down to the watermarks. E.g. better odds there is still unmovable free space, you just need to reclaim some movable/reclaimable space elsewhere to satisfy the watermarks.
I mean in general, yes, but the question is how much. This isn't really an argument for retaining what we have, and as you say below, it's not really clear what how much is.
Quoted 9 lines
I've been working on making this more robust with the huge page allocator /
defrag_modestuff: instead of falling back and poisoning a block, invoke reclaim/compaction to produce a neutral block that can be converted entirely.It's the same idea as the higher
min_free_kbytesand watermark boosting, but it is more targeted at the end result: readily available space in compatible or convertible blocks.
I wonder whether compaction needs to be improved before we can do that sanely?
Quoted 4 lines
But with that active regime, oversized pageblocks are even worse. You'd pay ongoing compaction work to produce a level of contiguity that you don't actually need.
I agree having 512 MiB page blocks on 64-KiB arm64 is a bad idea, but I feel like the entire mechanism needs some degree of rethinking before that kind of change is done.
Like I said in my other reply, I see this problem as consisting of the short term (resolving real world pain people experience now) vs. the long term (reworking compaction, reclaim, and how page blocks function).
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?
Quoted 2 lines
I'm just wondering what the easiest way forward is to fix the ARM 64k page problem.
I mean you're not alone :) we've had a whole host of proposals all of which were rejected, and you just rejected another...
This is why I suggested the 'if you are concerned about reserves being >X then cap them at X' proposal.
(v2 of this patch)
https://lore.kernel.org/linux-mm/[email protected]/
That way nobody is changed, and only really huge page size gets capped.
I mean that's still a possible short-term way forward given this approach was rejected?
Quoted 12 lines
Yes, optimally, reclaim would work to satisfy compaction space by itself. We've seen it fail at that before, though.
How critical
set_recommended_min_free_kbytes()is today is a question that neither of us has a clear answer to. It's from 2011 and a lot has changed. However, knowing Andrea, I'm willing to bet he added this based on seeing a need in testing data. And I would actually expect it to work better now with proactive compaction, since that has a better chance of turning low-order chunks of that volume into pageblocks that can be converted instead of needing a poisoning steal.
Quoted 2 lines
It's a change of long-standing behavior for everybody. It has a regression risk and requires careful evaluation and testing.
See other mail for thoughts on this :) I mean I share your concerns, but I worry that we continue to persist in doing things because 'that's how they were always done'.
Quoted 4 lines
Meanwhile, adjusting the pageblock size on 64k page arm configs has a much smaller blast radius, appears to be the right move ANYWAY given what pageblocks are for, and makes the
min_free_kbytesa non-issue.
Well it doesn't make it a non-issue, it redefines the size of page blocks while the rest of the THP code continues to treat PMDs as first class citizens.
I wonder if Kiryl's work to make PMD _not_ be the 1st class citizen of THP will make such a change more reasonable.
Perhaps he has thoughts?
--
Cheers, Lorenzo
Zi Yan
reviewerQuoted 9 lines
If you look at
try_to_claim_block(), you need half of a block to be free or compatible with the requested migratetype in order to convert it. When memory is full, LRU pages are scattered all over, and compaction is not involved (order-0), this gets more difficult the bigger the block is. You can get into a situation where LRU reclaim will not clear sufficient room for conversion in any given pageblock anymore and you're stuck with the type distribution. A large share of buddy requests then go permanently through the slower fallback path.
I think we are digressing from the min_free_kbytes discussion, but it is good to understand the limitations of our compaction and reclaim. Nimrod is going to run some workloads to show us how his patch affect 4KB/2MB THP systems. Based on the min_free_kbytes calculation, this patch has no effect (on 4KB base page system) for a single node system with 272GB memory and a two-node system with 484GB memory. So smaller machines would see the impact of this patch. For example, a 16GB single node machine will have 16MB min_free_kbytes instead of 66MB.
Quoted 36 lines
Usama knows more about this, but we have seen this on GB300 hosts, and have JUST started to deploy kernels with smaller pageblocks (2M).
Quoted 7 lines
Basically it indicates at some point kernel allocates a lot of unmovable pages that use many 512MB pageblocks and the life time of these unmovable pages are so diverse, leading to all these pageblocks remain unmovable and free pages spread across all these pageblocks. I thought bigger pageblocks can keep unmovable pages constrained within fewer pageblocks, leaving more contiguous free memory.
The idea is that the pageblock maintains contiguity for the largest size you routinely expect to allocate.
The page allocator is very passive right now, and it doesn't work super reliably. But even in the current regime, smaller blocks have a better chance of containment.
For example, when the ever-growing page cache runs out of movable block space, it spills into unmovable free space. When the next unmovable request finds no space, it runs LRU reclaim - which is more likely to free space in one of the many movable blocks. And so the next block is poisoned. Smaller blocks have a better chance of filling up natively, means less pressure to spill into incompatible ones.
And the higher
min_free_kbytes, the more likely there are still native options when the zones are down to the watermarks. E.g. better odds there is still unmovable free space, you just need to reclaim some movable/reclaimable space elsewhere to satisfy the watermarks.I've been working on making this more robust with the huge page allocator /
defrag_modestuff: instead of falling back and poisoning a block, invoke reclaim/compaction to produce a neutral block that can be converted entirely.
Quoted 7 lines
It's the same idea as the higher
min_free_kbytesand watermark boosting, but it is more targeted at the end result: readily available space in compatible or convertible blocks.But with that active regime, oversized pageblocks are even worse. You'd pay ongoing compaction work to produce a level of contiguity that you don't actually need.
OK, digress to the compaction and reclaim discussion. Isn't this 512MB pageblock size issue also telling us 1GB THP allocation will not work at all? It is a proxy of 1GB super-pageblock.
I need to spend some time on it to understand all the heuristics and see if anything can be improved.
A quick chat with Claude results in some items for me to explore:
1. reduce pageblock conversion threshold from half of the pageblock. That can reduce the number of slowpath entries.
2. make reclaim pageblock aware, so when an order-0 unmovable page allocation incurs a reclaim, the reclaim will try to get free movable pages from an unmovable pageblock first.
3. in addition to produce a neutral block, compaction/reclaim need to repair unmovable blocks by getting rid of movable pages. Proactive compaction should do that too.
Quoted 25 lines
Quoted 10 lines
Quoted 3 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?I'm just wondering what the easiest way forward is to fix the ARM 64k page problem.
Yes, optimally, reclaim would work to satisfy compaction space by itself. We've seen it fail at that before, though.
How critical
set_recommended_min_free_kbytes()is today is a question that neither of us has a clear answer to. It's from 2011 and a lot has changed. However, knowing Andrea, I'm willing to bet he added this based on seeing a need in testing data. And I would actually expect it to work better now with proactive compaction, since that has a better chance of turning low-order chunks of that volume into pageblocks that can be converted instead of needing a poisoning steal.
Quoted 2 lines
It's a change of long-standing behavior for everybody. It has a regression risk and requires careful evaluation and testing.
Not for 4KB systems with large memory, since khugepaged's min_free_kbytes is capped at 66MB for one-node, 88MB for two-node, and 44MB + N*22MB for N-node. For a GB300 host you mentioned above, it has 494GB memory and with pageblock size set to 2MB, it should be a nop, right?
Quoted 4 lines
Meanwhile, adjusting the pageblock size on 64k page arm configs has a much smaller blast radius, appears to be the right move ANYWAY given what pageblocks are for, and makes the
min_free_kbytesa non-issue.
Signature · 2 lines
Best Regards, Yan, Zi
Johannes Weiner
reviewerQuoted 3 lines · Zi Yan
OK, digress to the compaction and reclaim discussion. Isn't this 512MB pageblock size issue also telling us 1GB THP allocation will not work at all? It is a proxy of 1GB super-pageblock.
Yes. I doubt this will work well with the conventional page allocator.
If you recall, when Usama floated 1G THP, one of the first questions was how do you allocate this. Rik's gigablock patches are an interesting push in that direction. CMA as a stop-gap measure.
Meanwhile I'm trying to also make the native pageblock defragmentation better to make order-9 requests more reliable to allocate at runtime.
Both of these efforts are kind of complicated by this config saying pageblock is now half a gig. It's too large for gigablocks. It's too large for watermarking/pageblock-aware reserves (as this patch shows).
We're struggling with order-9 blocks and you're saying this should work fine with order-13 blocks - with a 16x larger page - but with defrag reserves that are reasonable in absolute bytes. ;)
Quoted 7 lines
I need to spend some time on it to understand all the heuristics and see if anything can be improved.
A quick chat with Claude results in some items for me to explore:
1. reduce pageblock conversion threshold from half of the pageblock. That can reduce the number of slowpath entries.
That means you have pageblocks that contain a supermajority of a foreign type. What does the block type mean at that point?
Quoted 7 lines
2. make reclaim pageblock aware, so when an order-0 unmovable page allocation incurs a reclaim, the reclaim will try to get free movable pages from an unmovable pageblock first.
3. in addition to produce a neutral block, compaction/reclaim need to repair unmovable blocks by getting rid of movable pages. Proactive compaction should do that too.
This makes a lot of sense. I believe Rik was looking at more directed evacuations of exactly that type. CCing him. You guys might be able to collaborate on this.
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 2 lines
Seems to me the excessive
min_free_kbytesis just a symptom of a deeper problem.Yes, our anti-fragmentation mechanism does not work as we expected, so that we need an excessive
min_free_kbytesto get khugepaged working. I wonder why reclaim cannot get the extra free memory instead of reserving it viamin_free_kbytes. Maybe we need a watermark boost when some consecutive THP allocations are seen to achieve similar effect of boostingmin_free_kbytes?I'm just wondering what the easiest way forward is to fix the ARM 64k page problem.
Yes, optimally, reclaim would work to satisfy compaction space by itself. We've seen it fail at that before, though.
How critical
set_recommended_min_free_kbytes()is today is a question that neither of us has a clear answer to. It's from 2011 and a lot has changed. However, knowing Andrea, I'm willing to bet he added this based on seeing a need in testing data. And I would actually expect it to work better now with proactive compaction, since that has a better chance of turning low-order chunks of that volume into pageblocks that can be converted instead of needing a poisoning steal.
Quoted 8 lines
Quoted 2 lines
It's a change of long-standing behavior for everybody. It has a regression risk and requires careful evaluation and testing.
Not for 4KB systems with large memory, since khugepaged's
min_free_kbytesis capped at 66MB for one-node, 88MB for two-node, and 44MB + N*22MB for N-node. For a GB300 host you mentioned above, it has 494GB memory and with pageblock size set to 2MB, it should be a nop, right?
For single node, normal minfree catches up with recommended at 256G. For two nodes, it catches up at 484G.
I would say that affects the majority, or at least a sizable share, of machines out there today.
Barry Song
reviewerOn Thu, Sep 3, 2026 at 12:04 AM Johannes Weiner <[email protected]> wrote:
[...]
Quoted 25 lines
The idea is that the pageblock maintains contiguity for the largest size you routinely expect to allocate.
The page allocator is very passive right now, and it doesn't work super reliably. But even in the current regime, smaller blocks have a better chance of containment.
For example, when the ever-growing page cache runs out of movable block space, it spills into unmovable free space. When the next unmovable request finds no space, it runs LRU reclaim - which is more likely to free space in one of the many movable blocks. And so the next block is poisoned. Smaller blocks have a better chance of filling up natively, means less pressure to spill into incompatible ones.
And the higher
min_free_kbytes, the more likely there are still native options when the zones are down to the watermarks. E.g. better odds there is still unmovable free space, you just need to reclaim some movable/reclaimable space elsewhere to satisfy the watermarks.I've been working on making this more robust with the huge page allocator /
defrag_modestuff: instead of falling back and poisoning a block, invoke reclaim/compaction to produce a neutral block that can be converted entirely.
Quoted 7 lines
It's the same idea as the higher
min_free_kbytesand watermark boosting, but it is more targeted at the end result: readily available space in compatible or convertible blocks.But with that active regime, oversized pageblocks are even worse. You'd pay ongoing compaction work to produce a level of contiguity that you don't actually need.
Hi Johannes,
This could also be true when running 16KB or other mTHP sizes, such as 32KB or 64KB, on a system with 4KB base pages and 2MB pageblocks.
Proactive compaction would try to create 2MB pageblocks, even by migrating mTHPs, but 2MB isn't actually needed for large folios on the system.
So here is an mTHP-friendly compaction RFC from Bo (Cc: Bo):
https://lore.kernel.org/linux-mm/[email protected]/
The idea is to let compaction create mTHP orders rather than 2MB pageblocks, with much lower cost and greater efficiency.
I'd be interested to hear if you have any comments on it.
Best Regards
Barry
Quoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
Tested this on Grace.
Quoted 284 lines
---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/RFC v1:
https://lore.kernel.org/r/[email protected]/--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)include/linux/khugepaged.h +0
-5@@ -15,7 +15,6 @@ extern void __khugepaged_enter(struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm);extern void khugepaged_enter_vma (struct vm_area_struct *vma,vm_flags_t vm_flags); -extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void );void collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd);@@ -47,10 +46,6 @@ static inline void collapse_pte_mapped_thp(struct mm_struct *mm, { } -static inline void khugepaged_min_free_kbytes_update (void ) -{ -} -static inline bool current_is_khugepaged (void ) {return false;mm/huge_memory.c +0
-14@@ -499,13 +499,6 @@ static ssize_t enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count; }@@ -735,13 +728,6 @@ static ssize_t anon_enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count;mm/internal.h +0
-8@@ -689,11 +689,6 @@ int user_proactive_reclaim(char *buf,*/ pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address); -/* -* in mm/khugepaged.c -*/ -void set_recommended_min_free_kbytes (void ); -/* * in mm/page_alloc.c */ @@ -701,11 +696,8 @@ void set_recommended_min_free_kbytes(void);extern char *const zone_names[MAX_NR_ZONES ]; -extern int min_free_kbytes;extern int defrag_mode; -void setup_per_zone_wmarks (void ); -void calculate_min_free_kbytes (void );int __meminit init_per_zone_wmark_min (void );extern int __isolate_free_page (struct page *page,unsigned int order);mm/khugepaged.c +0
-60@@ -27,7 +27,6 @@#include <asm/tlb.h> #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" enum scan_result {@@ -3086,57 +3085,6 @@ static int khugepaged(void *none)return 0 ; } -void set_recommended_min_free_kbytes (void ) -{ -struct zone *zone; -int nr_zones =0 ; -unsigned long recommended_min; - -if (!hugepage_enabled ()) { -calculate_min_free_kbytes (); - goto update_wmarks; - } - -for_each_populated_zone (zone) { -/* -* We don't need to worry about fragmentation of -* ZONE_MOVABLE since it only has movable pages. -*/ -if (zone_idx (zone) >gfp_zone (GFP_USER )) -continue ; - - nr_zones++; - } - -/* Ensure 2 pageblocks are free to assist fragmentation avoidance */ - recommended_min = pageblock_nr_pages * nr_zones *2 ; - -/* -* Make sure that on average at least two pageblocks are almost free -* of another type, one for a migratetype to fall back to and a -* second to avoid subsequent fallbacks of other types There are 3 -* MIGRATE_TYPES we care about. -*/ - recommended_min += pageblock_nr_pages * nr_zones * -MIGRATE_PCPTYPES *MIGRATE_PCPTYPES ; - -/* don't ever allow to reserve more than 5% of the lowmem */ - recommended_min =min (recommended_min, - (unsigned long )nr_free_buffer_pages () /20 ); - recommended_min <<= (PAGE_SHIFT -10 ); - -if (recommended_min > min_free_kbytes) { -if (user_min_free_kbytes >=0 ) -pr_info_ratelimited ("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n" , - min_free_kbytes, recommended_min); - - min_free_kbytes = recommended_min; - } - -update_wmarks: -setup_per_zone_wmarks (); -} -int start_stop_khugepaged (void ) {guard (mutex)(&khugepaged_mutex);@@ -3160,17 +3108,9 @@ int start_stop_khugepaged(void)kthread_stop (khugepaged_thread); khugepaged_thread =NULL ; } -set_recommended_min_free_kbytes ();return 0 ; } -void khugepaged_min_free_kbytes_update (void ) -{ -guard (mutex)(&khugepaged_mutex); -if (hugepage_enabled () && khugepaged_thread) -set_recommended_min_free_kbytes (); -} -bool current_is_khugepaged (void ) {return kthread_func (current) == khugepaged;mm/page_alloc.c +4
-7@@ -50,7 +50,6 @@#include <linux/ftrace.h> #include <linux/lockdep.h> #include <linux/psi.h> -#include <linux/khugepaged.h> #include <linux/delayacct.h> #include <linux/cacheinfo.h> #include <linux/pgalloc_tag.h> @@ -272,8 +271,8 @@ const char * const migratetype_names[MIGRATE_TYPES] = {#endif }; -int min_free_kbytes =1024 ; -int user_min_free_kbytes =-1 ; +static int min_free_kbytes =1024 ; +static int user_min_free_kbytes =-1 ;static int watermark_boost_factor__read_mostly =15000 ;static int watermark_scale_factor =10 ;int defrag_mode;@@ -6663,7 +6662,7 @@ static void __setup_per_zone_wmarks(void)* Ensures that the watermark[min,low,high] values for each zone are set * correctly with respect to min_free_kbytes. */ -void setup_per_zone_wmarks (void ) +static void setup_per_zone_wmarks (void ) {struct zone *zone;static DEFINE_SPINLOCK (lock);@@ -6704,7 +6703,7 @@ void setup_per_zone_wmarks(void)* 8192MB: 11584k * 16384MB: 16384k */ -void calculate_min_free_kbytes (void ) +static void calculate_min_free_kbytes (void ) {unsigned long lowmem_kbytes;int new_min_free_kbytes;@@ -6732,8 +6731,6 @@ int __meminit init_per_zone_wmark_min(void)setup_min_slab_ratio ();#endif -khugepaged_min_free_kbytes_update (); -return 0 ; }postcore_initcall (init_per_zone_wmark_min)mm/page_alloc.h +0
-2@@ -227,8 +227,6 @@ void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order); -extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid,nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \mm/shmem.c +0
-7@@ -5782,13 +5782,6 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,int err =start_stop_khugepaged ();if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode hasn't changed -* to preserve the legacy behavior, as this is always called -* inside start_stop_khugepaged(). -*/ -set_recommended_min_free_kbytes (); }return count;
Quoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
Code all looks good, ran locally and with 16 KiB page size arm64 + THP enabled (tried with defrag -> madvise):
Before:
$ cat /proc/sys/vm/min_free_kbytes 360448
After:
$ cat /proc/sys/vm/min_free_kbytes 11472
And no delta on THP selftests.
$ cat /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed 936 $ cat /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans 2002
So (32 MiB) THPs are coming no problem with low memory pressure and without needing egregious defrag options, i.e. proactive compaction is doing its job fine.
See https://lore.kernel.org/linux-mm/apgdNQgHw4BW3bPG@gremlin/ for my thoughts on this approach in general but TL;DR am in favour :)
So LGTM and:
Quoted 285 lines
---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/RFC v1:
https://lore.kernel.org/r/[email protected]/--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)include/linux/khugepaged.h +0
-5@@ -15,7 +15,6 @@ extern void __khugepaged_enter(struct mm_struct *mm);extern void __khugepaged_exit (struct mm_struct *mm);extern void khugepaged_enter_vma (struct vm_area_struct *vma,vm_flags_t vm_flags); -extern void khugepaged_min_free_kbytes_update (void );extern bool current_is_khugepaged (void );void collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd);@@ -47,10 +46,6 @@ static inline void collapse_pte_mapped_thp(struct mm_struct *mm, { } -static inline void khugepaged_min_free_kbytes_update (void ) -{ -} -static inline bool current_is_khugepaged (void ) {return false;mm/huge_memory.c +0
-14@@ -499,13 +499,6 @@ static ssize_t enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count; }@@ -735,13 +728,6 @@ static ssize_t anon_enabled_store(struct kobject *kobj,if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode didn't -* change, as the previous code always called -* start_stop_khugepaged() which does this internally. -*/ -set_recommended_min_free_kbytes (); }return count;mm/internal.h +0
-8@@ -689,11 +689,6 @@ int user_proactive_reclaim(char *buf,*/ pmd_t *mm_find_pmd (struct mm_struct *mm,unsigned long address); -/* -* in mm/khugepaged.c -*/ -void set_recommended_min_free_kbytes (void ); -/* * in mm/page_alloc.c */ @@ -701,11 +696,8 @@ void set_recommended_min_free_kbytes(void);extern char *const zone_names[MAX_NR_ZONES ]; -extern int min_free_kbytes;extern int defrag_mode; -void setup_per_zone_wmarks (void ); -void calculate_min_free_kbytes (void );int __meminit init_per_zone_wmark_min (void );extern int __isolate_free_page (struct page *page,unsigned int order);mm/khugepaged.c +0
-60@@ -27,7 +27,6 @@#include <asm/tlb.h> #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" enum scan_result {@@ -3086,57 +3085,6 @@ static int khugepaged(void *none)return 0 ; } -void set_recommended_min_free_kbytes (void ) -{ -struct zone *zone; -int nr_zones =0 ; -unsigned long recommended_min; - -if (!hugepage_enabled ()) { -calculate_min_free_kbytes (); - goto update_wmarks; - } - -for_each_populated_zone (zone) { -/* -* We don't need to worry about fragmentation of -* ZONE_MOVABLE since it only has movable pages. -*/ -if (zone_idx (zone) >gfp_zone (GFP_USER )) -continue ; - - nr_zones++; - } - -/* Ensure 2 pageblocks are free to assist fragmentation avoidance */ - recommended_min = pageblock_nr_pages * nr_zones *2 ; - -/* -* Make sure that on average at least two pageblocks are almost free -* of another type, one for a migratetype to fall back to and a -* second to avoid subsequent fallbacks of other types There are 3 -* MIGRATE_TYPES we care about. -*/ - recommended_min += pageblock_nr_pages * nr_zones * -MIGRATE_PCPTYPES *MIGRATE_PCPTYPES ; - -/* don't ever allow to reserve more than 5% of the lowmem */ - recommended_min =min (recommended_min, - (unsigned long )nr_free_buffer_pages () /20 ); - recommended_min <<= (PAGE_SHIFT -10 ); - -if (recommended_min > min_free_kbytes) { -if (user_min_free_kbytes >=0 ) -pr_info_ratelimited ("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n" , - min_free_kbytes, recommended_min); - - min_free_kbytes = recommended_min; - } - -update_wmarks: -setup_per_zone_wmarks (); -} -int start_stop_khugepaged (void ) {guard (mutex)(&khugepaged_mutex);@@ -3160,17 +3108,9 @@ int start_stop_khugepaged(void)kthread_stop (khugepaged_thread); khugepaged_thread =NULL ; } -set_recommended_min_free_kbytes ();return 0 ; } -void khugepaged_min_free_kbytes_update (void ) -{ -guard (mutex)(&khugepaged_mutex); -if (hugepage_enabled () && khugepaged_thread) -set_recommended_min_free_kbytes (); -} -bool current_is_khugepaged (void ) {return kthread_func (current) == khugepaged;mm/page_alloc.c +4
-7@@ -50,7 +50,6 @@#include <linux/ftrace.h> #include <linux/lockdep.h> #include <linux/psi.h> -#include <linux/khugepaged.h> #include <linux/delayacct.h> #include <linux/cacheinfo.h> #include <linux/pgalloc_tag.h> @@ -272,8 +271,8 @@ const char * const migratetype_names[MIGRATE_TYPES] = {#endif }; -int min_free_kbytes =1024 ; -int user_min_free_kbytes =-1 ; +static int min_free_kbytes =1024 ; +static int user_min_free_kbytes =-1 ;static int watermark_boost_factor__read_mostly =15000 ;static int watermark_scale_factor =10 ;int defrag_mode;@@ -6663,7 +6662,7 @@ static void __setup_per_zone_wmarks(void)* Ensures that the watermark[min,low,high] values for each zone are set * correctly with respect to min_free_kbytes. */ -void setup_per_zone_wmarks (void ) +static void setup_per_zone_wmarks (void ) {struct zone *zone;static DEFINE_SPINLOCK (lock);@@ -6704,7 +6703,7 @@ void setup_per_zone_wmarks(void)* 8192MB: 11584k * 16384MB: 16384k */ -void calculate_min_free_kbytes (void ) +static void calculate_min_free_kbytes (void ) {unsigned long lowmem_kbytes;int new_min_free_kbytes;@@ -6732,8 +6731,6 @@ int __meminit init_per_zone_wmark_min(void)setup_min_slab_ratio ();#endif -khugepaged_min_free_kbytes_update (); -return 0 ; }postcore_initcall (init_per_zone_wmark_min)mm/page_alloc.h +0
-2@@ -227,8 +227,6 @@ void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags);extern bool free_pages_prepare (struct page *page,unsigned int order); -extern int user_min_free_kbytes; -struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order,int nid,nodemask_t *nodemask,unsigned int alloc_flags);#define __alloc_frozen_pages (...) \mm/shmem.c +0
-8@@ -5782,13 +5782,6 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,int err =start_stop_khugepaged ();if (err)return err; - }else { -/* -* Recalculate watermarks even when the mode hasn't changed -* to preserve the legacy behavior, as this is always called -* inside start_stop_khugepaged(). -*/ -set_recommended_min_free_kbytes (); }return count; --
Quoted 2 lines
2.45.0
--
Cheers, Lorenzo
Quoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
Just to summarize my take from the subthread with Zi: the premise of this patch is to roll the regression dice on every THP setup out there because certain ARM configurations result in a questionable pageblock size.
I'm not against carefully evaluating and testing out today's need for set_recommended_min_free_kbytes() in real world examples. But this is not that.
Lorenzo Stoakes (ARM)
reviewerQuoted 9 lines · Johannes Weiner
Just to summarize my take from the subthread with Zi: the premise of this patch is to roll the regression dice on every THP setup out there because certain ARM configurations result in a questionable pageblock size.
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>
Well you don't have to listen to me any more as ex-THP M ;) but my 2 pence...
Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
It feels like in certain areas we paint ourselves into a corner where everybody's too scared to change anything until we're sure nobody in the world is broken*.
And so we continue to ride the merry-go-round of proposals/rejections indefinitely.
All the while regressions in tip kernel are a regular occurrence (yes we don't want that, but they happen), and they are resolved as they arise.
I wonder if we aren't limiting ourselves by thinking this way.
Michal's proposal was that the original code was written _long_ before improvements in the compaction algorithm and fails to account for those.
It seems odd to retain the same constraints given the rest of the kernel has changed.
Perhaps a compromise would be to put the ability to disable this behind a config option or maybe a kernel arg? Of course that becomes something of a uAPI... but at least it gives the option to constrian this for those who want it.
--
Cheers, Lorenzo
* I think, for instance, about how the hell we can test changes to the compaction algorithm in a way that's sane and acceptable to people?
Though that is something that really _does_ need it, as it's vastly more complicated and impactful than the proposed change here.
I'm not sure what the answer is really :) ideally we'd have some vast farm of machines doing representative workloads with software that does careful statistically-signficant analysis of how they perform, but that seems rather like a pipe dream.
Johannes Weiner
reviewerQuoted 5 lines · Lorenzo Stoakes (ARM)
Quoted 4 lines · Johannes Weiner
Just to summarize my take from the subthread with Zi: the premise of this patch is to roll the regression dice on every THP setup out there because certain ARM configurations result in a questionable pageblock size.
Quoted 8 lines
Quoted 5 lines
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>Well you don't have to listen to me any more as ex-THP M ;) but my 2 pence...
I'll always listen to you, Lorenzo. <3
Quoted 5 lines
Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change. Or be more specific about which changes obviated the additional pageblock reserves, and how.
Quoted 3 lines
It feels like in certain areas we paint ourselves into a corner where everybody's too scared to change anything until we're sure nobody in the world is broken*.
I'm fine with calculated risks, actually. A bit more surprised that Michal was so readily on board with this :)
Quoted 6 lines
And so we continue to ride the merry-go-round of proposals/rejections indefinitely.
All the while regressions in tip kernel are a regular occurrence (yes we don't want that, but they happen), and they are resolved as they arise.
Quoted 7 lines
I wonder if we aren't limiting ourselves by thinking this way.
Michal's proposal was that the original code was written _long_ before improvements in the compaction algorithm and fails to account for those.
It seems odd to retain the same constraints given the rest of the kernel has changed.
That's a great motivation to take a closer look at whether we still need it. I'm not attached to anything that's plausibly shown to be unnecessary.
But I think there are levels of argument quality:
1. This code is old as in time
2. This code is old as in the surroundings have changed
3. This code is not needed due to sha1, sha2, sha3 supplanting it thusly: ...
4. This code is not making a difference in represenative tests
The patch is at 2 and I would really prefer we get to 3 or 4. That's not the same as saying we should stop making changes.
And honestly, while we tend to claim we want 4 for everything, we're happy many times to roll the dice at 3 - iff the story is specific and plausible. And I believe that touches on your footnote ;)
But let's talk about plausible. Because I'm reading what set_recommended_min_free_kbytes() does, along with its comments, and can't help but think that this still applies in the current world.
The compaction aid is/was always incidental. Yeah it would suck if that is/was (still) an implicit dependency, but ridding us of that could be a more deeper-reaching change than you would want to make to fix that pressing reserves issue with 64k pages.
The fallback avoidance reasoning OTOH still seems cogent to me. The mechanism it references is about passive fragmentation avoidance by giving the allocator placement options, before compaction gets involved. That makes sense in how I understand page_alloc.c today.
Quoted 4 lines
Perhaps a compromise would be to put the ability to disable this behind a config option or maybe a kernel arg? Of course that becomes something of a uAPI... but at least it gives the option to constrian this for those who want it.
I can't force you to engage with the idea of capping the pageblock instead. But I'm still kind of dying to know what the reluctance is ;) And I apologize if I missed any prior arguments on this specifically.
Lorenzo Stoakes (ARM)
reviewerQuoted 10 lines · Johannes Weiner
Quoted 9 lines · Lorenzo Stoakes (ARM)
Quoted 8 lines · Johannes Weiner
Just to summarize my take from the subthread with Zi: the premise of this patch is to roll the regression dice on every THP setup out there because certain ARM configurations result in a questionable pageblock size.
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Nacked-by Johannes Weiner <[email protected]>Well you don't have to listen to me any more as ex-THP M ;) but my 2 pence...
I'll always listen to you, Lorenzo. <3
<3 ;)
Quoted 4 lines
Quoted 4 lines
Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
Quoted 8 lines
Quoted 2 lines
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change. Or be more specific about which changes obviated the additional pageblock reserves, and how.
Looking over the sub-thread (correct me if I'm wrong) the issues seem to be:
- 512 MB pageblocks become unmoveable quicker than expected
- When trying to convert a pageblock in try_to_claim_block() 256 MiB is required to be of the desired migratetype, and this is difficult to achieve vs. 1 MiB (yes clearly :)
- AI training checkpointing was a problematic workload - big latency spikes and timeouts. Tonnes of unmoveable memory, order-0 allocations falling back to MIGRATE_MOVABLE (ugh), exhibiting try_to_claim_block() symptoms above.
I hear all of this, and to be clear - this kind of real-world data, at scale, is the kind of thing we should base decisions on more than anything else.
Reality > theory every time (and the more you look into the kernel you more you realise it's a tower of heuristics anyway, especially in classical reclaim :)
I guess what you're trying to say here is the only way in these circumstances to make headway would be to have more memory reserved.
But is that the right conclusion? Aren't you still screwed once those reserves are chomped up?
Or are you saying the increased watermark levels gets you effective kcompactd/kswapd sooner?
The TL;DR for me is - you have a workload that's broken already with larger pageblock size - maybe you could test that with/without this patch and see if it really does help?
Anyway it seems to me all of this is essentially a (valid!) critique of assumptions backed into the page allocator code.
Which is again what I think about this change - we need a short-term fix of _some kind,_ and a longer term fix in compaction, page alloc, reclaim.
Quoted 28 lines
Quoted 4 lines
It feels like in certain areas we paint ourselves into a corner where everybody's too scared to change anything until we're sure nobody in the world is broken*.
I'm fine with calculated risks, actually. A bit more surprised that Michal was so readily on board with this :)
Quoted 6 lines
And so we continue to ride the merry-go-round of proposals/rejections indefinitely.
All the while regressions in tip kernel are a regular occurrence (yes we don't want that, but they happen), and they are resolved as they arise.
Quoted 7 lines
I wonder if we aren't limiting ourselves by thinking this way.
Michal's proposal was that the original code was written _long_ before improvements in the compaction algorithm and fails to account for those.
It seems odd to retain the same constraints given the rest of the kernel has changed.
That's a great motivation to take a closer look at whether we still need it. I'm not attached to anything that's plausibly shown to be unnecessary.
But I think there are levels of argument quality:
Quoted 7 lines
1. This code is old as in time
2. This code is old as in the surroundings have changed
3. This code is not needed due to sha1, sha2, sha3 supplanting it thusly: ...
4. This code is not making a difference in represenative testsThe patch is at 2 and I would really prefer we get to 3 or 4. That's not the same as saying we should stop making changes.
1 nearly always implies 2 because computers become radically different over time (cache hierarchy + latencies, memory model, I/O bandwidth etc.)
3 feels like a totally different category - there's just no heuristic in mm that would match that.
and 4 I think is the crux of it - we lack any such agreed-upon 'representative tests' and I fear that getting them might be an impossible problem (TM) as people will argue endlessly about what's representative and what's not.
(I'm reminded of discussions at LSF about MGLRU and how exactly we decide classic vs. MGLRU - Kairu had some nice ideas - but it's another of those 'will we ever agree on this?' situations).
Quoted 4 lines
And honestly, while we tend to claim we want 4 for everything, we're happy many times to roll the dice at 3 - iff the story is specific and plausible. And I believe that touches on your footnote ;)
I really feel 3 is a different class of problem. When would that apply to a heuristic?
Quoted 4 lines
But let's talk about plausible. Because I'm reading what
set_recommended_min_free_kbytes()does, along with its comments, and can't help but think that this still applies in the current world.
Hmm really? The calculation is:
min_pages = nr_zones * nr_pblock_pgs * (2 + MIGRATE_PCPTYPES ^ 2)
But __setup_per_zone_wmarks() does:
tmp = min_pages * (managed pages) / (total lowmem pages)
Looking at an x86 box:
dma 3840 <- give an extra nr_pblock_pgs * (2 + MIGRATE_PCPTYPES ^ 2) pgs
dma32 592222 <- give an extra nr_pblock_pgs * (2 + MIGRATE_PCPTYPES ^ 2) pgs
normal 15682190 <- give an extra nr_pblock_pgs * (2 + MIGRATE_PCPTYPES ^ 2) pgs
So you end up with 6 * nr_pblock_pgs + 3 * MIGRATE_PCPTYPES ^2 but 96.3% of that goes to ZONE_NORMAL, which is now arbitrarily bumped up by a factor of ~3 for... what reason? And dma/dma32 get tiny fractions (more understandable).
So that definitely doesn't make sense.
Many arm64 machines only have one zone and solve things that way, but that's effectively by mistake (the algorithm says scale by zone irrelevant of further scaling).
And then the arbitrary 5% rule - it's mostly irrelevant for modern systems, and a 64 KiB page size system is likely to run significant RAM (my 16 KiB page system with 352 MiB min_kbytes would need < 8 GiB to get capped.
So I honestly think that sucks too.
And also - MADV_COLLAPSE can get you THP _even with all the /sys/kernel/mm/transparent_hugepage knobs turned off_, i.e. with no additional reserves. So turning it on/off on based on the knobs seems broken in itself, at least partly.
I guess Michal can expand on why he feels that perhaps even the nr_pblock_pgs * (2 + MIGRATE_PCPTYPES ^2).
Now that's not to say the non-THP heuristic isn't open to scrutiny too :)
Quoted 5 lines
The compaction aid is/was always incidental. Yeah it would suck if that is/was (still) an implicit dependency, but ridding us of that could be a more deeper-reaching change than you would want to make to fix that pressing reserves issue with 64k pages.
I don't think it is incidental?
How else do you get huge pages under memory pressure? It feels like compaction + defrag are intimately linked to THP.
Quoted 5 lines
The fallback avoidance reasoning OTOH still seems cogent to me. The mechanism it references is about passive fragmentation avoidance by giving the allocator placement options, before compaction gets involved. That makes sense in how I understand
page_alloc.ctoday.
See above.
I guess we have to answer whether the heuristic in calculate_min_free_kbytes() suffices vs. set_recommended_min_free_kbytes()...
That's not to say we couldn't just change calculate_min_free_kbytes() to a sensible heuristic.
I guess the v2 hard cap wasn't desirable, but since the heuristic already caps regardless of the rest of the arithmetic maybe:
- Cap to a hardcoded max reserve value [v2 of this series]
- Cap to a bigger % of free memory?
Are short-term options here?
And on the 'changes are outlandish' side of things, well the vast majority of users are at 4 KiB base page size, 2 MiB PMD - so maybe we could retain the exact same values you have now in that case, and get experimental with bigger page sizes?
Would that then carry less risk in your view?
Quoted 9 lines
Quoted 5 lines
Perhaps a compromise would be to put the ability to disable this behind a config option or maybe a kernel arg? Of course that becomes something of a uAPI... but at least it gives the option to constrian this for those who want it.
I can't force you to engage with the idea of capping the pageblock instead. But I'm still kind of dying to know what the reluctance is ;) And I apologize if I missed any prior arguments on this specifically.
Because fragmentation avoidance, compaction and the highatomic/boost reserves are all at the granularity of page blocks, so a PMD spanning several pageblocks is not given any help at all.
The underlying problem is that "THP == PMD" is a fiction at 64 KiB.
A 512 MiB PMD is not a useful default unit, so both the pageblock size and the reserve derived from it are sized for something nobody wants.
You can't have it both ways:
1. Complain the defaults reserve too much on 64 KiB while THP defaults to madvise/always for a 512 MiB PMD you'll never use, with no effort to set mTHP sizes instead.
2. "Fix" it by shrinking pageblocks below PMD, keeping the reserve whose sole purpose is PMD THP allocation while removing its ability to serve that purpose.
Really the issue is that actually PMD THP is _not_ the primary need here. For most normal uses at 16 KiB page it's dubious at best, at 64 KiB page size it's ludicrous.
What we want to do is to stop treating PMD as the unit-of-measure everywhere instead of hacking stuff in. Kiryl's work on THP promises us this.
(Note that it's not as simple btw as saying 'oh mTHP is available for 64 KiB page size, let's reduce reserves to scale to mTHP sizes' - you require mappings and alignment at PMD size even with mTHP, so it's still useless :)
We want to account for the fact we want defragmentation for:
- PMD THP at smaller page size
- mTHP
- page cache large folios
- drivers-wanting-higher-order allocations
So I think the long-term solution is to stop pretending and to do this.
Also we have this weird thing of CONFIG_TRANSPARENT_HUGEPAGE gating large folio stuff in general.
Oh and as noted above MADV_COLLAPSE can get you huge pages even when all the knobs at /sys/kernel/mm/transparent_hugepage are switched off - so _without_ any additional reserves :)
I feel the short-term solution is to attack the issue where it exists - the uptick in reserves taking a pragmatic approach. And long term the above.
Anyway a request:
can you actually run the problematic workloads with and without this patch and tell us whether you see a meaningful regression?
Let's get some data for this :)
--
Cheers, Lorenzo
Johannes Weiner
reviewerQuoted 18 lines · Lorenzo Stoakes (ARM)
Looking over the sub-thread (correct me if I'm wrong) the issues seem to be:
- 512 MB pageblocks become unmoveable quicker than expected
- When trying to convert a pageblock in
try_to_claim_block()256 MiB is required to be of the desired migratetype, and this is difficult to achieve vs. 1 MiB (yes clearly :)- AI training checkpointing was a problematic workload - big latency spikes and timeouts. Tonnes of unmoveable memory, order-0 allocations falling back to
MIGRATE_MOVABLE(ugh), exhibitingtry_to_claim_block()symptoms above.I hear all of this, and to be clear - this kind of real-world data, at scale, is the kind of thing we should base decisions on more than anything else.
Reality > theory every time (and the more you look into the kernel you more you realise it's a tower of heuristics anyway, especially in classical reclaim :)
Quoted 8 lines
I guess what you're trying to say here is the only way in these circumstances to make headway would be to have more memory reserved.
But is that the right conclusion? Aren't you still screwed once those reserves are chomped up?
Or are you saying the increased watermark levels gets you effective kcompactd/kswapd sooner?
+1 Exactly! The watermarks sit on top of that reserve. Both background reclaim and direct reclaim are thresholded such that there are always a few pageblocks worth of free space for the allocator to choose from, thus reducing the risk of fallbacks and block poisoning.
Quoted 6 lines
The TL;DR for me is - you have a workload that's broken already with larger pageblock size - maybe you could test that with/without this patch and see if it really does help?
Anyway it seems to me all of this is essentially a (valid!) critique of assumptions backed into the page allocator code.
This part I don't quite follow. Why is the page allocator doing anything wrong here?
You tell it your largest routine allocation size. It groups smaller allocations by their ability to move into buckets of this size, coordinates a headroom of buckets for non-violating placements, and ensures reclaim kicks in when that headroom depletes.
You're giving it a very large bucket size and are not happy with the headroom that commands.
[ I'll reply to the other points in your email later. ]
Lorenzo Stoakes (ARM)
reviewerQuoted 25 lines · Johannes Weiner
Quoted 24 lines · Lorenzo Stoakes (ARM)
Looking over the sub-thread (correct me if I'm wrong) the issues seem to be:
- 512 MB pageblocks become unmoveable quicker than expected
- When trying to convert a pageblock in
try_to_claim_block()256 MiB is required to be of the desired migratetype, and this is difficult to achieve vs. 1 MiB (yes clearly :)- AI training checkpointing was a problematic workload - big latency spikes and timeouts. Tonnes of unmoveable memory, order-0 allocations falling back to
MIGRATE_MOVABLE(ugh), exhibitingtry_to_claim_block()symptoms above.I hear all of this, and to be clear - this kind of real-world data, at scale, is the kind of thing we should base decisions on more than anything else.
Reality > theory every time (and the more you look into the kernel you more you realise it's a tower of heuristics anyway, especially in classical reclaim :)
I guess what you're trying to say here is the only way in these circumstances to make headway would be to have more memory reserved.
But is that the right conclusion? Aren't you still screwed once those reserves are chomped up?
Quoted 7 lines
Quoted 2 lines
Or are you saying the increased watermark levels gets you effective kcompactd/kswapd sooner?
+1 Exactly! The watermarks sit on top of that reserve. Both background reclaim and direct reclaim are thresholded such that there are always a few pageblocks worth of free space for the allocator to choose from, thus reducing the risk of fallbacks and block poisoning.
Right.
Quoted 10 lines
Quoted 7 lines
The TL;DR for me is - you have a workload that's broken already with larger pageblock size - maybe you could test that with/without this patch and see if it really does help?
Anyway it seems to me all of this is essentially a (valid!) critique of assumptions backed into the page allocator code.
This part I don't quite follow. Why is the page allocator doing anything wrong here?
In this bit of the email we were talking about a function that lives in page_alloc.c :)
Quoted 5 lines
You tell it your largest routine allocation size. It groups smaller allocations by their ability to move into buckets of this size, coordinates a headroom of buckets for non-violating placements, and ensures reclaim kicks in when that headroom depletes.
Yup, but as discussed if something straddles the buckets it's useless (it can only make guarantees about the buckets themselves), the target everything has in mind is PMD (since we are bumping the reserves for THP explicitly), and alignment/size requirements make smaller sizing useless for mTHP.
Quoted 3 lines
You're giving it a very large bucket size and are not happy with the headroom that commands.
Yup.
Quoted 2 lines
[ I'll reply to the other points in your email later. ]
--
Cheers, Lorenzo
Quoted 11 lines · Johannes Weiner
Quoted 10 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Johannes Weiner
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
Quoted 7 lines
Quoted 2 lines
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change.
Hi,
I tested this on an x86-64 (4K/2M) virtual machine with one NUMA node and 16 GiB online memory, using mmtests config-workload-thpchallenge-fio with THPCHALLENGE_MADV_HUGEPAGE=yes.min_free_kbytes was 16 MiB patched and 66 MiB unpatched.
I ran each kernel 30 times, rebooting before each run. The results did not show a regression in THP fault success rate or latency:
Average THP fault success (Percentage Faults Huge) increased from 13.07% unpatched to 13.34% patched, and average fault latency (Fault Latencies) decreased by 6.5%.
In contrast, compaction metrics were higher on average with the patch:
Compaction stalls: 1,635 -> 1,713 (+4.7%) Compaction failures: 1,350 -> 1,416 (+4.9%)
Compaction migrate scanned: 4,812,254 -> 5,587,587 (+16.1%)
Thanks,
Johannes Weiner
reviewerQuoted 34 lines · Nimrod Oren
Quoted 11 lines · Johannes Weiner
Quoted 10 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Johannes Weiner
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
Quoted 7 lines
Quoted 2 lines
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change.
Hi,
I tested this on an x86-64 (4K/2M) virtual machine with one NUMA node and 16 GiB online memory, using mmtests config-workload-thpchallenge-fio with THPCHALLENGE_MADV_HUGEPAGE=yes.
min_free_kbyteswas 16 MiB patched and 66 MiB unpatched.I ran each kernel 30 times, rebooting before each run. The results did not show a regression in THP fault success rate or latency:
Average THP fault success (Percentage Faults Huge) increased from 13.07% unpatched to 13.34% patched, and average fault latency (Fault Latencies) decreased by 6.5%.
Quoted 5 lines
In contrast, compaction metrics were higher on average with the patch:
Compaction stalls: 1,635 -> 1,713 (+4.7%) Compaction failures: 1,350 -> 1,416 (+4.9%)Compaction migrate scanned: 4,812,254 -> 5,587,587 (+16.1%)
A 2% increase in THP success bought with a 16.1% increase in compaction work looks like a sizable efficiency regression.
A scan efficiency drop is in line with expectations of what happens when non-frag placement reserves are taken from the allocator. A comparison of trace_mm_page_alloc_extfrag rates could be instructive.
Why the 2% success boost isn't quite clear to me. Allocation latency improving suggests the extra work is primarily picked up by background compaction. Reduced reserves could be making proactive compaction more aggressive. But the improvement is unlikely to hold once you run out of idle CPUs and the additional compaction work actually eats into the workload. It could be useful to look closer at who is doing the extra work and based on what triggers.
Quoted 48 lines · Johannes Weiner
Quoted 34 lines · Nimrod Oren
Quoted 11 lines · Johannes Weiner
Quoted 10 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Johannes Weiner
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
Quoted 7 lines
Quoted 2 lines
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change.
Hi,
I tested this on an x86-64 (4K/2M) virtual machine with one NUMA node and 16 GiB online memory, using mmtests config-workload-thpchallenge-fio with THPCHALLENGE_MADV_HUGEPAGE=yes.
min_free_kbyteswas 16 MiB patched and 66 MiB unpatched.I ran each kernel 30 times, rebooting before each run. The results did not show a regression in THP fault success rate or latency:
Average THP fault success (Percentage Faults Huge) increased from 13.07% unpatched to 13.34% patched, and average fault latency (Fault Latencies) decreased by 6.5%.
Quoted 5 lines
In contrast, compaction metrics were higher on average with the patch:
Compaction stalls: 1,635 -> 1,713 (+4.7%) Compaction failures: 1,350 -> 1,416 (+4.9%)Compaction migrate scanned: 4,812,254 -> 5,587,587 (+16.1%)
A 2% increase in THP success bought with a 16.1% increase in compaction work looks like a sizable efficiency regression.
A scan efficiency drop is in line with expectations of what happens when non-frag placement reserves are taken from the allocator. A comparison of
trace_mm_page_alloc_extfragrates could be instructive.
Quoted 7 lines
Why the 2% success boost isn't quite clear to me. Allocation latency improving suggests the extra work is primarily picked up by background compaction. Reduced reserves could be making proactive compaction more aggressive. But the improvement is unlikely to hold once you run out of idle CPUs and the additional compaction work actually eats into the workload. It could be useful to look closer at who is doing the extra work and based on what triggers.
Thanks.
I ran another 30 pairs with the same setup and workload, this time adding identical CPU load to both conditions. CPU utilization during the measured workload averaged approximately 99.7% in both the baseline and the patched runs. The aggregate mpstat %sys value averaged 2.25% in the baseline runs and 2.22% in the patched runs.
THP results:
baseline avg patched avg change Percentage Faults Huge 9.92% 9.12% -0.80 pp Fault Latencies 4923 us 4856 us -1.36%
The average difference in Percentage Faults Huge changed from +0.26 percentage points in the earlier runs to -0.80 points under CPU load. It's worth clarifying, though, that both differences were small relative to the observed run-to-run variability: the baseline/patched standard deviations were 4.52/4.59 percentage points earlier and 2.64/2.95 points under CPU load.
The same applies to Fault Latencies, whose baseline/patched SDs were 432/368 us earlier and 181/132 us under CPU load.
Compaction and reclaim results:
baseline avg patched avg change Compaction migrate scanned 3,983,598 4,563,788 +14.6% Kcompactd migrate scanned 1,242,728 1,944,022 +56.4% Compaction free scanned 17,689,565 20,461,694 +15.7% Kcompactd free scanned 2,608,710 4,426,810 +69.7% Compaction stalls 1,476 1,514 +2.6% Compaction failures 1,265 1,319 +4.3% Direct pages scanned 1,339,044 1,344,907 +0.4% Kswapd pages scanned 2,011,323 2,141,355 +6.5%
End-to-end workload results:
baseline avg patched avg change
Duration Elapsed 98.82 s 98.33 s -0.50%
Aggregate fio read bandwidth 0.2142 GiB/s 0.2152 GiB/s +0.48%
Aggregate fio read bandwidth is the sum of the per-job read bandwidth values in the fio output.
Quoted 93 lines · Nimrod Oren
Quoted 48 lines · Johannes Weiner
Quoted 34 lines · Nimrod Oren
Quoted 11 lines · Johannes Weiner
Quoted 10 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Johannes Weiner
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.Nacked-by Johannes Weiner <[email protected]>Isn't every possible change to address this kind of issue subject to exactly the same kind of constraint?
Quoted 7 lines
Quoted 2 lines
I'd like to know what not rolling that dice looks like :) or what constitutes 'careful evaluation'.
Usama gave some great examples in his other email. I'm not really arguing to keep things out of tradition. But I think it's fair to say let's at least test the common 4k/2M THP setups under memory pressure before and after the change.
Hi,
I tested this on an x86-64 (4K/2M) virtual machine with one NUMA node and 16 GiB online memory, using mmtests config-workload-thpchallenge-fio with THPCHALLENGE_MADV_HUGEPAGE=yes.
min_free_kbyteswas 16 MiB patched and 66 MiB unpatched.I ran each kernel 30 times, rebooting before each run. The results did not show a regression in THP fault success rate or latency:
Average THP fault success (Percentage Faults Huge) increased from 13.07% unpatched to 13.34% patched, and average fault latency (Fault Latencies) decreased by 6.5%.
Quoted 5 lines
In contrast, compaction metrics were higher on average with the patch:
Compaction stalls: 1,635 -> 1,713 (+4.7%) Compaction failures: 1,350 -> 1,416 (+4.9%)Compaction migrate scanned: 4,812,254 -> 5,587,587 (+16.1%)
A 2% increase in THP success bought with a 16.1% increase in compaction work looks like a sizable efficiency regression.
A scan efficiency drop is in line with expectations of what happens when non-frag placement reserves are taken from the allocator. A comparison of
trace_mm_page_alloc_extfragrates could be instructive.Quoted 7 lines
Why the 2% success boost isn't quite clear to me. Allocation latency improving suggests the extra work is primarily picked up by background compaction. Reduced reserves could be making proactive compaction more aggressive. But the improvement is unlikely to hold once you run out of idle CPUs and the additional compaction work actually eats into the workload. It could be useful to look closer at who is doing the extra work and based on what triggers.
Thanks.
I ran another 30 pairs with the same setup and workload, this time adding identical CPU load to both conditions. CPU utilization during the measured workload averaged approximately 99.7% in both the baseline and the patched runs. The aggregate mpstat %sys value averaged 2.25% in the baseline runs and 2.22% in the patched runs.
THP results:
baseline avg patched avg change Percentage Faults Huge 9.92% 9.12% -0.80 pp Fault Latencies 4923 us 4856 us -1.36%The average difference in Percentage Faults Huge changed from +0.26 percentage points in the earlier runs to -0.80 points under CPU load. It's worth clarifying, though, that both differences were small relative to the observed run-to-run variability: the baseline/patched standard deviations were 4.52/4.59 percentage points earlier and 2.64/2.95 points under CPU load.
The same applies to Fault Latencies, whose baseline/patched SDs were 432/368 us earlier and 181/132 us under CPU load.
Compaction and reclaim results:
baseline avg patched avg change Compaction migrate scanned 3,983,598 4,563,788 +14.6% Kcompactd migrate scanned 1,242,728 1,944,022 +56.4% Compaction free scanned 17,689,565 20,461,694 +15.7% Kcompactd free scanned 2,608,710 4,426,810 +69.7% Compaction stalls 1,476 1,514 +2.6% Compaction failures 1,265 1,319 +4.3% Direct pages scanned 1,339,044 1,344,907 +0.4% Kswapd pages scanned 2,011,323 2,141,355 +6.5%
Quoted 8 lines
End-to-end workload results:
baseline avg patched avg change
Duration Elapsed 98.82 s 98.33 s -0.50%
Aggregate fio read bandwidth 0.2142 GiB/s 0.2152 GiB/s +0.48%Aggregate fio read bandwidth is the sum of the per-job read bandwidth values in the fio output.
Following up with tracing data.
I ran another eight baseline/patched pairs with the same thpchallenge-fio workload, without additional CPU load.
For mm_page_alloc_extfrag, the mean was 3,334 events/second baseline and 3,586 patched. The patched rate was higher in six pairs and lower in two. The pairwise changes ranged from -1,281 to +1,943 events/second.
I extended mm_compaction_end identically in both kernels to record total_migrate_scanned and the direct_compaction and proactive_compaction flags.
Summing total_migrate_scanned per run gave these averages:
baseline avg patched avg
proactive kcompactd 2,826,839 3,145,912
non-proactive kcompactd 235,081 275,834
direct compaction 2,518,875 2,295,135
Proactive and non-proactive kcompactd scanning were higher with the patched kernel in five of eight pairs each, while direct scanning was lower in six of eight.
Michal Hocko
reviewerQuoted 32 lines · Johannes Weiner
Quoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
Quoted 7 lines
Just to summarize my take from the subthread with Zi: the premise of this patch is to roll the regression dice on every THP setup out there because certain ARM configurations result in a questionable pageblock size.
I'm not against carefully evaluating and testing out today's need for
set_recommended_min_free_kbytes()in real world examples. But this is not that.
I do hear your concerns Johannes. They are valid but I am not really sure wht kind of testing would prove such a change to be safe.
Quoted 1 line
Nacked-by Johannes Weiner <[email protected]>
Signature · 2 lines
Michal Hocko SUSE Labs
Usama Arif
reviewerQuoted 27 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.
Quoted 3 lines
Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>
Could this negatively affect x86 and arm64 systems using 4 KiB pages?
This change reduces the free-memory headroom available for compaction. That could reduce THP allocation success, especially for GFP_TRANSHUGE_LIGHT allocations, which fail quickly rather than performing direct reclaim or compaction.
With defrag=madvise, faults in MADV_HUGEPAGE regions may instead incur additional synchronous reclaim or compaction latency. khugepaged itself will not start later, but it may encounter more fragmented memory and perform more work or fail more often.
I think it would be good to provide THP success, fallback, and allocation-latency results under memory pressure for x86 and arm64 4 KiB systems?
Vlastimil Babka (SUSE)
maintainerQuoted 52 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/
Quoted 9 lines
--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of min_free_kbytes the first moment the sysctl gets adjusted from userspace?
Zi Yan
reviewerQuoted 63 lines · Vlastimil Babka (SUSE)
Quoted 52 lines · Nimrod Oren
When THP is enabled,
set_recommended_min_free_kbytes()may raisemin_free_kbytesusing a heuristic that scales withpageblock_nr_pages. Commitf000565adb77("thp: set recommended min free kbytes") added this heuristic to help keep pageblocks free and reduce fragmentation for THP allocations.The recommendation scales poorly with larger base page sizes. With the default arm64 pageblock sizes, the contribution per eligible zone before applying the existing cap of 5% of low memory is:
4 KiB pages: 2 MiB pageblock, 22 MiB per zone
16 KiB pages: 32 MiB pageblock, 352 MiB per zone 64 KiB pages: 512 MiB pageblock, 5.5 GiB per zoneEven with that cap,
min_free_kbytescan reach excessive levels.The automatic
min_free_kbytesincrease predates proactive compaction and many subsequent changes to compaction. Given those changes, increasingmin_free_kbytesfor THP by default is no longer clearly justified.Remove
set_recommended_min_free_kbytes()and all associated recalculation paths. With this policy gone,min_free_kbytesis controlled only by the page allocator's default calculation and the vm.min_free_kbytes sysctl. Users who want additional headroom may set a higher value via that sysctl.Suggested-by Michal Hocko <[email protected]>Signed-off-by Nimrod Oren <[email protected]>---
v3:
* Replace the 1 GiB cap with removal of the THP-drivenmin_free_kbytesincrease as suggested by Michal.
* Remove all associated recalculation paths.
* Makemin_free_kbytes,user_min_free_kbytes,calculate_min_free_kbytes(), andsetup_per_zone_wmarks()static.
* Drop the now-unused declarations and khugepaged'spage_alloc.hinclude.
* Drop the obsolete v2 documentation addition.v2:
* Use a named constant for the cap.
* Drop explicitlinux/sizes.hinclude.
* Updatemin_free_kbytesdocumentation.
https://lore.kernel.org/r/[email protected]/v1:
* Cap the final recommendation at 1 GiB as suggested by Lorenzo.
https://lore.kernel.org/r/[email protected]/Quoted 9 lines
--- include/linux/khugepaged.h | 5 ---- mm/huge_memory.c | 14 --------- mm/internal.h | 8 ----- mm/khugepaged.c | 60 -------------------------------------- mm/page_alloc.c | 11 +++---- mm/page_alloc.h | 2 -- mm/shmem.c | 7 ----- 7 files changed, 4 insertions(+), 103 deletions(-)
Quoted 3 lines
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?
Sounds good to me. Johannes, WDYT?
Signature · 2 lines
Best Regards, Yan, Zi
Johannes Weiner
reviewerQuoted 6 lines · Zi Yan
Quoted 3 lines · Vlastimil Babka (SUSE)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?Sounds good to me. Johannes, WDYT?
+1 That sounds good to me as well.
Lorenzo Stoakes (ARM)
reviewerQuoted 9 lines · Johannes Weiner
Quoted 6 lines · Zi Yan
Quoted 3 lines · Vlastimil Babka (SUSE)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?Sounds good to me. Johannes, WDYT?
+1 That sounds good to me as well.
Me also! :>)
--
Cheers, Lorenzo
Quoted 7 lines · On Wed, 9 Sep 2026 09:11:04 +0100 "Lorenzo Stoakes (ARM)" <[email protected]>…
Quoted 6 lines · Johannes Weiner
Quoted 5 lines · Zi Yan
Quoted 3 lines · Vlastimil Babka (SUSE)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Sounds good to me. Johannes, WDYT?
+1 That sounds good to me as well.
Me also! :>)
You might want some method of restoring the default behaviour. Maybe setting to 0 can be used (if that is just silly).
David
Quoted 4 lines
--
Cheers, Lorenzo
Lorenzo Stoakes (ARM)
reviewerQuoted 10 lines · David Laight
Quoted 7 lines · On Wed, 9 Sep 2026 09:11:04 +0100 "Lorenzo Stoakes (ARM)" <[email protected]>…
Quoted 6 lines · Johannes Weiner
Quoted 5 lines · Zi Yan
Quoted 3 lines · Vlastimil Babka (SUSE)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Sounds good to me. Johannes, WDYT?
+1 That sounds good to me as well.
Me also! :>)
You might want some method of restoring the default behaviour. Maybe setting to 0 can be used (if that is just silly).
I suppose the concern would be an embedded user for some reason enabling CONFIG_TRANSPARENT_HUGEPAGE and then defaulting to THP ->madvise and getting the bigger reserves before somehow disabling it after boot.
But I'm not sure if this is something we really need to worry about?
Quoted 8 lines
David
Quoted 4 lines
--
Cheers, Lorenzo
--
Cheers, Lorenzo
Quoted 15 lines · On Wed, 9 Sep 2026 12:18:03 +0100 "Lorenzo Stoakes (ARM)" <[email protected]>…
Quoted 14 lines · David Laight
Quoted 11 lines · On Wed, 9 Sep 2026 09:11:04 +0100 "Lorenzo Stoakes (ARM)" <[email protected]>…
Quoted 9 lines · Johannes Weiner
Quoted 6 lines · Zi Yan
Quoted 3 lines · Vlastimil Babka (SUSE)
Given neither of the patchset versions achieved universal acceptance, could we try just disabling all the automatic (re)calculations of
min_free_kbytesthe first moment the sysctl gets adjusted from userspace?Sounds good to me. Johannes, WDYT?
+1 That sounds good to me as well.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Me also! :>)
You might want some method of restoring the default behaviour. Maybe setting to 0 can be used (if that is just silly).
I suppose the concern would be an embedded user for some reason enabling
CONFIG_TRANSPARENT_HUGEPAGEand then defaulting to THP ->madvise and getting the bigger reserves before somehow disabling it after boot.
Or just testing whether it makes any difference to a workload.
David
Quoted 14 lines
But I'm not sure if this is something we really need to worry about?
Quoted 8 lines
David
Quoted 4 lines
--
Cheers, Lorenzo--
Cheers, Lorenzo