mm: split underutilized THPs
Subsystems (13)
v2 cover mm: split underutilized THPs
The current upstream default policy for THP is always. However, Meta uses madvise in production as the current THP=always policy vastly overprovisions THPs in sparsely accessed memory areas, resulting in excessive memory pressure and premature OOM killing.
Using madvise + relying on khugepaged has certain drawbacks over THP=always. Using madvise hints mean THPs aren't "transparent" and require userspace changes. Waiting for khugepaged to scan memory and collapse pages into THP can be slow and unpredictable in terms of performance (i.e. you dont know when the collapse will happen), while production environments require predictable performance. If there is enough memory available, its better for both performance and predictability to have a THP from fault time, i.e. THP=always rather than wait for khugepaged to collapse it, and deal with sparsely populated THPs when the system is running out of memory.
This patch-series is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in or collapsed by khugepaged, the THP is added to a list. Whenever memory reclaim happens, the kernel runs the deferred_split shrinker which goes through the list and checks if the THP was underutilized, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold, the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory. This method avoids the downside of wasting memory in areas where THP is sparsely filled when THP is always enabled, while still providing the upside THPs like reduced TLB misses without having to use madvise.
Meta production workloads that were CPU bound (>99% CPU utilzation) were tested with THP shrinker. The results after 2 hours are as follows:
| THP=madvise | THP=always | THP=always
| | | + shrinker series
| | | + max_ptes_none=409
-----------------------------------------------------------------------------
Performance improvement | - | +1.8% | +1.7%
(over THP=madvise) | | |-----------------------------------------------------------------------------
Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%)
-----------------------------------------------------------------------------
max_ptes_none=409 means that any THP that has more than 409 out of 512 (80%) zero filled filled pages will be split.
To test out the patches, the below commands without the shrinker will invoke OOM killer immediately and kill stress, but will not fail with the shrinker:
echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none mkdir /sys/fs/cgroup/test
echo $$ > /sys/fs/cgroup/test/cgroup.procs
echo 20M > /sys/fs/cgroup/test/memory.max
echo 0 > /sys/fs/cgroup/test/memory.swap.max
# allocate twice memory.max for each stress worker and touch 40/512 of # each THP, i.e. vm-stride 50K. # With the shrinker, max_ptes_none of 470 and below won't invoke OOM # killer. # Without the shrinker, OOM killer is invoked immediately irrespective # of max_ptes_none value and kills stress. stress --vm 1 --vm-bytes 40M --vm-stride 50K
v1 -> v2:
- Turn page checks and operations to folio versions in __split_huge_page. This means patches 1 and 2 from v1 are no longer needed. (David Hildenbrand)
- Map to shared zeropage in all cases if the base page is zero-filled. The uffd selftest was removed.
(David Hildenbrand).
- rename 'dirty' to 'contains_data' in try_to_map_unused_to_zeropage (Rik van Riel).
- Use unsigned long instead of uint64_t (kernel test robot).
Alexander Zhu (1):
mm: selftest to verify zero-filled pages are mapped to zeropage
Usama Arif (1):
mm: split underutilized THPs
Yu Zhao (2):
mm: free zapped tail pages when splitting isolated thp
mm: remap unused subpages to shared zeropage when splitting isolated
thp
Documentation/admin-guide/mm/transhuge.rst | 6 + include/linux/huge_mm.h | 4 +- include/linux/khugepaged.h | 1 + include/linux/mm_types.h | 2 + include/linux/rmap.h | 3 +- include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 153 +++++++++++++++--- mm/hugetlb.c | 1 + mm/internal.h | 4 +- mm/khugepaged.c | 3 +- mm/memcontrol.c | 3 +- mm/migrate.c | 73 +++++++-- mm/migrate_device.c | 4 +- mm/rmap.c | 2 +- mm/vmscan.c | 3 +- mm/vmstat.c | 1 + .../selftests/mm/split_huge_page_test.c | 71 ++++++++ tools/testing/selftests/mm/vm_util.c | 22 +++ tools/testing/selftests/mm/vm_util.h | 1 + 19 files changed, 321 insertions(+), 37 deletions(-)
Signature · 1 line
2.43.5
PATCH v2 1/4 mm: free zapped tail pages when splitting isolated thp
From: Yu Zhao <[email protected]>
If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.
--- mm/huge_memory.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
mm/huge_memory.c +27 -0
@@ -2923,7 +2923,9 @@ static void __split_huge_page(struct page *page, struct list_head *list,unsigned int new_nr =1 << new_order;int order =folio_order (folio);unsigned int nr =1 << order; +struct folio_batch free_folios; +folio_batch_init (&free_folios);/* complete memcg works before add pages to LRU */ split_page_memcg (head, order, new_order);
@@ -3007,6 +3009,26 @@ static void __split_huge_page(struct page *page, struct list_head *list,if (subpage == page)continue ;folio_unlock (new_folio); +/* +* If a folio has only two references left, one inherited +* from the isolation of its head and the other from +* lru_add_page_tail() which we are about to drop, it means this +* folio was concurrently zapped. Then we can safely free it +* and save page reclaim or migration the trouble of trying it. +*/ +if (list &&page_ref_freeze (subpage,2 )) { +VM_WARN_ON_ONCE_FOLIO (folio_test_lru (new_folio), new_folio); +VM_WARN_ON_ONCE_FOLIO (folio_test_large (new_folio), new_folio); +VM_WARN_ON_ONCE_FOLIO (folio_mapped (new_folio), new_folio); + +folio_clear_active (new_folio); +folio_clear_unevictable (new_folio); +if (folio_batch_add (&free_folios, folio) ==0 ) { +mem_cgroup_uncharge_folios (&free_folios); +free_unref_folios (&free_folios); + } +continue ; + }/* * Subpages may be freed if there wasn't any mapping
@@ -3017,6 +3039,11 @@ static void __split_huge_page(struct page *page, struct list_head *list,*/ free_page_and_swap_cache (subpage); } + +if (free_folios.nr) { +mem_cgroup_uncharge_folios (&free_folios); +free_unref_folios (&free_folios); + } }/* Racy check whether the huge page can be split */
Signature · 1 line
2.43.5
Re: [PATCH v2 1/4] mm: free zapped tail pages when splitting isolated thp
Quoted 31 lines · Usama Arif
From: Yu Zhao <[email protected]>
If a tail page has only two references left, one inherited from the isolation of its head and the other from
lru_add_page_tail()which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.Signed-off-by Yu Zhao <[email protected]>Tested-by Shuang Zhai <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/huge_memory.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)mm/huge_memory.c +2
-0@@ -2923,7 +2923,9 @@ static void __split_huge_page(struct page *page, struct list_head *list,unsigned int new_nr =1 << new_order;int order =folio_order (folio);unsigned int nr =1 << order; +struct folio_batch free_folios; +folio_batch_init (&free_folios);/* complete memcg works before add pages to LRU */ split_page_memcg (head, order, new_order);@@ -3007,6 +3009,26 @@ static void __split_huge_page(struct page *page, struct list_head *list,if (subpage == page)
mm/huge_memory.c static void __split_huge_page(struct page *page, struct list_head *list, · line 3019
Quoted 10 lines
mm/huge_memory.c +8
-0continue ;folio_unlock (new_folio); +/* +* If a folio has only two references left, one inherited +* from the isolation of its head and the other from +* lru_add_page_tail() which we are about to drop, it means this +* folio was concurrently zapped. Then we can safely free it +* and save page reclaim or migration the trouble of trying it. +*/ +if (list &&page_ref_freeze (subpage,2 )) {
folio_ref_freeze(new_folio, 2)?
Otherwise looks good to me
Re: [PATCH v2 1/4] mm: free zapped tail pages when splitting isolated thp
In reply to Usama ArifQuoted 38 lines · Usama Arif
From: Yu Zhao <[email protected]>
If a tail page has only two references left, one inherited from the isolation of its head and the other from
lru_add_page_tail()which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.Signed-off-by Yu Zhao <[email protected]>Tested-by Shuang Zhai <[email protected]>Signed-off-by Usama Arif <[email protected]>--- mm/huge_memory.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)mm/huge_memory.c +7
-0@@ -2923,7 +2923,9 @@ static void __split_huge_page(struct page *page, struct list_head *list,unsigned int new_nr =1 << new_order;int order =folio_order (folio);unsigned int nr =1 << order; +struct folio_batch free_folios; +folio_batch_init (&free_folios);/* complete memcg works before add pages to LRU */ split_page_memcg (head, order, new_order);@@ -3007,6 +3009,26 @@ static void __split_huge_page(struct page *page, struct list_head *list,if (subpage == page)continue ;folio_unlock (new_folio); +/* +* If a folio has only two references left, one inherited +* from the isolation of its head and the other from +* lru_add_page_tail() which we are about to drop, it means this +* folio was concurrently zapped. Then we can safely free it
mm/huge_memory.c static void __split_huge_page(struct page *page, struct list_head *list, · line 3026
Quoted 10 lines
mm/huge_memory.c +10
-0+* and save page reclaim or migration the trouble of trying it. +*/ +if (list &&page_ref_freeze (subpage,2 )) { +VM_WARN_ON_ONCE_FOLIO (folio_test_lru (new_folio), new_folio); +VM_WARN_ON_ONCE_FOLIO (folio_test_large (new_folio), new_folio); +VM_WARN_ON_ONCE_FOLIO (folio_mapped (new_folio), new_folio); + +folio_clear_active (new_folio); +folio_clear_unevictable (new_folio); +if (folio_batch_add (&free_folios, folio) ==0 ) {
nit: "!folio_batch_add(&free_folios, folio)"
Nothing else jumped at me.
Signature · 3 lines
Cheers, David / dhildenb
PATCH v2 2/4 mm: remap unused subpages to shared zeropage when splitting isolated thp
In reply to Usama ArifFrom: Yu Zhao <[email protected]>
Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.
This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.
--- include/linux/rmap.h | 3 +- mm/huge_memory.c | 8 ++--- mm/migrate.c | 70 +++++++++++++++++++++++++++++++++++++++----- mm/migrate_device.c | 4 +-- 4 files changed, 70 insertions(+), 15 deletions(-)
include/linux/rmap.h +2 -1
@@ -745,7 +745,8 @@ int folio_mkclean(struct folio *);int pfn_mkclean_range (unsigned long pfn,unsigned long nr_pages,pgoff_t pgoff,struct vm_area_struct *vma); -void remove_migration_ptes (struct folio *src,struct folio *dst,bool locked); +void remove_migration_ptes (struct folio *src,struct folio *dst,bool locked, +bool map_unused_to_zeropage);/* * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
@@ -2775,7 +2775,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,return false; } -static void remap_page (struct folio *folio,unsigned long nr) +static void remap_page (struct folio *folio,unsigned long nr,bool map_unused_to_zeropage) {int i =0 ;
@@ -2783,7 +2783,7 @@ static void remap_page(struct folio *folio, unsigned long nr)if (!folio_test_anon (folio))return ;for (;;) { -remove_migration_ptes (folio, folio, true); +remove_migration_ptes (folio, folio, true, map_unused_to_zeropage); i +=folio_nr_pages (folio);if (i >= nr)break ;
@@ -2993,7 +2993,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,if (nr_dropped)shmem_uncharge (folio->mapping->host, nr_dropped); -remap_page (folio, nr); +remap_page (folio, nr,PageAnon (head));/* * set page to its compound_head when split to non order-0 pages, so
@@ -3287,7 +3287,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,if (mapping)xas_unlock (&xas);local_irq_enable (); -remap_page (folio,folio_nr_pages (folio)); +remap_page (folio,folio_nr_pages (folio), false); ret = -EAGAIN ; }
mm/migrate.c +62 -8
122 lines not on this page · raw message
mm/migrate_device.c +2 -2
@@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,continue ; folio =page_folio (page); -remove_migration_ptes (folio, folio, false); +remove_migration_ptes (folio, folio, false, false); src_pfns[i] =0 ;folio_unlock (folio);
@@ -837,7 +837,7 @@ void migrate_device_finalize(unsigned long *src_pfns, src =page_folio (page); dst =page_folio (newpage); -remove_migration_ptes (src, dst, false); +remove_migration_ptes (src, dst, false, false);folio_unlock (src);if (is_zone_device_page (page))
Signature · 1 line
2.43.5
Johannes Weiner
maintainerRe: [PATCH v2 2/4] mm: remap unused subpages to shared zeropage when splitting isolated thp
Quoted 25 lines · Usama Arif
mm/migrate.c +21
-0@@ -177,13 +177,56 @@ void putback_movable_pages(struct list_head *l) } } +static bool try_to_map_unused_to_zeropage (struct page_vma_mapped_walk *pvmw, +struct folio *folio, +unsigned long idx) +{ +struct page *page =folio_page (folio, idx); +bool contains_data; +pte_t newpte; +void *addr; + +VM_BUG_ON_PAGE (PageCompound (page), page); +VM_BUG_ON_PAGE (!PageAnon (page), page); +VM_BUG_ON_PAGE (!PageLocked (page), page); +VM_BUG_ON_PAGE (pte_present (*pvmw->pte), page); + +if (PageMlocked (page) || (pvmw->vma->vm_flags &VM_LOCKED )) +return false; + +/* +* The pmd entry mapping the old thp was flushed and the pte mapping +* this subpage has been non present. If the subpage is only zero-filled +* then map it to the shared zeropage.
mm/migrate.c void putback_movable_pages(struct list_head *l) · line 210
Quoted 10 lines
mm/migrate.c +10
-0+*/ + addr =kmap_local_page (page); + contains_data =memchr_inv (addr,0 ,PAGE_SIZE ); +kunmap_local (addr); + +if (contains_data ||mm_forbids_zeropage (pvmw->vma->vm_mm)) +return false; + + newpte =pte_mkspecial (pfn_pte (page_to_pfn (ZERO_PAGE (pvmw->address)), + pvmw->vma->vm_page_prot));
Why not use my_zero_pfn() here? On many configurations this just returns zero_pfn and avoids the indirection through mem_map.
mm/migrate.c static int writeout(struct address_space *mapping, struct folio *folio) · line 961
Quoted 6 lines
mm/migrate.c +1
-1@@ -904,7 +958,7 @@ static int writeout(struct address_space *mapping, struct folio *folio)* At this point we know that the migration attempt cannot * be successful. */ -remove_migration_ptes (folio, folio, false); +remove_migration_ptes (folio, folio, false, false);
bool params are not great for callsite readability.
How about a flags parameter and using names?
enum rmp_flags {
RMP_LOCKED = 1 << 0,
RMP_ZEROPAGES = 1 << 1,
}Re: [PATCH v2 2/4] mm: remap unused subpages to shared zeropage when splitting isolated thp
Quoted 49 lines · Johannes Weiner
Quoted 25 lines · Usama Arif
mm/migrate.c +21
-0@@ -177,13 +177,56 @@ void putback_movable_pages(struct list_head *l) } } +static bool try_to_map_unused_to_zeropage (struct page_vma_mapped_walk *pvmw, +struct folio *folio, +unsigned long idx) +{ +struct page *page =folio_page (folio, idx); +bool contains_data; +pte_t newpte; +void *addr; + +VM_BUG_ON_PAGE (PageCompound (page), page); +VM_BUG_ON_PAGE (!PageAnon (page), page); +VM_BUG_ON_PAGE (!PageLocked (page), page); +VM_BUG_ON_PAGE (pte_present (*pvmw->pte), page); + +if (PageMlocked (page) || (pvmw->vma->vm_flags &VM_LOCKED )) +return false; + +/* +* The pmd entry mapping the old thp was flushed and the pte mapping +* this subpage has been non present. If the subpage is only zero-filled +* then map it to the shared zeropage. mm/migrate.c void putback_movable_pages(struct list_head *l) · line 210
Quoted 10 lines
mm/migrate.c +10
-0+*/ + addr =kmap_local_page (page); + contains_data =memchr_inv (addr,0 ,PAGE_SIZE ); +kunmap_local (addr); + +if (contains_data ||mm_forbids_zeropage (pvmw->vma->vm_mm)) +return false; + + newpte =pte_mkspecial (pfn_pte (page_to_pfn (ZERO_PAGE (pvmw->address)), + pvmw->vma->vm_page_prot));Why not use
my_zero_pfn()here? On many configurations this just returnszero_pfnand avoids the indirection throughmem_map.mm/migrate.c static int writeout(struct address_space *mapping, struct folio *folio) · line 961
Quoted 6 lines
mm/migrate.c +1
-1@@ -904,7 +958,7 @@ static int writeout(struct address_space *mapping, struct folio *folio)* At this point we know that the migration attempt cannot * be successful. */ -remove_migration_ptes (folio, folio, false); +remove_migration_ptes (folio, folio, false, false);bool params are not great for callsite readability.
Quoted 6 lines
How about a flags parameter and using names?
enum rmp_flags { RMP_LOCKED = 1 << 0, RMP_ZEROPAGES = 1 << 1, }
Thanks! Will include both of the above changes in the next revision.
PATCH v2 3/4 mm: selftest to verify zero-filled pages are mapped to zeropage
In reply to Usama ArifFrom: Alexander Zhu <[email protected]>
When a THP is split, any subpage that is zero-filled will be mapped to the shared zeropage, hence saving memory. Add selftest to verify this by allocating zero-filled THP and comparing RssAnon before and after split.
--- .../selftests/mm/split_huge_page_test.c | 71 +++++++++++++++++++ tools/testing/selftests/mm/vm_util.c | 22 ++++++ tools/testing/selftests/mm/vm_util.h | 1 + 3 files changed, 94 insertions(+)
tools/testing/selftests/mm/split_huge_page_test.c +71 -0
@@ -84,6 +84,76 @@ static void write_debugfs(const char *fmt, ...)write_file (SPLIT_DEBUGFS , input, ret +1 ); } +static char *allocate_zero_filled_hugepage (size_t len) +{ +char *result; +size_t i; + + result =memalign (pmd_pagesize, len); +if (!result) { +printf ("Fail to allocate memory\n" ); +exit (EXIT_FAILURE ); + } + +madvise (result, len,MADV_HUGEPAGE ); + +for (i =0 ; i < len; i++) + result[i] = (char )0 ; + +return result; +} + +static void verify_rss_anon_split_huge_page_all_zeroes (char *one_page,int nr_hpages,size_t len) +{ +unsigned long rss_anon_before, rss_anon_after; +size_t i; + +if (!check_huge_anon (one_page,4 , pmd_pagesize)) { +printf ("No THP is allocated\n" ); +exit (EXIT_FAILURE ); + } + + rss_anon_before =rss_anon (); +if (!rss_anon_before) { +printf ("No RssAnon is allocated before split\n" ); +exit (EXIT_FAILURE ); + } + +/* split all THPs */ +write_debugfs (PID_FMT ,getpid (), (uint64_t )one_page, + (uint64_t )one_page + len,0 ); + +for (i =0 ; i < len; i++) +if (one_page[i] != (char )0 ) { +printf ("%ld byte corrupted\n" , i); +exit (EXIT_FAILURE ); + } + +if (!check_huge_anon (one_page,0 , pmd_pagesize)) { +printf ("Still AnonHugePages not split\n" ); +exit (EXIT_FAILURE ); + } + + rss_anon_after =rss_anon (); +if (rss_anon_after >= rss_anon_before) { +printf ("Incorrect RssAnon value. Before: %ld After: %ld\n" , + rss_anon_before, rss_anon_after); +exit (EXIT_FAILURE ); + } +} + +void split_pmd_zero_pages (void ) +{ +char *one_page; +int nr_hpages =4 ; +size_t len = nr_hpages * pmd_pagesize; + + one_page =allocate_zero_filled_hugepage (len); +verify_rss_anon_split_huge_page_all_zeroes (one_page, nr_hpages, len); +printf ("Split zero filled huge pages successful\n" ); +free (one_page); +} +void split_pmd_thp (void ) {char *one_page;
@@ -431,6 +501,7 @@ int main(int argc, char **argv) fd_size =2 * pmd_pagesize; +split_pmd_zero_pages ();split_pmd_thp ();split_pte_mapped_thp ();split_file_backed_thp ();
tools/testing/selftests/mm/vm_util.c +22 -0
@@ -12,6 +12,7 @@#define PMD_SIZE_FILE_PATH "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size"#define SMAP_FILE_PATH "/proc/self/smaps" +#define STATUS_FILE_PATH "/proc/self/status"#define MAX_LINE_LENGTH 500unsigned int __page_size;
@@ -171,6 +172,27 @@ uint64_t read_pmd_pagesize(void)return strtoul (buf,NULL ,10 ); } +unsigned long rss_anon (void ) +{ +unsigned long rss_anon =0 ; +FILE *fp; +char buffer[MAX_LINE_LENGTH ]; + + fp =fopen (STATUS_FILE_PATH ,"r" ); +if (!fp) +ksft_exit_fail_msg ("%s: Failed to open file %s\n" , __func__,STATUS_FILE_PATH ); + +if (!check_for_pattern (fp,"RssAnon:" , buffer,sizeof (buffer))) + goto err_out; + +if (sscanf (buffer,"RssAnon:%10lu kB" , &rss_anon) !=1 ) +ksft_exit_fail_msg ("Reading status error\n" ); + +err_out: +fclose (fp); +return rss_anon; +} +bool __check_huge (void *addr,char *pattern,int nr_hpages,uint64_t hpage_size) {
tools/testing/selftests/mm/vm_util.h +1 -0
@@ -39,6 +39,7 @@ unsigned long pagemap_get_pfn(int fd, char *start);void clear_softdirty (void );bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void ); +uint64_t rss_anon (void );bool check_huge_anon (void *addr,int nr_hpages,uint64_t hpage_size);bool check_huge_file (void *addr,int nr_hpages,uint64_t hpage_size);bool check_huge_shmem (void *addr,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.43.5
PATCH v2 4/4 mm: split underutilized THPs
In reply to Usama ArifThis is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to _deferred_list. Whenever memory reclaim happens in linux, the kernel runs the deferred_split shrinker which goes through the _deferred_list.
If the folio was partially mapped, the shrinker attempts to split it. A new boolean is added to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements the folio mapcount elements, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan without the boolean.
If folio->_partially_mapped is not set, the shrinker checks if the THP was underutilized, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold (decided by /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none), the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory.
--- Documentation/admin-guide/mm/transhuge.rst | 6 ++ include/linux/huge_mm.h | 4 +- include/linux/khugepaged.h | 1 + include/linux/mm_types.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 118 ++++++++++++++++++--- mm/hugetlb.c | 1 + mm/internal.h | 4 +- mm/khugepaged.c | 3 +- mm/memcontrol.c | 3 +- mm/migrate.c | 3 +- mm/rmap.c | 2 +- mm/vmscan.c | 3 +- mm/vmstat.c | 1 + 14 files changed, 130 insertions(+), 22 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +6 -0
@@ -447,6 +447,12 @@ thp_deferred_split_page splitting it would free up some memory. Pages on split queue are going to be split under memory pressure. +thp_underutilized_split_page + is incremented when a huge page on the split queue was split + because it was underutilized. A THP is underutilized if the + number of zero pages in the THP are above a certain threshold + (/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none). + thp_split_pmd is incremented every time a PMD split into table of PTEs. This can happen, for instance, when application calls mprotect() or
include/linux/huge_mm.h +2 -2
@@ -321,7 +321,7 @@ static inline int split_huge_page(struct page *page) {return split_huge_page_to_list_to_order (page,NULL ,0 ); } -void deferred_split_folio (struct folio *folio); +void deferred_split_folio (struct folio *folio,bool partially_mapped);void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long address,bool freeze,struct folio *folio);
@@ -484,7 +484,7 @@ static inline int split_huge_page(struct page *page) {return 0 ; } -static inline void deferred_split_folio (struct folio *folio) {} +static inline void deferred_split_folio (struct folio *folio,bool partially_mapped) {}#define split_huge_pmd (__vma, __pmd, __address) \ do { } while (0)
include/linux/khugepaged.h +1 -0
@@ -4,6 +4,7 @@#include <linux/sched/coredump.h> /* MMF_VM_HUGEPAGE */ +extern unsigned int khugepaged_max_ptes_none__read_mostly ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE extern struct attribute_group khugepaged_attr_group;
include/linux/mm_types.h +2 -0
@@ -311,6 +311,7 @@ typedef struct {* @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_hwpoison: Do not use directly, call raw_hwp_list_head(). * @_deferred_list: Folios to be split under memory pressure. +* @_partially_mapped: Folio was partially mapped. * @_unused_slab_obj_exts: Placeholder to match obj_exts in struct slab. * * A folio is a physically, virtually and logically contiguous set
@@ -393,6 +394,7 @@ struct folio {unsigned long _head_2a;/* public: */ struct list_head _deferred_list; +bool _partially_mapped;/* private: the union with struct page is transitional */ };struct page __page_2;
include/linux/vm_event_item.h +1 -0
@@ -105,6 +105,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_SPLIT_PAGE ,THP_SPLIT_PAGE_FAILED ,THP_DEFERRED_SPLIT_PAGE , +THP_UNDERUTILIZED_SPLIT_PAGE ,THP_SPLIT_PMD ,THP_SCAN_EXCEED_NONE_PTE , THP_SCAN_EXCEED_SWAP_PTE,
mm/huge_memory.c +104 -14
195 lines not on this page · raw message
mm/hugetlb.c +1 -0
@@ -1758,6 +1758,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h,free_gigantic_folio (folio,huge_page_order (h)); }else {INIT_LIST_HEAD (&folio->_deferred_list); + folio->_partially_mapped = false;folio_put (folio); } }
mm/internal.h +3 -1
@@ -657,8 +657,10 @@ static inline void prep_compound_head(struct page *page, unsigned int order)atomic_set (&folio->_entire_mapcount,-1 );atomic_set (&folio->_nr_pages_mapped,0 );atomic_set (&folio->_pincount,0 ); -if (order >1 ) +if (order >1 ) {INIT_LIST_HEAD (&folio->_deferred_list); + folio->_partially_mapped = false; + } }static inline void prep_compound_tail (struct page *head,int tail_idx)
mm/khugepaged.c +2 -1
@@ -85,7 +85,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);* * Note that these are only respected if collapse was initiated by khugepaged. */ -static unsigned int khugepaged_max_ptes_none__read_mostly ; +unsigned int khugepaged_max_ptes_none__read_mostly ;static unsigned int khugepaged_max_ptes_swap__read_mostly ;static unsigned int khugepaged_max_ptes_shared__read_mostly ;
@@ -1235,6 +1235,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,pgtable_trans_huge_deposit (mm, pmd, pgtable);set_pmd_at (mm, address, pmd, _pmd);update_mmu_cache_pmd (vma, address, pmd); +deferred_split_folio (folio, false);spin_unlock (pmd_ptl); folio =NULL ;
mm/memcontrol.c +2 -1
@@ -4651,7 +4651,8 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug)VM_BUG_ON_FOLIO (folio_test_lru (folio), folio);VM_BUG_ON_FOLIO (folio_order (folio) >1 && !folio_test_hugetlb (folio) && - !list_empty (&folio->_deferred_list), folio); + !list_empty (&folio->_deferred_list) && + folio->_partially_mapped, folio);/* * Nobody should be changing or seriously looking at
mm/migrate.c +2 -1
@@ -1731,7 +1731,8 @@ static int migrate_pages_batch(struct list_head *from,* use _deferred_list. */ if (nr_pages >2 && - !list_empty (&folio->_deferred_list)) { + !list_empty (&folio->_deferred_list) && + folio->_partially_mapped) {if (try_split_folio (folio, split_folios) ==0 ) { nr_failed++; stats->nr_thp_failed += is_thp;
mm/rmap.c +1 -1
@@ -1582,7 +1582,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,*/ if (folio_test_anon (folio) && partially_mapped &&list_empty (&folio->_deferred_list)) -deferred_split_folio (folio); +deferred_split_folio (folio, true); }__folio_mod_stat (folio, -nr, -nr_pmdmapped);
mm/vmscan.c +2 -1
@@ -1233,7 +1233,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* Split partially mapped folios right away. * We can free the unmapped pages without IO. */ -if (data_race (!list_empty (&folio->_deferred_list)) && +if (data_race (!list_empty (&folio->_deferred_list) && + folio->_partially_mapped) &&split_folio_to_list (folio, folio_list)) goto activate_locked; }
mm/vmstat.c +1 -0
@@ -1367,6 +1367,7 @@ const char * const vmstat_text[] = {"thp_split_page" ,"thp_split_page_failed" ,"thp_deferred_split_page" , +"thp_underutilized_split_page" ,"thp_split_pmd" ,"thp_scan_exceed_none_pte" , "thp_scan_exceed_swap_pte",
Signature · 1 line
2.43.5
Quoted 7 lines · Usama Arif
This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (
__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to_deferred_list. Whenever memory reclaim happens in linux, the kernel runs thedeferred_splitshrinker which goes through the_deferred_list.
Quoted 7 lines
If the folio was partially mapped, the shrinker attempts to split it. A new boolean is added to be able to distinguish between partially mapped folios and others in the
deferred_listat split time indeferred_split_scan. Its needed as__folio_remove_rmapdecrements the folio mapcount elements, hence it won't be possible to distinguish between partially mapped folios and others indeferred_split_scanwithout the boolean.
Just so I get this right: Are you saying that we might now add fully mapped folios to the deferred split queue and that's what you want to distinguish?
If that's the case, then could we use a bit in folio->_flags_1 instead?
Further, I think you forgot to update at least one instance if a list_empty(&folio->_deferred_list) check where we want to detect "partially mapped". Please go over all and see what needs adjustments.
I would actually suggest to split decoupling of "_deferred_list" and "partially mapped" into a separate preparation patch.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 16 lines · David Hildenbrand
Quoted 7 lines · Usama Arif
This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (
__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to_deferred_list. Whenever memory reclaim happens in linux, the kernel runs thedeferred_splitshrinker which goes through the_deferred_list.Quoted 7 lines
If the folio was partially mapped, the shrinker attempts to split it. A new boolean is added to be able to distinguish between partially mapped folios and others in the
deferred_listat split time indeferred_split_scan. Its needed as__folio_remove_rmapdecrements the folio mapcount elements, hence it won't be possible to distinguish between partially mapped folios and others indeferred_split_scanwithout the boolean.
Quoted 1 line
Just so I get this right: Are you saying that we might now add fully mapped folios to the deferred split queue and that's what you want to distinguish?
Yes
Quoted 2 lines
If that's the case, then could we use a bit in
folio->_flags_1instead?
Yes, thats a good idea. Will create the below flag for the next revision
include/linux/page-flags.h +5 -0
@@ -189,6 +189,11 @@ enum pageflags {#define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) +enum folioflags_1 { +/* The first 8 bits of folio->_flags_1 are used to keep track of folio order */ +FOLIO_PARTIALLY_MAPPED =8 ,/* folio is partially mapped */ +} + #ifndef __GENERATING_BOUNDS_H
and use set/clear/test_bit(FOLIO_PARTIALLY_MAPPED, &folio->_flags_1) in the respective places.
Quoted 3 lines
Further, I think you forgot to update at least one instance if a
list_empty(&folio->_deferred_list)check where we want to detect "partially mapped". Please go over all and see what needs adjustments.
Ah I think its the one in free_tail_page_prepare? The way I wrote this patch is by going through all instances of "folio->_deferred_list" and deciding if partially_mapped needs to be set/cleared/tested. I think I missed it when rebasing to mm-unstable. Double checked now and the only one missing is free_tail_page_prepare ([1] was removed recently by Barry)
[1] https://lore.kernel.org/lkml/[email protected]/
Will include the below diff in the next revision.
mm/page_alloc.c +3 -2
@@ -957,8 +957,9 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)break ;case 2 :/* the second tail page: deferred_list overlaps ->mapping */ -if (unlikely (!list_empty (&folio->_deferred_list))) { -bad_page (page,"on deferred list" ); +if (unlikely (!list_empty (&folio->_deferred_list) && +test_bit (FOLIO_PARTIALLY_MAPPED , &folio->_flags_1))) { +bad_page (page,"partially mapped folio on deferred list" ); goto out; }break ;
Quoted 2 lines
I would actually suggest to split decoupling of "
_deferred_list" and "partially mapped" into a separate preparation patch.
Yes, will do. I will split it into 3 patches, 1st one that introduces FOLIO_PARTIALLY_MAPPED and sets/clear it in the right place without introducing any functional change, 2nd to split underutilized THPs and 3rd to add sysfs entry to enable/disable the shrinker. Should make the patches quite small and easy to review.
Quoted 32 lines · Usama Arif
Quoted 16 lines · David Hildenbrand
Quoted 7 lines · Usama Arif
This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (
__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to_deferred_list. Whenever memory reclaim happens in linux, the kernel runs thedeferred_splitshrinker which goes through the_deferred_list.Quoted 7 lines
If the folio was partially mapped, the shrinker attempts to split it. A new boolean is added to be able to distinguish between partially mapped folios and others in the
deferred_listat split time indeferred_split_scan. Its needed as__folio_remove_rmapdecrements the folio mapcount elements, hence it won't be possible to distinguish between partially mapped folios and others indeferred_split_scanwithout the boolean.Quoted 1 line
Just so I get this right: Are you saying that we might now add fully mapped folios to the deferred split queue and that's what you want to distinguish?
Yes
Quoted 2 lines
If that's the case, then could we use a bit in
folio->_flags_1instead?Yes, thats a good idea. Will create the below flag for the next revision
include/linux/page-flags.h +0
-0
include/linux/page-flags.h enum pageflags { · line 195
Quoted 7 lines
include/linux/page-flags.h +4
-0@@ -189,6 +189,11 @@ enum pageflags {#define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) +enum folioflags_1 { +/* The first 8 bits of folio->_flags_1 are used to keep track of folio order */ +FOLIO_PARTIALLY_MAPPED =8 ,/* folio is partially mapped */ +}
This might be what you want to achieve:
include/linux/page-flags.h +3 -1
@@ -182,6 +182,7 @@ enum pageflags {/* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_active, PG_large_rmappable = PG_workingset,/* anon or file-backed */ + PG_partially_mapped, /* was identified to be partially mapped */ };#define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1)
@@ -861,8 +862,9 @@ static inline void ClearPageCompound(struct page *page)ClearPageHead (page); } FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE) +FOLIO_FLAG(partially_mapped, FOLIO_SECOND_PAGE)#else -FOLIO_FLAG_FALSE(large_rmappable) +FOLIO_FLAG_FALSE(partially_mapped)#endif #define PG_head_mask ((1UL << PG_head))
The downside is an atomic op to set/clear, but it should likely not really matter (initially, the flag will be clear, and we should only ever set it once when partially unmapping). If it hurts, we can reconsider.
[...]
Quoted 3 lines
Quoted 2 lines
I would actually suggest to split decoupling of "
_deferred_list" and "partially mapped" into a separate preparation patch.Yes, will do. I will split it into 3 patches, 1st one that introduces
FOLIO_PARTIALLY_MAPPEDand sets/clear it in the right place without introducing any functional change, 2nd to split underutilized THPs and 3rd to add sysfs entry to enable/disable the shrinker. Should make the patches quite small and easy to review.
Great! As always, please shout if you disagree with something I propose :)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 65 lines · David Hildenbrand
Quoted 32 lines · Usama Arif
Quoted 16 lines · David Hildenbrand
Quoted 7 lines · Usama Arif
This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (
__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to_deferred_list. Whenever memory reclaim happens in linux, the kernel runs thedeferred_splitshrinker which goes through the_deferred_list.Quoted 7 lines
If the folio was partially mapped, the shrinker attempts to split it. A new boolean is added to be able to distinguish between partially mapped folios and others in the
deferred_listat split time indeferred_split_scan. Its needed as__folio_remove_rmapdecrements the folio mapcount elements, hence it won't be possible to distinguish between partially mapped folios and others indeferred_split_scanwithout the boolean.Quoted 1 line
Just so I get this right: Are you saying that we might now add fully mapped folios to the deferred split queue and that's what you want to distinguish?
Yes
Quoted 2 lines
If that's the case, then could we use a bit in
folio->_flags_1instead?Yes, thats a good idea. Will create the below flag for the next revision
include/linux/page-flags.h +0
-0include/linux/page-flags.h enum pageflags { · line 193
Quoted 5 lines
include/linux/page-flags.h +3
-0@@ -189,6 +189,11 @@ enum pageflags { #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) +enum folioflags_1 { + /* The first 8 bits of folio->_flags_1 are used to keep track of folio order */ + FOLIO_PARTIALLY_MAPPED = 8, /* folio is partially mapped */ +}This might be what you want to achieve:
include/linux/page-flags.h +0
-0@@ -182,6 +182,7 @@ enum pageflags { /* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_active,
PG_large_rmappable=PG_workingset, /* anon or file-backed */ +PG_partially_mapped, /* was identified to be partially mapped */ };
#define PAGEFLAGS_MASK ((1UL <<NR_PAGEFLAGS) - 1)include/linux/page-flags.h +2
-1@@ -861,8 +862,9 @@ static inline void ClearPageCompound(struct page *page) ClearPageHead(page); } FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE) +FOLIO_FLAG(partially_mapped, FOLIO_SECOND_PAGE) #else -FOLIO_FLAG_FALSE(large_rmappable) +FOLIO_FLAG_FALSE(partially_mapped) #endif
#definePG_head_mask((1UL <<PG_head))
Quoted 5 lines
The downside is an atomic op to set/clear, but it should likely not really matter (initially, the flag will be clear, and we should only ever set it once when partially unmapping). If it hurts, we can reconsider.
[...]
I was looking for where the bits for flags_1 were specified! I just saw the start of enum pageflags, saw that compound order isn't specified anywhere over there and ignored the end :)
Yes, this is what I wanted to do. Thanks.