THP Shrinker
Subsystems (10)
v5 cover THP Shrinker
From: Alexander Zhu <[email protected]>
Transparent Hugepages use a larger page size of 2MB in comparison to normal sized pages that are 4kb. A larger page size allows for fewer TLB cache misses and thus more efficient use of the CPU. Using a larger page size also results in more memory waste, which can hurt performance in some use cases. THPs are currently enabled in the Linux Kernel by applications in limited virtual address ranges via the madvise system call. The THP shrinker tries to find a balance between increased use of THPs, and increased use of memory. It shrinks the size of memory by removing the underutilized THPs that are identified by the thp_utilization scanner.
In our experiments we have noticed that the least utilized THPs are almost entirely unutilized.
Sample Output:
Utilized[0-50]: 1331 680884
Utilized[51-101]: 9 3983
Utilized[102-152]: 3 1187
Utilized[153-203]: 0 0
Utilized[204-255]: 2 539
Utilized[256-306]: 5 1135
Utilized[307-357]: 1 192
Utilized[358-408]: 0 0
Utilized[409-459]: 1 57
Utilized[460-512]: 400 13
Last Scan Time: 223.98s
Last Scan Duration: 70.65s
Above is a sample obtained from one of our test machines when THP is always enabled. Of the 1331 THPs in this thp_utilization sample that have from 0-50 utilized subpages, we see that there are 680884 free pages. This comes out to 680884 / (512 * 1331) = 99.91% zero pages in the least utilized bucket. This represents 680884 * 4KB = 2.7GB memory waste.
Also note that the vast majority of pages are either in the least utilized [0-50] or most utilized [460-512] buckets. The least utilized THPs are responsible for almost all of the memory waste when THP is always enabled. Thus by clearing out THPs in the lowest utilization bucket we extract most of the improvement in CPU efficiency. We have seen similar results on our production hosts.
This patchset introduces the THP shrinker we have developed to identify and split the least utilized THPs. It includes the thp_utilization changes that groups anonymous THPs into buckets, the split_huge_page() changes that identify and zap zero 4KB pages within THPs and the shrinker changes. It should be noted that the split_huge_page() changes are based off previous work done by Yu Zhao.
In the future, we intend to allow additional tuning to the shrinker based on workload depending on CPU/IO/Memory pressure and the amount of anonymous memory. The long term goal is to eventually always enable THP for all applications and deprecate madvise entirely.
In production we thus far have observed 2-3% reduction in overall cpu usage on stateless web servers when THP is always enabled.
Alexander Zhu (5):
mm: add thp_utilization metrics to debugfs
mm: changes to split_huge_page() to free zero filled tail pages
mm: do not remap clean subpages when splitting isolated thp
mm: add selftests to split_huge_page() to verify unmap/zap of zero
pages
mm: THP low utilization shrinker
Documentation/admin-guide/mm/transhuge.rst | 9 + include/linux/huge_mm.h | 9 + include/linux/list_lru.h | 24 ++ include/linux/mm_types.h | 5 + include/linux/rmap.h | 2 +- include/linux/vm_event_item.h | 3 + mm/Makefile | 2 +- mm/huge_memory.c | 155 +++++++++++- mm/list_lru.c | 49 ++++ mm/migrate.c | 73 +++++- mm/migrate_device.c | 4 +- mm/page_alloc.c | 6 + mm/thp_utilization.c | 222 ++++++++++++++++++ mm/vmstat.c | 3 + .../selftests/vm/split_huge_page_test.c | 115 ++++++++- tools/testing/selftests/vm/vm_util.c | 23 ++ tools/testing/selftests/vm/vm_util.h | 3 + 17 files changed, 689 insertions(+), 18 deletions(-) create mode 100644 mm/thp_utilization.c
Signature · 1 line
2.30.2
PATCH v5 1/5 mm: add thp_utilization metrics to debugfs
From: Alexander Zhu <[email protected]>
This change introduces a tool that scans through all of physical memory for anonymous THPs and groups them into buckets based on utilization. It also includes an interface under /sys/kernel/debug/thp_utilization.
Sample Output:
Utilized[0-50]: 1331 680884
Utilized[51-101]: 9 3983
Utilized[102-152]: 3 1187
Utilized[153-203]: 0 0
Utilized[204-255]: 2 539
Utilized[256-306]: 5 1135
Utilized[307-357]: 1 192
Utilized[358-408]: 0 0
Utilized[409-459]: 1 57
Utilized[460-512]: 400 13
Last Scan Time: 223.98s
Last Scan Duration: 70.65s
This indicates that there are 1331 THPs that have between 0 and 50 utilized (non zero) pages. In total there are 680884 zero pages in this utilization bucket. THPs in the [0-50] bucket compose 76% of total THPs, and are responsible for 99% of total zero pages across all THPs. In other words, the least utilized THPs are responsible for almost all of the memory waste when THP is always enabled. Similar results have been observed across production workloads.
The last two lines indicate the timestamp and duration of the most recent scan through all of physical memory. Here we see that the last scan occurred 223.98 seconds after boot time and took 70.65 seconds.
Utilization of a THP is defined as the percentage of nonzero pages in the THP. The worker thread will scan through all of physical memory and obtain utilization of all anonymous THPs. It will gather this information by periodically scanning through all of physical memory for anonymous THPs, group them into buckets based on utilization, and report utilization information through debugfs under /sys/kernel/debug/thp_utilization.
---
v3 to v4
-changed thp_utilization_bucket() function to take folios, saves conversion between page and folio -added newlines where they were previously missing in v2-v3
-moved the thp utilization code out into its own file under mm/thp_utilization.c
-removed is_anonymous_transparent_hugepage function. Use folio_test_anon and folio_test_trans_huge instead. -changed thp_number_utilized_pages to use memchr_inv
v1 to v2
-reversed ordering of is_transparent_hugepage and PageAnon in is_anon_transparent_hugepage, page->mapping is only meaningful for user pages
RFC to v1
-Refactored out the code to obtain the thp_utilization_bucket, as that now has to be used in multiple places.
Documentation/admin-guide/mm/transhuge.rst | 9 + mm/Makefile | 2 +- mm/thp_utilization.c | 203 +++++++++++++++++++++ 3 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 mm/thp_utilization.c
Documentation/admin-guide/mm/transhuge.rst +9 -0
@@ -304,6 +304,15 @@ To identify what applications are mapping file transparent huge pages, it is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields for each mapping. +The utilization of transparent hugepages can be viewed by reading +``/sys/kernel/debug/thp_utilization``. The utilization of a THP is defined +as the ratio of non zero filled 4kb pages to the total number of pages in a +THP. The buckets are labelled by the range of total utilized 4kb pages with +one line per utilization bucket. Each line contains the total number of +THPs in that bucket and the total number of zero filled 4kb pages summed +over all THPs in that bucket. The last two lines show the timestamp and +duration respectively of the most recent scan over all of physical memory. + Note that reading the smaps file is expensive and reading it frequently will incur overhead.
mm/Makefile +1 -1
@@ -95,7 +95,7 @@ obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o -obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o thp_utilization.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o ifdef CONFIG_SWAP
mm/thp_utilization.c new file +203 -0
203 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v5 2/5 mm: changes to split_huge_page() to free zero filled tail pages
In reply to [email protected]From: Alexander Zhu <[email protected]>
Currently, when /sys/kernel/mm/transparent_hugepage/enabled=always is set there are a large number of transparent hugepages that are almost entirely zero filled. This is mentioned in a number of previous patchsets including:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/
[email protected]/
Currently, split_huge_page() does not have a way to identify zero filled pages within the THP. Thus these zero pages get remapped and continue to create memory waste. In this patch, we identify and free tail pages that are zero filled in split_huge_page(). In this way, we avoid mapping these pages back into page table entries and can free up unused memory within THPs. This is based off the previously mentioned patchset by Yu Zhao. However, we chose to free anonymous zero tail pages whenever they are encountered instead of only on reclaim or migration.
---
v4 to v5
-split out split_huge_page changes into three different patches. One for zapping zero pages, one for not remapping zero pages, and one for self tests.
include/linux/vm_event_item.h | 3 +++ mm/huge_memory.c | 37 +++++++++++++++++++++++++++++++++++ mm/vmstat.c | 3 +++ 3 files changed, 43 insertions(+)
include/linux/vm_event_item.h +3 -0
@@ -111,6 +111,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD THP_SPLIT_PUD , #endif +THP_SPLIT_FREE , +THP_SPLIT_UNMAP , +THP_SPLIT_REMAP_READONLY_ZERO_PAGE ,THP_ZERO_PAGE_ALLOC ,THP_ZERO_PAGE_ALLOC_FAILED , THP_SWPOUT,
mm/huge_memory.c +37 -0
@@ -2496,6 +2496,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,struct address_space *swap_cache =NULL ;unsigned long offset =0 ;unsigned int nr =thp_nr_pages (head); +LIST_HEAD (pages_to_free); +int nr_pages_to_free =0 ;int i;/* complete memcg works before add pages to LRU */
@@ -2572,6 +2574,34 @@ static void __split_huge_page(struct page *page, struct list_head *list,continue ;unlock_page (subpage); +/* +* If a tail page has only two references left, one inherited +* from the isolation of its head and the other from +* lru_add_page_tail() which we are about to drop, it means this +* tail page was concurrently zapped. Then we can safely free it +* and save page reclaim or migration the trouble of trying it. +*/ +if (list &&page_ref_freeze (subpage,2 )) { +VM_BUG_ON_PAGE (PageLRU (subpage), subpage); +VM_BUG_ON_PAGE (PageCompound (subpage), subpage); +VM_BUG_ON_PAGE (page_mapped (subpage), subpage); + +ClearPageActive (subpage); +ClearPageUnevictable (subpage); +list_move (&subpage->lru, &pages_to_free); + nr_pages_to_free++; +continue ; + } + +/* +* If a tail page has only one reference left, it will be freed +* by the call to free_page_and_swap_cache below. Since zero +* subpages are no longer remapped, there will only be one +* reference left in cases outside of reclaim or migration. +*/ +if (page_ref_count (subpage) ==1 ) + nr_pages_to_free++; +/* * Subpages may be freed if there wasn't any mapping * like if add_to_swap() is running on a lru page that
@@ -2581,6 +2611,13 @@ static void __split_huge_page(struct page *page, struct list_head *list,*/ free_page_and_swap_cache (subpage); } + +if (!nr_pages_to_free) +return ; + +mem_cgroup_uncharge_list (&pages_to_free); +free_unref_page_list (&pages_to_free); +count_vm_events (THP_SPLIT_FREE , nr_pages_to_free); }/* Racy check whether the huge page can be split */
mm/vmstat.c +3 -0
@@ -1359,6 +1359,9 @@ const char * const vmstat_text[] = {#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD "thp_split_pud" , #endif +"thp_split_free" , +"thp_split_unmap" , +"thp_split_remap_readonly_zero_page" ,"thp_zero_page_alloc" ,"thp_zero_page_alloc_failed" , "thp_swpout",
Signature · 1 line
2.30.2
PATCH v5 3/5 mm: do not remap clean subpages when splitting isolated thp
In reply to [email protected]From: Alexander Zhu <[email protected]>
Changes to avoid remap on zero pages that are free'd in split_huge_page(). Pages are not remapped except in the case of userfaultfd. In the case of userfaultfd we remap to the shared zero page, similar to what is done by KSM.
---
v1 to v2
-Only trigger the unmap_clean/zap in split_huge_page on anonymous THPs. We cannot zap zero pages for file THPs.
RFC to v1
-Added support to map to the read only zero page when splitting a THP registered with userfaultfd.
include/linux/rmap.h | 2 +- mm/huge_memory.c | 8 ++--- mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++----- mm/migrate_device.c | 4 +-- 4 files changed, 73 insertions(+), 14 deletions(-)
include/linux/rmap.h +1 -1
@@ -428,7 +428,7 @@ int folio_mkclean(struct folio *);int pfn_mkclean_range (unsigned long pfn,unsigned long nr_pages,pgoff_t pgoff,struct vm_area_struct *vma); -void remove_migration_ptes (struct folio *src,struct folio *dst,bool locked); +void remove_migration_ptes (struct folio *src,struct folio *dst,bool locked,bool unmap_clean);int page_mapped_in_vma (struct page *page,struct vm_area_struct *vma);
mm/huge_memory.c +4 -4
@@ -2373,7 +2373,7 @@ static void unmap_folio(struct folio *folio)try_to_unmap (folio, ttu_flags |TTU_IGNORE_MLOCK ); } -static void remap_page (struct folio *folio,unsigned long nr) +static void remap_page (struct folio *folio,unsigned long nr,bool unmap_clean) {int i =0 ;
@@ -2381,7 +2381,7 @@ static void remap_page(struct folio *folio, unsigned long nr)if (!folio_test_anon (folio))return ;for (;;) { -remove_migration_ptes (folio, folio, true); +remove_migration_ptes (folio, folio, true, unmap_clean); i +=folio_nr_pages (folio);if (i >= nr)break ;
@@ -2560,7 +2560,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }local_irq_enable (); -remap_page (folio, nr); +remap_page (folio, nr,PageAnon (head));if (PageSwapCache (head)) {swp_entry_t entry = { .val =page_private (head) };
@@ -2789,7 +2789,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)if (mapping)xas_unlock (&xas);local_irq_enable (); -remap_page (folio,folio_nr_pages (folio)); +remap_page (folio,folio_nr_pages (folio), false); ret = -EBUSY ; }
mm/migrate.c +66 -7
125 lines not on this page · raw message
mm/migrate_device.c +2 -2
@@ -421,7 +421,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,continue ; folio =page_folio (page); -remove_migration_ptes (folio, folio, false); +remove_migration_ptes (folio, folio, false, false); src_pfns[i] =0 ;folio_unlock (folio);
@@ -847,7 +847,7 @@ void migrate_device_finalize(unsigned long *src_pfns, src =page_folio (page); dst =page_folio (newpage); -remove_migration_ptes (src, dst, false); +remove_migration_ptes (src, dst, false, false);folio_unlock (src);if (is_zone_device_page (page))
Signature · 1 line
2.30.2
PATCH v5 4/5 mm: add selftests to split_huge_page() to verify unmap/zap of zero pages
In reply to [email protected]From: Alexander Zhu <[email protected]>
Self tests to verify the RssAnon value to make sure zero pages are not remapped except in the case of userfaultfd. Also includes a self test for the userfaultfd use case.
---
v1 to v2
-Modified split_huge_page self test based off more recent changes.
RFC to v1
Also added a self test to verify that userfaultfd change is working.
.../selftests/vm/split_huge_page_test.c | 115 +++++++++++++++++- tools/testing/selftests/vm/vm_util.c | 23 ++++ tools/testing/selftests/vm/vm_util.h | 3 + 3 files changed, 140 insertions(+), 1 deletion(-)
tools/testing/selftests/vm/split_huge_page_test.c +114 -1
@@ -16,6 +16,9 @@#include <sys/mount.h> #include <malloc.h> #include <stdbool.h> +#include <sys/syscall.h> /* Definition of SYS_* constants */ +#include <linux/userfaultfd.h> +#include <sys/ioctl.h> #include "vm_util.h" uint64_t pagesize;
@@ -88,6 +91,115 @@ static void write_debugfs(const char *fmt, ...) } } +static char *allocate_zero_filled_hugepage (size_t len) +{ +char *result; +size_t i; + + result =memalign (pmd_pagesize, len); +if (!result) { +printf ("Fail to allocate memory\n" ); +exit (EXIT_FAILURE ); + } + +madvise (result, len,MADV_HUGEPAGE ); + +for (i =0 ; i < len; i++) + result[i] = (char )0 ; + +return result; +} + +static void verify_rss_anon_split_huge_page_all_zeroes (char *one_page,int nr_hpages,size_t len) +{ +uint64_t rss_anon_before, rss_anon_after; +size_t i; + +if (!check_huge_anon (one_page,4 , pmd_pagesize)) { +printf ("No THP is allocated\n" ); +exit (EXIT_FAILURE ); + } + + rss_anon_before =rss_anon (); +if (!rss_anon_before) { +printf ("No RssAnon is allocated before split\n" ); +exit (EXIT_FAILURE ); + } + +/* split all THPs */ +write_debugfs (PID_FMT ,getpid (), (uint64_t )one_page, + (uint64_t )one_page + len); + +for (i =0 ; i < len; i++) +if (one_page[i] != (char )0 ) { +printf ("%ld byte corrupted\n" , i); +exit (EXIT_FAILURE ); + } + +if (!check_huge_anon (one_page,0 , pmd_pagesize)) { +printf ("Still AnonHugePages not split\n" ); +exit (EXIT_FAILURE ); + } + + rss_anon_after =rss_anon (); +if (rss_anon_after >= rss_anon_before) { +printf ("Incorrect RssAnon value. Before: %ld After: %ld\n" , + rss_anon_before, rss_anon_after); +exit (EXIT_FAILURE ); + } +} + +void split_pmd_zero_pages (void ) +{ +char *one_page; +int nr_hpages =4 ; +size_t len = nr_hpages * pmd_pagesize; + + one_page =allocate_zero_filled_hugepage (len); +verify_rss_anon_split_huge_page_all_zeroes (one_page, nr_hpages, len); +printf ("Split zero filled huge pages successful\n" ); +free (one_page); +} + +void split_pmd_zero_pages_uffd (void ) +{ +char *one_page; +int nr_hpages =4 ; +size_t len = nr_hpages * pmd_pagesize; +long uffd;/* userfaultfd file descriptor */ +struct uffdio_api uffdio_api; +struct uffdio_register uffdio_register; + +/* Create and enable userfaultfd object. */ + + uffd =syscall (__NR_userfaultfd,O_CLOEXEC |O_NONBLOCK ); +if (uffd ==-1 ) { +perror ("userfaultfd" ); +exit (1 ); + } + + uffdio_api.api =UFFD_API ; + uffdio_api.features =0 ; +if (ioctl (uffd,UFFDIO_API , &uffdio_api) ==-1 ) { +perror ("ioctl-UFFDIO_API" ); +exit (1 ); + } + + one_page =allocate_zero_filled_hugepage (len); + + uffdio_register.range.start = (unsigned long )one_page; + uffdio_register.range.len = len; + uffdio_register.mode =UFFDIO_REGISTER_MODE_WP ; +if (ioctl (uffd,UFFDIO_REGISTER , &uffdio_register) ==-1 ) { +perror ("ioctl-UFFDIO_REGISTER" ); +exit (1 ); + } + +verify_rss_anon_split_huge_page_all_zeroes (one_page, nr_hpages, len); +printf ("Split zero filled huge pages with uffd successful\n" ); +free (one_page); +} +void split_pmd_thp (void ) {char *one_page;
@@ -121,7 +233,6 @@ void split_pmd_thp(void)exit (EXIT_FAILURE ); } -if (check_huge_anon (one_page,0 , pmd_pagesize)) {printf ("Still AnonHugePages not split\n" );exit (EXIT_FAILURE );
@@ -301,6 +412,8 @@ int main(int argc, char **argv) pageshift =ffs (pagesize) -1 ; pmd_pagesize =read_pmd_pagesize (); +split_pmd_zero_pages (); +split_pmd_zero_pages_uffd ();split_pmd_thp ();split_pte_mapped_thp ();split_file_backed_thp ();
tools/testing/selftests/vm/vm_util.c +23 -0
@@ -6,6 +6,7 @@#define PMD_SIZE_FILE_PATH "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size"#define SMAP_FILE_PATH "/proc/self/smaps" +#define STATUS_FILE_PATH "/proc/self/status"#define MAX_LINE_LENGTH 500uint64_t pagemap_get_entry (int fd,char *start)
@@ -72,6 +73,28 @@ uint64_t read_pmd_pagesize(void)return strtoul (buf,NULL ,10 ); } +uint64_t rss_anon (void ) +{ +uint64_t rss_anon =0 ; +int ret; +FILE *fp; +char buffer[MAX_LINE_LENGTH ]; + + fp =fopen (STATUS_FILE_PATH ,"r" ); +if (!fp) +ksft_exit_fail_msg ("%s: Failed to open file %s\n" , __func__,STATUS_FILE_PATH ); + +if (!check_for_pattern (fp,"RssAnon:" , buffer,sizeof (buffer))) + goto err_out; + +if (sscanf (buffer,"RssAnon:%10ld kB" , &rss_anon) !=1 ) +ksft_exit_fail_msg ("Reading status error\n" ); + +err_out: +fclose (fp); +return rss_anon; +} +bool __check_huge (void *addr,char *pattern,int nr_hpages,uint64_t hpage_size) {
tools/testing/selftests/vm/vm_util.h +3 -0
@@ -1,12 +1,15 @@/* SPDX-License-Identifier: GPL-2.0 */ #include <stdint.h> #include <stdbool.h> +#include <stddef.h> +#include <stdio.h> uint64_t pagemap_get_entry (int fd,char *start);bool pagemap_is_softdirty (int fd,char *start);void clear_softdirty (void );bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void ); +uint64_t rss_anon (void );bool check_huge_anon (void *addr,int nr_hpages,uint64_t hpage_size);bool check_huge_file (void *addr,int nr_hpages,uint64_t hpage_size);bool check_huge_shmem (void *addr,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.30.2
PATCH v5 5/5 mm: THP low utilization shrinker
In reply to [email protected]From: Alexander Zhu <[email protected]>
This patch introduces a shrinker that will remove THPs in the lowest utilization bucket. As previously mentioned, we have observed that almost all of the memory waste when THPs are always enabled is contained in the lowest utilization bucket. The shrinker will add these THPs to a list_lru and split anonymous THPs based off information from kswapd. It requires the changes from thp_utilization to identify the least utilized THPs, and the changes to split_huge_page to identify and free zero pages within THPs.
---
v4 to v5
-fixed bug with lru_to_folio, was corrupting the folio
-fixed bug with memchr_inv in mm/thp_utilization. zero page should mean !memchr_inv(kaddr, 0, PAGE_SIZE)
v3 to v4
-added some comments regardling trylock
-change the relock to be unconditional in low_util_free_page
-only expose can_shrink_thp, abstract the thp_utilization and bucket logic to be private to mm/thp_utilization.c
v2 to v3
-put_page() after trylock_page in low_util_free_page. put() to be called after get() call -removed spin_unlock_irq in low_util_free_page above LRU_SKIP. There was a double unlock.
-moved spin_unlock_irq() to below list_lru_isolate() in low_util_free_page. This is to shorten the critical section. -moved lock_page in add_underutilized_thp such that we only lock when allocating and adding to the list_lru -removed list_lru_alloc in list_lru_add_page and list_lru_delete_page as these are no longer needed.
v1 to v2
-Changed lru_lock to be irq safe. Added irq_save and restore around list_lru adds/deletes.
-Changed low_util_free_page() to trylock the page, and if it fails, unlock lru_lock and return LRU_SKIP. This is to avoid deadlock between reclaim, which calls split_huge_page() and the THP Shrinker -Changed low_util_free_page() to unlock lru_lock, split_huge_page, then lock lru_lock. This way split_huge_page is not called with the lru_lock held. That leads to deadlock as split_huge_page calls on_each_cpu_mask -Changed list_lru_shrink_walk to list_lru_shrink_walk_irq.
RFC to v1
-Remove all THPs that are not in the top utilization bucket. This is what we have found to perform the best in production testing, we have found that there are an almost trivial number of THPs in the middle range of buckets that account for most of the memory waste. -Added check for THP utilization prior to split_huge_page for the THP Shrinker. This is to account for THPs that move to the top bucket, but were underutilized at the time they were added to the list_lru.
-Multiply the shrink_count and scan_count by HPAGE_PMD_NR. This is because a THP is 512 pages, and should count as 512 objects in reclaim. This way reclaim is triggered at a more appropriate frequency than in the RFC.
include/linux/huge_mm.h | 9 ++++ include/linux/list_lru.h | 24 +++++++++ include/linux/mm_types.h | 5 ++ mm/huge_memory.c | 110 ++++++++++++++++++++++++++++++++++++++- mm/list_lru.c | 49 +++++++++++++++++ mm/page_alloc.c | 6 +++ mm/thp_utilization.c | 21 +++++++- 7 files changed, 221 insertions(+), 3 deletions(-)
include/linux/huge_mm.h +9 -0
@@ -178,6 +178,8 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags); +bool can_shrink_thp (struct folio *folio); +void prep_transhuge_page (struct page *page);void free_transhuge_page (struct page *page);
@@ -189,6 +191,8 @@ static inline int split_huge_page(struct page *page) }void deferred_split_huge_page (struct page *page); +void add_underutilized_thp (struct page *page); +void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long address,bool freeze,struct folio *folio);
@@ -302,6 +306,11 @@ static inline struct list_head *page_deferred_list(struct page *page) return &page[2].deferred_list; } +static inline struct list_head *page_underutilized_thp_list (struct page *page) +{ +return &page[3 ].underutilized_thp_list; +} +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })#define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; })
include/linux/list_lru.h +24 -0
@@ -89,6 +89,18 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren*/ bool list_lru_add (struct list_lru *lru,struct list_head *item); +/** +* list_lru_add_page: add an element to the lru list's tail +* @list_lru: the lru pointer +* @page: the page containing the item +* @item: the item to be deleted. +* +* This function works the same as list_lru_add in terms of list +* manipulation. Used for non slab objects contained in the page. +* +* Return value: true if the list was updated, false otherwise +*/ +bool list_lru_add_page (struct list_lru *lru,struct page *page,struct list_head *item);/** * list_lru_del: delete an element to the lru list * @list_lru: the lru pointer
@@ -102,6 +114,18 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item);*/ bool list_lru_del (struct list_lru *lru,struct list_head *item); +/** +* list_lru_del_page: delete an element to the lru list +* @list_lru: the lru pointer +* @page: the page containing the item +* @item: the item to be deleted. +* +* This function works the same as list_lru_del in terms of list +* manipulation. Used for non slab objects contained in the page. +* +* Return value: true if the list was updated, false otherwise +*/ +bool list_lru_del_page (struct list_lru *lru,struct page *page,struct list_head *item);/** * list_lru_count_one: return the number of objects currently held by @lru * @lru: the lru pointer.
include/linux/mm_types.h +5 -0
@@ -152,6 +152,11 @@ struct page {/* For both global and memcg */ struct list_head deferred_list; }; +struct {/* Third tail page of compound page */ +unsigned long _compound_pad_3;/* compound_head */ +unsigned long _compound_pad_4; +struct list_head underutilized_thp_list; + };struct {/* Page table pages */ unsigned long _pt_pad_1;/* compound_head */ pgtable_t pmd_huge_pte;/* protected by page->ptl */
mm/huge_memory.c +108 -2
186 lines not on this page · raw message
mm/list_lru.c +49 -0
61 lines not on this page · raw message
mm/page_alloc.c +6 -0
@@ -1335,6 +1335,12 @@ static int free_tail_pages_check(struct page *head_page, struct page *page)* deferred_list.next -- ignore value. */ break ; +case 3 : +/* +* the third tail page: ->mapping is +* underutilized_thp_list.next -- ignore value. +*/ +break ;default :if (page->mapping !=TAIL_MAPPING ) {bad_page (page,"corrupted mapping in tail page" );
mm/thp_utilization.c +20 -1
@@ -98,13 +98,16 @@ static int thp_number_utilized_pages(struct folio *folio)int thp_nr_utilized_pages =HPAGE_PMD_NR ;void *kaddr;int i; +bool zero_page;if (!folio || !folio_test_anon (folio) || !folio_test_transhuge (folio))return -1 ;for (i =0 ; i <folio_nr_pages (folio); i++) { kaddr =kmap_local_folio (folio, i); -if (memchr_inv (kaddr,0 ,PAGE_SIZE )) + zero_page = !memchr_inv (kaddr,0 ,PAGE_SIZE ); + +if (zero_page) thp_nr_utilized_pages--;kunmap_local (kaddr);
@@ -113,6 +116,19 @@ static int thp_number_utilized_pages(struct folio *folio)return thp_nr_utilized_pages; } +bool can_shrink_thp (struct folio *folio) +{ +int bucket, num_utilized_pages; + +if (!folio || !folio_test_anon (folio) || !folio_test_transhuge (folio)) +return false; + + num_utilized_pages =thp_number_utilized_pages (folio); + bucket =thp_utilization_bucket (num_utilized_pages); + +return bucket >=0 && bucket <THP_UTIL_BUCKET_NR -1 ; +} +static void thp_scan_next_zone (void ) {struct timespec64 current_time;
@@ -167,6 +183,9 @@ static void thp_util_scan(unsigned long pfn_end)if (bucket <0 )continue ; +if (bucket <THP_UTIL_BUCKET_NR -1 ) +add_underutilized_thp (page); + thp_scan.buckets[bucket].nr_thps++; thp_scan.buckets[bucket].nr_zero_pages += (HPAGE_PMD_NR - num_utilized_pages); }
Signature · 1 line
2.30.2
kernel test robot
build bot[lkp] [+100 bytes kernel size regression] [i386-tinyconfig] [07b3ee621e] mm: THP low utilization shrinker
FYI, we noticed a +100 bytes kernel size regression due to commit:
FYI, we noticed a +100 bytes kernel size regression due to commit:
commit: 07b3ee621ef564a06c744fefe220980ae1f91f64 (mm: THP low utilization shrinker)
url: https://github.com/intel-lab-lkp/linux/commits/alexlzhu-fb-com/THP-Shrinker/20221026-083106
base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
patch link: https://lore.kernel.org/linux-mm/4880884498c6f0cf3de0d1b7875151a21869a493.1666743422.git.alexlzhu@fb.com
patch subject: [PATCH v5 5/5] mm: THP low utilization shrinker
Details as below (size data is obtained by nm --size-sort vmlinux):
df3f3903: mm: add selftests to split_huge_page() to verify unmap/zap of zero pages 07b3ee62: mm: THP low utilization shrinker
+------------------------+----------+----------+-------+ | symbol | df3f3903 | 07b3ee62 | delta | +------------------------+----------+----------+-------+ | bzImage | 491424 | 491616 | 192 | | nm.T.list_lru_add_page | 0 | 57 | 57 | | nm.T.list_lru_del_page | 0 | 55 | 55 | | nm.T.list_lru_isolate | 21 | 15 | -6 | | nm.T.list_lru_del | 42 | 36 | -6 | +------------------------+----------+----------+-------+
Thanks