Subsystems (8)
Subject: [PATCH mm-unstable v2 0/9] mm: add file/shmem support to MADV_COLLAPSE
v2 Forward
Mostly a RESEND: rebase on latest mm-unstable + minor bug fixes from kernel test robot.
--------------------------------
This series builds on top of the previous "mm: userspace hugepage collapse" series which introduced the MADV_COLLAPSE madvise mode and added support for private, anonymous mappings[1], by adding support for file and shmem backed memory to CONFIG_READ_ONLY_THP_FOR_FS=y kernels.
File and shmem support have been added with effort to align with existing MADV_COLLAPSE semantics and policy decisions[2]. Collapse of shmem-backed memory ignores kernel-guiding directives and heuristics including all sysfs settings (transparent_hugepage/shmem_enabled), and tmpfs huge= mount options (shmem always supports large folios). Like anonymous mappings, on successful return of MADV_COLLAPSE on file/shmem memory, the contents of memory mapped by the addresses provided will be synchronously pmd-mapped THPs.
This functionality unlocks two important uses:
(1) Immediately back executable text by THPs. Current support provided
by CONFIG_READ_ONLY_THP_FOR_FS may take a long time on a large system which might impair services from serving at their full rated load after (re)starting. Tricks like mremap(2)'ing text onto anonymous memory to immediately realize iTLB performance prevents page sharing and demand paging, both of which increase steady state memory footprint. Now, we can have the best of both worlds: Peak upfront performance and lower RAM footprints.
(2) userfaultfd-based live migration of virtual machines satisfy UFFD
faults by fetching native-sized pages over the network (to avoid latency of transferring an entire hugepage). However, after guest memory has been fully copied to the new host, MADV_COLLAPSE can be used to immediately increase guest performance.
khugepaged has received a small improvement by association and can now detect and collapse pte-mapped THPs. However, there is still work to be done along the file collapse path. Compound pages of arbitrary order still needs to be supported and THP collapse needs to be converted to using folios in general. Eventually, we'd like to move away from the read-only and executable-mapped constraints currently imposed on eligible files and support any inode claiming huge folio support. That said, I think the series as-is covers enough to claim that MADV_COLLAPSE supports file/shmem memory.
Patches 1-3 Implement the guts of the series.
Patch 4 Is a tracepoint for debugging.
Patches 5-8 Refactor existing khugepaged selftests to work with new
memory types.
Patch 9 Adds a userfaultfd selftest mode to mimic a functional test
of UFFDIO_REGISTER_MODE_MINOR+MADV_COLLAPSE live migration.
Applies against mm-unstable.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
v1 -> v2:
- Add missing definition for khugepaged_add_pte_mapped_thp() in !CONFIG_SHEM builds, in "mm/khugepaged: attempt to map file/shmem-backed pte-mapped THPs by pmds"
- Minor bugfixes in "mm/madvise: add file and shmem support to MADV_COLLAPSE" for !CONFIG_SHMEM, !CONFIG_TRANSPARENT_HUGEPAGE and some compiler settings.
- Rebased on latest mm-unstable
Zach O'Keefe (9):
mm/shmem: add flag to enforce shmem THP in hugepage_vma_check() mm/khugepaged: attempt to map file/shmem-backed pte-mapped THPs by
pmds
mm/madvise: add file and shmem support to MADV_COLLAPSE mm/khugepaged: add tracepoint to hpage_collapse_scan_file() selftests/vm: dedup THP helpers
selftests/vm: modularize thp collapse memory operations selftests/vm: add thp collapse file and tmpfs testing selftests/vm: add thp collapse shmem testing selftests/vm: add selftest for MADV_COLLAPSE of uffd-minor memory
include/linux/khugepaged.h | 13 +- include/linux/shmem_fs.h | 10 +- include/trace/events/huge_memory.h | 36 + kernel/events/uprobes.c | 2 +- mm/huge_memory.c | 2 +- mm/khugepaged.c | 289 ++++-- mm/shmem.c | 18 +- tools/testing/selftests/vm/Makefile | 2 + tools/testing/selftests/vm/khugepaged.c | 828 ++++++++++++------ tools/testing/selftests/vm/soft-dirty.c | 2 +- .../selftests/vm/split_huge_page_test.c | 12 +- tools/testing/selftests/vm/userfaultfd.c | 171 +++- tools/testing/selftests/vm/vm_util.c | 36 +- tools/testing/selftests/vm/vm_util.h | 5 +- 14 files changed, 1040 insertions(+), 386 deletions(-)
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 1/9 mm/shmem: add flag to enforce shmem THP in hugepage_vma_check()
Extend 'mm/thp: add flag to enforce sysfs THP in hugepage_vma_check()' to shmem, allowing callers to ignore /sys/kernel/transparent_hugepage/shmem_enabled and tmpfs huge= mount.
This is intended to be used by MADV_COLLAPSE, and the rationale is analogous to the anon/file case: MADV_COLLAPSE is not coupled to directives that advise the kernel's decisions on when THPs should be considered eligible. shmem/tmpfs always claims large folio support, regardless of sysfs or mount options.
--- include/linux/shmem_fs.h | 10 ++++++---- mm/huge_memory.c | 2 +- mm/shmem.c | 18 +++++++++--------- 3 files changed, 16 insertions(+), 14 deletions(-)
include/linux/shmem_fs.h +6 -4
@@ -92,11 +92,13 @@ extern struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,extern void shmem_truncate_range (struct inode *inode,loff_t start,loff_t end);int shmem_unuse (unsigned int type); -extern bool shmem_is_huge (struct vm_area_struct *vma, -struct inode *inode,pgoff_t index); -static inline bool shmem_huge_enabled (struct vm_area_struct *vma) +extern bool shmem_is_huge (struct vm_area_struct *vma,struct inode *inode, +pgoff_t index,bool shmem_huge_force); +static inline bool shmem_huge_enabled (struct vm_area_struct *vma, +bool shmem_huge_force) { -return shmem_is_huge (vma,file_inode (vma->vm_file), vma->vm_pgoff); +return shmem_is_huge (vma,file_inode (vma->vm_file), vma->vm_pgoff, + shmem_huge_force); }extern unsigned long shmem_swap_usage (struct vm_area_struct *vma); extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,
mm/huge_memory.c +1 -1
@@ -119,7 +119,7 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,* own flags. */ if (!in_pf &&shmem_file (vma->vm_file)) -return shmem_huge_enabled (vma); +return shmem_huge_enabled (vma, !enforce_sysfs);/* Enforce sysfs THP requirements as necessary */ if (enforce_sysfs &&
mm/shmem.c +9 -9
@@ -472,20 +472,20 @@ static bool shmem_confirm_swap(struct address_space *mapping,static int shmem_huge__read_mostly =SHMEM_HUGE_NEVER ; -bool shmem_is_huge (struct vm_area_struct *vma, -struct inode *inode,pgoff_t index) +bool shmem_is_huge (struct vm_area_struct *vma,struct inode *inode, +pgoff_t index,bool shmem_huge_force) {loff_t i_size;if (!S_ISREG (inode->i_mode))return false; -if (shmem_huge ==SHMEM_HUGE_DENY ) -return false;if (vma && ((vma->vm_flags &VM_NOHUGEPAGE ) ||test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags)))return false; -if (shmem_huge ==SHMEM_HUGE_FORCE ) +if (shmem_huge ==SHMEM_HUGE_FORCE || shmem_huge_force)return true; +if (shmem_huge ==SHMEM_HUGE_DENY ) +return false;switch (SHMEM_SB (inode->i_sb)->huge) {case SHMEM_HUGE_ALWAYS :
@@ -680,8 +680,8 @@ static long shmem_unused_huge_count(struct super_block *sb,#define shmem_huge SHMEM_HUGE_DENY -bool shmem_is_huge (struct vm_area_struct *vma, -struct inode *inode,pgoff_t index) +bool shmem_is_huge (struct vm_area_struct *vma,struct inode *inode, +pgoff_t index,bool shmem_huge_force) {return false; }
@@ -1069,7 +1069,7 @@ static int shmem_getattr(struct user_namespace *mnt_userns, STATX_ATTR_NODUMP);generic_fillattr (&init_user_ns, inode, stat); -if (shmem_is_huge (NULL , inode,0 )) +if (shmem_is_huge (NULL , inode,0 , false)) stat->blksize =HPAGE_PMD_SIZE ;if (request_mask &STATX_BTIME ) {
@@ -1910,7 +1910,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,return 0 ; } -if (!shmem_is_huge (vma, inode, index)) +if (!shmem_is_huge (vma, inode, index, false)) goto alloc_nohuge; huge_gfp =vma_thp_gfp_mask (vma);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 2/9 mm/khugepaged: attempt to map file/shmem-backed pte-mapped THPs by pmds
In reply to Zach O'KeefeThe main benefit of THPs are that they can be mapped at the pmd level, increasing the likelihood of TLB hit and spending less cycles in page table walks. pte-mapped hugepages - that is - hugepage-aligned compound pages of order HPAGE_PMD_ORDER - although being contiguous in physical memory, don't have this advantage. In fact, one could argue they are detrimental to system performance overall since they occupy a precious hugepage-aligned/sized region of physical memory that could otherwise be used more effectively. Additionally, pte-mapped hugepages can be the cheapest memory to collapse for khugepaged since no new hugepage allocation or copying of memory contents is necessary - we only need to update the mapping page tables.
In the anonymous collapse path, we are able to collapse pte-mapped hugepages (albeit, perhaps suboptimally), but the file/shmem path makes no effort when compound pages (of any order) are encountered.
Identify pte-mapped hugepages in the file/shmem collapse path. In khugepaged context, attempt to update page tables mapping this hugepage. Note that these collapses still count towards the /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed counter, and if the pte-mapped hugepage was also mapped into multiple process' address spaces, could be incremented for each page table update. Since we increment the counter when a pte-mapped hugepage is successfully added to the list of to-collapse pte-mapped THPs, it's possible that we never actually update the page table either. This is different from how file/shmem pages_collapsed accounting works today where only a successful page cache update is counted (it's also possible here that no page tables are actually changed). Though it incurs some slop, this is preferred to either not accounting for the event at all, or plumbing through data in struct mm_slot on whether to account for the collapse or not.
Note that work still needs to be done to support arbitrary compound pages, and that this should all be converted to using folios.
--- include/trace/events/huge_memory.h | 1 + mm/khugepaged.c | 49 ++++++++++++++++++++++++++---- 2 files changed, 44 insertions(+), 6 deletions(-)
include/trace/events/huge_memory.h +1 -0
@@ -17,6 +17,7 @@ EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \ + EM( SCAN_PTE_MAPPED_HUGEPAGE, "pte_mapped_hugepage") \ EM( SCAN_PAGE_RO, "no_writable_page") \ EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \EM (SCAN_PAGE_NULL ,"page_null" ) \
mm/khugepaged.c +43 -6
104 lines not on this page · raw message
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 3/9 mm/madvise: add file and shmem support to MADV_COLLAPSE
In reply to Zach O'KeefeAdd support for MADV_COLLAPSE to collapse shmem-backed and file-backed memory into THPs (requires CONFIG_READ_ONLY_THP_FOR_FS=y).
On success, the backing memory will be a hugepage. For the memory range and process provided, the page tables will synchronously have a huge pmd installed, mapping the THP. Other mappings of the file extent mapped by the memory range may be added to a set of entries that khugepaged will later process and attempt update their page tables to map the THP by a pmd.
This functionality unlocks two important uses:
(1) Immediately back executable text by THPs. Current support provided
by CONFIG_READ_ONLY_THP_FOR_FS may take a long time on a large system which might impair services from serving at their full rated load after (re)starting. Tricks like mremap(2)'ing text onto anonymous memory to immediately realize iTLB performance prevents page sharing and demand paging, both of which increase steady state memory footprint. Now, we can have the best of both worlds: Peak upfront performance and lower RAM footprints.
(2) userfaultfd-based live migration of virtual machines satisfy UFFD
faults by fetching native-sized pages over the network (to avoid latency of transferring an entire hugepage). However, after guest memory has been fully copied to the new host, MADV_COLLAPSE can be used to immediately increase guest performance.
--- include/linux/khugepaged.h | 13 +- include/trace/events/huge_memory.h | 1 + kernel/events/uprobes.c | 2 +- mm/khugepaged.c | 241 ++++++++++++++++++++++------- 4 files changed, 198 insertions(+), 59 deletions(-)
include/linux/khugepaged.h +8 -5
@@ -16,11 +16,13 @@ extern void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags);extern void khugepaged_min_free_kbytes_update (void );#ifdef CONFIG_SHMEM -extern void collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr); +extern int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr, +bool install_pmd);#else -static inline void collapse_pte_mapped_thp (struct mm_struct *mm, -unsigned long addr) +static inline int collapse_pte_mapped_thp (struct mm_struct *mm, +unsigned long addr,bool install_pmd) { +return 0 ; }#endif
@@ -46,9 +48,10 @@ static inline void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { } -static inline void collapse_pte_mapped_thp (struct mm_struct *mm, -unsigned long addr) +static inline int collapse_pte_mapped_thp (struct mm_struct *mm, +unsigned long addr,bool install_pmd) { +return 0 ; }static inline void khugepaged_min_free_kbytes_update (void )
include/trace/events/huge_memory.h +1 -0
@@ -11,6 +11,7 @@ EM( SCAN_FAIL, "failed") \ EM( SCAN_SUCCEED, "succeeded") \ EM( SCAN_PMD_NULL, "pmd_null") \ + EM( SCAN_PMD_NON_PRESENT, "pmd_non_present") \ EM( SCAN_PMD_MAPPED, "page_pmd_mapped") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \EM (SCAN_EXCEED_SWAP_PTE ,"exceed_swap_pte" ) \
kernel/events/uprobes.c +1 -1
@@ -553,7 +553,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,/* try collapse pmd for compound page */ if (!ret && orig_page_huge) -collapse_pte_mapped_thp (mm, vaddr); +collapse_pte_mapped_thp (mm, vaddr, false);return ret; }
mm/khugepaged.c +188 -53
450 lines not on this page · raw message
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 4/9 mm/khugepaged: add tracepoint to hpage_collapse_scan_file()
In reply to Zach O'KeefeAdd huge_memory:trace_mm_khugepaged_scan_file tracepoint to hpage_collapse_scan_file() analogously to hpage_collapse_scan_pmd(). While this change is targeted at debugging MADV_COLLAPSE pathway, the "mm_khugepaged" prefix is retained for symmetry with huge_memory:trace_mm_khugepaged_scan_pmd, which retains it's legacy name to prevent changing kernel ABI as much as possible.
--- include/trace/events/huge_memory.h | 34 ++++++++++++++++++++++++++++++ mm/khugepaged.c | 3 ++- 2 files changed, 36 insertions(+), 1 deletion(-)
include/trace/events/huge_memory.h +34 -0
@@ -169,5 +169,39 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, __entry->ret) ); +TRACE_EVENT (mm_khugepaged_scan_file, + + TP_PROTO(struct mm_struct *mm, struct page *page, const char *filename, + int present, int swap, int result), + +TP_ARGS (mm, page, filename, present, swap, result), + +TP_STRUCT__entry ( + __field(struct mm_struct *, mm) + __field(unsigned long, pfn) + __string(filename, filename) + __field(int, present) + __field(int, swap) +__field (int, result) + ), + +TP_fast_assign ( + __entry->mm = mm; + __entry->pfn = page ? page_to_pfn(page) : -1; + __assign_str(filename, filename); + __entry->present = present; + __entry->swap = swap; + __entry->result = result; + ), + +TP_printk ("mm=%p, scan_pfn=0x%lx, filename=%s, present=%d, swap=%d, result=%s" , + __entry->mm, + __entry->pfn, +__get_str (filename), + __entry->present, + __entry->swap, +__print_symbolic (__entry->result,SCAN_STATUS )) +); +#endif /* __HUGE_MEMORY_H */ #include <trace/define_trace.h>
mm/khugepaged.c +2 -1
@@ -2171,7 +2171,8 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } } -/* TODO: tracepoints */ +trace_mm_khugepaged_scan_file (mm, page, file->f_path.dentry->d_iname, + present, swap, result);return result; }#else
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 5/9 selftests/vm: dedup THP helpers
In reply to Zach O'KeefeThese files:
tools/testing/selftests/vm/vm_util.ctools/testing/selftests/vm/khugepaged.c
Both contain logic to:
1) Determine hugepage size on current system
2) Read /proc/self/smaps to determine number of THPs at an address
Refactor selftests/vm/khugepaged.c to use the vm_util common helpers and add it as a build dependency.
Since selftests/vm/khugepaged.c is the largest user of check_huge(), change the signature of check_huge() to match selftests/vm/khugepaged.c's useage: take an expected number of hugepages, and return a bool indicating if the correct number of hugepages were found. Add a wrapper, check_huge_anon(), in anticipation of checking smaps for file and shmem hugepages.
Update existing callsites to use the new pattern / function.
Likewise, check_for_pattern() was duplicated, and it's a general enough helper to include in vm_util helpers as well.
--- tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/khugepaged.c | 64 ++----------------- tools/testing/selftests/vm/soft-dirty.c | 2 +- .../selftests/vm/split_huge_page_test.c | 12 ++-- tools/testing/selftests/vm/vm_util.c | 26 +++++--- tools/testing/selftests/vm/vm_util.h | 3 +- 6 files changed, 32 insertions(+), 76 deletions(-)
tools/testing/selftests/vm/Makefile +1 -0
@@ -97,6 +97,7 @@ TEST_FILES += va_128TBswitch.sh include ../lib.mk +$(OUTPUT)/khugepaged: vm_util.c $(OUTPUT)/madv_populate: vm_util.c $(OUTPUT)/soft-dirty: vm_util.c $(OUTPUT)/split_huge_page_test: vm_util.c
tools/testing/selftests/vm/khugepaged.c +7 -57
98 lines not on this page · raw message
tools/testing/selftests/vm/soft-dirty.c +1 -1
@@ -91,7 +91,7 @@ static void test_hugepage(int pagemap_fd, int pagesize)for (i =0 ; i < hpage_len; i++) map[i] = (char )i; -if (check_huge (map)) { +if (check_huge_anon (map,1 , hpage_len)) {ksft_test_result_pass ("Test %s huge page allocation\n" , __func__);clear_softdirty ();
tools/testing/selftests/vm/split_huge_page_test.c +4 -8
@@ -92,7 +92,6 @@ void split_pmd_thp(void) {char *one_page;size_t len =4 * pmd_pagesize; -uint64_t thp_size;size_t i; one_page =memalign (pmd_pagesize, len);
@@ -107,8 +106,7 @@ void split_pmd_thp(void)for (i =0 ; i < len; i++) one_page[i] = (char )i; - thp_size =check_huge (one_page); -if (!thp_size) { +if (!check_huge_anon (one_page,1 , pmd_pagesize)) {printf ("No THP is allocated\n" );exit (EXIT_FAILURE ); }
@@ -124,9 +122,8 @@ void split_pmd_thp(void) } - thp_size =check_huge (one_page); -if (thp_size) { -printf ("Still %ld kB AnonHugePages not split\n" , thp_size); +if (check_huge_anon (one_page,0 , pmd_pagesize)) { +printf ("Still AnonHugePages not split\n" );exit (EXIT_FAILURE ); }
@@ -172,8 +169,7 @@ void split_pte_mapped_thp(void)for (i =0 ; i < len; i++) one_page[i] = (char )i; - thp_size =check_huge (one_page); -if (!thp_size) { +if (!check_huge_anon (one_page,1 , pmd_pagesize)) {printf ("No THP is allocated\n" );exit (EXIT_FAILURE ); }
tools/testing/selftests/vm/vm_util.c +17 -9
@@ -42,9 +42,9 @@ void clear_softdirty(void)ksft_exit_fail_msg ("writing clear_refs failed\n" ); } -static bool check_for_pattern (FILE *fp,const char *pattern,char *buf) +bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len) { -while (fgets (buf,MAX_LINE_LENGTH , fp) !=NULL ) { +while (fgets (buf, len, fp)) {if (!strncmp (buf, pattern,strlen (pattern)))return true; }
@@ -72,9 +72,10 @@ uint64_t read_pmd_pagesize(void)return strtoul (buf,NULL ,10 ); } -uint64_t check_huge (void *addr) +bool __check_huge (void *addr,char *pattern,int nr_hpages, +uint64_t hpage_size) { -uint64_t thp =0 ; +uint64_t thp =-1 ;int ret;FILE *fp;char buffer[MAX_LINE_LENGTH ];
@@ -89,20 +90,27 @@ uint64_t check_huge(void *addr)if (!fp)ksft_exit_fail_msg ("%s: Failed to open file %s\n" , __func__,SMAP_FILE_PATH ); -if (!check_for_pattern (fp, addr_pattern, buffer)) +if (!check_for_pattern (fp, addr_pattern, buffer,sizeof (buffer))) goto err_out;/* -* Fetch the AnonHugePages: in the same block and check the number of +* Fetch the pattern in the same block and check the number of * hugepages. */ -if (!check_for_pattern (fp,"AnonHugePages:" , buffer)) +if (!check_for_pattern (fp, pattern, buffer,sizeof (buffer))) goto err_out; -if (sscanf (buffer,"AnonHugePages:%10ld kB" , &thp) !=1 ) +snprintf (addr_pattern,MAX_LINE_LENGTH ,"%s%%9ld kB" , pattern); + +if (sscanf (buffer, addr_pattern, &thp) !=1 )ksft_exit_fail_msg ("Reading smap error\n" ); err_out:fclose (fp); -return thp; +return thp == (nr_hpages * (hpage_size >>10 )); +} + +bool check_huge_anon (void *addr,int nr_hpages,uint64_t hpage_size) +{ +return __check_huge (addr,"AnonHugePages: " , nr_hpages, hpage_size); }
tools/testing/selftests/vm/vm_util.h +2 -1
@@ -5,5 +5,6 @@uint64_t pagemap_get_entry (int fd,char *start);bool pagemap_is_softdirty (int fd,char *start);void clear_softdirty (void ); +bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void ); -uint64_t check_huge (void *addr); +bool check_huge_anon (void *addr,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 6/9 selftests/vm: modularize thp collapse memory operations
In reply to Zach O'KeefeModularize operations to setup, cleanup, fault, and check for huge pages, for a given memory type. This allows reusing existing tests with additional memory types by defining new memory operations. Following patches will add file and shmem memory types.
--- tools/testing/selftests/vm/khugepaged.c | 366 +++++++++++++----------- 1 file changed, 200 insertions(+), 166 deletions(-)
tools/testing/selftests/vm/khugepaged.c +200 -166
@@ -28,8 +28,16 @@ static int hpage_pmd_nr;#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"#define PID_SMAPS "/proc/self/smaps" +struct mem_ops { +void *(*setup_area)(int nr_hpages); +void (*cleanup_area)(void *p,unsigned long size); +void (*fault)(void *p,unsigned long start,unsigned long end); +bool (*check_huge)(void *addr,int nr_hpages); +}; +struct collapse_context { -void (*collapse)(const char *msg,char *p,int nr_hpages,bool expect); +void (*collapse)(const char *msg,char *p,int nr_hpages, +struct mem_ops *ops,bool expect);bool enforce_pte_scan_limits; };
@@ -353,11 +361,6 @@ static void save_settings(void)signal (SIGQUIT , restore_settings); } -static bool check_huge (void *addr,int nr_hpages) -{ -return check_huge_anon (addr, nr_hpages, hpage_pmd_size); -} -#define MAX_LINE_LENGTH 500static bool check_swap (void *addr,unsigned long size) {
@@ -431,18 +434,25 @@ static void fill_memory(int *p, unsigned long start, unsigned long end)* Returns pmd-mapped hugepage in VMA marked VM_HUGEPAGE, filled with * validate_memory()'able contents. */ -static void *alloc_hpage (void ) +static void *alloc_hpage (struct mem_ops *ops) { -void *p; +void *p = ops->setup_area (1 ); - p =alloc_mapping (1 ); + ops->fault (p,0 , hpage_pmd_size); +if (madvise (p, hpage_pmd_size,MADV_HUGEPAGE )) { +perror ("madvise(MADV_HUGEPAGE)" ); +exit (EXIT_FAILURE ); + }printf ("Allocate huge page..." ); -madvise (p, hpage_pmd_size,MADV_HUGEPAGE ); -fill_memory (p,0 , hpage_pmd_size); -if (check_huge (p,1 )) -success ("OK" ); -else -fail ("Fail" ); +if (madvise (p, hpage_pmd_size,MADV_COLLAPSE )) { +perror ("madvise(MADV_COLLAPSE)" ); +exit (EXIT_FAILURE ); + } +if (!ops->check_huge (p,1 )) { +perror ("madvise(MADV_COLLAPSE)" ); +exit (EXIT_FAILURE ); + } +success ("OK" );return p; }
@@ -459,18 +469,40 @@ static void validate_memory(int *p, unsigned long start, unsigned long end) } } -static void madvise_collapse (const char *msg,char *p,int nr_hpages, -bool expect) +static void *anon_setup_area (int nr_hpages) +{ +return alloc_mapping (nr_hpages); +} + +static void anon_cleanup_area (void *p,unsigned long size) +{ +munmap (p, size); +} + +static void anon_fault (void *p,unsigned long start,unsigned long end) +{ +fill_memory (p, start, end); +} + +static bool anon_check_huge (void *addr,int nr_hpages) +{ +return check_huge_anon (addr, nr_hpages, hpage_pmd_size); +} + +static struct mem_ops anon_ops = { + .setup_area = &anon_setup_area, + .cleanup_area = &anon_cleanup_area, + .fault = &anon_fault, + .check_huge = &anon_check_huge, +}; + +static void __madvise_collapse (const char *msg,char *p,int nr_hpages, +struct mem_ops *ops,bool expect) {int ret;struct settings settings = *current_settings ();printf ("%s..." , msg); -/* Sanity check */ -if (!check_huge (p,0 )) { -printf ("Unexpected huge page\n" ); -exit (EXIT_FAILURE ); - }/* * Prevent khugepaged interference and tests that MADV_COLLAPSE
@@ -482,9 +514,10 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages,/* Clear VM_NOHUGEPAGE */ madvise (p, nr_hpages * hpage_pmd_size,MADV_HUGEPAGE ); ret =madvise (p, nr_hpages * hpage_pmd_size,MADV_COLLAPSE ); +if (((bool )ret) == expect)fail ("Fail: Bad return value" ); -else if (check_huge (p, nr_hpages) != expect) +else if (!ops->check_huge (p, expect ? nr_hpages :0 ))fail ("Fail: check_huge()" );else success ("OK" );
@@ -492,14 +525,26 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages,pop_settings (); } +static void madvise_collapse (const char *msg,char *p,int nr_hpages, +struct mem_ops *ops,bool expect) +{ +/* Sanity check */ +if (!ops->check_huge (p,0 )) { +printf ("Unexpected huge page\n" ); +exit (EXIT_FAILURE ); + } +__madvise_collapse (msg, p, nr_hpages, ops, expect); +} +#define TICK 500000 -static bool wait_for_scan (const char *msg,char *p,int nr_hpages) +static bool wait_for_scan (const char *msg,char *p,int nr_hpages, +struct mem_ops *ops) {int full_scans;int timeout =6 ;/* 3 seconds */ /* Sanity check */ -if (!check_huge (p,0 )) { +if (!ops->check_huge (p,0 )) {printf ("Unexpected huge page\n" );exit (EXIT_FAILURE ); }
@@ -511,7 +556,7 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages)printf ("%s..." , msg);while (timeout--) { -if (check_huge (p, nr_hpages)) +if (ops->check_huge (p, nr_hpages))break ;if (read_num ("khugepaged/full_scans" ) >= full_scans)break ;
@@ -525,19 +570,20 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages) }static void khugepaged_collapse (const char *msg,char *p,int nr_hpages, -bool expect) +struct mem_ops *ops,bool expect) { -if (wait_for_scan (msg, p, nr_hpages)) { +if (wait_for_scan (msg, p, nr_hpages, ops)) {if (expect)fail ("Timeout" );else success ("OK" );return ; - }else if (check_huge (p, nr_hpages) == expect) { + } + +if (ops->check_huge (p, expect ? nr_hpages :0 ))success ("OK" ); - }else { +else fail ("Fail" ); - } }static void alloc_at_fault (void )
@@ -551,7 +597,7 @@ static void alloc_at_fault(void) p =alloc_mapping (1 ); *p =1 ;printf ("Allocate huge page on fault..." ); -if (check_huge (p,1 )) +if (check_huge_anon (p,1 , hpage_pmd_size))success ("OK" );else fail ("Fail" );
@@ -560,49 +606,48 @@ static void alloc_at_fault(void)madvise (p, page_size,MADV_DONTNEED );printf ("Split huge PMD on MADV_DONTNEED..." ); -if (check_huge (p,0 )) +if (check_huge_anon (p,0 , hpage_pmd_size))success ("OK" );else fail ("Fail" );munmap (p, hpage_pmd_size); } -static void collapse_full (struct collapse_context *c) +static void collapse_full (struct collapse_context *c,struct mem_ops *ops) {void *p;int nr_hpages =4 ;unsigned long size = nr_hpages * hpage_pmd_size; - p =alloc_mapping (nr_hpages); -fill_memory (p,0 , size); + p = ops->setup_area (nr_hpages); + ops->fault (p,0 , size); c->collapse ("Collapse multiple fully populated PTE table" , p, nr_hpages, - true); + ops, true);validate_memory (p,0 , size); -munmap (p, size); + ops->cleanup_area (p, size); } -static void collapse_empty (struct collapse_context *c) +static void collapse_empty (struct collapse_context *c,struct mem_ops *ops) {void *p; - p =alloc_mapping (1 ); - c->collapse ("Do not collapse empty PTE table" , p,1 , false); -munmap (p, hpage_pmd_size); + p = ops->setup_area (1 ); + c->collapse ("Do not collapse empty PTE table" , p,1 , ops, false); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_single_pte_entry (struct collapse_context *c) +static void collapse_single_pte_entry (struct collapse_context *c,struct mem_ops *ops) {void *p; - p =alloc_mapping (1 ); -fill_memory (p,0 , page_size); + p = ops->setup_area (1 ); + ops->fault (p,0 , page_size); c->collapse ("Collapse PTE table with single PTE entry present" , p, -1 , true); -validate_memory (p,0 , page_size); -munmap (p, hpage_pmd_size); +1 , ops, true); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_max_ptes_none (struct collapse_context *c) +static void collapse_max_ptes_none (struct collapse_context *c,struct mem_ops *ops) {int max_ptes_none = hpage_pmd_nr /2 ;struct settings settings = *current_settings ();
@@ -611,30 +656,30 @@ static void collapse_max_ptes_none(struct collapse_context *c) settings.khugepaged.max_ptes_none = max_ptes_none;push_settings (&settings); - p =alloc_mapping (1 ); + p = ops->setup_area (1 ); -fill_memory (p,0 , (hpage_pmd_nr - max_ptes_none -1 ) * page_size); + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_none -1 ) * page_size); c->collapse ("Maybe collapse with max_ptes_none exceeded" , p,1 , - !c->enforce_pte_scan_limits); + ops, !c->enforce_pte_scan_limits);validate_memory (p,0 , (hpage_pmd_nr - max_ptes_none -1 ) * page_size);if (c->enforce_pte_scan_limits) { -fill_memory (p,0 , (hpage_pmd_nr - max_ptes_none) * page_size); - c->collapse ("Collapse with max_ptes_none PTEs empty" , p,1 , + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_none) * page_size); + c->collapse ("Collapse with max_ptes_none PTEs empty" , p,1 , ops, true);validate_memory (p,0 , (hpage_pmd_nr - max_ptes_none) * page_size); } - -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size);pop_settings (); } -static void collapse_swapin_single_pte (struct collapse_context *c) +static void collapse_swapin_single_pte (struct collapse_context *c,struct mem_ops *ops) {void *p; - p =alloc_mapping (1 ); -fill_memory (p,0 , hpage_pmd_size); + + p = ops->setup_area (1 ); + ops->fault (p,0 , hpage_pmd_size);printf ("Swapout one page..." );if (madvise (p, page_size,MADV_PAGEOUT )) {
@@ -648,20 +693,21 @@ static void collapse_swapin_single_pte(struct collapse_context *c) goto out; } - c->collapse ("Collapse with swapping in single PTE entry" , p,1 , true); + c->collapse ("Collapse with swapping in single PTE entry" , p,1 , ops, + true);validate_memory (p,0 , hpage_pmd_size); out: -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_max_ptes_swap (struct collapse_context *c) +static void collapse_max_ptes_swap (struct collapse_context *c,struct mem_ops *ops) {int max_ptes_swap =read_num ("khugepaged/max_ptes_swap" );void *p; - p =alloc_mapping (1 ); + p = ops->setup_area (1 ); + ops->fault (p,0 , hpage_pmd_size); -fill_memory (p,0 , hpage_pmd_size);printf ("Swapout %d of %d pages..." , max_ptes_swap +1 , hpage_pmd_nr);if (madvise (p, (max_ptes_swap +1 ) * page_size,MADV_PAGEOUT )) {perror ("madvise(MADV_PAGEOUT)" );
@@ -674,12 +720,12 @@ static void collapse_max_ptes_swap(struct collapse_context *c) goto out; } - c->collapse ("Maybe collapse with max_ptes_swap exceeded" , p,1 , + c->collapse ("Maybe collapse with max_ptes_swap exceeded" , p,1 , ops, !c->enforce_pte_scan_limits);validate_memory (p,0 , hpage_pmd_size);if (c->enforce_pte_scan_limits) { -fill_memory (p,0 , hpage_pmd_size); + ops->fault (p,0 , hpage_pmd_size);printf ("Swapout %d of %d pages..." , max_ptes_swap, hpage_pmd_nr);if (madvise (p, max_ptes_swap * page_size,MADV_PAGEOUT )) {
@@ -694,63 +740,65 @@ static void collapse_max_ptes_swap(struct collapse_context *c) } c->collapse ("Collapse with max_ptes_swap pages swapped out" , p, -1 , true); +1 , ops, true);validate_memory (p,0 , hpage_pmd_size); } out: -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_single_pte_entry_compound (struct collapse_context *c) +static void collapse_single_pte_entry_compound (struct collapse_context *c,struct mem_ops *ops) {void *p; - p =alloc_hpage (); + p =alloc_hpage (ops); +madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE );printf ("Split huge page leaving single PTE mapping compound page..." );madvise (p + page_size, hpage_pmd_size - page_size,MADV_DONTNEED ); -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); c->collapse ("Collapse PTE table with single PTE mapping compound page" , - p,1 , true); + p,1 , ops, true);validate_memory (p,0 , page_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_full_of_compound (struct collapse_context *c) +static void collapse_full_of_compound (struct collapse_context *c,struct mem_ops *ops) {void *p; - p =alloc_hpage (); + p =alloc_hpage (ops);printf ("Split huge page leaving single PTE page table full of compound pages..." );madvise (p, page_size,MADV_NOHUGEPAGE );madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE ); -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); - c->collapse ("Collapse PTE table full of compound pages" , p,1 , true); + c->collapse ("Collapse PTE table full of compound pages" , p,1 , ops, + true);validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_compound_extreme (struct collapse_context *c) +static void collapse_compound_extreme (struct collapse_context *c,struct mem_ops *ops) {void *p;int i; - p =alloc_mapping (1 ); + p = ops->setup_area (1 );for (i =0 ; i < hpage_pmd_nr; i++) {printf ("\rConstruct PTE page table full of different PTE-mapped compound pages %3d/%d..." , i +1 , hpage_pmd_nr);madvise (BASE_ADDR , hpage_pmd_size,MADV_HUGEPAGE ); -fill_memory (BASE_ADDR ,0 , hpage_pmd_size); -if (!check_huge (BASE_ADDR ,1 )) { + ops->fault (BASE_ADDR ,0 , hpage_pmd_size); +if (!ops->check_huge (BASE_ADDR ,1 )) {printf ("Failed to allocate huge page\n" );exit (EXIT_FAILURE ); }
@@ -777,30 +825,30 @@ static void collapse_compound_extreme(struct collapse_context *c) } } -munmap (BASE_ADDR , hpage_pmd_size); -fill_memory (p,0 , hpage_pmd_size); -if (check_huge (p,0 )) + ops->cleanup_area (BASE_ADDR , hpage_pmd_size); + ops->fault (p,0 , hpage_pmd_size); +if (!ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" ); c->collapse ("Collapse PTE table full of different compound pages" , p,1 , - true); + ops, true);validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_fork (struct collapse_context *c) +static void collapse_fork (struct collapse_context *c,struct mem_ops *ops) {int wstatus;void *p; - p =alloc_mapping (1 ); + p = ops->setup_area (1 );printf ("Allocate small page..." ); -fill_memory (p,0 , page_size); -if (check_huge (p,0 )) + ops->fault (p,0 , page_size); +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" );
@@ -811,17 +859,17 @@ static void collapse_fork(struct collapse_context *c) skip_settings_restore = true; exit_status =0 ; -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); -fill_memory (p, page_size,2 * page_size); + ops->fault (p, page_size,2 * page_size); c->collapse ("Collapse PTE table with single page shared with parent process" , - p,1 , true); + p,1 , ops, true);validate_memory (p,0 , page_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size);exit (exit_status); }
@@ -829,27 +877,27 @@ static void collapse_fork(struct collapse_context *c) exit_status +=WEXITSTATUS (wstatus);printf ("Check if parent still has small page..." ); -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , page_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_fork_compound (struct collapse_context *c) +static void collapse_fork_compound (struct collapse_context *c,struct mem_ops *ops) {int wstatus;void *p; - p =alloc_hpage (); + p =alloc_hpage (ops);printf ("Share huge page over fork()..." );if (!fork ()) {/* Do not touch settings on child exit */ skip_settings_restore = true; exit_status =0 ; -if (check_huge (p,1 )) +if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );
@@ -857,20 +905,20 @@ static void collapse_fork_compound(struct collapse_context *c)printf ("Split huge page PMD in child process..." );madvise (p, page_size,MADV_NOHUGEPAGE );madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE ); -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); -fill_memory (p,0 , page_size); + ops->fault (p,0 , page_size);write_num ("khugepaged/max_ptes_shared" , hpage_pmd_nr -1 ); c->collapse ("Collapse PTE table full of compound pages in child" , - p,1 , true); + p,1 , ops, true);write_num ("khugepaged/max_ptes_shared" ,current_settings ()->khugepaged.max_ptes_shared);validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size);exit (exit_status); }
@@ -878,59 +926,59 @@ static void collapse_fork_compound(struct collapse_context *c) exit_status +=WEXITSTATUS (wstatus);printf ("Check if parent still has huge page..." ); -if (check_huge (p,1 )) +if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void collapse_max_ptes_shared (struct collapse_context *c) +static void collapse_max_ptes_shared (struct collapse_context *c,struct mem_ops *ops) {int max_ptes_shared =read_num ("khugepaged/max_ptes_shared" );int wstatus;void *p; - p =alloc_hpage (); + p =alloc_hpage (ops);printf ("Share huge page over fork()..." );if (!fork ()) {/* Do not touch settings on child exit */ skip_settings_restore = true; exit_status =0 ; -if (check_huge (p,1 )) +if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );printf ("Trigger CoW on page %d of %d..." , hpage_pmd_nr - max_ptes_shared -1 , hpage_pmd_nr); -fill_memory (p,0 , (hpage_pmd_nr - max_ptes_shared -1 ) * page_size); -if (check_huge (p,0 )) + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_shared -1 ) * page_size); +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); c->collapse ("Maybe collapse with max_ptes_shared exceeded" , p, -1 , !c->enforce_pte_scan_limits); +1 , ops, !c->enforce_pte_scan_limits);if (c->enforce_pte_scan_limits) {printf ("Trigger CoW on page %d of %d..." , hpage_pmd_nr - max_ptes_shared, hpage_pmd_nr); -fill_memory (p,0 , (hpage_pmd_nr - max_ptes_shared) * + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_shared) * page_size); -if (check_huge (p,0 )) +if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); c->collapse ("Collapse with max_ptes_shared PTEs shared" , - p,1 , true); + p,1 , ops, true); }validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size);exit (exit_status); }
@@ -938,42 +986,28 @@ static void collapse_max_ptes_shared(struct collapse_context *c) exit_status +=WEXITSTATUS (wstatus);printf ("Check if parent still has huge page..." ); -if (check_huge (p,1 )) +if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); } -static void madvise_collapse_existing_thps (void ) +static void madvise_collapse_existing_thps (struct collapse_context *c, +struct mem_ops *ops) {void *p; -int err; - p =alloc_mapping (1 ); -fill_memory (p,0 , hpage_pmd_size); + p = ops->setup_area (1 ); + ops->fault (p,0 , hpage_pmd_size); + c->collapse ("Collapse fully populated PTE table..." , p,1 , ops, true); +validate_memory (p,0 , hpage_pmd_size); -printf ("Collapse fully populated PTE table..." ); -/* -* Note that we don't set MADV_HUGEPAGE here, which -* also tests that VM_HUGEPAGE isn't required for -* MADV_COLLAPSE in "madvise" mode. -*/ - err =madvise (p, hpage_pmd_size,MADV_COLLAPSE ); -if (err ==0 &&check_huge (p,1 )) { -success ("OK" ); -printf ("Re-collapse PMD-mapped hugepage" ); - err =madvise (p, hpage_pmd_size,MADV_COLLAPSE ); -if (err ==0 &&check_huge (p,1 )) -success ("OK" ); -else -fail ("Fail" ); - }else { -fail ("Fail" ); - } +/* c->collapse() will find a hugepage and complain - call directly. */ +__madvise_collapse ("Re-collapse PMD-mapped hugepage" , p,1 , ops, true);validate_memory (p,0 , hpage_pmd_size); -munmap (p, hpage_pmd_size); + ops->cleanup_area (p, hpage_pmd_size); }int main (int argc,const char **argv)
@@ -1013,37 +1047,37 @@ int main(int argc, const char **argv) c.collapse = &khugepaged_collapse; c.enforce_pte_scan_limits = true; -collapse_full (&c); -collapse_empty (&c); -collapse_single_pte_entry (&c); -collapse_max_ptes_none (&c); -collapse_swapin_single_pte (&c); -collapse_max_ptes_swap (&c); -collapse_single_pte_entry_compound (&c); -collapse_full_of_compound (&c); -collapse_compound_extreme (&c); -collapse_fork (&c); -collapse_fork_compound (&c); -collapse_max_ptes_shared (&c); +collapse_full (&c, &anon_ops); +collapse_empty (&c, &anon_ops); +collapse_single_pte_entry (&c, &anon_ops); +collapse_max_ptes_none (&c, &anon_ops); +collapse_swapin_single_pte (&c, &anon_ops); +collapse_max_ptes_swap (&c, &anon_ops); +collapse_single_pte_entry_compound (&c, &anon_ops); +collapse_full_of_compound (&c, &anon_ops); +collapse_compound_extreme (&c, &anon_ops); +collapse_fork (&c, &anon_ops); +collapse_fork_compound (&c, &anon_ops); +collapse_max_ptes_shared (&c, &anon_ops); }if (!strcmp (tests,"madvise" ) || !strcmp (tests,"all" )) {printf ("\n*** Testing context: madvise ***\n" ); c.collapse = &madvise_collapse; c.enforce_pte_scan_limits = false; -collapse_full (&c); -collapse_empty (&c); -collapse_single_pte_entry (&c); -collapse_max_ptes_none (&c); -collapse_swapin_single_pte (&c); -collapse_max_ptes_swap (&c); -collapse_single_pte_entry_compound (&c); -collapse_full_of_compound (&c); -collapse_compound_extreme (&c); -collapse_fork (&c); -collapse_fork_compound (&c); -collapse_max_ptes_shared (&c); -madvise_collapse_existing_thps (); +collapse_full (&c, &anon_ops); +collapse_empty (&c, &anon_ops); +collapse_single_pte_entry (&c, &anon_ops); +collapse_max_ptes_none (&c, &anon_ops); +collapse_swapin_single_pte (&c, &anon_ops); +collapse_max_ptes_swap (&c, &anon_ops); +collapse_single_pte_entry_compound (&c, &anon_ops); +collapse_full_of_compound (&c, &anon_ops); +collapse_compound_extreme (&c, &anon_ops); +collapse_fork (&c, &anon_ops); +collapse_fork_compound (&c, &anon_ops); +collapse_max_ptes_shared (&c, &anon_ops); +madvise_collapse_existing_thps (&c, &anon_ops); }restore_settings (0 );
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 7/9 selftests/vm: add thp collapse file and tmpfs testing
In reply to Zach O'KeefeAdd memory operations for file-backed and tmpfs memory. Call existing tests with these new memory operations to test collapse functionality of khugepaged and MADV_COLLAPSE on file-backed and tmpfs memory. Not all tests are reusable; for example, collapse_swapin_single_pte() which checks swap usage.
Refactor test arguments. Usage is now:
Usage: ./khugepaged <test type> [dir]
<test type> : <context>:<mem_type>
<context> : [all|khugepaged|madvise]
<mem_type> : [all|anon|file] "file,all" mem_type requires [dir] argument
"file,all" mem_type requires kernel built with CONFIG_READ_ONLY_THP_FOR_FS=y
if [dir] is a (sub)directory of a tmpfs mount, tmpfs must be mounted with huge=madvise option for khugepaged tests to work
Refactor calling tests to make it clear what collapse context / memory operations they support, but only invoke tests requested by user. Also log what test is being ran, and with what context / memory, to make test logs more human readable.
A new test file is created and deleted for every test to ensure no pages remain in the page cache between tests. Also, disable read_ahead_kb so that pages don't find their way into the page cache without the tests faulting them in.
Add file and shmem wrappers to vm_utils check for file and shmem hugepages in smaps.
--- tools/testing/selftests/vm/khugepaged.c | 401 ++++++++++++++++++++---- tools/testing/selftests/vm/vm_util.c | 10 + tools/testing/selftests/vm/vm_util.h | 2 + 3 files changed, 357 insertions(+), 56 deletions(-)
tools/testing/selftests/vm/khugepaged.c +345 -56
@@ -1,6 +1,7 @@#define _GNU_SOURCE#include <fcntl.h> #include <limits.h> +#include <dirent.h> #include <signal.h> #include <stdio.h> #include <stdlib.h>
@@ -10,12 +11,20 @@#include <sys/mman.h> #include <sys/wait.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/vfs.h> + +#include "linux/magic.h" #include "vm_util.h" #ifndef MADV_PAGEOUT #define MADV_PAGEOUT 21#endif +#ifndef MADV_POPULATE_READ +#define MADV_POPULATE_READ 22 +#endif #ifndef MADV_COLLAPSE #define MADV_COLLAPSE 25#endif
@@ -26,21 +35,48 @@ static unsigned long page_size;static int hpage_pmd_nr;#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/" +#define READ_AHEAD_SYSFS "/sys/block/sda/queue/read_ahead_kb"#define PID_SMAPS "/proc/self/smaps" +#define TEST_FILE "collapse_test_file" + +#define MAX_LINE_LENGTH 500 + +enum vma_type { +VMA_ANON , +VMA_FILE , +VMA_SHMEM , +};struct mem_ops {void *(*setup_area)(int nr_hpages);void (*cleanup_area)(void *p,unsigned long size);void (*fault)(void *p,unsigned long start,unsigned long end);bool (*check_huge)(void *addr,int nr_hpages); +const char *name; }; +static struct mem_ops *file_ops; +static struct mem_ops *anon_ops; +struct collapse_context {void (*collapse)(const char *msg,char *p,int nr_hpages,struct mem_ops *ops,bool expect);bool enforce_pte_scan_limits; +const char *name; +}; + +static struct collapse_context *khugepaged_context; +static struct collapse_context *madvise_context; + +struct file_info { +const char *dir; +char path[MAX_LINE_LENGTH ]; +enum vma_type type; +int fd; }; +static struct file_info finfo; +enum thp_enabled {THP_ALWAYS ,THP_MADVISE ,
@@ -106,6 +142,7 @@ struct settings {enum shmem_enabled shmem_enabled;bool use_zero_page;struct khugepaged_settings khugepaged; +unsigned long read_ahead_kb; };static struct settings saved_settings;
@@ -124,6 +161,11 @@ static void fail(const char *msg) exit_status++; } +static void skip (const char *msg) +{ +printf (" \e[33m%s\e[0m\n" , msg); +} +static int read_file (const char *path,char *buf,size_t buflen) {int fd;
@@ -151,13 +193,19 @@ static int write_file(const char *path, const char *buf, size_t buflen)ssize_t numwritten; fd =open (path,O_WRONLY ); -if (fd ==-1 ) +if (fd ==-1 ) { +printf ("open(%s)\n" , path); +exit (EXIT_FAILURE );return 0 ; + } numwritten =write (fd, buf, buflen -1 );close (fd); -if (numwritten <1 ) +if (numwritten <1 ) { +printf ("write(%s)\n" , buf); +exit (EXIT_FAILURE );return 0 ; + }return (unsigned int ) numwritten; }
@@ -224,20 +272,11 @@ static void write_string(const char *name, const char *val) } } -static const unsigned long read_num (const char *name) +static const unsigned long _read_num (const char *path) { -char path[PATH_MAX ];char buf[21 ]; -int ret; - - ret =snprintf (path,PATH_MAX ,THP_SYSFS "%s" , name); -if (ret >=PATH_MAX ) { -printf ("%s: Pathname is too long\n" , __func__); -exit (EXIT_FAILURE ); - } - ret =read_file (path, buf,sizeof (buf)); -if (ret <0 ) { +if (read_file (path, buf,sizeof (buf)) <0 ) {perror ("read_file(read_num)" );exit (EXIT_FAILURE ); }
@@ -245,10 +284,9 @@ static const unsigned long read_num(const char *name)return strtoul (buf,NULL ,10 ); } -static void write_num (const char *name,unsigned long num) +static const unsigned long read_num (const char *name) {char path[PATH_MAX ]; -char buf[21 ];int ret; ret =snprintf (path,PATH_MAX ,THP_SYSFS "%s" , name);
@@ -256,6 +294,12 @@ static void write_num(const char *name, unsigned long num)printf ("%s: Pathname is too long\n" , __func__);exit (EXIT_FAILURE ); } +return _read_num (path); +} + +static void _write_num (const char *path,unsigned long num) +{ +char buf[21 ];sprintf (buf,"%ld" , num);if (!write_file (path, buf,strlen (buf) +1 )) {
@@ -264,6 +308,19 @@ static void write_num(const char *name, unsigned long num) } } +static void write_num (const char *name,unsigned long num) +{ +char path[PATH_MAX ]; +int ret; + + ret =snprintf (path,PATH_MAX ,THP_SYSFS "%s" , name); +if (ret >=PATH_MAX ) { +printf ("%s: Pathname is too long\n" , __func__); +exit (EXIT_FAILURE ); + } +_write_num (path, num); +} +static void write_settings (struct settings *settings) {struct khugepaged_settings *khugepaged = &settings->khugepaged;
@@ -283,6 +340,8 @@ static void write_settings(struct settings *settings)write_num ("khugepaged/max_ptes_swap" , khugepaged->max_ptes_swap);write_num ("khugepaged/max_ptes_shared" , khugepaged->max_ptes_shared);write_num ("khugepaged/pages_to_scan" , khugepaged->pages_to_scan); + +_write_num (READ_AHEAD_SYSFS , settings->read_ahead_kb); }#define MAX_SETTINGS_DEPTH 4
@@ -341,6 +400,7 @@ static void save_settings(void) .shmem_enabled = read_string("shmem_enabled", shmem_enabled_strings), .use_zero_page = read_num("use_zero_page"), + .read_ahead_kb = _read_num(READ_AHEAD_SYSFS), }; saved_settings.khugepaged = (struct khugepaged_settings ) { .defrag =read_num ("khugepaged/defrag" ),
@@ -361,7 +421,6 @@ static void save_settings(void)signal (SIGQUIT , restore_settings); } -#define MAX_LINE_LENGTH 500static bool check_swap (void *addr,unsigned long size) {bool swap = false;
@@ -489,11 +548,109 @@ static bool anon_check_huge(void *addr, int nr_hpages)return check_huge_anon (addr, nr_hpages, hpage_pmd_size); } -static struct mem_ops anon_ops = { +static void *file_setup_area (int nr_hpages) +{ +struct stat path_stat; +struct statfs fs; +int fd; +void *p; +unsigned long size; + +stat (finfo.dir, &path_stat); +if (!S_ISDIR (path_stat.st_mode)) { +printf ("%s: Not a directory (%s)\n" , __func__, finfo.dir); +exit (EXIT_FAILURE ); + } +if (snprintf (finfo.path,sizeof (finfo.path),"%s/" TEST_FILE , + finfo.dir) >=sizeof (finfo.path)) { +printf ("%s: Pathname is too long\n" , __func__); +exit (EXIT_FAILURE ); + } +if (statfs (finfo.dir, &fs)) { +perror ("statfs()" ); +exit (EXIT_FAILURE ); + } + finfo.type = fs.f_type ==TMPFS_MAGIC ?VMA_SHMEM :VMA_FILE ; + +unlink (finfo.path);/* Cleanup from previous failed tests */ +printf ("Creating %s for collapse%s..." , finfo.path, + finfo.type ==VMA_SHMEM ?" (tmpfs)" :"" ); + fd =open (finfo.path,O_DSYNC |O_CREAT |O_RDWR |O_TRUNC |O_EXCL , +777 ); +if (fd <0 ) { +perror ("open()" ); +exit (EXIT_FAILURE ); + } + + size = nr_hpages * hpage_pmd_size; + p =alloc_mapping (nr_hpages); +fill_memory (p,0 , size); +write (fd, p, size); +close (fd); +munmap (p, size); +success ("OK" ); + +printf ("Opening %s read only for collapse..." , finfo.path); + finfo.fd =open (finfo.path,O_RDONLY ,777 ); +if (finfo.fd <0 ) { +perror ("open()" ); +exit (EXIT_FAILURE ); + } + p =mmap (BASE_ADDR , size,PROT_READ |PROT_EXEC , +MAP_PRIVATE , finfo.fd,0 ); +if (p ==MAP_FAILED || p !=BASE_ADDR ) { +perror ("mmap()" ); +exit (EXIT_FAILURE ); + } + +/* Drop page cache */ +write_file ("/proc/sys/vm/drop_caches" ,"3" ,2 ); +success ("OK" ); +return p; +} + +static void file_cleanup_area (void *p,unsigned long size) +{ +munmap (p, size); +close (finfo.fd); +unlink (finfo.path); +} + +static void file_fault (void *p,unsigned long start,unsigned long end) +{ +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_READ )) { +perror ("madvise(MADV_POPULATE_READ" ); +exit (EXIT_FAILURE ); + } +} + +static bool file_check_huge (void *addr,int nr_hpages) +{ +switch (finfo.type) { +case VMA_FILE : +return check_huge_file (addr, nr_hpages, hpage_pmd_size); +case VMA_SHMEM : +return check_huge_shmem (addr, nr_hpages, hpage_pmd_size); +default : +exit (EXIT_FAILURE ); +return false; + } +} + +static struct mem_ops __anon_ops = { .setup_area = &anon_setup_area, .cleanup_area = &anon_cleanup_area, .fault = &anon_fault, .check_huge = &anon_check_huge, + .name ="anon" , +}; + +static struct mem_ops __file_ops = { + .setup_area = &file_setup_area, + .cleanup_area = &file_cleanup_area, + .fault = &file_fault, + .check_huge = &file_check_huge, + .name ="file" , }; static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
@@ -509,6 +666,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,* ignores /sys/kernel/mm/transparent_hugepage/enabled */ settings.thp_enabled =THP_NEVER ; + settings.shmem_enabled =SHMEM_NEVER ;push_settings (&settings);/* Clear VM_NOHUGEPAGE */
@@ -580,12 +738,37 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages,return ; } +/* +* For file and shmem memory, khugepaged only retracts pte entries after +* putting the new hugepage in the page cache. The hugepage must be +* subsequently refaulted to install the pmd mapping for the mm. +*/ +if (ops != &__anon_ops) + ops->fault (p,0 , nr_hpages * hpage_pmd_size); +if (ops->check_huge (p, expect ? nr_hpages :0 ))success ("OK" );else fail ("Fail" ); } +static struct collapse_context __khugepaged_context = { + .collapse = &khugepaged_collapse, + .enforce_pte_scan_limits = true, + .name ="khugepaged" , +}; + +static struct collapse_context __madvise_context = { + .collapse = &madvise_collapse, + .enforce_pte_scan_limits = false, + .name ="madvise" , +}; + +static bool is_tmpfs (struct mem_ops *ops) +{ +return ops == &__file_ops && finfo.type ==VMA_SHMEM ; +} +static void alloc_at_fault (void ) {struct settings settings = *current_settings ();
@@ -658,6 +841,13 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o p = ops->setup_area (1 ); +if (is_tmpfs (ops)) { +/* shmem pages always in the page cache */ +printf ("tmpfs..." ); +skip ("Skip" ); + goto skip; + } + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_none -1 ) * page_size); c->collapse ("Maybe collapse with max_ptes_none exceeded" , p,1 , ops, !c->enforce_pte_scan_limits);
@@ -670,6 +860,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *ovalidate_memory (p,0 , (hpage_pmd_nr - max_ptes_none) * page_size); } +skip: ops->cleanup_area (p, hpage_pmd_size);pop_settings (); }
@@ -753,6 +944,13 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, struc p =alloc_hpage (ops); +if (is_tmpfs (ops)) { +/* MADV_DONTNEED won't evict tmpfs pages */ +printf ("tmpfs..." ); +skip ("Skip" ); + goto skip; + } +madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE );printf ("Split huge page leaving single PTE mapping compound page..." );madvise (p + page_size, hpage_pmd_size - page_size,MADV_DONTNEED );
@@ -764,6 +962,7 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, struc c->collapse ("Collapse PTE table with single PTE mapping compound page" , p,1 , ops, true);validate_memory (p,0 , page_size); +skip: ops->cleanup_area (p, hpage_pmd_size); }
@@ -1010,9 +1209,72 @@ static void madvise_collapse_existing_thps(struct collapse_context *c, ops->cleanup_area (p, hpage_pmd_size); } +static void usage (void ) +{ +fprintf (stderr,"\nUsage: ./khugepaged <test type> [dir]\n\n" ); +fprintf (stderr,"\t<test type>\t: <context>:<mem_type>\n" ); +fprintf (stderr,"\t<context>\t: [all|khugepaged|madvise]\n" ); +fprintf (stderr,"\t<mem_type>\t: [all|anon|file]\n" ); +fprintf (stderr,"\n\t\"file,all\" mem_type requires [dir] argument\n" ); +fprintf (stderr,"\n\t\"file,all\" mem_type requires kernel built with\n" ); +fprintf (stderr,"\tCONFIG_READ_ONLY_THP_FOR_FS=y\n" ); +fprintf (stderr,"\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n" ); +fprintf (stderr,"\tmounted with huge=madvise option for khugepaged tests to work\n" ); +exit (1 ); +} + +static void parse_test_type (int argc,const char **argv) +{ +char *buf; +const char *token; + +if (argc ==1 ) { +/* Backwards compatibility */ + khugepaged_context = &__khugepaged_context; + madvise_context = &__madvise_context; + anon_ops = &__anon_ops; +return ; + } + + buf =strdup (argv[1 ]); + token =strsep (&buf,":" ); + +if (!strcmp (token,"all" )) { + khugepaged_context = &__khugepaged_context; + madvise_context = &__madvise_context; + }else if (!strcmp (token,"khugepaged" )) { + khugepaged_context = &__khugepaged_context; + }else if (!strcmp (token,"madvise" )) { + madvise_context = &__madvise_context; + }else { +usage (); + } + +if (!buf) +usage (); + +if (!strcmp (buf,"all" )) { + file_ops = &__file_ops; + anon_ops = &__anon_ops; + }else if (!strcmp (buf,"anon" )) { + anon_ops = &__anon_ops; + }else if (!strcmp (buf,"file" )) { + file_ops = &__file_ops; + }else { +usage (); + } + +if (!file_ops) +return ; + +if (argc !=3 ) +usage (); + + finfo.dir = argv[2 ]; +} +int main (int argc,const char **argv) { -struct collapse_context c;struct settings default_settings = { .thp_enabled =THP_MADVISE , .thp_defrag =THP_DEFRAG_ALWAYS ,
@@ -1023,8 +1285,17 @@ int main(int argc, const char **argv) .alloc_sleep_millisecs = 10, .scan_sleep_millisecs = 10, }, +/* +* When testing file-backed memory, the collapse path +* looks at how many pages are found in the page cache, not +* what pages are mapped. Disable read ahead optimization so +* pages don't find their way into the page cache unless +* we mem_ops->fault() them in. +*/ + .read_ahead_kb = 0, }; -const char *tests = argc ==1 ?"all" : argv[1 ]; + +parse_test_type (argc, argv);setbuf (stdout,NULL );
@@ -1042,43 +1313,61 @@ int main(int argc, const char **argv)alloc_at_fault (); -if (!strcmp (tests,"khugepaged" ) || !strcmp (tests,"all" )) { -printf ("\n*** Testing context: khugepaged ***\n" ); - c.collapse = &khugepaged_collapse; - c.enforce_pte_scan_limits = true; - -collapse_full (&c, &anon_ops); -collapse_empty (&c, &anon_ops); -collapse_single_pte_entry (&c, &anon_ops); -collapse_max_ptes_none (&c, &anon_ops); -collapse_swapin_single_pte (&c, &anon_ops); -collapse_max_ptes_swap (&c, &anon_ops); -collapse_single_pte_entry_compound (&c, &anon_ops); -collapse_full_of_compound (&c, &anon_ops); -collapse_compound_extreme (&c, &anon_ops); -collapse_fork (&c, &anon_ops); -collapse_fork_compound (&c, &anon_ops); -collapse_max_ptes_shared (&c, &anon_ops); - } -if (!strcmp (tests,"madvise" ) || !strcmp (tests,"all" )) { -printf ("\n*** Testing context: madvise ***\n" ); - c.collapse = &madvise_collapse; - c.enforce_pte_scan_limits = false; - -collapse_full (&c, &anon_ops); -collapse_empty (&c, &anon_ops); -collapse_single_pte_entry (&c, &anon_ops); -collapse_max_ptes_none (&c, &anon_ops); -collapse_swapin_single_pte (&c, &anon_ops); -collapse_max_ptes_swap (&c, &anon_ops); -collapse_single_pte_entry_compound (&c, &anon_ops); -collapse_full_of_compound (&c, &anon_ops); -collapse_compound_extreme (&c, &anon_ops); -collapse_fork (&c, &anon_ops); -collapse_fork_compound (&c, &anon_ops); -collapse_max_ptes_shared (&c, &anon_ops); -madvise_collapse_existing_thps (&c, &anon_ops); - } +#define TEST (t, c, o) do { \ + if (c && o) { \ + printf("\nRun test: " #t " (%s:%s)\n", c->name, o->name); \ + t(c, o); \ + } \ + } while (0) + +TEST (collapse_full, khugepaged_context, anon_ops); +TEST (collapse_full, khugepaged_context, file_ops); +TEST (collapse_full, madvise_context, anon_ops); +TEST (collapse_full, madvise_context, file_ops); + +TEST (collapse_empty, khugepaged_context, anon_ops); +TEST (collapse_empty, madvise_context, anon_ops); + +TEST (collapse_single_pte_entry, khugepaged_context, anon_ops); +TEST (collapse_single_pte_entry, khugepaged_context, file_ops); +TEST (collapse_single_pte_entry, madvise_context, anon_ops); +TEST (collapse_single_pte_entry, madvise_context, file_ops); + +TEST (collapse_max_ptes_none, khugepaged_context, anon_ops); +TEST (collapse_max_ptes_none, khugepaged_context, file_ops); +TEST (collapse_max_ptes_none, madvise_context, anon_ops); +TEST (collapse_max_ptes_none, madvise_context, file_ops); + +TEST (collapse_single_pte_entry_compound, khugepaged_context, anon_ops); +TEST (collapse_single_pte_entry_compound, khugepaged_context, file_ops); +TEST (collapse_single_pte_entry_compound, madvise_context, anon_ops); +TEST (collapse_single_pte_entry_compound, madvise_context, file_ops); + +TEST (collapse_full_of_compound, khugepaged_context, anon_ops); +TEST (collapse_full_of_compound, khugepaged_context, file_ops); +TEST (collapse_full_of_compound, madvise_context, anon_ops); +TEST (collapse_full_of_compound, madvise_context, file_ops); + +TEST (collapse_compound_extreme, khugepaged_context, anon_ops); +TEST (collapse_compound_extreme, madvise_context, anon_ops); + +TEST (collapse_swapin_single_pte, khugepaged_context, anon_ops); +TEST (collapse_swapin_single_pte, madvise_context, anon_ops); + +TEST (collapse_max_ptes_swap, khugepaged_context, anon_ops); +TEST (collapse_max_ptes_swap, madvise_context, anon_ops); + +TEST (collapse_fork, khugepaged_context, anon_ops); +TEST (collapse_fork, madvise_context, anon_ops); + +TEST (collapse_fork_compound, khugepaged_context, anon_ops); +TEST (collapse_fork_compound, madvise_context, anon_ops); + +TEST (collapse_max_ptes_shared, khugepaged_context, anon_ops); +TEST (collapse_max_ptes_shared, madvise_context, anon_ops); + +TEST (madvise_collapse_existing_thps, madvise_context, anon_ops); +TEST (madvise_collapse_existing_thps, madvise_context, file_ops);restore_settings (0 ); }
tools/testing/selftests/vm/vm_util.c +10 -0
@@ -114,3 +114,13 @@ bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size) {return __check_huge (addr,"AnonHugePages: " , nr_hpages, hpage_size); } + +bool check_huge_file (void *addr,int nr_hpages,uint64_t hpage_size) +{ +return __check_huge (addr,"FilePmdMapped:" , nr_hpages, hpage_size); +} + +bool check_huge_shmem (void *addr,int nr_hpages,uint64_t hpage_size) +{ +return __check_huge (addr,"ShmemPmdMapped:" , nr_hpages, hpage_size); +}
tools/testing/selftests/vm/vm_util.h +2 -0
@@ -8,3 +8,5 @@ void clear_softdirty(void);bool check_for_pattern (FILE *fp,const char *pattern,char *buf,size_t len);uint64_t read_pmd_pagesize (void );bool check_huge_anon (void *addr,int nr_hpages,uint64_t hpage_size); +bool check_huge_file (void *addr,int nr_hpages,uint64_t hpage_size); +bool check_huge_shmem (void *addr,int nr_hpages,uint64_t hpage_size);
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 8/9 selftests/vm: add thp collapse shmem testing
In reply to Zach O'KeefeAdd memory operations for shmem (memfd) memory, and reuse existing tests with the new memory operations.
Shmem tests can be called with "shmem" mem_type, and shmem tests are ran with "all" mem_type as well.
--- tools/testing/selftests/vm/khugepaged.c | 57 ++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 2 deletions(-)
tools/testing/selftests/vm/khugepaged.c +55 -2
@@ -57,6 +57,7 @@ struct mem_ops {static struct mem_ops *file_ops;static struct mem_ops *anon_ops; +static struct mem_ops *shmem_ops;struct collapse_context { void (*collapse)(const char *msg, char *p, int nr_hpages,
@@ -637,6 +638,40 @@ static bool file_check_huge(void *addr, int nr_hpages) } } +static void *shmem_setup_area (int nr_hpages) +{ +void *p; +unsigned long size = nr_hpages * hpage_pmd_size; + + finfo.fd =memfd_create ("khugepaged-selftest-collapse-shmem" ,0 ); +if (finfo.fd <0 ) { +perror ("memfd_create()" ); +exit (EXIT_FAILURE ); + } +if (ftruncate (finfo.fd, size)) { +perror ("ftruncate()" ); +exit (EXIT_FAILURE ); + } + p =mmap (BASE_ADDR , size,PROT_READ |PROT_WRITE ,MAP_SHARED , finfo.fd, +0 ); +if (p !=BASE_ADDR ) { +perror ("mmap()" ); +exit (EXIT_FAILURE ); + } +return p; +} + +static void shmem_cleanup_area (void *p,unsigned long size) +{ +munmap (p, size); +close (finfo.fd); +} + +static bool shmem_check_huge (void *addr,int nr_hpages) +{ +return check_huge_shmem (addr, nr_hpages, hpage_pmd_size); +} +static struct mem_ops __anon_ops = { .setup_area = &anon_setup_area, .cleanup_area = &anon_cleanup_area,
@@ -653,6 +688,14 @@ static struct mem_ops __file_ops = { .name = "file", }; +static struct mem_ops __shmem_ops = { + .setup_area = &shmem_setup_area, + .cleanup_area = &shmem_cleanup_area, + .fault = &anon_fault, + .check_huge = &shmem_check_huge, + .name ="shmem" , +}; +static void __madvise_collapse (const char *msg,char *p,int nr_hpages,struct mem_ops *ops,bool expect) {
@@ -1214,7 +1257,7 @@ static void usage(void)fprintf (stderr,"\nUsage: ./khugepaged <test type> [dir]\n\n" );fprintf (stderr,"\t<test type>\t: <context>:<mem_type>\n" );fprintf (stderr,"\t<context>\t: [all|khugepaged|madvise]\n" ); -fprintf (stderr,"\t<mem_type>\t: [all|anon|file]\n" ); +fprintf (stderr,"\t<mem_type>\t: [all|anon|file|shmem]\n" );fprintf (stderr,"\n\t\"file,all\" mem_type requires [dir] argument\n" );fprintf (stderr,"\n\t\"file,all\" mem_type requires kernel built with\n" );fprintf (stderr,"\tCONFIG_READ_ONLY_THP_FOR_FS=y\n" );
@@ -1256,10 +1299,13 @@ static void parse_test_type(int argc, const char **argv)if (!strcmp (buf,"all" )) { file_ops = &__file_ops; anon_ops = &__anon_ops; + shmem_ops = &__shmem_ops; }else if (!strcmp (buf,"anon" )) { anon_ops = &__anon_ops; }else if (!strcmp (buf,"file" )) { file_ops = &__file_ops; + }else if (!strcmp (buf,"shmem" )) { + shmem_ops = &__shmem_ops; }else {usage (); }
@@ -1278,7 +1324,7 @@ int main(int argc, const char **argv)struct settings default_settings = { .thp_enabled =THP_MADVISE , .thp_defrag =THP_DEFRAG_ALWAYS , - .shmem_enabled =SHMEM_NEVER , + .shmem_enabled =SHMEM_ADVISE , .use_zero_page =0 , .khugepaged = { .defrag =1 ,
@@ -1322,16 +1368,20 @@ int main(int argc, const char **argv)TEST (collapse_full, khugepaged_context, anon_ops);TEST (collapse_full, khugepaged_context, file_ops); +TEST (collapse_full, khugepaged_context, shmem_ops);TEST (collapse_full, madvise_context, anon_ops);TEST (collapse_full, madvise_context, file_ops); +TEST (collapse_full, madvise_context, shmem_ops);TEST (collapse_empty, khugepaged_context, anon_ops);TEST (collapse_empty, madvise_context, anon_ops);TEST (collapse_single_pte_entry, khugepaged_context, anon_ops);TEST (collapse_single_pte_entry, khugepaged_context, file_ops); +TEST (collapse_single_pte_entry, khugepaged_context, shmem_ops);TEST (collapse_single_pte_entry, madvise_context, anon_ops);TEST (collapse_single_pte_entry, madvise_context, file_ops); +TEST (collapse_single_pte_entry, madvise_context, shmem_ops);TEST (collapse_max_ptes_none, khugepaged_context, anon_ops);TEST (collapse_max_ptes_none, khugepaged_context, file_ops);
@@ -1345,8 +1395,10 @@ int main(int argc, const char **argv)TEST (collapse_full_of_compound, khugepaged_context, anon_ops);TEST (collapse_full_of_compound, khugepaged_context, file_ops); +TEST (collapse_full_of_compound, khugepaged_context, shmem_ops);TEST (collapse_full_of_compound, madvise_context, anon_ops);TEST (collapse_full_of_compound, madvise_context, file_ops); +TEST (collapse_full_of_compound, madvise_context, shmem_ops);TEST (collapse_compound_extreme, khugepaged_context, anon_ops);TEST (collapse_compound_extreme, madvise_context, anon_ops);
@@ -1368,6 +1420,7 @@ int main(int argc, const char **argv)TEST (madvise_collapse_existing_thps, madvise_context, anon_ops);TEST (madvise_collapse_existing_thps, madvise_context, file_ops); +TEST (madvise_collapse_existing_thps, madvise_context, shmem_ops);restore_settings (0 ); }
Signature · 1 line
2.37.2.672.g94769d06f0-goog
PATCH v2 9/9 selftests/vm: add selftest for MADV_COLLAPSE of uffd-minor memory
In reply to Zach O'KeefeAdd :collapse mod to userfaultfd selftest. Currently this mod is only valid for "shmem" test type, but could be used for other test types.
When provided, memory allocated by ->allocate_area() will be hugepage-aligned enforced to be hugepage-sized. userfaultf_minor_test, after the UFFD-registered mapping has been populated by UUFD minor fault handler, attempt to MADV_COLLAPSE the UFFD-registered mapping to collapse the memory into a pmd-mapped THP.
This test is meant to be a functional test of what occurs during UFFD-driven live migration of VMs backed by huge tmpfs where, after a hugepage-sized region has been successfully migrated (in native page-sized chunks, to avoid latency of fetched a hugepage over the network), we want to reclaim previous VM performance by remapping it at the PMD level.
--- tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/userfaultfd.c | 171 ++++++++++++++++++----- 2 files changed, 134 insertions(+), 38 deletions(-)
tools/testing/selftests/vm/Makefile +1 -0
@@ -101,6 +101,7 @@ $(OUTPUT)/khugepaged: vm_util.c $(OUTPUT)/madv_populate: vm_util.c $(OUTPUT)/soft-dirty: vm_util.c $(OUTPUT)/split_huge_page_test: vm_util.c +$(OUTPUT)/userfaultfd: vm_util.c ifeq ($(MACHINE),x86_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32))
tools/testing/selftests/vm/userfaultfd.c +133 -38
353 lines not on this page · raw message
Signature · 1 line
2.37.2.672.g94769d06f0-goog
Hi Zach,
I did a quick look at the series, basically no show stopper to me. But I didn't find time to review them thoroughly yet, quite busy on something else. Just a heads up, I didn't mean to ignore you. I will review them when I find some time.
Thanks,
Yang
Quoted 102 lines · Zach O'Keefe
Subject: [PATCH mm-unstable v2 0/9] mm: add file/shmem support to
MADV_COLLAPSEv2 Forward
Mostly a RESEND: rebase on latest mm-unstable + minor bug fixes from kernel test robot.
--------------------------------This series builds on top of the previous "mm: userspace hugepage collapse" series which introduced the
MADV_COLLAPSEmadvise mode and added support for private, anonymous mappings[1], by adding support for file and shmem backed memory toCONFIG_READ_ONLY_THP_FOR_FS=ykernels.File and shmem support have been added with effort to align with existing
MADV_COLLAPSEsemantics and policy decisions[2]. Collapse of shmem-backed memory ignores kernel-guiding directives and heuristics including all sysfs settings (transparent_hugepage/shmem_enabled), and tmpfs huge= mount options (shmem always supports large folios). Like anonymous mappings, on successful return ofMADV_COLLAPSEon file/shmem memory, the contents of memory mapped by the addresses provided will be synchronously pmd-mapped THPs.This functionality unlocks two important uses:
(1) Immediately back executable text by THPs. Current support provided
byCONFIG_READ_ONLY_THP_FOR_FSmay take a long time on a large system which might impair services from serving at their full rated load after (re)starting. Tricks like mremap(2)'ing text onto anonymous memory to immediately realize iTLB performance prevents page sharing and demand paging, both of which increase steady state memory footprint. Now, we can have the best of both worlds: Peak upfront performance and lower RAM footprints.(2) userfaultfd-based live migration of virtual machines satisfy UFFD
faults by fetching native-sized pages over the network (to avoid latency of transferring an entire hugepage). However, after guest memory has been fully copied to the new host,MADV_COLLAPSEcan be used to immediately increase guest performance.khugepaged has received a small improvement by association and can now detect and collapse pte-mapped THPs. However, there is still work to be done along the file collapse path. Compound pages of arbitrary order still needs to be supported and THP collapse needs to be converted to using folios in general. Eventually, we'd like to move away from the read-only and executable-mapped constraints currently imposed on eligible files and support any inode claiming huge folio support. That said, I think the series as-is covers enough to claim that
MADV_COLLAPSEsupports file/shmem memory.Patches 1-3 Implement the guts of the series. Patch 4 Is a tracepoint for debugging. Patches 5-8 Refactor existing khugepaged selftests to work with newmemory types.
Patch 9 Adds a userfaultfd selftest mode to mimic a functional test
of UFFDIO_REGISTER_MODE_MINOR+MADV_COLLAPSE live migration.Applies against mm-unstable.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/v1 -> v2:
- Add missing definition forkhugepaged_add_pte_mapped_thp()in !CONFIG_SHEM builds, in "mm/khugepaged: attempt to map file/shmem-backed pte-mapped THPs by pmds"
- Minor bugfixes in "mm/madvise: add file and shmem support toMADV_COLLAPSE" for !CONFIG_SHMEM, !CONFIG_TRANSPARENT_HUGEPAGE and some compiler settings.
- Rebased on latest mm-unstableZach O'Keefe (9):
mm/shmem: add flag to enforce shmem THP inhugepage_vma_check()mm/khugepaged: attempt to map file/shmem-backed pte-mapped THPs by
pmds
mm/madvise: add file and shmem support toMADV_COLLAPSEmm/khugepaged: add tracepoint tohpage_collapse_scan_file()selftests/vm: dedup THP helpers
selftests/vm: modularize thp collapse memory operations selftests/vm: add thp collapse file and tmpfs testing selftests/vm: add thp collapse shmem testing selftests/vm: add selftest forMADV_COLLAPSEof uffd-minor memoryinclude/linux/khugepaged.h | 13 +- include/linux/shmem_fs.h | 10 +- include/trace/events/huge_memory.h | 36 + kernel/events/uprobes.c | 2 +- mm/huge_memory.c | 2 +- mm/khugepaged.c | 289 ++++-- mm/shmem.c | 18 +- tools/testing/selftests/vm/Makefile | 2 + tools/testing/selftests/vm/khugepaged.c | 828 ++++++++++++------ tools/testing/selftests/vm/soft-dirty.c | 2 +- .../selftests/vm/split_huge_page_test.c | 12 +- tools/testing/selftests/vm/userfaultfd.c | 171 +++- tools/testing/selftests/vm/vm_util.c | 36 +- tools/testing/selftests/vm/vm_util.h | 5 +- 14 files changed, 1040 insertions(+), 386 deletions(-)--
2.37.2.672.g94769d06f0-goog
Quoted 10 lines · Yang Shi
Hi Zach,
I did a quick look at the series, basically no show stopper to me. But I didn't find time to review them thoroughly yet, quite busy on something else. Just a heads up, I didn't mean to ignore you. I will review them when I find some time.
Thanks,
Yang
Hey Yang,
Thanks for taking the time to look through, and thanks for giving me a heads up, and no rush!
In the last day or so, while porting this series around, I encountered some subtle edge cases I wanted to clean up / address - so it's good you didn't do a thorough review yet. I was *hoping* to have a v3 out last night (which evidently did not happen) and it does not seem like it will happen today, so I'll leave this message as a request for reviewers to hold off on a thorough review until v3.
Thanks for your time as always,
Zach