Remove CONFIG_READ_ONLY_THP_FOR_FS and enable file THP for writable files
Subsystems (9)
Zi Yan
reviewerv6 cover Remove CONFIG_READ_ONLY_THP_FOR_FS and enable file THP for writable files
Hi all,
This patchset removes READ_ONLY_THP_FOR_FS Kconfig and enables creating file-backed THPs for FSes with large folio support (the supported orders need to include PMD_ORDER) by default, including for writable files. It is an in-place replacement of V5 in mm-new. It affects Mike Rapoport's "make MM selftests more CI friendly", since "selftests/mm: khugepaged:
use kselftest framework" needs to be updated. I updated it and put it at the end of this cover letter.
Before the patchset, the status of creating read-only THPs is below:
| PF | MADV_COLLAPSE | khugepaged |
|-----------|---------------|------------|
large folio FSes only | ✓ | x | x |
READ_ONLY_THP_FOR_FS only | x | ✓ | ✓ |
both | ✓ | ✓ | ✓ |where READ_ONLY_THP_FOR_FS implies no large folio FSes.
Now without READ_ONLY_THP_FOR_FS:
| PF | MADV_COLLAPSE | khugepaged |
|-----------|---------------|------------|
large folio FSes (read-only fd) | ✓ | ✓ | ✓ |
large folio FSes (read-write fd) | ✓ | ✓ | ✓* |
no large folio FSes | x | x | x |* khugepaged only collapses clean folios from writable files. Userspace must flush dirty folios explicitly before khugepaged can collapse them. MADV_COLLAPSE handles the flush automatically via its writeback-and-retry path. Collapsing writable MAP_PRIVATE pagecache folios is still not supported, since PMD THP CoW only faults in at PTE level to avoid long CoW latency, and file_backed_vma_is_retractable() prevents it.
This means no-large-folio FSes need to add large folio support (the supported orders need to include PMD_ORDER), so that they can leverage file THP creation.
To prevent breaking file THP support for large folio FSes,
1. first 4 patches enable the support, so that without READ_ONLY_THP_FOR_FS, file THP still works for large folio FSes,
2. Patch 5 removes READ_ONLY_THP_FOR_FS Kconfig,
3. patches 6-12 remove code related to READ_ONLY_THP_FOR_FS,
4. patches 13-14 enable clean pagecache folio collapse for writable files.
NOTE: collapsing writable MAP_PRIVATE pagecache folios is not supported, since:
1. PMD THP CoW only faults in at PTE level to avoid long CoW latency,
2. the first check, due to 1, in file_backed_vma_is_retractable() prevents it.
Overview
===
1. collapse_file() checks for to-be-collapsed folio dirtiness after they are locked and unmapped to make sure no new write happens. Before, mapping->nr_thps and inode->i_writecount were used to cause read-only THP truncation before a fd becomes writable.
2. hugepage_enabled() is true for anon, shmem, and file-backed cases if the global khugepaged control is on, otherwise, khugepaged for file-backed case is turned off and anon and shmem depend on per-size control knobs.
3. collapse_file() from mm/khugepaged.c, instead of checking CONFIG_READ_ONLY_THP_FOR_FS, makes sure the mapping_max_folio_order() of struct address_space of the file is at least PMD_ORDER.
4. file_thp_enabled() checks mapping_max_folio_order() instead of CONFIG_READ_ONLY_THP_FOR_FS and no longer checks if the file is opened read-only. The dirty folio check after try_to_unmap() (Change 1) handles writable files correctly.
5. truncate_inode_partial_folio() calls folio_split() directly instead of the removed try_folio_split_to_order(), since large folios can only show up on a FS with large folio support.
6. nr_thps is removed from struct address_space, since it is no longer needed to drop all read-only THPs from a FS without large folio support when the fd becomes writable. Its related filemap_nr_thps*() are removed too.
7. folio_check_splittable() no longer checks READ_ONLY_THP_FOR_FS.
8. collapse_file() only calls filemap_flush() for read-only files. Blindly flushing dirty folios from writable files would cause undesirable system-wide writeback; userspace is expected to flush explicitly, or use MADV_COLLAPSE which handles it via its retry path.
9. Updated comments and selftests in various places.
Changelog
===
From V5[6]:
1. added mapping_min_folio_order(mapping) <= PMD_ORDER check to mapping_pmd_folio_support() in Patch 1 to correctly handle filesystems whose minimum folio order exceeds PMD_ORDER. Also improved the kernel-doc comment per David's suggestions.
2. cleaned up Patch 11 per David's review: use const for open_opt and mmap_prot, remove mmap_opt (use MAP_SHARED for both read-only and read-write mappings), inline file_fault_common() into separate file_fault_read() and file_fault_write() functions, fix "read only" typo to "read-only", update usage message to "with PMD-sized large folio support". Also fixed run_vmtests.sh to use elif test_selected thp for the SKIP case to avoid spurious [SKIP] output per Nico's report.
3. revised stale comment in Patch 13: removed "There won't be new dirty pages" and updated "khugepaged only works on read-only fd" to reflect that writable files are now supported; merged the comment blocks per David's suggestion.
From V4[5]:
1. fixed Patch 1's compilation error in !CONFIG_TRANSPARENT_HUGEPAGE
2. changed Patch 3 to no longer enable collapse for read-write fd but only allowe read-only fd.
3. added two new patches to enable clean pagecache folio collapse for writable files:
- Patch 13: remove inode_is_open_for_write() from file_thp_enabled() so that khugepaged and MADV_COLLAPSE can process writable files. filemap_flush() in collapse_file() is now conditionalized on the file being read-only, to avoid repeatedly writing back dirty folios from writable files.
- Patch 14: add read_write_file_read_ops and read_write_file_write_ops to the khugepaged selftest to cover the new writable-file collapse paths.
From V3[4]:
1. added a TODO comment in patch 1 noting that the is_shmem exception in the VM_WARN_ON_ONCE() check can be removed once shmem always calls mapping_set_large_folios() on its mapping. Used VM_WARN_ON_ONCE() in mapping_pmd_thp_support() instead.
2. fixed the dirty folio bail-out path in patch 2: add xas_unlock_irq() and folio_putback_lru() before the goto, which were missing and would have left the XA lock held and the LRU isolation ref leaked.
3. renamed hugepage_pmd_enabled() to hugepage_enabled() to reflect it controls khugepaged for all transparent hugepage types.
4. reverted the comment in hugepage_enabled() in patch 4 to the original; only removed the phrase "when configured in," which referred to CONFIG_READ_ONLY_THP_FOR_FS.
5. fixed commit message in patch 6: the dirty folio check is added after try_to_unmap() in collapse_file(), not after try_to_unmap_flush().
From V2[3]:
1. removed unnecessary check in collapse_scan_file().
2. removed inode_is_open_for_write() check in file_thp_enabled().
3. changed hugepage_enabled() to return true if khugepaged global control is on instead of false. cleaned up anon and shmem code in the function.
4. moved folio dirtiness check after try_to_unmap() but before try_to_unmap_flush(), since that is sufficient to prevent new writes.
5. reordered patch 4 and 5, so that khugepaged behavior does not change after READ_ONLY_THP_FOR_FS is removed.
6. added read-write file test in khugepaged selftest.
7. removed the read-only file restriction from guard-region selftest.
From V1[2]:
1. removed inode_is_open_for_write() check in collapse_file(), since the added folio dirtiness check after try_to_unmap_flush() should be sufficient to prevent writes to candidate folios.
2. removed READ_ONLY_THP_FOR_FS check in hugepage_enabled(), please see Patch 5 and item 2 in the overview for more details.
3. moved the patch removing READ_ONLY_THP_FOR_FS Kconfig after enabling khugepaged and MADV_COLLAPSE to create read-only THPs.
4. added mapping_pmd_thp_support() helper function.
5. used VM_WARN_ON_ONCE() in collapse_file() for mapping eligibility check and address alignment check instead of if + return error code. Always allow shmem, since MADV_COLLAPSE ignore shmem huge config.
6. added mapping eligibility check in collapse_scan_file().
7. removed trailing ; for folio_split() in the !CONFIG_TRANSPARENT_HUGEPAGE.
8. simplified code in folio_check_splittable() after removing READ_ONLY_THP_FOR_FS code.
9. clarified that read-only THP works for FSes with PMD THP support by default.
From RFC[1]:
1. instead of removing READ_ONLY_THP_FOR_FS function entirely, turn it on by default for all FSes with large folio support and the supported orders includes PMD_ORDER.
Suggestions and comments are welcome.
For Andrew to update "selftests/mm: khugepaged: use kselftest framework" from Mike Rapoport's "make MM selftests more CI friendly" series.
===
From 29f1e70373419e304ba7a69bc78fb43ba40ebfed Mon Sep 17 00:00:00 2001
From: "Mike Rapoport (Microsoft)" <[email protected]>
Date: Mon, 11 May 2026 19:27:58 +0300
Subject: [PATCH] selftests/mm: khugepaged: use kselftest framework
Convert khugepaged tests to use kselftest framework for reporting and tracking successful and failing runs.
The conversion is mostly about replacing printf()/perror() + exit() pairs with their ksft_ counterparts.
The nice colored success and failure indications are left intact.
Replace the progress report in collapse_compound_extreme() with a single ksft_print_msg() to avoid headache with formatting and make the test output more concise.
--- tools/testing/selftests/mm/khugepaged.c | 321 ++++++++++-------------- 1 file changed, 132 insertions(+), 189 deletions(-)
tools/testing/selftests/mm/khugepaged.c +132 -189
@@ -86,17 +86,19 @@ static int exit_status;static void success (const char *msg) {printf (" \e[32m%s\e[0m\n" , msg); + exit_status =KSFT_PASS ; }static void fail (const char *msg) {printf (" \e[31m%s\e[0m\n" , msg); - exit_status++; + exit_status =KSFT_FAIL ; }static void skip (const char *msg) {printf (" \e[33m%s\e[0m\n" , msg); + exit_status =KSFT_SKIP ; }static void restore_settings_atexit (void )
@@ -104,22 +106,24 @@ static void restore_settings_atexit(void)if (skip_settings_restore)return ; -printf ("Restore THP and khugepaged settings..." ); +ksft_print_msg ("Restore THP and khugepaged settings..." );thp_restore_settings ();success ("OK" ); skip_settings_restore = true; +ksft_print_cnts (); +exit (exit_status); }static void restore_settings (int sig) {/* exit() will invoke the restore_settings_atexit handler. */ -exit (sig ?EXIT_FAILURE : exit_status); +exit (sig ?KSFT_FAIL : exit_status); }static void save_settings (void ) { -printf ("Save THP and khugepaged settings..." ); +ksft_print_msg ("Save THP and khugepaged settings..." );if ((read_only_file_ops || read_write_file_read_ops || read_write_file_write_ops) && finfo.type ==VMA_FILE )
@@ -145,19 +149,13 @@ static void get_finfo(const char *dir) finfo.dir = dir;stat (finfo.dir, &path_stat); -if (!S_ISDIR (path_stat.st_mode)) { -printf ("%s: Not a directory (%s)\n" , __func__, finfo.dir); -exit (EXIT_FAILURE ); - } +if (!S_ISDIR (path_stat.st_mode)) +ksft_exit_fail_msg ("%s: Not a directory (%s)\n" , __func__, finfo.dir);if (snprintf (finfo.path,sizeof (finfo.path),"%s/" TEST_FILE , - finfo.dir) >=sizeof (finfo.path)) { -printf ("%s: Pathname is too long\n" , __func__); -exit (EXIT_FAILURE ); - } -if (statfs (finfo.dir, &fs)) { -perror ("statfs()" ); -exit (EXIT_FAILURE ); - } + finfo.dir) >=sizeof (finfo.path)) +ksft_exit_fail_msg ("%s: Pathname is too long\n" , __func__); +if (statfs (finfo.dir, &fs)) +ksft_exit_fail_perror ("statfs()" ); finfo.type = fs.f_type ==TMPFS_MAGIC ?VMA_SHMEM :VMA_FILE ;if (finfo.type ==VMA_SHMEM )return ;
@@ -165,40 +163,30 @@ static void get_finfo(const char *dir)/* Find owning device's queue/read_ahead_kb control */ if (snprintf (path,sizeof (path),"/sys/dev/block/%d:%d/uevent" ,major (path_stat.st_dev),minor (path_stat.st_dev)) - >=sizeof (path)) { -printf ("%s: Pathname is too long\n" , __func__); -exit (EXIT_FAILURE ); - } -if (read_file (path, buf,sizeof (buf)) <0 ) { -perror ("read_file(read_num)" ); -exit (EXIT_FAILURE ); - } + >=sizeof (path)) +ksft_exit_fail_msg ("%s: Pathname is too long\n" , __func__); +if (read_file (path, buf,sizeof (buf)) <0 ) +ksft_exit_fail_perror ("read_file(read_num)" );if (strstr (buf,"DEVTYPE=disk" )) {/* Found it */ if (snprintf (finfo.dev_queue_read_ahead_path,sizeof (finfo.dev_queue_read_ahead_path),"/sys/dev/block/%d:%d/queue/read_ahead_kb" ,major (path_stat.st_dev),minor (path_stat.st_dev)) - >=sizeof (finfo.dev_queue_read_ahead_path)) { -printf ("%s: Pathname is too long\n" , __func__); -exit (EXIT_FAILURE ); - } + >=sizeof (finfo.dev_queue_read_ahead_path)) +ksft_exit_fail_msg ("%s: Pathname is too long\n" , __func__);return ; } -if (!strstr (buf,"DEVTYPE=partition" )) { -printf ("%s: Unknown device type: %s\n" , __func__, path); -exit (EXIT_FAILURE ); - } +if (!strstr (buf,"DEVTYPE=partition" )) +ksft_exit_fail_msg ("%s: Unknown device type: %s\n" , __func__, path);/* * Partition of block device - need to find actual device. * Using naming convention that devnameN is partition of * device devname. */ str =strstr (buf,"DEVNAME=" ); -if (!str) { -printf ("%s: Could not read: %s" , __func__, path); -exit (EXIT_FAILURE ); - } +if (!str) +ksft_exit_fail_msg ("%s: Could not read: %s" , __func__, path); str +=8 ; end = str;while (*end) {
@@ -207,16 +195,13 @@ static void get_finfo(const char *dir)if (snprintf (finfo.dev_queue_read_ahead_path,sizeof (finfo.dev_queue_read_ahead_path),"/sys/block/%s/queue/read_ahead_kb" , - str) >=sizeof (finfo.dev_queue_read_ahead_path)) { -printf ("%s: Pathname is too long\n" , __func__); -exit (EXIT_FAILURE ); - } + str) >=sizeof (finfo.dev_queue_read_ahead_path)) +ksft_exit_fail_msg ("%s: Pathname is too long\n" , __func__);return ; } ++end; } -printf ("%s: Could not read: %s\n" , __func__, path); -exit (EXIT_FAILURE ); +ksft_exit_fail_msg ("%s: Could not read: %s\n" , __func__, path); }static bool check_swap (void *addr,unsigned long size)
@@ -229,26 +214,19 @@ static bool check_swap(void *addr, unsigned long size) ret =snprintf (addr_pattern,MAX_LINE_LENGTH ,"%08lx-" , (unsigned long ) addr); -if (ret >=MAX_LINE_LENGTH ) { -printf ("%s: Pattern is too long\n" , __func__); -exit (EXIT_FAILURE ); - } - +if (ret >=MAX_LINE_LENGTH ) +ksft_exit_fail_msg ("%s: Pattern is too long\n" , __func__); fp =fopen (PID_SMAPS ,"r" ); -if (!fp) { -printf ("%s: Failed to open file %s\n" , __func__,PID_SMAPS ); -exit (EXIT_FAILURE ); - } +if (!fp) +ksft_exit_fail_msg ("%s: Failed to open file %s\n" , __func__,PID_SMAPS );if (!check_for_pattern (fp, addr_pattern, buffer,sizeof (buffer))) goto err_out; ret =snprintf (addr_pattern,MAX_LINE_LENGTH ,"Swap:%19ld kB" , size >>10 ); -if (ret >=MAX_LINE_LENGTH ) { -printf ("%s: Pattern is too long\n" , __func__); -exit (EXIT_FAILURE ); - } +if (ret >=MAX_LINE_LENGTH ) +ksft_exit_fail_msg ("%s: Pattern is too long\n" , __func__);/* * Fetch the Swap: in the same block and check whether it got * the expected number of hugeepages next.
@@ -271,10 +249,8 @@ static void *alloc_mapping(int nr) p =mmap (BASE_ADDR , nr * hpage_pmd_size,PROT_READ |PROT_WRITE ,MAP_ANONYMOUS |MAP_PRIVATE ,-1 ,0 ); -if (p !=BASE_ADDR ) { -printf ("Failed to allocate VMA at %p\n" ,BASE_ADDR ); -exit (EXIT_FAILURE ); - } +if (p !=BASE_ADDR ) +ksft_exit_fail_msg ("Failed to allocate VMA at %p\n" ,BASE_ADDR );return p; }
@@ -324,19 +300,13 @@ static void *alloc_hpage(struct mem_ops *ops)* khugepaged on low-load system (like a test machine), which * would cause MADV_COLLAPSE to fail with EAGAIN. */ -printf ("Allocate huge page..." ); -if (madvise_collapse_retry (p, hpage_pmd_size)) { -perror ("madvise(MADV_COLLAPSE)" ); -exit (EXIT_FAILURE ); - } -if (!ops->check_huge (p,1 )) { -perror ("madvise(MADV_COLLAPSE)" ); -exit (EXIT_FAILURE ); - } -if (madvise (p, hpage_pmd_size,MADV_HUGEPAGE )) { -perror ("madvise(MADV_HUGEPAGE)" ); -exit (EXIT_FAILURE ); - } +ksft_print_msg ("Allocate huge page..." ); +if (madvise_collapse_retry (p, hpage_pmd_size)) +ksft_exit_fail_perror ("madvise(MADV_COLLAPSE)" ); +if (!ops->check_huge (p,1 )) +ksft_exit_fail_perror ("madvise(MADV_COLLAPSE)" ); +if (madvise (p, hpage_pmd_size,MADV_HUGEPAGE )) +ksft_exit_fail_perror ("madvise(MADV_HUGEPAGE)" );success ("OK" );return p; }
@@ -346,11 +316,9 @@ static void validate_memory(int *p, unsigned long start, unsigned long end)int i;for (i = start / page_size; i < end / page_size; i++) { -if (p[i * page_size /sizeof (*p)] != i +0xdead0000 ) { -printf ("Page %d is corrupted: %#x\n" , - i, p[i * page_size /sizeof (*p)]); -exit (EXIT_FAILURE ); - } +if (p[i * page_size /sizeof (*p)] != i +0xdead0000 ) +ksft_exit_fail_msg ("Page %d is corrupted: %#x\n" , + i, p[i * page_size /sizeof (*p)]); } }
@@ -383,14 +351,12 @@ static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup)unsigned long size;unlink (finfo.path);/* Cleanup from previous failed tests */ -printf ("Creating %s for collapse%s..." , finfo.path, - finfo.type ==VMA_SHMEM ?" (tmpfs)" :"" ); +ksft_print_msg ("Creating %s for collapse%s..." , finfo.path, + finfo.type ==VMA_SHMEM ?" (tmpfs)" :"" ); fd =open (finfo.path,O_CREAT |O_RDWR |O_TRUNC |O_EXCL ,777 ); -if (fd <0 ) { -perror ("open()" ); -exit (EXIT_FAILURE ); - } +if (fd <0 ) +ksft_exit_fail_perror ("open()" ); size = nr_hpages * hpage_pmd_size;if (ftruncate (fd, size)) {
@@ -411,22 +377,17 @@ static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup)close (fd);munmap (p, size);success ("OK" ); - -printf ("Opening %s %s for collapse..." , finfo.path, +ksft_print_msg ("Opening %s %s for collapse..." , finfo.path, setup ==FILE_SETUP_READ_ONLY_FS ?"read-only" : setup ==FILE_SETUP_READ_WRITE_FS_READ_DATA ?"read-write (read)" :"read-write (write)" ); finfo.fd =open (finfo.path, open_opt,777 ); -if (finfo.fd <0 ) { -perror ("open()" ); -exit (EXIT_FAILURE ); - } +if (finfo.fd <0 ) +ksft_exit_fail_perror ("open()" ); p =mmap (BASE_ADDR , size, mmap_prot,MAP_SHARED , finfo.fd,0 ); -if (p ==MAP_FAILED || p !=BASE_ADDR ) { -perror ("mmap()" ); -exit (EXIT_FAILURE ); - } +if (p ==MAP_FAILED || p !=BASE_ADDR ) +ksft_exit_fail_perror ("mmap()" );/* Drop page cache */ write_file ("/proc/sys/vm/drop_caches" ,"3" ,2 );
@@ -458,10 +419,8 @@ static void file_cleanup_area(void *p, unsigned long size)static void file_fault_read (void *p,unsigned long start,unsigned long end) { -if (madvise (((char *)p) + start, end - start,MADV_POPULATE_READ )) { -perror ("madvise(MADV_POPULATE_READ)" ); -exit (EXIT_FAILURE ); - } +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_READ )) +ksft_exit_fail_perror ("madvise(MADV_POPULATE_READ)" ); }static void file_fault_read_and_flush (void *p,unsigned long start,unsigned long end)
@@ -476,10 +435,8 @@ static void file_fault_read_and_flush(void *p, unsigned long start, unsigned lonstatic void file_fault_write (void *p,unsigned long start,unsigned long end) { -if (madvise (((char *)p) + start, end - start,MADV_POPULATE_WRITE )) { -perror ("madvise(MADV_POPULATE_WRITE)" ); -exit (EXIT_FAILURE ); - } +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_WRITE )) +ksft_exit_fail_perror ("madvise(MADV_POPULATE_WRITE)" ); }static bool file_check_huge (void *addr,int nr_hpages)
@@ -501,20 +458,14 @@ static void *shmem_setup_area(int nr_hpages)unsigned long size = nr_hpages * hpage_pmd_size; finfo.fd =memfd_create ("khugepaged-selftest-collapse-shmem" ,0 ); -if (finfo.fd <0 ) { -perror ("memfd_create()" ); -exit (EXIT_FAILURE ); - } -if (ftruncate (finfo.fd, size)) { -perror ("ftruncate()" ); -exit (EXIT_FAILURE ); - } +if (finfo.fd <0 ) +ksft_exit_fail_perror ("memfd_create()" ); +if (ftruncate (finfo.fd, size)) +ksft_exit_fail_perror ("ftruncate()" ); p =mmap (BASE_ADDR , size,PROT_READ |PROT_WRITE ,MAP_SHARED , finfo.fd,0 ); -if (p !=BASE_ADDR ) { -perror ("mmap()" ); -exit (EXIT_FAILURE ); - } +if (p !=BASE_ADDR ) +ksft_exit_fail_perror ("mmap()" );return p; }
@@ -588,7 +539,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,int ret;struct thp_settings settings = *thp_current_settings (); -printf ("%s..." , msg); +ksft_print_msg ("%s..." , msg);/* * read&write file collapse succeeds for MADV_COLLAPSE because dirty
@@ -621,10 +572,8 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages, struct mem_ops *ops, bool expect) {/* Sanity check */ -if (!ops->check_huge (p,0 )) { -printf ("Unexpected huge page\n" ); -exit (EXIT_FAILURE ); - } +if (!ops->check_huge (p,0 )) +ksft_exit_fail_msg ("Unexpected huge page\n" );__madvise_collapse (msg, p, nr_hpages, ops, expect); }
@@ -636,17 +585,15 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages,int timeout =6 ;/* 3 seconds */ /* Sanity check */ -if (!ops->check_huge (p,0 )) { -printf ("Unexpected huge page\n" ); -exit (EXIT_FAILURE ); - } +if (!ops->check_huge (p,0 )) +ksft_exit_fail_msg ("Unexpected huge page\n" );madvise (p, nr_hpages * hpage_pmd_size,MADV_HUGEPAGE );/* Wait until the second full_scan completed */ full_scans =thp_read_num ("khugepaged/full_scans" ) +2 ; -printf ("%s..." , msg); +ksft_print_msg ("%s..." , msg);while (timeout--) {if (ops->check_huge (p, nr_hpages))break ;
@@ -713,7 +660,7 @@ static void alloc_at_fault(void) p =alloc_mapping (1 ); *p =1 ; -printf ("Allocate huge page on fault..." ); +ksft_print_msg ("Allocate huge page on fault..." );if (check_huge_anon (p,1 , hpage_pmd_size))success ("OK" );else
@@ -722,12 +669,14 @@ static void alloc_at_fault(void)thp_pop_settings ();madvise (p, page_size,MADV_DONTNEED ); -printf ("Split huge PMD on MADV_DONTNEED..." ); +ksft_print_msg ("Split huge PMD on MADV_DONTNEED..." );if (check_huge_anon (p,0 , hpage_pmd_size))success ("OK" );else fail ("Fail" );munmap (p, hpage_pmd_size); + +ksft_test_result_report (exit_status,"allocate on fault and split\n" ); }static void collapse_full (struct collapse_context *c,struct mem_ops *ops)
@@ -742,6 +691,8 @@ static void collapse_full(struct collapse_context *c, struct mem_ops *ops) ops, true);validate_memory (p,0 , size); ops->cleanup_area (p, size); + +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_empty (struct collapse_context *c,struct mem_ops *ops)
@@ -751,6 +702,7 @@ static void collapse_empty(struct collapse_context *c, struct mem_ops *ops) p = ops->setup_area (1 ); c->collapse ("Do not collapse empty PTE table" , p,1 , ops, false); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_single_pte_entry (struct collapse_context *c,struct mem_ops *ops)
@@ -762,6 +714,7 @@ static void collapse_single_pte_entry(struct collapse_context *c, struct mem_ops c->collapse ("Collapse PTE table with single PTE entry present" , p,1 , ops, true); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_max_ptes_none (struct collapse_context *c,struct mem_ops *ops)
@@ -801,6 +754,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o skip: ops->cleanup_area (p, hpage_pmd_size);thp_pop_settings (); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_swapin_single_pte (struct collapse_context *c,struct mem_ops *ops)
@@ -810,11 +764,9 @@ static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_op p = ops->setup_area (1 ); ops->fault (p,0 , hpage_pmd_size); -printf ("Swapout one page..." ); -if (madvise (p, page_size,MADV_PAGEOUT )) { -perror ("madvise(MADV_PAGEOUT)" ); -exit (EXIT_FAILURE ); - } +ksft_print_msg ("Swapout one page..." ); +if (madvise (p, page_size,MADV_PAGEOUT )) +ksft_exit_fail_perror ("madvise(MADV_PAGEOUT)" );if (check_swap (p, page_size)) {success ("OK" ); }else {
@@ -827,6 +779,7 @@ static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_opvalidate_memory (p,0 , hpage_pmd_size); out: ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_max_ptes_swap (struct collapse_context *c,struct mem_ops *ops)
@@ -837,11 +790,9 @@ static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *o p = ops->setup_area (1 ); ops->fault (p,0 , hpage_pmd_size); -printf ("Swapout %d of %d pages..." , max_ptes_swap +1 , hpage_pmd_nr); -if (madvise (p, (max_ptes_swap +1 ) * page_size,MADV_PAGEOUT )) { -perror ("madvise(MADV_PAGEOUT)" ); -exit (EXIT_FAILURE ); - } +ksft_print_msg ("Swapout %d of %d pages..." , max_ptes_swap +1 , hpage_pmd_nr); +if (madvise (p, (max_ptes_swap +1 ) * page_size,MADV_PAGEOUT )) +ksft_exit_fail_perror ("madvise(MADV_PAGEOUT)" );if (check_swap (p, (max_ptes_swap +1 ) * page_size)) {success ("OK" ); }else {
@@ -855,12 +806,10 @@ static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *oif (c->enforce_pte_scan_limits) { ops->fault (p,0 , hpage_pmd_size); -printf ("Swapout %d of %d pages..." , max_ptes_swap, +ksft_print_msg ("Swapout %d of %d pages..." , max_ptes_swap, hpage_pmd_nr); -if (madvise (p, max_ptes_swap * page_size,MADV_PAGEOUT )) { -perror ("madvise(MADV_PAGEOUT)" ); -exit (EXIT_FAILURE ); - } +if (madvise (p, max_ptes_swap * page_size,MADV_PAGEOUT )) +ksft_exit_fail_perror ("madvise(MADV_PAGEOUT)" );if (check_swap (p, max_ptes_swap * page_size)) {success ("OK" ); }else {
@@ -874,6 +823,7 @@ static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *o } out: ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_single_pte_entry_compound (struct collapse_context *c,struct mem_ops *ops)
@@ -890,7 +840,7 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, struc }madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE ); -printf ("Split huge page leaving single PTE mapping compound page..." ); +ksft_print_msg ("Split huge page leaving single PTE mapping compound page..." );madvise (p + page_size, hpage_pmd_size - page_size,MADV_DONTNEED );if (ops->check_huge (p,0 ))success ("OK" );
@@ -902,6 +852,7 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, strucvalidate_memory (p,0 , page_size); skip: ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_full_of_compound (struct collapse_context *c,struct mem_ops *ops)
@@ -909,7 +860,7 @@ static void collapse_full_of_compound(struct collapse_context *c, struct mem_opsvoid *p; p =alloc_hpage (ops); -printf ("Split huge page leaving single PTE page table full of compound pages..." ); +ksft_print_msg ("Split huge page leaving single PTE page table full of compound pages..." );madvise (p, page_size,MADV_NOHUGEPAGE );madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE );if (ops->check_huge (p,0 ))
@@ -921,6 +872,7 @@ static void collapse_full_of_compound(struct collapse_context *c, struct mem_ops true);validate_memory (p,0 , hpage_pmd_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_compound_extreme (struct collapse_context *c,struct mem_ops *ops)
@@ -929,16 +881,12 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_opsint i; p = ops->setup_area (1 ); +ksft_print_msg ("Construct PTE page table full of different PTE-mapped compound pages\n" );for (i =0 ; i < hpage_pmd_nr; i++) { -printf ("\rConstruct PTE page table full of different PTE-mapped compound pages %3d/%d..." , - i +1 , hpage_pmd_nr); -madvise (BASE_ADDR , hpage_pmd_size,MADV_HUGEPAGE ); ops->fault (BASE_ADDR ,0 , hpage_pmd_size); -if (!ops->check_huge (BASE_ADDR ,1 )) { -printf ("Failed to allocate huge page\n" ); -exit (EXIT_FAILURE ); - } +if (!ops->check_huge (BASE_ADDR ,1 )) +ksft_exit_fail_msg ("Failed to allocate huge page\n" );madvise (BASE_ADDR , hpage_pmd_size,MADV_NOHUGEPAGE ); p = mremap(BASE_ADDR - i * page_size,
@@ -946,20 +894,16 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_ops (i +1 ) * page_size,MREMAP_MAYMOVE |MREMAP_FIXED , BASE_ADDR + 2 * hpage_pmd_size); -if (p ==MAP_FAILED ) { -perror ("mremap+unmap" ); -exit (EXIT_FAILURE ); - } +if (p ==MAP_FAILED ) +ksft_exit_fail_perror ("mremap+unmap" ); p =mremap (BASE_ADDR +2 * hpage_pmd_size, (i +1 ) * page_size, (i +1 ) * page_size + hpage_pmd_size,MREMAP_MAYMOVE |MREMAP_FIXED ,BASE_ADDR - (i +1 ) * page_size); -if (p ==MAP_FAILED ) { -perror ("mremap+alloc" ); -exit (EXIT_FAILURE ); - } +if (p ==MAP_FAILED ) +ksft_exit_fail_perror ("mremap+alloc" ); } ops->cleanup_area (BASE_ADDR , hpage_pmd_size);
@@ -974,6 +918,7 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_opsvalidate_memory (p,0 , hpage_pmd_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_fork (struct collapse_context *c,struct mem_ops *ops)
@@ -983,18 +928,17 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) p = ops->setup_area (1 ); -printf ("Allocate small page..." ); +ksft_print_msg ("Allocate small page..." ); ops->fault (p,0 , page_size);if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" ); -printf ("Share small page over fork()..." ); +ksft_print_msg ("Share small page over fork()..." );if (!fork ()) {/* Do not touch settings on child exit */ skip_settings_restore = true; - exit_status =0 ;if (ops->check_huge (p,0 ))success ("OK" );
@@ -1011,15 +955,16 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) }wait (&wstatus); - exit_status +=WEXITSTATUS (wstatus); + exit_status =WEXITSTATUS (wstatus); -printf ("Check if parent still has small page..." ); +ksft_print_msg ("Check if parent still has small page..." );if (ops->check_huge (p,0 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , page_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_fork_compound (struct collapse_context *c,struct mem_ops *ops)
@@ -1028,18 +973,17 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *ovoid *p; p =alloc_hpage (ops); -printf ("Share huge page over fork()..." ); +ksft_print_msg ("Share huge page over fork()..." );if (!fork ()) {/* Do not touch settings on child exit */ skip_settings_restore = true; - exit_status =0 ;if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" ); -printf ("Split huge page PMD in child process..." ); +ksft_print_msg ("Split huge page PMD in child process..." );madvise (p, page_size,MADV_NOHUGEPAGE );madvise (p, hpage_pmd_size,MADV_NOHUGEPAGE );if (ops->check_huge (p,0 ))
@@ -1060,15 +1004,16 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o }wait (&wstatus); - exit_status +=WEXITSTATUS (wstatus); + exit_status =WEXITSTATUS (wstatus); -printf ("Check if parent still has huge page..." ); +ksft_print_msg ("Check if parent still has huge page..." );if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , hpage_pmd_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void collapse_max_ptes_shared (struct collapse_context *c,struct mem_ops *ops)
@@ -1078,18 +1023,17 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_opsvoid *p; p =alloc_hpage (ops); -printf ("Share huge page over fork()..." ); +ksft_print_msg ("Share huge page over fork()..." );if (!fork ()) {/* Do not touch settings on child exit */ skip_settings_restore = true; - exit_status =0 ;if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" ); -printf ("Trigger CoW on page %d of %d..." , +ksft_print_msg ("Trigger CoW on page %d of %d..." , hpage_pmd_nr - max_ptes_shared -1 , hpage_pmd_nr); ops->fault (p,0 , (hpage_pmd_nr - max_ptes_shared -1 ) * page_size);if (ops->check_huge (p,0 ))
@@ -1101,7 +1045,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops 1, ops, !c->enforce_pte_scan_limits);if (c->enforce_pte_scan_limits) { -printf ("Trigger CoW on page %d of %d..." , +ksft_print_msg ("Trigger CoW on page %d of %d..." , hpage_pmd_nr - max_ptes_shared, hpage_pmd_nr); ops->fault (p,0 , (hpage_pmd_nr - max_ptes_shared) * page_size);
@@ -1120,15 +1064,16 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops }wait (&wstatus); - exit_status +=WEXITSTATUS (wstatus); + exit_status =WEXITSTATUS (wstatus); -printf ("Check if parent still has huge page..." ); +ksft_print_msg ("Check if parent still has huge page..." );if (ops->check_huge (p,1 ))success ("OK" );else fail ("Fail" );validate_memory (p,0 , hpage_pmd_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); } static void madvise_collapse_existing_thps(struct collapse_context *c,
@@ -1145,6 +1090,7 @@ static void madvise_collapse_existing_thps(struct collapse_context *c,__madvise_collapse ("Re-collapse PMD-mapped hugepage" , p,1 , ops, true);validate_memory (p,0 , hpage_pmd_size); ops->cleanup_area (p, hpage_pmd_size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }/*
@@ -1172,6 +1118,7 @@ static void madvise_retracted_page_tables(struct collapse_context *c, true);validate_memory (p,0 , size); ops->cleanup_area (p, size); +ksft_test_result_report (exit_status,"%s\n" , __func__); }static void usage (void )
@@ -1280,10 +1227,8 @@ static int nr_test_cases;#define TEST (t, c, o) do { \ if (c && o) { \ - if (nr_test_cases >= MAX_TEST_CASES) { \ - printf("MAX_TEST_CASES is too small\n"); \ - exit(EXIT_FAILURE); \ - } \ + if (nr_test_cases >= MAX_TEST_CASES) \ + ksft_exit_fail_msg("MAX_TEST_CASES is too small\n"); \ test_cases[nr_test_cases++] = (struct test_case){ \ .ctx = c, \ .ops = o, \
@@ -1316,10 +1261,10 @@ int main(int argc, char **argv) .read_ahead_kb = 0, }; -if (!thp_is_enabled ()) { -printf ("Transparent Hugepages not available\n" ); -return KSFT_SKIP ; - } +ksft_print_header (); + +if (!thp_is_enabled ()) +ksft_exit_skip ("Transparent Hugepages not available\n" );parse_test_type (argc, argv);
@@ -1327,10 +1272,8 @@ int main(int argc, char **argv) page_size =getpagesize (); hpage_pmd_size =read_pmd_pagesize (); -if (!hpage_pmd_size) { -printf ("Reading PMD pagesize failed" ); -exit (EXIT_FAILURE ); - } +if (!hpage_pmd_size) +ksft_exit_fail_msg ("Reading PMD pagesize failed\n" ); hpage_pmd_nr = hpage_pmd_size / page_size; hpage_pmd_order =__builtin_ctz (hpage_pmd_nr);
@@ -1346,8 +1289,6 @@ int main(int argc, char **argv)save_settings ();thp_push_settings (&default_settings); -alloc_at_fault (); -TEST (collapse_full, khugepaged_context, anon_ops);TEST (collapse_full, khugepaged_context, read_only_file_ops);TEST (collapse_full, khugepaged_context, read_write_file_read_ops);
@@ -1425,11 +1366,13 @@ int main(int argc, char **argv)TEST (madvise_retracted_page_tables, madvise_context, read_write_file_read_ops);TEST (madvise_retracted_page_tables, madvise_context, shmem_ops); - exit_status =KSFT_PASS ; +ksft_set_plan (nr_test_cases +1 ); + +alloc_at_fault ();for (int i =0 ; i < nr_test_cases; i++) {struct test_case *t = &test_cases[i]; -printf ("\nRun test: %s (%s:%s)\n" , t->desc, t->ctx->name, t->ops->name); +ksft_print_msg ("\nRun test: %s (%s:%s)\n" , t->desc, t->ctx->name, t->ops->name); t->fn (t->ctx, t->ops); }
Signature · 40 lines
2.53.0
Zi Yan (14):
mm/khugepaged: remove READ_ONLY_THP_FOR_FS check
mm/khugepaged: add folio dirty check after try_to_unmap()
mm/huge_memory: remove READ_ONLY_THP_FOR_FS from file_thp_enabled()
mm/khugepaged: remove READ_ONLY_THP_FOR_FS check in hugepage_enabled()
mm: remove READ_ONLY_THP_FOR_FS Kconfig option
mm: fs: remove filemap_nr_thps*() functions and their users
fs: remove nr_thps from struct address_space
mm/huge_memory: remove folio split check for READ_ONLY_THP_FOR_FS
mm/truncate: use folio_split() in truncate_inode_partial_folio()
fs/btrfs: remove a comment referring to READ_ONLY_THP_FOR_FS
selftests/mm: remove READ_ONLY_THP_FOR_FS in khugepaged
selftests/mm: remove READ_ONLY_THP_FOR_FS code from guard-regions
mm/khugepaged: enable clean pagecache folio collapse for writable
files
selftests/mm: add writable-file collapse tests for khugepaged
fs/btrfs/defrag.c | 3 -
fs/inode.c | 3 -
fs/open.c | 27 ---
include/linux/fs.h | 5 -
include/linux/huge_mm.h | 25 +--
include/linux/pagemap.h | 50 +++---
include/linux/shmem_fs.h | 2 +-
mm/Kconfig | 11 --
mm/filemap.c | 1 -
mm/huge_memory.c | 39 +----
mm/khugepaged.c | 107 ++++++------
mm/truncate.c | 8 +-
tools/testing/selftests/mm/guard-regions.c | 18 +-
tools/testing/selftests/mm/khugepaged.c | 184 ++++++++++++++++-----
tools/testing/selftests/mm/run_vmtests.sh | 12 +-
15 files changed, 254 insertions(+), 241 deletions(-)
--
2.53.0Zi Yan
reviewerPATCH v6 1/14 mm/khugepaged: remove READ_ONLY_THP_FOR_FS check
collapse_file() requires FSes supporting large folio with at least PMD_ORDER, so replace the READ_ONLY_THP_FOR_FS check with that. MADV_COLLAPSE ignores shmem huge config, so exclude the check for shmem.
While at it, replace VM_BUG_ON with VM_WARN_ON_ONCE.
Add a helper function mapping_pmd_folio_support() for FSes supporting large folio with at least PMD_ORDER.
--- include/linux/pagemap.h | 27 +++++++++++++++++++++++++++ mm/khugepaged.c | 10 ++++++++-- 2 files changed, 35 insertions(+), 2 deletions(-)
include/linux/pagemap.h +27 -0
@@ -513,6 +513,33 @@ static inline bool mapping_large_folio_support(const struct address_space *mappireturn mapping_max_folio_order (mapping) >0 ; } +/** +* mapping_pmd_folio_support() - Check if a mapping supports PMD-sized folio +* @mapping: The address_space +* +* While some mappings support large folios, they might not support PMD-sized +* folios. This function checks whether a mapping supports PMD-sized folios. +* For example, khugepaged needs this information before attempting to +* collapsing THPs. +* +* Return: True if PMD-sized folios are supported, otherwise false. +*/ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline bool mapping_pmd_folio_support (const struct address_space *mapping) +{ +/* AS_FOLIO_ORDER is only reasonable for pagecache folios */ +VM_WARN_ON_ONCE ((unsigned long )mapping &FOLIO_MAPPING_ANON ); + +return mapping_min_folio_order (mapping) <=PMD_ORDER && +mapping_max_folio_order (mapping) >=PMD_ORDER ; +} +#else +static inline bool mapping_pmd_folio_support (const struct address_space *mapping) +{ +return false; +} +#endif +/* Return the maximum folio size for this pagecache mapping, in bytes. */ static inline size_t mapping_max_folio_size (const struct address_space *mapping) {
mm/khugepaged.c +8 -2
@@ -2243,8 +2243,14 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,int nr_none =0 ;bool is_shmem =shmem_file (file); -VM_BUG_ON (!IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && !is_shmem); -VM_BUG_ON (start & (HPAGE_PMD_NR -1 )); +/* +* MADV_COLLAPSE ignores shmem huge config, so do not check shmem +* +* TODO: once shmem always calls mapping_set_large_folios() on its +* mapping, the shmem check can be removed. +*/ +VM_WARN_ON_ONCE (!is_shmem && !mapping_pmd_folio_support (mapping)); +VM_WARN_ON_ONCE (start & (HPAGE_PMD_NR -1 )); result =alloc_charge_folio (&new_folio, mm, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED )
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 2/14 mm/khugepaged: add folio dirty check after try_to_unmap()
In reply to Zi YanThis check ensures the correctness of read-only PMD folio collapse after it is enabled for all FSes supporting PMD pagecache folios and replaces READ_ONLY_THP_FOR_FS.
READ_ONLY_THP_FOR_FS only supports read-only fd and uses mapping->nr_thps and inode->i_writecount to prevent any write to read-only to-be-collapsed folios. In upcoming commits, READ_ONLY_THP_FOR_FS will be removed and the aforementioned mechanism will go away too. To ensure khugepaged functions as expected after the changes, skip if any folio is dirty after try_to_unmap(), since a dirty folio at that point means this read-only folio can get writes between try_to_unmap() and try_to_unmap_flush() via cached TLB entries and khugepaged does not support writable pagecache folio collapse yet.
--- mm/khugepaged.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
mm/khugepaged.c +24 -4
@@ -2335,8 +2335,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, } }else if (folio_test_dirty (folio)) {/* -* khugepaged only works on read-only fd, -* so this page is dirty because it hasn't +* This page is dirty because it hasn't * been flushed since first write. There * won't be new dirty pages. *
@@ -2394,8 +2393,8 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,if (!is_shmem && (folio_test_dirty (folio) ||folio_test_writeback (folio))) {/* -* khugepaged only works on read-only fd, so this -* folio is dirty because it hasn't been flushed +* khugepaged only works on clean file-backed folios, +* so this folio is dirty because it hasn't been flushed * since first write. */ result =SCAN_PAGE_DIRTY_OR_WRITEBACK ;
@@ -2439,6 +2438,27 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, goto out_unlock; } +/* +* At this point, the folio is locked and unmapped. If the PTE +* was dirty, try_to_unmap() has transferred the dirty bit to +* the folio and we must not collapse it into a clean +* file-backed folio. +* +* If the folio is clean here, no one can write it until we +* drop the folio lock. A write through a stale TLB entry came +* from a clean PTE and must fault because the PTE has been +* cleared; the fault path has to take the folio lock before +* installing a writable mapping. Buffered write paths also +* have to take the folio lock before modifying file contents +* without a mapping, typically via write_begin_get_folio(). +*/ +if (!is_shmem &&folio_test_dirty (folio)) { + result =SCAN_PAGE_DIRTY_OR_WRITEBACK ; +xas_unlock_irq (&xas); +folio_putback_lru (folio); + goto out_unlock; + } +/* * Accumulate the folios that are being collapsed. */
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 3/14 mm/huge_memory: remove READ_ONLY_THP_FOR_FS from file_thp_enabled()
In reply to Zi YanReplace it with a check on the max folio order of the file's address space mapping, making sure PMD folio is supported. Keep the inode open-for-write check, since even if collapse_file() now makes sure all to-be-collapsed folios are clean and the created PMD file THP can be handled by FSes properly, the filemap_flush() could perform undesirable write back.
--- mm/huge_memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/huge_memory.c +3 -3
@@ -86,9 +86,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) {struct inode *inode; -if (!IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) -return false; -if (!vma->vm_file)return false;
@@ -97,6 +94,9 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)if (IS_ANON_FILE (inode))return false; +if (!mapping_pmd_folio_support (vma->vm_file->f_mapping)) +return false; +return !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); }
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 4/14 mm/khugepaged: remove READ_ONLY_THP_FOR_FS check in hugepage_enabled()
In reply to Zi YanRemove the READ_ONLY_THP_FOR_FS gate and khugepaged for file-backed pmd-sized hugepages are enabled by the global transparent hugepage control. khugepaged can still be enabled by per-size control for anon and shmem when the global control is off.
Add shmem_hpage_pmd_enabled() stub for !CONFIG_SHMEM to remove IS_ENABLED(SHMEM) in hugepage_enabled().
Clean up hugepage_enabled() by moving anon code to anon_hpage_enabled().
--- include/linux/shmem_fs.h | 2 +- mm/khugepaged.c | 26 ++++++++++++++++---------- 2 files changed, 17 insertions(+), 11 deletions(-)
include/linux/shmem_fs.h +1 -1
@@ -127,7 +127,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,void shmem_truncate_range (struct inode *inode,loff_t start,uoff_t end);int shmem_unuse (unsigned int type); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && defined(CONFIG_SHMEM )unsigned long shmem_allowable_huge_orders (struct inode *inode,struct vm_area_struct *vma,pgoff_t index,loff_t write_end,bool shmem_huge_force);
mm/khugepaged.c +16 -10
@@ -522,26 +522,32 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); } +static inline bool anon_hpage_enabled (void ) +{ +if (READ_ONCE (huge_anon_orders_always)) +return true; +if (READ_ONCE (huge_anon_orders_madvise)) +return true; +if (READ_ONCE (huge_anon_orders_inherit) && +hugepage_global_enabled ()) +return true; +return false; +} +static bool hugepage_enabled (void ) {/* * We cover the anon, shmem and the file-backed case here; file-backed -* hugepages, when configured in, are determined by the global control. +* hugepages are determined by the global control. * Anon hugepages are determined by its per-size mTHP control. * Shmem pmd-sized hugepages are also determined by its pmd-size control, * except when the global shmem_huge is set to SHMEM_HUGE_DENY. */ -if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && -hugepage_global_enabled ()) -return true; -if (READ_ONCE (huge_anon_orders_always)) +if (hugepage_global_enabled ())return true; -if (READ_ONCE (huge_anon_orders_madvise)) -return true; -if (READ_ONCE (huge_anon_orders_inherit) && -hugepage_global_enabled ()) +if (anon_hpage_enabled ())return true; -if (IS_ENABLED (CONFIG_SHMEM ) &&shmem_hpage_pmd_enabled ()) +if (shmem_hpage_pmd_enabled ())return true;return false; }
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 5/14 mm: remove READ_ONLY_THP_FOR_FS Kconfig option
In reply to Zi YanAfter removing READ_ONLY_THP_FOR_FS check in file_thp_enabled(), khugepaged and MADV_COLLAPSE can run on FSes with PMD THP pagecache support even without READ_ONLY_THP_FOR_FS enabled. Remove the Kconfig first so that no one can use READ_ONLY_THP_FOR_FS as upcoming commits remove mapping->nr_thps, which its safe guard mechanism relies on.
--- mm/Kconfig | 11 ----------- 1 file changed, 11 deletions(-)
mm/Kconfig +0 -11
@@ -936,17 +936,6 @@ config THP_SWAP For selection by architectures with reasonable THP sizes. -config READ_ONLY_THP_FOR_FS - bool "Read-only THP for filesystems (EXPERIMENTAL)" - depends on TRANSPARENT_HUGEPAGE - - help - Allow khugepaged to put read-only file-backed pages in THP. - - This is marked experimental because it is a new feature. Write - support of file THPs will be developed in the next few release - cycles. - config NO_PAGE_MAPCOUNT bool "No per-page mapcount (EXPERIMENTAL)" help
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 6/14 mm: fs: remove filemap_nr_thps*() functions and their users
In reply to Zi YanThey are used by READ_ONLY_THP_FOR_FS to handle writes to FSes without large folio support, so that read-only THPs created in these FSes are not seen by the FSes when the underlying fd becomes writable. Now read-only PMD THPs only appear in a FS with large folio support and the supported orders include PMD_ORDER.
READ_ONLY_THP_FOR_FS was using mapping->nr_thps, inode->i_writecount, and smp_mb() to prevent writes to a read-only THP and collapsing writable folios into a THP. In collapse_file(), mapping->nr_thps is increased, then smp_mb(), and if inode->i_writecount > 0, collapse is stopped, while do_dentry_open() first increases inode->i_writecount, then a full memory fence, and if mapping->nr_thps > 0, all read-only THPs are truncated.
Now this mechanism can be removed along with READ_ONLY_THP_FOR_FS code, since a dirty folio check has been added after try_to_unmap() in collapse_file() to prevent dirty folios from being collapsed as clean.
--- fs/open.c | 27 --------------------------- include/linux/pagemap.h | 29 ----------------------------- mm/filemap.c | 1 - mm/huge_memory.c | 1 - mm/khugepaged.c | 28 ---------------------------- 5 files changed, 86 deletions(-)
fs/open.c +0 -27
@@ -968,33 +968,6 @@ static int do_dentry_open(struct file *f,if ((f->f_flags &O_DIRECT ) && !(f->f_mode &FMODE_CAN_ODIRECT ))return -EINVAL ; -/* -* XXX: Huge page cache doesn't support writing yet. Drop all page -* cache for this file before processing writes. -*/ -if (f->f_mode &FMODE_WRITE ) { -/* -* Depends on full fence from get_write_access() to synchronize -* against collapse_file() regarding i_writecount and nr_thps -* updates. Ensures subsequent insertion of THPs into the page -* cache will fail. -*/ -if (filemap_nr_thps (inode->i_mapping)) { -struct address_space *mapping = inode->i_mapping; - -filemap_invalidate_lock (inode->i_mapping); -/* -* unmap_mapping_range just need to be called once -* here, because the private pages is not need to be -* unmapped mapping (e.g. data segment of dynamic -* shared libraries here). -*/ -unmap_mapping_range (mapping,0 ,0 ,0 ); -truncate_inode_pages (mapping,0 ); -filemap_invalidate_unlock (inode->i_mapping); - } - } -return 0 ; cleanup_all:
include/linux/pagemap.h +0 -29
@@ -546,35 +546,6 @@ static inline size_t mapping_max_folio_size(const struct address_space *mapping)return PAGE_SIZE <<mapping_max_folio_order (mapping); } -static inline int filemap_nr_thps (const struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS -return atomic_read (&mapping->nr_thps); -#else -return 0 ; -#endif -} - -static inline void filemap_nr_thps_inc (struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS -if (!mapping_large_folio_support (mapping)) -atomic_inc (&mapping->nr_thps); -#else -WARN_ON_ONCE (mapping_large_folio_support (mapping) ==0 ); -#endif -} - -static inline void filemap_nr_thps_dec (struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS -if (!mapping_large_folio_support (mapping)) -atomic_dec (&mapping->nr_thps); -#else -WARN_ON_ONCE (mapping_large_folio_support (mapping) ==0 ); -#endif -} -struct address_space *folio_mapping (const struct folio *folio);/**
mm/filemap.c +0 -1
@@ -189,7 +189,6 @@ static void filemap_unaccount_folio(struct address_space *mapping,lruvec_stat_mod_folio (folio,NR_SHMEM_THPS , -nr); }else if (folio_test_pmd_mappable (folio)) {lruvec_stat_mod_folio (folio,NR_FILE_THPS , -nr); -filemap_nr_thps_dec (mapping); }if (test_bit (AS_KERNEL_FILE , &folio->mapping->flags)) mod_node_page_state(folio_pgdat(folio),
mm/huge_memory.c +0 -1
@@ -3951,7 +3951,6 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n }else {lruvec_stat_mod_folio (folio,NR_FILE_THPS , -nr); -filemap_nr_thps_dec (mapping); } } }
mm/khugepaged.c +0 -28
@@ -2477,21 +2477,6 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, goto xa_unlocked; } -if (!is_shmem) { -filemap_nr_thps_inc (mapping); -/* -* Paired with the fence in do_dentry_open() -> get_write_access() -* to ensure i_writecount is up to date and the update to nr_thps -* is visible. Ensures the page cache will be truncated if the -* file is opened writable. -*/ -smp_mb (); -if (inode_is_open_for_write (mapping->host)) { - result =SCAN_FAIL ; -filemap_nr_thps_dec (mapping); - } - } - xa_locked:xas_unlock_irq (&xas); xa_unlocked:
@@ -2669,19 +2654,6 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,folio_putback_lru (folio);folio_put (folio); } -/* -* Undo the updates of filemap_nr_thps_inc for non-SHMEM -* file only. This undo is not needed unless failure is -* due to SCAN_COPY_MC. -*/ -if (!is_shmem && result ==SCAN_COPY_MC ) { -filemap_nr_thps_dec (mapping); -/* -* Paired with the fence in do_dentry_open() -> get_write_access() -* to ensure the update to nr_thps is visible. -*/ -smp_mb (); - } new_folio->mapping =NULL ;
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 7/14 fs: remove nr_thps from struct address_space
In reply to Zi Yanfilemap_nr_thps*() are removed, the related field, address_space->nr_thps, is no longer needed. Remove it. This shrinks struct address_space by 8 bytes on 64-bit systems which may increase the number of inodes we can cache.
--- fs/inode.c | 3 --- include/linux/fs.h | 5 ----- 2 files changed, 8 deletions(-)
fs/inode.c +0 -3
@@ -279,9 +279,6 @@ int inode_init_always_gfp(struct super_block *sb, struct inode *inode, gfp_t gfp mapping->flags =0 ; mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 ); -#ifdef CONFIG_READ_ONLY_THP_FOR_FS -atomic_set (&mapping->nr_thps,0 ); -#endif mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->writeback_index =0 ;init_rwsem (&mapping->invalidate_lock);
include/linux/fs.h +0 -5
@@ -460,7 +460,6 @@ struct mapping_metadata_bhs {* memory mappings. * @gfp_mask: Memory allocation flags to use for allocating pages. * @i_mmap_writable: Number of VM_SHARED, VM_MAYWRITE mappings. -* @nr_thps: Number of THPs in the pagecache (non-shmem only). * @i_mmap: Tree of private and shared mappings. * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable. * @nrpages: Number of page entries, protected by the i_pages lock.
@@ -476,10 +475,6 @@ struct address_space {struct rw_semaphore invalidate_lock;gfp_t gfp_mask;atomic_t i_mmap_writable; -#ifdef CONFIG_READ_ONLY_THP_FOR_FS -/* number of thp, only for non-shmem files */ -atomic_t nr_thps; -#endif struct rb_root_cached i_mmap;unsigned long nrpages;pgoff_t writeback_index;
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 8/14 mm/huge_memory: remove folio split check for READ_ONLY_THP_FOR_FS
In reply to Zi YanWithout READ_ONLY_THP_FOR_FS, large file-backed folios cannot be created by a FS without large folio support. The check is no longer needed.
--- mm/huge_memory.c | 30 +++--------------------------- 1 file changed, 3 insertions(+), 27 deletions(-)
mm/huge_memory.c +3 -27
@@ -3846,33 +3846,9 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,if (!folio->mapping && !folio_test_anon (folio))return -EBUSY ; -if (folio_test_anon (folio)) { -/* order-1 is not supported for anonymous THP. */ -if (new_order ==1 ) -return -EINVAL ; - }else if (split_type ==SPLIT_TYPE_NON_UNIFORM || new_order) { -if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && - !mapping_large_folio_support (folio->mapping)) { -/* -* We can always split a folio down to a single page -* (new_order == 0) uniformly. -* -* For any other scenario -* a) uniform split targeting a large folio -* (new_order > 0) -* b) any non-uniform split -* we must confirm that the file system supports large -* folios. -* -* Note that we might still have THPs in such -* mappings, which is created from khugepaged when -* CONFIG_READ_ONLY_THP_FOR_FS is enabled. But in that -* case, the mapping does not actually support large -* folios properly. -*/ -return -EINVAL ; - } - } +/* order-1 is not supported for anonymous THP. */ +if (folio_test_anon (folio) && new_order ==1 ) +return -EINVAL ;/* * swapcache folio could only be split to order 0
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 9/14 mm/truncate: use folio_split() in truncate_inode_partial_folio()
In reply to Zi YanAfter READ_ONLY_THP_FOR_FS is removed, FS either supports large folio or not. folio_split() can be used on a FS with large folio support without worrying about getting a THP on a FS without large folio support.
When READ_ONLY_THP_FOR_FS was present, a PMD large pagecache folio can appear in a FS without large folio support after khugepaged or madvise(MADV_COLLAPSE) creates it. During truncate_inode_partial_folio(), such a PMD large pagecache folio is split and if the FS does not support large folio, it needs to be split to order-0 ones and could not be split non uniformly to ones with various orders. try_folio_split_to_order() was added to handle this situation by checking folio_check_splittable(..., SPLIT_TYPE_NON_UNIFORM) to detect if the large folio is created due to READ_ONLY_THP_FOR_FS and the FS does not support large folio. Now READ_ONLY_THP_FOR_FS is removed, all large pagecache folios are created with FSes supporting large folio, this function is no longer needed and all large pagecache folios can be split non uniformly.
--- include/linux/huge_mm.h | 25 ++----------------------- mm/truncate.c | 8 ++++---- 2 files changed, 6 insertions(+), 27 deletions(-)
include/linux/huge_mm.h +2 -23
@@ -394,27 +394,6 @@ static inline int split_huge_page_to_order(struct page *page, unsigned int new_oreturn split_huge_page_to_list_to_order (page,NULL , new_order); } -/** -* try_folio_split_to_order() - try to split a @folio at @page to @new_order -* using non uniform split. -* @folio: folio to be split -* @page: split to @new_order at the given page -* @new_order: the target split order -* -* Try to split a @folio at @page using non uniform split to @new_order, if -* non uniform split is not supported, fall back to uniform split. After-split -* folios are put back to LRU list. Use min_order_for_split() to get the lower -* bound of @new_order. -* -* Return: 0 - split is successful, otherwise split failed. -*/ -static inline int try_folio_split_to_order (struct folio *folio, -struct page *page,unsigned int new_order) -{ -if (folio_check_splittable (folio, new_order,SPLIT_TYPE_NON_UNIFORM )) -return split_huge_page_to_order (&folio->page, new_order); -return folio_split (folio, new_order, page,NULL ); -}static inline int split_huge_page (struct page *page) {return split_huge_page_to_list_to_order (page,NULL ,0 );
@@ -647,8 +626,8 @@ static inline int split_folio_to_list(struct folio *folio, struct list_head *lisreturn -EINVAL ; } -static inline int try_folio_split_to_order (struct folio *folio, -struct page *page,unsigned int new_order) +static inline int folio_split (struct folio *folio,unsigned int new_order, +struct page *page,struct list_head *list) {VM_WARN_ON_ONCE_FOLIO (1 , folio);return -EINVAL ;
mm/truncate.c +4 -4
@@ -177,7 +177,7 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)return 0 ; } -static int try_folio_split_or_unmap (struct folio *folio,struct page *split_at, +static int folio_split_or_unmap (struct folio *folio,struct page *split_at,unsigned long min_order) { enum ttu_flags ttu_flags =
@@ -186,7 +186,7 @@ static int try_folio_split_or_unmap(struct folio *folio, struct page *split_at,TTU_IGNORE_MLOCK ;int ret; - ret =try_folio_split_to_order (folio, split_at, min_order); + ret =folio_split (folio, min_order, split_at,NULL );/* * If the split fails, unmap the folio, so it will be refaulted
@@ -252,7 +252,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) min_order =mapping_min_folio_order (folio->mapping); split_at =folio_page (folio,PAGE_ALIGN_DOWN (offset) /PAGE_SIZE ); -if (!try_folio_split_or_unmap (folio, split_at, min_order)) { +if (!folio_split_or_unmap (folio, split_at, min_order)) {/* * try to split at offset + length to make sure folios within * the range can be dropped, especially to avoid memory waste
@@ -279,7 +279,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end)/* make sure folio2 is large and does not change its mapping */ if (folio_test_large (folio2) && folio2->mapping == folio->mapping) -try_folio_split_or_unmap (folio2, split_at2, min_order); +folio_split_or_unmap (folio2, split_at2, min_order);folio_unlock (folio2); out:
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 10/14 fs/btrfs: remove a comment referring to READ_ONLY_THP_FOR_FS
In reply to Zi YanREAD_ONLY_THP_FOR_FS is no longer present, remove related comment.
--- fs/btrfs/defrag.c | 3 --- 1 file changed, 3 deletions(-)
fs/btrfs/defrag.c +0 -3
@@ -860,9 +860,6 @@ static struct folio *defrag_prepare_one_folio(struct btrfs_inode *inode, pgoff_treturn folio;/* -* Since we can defragment files opened read-only, we can encounter -* transparent huge pages here (see CONFIG_READ_ONLY_THP_FOR_FS). -* * The IO for such large folios is not fully tested, thus return * an error to reject such folios unless it's an experimental build. *
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 11/14 selftests/mm: remove READ_ONLY_THP_FOR_FS in khugepaged
In reply to Zi YanChange the requirement to a file system with large folio support and the supported order needs to include PMD_ORDER.
Also add tests of opening a file with read write permission and populating folios with writes. Reuse the XFS image from split_huge_page_test.
--- tools/testing/selftests/mm/khugepaged.c | 125 +++++++++++++++------- tools/testing/selftests/mm/run_vmtests.sh | 12 ++- 2 files changed, 96 insertions(+), 41 deletions(-)
tools/testing/selftests/mm/khugepaged.c +86 -39
@@ -49,7 +49,8 @@ struct mem_ops {const char *name; }; -static struct mem_ops *file_ops; +static struct mem_ops *read_only_file_ops; +static struct mem_ops *read_write_file_ops;static struct mem_ops *anon_ops;static struct mem_ops *shmem_ops;
@@ -112,7 +113,8 @@ static void restore_settings(int sig)static void save_settings (void ) {printf ("Save THP and khugepaged settings..." ); -if (file_ops && finfo.type ==VMA_FILE ) +if ((read_only_file_ops || read_write_file_ops) && + finfo.type ==VMA_FILE )thp_set_read_ahead_path (finfo.dev_queue_read_ahead_path);thp_save_settings ();
@@ -364,8 +366,10 @@ static bool anon_check_huge(void *addr, int nr_hpages)return check_huge_anon (addr, nr_hpages, hpage_pmd_size); } -static void *file_setup_area (int nr_hpages) +static void *file_setup_area_common (int nr_hpages,bool read_only) { +const int open_opt = read_only ?O_RDONLY :O_RDWR ; +const int mmap_prot = read_only ?PROT_READ : (PROT_READ |PROT_WRITE );int fd;void *p;unsigned long size;
@@ -400,14 +404,14 @@ static void *file_setup_area(int nr_hpages)munmap (p, size);success ("OK" ); -printf ("Opening %s read only for collapse..." , finfo.path); - finfo.fd =open (finfo.path,O_RDONLY ,777 ); +printf ("Opening %s %s for collapse..." , finfo.path, + read_only ?"read-only" :"read-write" ); + finfo.fd =open (finfo.path, open_opt,777 );if (finfo.fd <0 ) {perror ("open()" );exit (EXIT_FAILURE ); } - p =mmap (BASE_ADDR , size,PROT_READ , -MAP_PRIVATE , finfo.fd,0 ); + p =mmap (BASE_ADDR , size, mmap_prot,MAP_SHARED , finfo.fd,0 );if (p ==MAP_FAILED || p !=BASE_ADDR ) {perror ("mmap()" );exit (EXIT_FAILURE );
@@ -419,6 +423,16 @@ static void *file_setup_area(int nr_hpages)return p; } +static void *file_setup_read_only_area (int nr_hpages) +{ +return file_setup_area_common (nr_hpages,/* read_only= */ true); +} + +static void *file_setup_read_write_area (int nr_hpages) +{ +return file_setup_area_common (nr_hpages,/* read_only= */ false); +} +static void file_cleanup_area (void *p,unsigned long size) {munmap (p, size);
@@ -426,10 +440,18 @@ static void file_cleanup_area(void *p, unsigned long size)unlink (finfo.path); } -static void file_fault (void *p,unsigned long start,unsigned long end) +static void file_fault_read (void *p,unsigned long start,unsigned long end) {if (madvise (((char *)p) + start, end - start,MADV_POPULATE_READ )) { -perror ("madvise(MADV_POPULATE_READ" ); +perror ("madvise(MADV_POPULATE_READ)" ); +exit (EXIT_FAILURE ); + } +} + +static void file_fault_write (void *p,unsigned long start,unsigned long end) +{ +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_WRITE )) { +perror ("madvise(MADV_POPULATE_WRITE)" );exit (EXIT_FAILURE ); } }
@@ -489,10 +511,18 @@ static struct mem_ops __anon_ops = { .name = "anon", }; -static struct mem_ops __file_ops = { - .setup_area = &file_setup_area, +static struct mem_ops __read_only_file_ops = { + .setup_area = &file_setup_read_only_area, .cleanup_area = &file_cleanup_area, - .fault = &file_fault, + .fault = &file_fault_read, + .check_huge = &file_check_huge, + .name ="file" , +}; + +static struct mem_ops __read_write_file_ops = { + .setup_area = &file_setup_read_write_area, + .cleanup_area = &file_cleanup_area, + .fault = &file_fault_write, .check_huge = &file_check_huge, .name ="file" , };
@@ -505,6 +535,17 @@ static struct mem_ops __shmem_ops = { .name = "shmem", }; +static bool is_tmpfs (struct mem_ops *ops) +{ +return (ops == &__read_only_file_ops || ops == &__read_write_file_ops) && + finfo.type ==VMA_SHMEM ; +} + +static bool is_anon (struct mem_ops *ops) +{ +return ops == &__anon_ops; +} +static void __madvise_collapse (const char *msg,char *p,int nr_hpages,struct mem_ops *ops,bool expect) {
@@ -513,6 +554,10 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,printf ("%s..." , msg); +/* read&write file collapse always fail */ +if (!is_tmpfs (ops) && ops == &__read_write_file_ops) + expect = false; +/* * Prevent khugepaged interference and tests that MADV_COLLAPSE * ignores /sys/kernel/mm/transparent_hugepage/enabled
@@ -579,6 +624,10 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages,static void khugepaged_collapse (const char *msg,char *p,int nr_hpages,struct mem_ops *ops,bool expect) { +/* read&write file collapse always fail */ +if (!is_tmpfs (ops) && ops == &__read_write_file_ops) + expect = false; +if (wait_for_scan (msg, p, nr_hpages, ops)) {if (expect)fail ("Timeout" );
@@ -613,16 +662,6 @@ static struct collapse_context __madvise_context = { .name = "madvise", }; -static bool is_tmpfs (struct mem_ops *ops) -{ -return ops == &__file_ops && finfo.type ==VMA_SHMEM ; -} - -static bool is_anon (struct mem_ops *ops) -{ -return ops == &__anon_ops; -} -static void alloc_at_fault (void ) {struct thp_settings settings = *thp_current_settings ();
@@ -1098,8 +1137,8 @@ static void usage(void)fprintf (stderr,"\t<context>\t: [all|khugepaged|madvise]\n" );fprintf (stderr,"\t<mem_type>\t: [all|anon|file|shmem]\n" );fprintf (stderr,"\n\t\"file,all\" mem_type requires [dir] argument\n" ); -fprintf (stderr,"\n\t\"file,all\" mem_type requires kernel built with\n" ); -fprintf (stderr,"\tCONFIG_READ_ONLY_THP_FOR_FS=y\n" ); +fprintf (stderr,"\n\t\"file,all\" mem_type requires a file system\n" ); +fprintf (stderr,"\twith PMD-sized large folio support\n" );fprintf (stderr,"\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n" );fprintf (stderr,"\tmounted with huge=advise option for khugepaged tests to work\n" );fprintf (stderr,"\n\tSupported Options:\n" );
@@ -1155,20 +1194,22 @@ static void parse_test_type(int argc, char **argv)usage ();if (!strcmp (buf,"all" )) { - file_ops = &__file_ops; + read_only_file_ops = &__read_only_file_ops; + read_write_file_ops = &__read_write_file_ops; anon_ops = &__anon_ops; shmem_ops = &__shmem_ops; }else if (!strcmp (buf,"anon" )) { anon_ops = &__anon_ops; }else if (!strcmp (buf,"file" )) { - file_ops = &__file_ops; + read_only_file_ops = &__read_only_file_ops; + read_write_file_ops = &__read_write_file_ops; }else if (!strcmp (buf,"shmem" )) { shmem_ops = &__shmem_ops; }else {usage (); } -if (!file_ops) +if (!read_only_file_ops && !read_write_file_ops)return ;if (argc !=2 )
@@ -1240,37 +1281,43 @@ int main(int argc, char **argv) }while (0 )TEST (collapse_full, khugepaged_context, anon_ops); -TEST (collapse_full, khugepaged_context, file_ops); +TEST (collapse_full, khugepaged_context, read_only_file_ops); +TEST (collapse_full, khugepaged_context, read_write_file_ops);TEST (collapse_full, khugepaged_context, shmem_ops);TEST (collapse_full, madvise_context, anon_ops); -TEST (collapse_full, madvise_context, file_ops); +TEST (collapse_full, madvise_context, read_only_file_ops); +TEST (collapse_full, madvise_context, read_write_file_ops);TEST (collapse_full, madvise_context, shmem_ops);TEST (collapse_empty, khugepaged_context, anon_ops);TEST (collapse_empty, madvise_context, anon_ops);TEST (collapse_single_pte_entry, khugepaged_context, anon_ops); -TEST (collapse_single_pte_entry, khugepaged_context, file_ops); +TEST (collapse_single_pte_entry, khugepaged_context, read_only_file_ops); +TEST (collapse_single_pte_entry, khugepaged_context, read_write_file_ops);TEST (collapse_single_pte_entry, khugepaged_context, shmem_ops);TEST (collapse_single_pte_entry, madvise_context, anon_ops); -TEST (collapse_single_pte_entry, madvise_context, file_ops); +TEST (collapse_single_pte_entry, madvise_context, read_only_file_ops); +TEST (collapse_single_pte_entry, madvise_context, read_write_file_ops);TEST (collapse_single_pte_entry, madvise_context, shmem_ops);TEST (collapse_max_ptes_none, khugepaged_context, anon_ops); -TEST (collapse_max_ptes_none, khugepaged_context, file_ops); +TEST (collapse_max_ptes_none, khugepaged_context, read_only_file_ops); +TEST (collapse_max_ptes_none, khugepaged_context, read_write_file_ops);TEST (collapse_max_ptes_none, madvise_context, anon_ops); -TEST (collapse_max_ptes_none, madvise_context, file_ops); +TEST (collapse_max_ptes_none, madvise_context, read_only_file_ops); +TEST (collapse_max_ptes_none, madvise_context, read_write_file_ops);TEST (collapse_single_pte_entry_compound, khugepaged_context, anon_ops); -TEST (collapse_single_pte_entry_compound, khugepaged_context, file_ops); +TEST (collapse_single_pte_entry_compound, khugepaged_context, read_only_file_ops);TEST (collapse_single_pte_entry_compound, madvise_context, anon_ops); -TEST (collapse_single_pte_entry_compound, madvise_context, file_ops); +TEST (collapse_single_pte_entry_compound, madvise_context, read_only_file_ops);TEST (collapse_full_of_compound, khugepaged_context, anon_ops); -TEST (collapse_full_of_compound, khugepaged_context, file_ops); +TEST (collapse_full_of_compound, khugepaged_context, read_only_file_ops);TEST (collapse_full_of_compound, khugepaged_context, shmem_ops);TEST (collapse_full_of_compound, madvise_context, anon_ops); -TEST (collapse_full_of_compound, madvise_context, file_ops); +TEST (collapse_full_of_compound, madvise_context, read_only_file_ops);TEST (collapse_full_of_compound, madvise_context, shmem_ops);TEST (collapse_compound_extreme, khugepaged_context, anon_ops);
@@ -1292,10 +1339,10 @@ int main(int argc, char **argv)TEST (collapse_max_ptes_shared, madvise_context, anon_ops);TEST (madvise_collapse_existing_thps, madvise_context, anon_ops); -TEST (madvise_collapse_existing_thps, madvise_context, file_ops); +TEST (madvise_collapse_existing_thps, madvise_context, read_only_file_ops);TEST (madvise_collapse_existing_thps, madvise_context, shmem_ops); -TEST (madvise_retracted_page_tables, madvise_context, file_ops); +TEST (madvise_retracted_page_tables, madvise_context, read_only_file_ops);TEST (madvise_retracted_page_tables, madvise_context, shmem_ops);restore_settings (0 );
tools/testing/selftests/mm/run_vmtests.sh +10 -2
@@ -490,8 +490,6 @@ CATEGORY="thp" run_test ./khugepaged all:shmem CATEGORY="thp" run_test ./khugepaged -s 4 all:shmem -CATEGORY="thp" run_test ./transhuge-stress -d 20 - # Try to create XFS if not provided if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then if [ "${HAVE_HUGEPAGES}" = "1" ]; then
@@ -508,6 +506,14 @@ if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then fi fi +if [ -n "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then +CATEGORY="thp" run_test ./khugepaged all:file ${SPLIT_HUGE_PAGE_TEST_XFS_PATH} +elif test_selected thp; then + count_total=$(( count_total + 1 )) + count_skip=$(( count_skip + 1 )) + echo "[SKIP] ./khugepaged all:file" | tap_prefix +fi + CATEGORY="thp" run_test ./split_huge_page_test ${SPLIT_HUGE_PAGE_TEST_XFS_PATH} if [ -n "${MOUNTED_XFS}" ]; then
@@ -516,6 +522,8 @@ if [ -n "${MOUNTED_XFS}" ]; then rm -f ${XFS_IMG} fi +CATEGORY="thp" run_test ./transhuge-stress -d 20 + CATEGORY="thp" run_test ./folio_split_race_test CATEGORY="migration" run_test ./migration
Signature · 1 line
2.53.0
Zi Yan
reviewerPATCH v6 12/14 selftests/mm: remove READ_ONLY_THP_FOR_FS code from guard-regions
In reply to Zi YanAny file system with large folio support and the supported orders include PMD_ORDER can be used. There is no need to open a file with read-only.
--- tools/testing/selftests/mm/guard-regions.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
tools/testing/selftests/mm/guard-regions.c +4 -14
@@ -2203,17 +2203,6 @@ TEST_F(guard_regions, collapse)if (variant->backing !=ANON_BACKED )ASSERT_EQ (ftruncate (self->fd, size),0 ); -/* -* We must close and re-open local-file backed as read-only for -* CONFIG_READ_ONLY_THP_FOR_FS to work. -*/ -if (variant->backing ==LOCAL_FILE_BACKED ) { -ASSERT_EQ (close (self->fd),0 ); - - self->fd =open (self->path,O_RDONLY ); -ASSERT_GE (self->fd,0 ); - } - ptr =mmap_ (self, variant,NULL , size,PROT_READ ,0 ,0 );ASSERT_NE (ptr,MAP_FAILED );
@@ -2237,9 +2226,10 @@ TEST_F(guard_regions, collapse)/* * Now collapse the entire region. This should fail in all cases. * -* The madvise() call will also fail if CONFIG_READ_ONLY_THP_FOR_FS is -* not set for the local file case, but we can't differentiate whether -* this occurred or if the collapse was rightly rejected. +* The madvise() call will also fail if the file system does not support +* large folio or the supported orders do not include PMD_ORDER for the +* local file case, but we can't differentiate whether this occurred or +* if the collapse was rightly rejected. */ EXPECT_NE (madvise (ptr, size,MADV_COLLAPSE ),0 );
Signature · 1 line
2.53.0
Re: [PATCH v6 12/14] selftests/mm: remove READ_ONLY_THP_FOR_FS code from guard-regions
Quoted 25 lines · Zi Yan
Any file system with large folio support and the supported orders include
PMD_ORDERcan be used. There is no need to open a file with read-only.Signed-off-by Zi Yan <[email protected]>Cc Al Viro <[email protected]>Cc Baolin Wang <[email protected]>Cc Barry Song <[email protected]>Cc Chris Mason <[email protected]>Cc Christian Brauner <[email protected]>Cc David Sterba <[email protected]>Cc Dev Jain <[email protected]>Cc Jan Kara <[email protected]>Cc Lance Yang <[email protected]>Cc Liam Howlett <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Michal Hocko <[email protected]>Cc Mike Rapoport <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Shuah Khan <[email protected]>Cc Song Liu <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Vlastimil Babka <[email protected]>
Quoted 1 line
---
Skimmed over it and nothing jumped at me
Signature · 3 lines
Cheers, David
Zi Yan
reviewerPATCH v6 13/14 mm/khugepaged: enable clean pagecache folio collapse for writable files
In reply to Zi Yancollapse_file() is capable of collapsing pagecache folios from writable files to PMD folios. Now enable clean pagecache folio collapse in addition to read-only pagecache folio collapse by removing the inode_is_open_for_write() from file_thp_enabled() and only performing filemap_flush() if the file is read-only.
This means userspace needs to explicitly flush the content of pagecache folios before khugepaged can collapse the folios, or use madvise(MADV_COLLAPSE), which does the flush in the retry. The reason is that blindly enabling dirty pagecache folio from writable files collapse makes khugepaged flush these folios all the time. It is undesirable to cause system level pagecache flushes.
To properly support dirty pagecache folio collapse, filemap_flush() needs to be avoided. Potentially, merging associated buffer instead of dropping it with filemap_release_folio() might be needed.
NOTE: this breaks khugepaged selftests for writable file pagecache collapse, which is set to fail all the time. The next commit fixes it.
--- mm/huge_memory.c | 2 +- mm/khugepaged.c | 15 +++++++++------ 2 files changed, 10 insertions(+), 7 deletions(-)
mm/huge_memory.c +1 -1
@@ -97,7 +97,7 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)if (!mapping_pmd_folio_support (vma->vm_file->f_mapping))return false; -return !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); +return S_ISREG (inode->i_mode); }/* If returns true, we are unable to access the VMA's folios. */
mm/khugepaged.c +9 -6
@@ -2342,18 +2342,21 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, }else if (folio_test_dirty (folio)) {/* * This page is dirty because it hasn't -* been flushed since first write. There -* won't be new dirty pages. +* been flushed since first write. * -* Trigger async flush here and hope the -* writeback is done when khugepaged -* revisits this page. +* Trigger async flush for read-only files and +* hope the writeback is done when khugepaged +* revisits this page. Writable files can have +* their folios dirty at any time; blindly +* flushing them would cause undesirable +* system-wide writeback. * * This is a one-off situation. We are not * forcing writeback in loop. */ xas_unlock_irq (&xas); -filemap_flush (mapping); +if (!inode_is_open_for_write (mapping->host)) +filemap_flush (mapping); result =SCAN_PAGE_DIRTY_OR_WRITEBACK ; goto xa_unlocked; }else if (folio_test_writeback (folio)) {
Signature · 1 line
2.53.0
Re: [PATCH v6 13/14] mm/khugepaged: enable clean pagecache folio collapse for writable files
Quoted 34 lines · Zi Yan
collapse_file()is capable of collapsing pagecache folios from writable files to PMD folios. Now enable clean pagecache folio collapse in addition to read-only pagecache folio collapse by removing theinode_is_open_for_write()fromfile_thp_enabled()and only performingfilemap_flush()if the file is read-only.This means userspace needs to explicitly flush the content of pagecache folios before khugepaged can collapse the folios, or use
madvise(MADV_COLLAPSE), which does the flush in the retry. The reason is that blindly enabling dirty pagecache folio from writable files collapse makes khugepaged flush these folios all the time. It is undesirable to cause system level pagecache flushes.To properly support dirty pagecache folio collapse,
filemap_flush()needs to be avoided. Potentially, merging associated buffer instead of dropping it withfilemap_release_folio()might be needed.NOTE: this breaks khugepaged selftests for writable file pagecache collapse, which is set to fail all the time. The next commit fixes it.
Signed-off-by Zi Yan <[email protected]>Reviewed-by Lance Yang <[email protected]>Cc Al Viro <[email protected]>Cc Baolin Wang <[email protected]>Cc Barry Song <[email protected]>Cc Chris Mason <[email protected]>Cc Christian Brauner <[email protected]>Cc David Hildenbrand (Arm) <[email protected]>Cc David Sterba <[email protected]>Cc Dev Jain <[email protected]>Cc Jan Kara <[email protected]>Cc Liam Howlett <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>
Quoted 8 lines
Cc Michal Hocko <[email protected]>Cc Mike Rapoport <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Shuah Khan <[email protected]>Cc Song Liu <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Vlastimil Babka <[email protected]>
Signature · 3 lines
Cheers, David
Zi Yan
reviewerPATCH v6 14/14 selftests/mm: add writable-file collapse tests for khugepaged
In reply to Zi Yancollapse_file() now supports collapsing clean pagecache folios from writable files, so add corresponding tests.
Note that madvise(MADV_COLLAPSE) works for dirty pagecache folios from writable files, because collapse_single_pmd() triggers a synchronous writeback when first attempt of collapse_file() fails. That writeback makes dirty folios clean and the retry of collapse_file() succeeds.
--- tools/testing/selftests/mm/khugepaged.c | 111 ++++++++++++++++++------ 1 file changed, 85 insertions(+), 26 deletions(-)
tools/testing/selftests/mm/khugepaged.c +85 -26
@@ -41,6 +41,12 @@ enum vma_type { VMA_SHMEM, }; +enum file_setup_ops { +FILE_SETUP_READ_ONLY_FS , +FILE_SETUP_READ_WRITE_FS_READ_DATA , +FILE_SETUP_READ_WRITE_FS_WRITE_DATA , +}; +struct mem_ops {void *(*setup_area)(int nr_hpages);void (*cleanup_area)(void *p,unsigned long size);
@@ -50,7 +56,8 @@ struct mem_ops { };static struct mem_ops *read_only_file_ops; -static struct mem_ops *read_write_file_ops; +static struct mem_ops *read_write_file_read_ops; +static struct mem_ops *read_write_file_write_ops;static struct mem_ops *anon_ops;static struct mem_ops *shmem_ops;
@@ -113,7 +120,8 @@ static void restore_settings(int sig)static void save_settings (void ) {printf ("Save THP and khugepaged settings..." ); -if ((read_only_file_ops || read_write_file_ops) && +if ((read_only_file_ops || read_write_file_read_ops || + read_write_file_write_ops) && finfo.type ==VMA_FILE )thp_set_read_ahead_path (finfo.dev_queue_read_ahead_path);thp_save_settings ();
@@ -366,10 +374,10 @@ static bool anon_check_huge(void *addr, int nr_hpages)return check_huge_anon (addr, nr_hpages, hpage_pmd_size); } -static void *file_setup_area_common (int nr_hpages,bool read_only) +static void *file_setup_area_common (int nr_hpages,enum file_setup_ops setup) { -const int open_opt = read_only ?O_RDONLY :O_RDWR ; -const int mmap_prot = read_only ?PROT_READ : (PROT_READ |PROT_WRITE ); +const int open_opt = setup ==FILE_SETUP_READ_ONLY_FS ?O_RDONLY :O_RDWR ; +const int mmap_prot = setup ==FILE_SETUP_READ_ONLY_FS ?PROT_READ : (PROT_READ |PROT_WRITE );int fd;void *p;unsigned long size;
@@ -405,7 +413,10 @@ static void *file_setup_area_common(int nr_hpages, bool read_only)success ("OK" );printf ("Opening %s %s for collapse..." , finfo.path, - read_only ?"read-only" :"read-write" ); + setup ==FILE_SETUP_READ_ONLY_FS ?"read-only" : + setup ==FILE_SETUP_READ_WRITE_FS_READ_DATA ? +"read-write (read)" : +"read-write (write)" ); finfo.fd =open (finfo.path, open_opt,777 );if (finfo.fd <0 ) {perror ("open()" );
@@ -425,12 +436,17 @@ static void *file_setup_area_common(int nr_hpages, bool read_only)static void *file_setup_read_only_area (int nr_hpages) { -return file_setup_area_common (nr_hpages,/* read_only= */ true); +return file_setup_area_common (nr_hpages,FILE_SETUP_READ_ONLY_FS ); +} + +static void *file_setup_read_write_fs_read_area (int nr_hpages) +{ +return file_setup_area_common (nr_hpages,FILE_SETUP_READ_WRITE_FS_READ_DATA ); } -static void *file_setup_read_write_area (int nr_hpages) +static void *file_setup_read_write_fs_write_area (int nr_hpages) { -return file_setup_area_common (nr_hpages,/* read_only= */ false); +return file_setup_area_common (nr_hpages,FILE_SETUP_READ_WRITE_FS_WRITE_DATA ); }static void file_cleanup_area (void *p,unsigned long size)
@@ -448,6 +464,16 @@ static void file_fault_read(void *p, unsigned long start, unsigned long end) } } +static void file_fault_read_and_flush (void *p,unsigned long start,unsigned long end) +{ +file_fault_read (p, start, end); +/* +* make folio clean, since dirty folios from read&write file are +* rejected and not flushed +*/ +msync ((char *)p + start, end - start,MS_SYNC ); +} +static void file_fault_write (void *p,unsigned long start,unsigned long end) {if (madvise (((char *)p) + start, end - start,MADV_POPULATE_WRITE )) {
@@ -519,8 +545,16 @@ static struct mem_ops __read_only_file_ops = { .name = "file", }; -static struct mem_ops __read_write_file_ops = { - .setup_area = &file_setup_read_write_area, +static struct mem_ops __read_write_file_read_ops = { + .setup_area = &file_setup_read_write_fs_read_area, + .cleanup_area = &file_cleanup_area, + .fault = &file_fault_read_and_flush, + .check_huge = &file_check_huge, + .name ="file" , +}; + +static struct mem_ops __read_write_file_write_ops = { + .setup_area = &file_setup_read_write_fs_write_area, .cleanup_area = &file_cleanup_area, .fault = &file_fault_write, .check_huge = &file_check_huge,
@@ -537,7 +571,9 @@ static struct mem_ops __shmem_ops = {static bool is_tmpfs (struct mem_ops *ops) { -return (ops == &__read_only_file_ops || ops == &__read_write_file_ops) && +return (ops == &__read_only_file_ops || + ops == &__read_write_file_read_ops || + ops == &__read_write_file_write_ops) && finfo.type ==VMA_SHMEM ; }
@@ -554,9 +590,11 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,printf ("%s..." , msg); -/* read&write file collapse always fail */ -if (!is_tmpfs (ops) && ops == &__read_write_file_ops) - expect = false; +/* +* read&write file collapse succeeds for MADV_COLLAPSE because dirty +* folios are written back after collapse fails for dirty folios and +* another collapse is attempted. +*/ /* * Prevent khugepaged interference and tests that MADV_COLLAPSE
@@ -624,8 +662,11 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages,static void khugepaged_collapse (const char *msg,char *p,int nr_hpages,struct mem_ops *ops,bool expect) { -/* read&write file collapse always fail */ -if (!is_tmpfs (ops) && ops == &__read_write_file_ops) +/* +* read&write file collapse fails since khugepaged does not flush +* the target dirty folios +*/ +if (!is_tmpfs (ops) && ops == &__read_write_file_write_ops) expect = false;if (wait_for_scan (msg, p, nr_hpages, ops)) {
@@ -748,6 +789,9 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *ovalidate_memory (p,0 , (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size);if (c->enforce_pte_scan_limits) { + ops->cleanup_area (p, hpage_pmd_size); + p = ops->setup_area (1 ); + ops->fault (p,0 , (hpage_pmd_nr - max_ptes_none) * page_size); c->collapse ("Collapse with max_ptes_none PTEs empty" , p,1 , ops, true);
@@ -1195,21 +1239,24 @@ static void parse_test_type(int argc, char **argv)if (!strcmp (buf,"all" )) { read_only_file_ops = &__read_only_file_ops; - read_write_file_ops = &__read_write_file_ops; + read_write_file_read_ops = &__read_write_file_read_ops; + read_write_file_write_ops = &__read_write_file_write_ops; anon_ops = &__anon_ops; shmem_ops = &__shmem_ops; }else if (!strcmp (buf,"anon" )) { anon_ops = &__anon_ops; }else if (!strcmp (buf,"file" )) { read_only_file_ops = &__read_only_file_ops; - read_write_file_ops = &__read_write_file_ops; + read_write_file_read_ops = &__read_write_file_read_ops; + read_write_file_write_ops = &__read_write_file_write_ops; }else if (!strcmp (buf,"shmem" )) { shmem_ops = &__shmem_ops; }else {usage (); } -if (!read_only_file_ops && !read_write_file_ops) +if (!read_only_file_ops && !read_write_file_read_ops && + !read_write_file_write_ops)return ;if (argc !=2 )
@@ -1282,11 +1329,13 @@ int main(int argc, char **argv)TEST (collapse_full, khugepaged_context, anon_ops);TEST (collapse_full, khugepaged_context, read_only_file_ops); -TEST (collapse_full, khugepaged_context, read_write_file_ops); +TEST (collapse_full, khugepaged_context, read_write_file_read_ops); +TEST (collapse_full, khugepaged_context, read_write_file_write_ops);TEST (collapse_full, khugepaged_context, shmem_ops);TEST (collapse_full, madvise_context, anon_ops);TEST (collapse_full, madvise_context, read_only_file_ops); -TEST (collapse_full, madvise_context, read_write_file_ops); +TEST (collapse_full, madvise_context, read_write_file_read_ops); +TEST (collapse_full, madvise_context, read_write_file_write_ops);TEST (collapse_full, madvise_context, shmem_ops);TEST (collapse_empty, khugepaged_context, anon_ops);
@@ -1294,30 +1343,38 @@ int main(int argc, char **argv)TEST (collapse_single_pte_entry, khugepaged_context, anon_ops);TEST (collapse_single_pte_entry, khugepaged_context, read_only_file_ops); -TEST (collapse_single_pte_entry, khugepaged_context, read_write_file_ops); +TEST (collapse_single_pte_entry, khugepaged_context, read_write_file_read_ops); +TEST (collapse_single_pte_entry, khugepaged_context, read_write_file_write_ops);TEST (collapse_single_pte_entry, khugepaged_context, shmem_ops);TEST (collapse_single_pte_entry, madvise_context, anon_ops);TEST (collapse_single_pte_entry, madvise_context, read_only_file_ops); -TEST (collapse_single_pte_entry, madvise_context, read_write_file_ops); +TEST (collapse_single_pte_entry, madvise_context, read_write_file_read_ops); +TEST (collapse_single_pte_entry, madvise_context, read_write_file_write_ops);TEST (collapse_single_pte_entry, madvise_context, shmem_ops);TEST (collapse_max_ptes_none, khugepaged_context, anon_ops);TEST (collapse_max_ptes_none, khugepaged_context, read_only_file_ops); -TEST (collapse_max_ptes_none, khugepaged_context, read_write_file_ops); +TEST (collapse_max_ptes_none, khugepaged_context, read_write_file_read_ops); +TEST (collapse_max_ptes_none, khugepaged_context, read_write_file_write_ops);TEST (collapse_max_ptes_none, madvise_context, anon_ops);TEST (collapse_max_ptes_none, madvise_context, read_only_file_ops); -TEST (collapse_max_ptes_none, madvise_context, read_write_file_ops); +TEST (collapse_max_ptes_none, madvise_context, read_write_file_read_ops); +TEST (collapse_max_ptes_none, madvise_context, read_write_file_write_ops);TEST (collapse_single_pte_entry_compound, khugepaged_context, anon_ops);TEST (collapse_single_pte_entry_compound, khugepaged_context, read_only_file_ops); +TEST (collapse_single_pte_entry_compound, khugepaged_context, read_write_file_read_ops);TEST (collapse_single_pte_entry_compound, madvise_context, anon_ops);TEST (collapse_single_pte_entry_compound, madvise_context, read_only_file_ops); +TEST (collapse_single_pte_entry_compound, madvise_context, read_write_file_read_ops);TEST (collapse_full_of_compound, khugepaged_context, anon_ops);TEST (collapse_full_of_compound, khugepaged_context, read_only_file_ops); +TEST (collapse_full_of_compound, khugepaged_context, read_write_file_read_ops);TEST (collapse_full_of_compound, khugepaged_context, shmem_ops);TEST (collapse_full_of_compound, madvise_context, anon_ops);TEST (collapse_full_of_compound, madvise_context, read_only_file_ops); +TEST (collapse_full_of_compound, madvise_context, read_write_file_read_ops);TEST (collapse_full_of_compound, madvise_context, shmem_ops);TEST (collapse_compound_extreme, khugepaged_context, anon_ops);
@@ -1340,9 +1397,11 @@ int main(int argc, char **argv)TEST (madvise_collapse_existing_thps, madvise_context, anon_ops);TEST (madvise_collapse_existing_thps, madvise_context, read_only_file_ops); +TEST (madvise_collapse_existing_thps, madvise_context, read_write_file_read_ops);TEST (madvise_collapse_existing_thps, madvise_context, shmem_ops);TEST (madvise_retracted_page_tables, madvise_context, read_only_file_ops); +TEST (madvise_retracted_page_tables, madvise_context, read_write_file_read_ops);TEST (madvise_retracted_page_tables, madvise_context, shmem_ops);restore_settings (0 );
Signature · 1 line
2.53.0
Re: [PATCH v6 14/14] selftests/mm: add writable-file collapse tests for khugepaged
Quoted 30 lines · Zi Yan
collapse_file()now supports collapsing clean pagecache folios from writable files, so add corresponding tests.Note that
madvise(MADV_COLLAPSE)works for dirty pagecache folios from writable files, becausecollapse_single_pmd()triggers a synchronous writeback when first attempt ofcollapse_file()fails. That writeback makes dirty folios clean and the retry ofcollapse_file()succeeds.Signed-off-by Zi Yan <[email protected]>Cc Al Viro <[email protected]>Cc Baolin Wang <[email protected]>Cc Barry Song <[email protected]>Cc Chris Mason <[email protected]>Cc Christian Brauner <[email protected]>Cc David Hildenbrand (Arm) <[email protected]>Cc David Sterba <[email protected]>Cc Dev Jain <[email protected]>Cc Jan Kara <[email protected]>Cc Lance Yang <[email protected]>Cc Liam Howlett <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Michal Hocko <[email protected]>Cc Mike Rapoport <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Shuah Khan <[email protected]>Cc Song Liu <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Vlastimil Babka <[email protected]>
Quoted 1 line
---
Signature · 3 lines
Cheers, David
Andrew Morton
maintainerQuoted 5 lines · On Sun, 17 May 2026 09:54:02 -0400 Zi Yan <[email protected]> wrote:
Hi all,
This patchset removes
READ_ONLY_THP_FOR_FSKconfigand enables creating file-backed THPs for FSes with large folio support (the supported orders need to includePMD_ORDER) by default, including for writable files.
Cool. Sashiko wasn't able to apply this (presumably because of Mike's CI-friendly series). I take it that the AI review from v5
(https://sashiko.dev/#/patchset/[email protected])
was considered?
Also, please check that the below were considered:
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
Quoted 4 lines
is an in-place replacement of V5 in mm-new. It affects Mike Rapoport's "make MM selftests more CI friendly", since "selftests/mm: khugepaged:
use kselftest framework" needs to be updated. I updated it and put it at the end of this cover letter.
Helpful, thanks. It was a little complicated because your email client messes with whitespace (it always has!), but I figured it out.
Quoted 17 lines
Changelog
===Quoted 1 line
From V5[6]:
1. added
mapping_min_folio_order(mapping)<=PMD_ORDERcheck tomapping_pmd_folio_support()in Patch 1 to correctly handle filesystems whose minimum folio order exceedsPMD_ORDER. Also improved the kernel-doc comment per David's suggestions.2. cleaned up Patch 11 per David's review: use const for
open_optandmmap_prot, removemmap_opt(useMAP_SHAREDfor both read-only and read-write mappings), inlinefile_fault_common()into separatefile_fault_read()andfile_fault_write()functions, fix "read only" typo to "read-only", update usage message to "with PMD-sized large folio support". Also fixedrun_vmtests.shto use eliftest_selectedthp for the SKIP case to avoid spurious [SKIP] output per Nico's report.
Quoted 5 lines
3. revised stale comment in Patch 13: removed "There won't be new dirty pages" and updated "khugepaged only works on read-only fd" to reflect that writable files are now supported; merged the comment blocks per David's suggestion.
Here's how v6 altered mm.git:
include/linux/pagemap.h | 12 +++---- mm/khugepaged.c | 18 ++++------- tools/testing/selftests/mm/khugepaged.c | 35 ++++++++-------------- 3 files changed, 26 insertions(+), 39 deletions(-)
include/linux/pagemap.h renamed from include/linux/pagemap.h~b +6 -6
@@ -514,15 +514,15 @@ static inline bool mapping_large_folio_s }/** -* mapping_pmd_folio_support() - Check if a mapping support PMD-sized folio +* mapping_pmd_folio_support() - Check if a mapping supports PMD-sized folio * @mapping: The address_space * -* Some file supports large folio but does not support as large as PMD order. -* If a PMD-sized pagecache folio is attempted to be created on a filesystem, -* this check needs to be performed first. +* While some mappings support large folios, they might not support PMD-sized +* folios. This function checks whether a mapping supports PMD-sized folios. +* For example, khugepaged needs this information before attempting to +* collapsing THPs. * -* Return: true - PMD-sized folio is supported, false - PMD-sized folio is not -* supported. +* Return: True if PMD-sized folios are supported, otherwise false. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline bool mapping_pmd_folio_support (const struct address_space *mapping)
mm/khugepaged.c renamed from mm/khugepaged.c~b +7 -11
@@ -2342,23 +2342,19 @@ static enum scan_result collapse_file(st }else if (folio_test_dirty (folio)) {/* * This page is dirty because it hasn't -* been flushed since first write. There -* won't be new dirty pages. +* been flushed since first write. * -* Trigger async flush here and hope the -* writeback is done when khugepaged -* revisits this page. +* Trigger async flush for read-only files and +* hope the writeback is done when khugepaged +* revisits this page. Writable files can have +* their folios dirty at any time; blindly +* flushing them would cause undesirable +* system-wide writeback. * * This is a one-off situation. We are not * forcing writeback in loop. */ xas_unlock_irq (&xas); -/* -* Only flush for read-only files. Writable -* files can have their folios dirty at any -* time; blindly flushing them would cause -* undesirable system-wide writeback. -*/ if (!inode_is_open_for_write (mapping->host))filemap_flush (mapping); result =SCAN_PAGE_DIRTY_OR_WRITEBACK ;
tools/testing/selftests/mm/khugepaged.c renamed from tools/testing/selftests/mm/khugepaged.c~b +13 -22
@@ -376,12 +376,11 @@ static bool anon_check_huge(void *addr,static void *file_setup_area_common (int nr_hpages,enum file_setup_ops setup) { +const int open_opt = setup ==FILE_SETUP_READ_ONLY_FS ?O_RDONLY :O_RDWR ; +const int mmap_prot = setup ==FILE_SETUP_READ_ONLY_FS ?PROT_READ : (PROT_READ |PROT_WRITE );int fd;void *p;unsigned long size; -int open_opt = setup ==FILE_SETUP_READ_ONLY_FS ?O_RDONLY :O_RDWR ; -int mmap_prot = setup ==FILE_SETUP_READ_ONLY_FS ?PROT_READ : (PROT_READ |PROT_WRITE ); -int mmap_opt = setup ==FILE_SETUP_READ_ONLY_FS ?MAP_PRIVATE :MAP_SHARED ;unlink (finfo.path);/* Cleanup from previous failed tests */ printf("Creating %s for collapse%s...", finfo.path,
@@ -414,7 +413,7 @@ static void *file_setup_area_common(intsuccess ("OK" );printf ("Opening %s %s for collapse..." , finfo.path, - setup ==FILE_SETUP_READ_ONLY_FS ?"read only" : + setup ==FILE_SETUP_READ_ONLY_FS ?"read-only" : setup ==FILE_SETUP_READ_WRITE_FS_READ_DATA ?"read-write (read)" :"read-write (write)" );
@@ -423,8 +422,7 @@ static void *file_setup_area_common(intperror ("open()" );exit (EXIT_FAILURE ); } - p =mmap (BASE_ADDR , size, mmap_prot, - mmap_opt, finfo.fd,0 ); + p =mmap (BASE_ADDR , size, mmap_prot,MAP_SHARED , finfo.fd,0 );if (p ==MAP_FAILED || p !=BASE_ADDR ) {perror ("mmap()" );exit (EXIT_FAILURE );
@@ -458,27 +456,17 @@ static void file_cleanup_area(void *p, uunlink (finfo.path); } -static void file_fault_common (void *p,unsigned long start,unsigned long end, -int madv_ops) +static void file_fault_read (void *p,unsigned long start,unsigned long end) { -if (madvise (((char *)p) + start, end - start, madv_ops)) { -if (madv_ops ==MADV_POPULATE_READ ) -perror ("madvise(MADV_POPULATE_READ" ); -else if (madv_ops ==MADV_POPULATE_WRITE ) -perror ("madvise(MADV_POPULATE_WRITE" ); +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_READ )) { +perror ("madvise(MADV_POPULATE_READ)" );exit (EXIT_FAILURE ); } } -static void file_fault_read (void *p,unsigned long start,unsigned long end) -{ -file_fault_common (p, start, end,MADV_POPULATE_READ ); -} -static void file_fault_read_and_flush (void *p,unsigned long start,unsigned long end) { -file_fault_common (p, start, end,MADV_POPULATE_READ ); - +file_fault_read (p, start, end);/* * make folio clean, since dirty folios from read&write file are * rejected and not flushed
@@ -488,7 +476,10 @@ static void file_fault_read_and_flush(vostatic void file_fault_write (void *p,unsigned long start,unsigned long end) { -file_fault_common (p, start, end,MADV_POPULATE_WRITE ); +if (madvise (((char *)p) + start, end - start,MADV_POPULATE_WRITE )) { +perror ("madvise(MADV_POPULATE_WRITE)" ); +exit (EXIT_FAILURE ); + } }static bool file_check_huge (void *addr,int nr_hpages)
@@ -1191,7 +1182,7 @@ static void usage(void)fprintf (stderr,"\t<mem_type>\t: [all|anon|file|shmem]\n" );fprintf (stderr,"\n\t\"file,all\" mem_type requires [dir] argument\n" );fprintf (stderr,"\n\t\"file,all\" mem_type requires a file system\n" ); -fprintf (stderr,"\twith large folio support (order >= PMD order)\n" ); +fprintf (stderr,"\twith PMD-sized large folio support\n" );fprintf (stderr,"\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n" );fprintf (stderr,"\tmounted with huge=advise option for khugepaged tests to work\n" );fprintf (stderr,"\n\tSupported Options:\n" );
_
Zi Yan
reviewerQuoted 15 lines · Andrew Morton
Quoted 5 lines · On Sun, 17 May 2026 09:54:02 -0400 Zi Yan <[email protected]> wrote:
Hi all,
This patchset removes
READ_ONLY_THP_FOR_FSKconfigand enables creating file-backed THPs for FSes with large folio support (the supported orders need to includePMD_ORDER) by default, including for writable files.Cool. Sashiko wasn't able to apply this (presumably because of Mike's CI-friendly series). I take it that the AI review from v5
(https://sashiko.dev/#/patchset/[email protected])
was considered?Also, please check that the below were considered:
Quoted 8 lines
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
I addressed all of them, except the second one and I will reply to the second one soon.
Quoted 8 lines
Quoted 5 lines
is an in-place replacement of V5 in mm-new. It affects Mike Rapoport's "make MM selftests more CI friendly", since "selftests/mm: khugepaged:
use kselftest framework" needs to be updated. I updated it and put it at the end of this cover letter.Helpful, thanks. It was a little complicated because your email client messes with whitespace (it always has!), but I figured it out.
You mean tabs become whitespace? And does all my patches have the whitespace issue or the one I put at the bottom? I am trying to figure out the issue. BTW, All patches are sent via git send-email using mstmp and
I copied the bottom one from git format-patch. I wonder which part went wrong.
Thanks.
Best Regards,
Yan, Zi
Andrew Morton
maintainerQuoted 22 lines · On Tue, 19 May 2026 07:39:07 +0800 Zi Yan <[email protected]> wrote:
Quoted 20 lines · Andrew Morton
Quoted 5 lines · On Sun, 17 May 2026 09:54:02 -0400 Zi Yan <[email protected]> wrote:
Hi all,
This patchset removes
READ_ONLY_THP_FOR_FSKconfigand enables creating file-backed THPs for FSes with large folio support (the supported orders need to includePMD_ORDER) by default, including for writable files.Cool. Sashiko wasn't able to apply this (presumably because of Mike's CI-friendly series). I take it that the AI review from v5
(https://sashiko.dev/#/patchset/[email protected])
was considered?Also, please check that the below were considered:
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
Quoted 6 lines
I addressed all of them, except the second one and I will reply to the second one soon.
Great.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
is an in-place replacement of V5 in mm-new. It affects Mike Rapoport's "make MM selftests more CI friendly", since "selftests/mm: khugepaged:
use kselftest framework" needs to be updated. I updated it and put it at the end of this cover letter.
Quoted 8 lines
Quoted 2 lines
Helpful, thanks. It was a little complicated because your email client messes with whitespace (it always has!), but I figured it out.
You mean tabs become whitespace? And does all my patches have the whitespace issue or the one I put at the bottom? I am trying to figure out the issue. BTW, All patches are sent via git send-email using mstmp and
I copied the bottom one from git format-patch. I wonder which part went wrong.
Your regular patches are OK - but when you informally paste a patch into the message body the tabs turn into whitespace.
You aren't the only one ;) I usually don't comment...
Zi Yan
reviewerQuoted 42 lines · Andrew Morton
Quoted 22 lines · On Tue, 19 May 2026 07:39:07 +0800 Zi Yan <[email protected]> wrote:
Quoted 20 lines · Andrew Morton
Quoted 5 lines · On Sun, 17 May 2026 09:54:02 -0400 Zi Yan <[email protected]> wrote:
Hi all,
This patchset removes
READ_ONLY_THP_FOR_FSKconfigand enables creating file-backed THPs for FSes with large folio support (the supported orders need to includePMD_ORDER) by default, including for writable files.Cool. Sashiko wasn't able to apply this (presumably because of Mike's CI-friendly series). I take it that the AI review from v5
(https://sashiko.dev/#/patchset/[email protected])
was considered?Also, please check that the below were considered:
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]
https://lore.kernel.org/[email protected]Quoted 6 lines
I addressed all of them, except the second one and I will reply to the second one soon.
Great.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
is an in-place replacement of V5 in mm-new. It affects Mike Rapoport's "make MM selftests more CI friendly", since "selftests/mm: khugepaged:
use kselftest framework" needs to be updated. I updated it and put it at the end of this cover letter.Helpful, thanks. It was a little complicated because your email client messes with whitespace (it always has!), but I figured it out.
Quoted 8 lines
Quoted 5 lines
You mean tabs become whitespace? And does all my patches have the whitespace issue or the one I put at the bottom? I am trying to figure out the issue. BTW, All patches are sent via git send-email using mstmp and
I copied the bottom one from git format-patch. I wonder which part went wrong.Your regular patches are OK - but when you informally paste a patch into the message body the tabs turn into whitespace.
I did that before, since I was copying diffs directly from a GUI terminal that converts tabs to whitespaces. Since you mentioned it last time, I always do git format-patch and copy the content from the patch file and it should work.
For the patch I pasted in this cover letter, I copied it from the patch file directly using neovim, but somehow tabs from surrounding code are converted to whitespaces (tabs from +/- are not). I noticed it when I checked my original cover letter just now. So my email client does not cause the issue. Anyway, I will be more careful next time. Sorry for the trouble.
Quoted 2 lines
You aren't the only one ;) I usually don't comment...
Thank you for reminding me of the issue.
Best Regards,
Yan, Zi