add shmem mTHP collapse support
Subsystems (7)
Baolin Wang
reviewerPATCH 2/12 mm: khugepaged: drop the hugepage_madvise()
In reply to Baolin WangAfter commit 2fd4e7693674 ("mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs"), hugepage_madvise() no longer calls any khugepaged related functions and only updates the VMA flags. So remove this helper and update the VMA flags directly in madvise_update_vma(), which simplifies the code.
--- include/linux/huge_mm.h | 8 -------- mm/khugepaged.c | 22 ---------------------- mm/madvise.c | 8 +++++--- 3 files changed, 5 insertions(+), 33 deletions(-)
include/linux/huge_mm.h +0 -8
@@ -508,8 +508,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,__split_huge_pud (__vma, __pud, __address); \ }while (0 ) -int hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags, - int advice); void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct vm_area_struct *next);spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma);
@@ -707,12 +705,6 @@ static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma,#define split_huge_pud (__vma, __pmd, __address) \ do { } while (0) -static inline int hugepage_madvise (struct vm_area_struct *vma, -vm_flags_t *vm_flags,int advice) -{ -return -EINVAL ; -} - static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end,
mm/khugepaged.c +0 -22
@@ -380,28 +380,6 @@ static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,return cc->policy.max_ptes_swap; } -int hugepage_madvise (struct vm_area_struct *vma, -vm_flags_t *vm_flags,int advice) -{ -switch (advice) { -case MADV_HUGEPAGE : - *vm_flags &= ~VM_NOHUGEPAGE ; - *vm_flags |=VM_HUGEPAGE ; -break ; -case MADV_NOHUGEPAGE : - *vm_flags &= ~VM_HUGEPAGE ; - *vm_flags |=VM_NOHUGEPAGE ; -/* -* Setting VM_NOHUGEPAGE will prevent khugepaged from scanning -* this vma even if we leave the mm registered in khugepaged if -* it got registered before VM_NOHUGEPAGE was set. -*/ -break ; - } - -return 0 ; -} -int __init khugepaged_init (void ) { mm_slot_cache =KMEM_CACHE (mm_slot,0 );
mm/madvise.c +5 -3
@@ -1598,10 +1598,12 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior) goto out;break ;case MADV_HUGEPAGE : + new_flags &= ~VM_NOHUGEPAGE ; + new_flags |=VM_HUGEPAGE ; +break ;case MADV_NOHUGEPAGE : - error =hugepage_madvise (vma, &new_flags, behavior); -if (error) - goto out; + new_flags &= ~VM_HUGEPAGE ; + new_flags |=VM_NOHUGEPAGE ;break ;case __MADV_SET_ANON_VMA_NAME:/* Only anonymous mappings can be named */
Signature · 1 line
2.47.3
Baolin Wang
reviewercover add shmem mTHP collapse support
Based on mm-unstable branch.
This patchset is split from the previous shmem mTHP collapse patchset [1], as the anon mTHP collapse selftests [2] have already been merged.
This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse, which is based on Nico's patchset[3].
The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
track present pages via a bitmap while scanning PMD ranges for collapse candidates, then use the bitmap after the scan completes to determine the most efficient mTHP order to collapse to. Built on the basic framework added for anonymous mTHP collapse, the shmem mTHP collapse implementation is straightforward (Thanks for Nico's work).
Note: I have not yet enabled large order collapse for file folios (file folios currently only support PMD-sized large folio collapse). Although file large order collapse would be more straightforward to implement after shmem mTHP collapse support is added (requiring some changes to file_thp_enabled()), I think this still need some discussion on whether it is necessary to support other large orders collapse for file folios.
Patch 1-4: some cleanups.
Patch 5-8: preparation patches.
Patch 9-11: shmem mTHP collapse support.
Patch 12: add selftests.
Comments are welcome. Thanks.
Changes from RFC v2:
- Rebase on the new mm-unstable branch.
- Add some cleanup patches.
- Drop anon mTHP collapse selftests patches as they have been merged.
- Record swap entries when calling mthp_collapse() for shmem (per Lance).
- Return SCAN_PTE_MAPPED_HUGEPAGE if page cache already holds a PMD-sized folio (per Lance).
- Allow collapse order 1 folios for shmem (per Lance).
Changes from RFC v1:
https://lore.kernel.org/all/[email protected]/
- Rebase on the new code, and update to use the new functions.
- Add more test cases.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Baolin Wang (12):
mm: khugepaged: remove outdated comments and check for shmem
mm: khugepaged: drop the hugepage_madvise()
mm: shmem: don't expose 'deny' and 'force' via the kernel command line
mm: khugepaged: move cc->scan_file and cc->scan_pgoff assignment into
collapse_scan_file()
mm: khugepaged: add max_ptes_none check in collapse_file()
mm: khugepaged: generalize collapse_file() for shmem mTHP support
mm: khugepaged: add an order check for PMD-sized THP statistics
mm: khugepaged: add a helper to get the minimal collapse order
mm: khugepaged: add shmem mTHP collapse support
mm: shmem: run khugepaged for all shmem mTHP orders
mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
selftests: mm: add shmem mTHP collapse test cases
Documentation/admin-guide/mm/transhuge.rst | 4 +- include/linux/huge_mm.h | 8 - include/linux/shmem_fs.h | 4 +- mm/collapse.h | 3 +- mm/khugepaged.c | 220 +++++++++++++-------- mm/madvise.c | 8 +- mm/shmem.c | 22 +-- tools/testing/selftests/mm/khugepaged.c | 7 +- tools/testing/selftests/mm/run_vmtests.sh | 2 + 9 files changed, 168 insertions(+), 110 deletions(-)
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 1/12 mm: khugepaged: remove outdated comments and check for shmem
After commit eb92d97f7e6a ("Revert 'tmpfs: don't enable large folios if not supported'"), all shmem mounts can support large folios unconditionally, so drop the outdated comments and the check for shmem in collapse_file().
--- mm/khugepaged.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
mm/khugepaged.c +2 -5
@@ -2192,12 +2192,9 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,bool is_shmem =shmem_file (file);/* -* MADV_COLLAPSE ignores shmem huge config, so do not check shmem -* -* TODO: once shmem always calls mapping_set_large_folios() on its -* mapping, the shmem check can be removed. +* MADV_COLLAPSE ignores shmem huge config, so do not check shmem. */ -VM_WARN_ON_ONCE (!is_shmem && !mapping_pmd_folio_support (mapping)); +VM_WARN_ON_ONCE (!mapping_pmd_folio_support (mapping));VM_WARN_ON_ONCE (start & (HPAGE_PMD_NR -1 )); result =alloc_charge_folio (&new_folio, mm, cc,HPAGE_PMD_ORDER );
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 7/12 mm: khugepaged: add an order check for PMD-sized THP statistics
In reply to Baolin WangIn order to support shmem mTHP collapse in the following patches, add an PMD-sized THP order check to avoid PMD-sized THP statistics errors.
No functional changes.
--- mm/khugepaged.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/khugepaged.c +3 -2
@@ -2518,8 +2518,9 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,if (is_shmem) {lruvec_stat_mod_folio (new_folio,NR_SHMEM , nr_pages); -lruvec_stat_mod_folio (new_folio,NR_SHMEM_THPS ,HPAGE_PMD_NR ); - }else { +if (is_pmd_order (order)) +lruvec_stat_mod_folio (new_folio,NR_SHMEM_THPS ,HPAGE_PMD_NR ); + }else if (is_pmd_order (order)) {lruvec_stat_mod_folio (new_folio,NR_FILE_THPS ,HPAGE_PMD_NR ); }lruvec_stat_mod_folio (new_folio,NR_FILE_PAGES , nr_pages);
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 8/12 mm: khugepaged: add a helper to get the minimal collapse order
In reply to Baolin Wangshmem mTHP collapse will need to support order-1 collapse in the future, so add a helper to retrieve the minimal collapse order as a preparation. Also rename KHUGEPAGED_MIN_MTHP_ORDER to KHUGEPAGED_MIN_ANON_MTHP_ORDER for better readability, and add a new COLLAPSE_MIN_FILE_MTHP_ORDER for the shmem minimal collapse order.
--- mm/collapse.h | 3 ++- mm/khugepaged.c | 10 +++++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
mm/collapse.h +2 -1
@@ -8,7 +8,8 @@#include <linux/types.h> #define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) -#define COLLAPSE_MIN_MTHP_ORDER 2 +#define COLLAPSE_MIN_ANON_MTHP_ORDER 2 +#define COLLAPSE_MIN_FILE_MTHP_ORDER 1enum scan_result {SCAN_FAIL ,
mm/khugepaged.c +9 -1
@@ -302,6 +302,14 @@ static bool pte_none_or_zero(pte_t pte)return pte_present (pte) &&is_zero_pfn (pte_pfn (pte)); } +static unsigned int collapse_min_mthp_order (struct file *file) +{ +if (file) +return COLLAPSE_MIN_FILE_MTHP_ORDER ; + +return COLLAPSE_MIN_ANON_MTHP_ORDER ; +} +/** * collapse_max_ptes_none - Calculate maximum allowed empty PTEs or PTEs mapping * the shared zeropage for the given collapse operation.
@@ -1485,7 +1493,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addres* any smaller order enabled. When at the smallest order * we must always move to the next offset. */ -if (order >COLLAPSE_MIN_MTHP_ORDER && +if (order >collapse_min_mthp_order (NULL ) && (cc->scan_orders &GENMASK (order -1 ,0 ))) { order--;continue ;
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 3/12 mm: shmem: don't expose 'deny' and 'force' via the kernel command line
In reply to Baolin WangBased on the previous discussion with Lorenzo[1], the 'deny' and 'force' options are testing-only configurations and shouldn't be exposed via the shmem kernel command line.
So drop these debug options from the shmem kernel command line and update the documentation to make this clear.
[1] https://lore.kernel.org/all/[email protected]/
--- Documentation/admin-guide/mm/transhuge.rst | 4 ++-- mm/shmem.c | 12 ++++++------ 2 files changed, 8 insertions(+), 8 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +2 -2
@@ -389,8 +389,8 @@ is not defined within a valid ``thp_anon``, its policy will default to Similarly to ``transparent_hugepage``, you can control the hugepage allocation policy for the internal shmem mount by using the kernel parameter ``transparent_hugepage_shmem=<policy>``, where ``<policy>`` is one of the -seven valid policies for shmem (``always``, ``within_size``, ``advise``, -``never``, ``deny``, and ``force``). +four valid policies for shmem (``always``, ``within_size``, ``advise``, +``never``). Similarly to ``transparent_hugepage_shmem``, you can control the default hugepage allocation policy for the tmpfs mount by using the kernel parameter
mm/shmem.c +6 -6
@@ -668,7 +668,7 @@ static unsigned int shmem_huge_global_enabled(struct inode *inode, pgoff_t index } } -static int shmem_parse_huge (const char *str) +static int shmem_parse_huge (const char *str,bool skip_deny_force) {int huge;
@@ -683,9 +683,9 @@ static int shmem_parse_huge(const char *str) huge =SHMEM_HUGE_WITHIN_SIZE ; else if (!strcmp(str, "advise")) huge =SHMEM_HUGE_ADVISE ; - else if (!strcmp(str, "deny")) + else if (!strcmp(str, "deny") && !skip_deny_force) huge =SHMEM_HUGE_DENY ; - else if (!strcmp(str, "force")) + else if (!strcmp(str, "force") && !skip_deny_force) huge =SHMEM_HUGE_FORCE ; else return -EINVAL;
@@ -5680,7 +5680,7 @@ static ssize_t shmem_enabled_store(struct kobject *kobj,if (count && tmp[count -1 ] =='\n' ) tmp[count -1 ] ='\0' ; - huge =shmem_parse_huge (tmp); + huge =shmem_parse_huge (tmp,/*skip_deny_force=*/ false);if (huge == -EINVAL )return huge;
@@ -5807,7 +5807,7 @@ static int __init setup_transparent_hugepage_shmem(char *str) {int huge; - huge =shmem_parse_huge (str); + huge =shmem_parse_huge (str,/*skip_deny_force=*/ true);if (huge == -EINVAL ) {pr_warn ("transparent_hugepage_shmem= cannot parse, ignored\n" );return huge;
@@ -5822,7 +5822,7 @@ static int __init setup_transparent_hugepage_tmpfs(char *str) {int huge; - huge =shmem_parse_huge (str); + huge =shmem_parse_huge (str,/*skip_deny_force=*/ true);if (huge <0 ) {pr_warn ("transparent_hugepage_tmpfs= cannot parse, ignored\n" );return huge;
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 9/12 mm: khugepaged: add shmem mTHP collapse support
In reply to Baolin WangKhugepaged already supports the anonymous mTHP collapse. Similarly, let khugepaged also support the shmem mTHP collapse. The strategy for shmem mTHP collapse follows the anonymous mTHP collapse:
Track present pages via a bitmap while scanning PMD ranges for collapse candidates. After the scan completes, use the bitmap to determine the most efficient mTHP order to collapse to. Scale 'max_ptes_none' by the attempted collapse order to determine the minimum fill threshold for eligibility. Similarly, shmem mTHP collapse rejects regions containing swapped-out pages to avoid creep.
Currently, the collapse_pte_mapped_thp() does not build the mapping for mTHP. Cause we still expect to establish the mTHP mapping via refault under the control of fault_around. So collapse_pte_mapped_thp() remains responsible only for building the mapping for PMD-sized THP, which is reasonable and makes life easier.
Note that we do not need to remove pte page tables for shmem mTHP collapse.
--- mm/khugepaged.c | 100 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 81 insertions(+), 19 deletions(-)
mm/khugepaged.c +81 -19
@@ -84,6 +84,10 @@ static struct khugepaged_scan khugepaged_scan = { .mm_head = LIST_HEAD_INIT(khugepaged_scan.mm_head), }; +static enum scan_result collapse_file (struct mm_struct *mm,unsigned long addr, +struct file *file,pgoff_t start, +struct collapse_control *cc,int order); +#ifdef CONFIG_SYSFS static ssize_t scan_sleep_millisecs_show(struct kobject *kobj, struct kobj_attribute *attr,
@@ -1452,7 +1456,11 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addresenum scan_result ret; collapse_address = address + offset *PAGE_SIZE ; - ret =collapse_huge_page (mm, collapse_address, cc, order); +if (cc->scan_file) + ret =collapse_file (mm, collapse_address, cc->scan_file, + cc->scan_pgoff + offset, cc, order); +else + ret =collapse_huge_page (mm, collapse_address, cc, order);switch (ret) {/* Cases where we continue to next collapse candidate */
@@ -1460,6 +1468,13 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addres collapsed += nr_ptes; fallthrough;case SCAN_PTE_MAPPED_HUGEPAGE : +/* +* Return SCAN_PTE_MAPPED_HUGEPAGE to call +* collapse_pte_mapped_thp() if the page cache +* already holds the PMD folio. +*/ +if (order ==HPAGE_PMD_ORDER ) + last_result = ret; goto next_offset;/* Cases where lower orders might still succeed */ case SCAN_ALLOC_HUGE_PAGE_FAIL :
@@ -1493,7 +1508,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addres* any smaller order enabled. When at the smallest order * we must always move to the next offset. */ -if (order >collapse_min_mthp_order (NULL ) && +if (order >collapse_min_mthp_order (cc->scan_file) && (cc->scan_orders &GENMASK (order -1 ,0 ))) { order--;continue ;
@@ -2237,7 +2252,8 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,if (++nr_none > max_ptes_none) { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_collapse_event (order,THP_SCAN_EXCEED_NONE_PTE , +MTHP_STAT_COLLAPSE_EXCEED_NONE ); goto xa_locked; }
@@ -2247,6 +2263,19 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,if (xa_is_value (folio) || !folio_test_uptodate (folio)) {xas_unlock_irq (&xas); + +/* +* TODO: Support swapin without leading to further mTHP +* collapses. Currently bringing in new pages via swapin may +* cause a future higher order collapse on a rescan of the same +* range. +*/ +if (!is_pmd_order (order)) { +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SWAP ); + result =SCAN_EXCEED_SWAP_PTE ; + goto xa_unlocked; + } +/* swap in or instantiate fallocated page */ if (shmem_get_folio (mapping->host, index,0 , &folio,SGP_NOALLOC )) {
@@ -2330,6 +2359,15 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, goto out_unlock; } +/* +* If the folio order is greater than the collapse order, there is +* no need to continue attempting to collapse. +*/ +if (folio_order (folio) >= order) { + result =SCAN_PTE_MAPPED_HUGEPAGE ; + goto out_unlock; + } +if (folio_mapping (folio) != mapping) { result =SCAN_TRUNCATED ; goto out_unlock;
@@ -2552,10 +2590,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,xas_unlock_irq (&xas);/* -* Remove pte page tables, so we can re-fault the page as huge. A -* caller that wants the PMD mapped now is told to go and do that. +* Remove pte page tables for PMD-sized THP collapse, so we can +* re-fault the page as huge. */ -retract_page_tables (mapping, start); +if (is_pmd_order (order)) +retract_page_tables (mapping, start);if (cc->policy.install_pmd) result =SCAN_PTE_MAPPED_HUGEPAGE ;folio_unlock (new_folio);
@@ -2606,10 +2645,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, }static enum scan_result collapse_scan_file (struct vm_area_struct *vma, -unsigned long addr,struct collapse_control *cc) +unsigned long addr,struct collapse_control *cc, +unsigned long enabled_orders) { -const unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL ,HPAGE_PMD_ORDER );const unsigned int max_ptes_swap =collapse_max_ptes_swap (cc,HPAGE_PMD_ORDER ); +unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL ,HPAGE_PMD_ORDER );struct file *file = vma->vm_file;struct folio *folio =NULL ;struct address_space *mapping = file->f_mapping;
@@ -2619,10 +2659,19 @@ static enum scan_result collapse_scan_file(struct vm_area_struct *vma,int node =NUMA_NO_NODE ;enum scan_result result =SCAN_SUCCEED ;unsigned long failed_pfn =-1 ; +unsigned long nr_pages; +pgoff_t pgoff; present =0 ; swap =0 ;collapse_scan_reset (cc); +/* +* If PMD is the only enabled order, enforce max_ptes_none, otherwise +* scan all pages to populate the bitmap for mTHP collapse. +*/ +if (enabled_orders !=BIT (HPAGE_PMD_ORDER )) + max_ptes_none =COLLAPSE_MAX_PTES_LIMIT ; +rcu_read_lock ();xas_for_each (&xas, folio, start +HPAGE_PMD_NR -1 ) {if (xas_retry (&xas, folio))
@@ -2690,7 +2739,17 @@ static enum scan_result collapse_scan_file(struct vm_area_struct *vma,* is just too costly... */ - present +=folio_nr_pages (folio); + nr_pages =folio_nr_pages (folio); + present += nr_pages; + +/* +* If there are folios present, keep track of it in the bitmap +* for file/shmem mTHP collapse. +*/ + pgoff =max_t (pgoff_t, start, folio->index) - start; + nr_pages =min_t (int,HPAGE_PMD_NR - pgoff, nr_pages); +bitmap_set (cc->eligible_ptes, pgoff, nr_pages); +folio_put (folio);if (need_resched ()) {
@@ -2706,7 +2765,8 @@ static enum scan_result collapse_scan_file(struct vm_area_struct *vma,if (result ==SCAN_SUCCEED && present <HPAGE_PMD_NR - max_ptes_none) { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_collapse_event (HPAGE_PMD_ORDER ,THP_SCAN_EXCEED_NONE_PTE , +MTHP_STAT_COLLAPSE_EXCEED_NONE ); }/*
@@ -2721,6 +2781,8 @@ static enum scan_result collapse_scan_file(struct vm_area_struct *vma,*/ cc->scan_file =get_file (file); cc->scan_pgoff = start; + cc->scan_orders = enabled_orders; + cc->scan_unmapped = swap; } trace_mm_khugepaged_scan_file(vma->vm_mm, failed_pfn, file, present, swap,
@@ -2778,7 +2840,7 @@ enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,if (vma_is_anonymous (vma))return collapse_scan_anon_pmd (vma, addr, cc, orders); -return collapse_scan_file (vma, addr, cc); +return collapse_scan_file (vma, addr, cc, orders); }/*
@@ -2796,23 +2858,21 @@ enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, {struct file *file = cc->scan_file;bool triggered_wb = false; -pgoff_t pgoff; - -if (!file) -return mthp_collapse (mm, addr, cc); - - cc->scan_file =NULL ; - pgoff = cc->scan_pgoff;/* The scan found the PMD folio in place: nothing to collapse */ if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) goto retract; + retry: - result =collapse_file (mm, addr, file, pgoff, cc,HPAGE_PMD_ORDER ); + result =mthp_collapse (mm, addr, cc); +/* Just return the results for anonymous folio collapse. */ +if (!file) +return result;/* Dirty pages are worth a writeback and one more try, if asked for */ if (cc->policy.writeback_dirty && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +pgoff_t pgoff = cc->scan_pgoff;const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ;const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ;
@@ -2821,6 +2881,8 @@ enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, goto retry; } retract: + cc->scan_file =NULL ; +VM_WARN_ON_ONCE (!file);fput (file);/*
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 4/12 mm: khugepaged: move cc->scan_file and cc->scan_pgoff assignment into collapse_scan_file()
In reply to Baolin WangMove the assignment of cc->scan_file and cc->scan_pgoff into collapse_scan_file(). This simplifies the parameter passing to collapse_scan_file() and cleans up the code in preparation for the upcoming shmem mTHP collapse support.
--- mm/khugepaged.c | 43 ++++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 23 deletions(-)
mm/khugepaged.c +20 -23
@@ -2586,14 +2586,15 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,return result; } -static enum scan_result collapse_scan_file (struct mm_struct *mm, -unsigned long addr,struct file *file,pgoff_t start, -struct collapse_control *cc) +static enum scan_result collapse_scan_file (struct vm_area_struct *vma, +unsigned long addr,struct collapse_control *cc) {const unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL ,HPAGE_PMD_ORDER );const unsigned int max_ptes_swap =collapse_max_ptes_swap (cc,HPAGE_PMD_ORDER ); +struct file *file = vma->vm_file;struct folio *folio =NULL ;struct address_space *mapping = file->f_mapping; +pgoff_t start =linear_page_index (vma, addr);XA_STATE (xas, &mapping->i_pages, start);int present, swap;int node =NUMA_NO_NODE ;
@@ -2689,7 +2690,21 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); } -trace_mm_khugepaged_scan_file (mm, failed_pfn, file, present, swap, +/* +* SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds +* the PMD folio, and retracting the PTE table is the run's job. +*/ +if (result ==SCAN_SUCCEED || result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +/* +* A file collapse works on the page cache and never sees a +* VMA, so take what it needs from this one while it is still +* here. +*/ + cc->scan_file =get_file (file); + cc->scan_pgoff = start; + } + +trace_mm_khugepaged_scan_file (vma->vm_mm, failed_pfn, file, present, swap, result);return result; }
@@ -2737,9 +2752,6 @@ enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, unsigned long addr, struct collapse_control *cc, unsigned long orders) { - enum scan_result result; -pgoff_t pgoff; - mmap_assert_locked(vma->vm_mm);/* Whatever the last scan found has to have been run by now */ collapse_put_scan_file (cc);
@@ -2747,22 +2759,7 @@ enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,if (vma_is_anonymous (vma))return collapse_scan_anon_pmd (vma, addr, cc, orders); - pgoff =linear_page_index (vma, addr); - result =collapse_scan_file (vma->vm_mm, addr, vma->vm_file, pgoff, cc); -/* -* SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds -* the PMD folio, and retracting the PTE table is the run's job. -*/ -if (result !=SCAN_SUCCEED && result !=SCAN_PTE_MAPPED_HUGEPAGE ) -return result; - -/* -* A file collapse works on the page cache and never sees a VMA, so take -* what it needs from this one while it is still here. -*/ - cc->scan_file =get_file (vma->vm_file); - cc->scan_pgoff = pgoff; -return result; +return collapse_scan_file (vma, addr, cc); }/*
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 5/12 mm: khugepaged: add max_ptes_none check in collapse_file()
In reply to Baolin WangSimilar to the anonymous folios collapse, we should also check the 'max_ptes_none' when trying to collapse shmem/file folios, which is also intended as preparation for shmem mTHP collapse in the following patches.
--- mm/khugepaged.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/khugepaged.c +8 -1
@@ -2158,6 +2158,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)static enum scan_result collapse_file (struct mm_struct *mm,unsigned long addr,struct file *file,pgoff_t start,struct collapse_control *cc) { +const unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL ,HPAGE_PMD_ORDER );struct address_space *mapping = file->f_mapping;struct page *dst;struct folio *folio, *tmp, *new_folio;
@@ -2222,7 +2223,13 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, goto xa_locked; } } - nr_none++; + +if (++nr_none > max_ptes_none) { + result =SCAN_EXCEED_NONE_PTE ; +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); + goto xa_locked; + } + index++;continue ; }
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 11/12 mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
In reply to Baolin WangWe are now ready to enable shmem mTHP collapse, allowing thp_vma_allowable_orders() to check all permissible shmem large orders.
--- mm/khugepaged.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/khugepaged.c +6 -1
@@ -493,9 +493,14 @@ unsigned long collapse_possible_orders(struct vm_area_struct *vma, {unsigned long orders; -/* If khugepaged is scanning an anonymous vma, allow mTHP collapse */ +/* +* If khugepaged is scanning an anonymous or shmem vma, +* allow mTHP collapse. +*/ if ((tva_flags ==TVA_KHUGEPAGED ) &&vma_is_anonymous (vma)) orders =THP_ORDERS_ALL_ANON ; +else if ((tva_flags ==TVA_KHUGEPAGED ) &&vma_is_shmem (vma)) + orders =THP_ORDERS_ALL_FILE_DEFAULT ;else orders =BIT (HPAGE_PMD_ORDER );
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 6/12 mm: khugepaged: generalize collapse_file() for shmem mTHP support
In reply to Baolin WangGeneralize the order of the collapse_file() function to support future shmem mTHP collapse.
No functional changes in this patch.
--- mm/khugepaged.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-)
mm/khugepaged.c +15 -12
@@ -2138,6 +2138,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)* @file: file that collapse on * @start: collapse start address * @cc: collapse context and scratchpad +* @order: folio order being collapsed to * * Basic scheme is simple, details are more complex: * - allocate and lock a new huge page;
@@ -2156,16 +2157,18 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)* + unlock and free huge page; */ static enum scan_result collapse_file (struct mm_struct *mm,unsigned long addr, -struct file *file,pgoff_t start,struct collapse_control *cc) +struct file *file,pgoff_t start,struct collapse_control *cc, +int order) { -const unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL ,HPAGE_PMD_ORDER ); +const unsigned int max_ptes_none =collapse_max_ptes_none (cc,NULL , order);struct address_space *mapping = file->f_mapping; +const unsigned long nr_pages =1UL << order;struct page *dst;struct folio *folio, *tmp, *new_folio;unsigned long new_pfn =-1 ; -pgoff_t index =0 , end = start +HPAGE_PMD_NR ; +pgoff_t index =0 , end = start + nr_pages;LIST_HEAD (pagelist); -XA_STATE_ORDER (xas, &mapping->i_pages, start,HPAGE_PMD_ORDER ); +XA_STATE_ORDER (xas, &mapping->i_pages, start, order);enum scan_result result =SCAN_SUCCEED ;int nr_none =0 ;bool is_shmem =shmem_file (file);
@@ -2174,9 +2177,9 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,* MADV_COLLAPSE ignores shmem huge config, so do not check shmem. */ VM_WARN_ON_ONCE (!mapping_pmd_folio_support (mapping)); -VM_WARN_ON_ONCE (start & (HPAGE_PMD_NR -1 )); +VM_WARN_ON_ONCE (start & (nr_pages -1 )); - result =alloc_charge_folio (&new_folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&new_folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out; new_pfn =folio_pfn (new_folio);
@@ -2514,12 +2517,12 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, }if (is_shmem) { -lruvec_stat_mod_folio (new_folio,NR_SHMEM ,HPAGE_PMD_NR ); +lruvec_stat_mod_folio (new_folio,NR_SHMEM , nr_pages);lruvec_stat_mod_folio (new_folio,NR_SHMEM_THPS ,HPAGE_PMD_NR ); }else {lruvec_stat_mod_folio (new_folio,NR_FILE_THPS ,HPAGE_PMD_NR ); } -lruvec_stat_mod_folio (new_folio,NR_FILE_PAGES ,HPAGE_PMD_NR ); +lruvec_stat_mod_folio (new_folio,NR_FILE_PAGES , nr_pages);/* * Mark new_folio as uptodate before inserting it into the
@@ -2527,14 +2530,14 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,* unwritten page. */ folio_mark_uptodate (new_folio); -folio_ref_add (new_folio,HPAGE_PMD_NR -1 ); +folio_ref_add (new_folio, nr_pages -1 );if (is_shmem)folio_mark_dirty (new_folio);folio_add_lru (new_folio);/* Join all the small entries into a single multi-index entry. */ -xas_set_order (&xas, start,HPAGE_PMD_ORDER ); +xas_set_order (&xas, start, order);xas_store (&xas, new_folio);WARN_ON_ONCE (xas_error (&xas));xas_unlock_irq (&xas);
@@ -2589,7 +2592,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,folio_put (new_folio); out:VM_BUG_ON (!list_empty (&pagelist)); -trace_mm_khugepaged_collapse_file (mm, new_pfn, index, addr, is_shmem, file,HPAGE_PMD_NR , result); +trace_mm_khugepaged_collapse_file (mm, new_pfn, index, addr, is_shmem, file, nr_pages, result);return result; }
@@ -2796,7 +2799,7 @@ enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr,if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) goto retract; retry: - result =collapse_file (mm, addr, file, pgoff, cc); + result =collapse_file (mm, addr, file, pgoff, cc,HPAGE_PMD_ORDER );/* Dirty pages are worth a writeback and one more try, if asked for */ if (cc->policy.writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 10/12 mm: shmem: run khugepaged for all shmem mTHP orders
In reply to Baolin WangWhen only non-PMD-sized mTHP is enabled (such as only 64K mTHP enabled), we should also allow kicking khugepaged to attempt scanning and collapsing 64K shmem mTHP. Modify shmem_hpage_pmd_enabled() to support shmem mTHP collapse, and while we are at it, rename it to make the function name more clear.
--- include/linux/shmem_fs.h | 4 ++-- mm/khugepaged.c | 2 +- mm/shmem.c | 10 +++++----- 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/shmem_fs.h +2 -2
@@ -135,7 +135,7 @@ int shmem_unuse(unsigned int type);unsigned long shmem_allowable_huge_orders (struct inode *inode,struct vm_area_struct *vma,pgoff_t index,loff_t write_end,bool shmem_huge_force); -bool shmem_hpage_pmd_enabled (void ); +bool shmem_hpage_enabled (void );#else static inline unsigned long shmem_allowable_huge_orders(struct inode *inode, struct vm_area_struct *vma, pgoff_t index,
@@ -144,7 +144,7 @@ static inline unsigned long shmem_allowable_huge_orders(struct inode *inode,return 0 ; } -static inline bool shmem_hpage_pmd_enabled (void ) +static inline bool shmem_hpage_enabled (void ) {return false; }
mm/khugepaged.c +1 -1
@@ -447,7 +447,7 @@ static bool hugepage_enabled(void)return true;if (anon_hpage_enabled ())return true; -if (shmem_hpage_pmd_enabled ()) +if (shmem_hpage_enabled ())return true;return false; }
mm/shmem.c +5 -5
@@ -2018,17 +2018,17 @@ static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -bool shmem_hpage_pmd_enabled (void ) +bool shmem_hpage_enabled (void ) {if (shmem_huge ==SHMEM_HUGE_DENY )return false; -if (test_bit (HPAGE_PMD_ORDER , &huge_shmem_orders_always)) +if (READ_ONCE (huge_shmem_orders_always))return true; -if (test_bit (HPAGE_PMD_ORDER , &huge_shmem_orders_madvise)) +if (READ_ONCE (huge_shmem_orders_madvise))return true; -if (test_bit (HPAGE_PMD_ORDER , &huge_shmem_orders_within_size)) +if (READ_ONCE (huge_shmem_orders_within_size))return true; -if (test_bit (HPAGE_PMD_ORDER , &huge_shmem_orders_inherit) && +if (READ_ONCE (huge_shmem_orders_inherit) && shmem_huge !=SHMEM_HUGE_NEVER )return true;
Signature · 1 line
2.47.3
Baolin Wang
reviewerPATCH 12/12 selftests: mm: add shmem mTHP collapse test cases
In reply to Baolin WangAdd shmem mTHP collapse test cases for 'collapse_full', and 'collapse_single_mthp' for shmem. All khugepaged test cases passed.
--- tools/testing/selftests/mm/khugepaged.c | 7 +++++-- tools/testing/selftests/mm/run_vmtests.sh | 2 ++ 2 files changed, 7 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/khugepaged.c +5 -2
@@ -627,6 +627,9 @@ static void mthp_khugepaged_collapse(const char *msg, char *p, int nr_hpages,if (ops == &__anon_ops) { settings.thp_enabled =THP_NEVER ; settings.hugepages[collapse_order].enabled =THP_MADVISE ; + }else if (ops == &__shmem_ops) { + settings.shmem_enabled =SHMEM_NEVER ; + settings.shmem_hugepages[collapse_order].enabled =SHMEM_ADVISE ; }thp_push_settings (&settings);
@@ -1260,8 +1263,6 @@ static void parse_test_type(int argc, char **argv)usage (); }else if (!strcmp (buf,"shmem" )) { shmem_ops = &__shmem_ops; -if (mthp_khugepaged_context) -usage (); }else {usage (); }
@@ -1358,6 +1359,7 @@ int main(int argc, char **argv)TEST (collapse_full, khugepaged_context, read_write_file_write_ops);TEST (collapse_full, khugepaged_context, shmem_ops);TEST (collapse_full, mthp_khugepaged_context, anon_ops); +TEST (collapse_full, mthp_khugepaged_context, shmem_ops);TEST (collapse_full, madvise_context, anon_ops);TEST (collapse_full, madvise_context, read_only_file_ops);TEST (collapse_full, madvise_context, read_write_file_read_ops);
@@ -1369,6 +1371,7 @@ int main(int argc, char **argv)TEST (collapse_empty, madvise_context, anon_ops);TEST (collapse_single_mthp, mthp_khugepaged_context, anon_ops); +TEST (collapse_single_mthp, mthp_khugepaged_context, shmem_ops);TEST (collapse_single_pte_entry, khugepaged_context, anon_ops);TEST (collapse_single_pte_entry, khugepaged_context, read_only_file_ops);
tools/testing/selftests/mm/run_vmtests.sh +2 -0
@@ -427,6 +427,8 @@ CATEGORY="thp" run_test ./khugepaged -s 4 all:shmem CATEGORY="thp" run_test ./khugepaged -c 4 mthp_khugepaged:anon +CATEGORY="thp" run_test ./khugepaged -c 4 mthp_khugepaged:shmem + # Try to create XFS if not provided if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then if test_selected "thp"; then
Signature · 1 line
2.47.3
Re: [PATCH 01/12] mm: khugepaged: remove outdated comments and check for shmem
In reply to Baolin WangQuoted 7 lines · Baolin Wang
After commit
eb92d97f7e6a("Revert 'tmpfs: don't enable large folios if not supported'"), all shmem mounts can support large folios unconditionally, so drop the outdated comments and the check for shmem incollapse_file().Signed-off-by Baolin Wang <[email protected]>---
LGTM,
Re: [PATCH 02/12] mm: khugepaged: drop the hugepage_madvise()
In reply to Baolin WangQuoted 9 lines · Baolin Wang
After commit
2fd4e7693674("mm: fix incorrectvm_flagsusage when checking allowable orders for tmpfs"),hugepage_madvise()no longer calls any khugepaged related functions and only updates the VMA flags. So remove this helper and update the VMA flags directly inmadvise_update_vma(), which simplifies the code.Signed-off-by Baolin Wang <[email protected]>---
LGTM,
Barry Song
reviewerRe: [PATCH 03/12] mm: shmem: don't expose 'deny' and 'force' via the kernel command line
In reply to Baolin WangQuoted 7 lines · Baolin Wang
Based on the previous discussion with Lorenzo[1], the 'deny' and 'force' options are testing-only configurations and shouldn't be exposed via the shmem kernel command line.
So drop these debug options from the shmem kernel command line and update the documentation to make this clear.
If it is a debug option, it should not be exposed through sysfs :-)
Maybe we could rephrase this to say that it can only be switched at runtime through the shmem_enabled sysfs interface?
Quoted 5 lines
[1] https://lore.kernel.org/all/[email protected]/
Suggested-by Lorenzo Stoakes (ARM) <[email protected]>Signed-off-by Baolin Wang <[email protected]>---
[...]
Quoted 2 lines
+static int
shmem_parse_huge(const char *str, bool skip_deny_force)
{
The name is a bit weird. What about allow_deny_force?
Quoted 15 lines
int huge;mm/shmem.c +2
-2@@ -683,9 +683,9 @@ static int shmem_parse_huge(const char *str) huge =SHMEM_HUGE_WITHIN_SIZE ; else if (!strcmp(str, "advise")) huge =SHMEM_HUGE_ADVISE ; - else if (!strcmp(str, "deny")) + else if (!strcmp(str, "deny") && !skip_deny_force) huge =SHMEM_HUGE_DENY ; - else if (!strcmp(str, "force")) + else if (!strcmp(str, "force") && !skip_deny_force) huge =SHMEM_HUGE_FORCE ; else return -EINVAL;
mm/shmem.c static ssize_t shmem_enabled_store(struct kobject *kobj, · line 5683
Quoted 5 lines
mm/shmem.c +1
-1@@ -5680,7 +5680,7 @@ static ssize_t shmem_enabled_store(struct kobject *kobj,if (count && tmp[count -1 ] =='\n' ) tmp[count -1 ] ='\0' ; - huge =shmem_parse_huge (tmp); + huge =shmem_parse_huge (tmp,/*skip_deny_force=*/ false);
/*allow_deny_force=*/true
mm/shmem.c static int __init setup_transparent_hugepage_shmem(char *str) · line 5810
Quoted 9 lines
mm/shmem.c +1
-1if (huge == -EINVAL )return huge;@@ -5807,7 +5807,7 @@ static int __init setup_transparent_hugepage_shmem(char *str) {int huge; - huge =shmem_parse_huge (str); + huge =shmem_parse_huge (str,/*skip_deny_force=*/ true);
/*allow_deny_force=*/false
mm/shmem.c static int __init setup_transparent_hugepage_tmpfs(char *str) · line 5825
Quoted 9 lines
mm/shmem.c +1
-1if (huge == -EINVAL ) {pr_warn ("transparent_hugepage_shmem= cannot parse, ignored\n" );return huge;@@ -5822,7 +5822,7 @@ static int __init setup_transparent_hugepage_tmpfs(char *str) {int huge; - huge =shmem_parse_huge (str); + huge =shmem_parse_huge (str,/*skip_deny_force=*/ true);
/*allow_deny_force=*/false
Quoted 6 lines
mm/shmem.c +0
-1if (huge <0 ) {pr_warn ("transparent_hugepage_tmpfs= cannot parse, ignored\n" );return huge; --2.47.3
Best Regards
Barry
Baolin Wang
reviewerRe: [PATCH 03/12] mm: shmem: don't expose 'deny' and 'force' via the kernel command line
Quoted 11 lines · Barry Song
Quoted 7 lines · Baolin Wang
Based on the previous discussion with Lorenzo[1], the 'deny' and 'force' options are testing-only configurations and shouldn't be exposed via the shmem kernel command line.
So drop these debug options from the shmem kernel command line and update the documentation to make this clear.
If it is a debug option, it should not be exposed through sysfs :-)
Ah, let's keep it compatible for now.
Quoted 2 lines
Maybe we could rephrase this to say that it can only be switched at runtime through the
shmem_enabledsysfs interface?
After the changes to transhuge.rst, this is essentially what it means. Anyway, I can make it more explicit in the commit message.
Quoted 9 lines
Quoted 4 lines
[1] https://lore.kernel.org/all/[email protected]/
Suggested-by Lorenzo Stoakes (ARM) <[email protected]>Signed-off-by Baolin Wang <[email protected]>---
[...]
Quoted 2 lines
+static int
shmem_parse_huge(const char *str, bool skip_deny_force)
{The name is a bit weird. What about
allow_deny_force?
Sounds good. Will do if no other objections.
Thanks for reviewing.