khugepaged: mTHP support
Subsystems (5)
v10 cover khugepaged: mTHP support
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
To achieve this we generalize the khugepaged functions to no longer depend on PMD_ORDER. Then during the PMD scan, we use a bitmap to track chunks of pages (defined by KHUGEPAGED_MTHP_MIN_ORDER) that are utilized. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction on max_ptes_none is removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained. max_ptes_none will be scaled by the attempted collapse order to determine how full a mTHP must be to be eligible for the collapse to occur. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. It is now also possible to collapse to mTHPs without requiring the PMD THP size to be enabled.
With the default max_ptes_none=511, the code should keep its most of its original behavior. When enabling multiple adjacent (m)THP sizes we need to set max_ptes_none<=255. With max_ptes_none > HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. This is due the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.
Patch 1: Refactor/rename hpage_collapse Patch 2: Some refactoring to combine madvise_collapse and khugepaged Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-8: The mTHP patches Patch 9-10: Allow khugepaged to operate without PMD enabled Patch 11-12: Tracing/stats Patch 13: Documentation
---------
Testing
---------
- Built for x86_64, aarch64, ppc64le, and s390x
- selftests mm
- I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit)
The summary from my testings was that there was no significant regression noticed through this test. In some cases my changes had better collapse latencies, and was able to scan more pages in the same amount of time/work, but for the most part the results were consistent.
- redis testing. I tested these changes along with my defer changes (see followup [4] post for more details). We've decided to get the mTHP changes merged first before attempting the defer series.
- some basic testing on 64k page size.
- lots of general use.
V10 Changes:
- Fixed bug where bitmap tracking was off by one leading to weird behavior in some test cases.
- Track mTHP stats for PMD order too (Baolin)
- indentation cleanup (David)
- add review/ack tags
- Improve the control flow, readability, and result handling in collapse_scan_bitmap (Baolin)
- Indentation nits/cleanup (David)
- Converted u8 orders to unsigned int to be consistent with other folio callers (David)
- Handled conflicts with Devs work on pte batching
- Changed SWAP/SHARED restriction comments to a TODO comment (David)
- Squashed main mTHP patch and the introduce bitmap patch (David)
- Other small nits
V9 Changes: [3]
- Drop madvise_collapse support [2]. Further discussion needed.
- Add documentation entries for new stats (Baolin)
- Fix missing stat update (MTHP_STAT_COLLAPSE_EXCEED_SWAP) that was accidentally dropped in v7 (Baolin)
- Fix mishandled conflict noted in v8 (merged into wrong commit)
- change rename from khugepaged to collapse (Dev)
V8 Changes:
- Fix mishandled conflict with shmem config changes (Baolin)
- Add Baolin's patches for allowing collapse without PMD enabled
- Add additional patch for allowing madvise_collapse without PMD enabled
- Documentations nits (Randy)
- Simplify SCAN_ANY_PROCESS lock jumbling (Liam)
- Add a BUG_ON to the mTHP collapse similar to PMD (Dev)
- Remove doc comment about khugepaged PMD only limitation (Dev)
- Change revalidation function to accept multiple orders
- Handled conflicts introduced by Lorenzo's madvise changes
V7 (RESEND)
V6 Changes:
- Dont release the anon_vma_lock early (like in the PMD case), as not all pages are isolated.
- Define the PTE as null to avoid a uninitilized condition
- minor nits and newline cleanup
- make sure to unmap and unlock the pte for the swapin case
- change the revalidation to always check the PMD order (as this will make sure that no other VMA spans it)
V5 Changes:
- switched the order of patches 1 and 2
- fixed some edge cases on the unified madvise_collapse and khugepaged
- Explained the "creep" some more in the docs
- fix EXCEED_SHARED vs EXCEED_SWAP accounting issue
- fix potential highmem issue caused by a early unmap of the PTE
V4 Changes:
- Rebased onto mm-unstable
- small changes to Documentation
V3 Changes:
- corrected legacy behavior for khugepaged and madvise_collapse
- added proper mTHP stat tracking
- Minor changes to prevent a nested lock on non-split-lock arches
- Took Devs version of alloc_charge_folio as it has the proper stats
- Skip cases were trying to collapse to a lower order would still fail
- Fixed cases were the bitmap was not being updated properly
- Moved Documentation update to this series instead of the defer set
- Minor bugs discovered during testing and review
- Minor "nit" cleanup
V2 Changes:
- Minor bug fixes discovered during review and testing
- removed dynamic allocations for bitmaps, and made them stack based
- Adjusted bitmap offset from u8 to u16 to support 64k pagesize.
- Updated trace events to include collapsing order info.
- Scaled max_ptes_none by order rather than scaling to a 0-100 scale.
- No longer require a chunk to be fully utilized before setting the bit. Use the same max_ptes_none scaling principle to achieve this.
- Skip mTHP collapse that requires swapin or shared handling. This helps prevent some of the "creep" that was discovered in v1.
A big thanks to everyone that has reviewed, tested, and participated in the development process. Its been a great experience working with all of you on this long endeavour.
[1] - https://gitlab.com/npache/khugepaged_mthp_test
[2] - https://lore.kernel.org/all/[email protected]/
[3] - https://lore.kernel.org/lkml/[email protected]/
[4] - https://lore.kernel.org/lkml/[email protected]/
Baolin Wang (2):
khugepaged: enable collapsing mTHPs even when PMD THPs are disabled
khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
Dev Jain (1):
khugepaged: generalize alloc_charge_folio()
Nico Pache (10):
khugepaged: rename hpage_collapse_* to collapse_* introduce collapse_single_pmd to unify khugepaged and madvise_collapse
khugepaged: generalize hugepage_vma_revalidate for mTHP support
khugepaged: generalize __collapse_huge_page_* for mTHP support
khugepaged: add mTHP support
khugepaged: skip collapsing mTHP to smaller orders
khugepaged: avoid unnecessary mTHP collapse attempts
khugepaged: improve tracepoints for mTHP orders
khugepaged: add per-order mTHP khugepaged stats
Documentation: mm: update the admin guide for mTHP collapse
Documentation/admin-guide/mm/transhuge.rst | 44 +- include/linux/huge_mm.h | 5 + include/linux/khugepaged.h | 4 + include/trace/events/huge_memory.h | 34 +- mm/huge_memory.c | 11 + mm/khugepaged.c | 552 +++++++++++++++------ 6 files changed, 468 insertions(+), 182 deletions(-)
Signature · 1 line
2.50.1
PATCH v10 1/13 khugepaged: rename hpage_collapse_* to collapse_*
The hpage_collapse functions describe functions used by madvise_collapse and khugepaged. remove the unnecessary hpage prefix to shorten the function name.
--- mm/khugepaged.c | 73 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 37 deletions(-)
mm/khugepaged.c +36 -37
@@ -402,14 +402,14 @@ void __init khugepaged_destroy(void)kmem_cache_destroy (mm_slot_cache); } -static inline int hpage_collapse_test_exit (struct mm_struct *mm) +static inline int collapse_test_exit (struct mm_struct *mm) {return atomic_read (&mm->mm_users) ==0 ; } -static inline int hpage_collapse_test_exit_or_disable (struct mm_struct *mm) +static inline int collapse_test_exit_or_disable (struct mm_struct *mm) { -return hpage_collapse_test_exit (mm) || +return collapse_test_exit (mm) ||mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); }
@@ -444,7 +444,7 @@ void __khugepaged_enter(struct mm_struct *mm)int wakeup;/* __khugepaged_exit() must not run from under us */ -VM_BUG_ON_MM (hpage_collapse_test_exit (mm), mm); +VM_BUG_ON_MM (collapse_test_exit (mm), mm);if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm)))return ;
@@ -502,7 +502,7 @@ void __khugepaged_exit(struct mm_struct *mm) }else if (mm_slot) {/* * This is required to serialize against -* hpage_collapse_test_exit() (which is guaranteed to run +* collapse_test_exit() (which is guaranteed to run * under mmap sem read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock.
@@ -592,7 +592,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); -/* See hpage_collapse_scan_pmd(). */ +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; if (cc->is_khugepaged &&
@@ -848,7 +848,7 @@ struct collapse_control khugepaged_collapse_control = { .is_khugepaged = true, }; -static bool hpage_collapse_scan_abort (int nid,struct collapse_control *cc) +static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;
@@ -883,7 +883,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) }#ifdef CONFIG_NUMA -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {int nid, target_node =0 , max_value =0 ;
@@ -902,7 +902,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) return target_node; }#else -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {return 0 ; }
@@ -923,7 +923,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,enum tva_type type = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm)))return SCAN_ANY_PROCESS ; *vmap = vma =find_vma (mm, address);
@@ -996,7 +996,7 @@ static int check_pmd_still_valid(struct mm_struct *mm,/* * Bring missing pages in from swap, to complete THP collapse. -* Only done if hpage_collapse_scan_pmd believes it is worthwhile. +* Only done if khugepaged_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released.
@@ -1082,7 +1082,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE ); -int node =hpage_collapse_find_target_node (cc); +int node =collapse_find_target_node (cc);struct folio *folio; folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask);
@@ -1268,10 +1268,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,return result; } -static int hpage_collapse_scan_pmd (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long address,bool *mmap_locked, -struct collapse_control *cc) +static int collapse_scan_pmd (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long address,bool *mmap_locked, +struct collapse_control *cc) {pmd_t *pmd;pte_t *pte, *_pte;
@@ -1382,7 +1382,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* hit record. */ node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; goto out_unmap; }
@@ -1451,7 +1451,7 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot)lockdep_assert_held (&khugepaged_mm_lock); -if (hpage_collapse_test_exit (mm)) { +if (collapse_test_exit (mm)) {/* free mm_slot */ hash_del (&slot->hash);list_del (&slot->mm_node);
@@ -1753,7 +1753,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)if (find_pmd_or_thp_or_none (mm, addr, &pmd) !=SCAN_SUCCEED )continue ; -if (hpage_collapse_test_exit (mm)) +if (collapse_test_exit (mm))continue ;/* * When a vma is registered with uffd-wp, we cannot recycle
@@ -2275,9 +2275,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,return result; } -static int hpage_collapse_scan_file (struct mm_struct *mm,unsigned long addr, -struct file *file,pgoff_t start, -struct collapse_control *cc) +static int collapse_scan_file (struct mm_struct *mm,unsigned long addr, +struct file *file,pgoff_t start, +struct collapse_control *cc) {struct folio *folio =NULL ;struct address_space *mapping = file->f_mapping;
@@ -2332,7 +2332,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ;folio_put (folio);break ;
@@ -2382,7 +2382,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; } -static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, +static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) __acquires(&khugepaged_mm_lock)
@@ -2420,7 +2420,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, goto breakouterloop_mmap_lock; progress++; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop;vma_iter_init (&vmi, mm, khugepaged_scan.address);
@@ -2428,7 +2428,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,unsigned long hstart, hend;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) { +if (unlikely (collapse_test_exit_or_disable (mm))) { progress++;break ; }
@@ -2449,7 +2449,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,bool mmap_locked = true;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart ||
@@ -2462,12 +2462,12 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); mmap_locked = false; - *result =hpage_collapse_scan_file (mm, + *result =collapse_scan_file (mm, khugepaged_scan.address, file, pgoff, cc);fput (file);if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) {mmap_read_lock (mm); -if (hpage_collapse_test_exit_or_disable (mm)) +if (collapse_test_exit_or_disable (mm)) goto breakouterloop; *result =collapse_pte_mapped_thp (mm, khugepaged_scan.address, false);
@@ -2476,7 +2476,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); } }else { - *result =hpage_collapse_scan_pmd (mm, vma, + *result =collapse_scan_pmd (mm, vma, khugepaged_scan.address, &mmap_locked, cc); }
@@ -2509,7 +2509,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,* Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ -if (hpage_collapse_test_exit (mm) || !vma) { +if (collapse_test_exit (mm) || !vma) {/* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find
@@ -2563,8 +2563,8 @@ static void khugepaged_do_scan(struct collapse_control *cc) pass_through_head++;if (khugepaged_has_work () && pass_through_head <2 ) - progress +=khugepaged_scan_mm_slot (pages - progress, - &result, cc); + progress +=collapse_scan_mm_slot (pages - progress, + &result, cc);else progress = pages;spin_unlock (&khugepaged_mm_lock);
@@ -2805,12 +2805,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_unlock (mm); mmap_locked = false; - result =hpage_collapse_scan_file (mm, addr, file, pgoff, - cc); + result =collapse_scan_file (mm, addr, file, pgoff, cc);fput (file); }else { - result =hpage_collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); + result =collapse_scan_pmd (mm, vma, addr, + &mmap_locked, cc); }if (!mmap_locked) *lock_dropped = true;
Signature · 1 line
2.50.1
Re: [PATCH v10 01/13] khugepaged: rename hpage_collapse_* to collapse_*
Quoted 4 lines · Nico Pache
The
hpage_collapsefunctions describe functions used bymadvise_collapseand khugepaged. remove the unnecessary hpage prefix to shorten the function name.
Not a big deal, but You missed a comment in mm/mremap.c:
* Now new_pte is none, so hpage_collapse_scan_file() path can not find
In move_ptes().
Quoted 5 lines
Reviewed-by Liam R. Howlett <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>
Apart from nit above LGTM so:
Quoted 270 lines
--- mm/khugepaged.c | 73 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 37 deletions(-)mm/khugepaged.c +36
-38@@ -402,14 +402,14 @@ void __init khugepaged_destroy(void)kmem_cache_destroy (mm_slot_cache); } -static inline int hpage_collapse_test_exit (struct mm_struct *mm) +static inline int collapse_test_exit (struct mm_struct *mm) {return atomic_read (&mm->mm_users) ==0 ; } -static inline int hpage_collapse_test_exit_or_disable (struct mm_struct *mm) +static inline int collapse_test_exit_or_disable (struct mm_struct *mm) { -return hpage_collapse_test_exit (mm) || +return collapse_test_exit (mm) ||mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); }@@ -444,7 +444,7 @@ void __khugepaged_enter(struct mm_struct *mm)int wakeup;/* __khugepaged_exit() must not run from under us */ -VM_BUG_ON_MM (hpage_collapse_test_exit (mm), mm); +VM_BUG_ON_MM (collapse_test_exit (mm), mm);if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm)))return ;@@ -502,7 +502,7 @@ void __khugepaged_exit(struct mm_struct *mm) }else if (mm_slot) {/* * This is required to serialize against -* hpage_collapse_test_exit() (which is guaranteed to run +* collapse_test_exit() (which is guaranteed to run * under mmap sem read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock. @@ -592,7 +592,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); -/* See hpage_collapse_scan_pmd(). */ +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; if (cc->is_khugepaged &&@@ -848,7 +848,7 @@ struct collapse_control khugepaged_collapse_control = { .is_khugepaged = true, }; -static bool hpage_collapse_scan_abort (int nid,struct collapse_control *cc) +static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -883,7 +883,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) }#ifdef CONFIG_NUMA -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {int nid, target_node =0 , max_value =0 ;@@ -902,7 +902,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) return target_node; }#else -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {return 0 ; }@@ -923,7 +923,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,enum tva_type type = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm)))return SCAN_ANY_PROCESS ; *vmap = vma =find_vma (mm, address);@@ -996,7 +996,7 @@ static int check_pmd_still_valid(struct mm_struct *mm,/* * Bring missing pages in from swap, to complete THP collapse. -* Only done if hpage_collapse_scan_pmd believes it is worthwhile. +* Only done if khugepaged_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. @@ -1082,7 +1082,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE ); -int node =hpage_collapse_find_target_node (cc); +int node =collapse_find_target_node (cc);struct folio *folio; folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask);@@ -1268,10 +1268,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,return result; } -static int hpage_collapse_scan_pmd (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long address,bool *mmap_locked, -struct collapse_control *cc) +static int collapse_scan_pmd (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long address,bool *mmap_locked, +struct collapse_control *cc) {pmd_t *pmd;pte_t *pte, *_pte;@@ -1382,7 +1382,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* hit record. */ node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; goto out_unmap; }@@ -1451,7 +1451,7 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot)lockdep_assert_held (&khugepaged_mm_lock); -if (hpage_collapse_test_exit (mm)) { +if (collapse_test_exit (mm)) {/* free mm_slot */ hash_del (&slot->hash);list_del (&slot->mm_node);@@ -1753,7 +1753,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)if (find_pmd_or_thp_or_none (mm, addr, &pmd) !=SCAN_SUCCEED )continue ; -if (hpage_collapse_test_exit (mm)) +if (collapse_test_exit (mm))continue ;/* * When a vma is registered with uffd-wp, we cannot recycle @@ -2275,9 +2275,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,return result; } -static int hpage_collapse_scan_file (struct mm_struct *mm,unsigned long addr, -struct file *file,pgoff_t start, -struct collapse_control *cc) +static int collapse_scan_file (struct mm_struct *mm,unsigned long addr, +struct file *file,pgoff_t start, +struct collapse_control *cc) {struct folio *folio =NULL ;struct address_space *mapping = file->f_mapping;@@ -2332,7 +2332,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ;folio_put (folio);break ;@@ -2382,7 +2382,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; } -static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, +static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) __acquires(&khugepaged_mm_lock)@@ -2420,7 +2420,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, goto breakouterloop_mmap_lock; progress++; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop;vma_iter_init (&vmi, mm, khugepaged_scan.address);@@ -2428,7 +2428,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,unsigned long hstart, hend;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) { +if (unlikely (collapse_test_exit_or_disable (mm))) { progress++;break ; }@@ -2449,7 +2449,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,bool mmap_locked = true;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart ||@@ -2462,12 +2462,12 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); mmap_locked = false; - *result =hpage_collapse_scan_file (mm, + *result =collapse_scan_file (mm, khugepaged_scan.address, file, pgoff, cc);fput (file);if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) {mmap_read_lock (mm); -if (hpage_collapse_test_exit_or_disable (mm)) +if (collapse_test_exit_or_disable (mm)) goto breakouterloop; *result =collapse_pte_mapped_thp (mm, khugepaged_scan.address, false);@@ -2476,7 +2476,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); } }else { - *result =hpage_collapse_scan_pmd (mm, vma, + *result =collapse_scan_pmd (mm, vma, khugepaged_scan.address, &mmap_locked, cc); }@@ -2509,7 +2509,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,* Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ -if (hpage_collapse_test_exit (mm) || !vma) { +if (collapse_test_exit (mm) || !vma) {/* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find @@ -2563,8 +2563,8 @@ static void khugepaged_do_scan(struct collapse_control *cc) pass_through_head++;if (khugepaged_has_work () && pass_through_head <2 ) - progress +=khugepaged_scan_mm_slot (pages - progress, - &result, cc); + progress +=collapse_scan_mm_slot (pages - progress, + &result, cc);else progress = pages;spin_unlock (&khugepaged_mm_lock);@@ -2805,12 +2805,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_unlock (mm); mmap_locked = false; - result =hpage_collapse_scan_file (mm, addr, file, pgoff, - cc); + result =collapse_scan_file (mm, addr, file, pgoff, cc);fput (file); }else { - result =hpage_collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); + result =collapse_scan_pmd (mm, vma, addr, + &mmap_locked, cc); }if (!mmap_locked) *lock_dropped = true; --2.50.1
Re: [PATCH v10 01/13] khugepaged: rename hpage_collapse_* to collapse_*
In reply to Nico PacheQuoted 10 lines · Nico Pache
The
hpage_collapsefunctions describe functions used bymadvise_collapseand khugepaged. remove the unnecessary hpage prefix to shorten the function name.Reviewed-by Liam R. Howlett <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>
LGTM.
Cheers,
Lance
Quoted 271 lines
--- mm/khugepaged.c | 73 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 37 deletions(-)mm/khugepaged.c +36
-38@@ -402,14 +402,14 @@ void __init khugepaged_destroy(void)kmem_cache_destroy (mm_slot_cache); } -static inline int hpage_collapse_test_exit (struct mm_struct *mm) +static inline int collapse_test_exit (struct mm_struct *mm) {return atomic_read (&mm->mm_users) ==0 ; } -static inline int hpage_collapse_test_exit_or_disable (struct mm_struct *mm) +static inline int collapse_test_exit_or_disable (struct mm_struct *mm) { -return hpage_collapse_test_exit (mm) || +return collapse_test_exit (mm) ||mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); }@@ -444,7 +444,7 @@ void __khugepaged_enter(struct mm_struct *mm)int wakeup;/* __khugepaged_exit() must not run from under us */ -VM_BUG_ON_MM (hpage_collapse_test_exit (mm), mm); +VM_BUG_ON_MM (collapse_test_exit (mm), mm);if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm)))return ;@@ -502,7 +502,7 @@ void __khugepaged_exit(struct mm_struct *mm) }else if (mm_slot) {/* * This is required to serialize against -* hpage_collapse_test_exit() (which is guaranteed to run +* collapse_test_exit() (which is guaranteed to run * under mmap sem read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock. @@ -592,7 +592,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); -/* See hpage_collapse_scan_pmd(). */ +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; if (cc->is_khugepaged &&@@ -848,7 +848,7 @@ struct collapse_control khugepaged_collapse_control = { .is_khugepaged = true, }; -static bool hpage_collapse_scan_abort (int nid,struct collapse_control *cc) +static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -883,7 +883,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) }#ifdef CONFIG_NUMA -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {int nid, target_node =0 , max_value =0 ;@@ -902,7 +902,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) return target_node; }#else -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {return 0 ; }@@ -923,7 +923,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,enum tva_type type = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm)))return SCAN_ANY_PROCESS ; *vmap = vma =find_vma (mm, address);@@ -996,7 +996,7 @@ static int check_pmd_still_valid(struct mm_struct *mm,/* * Bring missing pages in from swap, to complete THP collapse. -* Only done if hpage_collapse_scan_pmd believes it is worthwhile. +* Only done if khugepaged_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. @@ -1082,7 +1082,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE ); -int node =hpage_collapse_find_target_node (cc); +int node =collapse_find_target_node (cc);struct folio *folio; folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask);@@ -1268,10 +1268,10 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,return result; } -static int hpage_collapse_scan_pmd (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long address,bool *mmap_locked, -struct collapse_control *cc) +static int collapse_scan_pmd (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long address,bool *mmap_locked, +struct collapse_control *cc) {pmd_t *pmd;pte_t *pte, *_pte;@@ -1382,7 +1382,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,* hit record. */ node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; goto out_unmap; }@@ -1451,7 +1451,7 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot)lockdep_assert_held (&khugepaged_mm_lock); -if (hpage_collapse_test_exit (mm)) { +if (collapse_test_exit (mm)) {/* free mm_slot */ hash_del (&slot->hash);list_del (&slot->mm_node);@@ -1753,7 +1753,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)if (find_pmd_or_thp_or_none (mm, addr, &pmd) !=SCAN_SUCCEED )continue ; -if (hpage_collapse_test_exit (mm)) +if (collapse_test_exit (mm))continue ;/* * When a vma is registered with uffd-wp, we cannot recycle @@ -2275,9 +2275,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,return result; } -static int hpage_collapse_scan_file (struct mm_struct *mm,unsigned long addr, -struct file *file,pgoff_t start, -struct collapse_control *cc) +static int collapse_scan_file (struct mm_struct *mm,unsigned long addr, +struct file *file,pgoff_t start, +struct collapse_control *cc) {struct folio *folio =NULL ;struct address_space *mapping = file->f_mapping;@@ -2332,7 +2332,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ;folio_put (folio);break ;@@ -2382,7 +2382,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; } -static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, +static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) __acquires(&khugepaged_mm_lock)@@ -2420,7 +2420,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, goto breakouterloop_mmap_lock; progress++; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop;vma_iter_init (&vmi, mm, khugepaged_scan.address);@@ -2428,7 +2428,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,unsigned long hstart, hend;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) { +if (unlikely (collapse_test_exit_or_disable (mm))) { progress++;break ; }@@ -2449,7 +2449,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,bool mmap_locked = true;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart ||@@ -2462,12 +2462,12 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); mmap_locked = false; - *result =hpage_collapse_scan_file (mm, + *result =collapse_scan_file (mm, khugepaged_scan.address, file, pgoff, cc);fput (file);if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) {mmap_read_lock (mm); -if (hpage_collapse_test_exit_or_disable (mm)) +if (collapse_test_exit_or_disable (mm)) goto breakouterloop; *result =collapse_pte_mapped_thp (mm, khugepaged_scan.address, false);@@ -2476,7 +2476,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,mmap_read_unlock (mm); } }else { - *result =hpage_collapse_scan_pmd (mm, vma, + *result =collapse_scan_pmd (mm, vma, khugepaged_scan.address, &mmap_locked, cc); }@@ -2509,7 +2509,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,* Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ -if (hpage_collapse_test_exit (mm) || !vma) { +if (collapse_test_exit (mm) || !vma) {/* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find @@ -2563,8 +2563,8 @@ static void khugepaged_do_scan(struct collapse_control *cc) pass_through_head++;if (khugepaged_has_work () && pass_through_head <2 ) - progress +=khugepaged_scan_mm_slot (pages - progress, - &result, cc); + progress +=collapse_scan_mm_slot (pages - progress, + &result, cc);else progress = pages;spin_unlock (&khugepaged_mm_lock);@@ -2805,12 +2805,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_unlock (mm); mmap_locked = false; - result =hpage_collapse_scan_file (mm, addr, file, pgoff, - cc); + result =collapse_scan_file (mm, addr, file, pgoff, cc);fput (file); }else { - result =hpage_collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); + result =collapse_scan_pmd (mm, vma, addr, + &mmap_locked, cc); }if (!mmap_locked) *lock_dropped = true; --2.50.1
PATCH v10 2/13 introduce collapse_single_pmd to unify khugepaged and madvise_collapse
In reply to Nico PacheThe khugepaged daemon and madvise_collapse have two different implementations that do almost the same thing.
Create collapse_single_pmd to increase code reuse and create an entry point to these two users.
Refactor madvise_collapse and collapse_scan_mm_slot to use the new collapse_single_pmd function. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged tests collapse_test_exit_or_disable before calling collapse_pte_mapped_thp, but we weren't doing it in the madvise_collapse case. By unifying these two callers madvise_collapse now also performs this check.
--- mm/khugepaged.c | 94 ++++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 45 deletions(-)
mm/khugepaged.c +49 -45
@@ -2382,6 +2382,50 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static int collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +int result =SCAN_FAIL ; +struct mm_struct *mm = vma->vm_mm; + +if (!vma_is_anonymous (vma)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; + result =collapse_scan_file (mm, addr, file, pgoff, cc); +fput (file); +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); + *mmap_locked = true; +if (collapse_test_exit_or_disable (mm)) { +mmap_read_unlock (mm); + *mmap_locked = false; + result =SCAN_ANY_PROCESS ; + goto end; + } + result =collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + *mmap_locked = false; + } + }else { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + } +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +end: +return result; +} + static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)
@@ -2455,34 +2499,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; progress +=HPAGE_PMD_NR ;
@@ -2799,34 +2818,19 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_assert_locked (mm);memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - result =collapse_scan_file (mm, addr, file, pgoff, cc); -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); - } + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); +if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PMD_NULL :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :
Signature · 1 line
2.50.1
Re: [PATCH v10 02/13] introduce collapse_single_pmd to unify khugepaged and madvise_collapse
Quoted 28 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing.Create
collapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check.Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 94 ++++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 45 deletions(-)mm/khugepaged.c +0
-0@@ -2382,6 +2382,50 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; }
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2393
Quoted 10 lines
mm/khugepaged.c +9
-0+/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static int collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +int result =SCAN_FAIL ;
You assign result in all branches, so this can be uninitialised.
Quoted 8 lines
mm/khugepaged.c +8
-0+struct mm_struct *mm = vma->vm_mm; + +if (!vma_is_anonymous (vma)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false;
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2411
Quoted 10 lines
mm/khugepaged.c +10
-0+ result =collapse_scan_file (mm, addr, file, pgoff, cc); +fput (file); +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); + *mmap_locked = true; +if (collapse_test_exit_or_disable (mm)) { +mmap_read_unlock (mm); + *mmap_locked = false; + result =SCAN_ANY_PROCESS ; + goto end;
Don't love that in e.g. collapse_scan_mm_slot() we are using the mmap lock being disabled as in effect an error code.
Is SCAN_ANY_PROCESS correct here? Because in collapse_scan_mm_slot() you'd previously:
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) {
mmap_read_lock(mm);
if (collapse_test_exit_or_disable(mm))
goto breakouterloop;
...
}But now you're setting result = SCAN_ANY_PROCESS rather than SCAN_PTE_MAPPED_HUGEPAGE in this instance?
You don't mention that you're changing this, or at least explicitly enough, the commit message should state that you're changing this and explain why it's ok.
This whole file is horrid, and it's kinda an aside, but I really wish we had some comment going through each of the scan_result cases and explaining what each one meant.
Also I think:
return SCAN_ANY_PROCESS;
Is better than:
result = SCAN_ANY_PROCESS; goto end; ... end: return result;
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2414
Quoted 3 lines
mm/khugepaged.c +3
-0+ } + result =collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged);
Hm another change here, in the original code in collapse_scan_mm_slot() this is:
*result = collapse_pte_mapped_thp(mm, khugepaged_scan.address, false);
Presumably collapse_scan_mm_slot() is only ever invoked with cc->is_khugepaged?
Maybe worth adding a VM_WARN_ON_ONCE(!cc->is_khugepaged) at the top of collapse_scan_mm_slot() to assert this (and other places where your change assumes this to be the case).
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2424
Quoted 10 lines
mm/khugepaged.c +10
-0+if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + *mmap_locked = false; + } + }else { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + } +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed;
Similarly, presumably because collapse_scan_mm_slot() only ever invoked khugepaged case this didn't have the cc->is_khugepaged check?
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2427
Quoted 3 lines
mm/khugepaged.c +3
-0+end: +return result; +}
There's a LOT of nesting going on here, I think we can simplify this a lot. If we make the change I noted above re: returning SCAN_ANY_PROCESS< we can move the end label up a bit and avoid a ton of nesting, e.g.:
static int collapse_single_pmd(unsigned long addr,
struct vm_area_struct *vma, bool *mmap_locked, struct collapse_control *cc)
{
struct mm_struct *mm = vma->vm_mm;
struct file *file;
pgoff_t pgoff;
int result;
if (vma_is_anonymous(vma)) {
result = collapse_scan_pmd(mm, vma, addr, mmap_locked, cc);
goto end:
}
file = get_file(vma->vm_file);
pgoff = linear_page_index(vma, addr);
mmap_read_unlock(mm);
*mmap_locked = false;
result = collapse_scan_file(mm, addr, file, pgoff, cc);
fput(file);
if (result != SCAN_PTE_MAPPED_HUGEPAGE)
goto end;
mmap_read_lock(mm);
*mmap_locked = true;
if (collapse_test_exit_or_disable(mm)) {
mmap_read_unlock(mm);
*mmap_locked = false;
return SCAN_ANY_PROCESS;
}
result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged);
if (result == SCAN_PMD_MAPPED)
result = SCAN_SUCCEED;
mmap_read_unlock(mm);
*mmap_locked = false;end:
if (cc->is_khugepaged && result == SCAN_SUCCEED) ++khugepaged_pages_collapsed; return result; }
(untested, thrown together so do double check!)
Quoted 49 lines
mm/khugepaged.c +3
-30+ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)@@ -2455,34 +2499,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; progress +=HPAGE_PMD_NR ;@@ -2799,34 +2818,19 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_assert_locked (mm);memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr);
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2823
Quoted 10 lines
mm/khugepaged.c +2
-8-mmap_read_unlock (mm); - mmap_locked = false; - result =collapse_scan_file (mm, addr, file, pgoff, cc); -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); - } + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); +
Ack the fact you noted the behaviour change re:collapse_test_exit_or_disable() that seems fine.
Quoted 5 lines
mm/khugepaged.c +0
-1if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2828
Quoted 10 lines
mm/khugepaged.c +0
-6case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;
One thing that differs with new code her is we filter SCAN_PMD_MAPPED to SCAN_SUCCEED.
I was about to say 'but ++thps - is this correct' but now I realise this was looping back on itself with a goto to do just that... ugh ye gads.
Anwyay that's fine because it doesn't change anything.
Re: switch statement in general, again would be good to always have each scan possibility in switch statements, but perhaps given so many not practical :)
(that way the compiler warns on missing a newly added enum val)
Quoted 8 lines
mm/khugepaged.c +1
-1/* Whitelisted set of results where continuing OK */ +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PMD_NULL :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP : --2.50.1
Re: [PATCH v10 02/13] introduce collapse_single_pmd to unify khugepaged and madvise_collapse
Quoted 31 lines · Lorenzo Stoakes
Quoted 29 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing.Create
collapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check.Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 94 ++++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 45 deletions(-)mm/khugepaged.c +0
-0@@ -2382,6 +2382,50 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; }
Quoted 11 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2393
Quoted 9 lines
mm/khugepaged.c +9
-0+/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static int collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +int result =SCAN_FAIL ;You assign result in all branches, so this can be uninitialised.
ack, thanks.
Quoted 23 lines
Quoted 9 lines
mm/khugepaged.c +8
-0+struct mm_struct *mm = vma->vm_mm; + +if (!vma_is_anonymous (vma)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2411
Quoted 10 lines
mm/khugepaged.c +10
-0+ result =collapse_scan_file (mm, addr, file, pgoff, cc); +fput (file); +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); + *mmap_locked = true; +if (collapse_test_exit_or_disable (mm)) { +mmap_read_unlock (mm); + *mmap_locked = false; + result =SCAN_ANY_PROCESS ; + goto end;Don't love that in e.g.
collapse_scan_mm_slot()we are using the mmap lock being disabled as in effect an error code.
Quoted 2 lines
Is
SCAN_ANY_PROCESScorrect here? Because incollapse_scan_mm_slot()you'd previously:
https://lore.kernel.org/lkml/[email protected]/
Baolin brought up a good point a while back that if collapse_test_exit_or_disable returns true we will be breaking out of the loop and should change the return value to indicate this. So to combine the madvise breakout and the scan_slot breakout we drop the lock and return SCAN_ANY_PROCESS.
Quoted 8 lines
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) {
mmap_read_lock(mm);
if (collapse_test_exit_or_disable(mm))
goto breakouterloop;
...
}Quoted 6 lines
But now you're setting result =
SCAN_ANY_PROCESSrather thanSCAN_PTE_MAPPED_HUGEPAGEin this instance?You don't mention that you're changing this, or at least explicitly enough, the commit message should state that you're changing this and explain why it's ok.
I do state it but perhaps I need to be more verbose! I will update the message to state we are also changing the result value too.
Quoted 4 lines
This whole file is horrid, and it's kinda an aside, but I really wish we had some comment going through each of the
scan_resultcases and explaining what each one meant.
Yeah its been a huge pain to have to investigate what everything is supposed to mean, and I often have to go searching to confirm things. include/trace/events/huge_memory.h has a "good" summary of them
Quoted 9 lines
Also I think:
return SCAN_ANY_PROCESS;Is better than:
result = SCAN_ANY_PROCESS; goto end;
I agree! I will change that :)
Quoted 8 lines
... end: return result;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2414
Quoted 3 lines
mm/khugepaged.c +3
-0+ } + result =collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged);
Quoted 8 lines
Hm another change here, in the original code in
collapse_scan_mm_slot()this is:*result = collapse_pte_mapped_thp(mm,
khugepaged_scan.address, false);Presumably
collapse_scan_mm_slot()is only ever invoked withcc->is_khugepaged?
Correct, but the madvise_collapse calls this with true, hence why it now depends on the is_khugepaged variable. No functional change here.
Quoted 4 lines
Maybe worth adding a
VM_WARN_ON_ONCE(!cc->is_khugepaged)at the top ofcollapse_scan_mm_slot()to assert this (and other places where your change assumes this to be the case).
Ok I will investigate doing that but it would take a huge mistake to hit that assertion.
Quoted 13 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2424
Quoted 12 lines
mm/khugepaged.c +10
-0+if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + *mmap_locked = false; + } + }else { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + } +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed;
Quoted 2 lines
Similarly, presumably because
collapse_scan_mm_slot()only ever invoked khugepaged case this didn't have thecc->is_khugepagedcheck?
Correct, we only increment this when its khugepaged, so we need to guard it so madvise collapse wont increment this.
Quoted 8 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2427
Quoted 4 lines
mm/khugepaged.c +3
-0+end: +return result; +}There's a LOT of nesting going on here, I think we can simplify this a lot. If we make the change I noted above re: returning SCAN_ANY_PROCESS< we can move the end label up a bit and avoid a ton of nesting, e.g.:
Ah I like this much more, I will try to implement/test it.
Quoted 129 lines
static int collapse_single_pmd(unsigned long addr,
structvm_area_struct*vma, bool*mmap_locked, structcollapse_control*cc){ struct mm_struct *mm = vma->vm_mm; struct file *file; pgoff_t pgoff; int result; if (vma_is_anonymous(vma)) { result = collapse_scan_pmd(mm, vma, addr, mmap_locked, cc); goto end: } file = get_file(vma->vm_file); pgoff = linear_page_index(vma, addr); mmap_read_unlock(mm); *mmap_locked = false; result = collapse_scan_file(mm, addr, file, pgoff, cc); fput(file); if (result != SCAN_PTE_MAPPED_HUGEPAGE) goto end; mmap_read_lock(mm); *mmap_locked = true; if (collapse_test_exit_or_disable(mm)) { mmap_read_unlock(mm); *mmap_locked = false; return SCAN_ANY_PROCESS; } result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged); if (result == SCAN_PMD_MAPPED) result = SCAN_SUCCEED; mmap_read_unlock(mm); *mmap_locked = false;end:
if (cc->is_khugepaged && result == SCAN_SUCCEED) ++khugepaged_pages_collapsed; return result; }(untested, thrown together so do double check!)
Quoted 49 lines
mm/khugepaged.c +3
-30+ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)@@ -2455,34 +2499,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; progress +=HPAGE_PMD_NR ;@@ -2799,34 +2818,19 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_assert_locked (mm);memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr);mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2823
Quoted 10 lines
mm/khugepaged.c +2
-8-mmap_read_unlock (mm); - mmap_locked = false; - result =collapse_scan_file (mm, addr, file, pgoff, cc); -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); - } + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); +Ack the fact you noted the behaviour change re:
collapse_test_exit_or_disable()that seems fine.Quoted 5 lines
mm/khugepaged.c +0
-1if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2828
Quoted 10 lines
mm/khugepaged.c +0
-6case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;One thing that differs with new code her is we filter
SCAN_PMD_MAPPEDtoSCAN_SUCCEED.
Quoted 8 lines
I was about to say 'but ++thps - is this correct' but now I realise this was looping back on itself with a goto to do just that... ugh ye gads.
Anwyay that's fine because it doesn't change anything.
Re: switch statement in general, again would be good to always have each scan possibility in switch statements, but perhaps given so many not practical :)
Yeah it may be worth investigating for future changes I have for khugepaged (including the new switch statement I implement later and you commented on)
Quoted 9 lines
(that way the compiler warns on missing a newly added enum val)
Quoted 6 lines
mm/khugepaged.c +1
-1/* Whitelisted set of results where continuing OK */ +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PMD_NULL :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP : --
Thanks for the review :)
-- Nico
Quoted 3 lines
Quoted 2 lines
2.50.1
Re: [PATCH v10 02/13] introduce collapse_single_pmd to unify khugepaged and madvise_collapse
Quoted 68 lines · Nico Pache
Quoted 31 lines · Lorenzo Stoakes
Quoted 29 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing.Create
collapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check.Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 94 ++++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 45 deletions(-)mm/khugepaged.c +0
-0@@ -2382,6 +2382,50 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; }Quoted 11 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2393
Quoted 9 lines
mm/khugepaged.c +9
-0+/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static int collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +int result =SCAN_FAIL ;You assign result in all branches, so this can be uninitialised.
ack, thanks.
Quoted 23 lines
Quoted 9 lines
mm/khugepaged.c +8
-0+struct mm_struct *mm = vma->vm_mm; + +if (!vma_is_anonymous (vma)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2411
Quoted 10 lines
mm/khugepaged.c +10
-0+ result =collapse_scan_file (mm, addr, file, pgoff, cc); +fput (file); +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); + *mmap_locked = true; +if (collapse_test_exit_or_disable (mm)) { +mmap_read_unlock (mm); + *mmap_locked = false; + result =SCAN_ANY_PROCESS ; + goto end;Don't love that in e.g.
collapse_scan_mm_slot()we are using the mmap lock being disabled as in effect an error code.
Quoted 8 lines
Quoted 2 lines
Is
SCAN_ANY_PROCESScorrect here? Because incollapse_scan_mm_slot()you'd previously:https://lore.kernel.org/lkml/[email protected]/
Baolin brought up a good point a while back that ifcollapse_test_exit_or_disablereturns true we will be breaking out of the loop and should change the return value to indicate this. So to combine the madvise breakout and thescan_slotbreakout we drop the lock and returnSCAN_ANY_PROCESS.
Let's document in commit msg, as Liam's pointed out it's really important to track things, and part of that as well is detailing in the commit message what you're doing + why.
With the THP code being as 'organically grown' as it is shall we say :) it's even more mportant to be specific.
Quoted 8 lines
Quoted 8 lines
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) goto breakouterloop; ... }
Quoted 8 lines
Quoted 6 lines
But now you're setting result =
SCAN_ANY_PROCESSrather thanSCAN_PTE_MAPPED_HUGEPAGEin this instance?You don't mention that you're changing this, or at least explicitly enough, the commit message should state that you're changing this and explain why it's ok.
I do state it but perhaps I need to be more verbose! I will update the message to state we are also changing the result value too.
Thanks!
Quoted 54 lines
Quoted 4 lines
This whole file is horrid, and it's kinda an aside, but I really wish we had some comment going through each of the
scan_resultcases and explaining what each one meant.Yeah its been a huge pain to have to investigate what everything is supposed to mean, and I often have to go searching to confirm things.
include/trace/events/huge_memory.hhas a "good" summary of themQuoted 9 lines
Also I think:
return SCAN_ANY_PROCESS;Is better than:
result = SCAN_ANY_PROCESS; goto end;I agree! I will change that :)
Quoted 8 lines
... end: return result;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2414
Quoted 3 lines
mm/khugepaged.c +3
-0+ } + result =collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged);Quoted 8 lines
Hm another change here, in the original code in
collapse_scan_mm_slot()this is:*result = collapse_pte_mapped_thp(mm,
khugepaged_scan.address, false);Presumably
collapse_scan_mm_slot()is only ever invoked withcc->is_khugepaged?Correct, but the
madvise_collapsecalls this with true, hence why it now depends on theis_khugepagedvariable. No functional change here.Quoted 4 lines
Maybe worth adding a
VM_WARN_ON_ONCE(!cc->is_khugepaged)at the top ofcollapse_scan_mm_slot()to assert this (and other places where your change assumes this to be the case).Ok I will investigate doing that but it would take a huge mistake to hit that assertion.
Quoted 13 lines
Quoted 12 lines
mm/khugepaged.c +10
-0+if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + *mmap_locked = false; + } + }else { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + } +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed;
Quoted 4 lines
Quoted 2 lines
Similarly, presumably because
collapse_scan_mm_slot()only ever invoked khugepaged case this didn't have thecc->is_khugepagedcheck?Correct, we only increment this when its khugepaged, so we need to guard it so madvise collapse wont increment this.
You know what I'm going to say :) commit message please!
Quoted 55 lines
Quoted 8 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2427
Quoted 4 lines
mm/khugepaged.c +3
-0+end: +return result; +}There's a LOT of nesting going on here, I think we can simplify this a lot. If we make the change I noted above re: returning SCAN_ANY_PROCESS< we can move the end label up a bit and avoid a ton of nesting, e.g.:
Ah I like this much more, I will try to implement/test it.
Quoted 46 lines
static int collapse_single_pmd(unsigned long addr,
structvm_area_struct*vma, bool*mmap_locked, structcollapse_control*cc){ struct mm_struct *mm = vma->vm_mm; struct file *file; pgoff_t pgoff; int result; if (vma_is_anonymous(vma)) { result = collapse_scan_pmd(mm, vma, addr, mmap_locked, cc); goto end: } file = get_file(vma->vm_file); pgoff = linear_page_index(vma, addr); mmap_read_unlock(mm); *mmap_locked = false; result = collapse_scan_file(mm, addr, file, pgoff, cc); fput(file); if (result != SCAN_PTE_MAPPED_HUGEPAGE) goto end; mmap_read_lock(mm); *mmap_locked = true; if (collapse_test_exit_or_disable(mm)) { mmap_read_unlock(mm); *mmap_locked = false; return SCAN_ANY_PROCESS; } result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged); if (result == SCAN_PMD_MAPPED) result = SCAN_SUCCEED; mmap_read_unlock(mm); *mmap_locked = false;end:
if (cc->is_khugepaged && result == SCAN_SUCCEED) ++khugepaged_pages_collapsed; return result; }(untested, thrown together so do double check!)
This suggested refactoring work for you?
Quoted 88 lines
Quoted 88 lines
Quoted 50 lines
mm/khugepaged.c +3
-30+ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)@@ -2455,34 +2499,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; progress +=HPAGE_PMD_NR ;@@ -2799,34 +2818,19 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_assert_locked (mm);memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr);mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2823
Quoted 10 lines
mm/khugepaged.c +2
-8-mmap_read_unlock (mm); - mmap_locked = false; - result =collapse_scan_file (mm, addr, file, pgoff, cc); -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); - } + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); +Ack the fact you noted the behaviour change re:
collapse_test_exit_or_disable()that seems fine.Quoted 5 lines
mm/khugepaged.c +0
-1if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2828
Quoted 10 lines
mm/khugepaged.c +0
-6case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;One thing that differs with new code her is we filter
SCAN_PMD_MAPPEDtoSCAN_SUCCEED.I was about to say 'but ++thps - is this correct' but now I realise this was looping back on itself with a goto to do just that... ugh ye gads.
Anwyay that's fine because it doesn't change anything.
Quoted 7 lines
Quoted 3 lines
Re: switch statement in general, again would be good to always have each scan possibility in switch statements, but perhaps given so many not practical :)
Yeah it may be worth investigating for future changes I have for khugepaged (including the new switch statement I implement later and you commented on)
Ack yeah this can be one for the future!
Quoted 11 lines
Quoted 9 lines
(that way the compiler warns on missing a newly added enum val)
Quoted 6 lines
mm/khugepaged.c +1
-1/* Whitelisted set of results where continuing OK */ +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PMD_NULL :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP : --Thanks for the review :)
No probs, to underline as well - the critique is to make sure we get this right, my aim here is to get your series landed in as good a form as possible :)
Quoted 6 lines
-- Nico
Quoted 3 lines
Quoted 2 lines
2.50.1
Cheers, Lorenzo
Re: [PATCH v10 02/13] introduce collapse_single_pmd to unify khugepaged and madvise_collapse
Quoted 79 lines · Lorenzo Stoakes
Quoted 68 lines · Nico Pache
Quoted 31 lines · Lorenzo Stoakes
Quoted 29 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing.Create
collapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check.Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 94 ++++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 45 deletions(-)mm/khugepaged.c +0
-0@@ -2382,6 +2382,50 @@ static int collapse_scan_file(struct mm_struct *mm, unsigned long addr,return result; }Quoted 11 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2393
Quoted 9 lines
mm/khugepaged.c +9
-0+/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static int collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +int result =SCAN_FAIL ;You assign result in all branches, so this can be uninitialised.
ack, thanks.
Quoted 23 lines
Quoted 9 lines
mm/khugepaged.c +8
-0+struct mm_struct *mm = vma->vm_mm; + +if (!vma_is_anonymous (vma)) { +struct file *file =get_file (vma->vm_file); +pgoff_t pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2411
Quoted 10 lines
mm/khugepaged.c +10
-0+ result =collapse_scan_file (mm, addr, file, pgoff, cc); +fput (file); +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); + *mmap_locked = true; +if (collapse_test_exit_or_disable (mm)) { +mmap_read_unlock (mm); + *mmap_locked = false; + result =SCAN_ANY_PROCESS ; + goto end;Don't love that in e.g.
collapse_scan_mm_slot()we are using the mmap lock being disabled as in effect an error code.Quoted 8 lines
Quoted 2 lines
Is
SCAN_ANY_PROCESScorrect here? Because incollapse_scan_mm_slot()you'd previously:https://lore.kernel.org/lkml/[email protected]/
Baolin brought up a good point a while back that ifcollapse_test_exit_or_disablereturns true we will be breaking out of the loop and should change the return value to indicate this. So to combine the madvise breakout and thescan_slotbreakout we drop the lock and returnSCAN_ANY_PROCESS.
Quoted 3 lines
Let's document in commit msg, as Liam's pointed out it's really important to track things, and part of that as well is detailing in the commit message what you're doing + why.
ack! thanks
Quoted 77 lines
With the THP code being as 'organically grown' as it is shall we say :) it's even more mportant to be specific.
Quoted 8 lines
Quoted 8 lines
if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); if (collapse_test_exit_or_disable(mm)) goto breakouterloop; ... }Quoted 8 lines
Quoted 6 lines
But now you're setting result =
SCAN_ANY_PROCESSrather thanSCAN_PTE_MAPPED_HUGEPAGEin this instance?You don't mention that you're changing this, or at least explicitly enough, the commit message should state that you're changing this and explain why it's ok.
I do state it but perhaps I need to be more verbose! I will update the message to state we are also changing the result value too.
Thanks!
Quoted 54 lines
Quoted 4 lines
This whole file is horrid, and it's kinda an aside, but I really wish we had some comment going through each of the
scan_resultcases and explaining what each one meant.Yeah its been a huge pain to have to investigate what everything is supposed to mean, and I often have to go searching to confirm things.
include/trace/events/huge_memory.hhas a "good" summary of themQuoted 9 lines
Also I think:
return SCAN_ANY_PROCESS;Is better than:
result = SCAN_ANY_PROCESS; goto end;I agree! I will change that :)
Quoted 8 lines
... end: return result;mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2414
Quoted 3 lines
mm/khugepaged.c +3
-0+ } + result =collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged);Quoted 8 lines
Hm another change here, in the original code in
collapse_scan_mm_slot()this is:*result = collapse_pte_mapped_thp(mm,
khugepaged_scan.address, false);Presumably
collapse_scan_mm_slot()is only ever invoked withcc->is_khugepaged?Correct, but the
madvise_collapsecalls this with true, hence why it now depends on theis_khugepagedvariable. No functional change here.Quoted 4 lines
Maybe worth adding a
VM_WARN_ON_ONCE(!cc->is_khugepaged)at the top ofcollapse_scan_mm_slot()to assert this (and other places where your change assumes this to be the case).Ok I will investigate doing that but it would take a huge mistake to hit that assertion.
Quoted 13 lines
Quoted 12 lines
mm/khugepaged.c +10
-0+if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + *mmap_locked = false; + } + }else { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + } +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed;
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Similarly, presumably because
collapse_scan_mm_slot()only ever invoked khugepaged case this didn't have thecc->is_khugepagedcheck?Correct, we only increment this when its khugepaged, so we need to guard it so madvise collapse wont increment this.
You know what I'm going to say :) commit message please!
ack, although this isnt anything new. I just needed to add it because madvise collapse doesnt increment this. Still I'll add a blurb.
Quoted 52 lines
Quoted 52 lines
Quoted 9 lines
mm/khugepaged.c static int collapse_scan_file(struct mm_struct *mm, unsigned long addr, · line 2427
Quoted 5 lines
mm/khugepaged.c +3
-0+end: +return result; +}There's a LOT of nesting going on here, I think we can simplify this a lot. If we make the change I noted above re: returning SCAN_ANY_PROCESS< we can move the end label up a bit and avoid a ton of nesting, e.g.:
Ah I like this much more, I will try to implement/test it.
Quoted 42 lines
static int collapse_single_pmd(unsigned long addr,
structvm_area_struct*vma, bool*mmap_locked, structcollapse_control*cc){ struct mm_struct *mm = vma->vm_mm; struct file *file; pgoff_t pgoff; int result; if (vma_is_anonymous(vma)) { result = collapse_scan_pmd(mm, vma, addr, mmap_locked, cc); goto end: } file = get_file(vma->vm_file); pgoff = linear_page_index(vma, addr); mmap_read_unlock(mm); *mmap_locked = false; result = collapse_scan_file(mm, addr, file, pgoff, cc); fput(file); if (result != SCAN_PTE_MAPPED_HUGEPAGE) goto end; mmap_read_lock(mm); *mmap_locked = true; if (collapse_test_exit_or_disable(mm)) { mmap_read_unlock(mm); *mmap_locked = false; return SCAN_ANY_PROCESS; } result = collapse_pte_mapped_thp(mm, addr, !cc->is_khugepaged); if (result == SCAN_PMD_MAPPED) result = SCAN_SUCCEED; mmap_read_unlock(mm); *mmap_locked = false;end:
if (cc->is_khugepaged && result == SCAN_SUCCEED) ++khugepaged_pages_collapsed;
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
return result; }(untested, thrown together so do double check!)
This suggested refactoring work for you?
Looks correct, I'm going to implement all the changes then test to make sure it works as intended.
Quoted 109 lines
Quoted 89 lines
Quoted 89 lines
Quoted 51 lines
mm/khugepaged.c +3
-30+ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)@@ -2455,34 +2499,9 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; progress +=HPAGE_PMD_NR ;@@ -2799,34 +2818,19 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_assert_locked (mm);memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr);mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2823
Quoted 10 lines
mm/khugepaged.c +2
-8-mmap_read_unlock (mm); - mmap_locked = false; - result =collapse_scan_file (mm, addr, file, pgoff, cc); -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); - } + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); +Ack the fact you noted the behaviour change re:
collapse_test_exit_or_disable()that seems fine.Quoted 5 lines
mm/khugepaged.c +0
-1if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2828
Quoted 10 lines
mm/khugepaged.c +0
-6case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;One thing that differs with new code her is we filter
SCAN_PMD_MAPPEDtoSCAN_SUCCEED.I was about to say 'but ++thps - is this correct' but now I realise this was looping back on itself with a goto to do just that... ugh ye gads.
Anwyay that's fine because it doesn't change anything.
Quoted 7 lines
Quoted 3 lines
Re: switch statement in general, again would be good to always have each scan possibility in switch statements, but perhaps given so many not practical :)
Yeah it may be worth investigating for future changes I have for khugepaged (including the new switch statement I implement later and you commented on)
Ack yeah this can be one for the future!
Quoted 10 lines
Quoted 9 lines
(that way the compiler warns on missing a newly added enum val)
Quoted 6 lines
mm/khugepaged.c +1
-1/* Whitelisted set of results where continuing OK */ +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PMD_NULL :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP : --
Quoted 4 lines
Quoted 1 line
Thanks for the review :)
No probs, to underline as well - the critique is to make sure we get this right, my aim here is to get your series landed in as good a form as possible :)
All critiquing is welcome and appreciated :) The refactoring looks much better now too!
Cheers,
-- Nico
Quoted 10 lines
Quoted 7 lines
-- Nico
Quoted 3 lines
Quoted 2 lines
2.50.1
Cheers, Lorenzo
PATCH v10 3/13 khugepaged: generalize hugepage_vma_revalidate for mTHP support
In reply to Nico PacheFor khugepaged to support different mTHP orders, we must generalize this to check if the PMD is not shared by another VMA and the order is enabled.
To ensure madvise_collapse can support working on mTHP orders without the PMD order enabled, we need to convert hugepage_vma_revalidate to take a bitmap of orders.
No functional change in this patch.
--- mm/khugepaged.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
mm/khugepaged.c +8 -5
@@ -917,7 +917,7 @@ static int collapse_find_target_node(struct collapse_control *cc)static int hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap, -struct collapse_control *cc) +struct collapse_control *cc,unsigned long orders) {struct vm_area_struct *vma; enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
@@ -930,9 +930,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!vma)return SCAN_VMA_NULL ; +/* Always check the PMD order to insure its not shared by another VMA */ if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then
@@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;
@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */
@@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock;
Signature · 1 line
2.50.1
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
Quoted 7 lines · Nico Pache
For khugepaged to support different mTHP orders, we must generalize this to check if the PMD is not shared by another VMA and the order is enabled.
To ensure
madvise_collapsecan support working on mTHP orders without the PMD order enabled, we need to converthugepage_vma_revalidateto take a bitmap of orders.
Quoted 7 lines
No functional change in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>
LGTM (modulo nit/query below) so:
Quoted 18 lines
--- mm/khugepaged.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)mm/khugepaged.c +1
-1@@ -917,7 +917,7 @@ static int collapse_find_target_node(struct collapse_control *cc)static int hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap, -struct collapse_control *cc) +struct collapse_control *cc,unsigned long orders) {struct vm_area_struct *vma; enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
mm/khugepaged.c static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, · line 933
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -930,9 +930,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!vma)return SCAN_VMA_NULL ; +/* Always check the PMD order to insure its not shared by another VMA */
NIT: ensure not insure.
Quoted 8 lines
mm/khugepaged.c +1
-1if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1139
Quoted 6 lines
mm/khugepaged.c +2
-1@@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));
Shouldn't this be PMD order? Seems equivalent.
Quoted 25 lines
mm/khugepaged.c +3
-3if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock; --2.50.1
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
Quoted 17 lines · Lorenzo Stoakes
Quoted 7 lines · Nico Pache
For khugepaged to support different mTHP orders, we must generalize this to check if the PMD is not shared by another VMA and the order is enabled.
To ensure
madvise_collapsecan support working on mTHP orders without the PMD order enabled, we need to converthugepage_vma_revalidateto take a bitmap of orders.Quoted 7 lines
No functional change in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>
Quoted 3 lines
LGTM (modulo nit/query below) so:
Reviewed-by Lorenzo Stoakes <[email protected]>
Thanks :)
Quoted 22 lines
Quoted 22 lines
--- mm/khugepaged.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)mm/khugepaged.c +1
-1@@ -917,7 +917,7 @@ static int collapse_find_target_node(struct collapse_control *cc)static int hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap, -struct collapse_control *cc) +struct collapse_control *cc,unsigned long orders) {struct vm_area_struct *vma; enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :@@ -930,9 +930,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!vma)return SCAN_VMA_NULL ;
Quoted 3 lines
mm/khugepaged.c static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, · line 933
Quoted 1 line
mm/khugepaged.c +1
-0+/* Always check the PMD order to insure its not shared by another VMA */ NIT: ensure not insure.
ack, ill fix that!
Quoted 11 lines
Quoted 11 lines
mm/khugepaged.c +1
-1if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then @@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;
Quoted 6 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1139
Quoted 4 lines
mm/khugepaged.c +2
-1mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));Shouldn't this be PMD order? Seems equivalent.
Yeah i'm actually not sure why we have both... they seem to be the same thing, but perhaps there is some reason for having two...
Quoted 27 lines
Quoted 26 lines
mm/khugepaged.c +3
-3if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock; --2.50.1
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
On Wed, Aug 20, 2025 at 09:40:40AM -0600, Nico Pache wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
if (!thp_vma_suitable_order(vma, address,
PMD_ORDER))
returnSCAN_ADDRESS_RANGE;mm/khugepaged.c +1
-1-if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then @@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;
Quoted 8 lines
Quoted 6 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1139
Quoted 4 lines
mm/khugepaged.c +2
-1mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));Shouldn't this be PMD order? Seems equivalent.
Yeah i'm actually not sure why we have both... they seem to be the same thing, but perhaps there is some reason for having two...
I am confused with these two, PMD_ORDER above and HPAGE_PMD_ORDER from here.
Do we have a guide on when to use which?
Quoted 28 lines
Quoted 27 lines
Quoted 26 lines
mm/khugepaged.c +3
-3if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock; --2.50.1
Signature · 2 lines
Wei Yang Help you, Help me
Zi Yan
reviewerRe: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
Quoted 19 lines · Wei Yang
On Wed, Aug 20, 2025 at 09:40:40AM -0600, Nico Pache wrote:
[...]Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
if (!thp_vma_suitable_order(vma, address,
PMD_ORDER))
returnSCAN_ADDRESS_RANGE;mm/khugepaged.c +1
-1-if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1139
Quoted 6 lines
mm/khugepaged.c +2
-1@@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));Shouldn't this be PMD order? Seems equivalent.
Quoted 6 lines
Quoted 2 lines
Yeah i'm actually not sure why we have both... they seem to be the same thing, but perhaps there is some reason for having two...
I am confused with these two,
PMD_ORDERabove andHPAGE_PMD_ORDERfrom here.Do we have a guide on when to use which?
Looking at the definition of HPAGE_PMD_SHIFT in huge_mm.h, it will cause a build bug when PMD level huge page is not supported. So I think HPAGE_PMD_ORDER should be used for all huge pages (both THP and hugetlb).
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
mm/khugepaged.c +3
-2if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));
Quoted 12 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
mm/khugepaged.c +0
-1if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock; --2.50.1
Signature · 2 lines
Wei Yang Help you, Help me
--
Best Regards,
Yan, Zi
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
Quoted 24 lines · Zi Yan
Quoted 22 lines · Wei Yang
On Wed, Aug 20, 2025 at 09:40:40AM -0600, Nico Pache wrote:
[...]Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
if (!thp_vma_suitable_order(vma, address,
PMD_ORDER))
returnSCAN_ADDRESS_RANGE;mm/khugepaged.c +1
-1-if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then @@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;Quoted 6 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1139
Quoted 4 lines
mm/khugepaged.c +2
-1mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));Shouldn't this be PMD order? Seems equivalent.
Yeah i'm actually not sure why we have both... they seem to be the same thing, but perhaps there is some reason for having two...
Quoted 7 lines
Quoted 3 lines
I am confused with these two,
PMD_ORDERabove andHPAGE_PMD_ORDERfrom here.Do we have a guide on when to use which?
Looking at the definition of
HPAGE_PMD_SHIFTinhuge_mm.h, it will cause a build bug when PMD level huge page is not supported. So I thinkHPAGE_PMD_ORDERshould be used for all huge pages (both THP and hugetlb).
Thanks yeah that makes sense, so let's keep it all as HPAGE_PMD_ORDER then!
Cheers, Lorenzo
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
In reply to Nico PacheHi, Nico
Some nit below.
mm/khugepaged.c
Quoted 22 lines · Nico Pache
For khugepaged to support different mTHP orders, we must generalize this to check if the PMD is not shared by another VMA and the order is enabled.
To ensure
madvise_collapsecan support working on mTHP orders without the PMD order enabled, we need to converthugepage_vma_revalidateto take a bitmap of orders.No functional change in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>---
mm/khugepaged.c| 13 ++++++++-----1 file changed, 8 insertions(+), 5 deletions(-)mm/khugepaged.c +0
-0
There is a comment above this function, which says "revalidate vma before taking mmap_lock".
I am afraid it is "after taking mmap_lock"? or "after taking mmap_lock again"?
Quoted 11 lines
mm/khugepaged.c +1
-1@@ -917,7 +917,7 @@ static int collapse_find_target_node(struct collapse_control *cc)static int hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap, -struct collapse_control *cc) +struct collapse_control *cc,unsigned long orders) {struct vm_area_struct *vma; enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :@@ -930,9 +930,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!vma)
mm/khugepaged.c static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, · line 939
Quoted 10 lines
mm/khugepaged.c +2
-1return SCAN_VMA_NULL ; +/* Always check the PMD order to insure its not shared by another VMA */ if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then
Below is a comment, "thp_vma_allowable_order may return".
Since you use thp_vma_allowable_orders, maybe we need to change the comment too.
Quoted 32 lines
mm/khugepaged.c +5
-3@@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock;Signature · 1 line
2.50.1
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH v10 03/13] khugepaged: generalize hugepage_vma_revalidate for mTHP support
Quoted 32 lines · Wei Yang
Hi, Nico
Some nit below.
mm/khugepaged.c
Quoted 22 lines · Nico Pache
For khugepaged to support different mTHP orders, we must generalize this to check if the PMD is not shared by another VMA and the order is enabled.
To ensure
madvise_collapsecan support working on mTHP orders without the PMD order enabled, we need to converthugepage_vma_revalidateto take a bitmap of orders.No functional change in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>---
mm/khugepaged.c| 13 ++++++++-----1 file changed, 8 insertions(+), 5 deletions(-)mm/khugepaged.c +0
-0There is a comment above this function, which says "revalidate vma before taking
mmap_lock".
Quoted 1 line
I am afraid it is "after taking
mmap_lock"? or "after takingmmap_lockagain"?
Good catch, never noticed that. I updated the comment!
Quoted 25 lines
Quoted 12 lines
mm/khugepaged.c +1
-1@@ -917,7 +917,7 @@ static int collapse_find_target_node(struct collapse_control *cc)static int hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap, -struct collapse_control *cc) +struct collapse_control *cc,unsigned long orders) {struct vm_area_struct *vma; enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :@@ -930,9 +930,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,if (!vma)mm/khugepaged.c static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, · line 939
Quoted 10 lines
mm/khugepaged.c +2
-1return SCAN_VMA_NULL ; +/* Always check the PMD order to insure its not shared by another VMA */ if (!thp_vma_suitable_order (vma, address,PMD_ORDER ))return SCAN_ADDRESS_RANGE ; -if (!thp_vma_allowable_order (vma, vma->vm_flags, type,PMD_ORDER )) +if (!thp_vma_allowable_orders (vma, vma->vm_flags, type, orders))return SCAN_VMA_CHECK ;/* * Anon VMA expected, the address may be unmapped then Below is a comment, "
thp_vma_allowable_ordermay return".
Quoted 2 lines
Since you use
thp_vma_allowable_orders, maybe we need to change the comment too.
Ack! Thanks for the review!
Quoted 38 lines
Quoted 31 lines
mm/khugepaged.c +5
-4@@ -1134,7 +1135,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1168,7 +1170,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc, +BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ @@ -2807,7 +2810,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_lock (mm); mmap_locked = true; result =hugepage_vma_revalidate (mm, addr, false, &vma, - cc); + cc,BIT (HPAGE_PMD_ORDER ));if (result !=SCAN_SUCCEED ) { last_fail = result; goto out_nolock; --Quoted 2 lines
2.50.1
--
Wei Yang
Help you, Help me
PATCH v10 4/13 khugepaged: generalize alloc_charge_folio()
In reply to Nico PacheFrom: Dev Jain <[email protected]>
Pass order to alloc_charge_folio() and update mTHP statistics.
--- Documentation/admin-guide/mm/transhuge.rst | 8 ++++++++ include/linux/huge_mm.h | 2 ++ mm/huge_memory.c | 4 ++++ mm/khugepaged.c | 17 +++++++++++------ 4 files changed, 25 insertions(+), 6 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +8 -0
@@ -630,6 +630,14 @@ anon_fault_fallback_charge instead falls back to using huge pages with lower orders or small pages even though the allocation was successful. +collapse_alloc + is incremented every time a huge page is successfully allocated for a + khugepaged collapse. + +collapse_alloc_failed + is incremented every time a huge page allocation fails during a + khugepaged collapse. + zswpout is incremented every time a huge page is swapped out to zswap in one piece without splitting.
include/linux/huge_mm.h +2 -0
@@ -128,6 +128,8 @@ enum mthp_stat_item {MTHP_STAT_ANON_FAULT_ALLOC ,MTHP_STAT_ANON_FAULT_FALLBACK ,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE , +MTHP_STAT_COLLAPSE_ALLOC , +MTHP_STAT_COLLAPSE_ALLOC_FAILED ,MTHP_STAT_ZSWPOUT ,MTHP_STAT_SWPIN , MTHP_STAT_SWPIN_FALLBACK,
mm/huge_memory.c +4 -0
@@ -621,6 +621,8 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)DEFINE_MTHP_STAT_ATTR (anon_fault_alloc,MTHP_STAT_ANON_FAULT_ALLOC );DEFINE_MTHP_STAT_ATTR (anon_fault_fallback,MTHP_STAT_ANON_FAULT_FALLBACK );DEFINE_MTHP_STAT_ATTR (anon_fault_fallback_charge,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE ); +DEFINE_MTHP_STAT_ATTR (collapse_alloc,MTHP_STAT_COLLAPSE_ALLOC ); +DEFINE_MTHP_STAT_ATTR (collapse_alloc_failed,MTHP_STAT_COLLAPSE_ALLOC_FAILED );DEFINE_MTHP_STAT_ATTR (zswpout,MTHP_STAT_ZSWPOUT );DEFINE_MTHP_STAT_ATTR (swpin,MTHP_STAT_SWPIN );DEFINE_MTHP_STAT_ATTR (swpin_fallback,MTHP_STAT_SWPIN_FALLBACK );
@@ -686,6 +688,8 @@ static struct attribute *any_stats_attrs[] = {#endif &split_attr.attr, &split_failed_attr.attr, + &collapse_alloc_attr.attr, + &collapse_alloc_failed_attr.attr, NULL, };
mm/khugepaged.c +11 -6
@@ -1079,21 +1079,26 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }static int alloc_charge_folio (struct folio **foliop,struct mm_struct *mm, -struct collapse_control *cc) +struct collapse_control *cc,unsigned int order) {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE );int node =collapse_find_target_node (cc);struct folio *folio; - folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask); + folio =__folio_alloc (gfp, order, node, &cc->alloc_nmask);if (!folio) { *foliop =NULL ; -count_vm_event (THP_COLLAPSE_ALLOC_FAILED ); +if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_COLLAPSE_ALLOC_FAILED ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_ALLOC_FAILED );return SCAN_ALLOC_HUGE_PAGE_FAIL ; } -count_vm_event (THP_COLLAPSE_ALLOC ); +if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_COLLAPSE_ALLOC ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_ALLOC ); +if (unlikely (mem_cgroup_charge (folio, mm, gfp))) {folio_put (folio); *foliop =NULL ;
@@ -1130,7 +1135,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,*/ mmap_read_unlock (mm); - result =alloc_charge_folio (&folio, mm, cc); + result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out_nolock;
@@ -1863,7 +1868,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,VM_BUG_ON (!IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && !is_shmem);VM_BUG_ON (start & (HPAGE_PMD_NR -1 )); - result =alloc_charge_folio (&new_folio, mm, cc); + result =alloc_charge_folio (&new_folio, mm, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out;
Signature · 1 line
2.50.1
Re: [PATCH v10 04/13] khugepaged: generalize alloc_charge_folio()
Quoted 9 lines · Nico Pache
From: Dev Jain <[email protected]>
Pass order to
alloc_charge_folio()and update mTHP statistics.Reviewed-by Baolin Wang <[email protected]>Co-developed-by Nico Pache <[email protected]>Signed-off-by Nico Pache <[email protected]>Signed-off-by Dev Jain <[email protected]>
LGTM so:
Quoted 117 lines
--- Documentation/admin-guide/mm/transhuge.rst | 8 ++++++++ include/linux/huge_mm.h | 2 ++ mm/huge_memory.c | 4 ++++ mm/khugepaged.c | 17 +++++++++++------ 4 files changed, 25 insertions(+), 6 deletions(-)Documentation/admin-guide/mm/transhuge.rst +8
-0@@ -630,6 +630,14 @@ anon_fault_fallback_charge instead falls back to using huge pages with lower orders or small pages even though the allocation was successful. +collapse_alloc + is incremented every time a huge page is successfully allocated for a + khugepaged collapse. + +collapse_alloc_failed + is incremented every time a huge page allocation fails during a + khugepaged collapse. + zswpout is incremented every time a huge page is swapped out to zswap in one piece without splitting.include/linux/huge_mm.h +2
-0@@ -128,6 +128,8 @@ enum mthp_stat_item {MTHP_STAT_ANON_FAULT_ALLOC ,MTHP_STAT_ANON_FAULT_FALLBACK ,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE , +MTHP_STAT_COLLAPSE_ALLOC , +MTHP_STAT_COLLAPSE_ALLOC_FAILED ,MTHP_STAT_ZSWPOUT ,MTHP_STAT_SWPIN , MTHP_STAT_SWPIN_FALLBACK,mm/huge_memory.c +4
-0@@ -621,6 +621,8 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)DEFINE_MTHP_STAT_ATTR (anon_fault_alloc,MTHP_STAT_ANON_FAULT_ALLOC );DEFINE_MTHP_STAT_ATTR (anon_fault_fallback,MTHP_STAT_ANON_FAULT_FALLBACK );DEFINE_MTHP_STAT_ATTR (anon_fault_fallback_charge,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE ); +DEFINE_MTHP_STAT_ATTR (collapse_alloc,MTHP_STAT_COLLAPSE_ALLOC ); +DEFINE_MTHP_STAT_ATTR (collapse_alloc_failed,MTHP_STAT_COLLAPSE_ALLOC_FAILED );DEFINE_MTHP_STAT_ATTR (zswpout,MTHP_STAT_ZSWPOUT );DEFINE_MTHP_STAT_ATTR (swpin,MTHP_STAT_SWPIN );DEFINE_MTHP_STAT_ATTR (swpin_fallback,MTHP_STAT_SWPIN_FALLBACK );@@ -686,6 +688,8 @@ static struct attribute *any_stats_attrs[] = {#endif &split_attr.attr, &split_failed_attr.attr, + &collapse_alloc_attr.attr, + &collapse_alloc_failed_attr.attr, NULL, };mm/khugepaged.c +11
-7@@ -1079,21 +1079,26 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }static int alloc_charge_folio (struct folio **foliop,struct mm_struct *mm, -struct collapse_control *cc) +struct collapse_control *cc,unsigned int order) {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE );int node =collapse_find_target_node (cc);struct folio *folio; - folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask); + folio =__folio_alloc (gfp, order, node, &cc->alloc_nmask);if (!folio) { *foliop =NULL ; -count_vm_event (THP_COLLAPSE_ALLOC_FAILED ); +if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_COLLAPSE_ALLOC_FAILED ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_ALLOC_FAILED );return SCAN_ALLOC_HUGE_PAGE_FAIL ; } -count_vm_event (THP_COLLAPSE_ALLOC ); +if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_COLLAPSE_ALLOC ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_ALLOC ); +if (unlikely (mem_cgroup_charge (folio, mm, gfp))) {folio_put (folio); *foliop =NULL ;@@ -1130,7 +1135,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,*/ mmap_read_unlock (mm); - result =alloc_charge_folio (&folio, mm, cc); + result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out_nolock;@@ -1863,7 +1868,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,VM_BUG_ON (!IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && !is_shmem);VM_BUG_ON (start & (HPAGE_PMD_NR -1 )); - result =alloc_charge_folio (&new_folio, mm, cc); + result =alloc_charge_folio (&new_folio, mm, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out; --2.50.1
PATCH v10 5/13 khugepaged: generalize __collapse_huge_page_* for mTHP support
In reply to Nico Pachegeneralize the order of the __collapse_huge_page_* functions to support future mTHP collapse.
mTHP collapse can suffer from incosistant behavior, and memory waste "creep". disable swapin and shared support for mTHP collapse.
No functional changes in this patch.
--- mm/khugepaged.c | 62 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 43 insertions(+), 19 deletions(-)
mm/khugepaged.c +43 -19
@@ -551,15 +551,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order) { struct page *page = NULL;struct folio *folio =NULL ;pte_t *_pte;int none_or_zero =0 , shared =0 , result =SCAN_FAIL , referenced =0 ;bool writable = false; +int scaled_max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; +for (_pte = pte; _pte < pte + (1 << order); _pte++, address +=PAGE_SIZE ) {pte_t pteval =ptep_get (_pte); if (pte_none(pteval) || (pte_present(pteval) &&
@@ -567,7 +569,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, ++none_or_zero;if (!userfaultfd_armed (vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= scaled_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;
@@ -595,8 +597,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; -if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { +/* +* TODO: Support shared pages without leading to further +* mTHP collapses. Currently bringing in new pages via +* shared may cause a future higher order collapse on a +* rescan of the same range. +*/ +if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared)) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); goto out;
@@ -697,15 +705,16 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order) { - unsigned long end = address + HPAGE_PMD_SIZE; + unsigned long end = address + (PAGE_SIZE << order);struct folio *src, *tmp;pte_t pteval;pte_t *_pte;unsigned int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte += nr_ptes, +for (_pte = pte; _pte < pte + (1 << order); _pte += nr_ptes, address += nr_ptes *PAGE_SIZE ) { nr_ptes =1 ; pteval =ptep_get (_pte);
@@ -761,7 +770,8 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order) { spinlock_t *pmd_ptl;
@@ -778,7 +788,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,* Release both raw and compound pages isolated * in __collapse_huge_page_isolate. */ -release_pte_pages (pte, pte +HPAGE_PMD_NR , compound_pagelist); +release_pte_pages (pte, pte + (1 << order), compound_pagelist); }/*
@@ -799,7 +809,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,static int __collapse_huge_page_copy (pte_t *pte,struct folio *folio,pmd_t *pmd,pmd_t orig_pmd,struct vm_area_struct *vma,unsigned long address,spinlock_t *ptl, -struct list_head *compound_pagelist) +struct list_head *compound_pagelist,unsigned int order) {unsigned int i;int result =SCAN_SUCCEED ;
@@ -807,7 +817,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,/* * Copying pages' contents is subject to memory poison at any iteration. */ -for (i =0 ; i <HPAGE_PMD_NR ; i++) { +for (i =0 ; i < (1 << order); i++) {pte_t pteval =ptep_get (pte + i);struct page *page =folio_page (folio, i);unsigned long src_addr = address + i *PAGE_SIZE ;
@@ -826,10 +836,10 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,if (likely (result ==SCAN_SUCCEED ))__collapse_huge_page_copy_succeeded (pte, vma, address, ptl, - compound_pagelist); + compound_pagelist, order);else __collapse_huge_page_copy_failed (pte, pmd, orig_pmd, vma, - compound_pagelist); + compound_pagelist, order);return result; }
@@ -1005,11 +1015,11 @@ static int check_pmd_still_valid(struct mm_struct *mm,static int __collapse_huge_page_swapin (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long haddr,pmd_t *pmd, -int referenced) +int referenced,unsigned int order) {int swapped_in =0 ;vm_fault_t ret =0 ; -unsigned long address, end = haddr + (HPAGE_PMD_NR *PAGE_SIZE ); +unsigned long address, end = haddr + (PAGE_SIZE << order);int result;pte_t *pte =NULL ;spinlock_t *ptl;
@@ -1040,6 +1050,19 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +/* +* TODO: Support swapin without leading to further mTHP +* collapses. Currently bringing in new pages via swapin may +* cause a future higher order collapse on a rescan of the same +* range. +*/ +if (order !=HPAGE_PMD_ORDER ) { +pte_unmap (pte); +mmap_read_unlock (mm); + result =SCAN_EXCEED_SWAP_PTE ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);
@@ -1160,7 +1183,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* that case. Continuing to collapse causes inconsistency. */ result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced); + referenced,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out_nolock; }
@@ -1208,7 +1231,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl);if (pte) { result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist); + &compound_pagelist, +HPAGE_PMD_ORDER );spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ;
@@ -1238,7 +1262,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, vma, address, pte_ptl, - &compound_pagelist); + &compound_pagelist,HPAGE_PMD_ORDER );pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) goto out_up_write;
Signature · 1 line
2.50.1
Re: [PATCH v10 05/13] khugepaged: generalize __collapse_huge_page_* for mTHP support
Quoted 22 lines · Nico Pache
generalize the order of the __collapse_huge_page_* functions to support future mTHP collapse.
mTHP collapse can suffer from incosistant behavior, and memory waste "creep". disable swapin and shared support for mTHP collapse.
No functional changes in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 62 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 43 insertions(+), 19 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 555
Quoted 6 lines
mm/khugepaged.c +2
-1@@ -551,15 +551,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order)
I think it's better if we keep the output var as the last in the order. It's a bit weird to have the order specified here.
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 562
Quoted 7 lines
mm/khugepaged.c +1
-0{struct page *page =NULL ;struct folio *folio =NULL ;pte_t *_pte;int none_or_zero =0 , shared =0 , result =SCAN_FAIL , referenced =0 ;bool writable = false; +int scaled_max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order);
This is a weird formulation, I guess we have to go with it to keep things consistent-ish, but it's like we have a value for this that is reliant on the order always being PMD and then sort of awkwardly adjusting for MTHP.
I guess we're stuck with it though since we have:
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
I guess a more sane version of this would be a ratio or something...
Anyway probably out of scope here.
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 564
Quoted 3 lines
mm/khugepaged.c +1
-1- for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_pte = pte; _pte < pte + (1 << order);
Hmm is this correct? I think shifting an int is probably a bad idea even if we can get away with it for even PUD order atm (though... 64KB ARM hm), wouldn't _BITUL(order) be better?
Might also be worth putting into a separate local var.
Quoted 14 lines
mm/khugepaged.c +1
-1_pte++, address += PAGE_SIZE) {pte_t pteval =ptep_get (_pte); if (pte_none(pteval) || (pte_present(pteval) &&@@ -567,7 +569,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, ++none_or_zero;if (!userfaultfd_armed (vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= scaled_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -595,8 +597,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,/* See collapse_scan_pmd(). */
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 605
Quoted 10 lines
mm/khugepaged.c +6
-2if (folio_maybe_mapped_shared (folio)) { ++shared; -if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { +/* +* TODO: Support shared pages without leading to further +* mTHP collapses. Currently bringing in new pages via +* shared may cause a future higher order collapse on a +* rescan of the same range. +*/
Can we document this if not already in a subsequent commit? :) Thanks
Quoted 12 lines
mm/khugepaged.c +4
-1+if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared)) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); goto out;@@ -697,15 +705,16 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order)
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 717
Quoted 10 lines
mm/khugepaged.c +2
-2{ -unsigned long end = address +HPAGE_PMD_SIZE ; +unsigned long end = address + (PAGE_SIZE << order);struct folio *src, *tmp;pte_t pteval;pte_t *_pte;unsigned int nr_ptes; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes, + for (_pte = pte; _pte < pte + (1 << order); _pte += nr_ptes,
Same comment as above re: 1 << order.
mm/khugepaged.c static void __collapse_huge_page_copy_failed(pte_t *pte, · line 774
Quoted 10 lines
mm/khugepaged.c +2
-1address += nr_ptes * PAGE_SIZE) { nr_ptes =1 ; pteval =ptep_get (_pte);@@ -761,7 +770,8 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order)
Same comment as above re: parameter ordering.
Quoted 13 lines
mm/khugepaged.c +1
-1{spinlock_t *pmd_ptl;@@ -778,7 +788,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,* Release both raw and compound pages isolated * in __collapse_huge_page_isolate. */ -release_pte_pages (pte, pte +HPAGE_PMD_NR , compound_pagelist); +release_pte_pages (pte, pte + (1 << order), compound_pagelist); }/*
mm/khugepaged.c static void __collapse_huge_page_copy_failed(pte_t *pte, · line 812
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -799,7 +809,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,static int __collapse_huge_page_copy (pte_t *pte,struct folio *folio,pmd_t *pmd,pmd_t orig_pmd,struct vm_area_struct *vma,unsigned long address,spinlock_t *ptl, -struct list_head *compound_pagelist) +struct list_head *compound_pagelist,unsigned int order)
Same comment as before re: parameter ordering
mm/khugepaged.c static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, · line 820
Quoted 9 lines
mm/khugepaged.c +1
-1{unsigned int i;int result =SCAN_SUCCEED ;@@ -807,7 +817,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,/* * Copying pages' contents is subject to memory poison at any iteration. */ -for (i =0 ; i <HPAGE_PMD_NR ; i++) { +for (i =0 ; i < (1 << order); i++) {
Same comment as before about 1 << order
Quoted 81 lines
mm/khugepaged.c +21
-8pte_t pteval =ptep_get (pte + i);struct page *page =folio_page (folio, i);unsigned long src_addr = address + i *PAGE_SIZE ;@@ -826,10 +836,10 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,if (likely (result ==SCAN_SUCCEED ))__collapse_huge_page_copy_succeeded (pte, vma, address, ptl, - compound_pagelist); + compound_pagelist, order);else __collapse_huge_page_copy_failed (pte, pmd, orig_pmd, vma, - compound_pagelist); + compound_pagelist, order);return result; }@@ -1005,11 +1015,11 @@ static int check_pmd_still_valid(struct mm_struct *mm,static int __collapse_huge_page_swapin (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long haddr,pmd_t *pmd, -int referenced) +int referenced,unsigned int order) {int swapped_in =0 ;vm_fault_t ret =0 ; -unsigned long address, end = haddr + (HPAGE_PMD_NR *PAGE_SIZE ); +unsigned long address, end = haddr + (PAGE_SIZE << order);int result;pte_t *pte =NULL ;spinlock_t *ptl;@@ -1040,6 +1050,19 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,if (!is_swap_pte (vmf.orig_pte))continue ; +/* +* TODO: Support swapin without leading to further mTHP +* collapses. Currently bringing in new pages via swapin may +* cause a future higher order collapse on a rescan of the same +* range. +*/ +if (order !=HPAGE_PMD_ORDER ) { +pte_unmap (pte); +mmap_read_unlock (mm); + result =SCAN_EXCEED_SWAP_PTE ; + goto out; + } + vmf.pte = pte; vmf.ptl = ptl; ret =do_swap_page (&vmf);@@ -1160,7 +1183,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* that case. Continuing to collapse causes inconsistency. */ result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced); + referenced,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) goto out_nolock; }@@ -1208,7 +1231,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl);if (pte) { result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist); + &compound_pagelist, +HPAGE_PMD_ORDER );spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ;@@ -1238,7 +1262,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, vma, address, pte_ptl, - &compound_pagelist); + &compound_pagelist,HPAGE_PMD_ORDER );pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) goto out_up_write; --2.50.1
Re: [PATCH v10 05/13] khugepaged: generalize __collapse_huge_page_* for mTHP support
Quoted 6 lines · Lorenzo Stoakes
Quoted 5 lines · Nico Pache
generalize the order of the __collapse_huge_page_* functions to support future mTHP collapse.
mTHP collapse can suffer from incosistant behavior, and memory waste "creep". disable swapin and shared support for mTHP collapse.
I'd just note that mTHP collapse will initially not honor these two parameters (spell them out), failing if anything is swapped out or shared.
Quoted 12 lines
Quoted 12 lines
No functional changes in this patch.
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 62 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 43 insertions(+), 19 deletions(-)
Quoted 14 lines
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 555
Quoted 11 lines
mm/khugepaged.c +2
-1@@ -551,15 +551,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, + unsigned int order)I think it's better if we keep the output var as the last in the order. It's a bit weird to have the order specified here.
Also, while at it, just double-tab indent.
Quoted 13 lines
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 562
Quoted 8 lines
mm/khugepaged.c +1
-0{struct page *page =NULL ;struct folio *folio =NULL ;pte_t *_pte;int none_or_zero =0 , shared =0 , result =SCAN_FAIL , referenced =0 ;bool writable = false; +int scaled_max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order);This is a weird formulation, I guess we have to go with it to keep things consistent-ish, but it's like we have a value for this that is reliant on the order always being PMD and then sort of awkwardly adjusting for MTHP.
Quoted 5 lines
I guess we're stuck with it though since we have:
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneI guess a more sane version of this would be a ratio or something...
Yeah, ratios would have made much more sense ... but people didn't plan for having something that is not a PMD size.
Quoted 10 lines
Anyway probably out of scope here.
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 564
Quoted 3 lines
mm/khugepaged.c +1
-1- for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_pte = pte; _pte < pte + (1 << order);Hmm is this correct? I think shifting an int is probably a bad idea even if we can get away with it for even PUD order atm (though... 64KB ARM hm), wouldn't
_BITUL(order)be better?
Just for completeness: we discussed that recently in other context. It would better be BIT() instead of _BITUL().
But I am not a fan of using BIT() when not working with bitmaps etc.
"1ul << order" etc is used throughout the code base.
What makes this easier to read is:
const unsigned long nr_pages = 1ul << order;
Maybe in the future we want an ORDER_PAGES(), maybe not. Have not made up my mind yet :)
Signature · 3 lines
Cheers David / dhildenb
PATCH v10 6/13 khugepaged: add mTHP support
In reply to Nico PacheIntroduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in KHUGEPAGED_MIN_MTHP_ORDER chunks via a bitmap. Each bit represents a utilized region of order KHUGEPAGED_MIN_MTHP_ORDER ptes. If mTHPs are enabled we remove the restriction of max_ptes_none during the scan phase so we don't bailout early and miss potential mTHP candidates.
A new function collapse_scan_bitmap is used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion. max_ptes_none will be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.
Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "max_ptes_none > HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently madv_collapse is not supported and will only attempt PMD collapse.
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)
include/linux/khugepaged.h +4 -0
@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2 +#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER) +#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER)) +#define MTHP_BITMAP_SIZE (1 << (HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER))#include <linux/mm.h>
mm/khugepaged.c +184 -52
397 lines not on this page · raw message
Signature · 1 line
2.50.1
Re: [PATCH v10 06/13] khugepaged: add mTHP support
Quoted 28 lines · Nico Pache
Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in
KHUGEPAGED_MIN_MTHP_ORDERchunks via a bitmap. Each bit represents a utilized region of orderKHUGEPAGED_MIN_MTHP_ORDERptes. If mTHPs are enabled we remove the restriction ofmax_ptes_noneduring the scan phase so we don't bailout early and miss potential mTHP candidates.A new function
collapse_scan_bitmapis used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion.max_ptes_nonewill be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "
max_ptes_none> HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.
Quoted 7 lines
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently
madv_collapseis not supported and will only attempt PMD collapse.
Yes I think this has to remain the case unfortunately as we override sysfs-specified orders for MADV_COLLAPSE and there's no sensible way to determine what order we ought to be using.
Quoted 2 lines
Signed-off-by Nico Pache <[email protected]>
You've gone from small incremental changes to a huge one here... for the sake of reviewer sanity at least, any chance of breaking this up?
Quoted 10 lines
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)include/linux/khugepaged.h +0
-0
include/linux/khugepaged.h line 4
Quoted 4 lines
include/linux/khugepaged.h +1
-0@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2
I guess this makes sense as by definition 2 pages is least it could possibly be.
include/linux/khugepaged.h line 5
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER)
Surely KHUGEPAGED_MIN_NR_MTHP_PTES would be more meaningful?
include/linux/khugepaged.h line 6
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))
This is confusing - size of what?
If it's number of bits surely this should be ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER?
This seems to be more so 'the maximum value that could contain the bits right?
I think this is just wrong though, see below at DECLARE_BITMAP() stuff.
include/linux/khugepaged.h line 7
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define MTHP_BITMAP_SIZE (1 << (HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER))
Hard to know how this relates to MAX_MTHP_BITMAP_SIZE?
I guess this is the current bitmap size indicating all that is possible, but if these are all #define's what is this accomplishing?
For all - please do not do (1 << xxx)! This can lead to sign-extension bugs at least in theory, use _BITUL(...), it's neater too.
NIT but the whitespace is all screwed up here.
KHUGEPAGED_MIN_MTHP_ORDER and KHUGEPAGED_MIN_MTHP_NR
Quoted 20 lines
include/linux/khugepaged.h +0
-0#include <linux/mm.h> mm/khugepaged.c +5
-0@@ -94,6 +94,11 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);static struct kmem_cache *mm_slot_cache__ro_after_init ; +struct scan_bit_state { +u8 order; +u16 offset; +}; +struct collapse_control {bool is_khugepaged;
mm/khugepaged.c struct collapse_control { · line 113
Quoted 7 lines
mm/khugepaged.c +4
-0@@ -102,6 +107,18 @@ struct collapse_control {/* nodemask for allocation fallback */ nodemask_t alloc_nmask; + +/* +* bitmap used to collapse mTHP sizes. +* 1bit = order KHUGEPAGED_MIN_MTHP_ORDER mTHP
I'm not sure what this '1bit = xxx' comment means?
mm/khugepaged.c struct collapse_control { · line 115
Quoted 2 lines
mm/khugepaged.c +2
-0+*/ +DECLARE_BITMAP (mthp_bitmap,MAX_MTHP_BITMAP_SIZE );
Hmm this seems wrong.
DECLARE_BITMAP(..., val) is expessed as:
#define DECLARE_BITMAP(name,bits) \
unsigned long name[BITS_TO_LONGS(bits)]
So the 2nd param should be number of bits.
But MAX_MTHP_BITMAP_SIZE is:
(1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))
So typically:
(1 << (9 - 2)) = 128
And BITS_TO_LONGS is defined as:
__KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))
So essentially this will be 128 / 8 on a 64-bit system so 16 bytes to store... 7 bits?
Unless I'm missing something here?
mm/khugepaged.c struct collapse_control { · line 116
Quoted 1 line
mm/khugepaged.c +1
-0+DECLARE_BITMAP (mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );
Same comment as above obviously. But also this is kind of horrible, why are we putting a copy of this entire bitmap on the stack every time we declare a cc?
mm/khugepaged.c struct collapse_control { · line 122
Quoted 6 lines
mm/khugepaged.c +5
-0+struct scan_bit_state mthp_bitmap_stack[MAX_MTHP_BITMAP_SIZE ]; +}; + +struct collapse_control khugepaged_collapse_control = { + .is_khugepaged = true, };
Why are we moving this here?
Quoted 22 lines
mm/khugepaged.c +2
-5/** @@ -854,10 +871,6 @@ static void khugepaged_alloc_sleep(void)remove_wait_queue (&khugepaged_wait, &wait); } -struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; -static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -1136,17 +1149,19 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,static int collapse_huge_page (struct mm_struct *mm,unsigned long address,int referenced,int unmapped, -struct collapse_control *cc) +struct collapse_control *cc,bool *mmap_locked, +unsigned int order,unsigned long offset) {LIST_HEAD (compound_pagelist);
mm/khugepaged.c static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, · line 1164
Quoted 10 lines
mm/khugepaged.c +2
-1pmd_t *pmd, _pmd; -pte_t *pte; +pte_t *pte =NULL , mthp_pte;pgtable_t pgtable;struct folio *folio;spinlock_t *pmd_ptl, *pte_ptl;int result =SCAN_FAIL ;struct vm_area_struct *vma;struct mmu_notifier_range range; +unsigned long _address = address + offset *PAGE_SIZE ;
This name is really horrible. please name it sensibly.
It feels like address ought to be consistently the base of the THP or mTHP we wish to collapse, and if we need something PMD aligned for some reason we should rename _that_ to e.g. pmd_address.
Orrr it could be mthp_address...
Perhaps we could just figure that out here and pass only the address... aligning to PMD boundary shouldn't be hard/costly.
But it may indicate we need further refactorisation so we don't need to paper over cracks + pass around a PMD address to do things when that may not be where the (m)THP range begins.
Quoted 15 lines
mm/khugepaged.c +5
-1VM_BUG_ON (address & ~HPAGE_PMD_MASK );@@ -1155,16 +1170,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* The allocation can take potentially a long time if it involves * sync compaction, and we do not need to hold the mmap_lock during * that. We will recheck the vma after taking it again in write mode. +* If collapsing mTHPs we may have already released the read_lock. */ -mmap_read_unlock (mm); +if (*mmap_locked) { +mmap_read_unlock (mm); + *mmap_locked = false; + }
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1186
Quoted 10 lines
mm/khugepaged.c +3
-3- result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + *mmap_locked = true; + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));
I mean this is kind of going back to previous commits, but it's really ugly to pass a BIT(xxx) here, is that really necessary? Can't we just pass in the order?
It's also inconsistent with other calls like
e.g. __collapse_huge_page_swapin() below which passes the order.
Same goes obv. for all such invocations.
Quoted 31 lines
mm/khugepaged.c +4
-4if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1182,13 +1201,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced,HPAGE_PMD_ORDER ); + result =__collapse_huge_page_swapin (mm, vma, _address, pmd, + referenced, order);if (result !=SCAN_SUCCEED ) goto out_nolock; }mmap_read_unlock (mm); + *mmap_locked = false;/* * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM @@ -1198,8 +1218,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1233
Quoted 7 lines
mm/khugepaged.c +2
-2@@ -1210,11 +1229,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_lock_write (vma->anon_vma); -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, address, - address +HPAGE_PMD_SIZE ); +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, _address, + _address + (PAGE_SIZE << order));
This _address is horrible. That really does have to change.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1237
Quoted 4 lines
mm/khugepaged.c +1
-0mmu_notifier_invalidate_range_start (&range); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ +
Odd whitespace...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1251
Quoted 9 lines
mm/khugepaged.c +1
-1/* * This removes any huge TLB entry from the CPU so we won't allow * huge and small TLB entries for the same virtual address to @@ -1228,19 +1248,16 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,mmu_notifier_invalidate_range_end (&range);tlb_remove_table_sync_one (); - pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl); + pte =pte_offset_map_lock (mm, &_pmd, _address, &pte_ptl);
I see we already have a 'convention' of _ prefix on the pmd param, but two wrongs don't make a right...
Quoted 4 lines
mm/khugepaged.c +0
-3if (pte) { - result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist, -HPAGE_PMD_ORDER );
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1243
Quoted 10 lines
mm/khugepaged.c +2
-2+ result =__collapse_huge_page_isolate (vma, _address, pte, cc, + &compound_pagelist, order);spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ; }if (unlikely (result !=SCAN_SUCCEED )) { -if (pte) -pte_unmap (pte);
Why are we removing this?
Quoted 5 lines
mm/khugepaged.c +0
-0spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));/* @@ -1255,17 +1272,17 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, }
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1279
Quoted 10 lines
mm/khugepaged.c +4
-3/* -* All pages are isolated and locked so anon_vma rmap -* can't run anymore. +* For PMD collapse all pages are isolated and locked so anon_vma +* rmap can't run anymore */ -anon_vma_unlock_write (vma->anon_vma); +if (order ==HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);
Hmm this is introducing a horrible new way for things to go wrong. And there's now a whole host of terrible error paths that can go wrong very easily around rmap locks and yeah, no way we cannot do it this way.
rmap locks are VERY sensitive and the ordering of the locking matters a great deal (see top of mm/rmap.c). So we have to be SO careful here.
I suggest you simply have a boolean 'anon_vma_locked' or something like this, and get rid of these horrible additional code paths and the second order == HPAGE_PMD_ORDER check.
We'll track whether or not the lock is held and thereby needs releasing that way instead.
Also, and very importantly - are you 100% sure you can't possibly have a deadlock or issue beyond this point if you don't release the rmap lock?
This is veeeery important, as there can be implicit assumptions around whether or not one can acquire these locks and you basically have to audit ALL code over which this lock is held.
I'm speaking from hard experience here having bumped into this in various attempts at work relating to this stuff...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1285
Quoted 10 lines
mm/khugepaged.c +3
-4result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, - vma, address, pte_ptl, - &compound_pagelist,HPAGE_PMD_ORDER ); -pte_unmap (pte); + vma, _address, pte_ptl, + &compound_pagelist, order);if (unlikely (result !=SCAN_SUCCEED )) - goto out_up_write; + goto out_unlock_anon_vma;
See above...
Quoted 18 lines
mm/khugepaged.c +0
-11/* * The smp_wmb() inside __folio_mark_uptodate() ensures the @@ -1273,33 +1290,115 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* write. */ __folio_mark_uptodate (folio); - pgtable =pmd_pgtable (_pmd); - - _pmd =folio_mk_pmd (folio, vma->vm_page_prot); - _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); - -spin_lock (pmd_ptl); -BUG_ON (!pmd_none (*pmd)); -folio_add_new_anon_rmap (folio, vma, address,RMAP_EXCLUSIVE ); -folio_add_lru_vma (folio, vma); -pgtable_trans_huge_deposit (mm, pmd, pgtable); -set_pmd_at (mm, address, pmd, _pmd);
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1299
Quoted 10 lines
mm/khugepaged.c +7
-3-update_mmu_cache_pmd (vma, address, pmd); -deferred_split_folio (folio, false); -spin_unlock (pmd_ptl); +if (order ==HPAGE_PMD_ORDER ) { + pgtable =pmd_pgtable (_pmd); + _pmd =folio_mk_pmd (folio, vma->vm_page_prot); + _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));
I know you're refactoring this, but be good to change this to a WARN_ON_ONCE(), BUG_ON() is verboten unless it's absolutely definitely going to be a kernel nuclear event, so worth changing things up as we go.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1308
Quoted 9 lines
mm/khugepaged.c +9
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, address, pmd, _pmd); +update_mmu_cache_pmd (vma, address, pmd); +deferred_split_folio (folio, false); +spin_unlock (pmd_ptl); + }else {/* mTHP collapse */ + mthp_pte =mk_pte (&folio->page, vma->vm_page_prot);
I guess it's a rule that each THP or mTHP range spanned must span one and only one folio.
Not sure &folio->page has a future though.
Maybe better to use folio_page(folio, 0)?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1312
Quoted 4 lines
mm/khugepaged.c +4
-0+ mthp_pte =maybe_mkwrite (pte_mkdirty (mthp_pte), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));
having said the above, this is trictly introducing a new BUG_ON() which is a no-no, please make it a WARN_ON_ONCE().
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1313
Quoted 1 line
mm/khugepaged.c +1
-0+folio_ref_add (folio, (1 << order) -1 );
Again no 1 << x please.
Do we do something similar somewhere else for mthp ref counting? Can we share code somehow?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1316
Quoted 3 lines
mm/khugepaged.c +3
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, _address, pte, mthp_pte, (1 << order));
Please avoid 1 << order, and I think at this point since you reference it a bunch of times, just store a local var like nr_pages or sth?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1319
Quoted 3 lines
mm/khugepaged.c +3
-0+update_mmu_cache_range (NULL , vma, _address, pte, (1 << order)); + +smp_wmb ();/* make pte visible before pmd */
Can you give some detail as to why this will work here and why it is necessary?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1320
Quoted 1 line
mm/khugepaged.c +1
-0+pmd_populate (mm, pmd,pmd_pgtable (_pmd));
If we're updating PTE entriess why do we need to assign the PMD entry?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1322
Quoted 2 lines
mm/khugepaged.c +2
-0+spin_unlock (pmd_ptl); + }
This deeply, badly needs to be refactored into something that both shares code and separates out these two operations.
This function is disgustingly long as it is, and that's not your fault, but let's try to make things better as we go.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1329
Quoted 7 lines
mm/khugepaged.c +3
-0folio =NULL ; result =SCAN_SUCCEED ; +out_unlock_anon_vma: +if (order !=HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);
Obviously again as above, we need to simplify this and get rid of this whole bit.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1332
Quoted 3 lines
mm/khugepaged.c +2
-0out_up_write: +if (pte) +pte_unmap (pte);
OK I guess you moved this from above down here? Is this a valid place to do this?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1335
Quoted 3 lines
mm/khugepaged.c +1
-0mmap_write_unlock (mm); out_nolock: + *mmap_locked = false;
This is kind of horrible, we now have pretty mad logic around who sets mmap_locked and where.
Can we just do this at the call sites so we avoid that?
I mean anything we do with this is hideous, but that'd be less confusing It hink.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1342
Quoted 7 lines
mm/khugepaged.c +1
-0if (folio)folio_put (folio);trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result);return result; } +/* Recursive function to consume the bitmap */
Err... please don't? Kernel stack is a seriously finite resource, we do not want recursion at all.
But I'm not actually seeing any recursion here? Am I missing something?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1345
Quoted 3 lines
mm/khugepaged.c +3
-0+static int collapse_scan_bitmap (struct mm_struct *mm,unsigned long address, +int referenced,int unmapped,struct collapse_control *cc, +bool *mmap_locked,unsigned long enabled_orders)
This is a complicated and confusing function, it requires a comment describing how it works.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1351
Quoted 6 lines
mm/khugepaged.c +6
-0+{ +u8 order, next_order; +u16 offset, mid_offset; +int num_chunks; +int bits_set, threshold_bits; +int top =-1 ;
Err why do we start at -1 then immediately increment it?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1355
Quoted 4 lines
mm/khugepaged.c +4
-0+int collapsed =0 ; +int ret; +struct scan_bit_state state; +bool is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));
Extraneous outer parens.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1358
Quoted 3 lines
mm/khugepaged.c +3
-0+ + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + {HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ,0 };
This is the same as
cc->mthp_bitmap_stack[0] = ...; top = 1;
No?
This is really horrible. Can we just have a helper function for this please?
Like:
static int mthp_push_stack(struct collapse_control *cc,
int index, u8 order, u16 offset)
{
struct scan_bit_state *state = &cc->mthp_bitmap_stack[index];
VM_WARN_ON(index >= MAX_MTHP_BITMAP_SIZE);
state->order = order;
state->offset = offset;
return index + 1;
}And can invoke via:
top = mthp_push_stack(cc, top, order, offset);
Or pass index as a pointer possibly also.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1361
Quoted 3 lines
mm/khugepaged.c +3
-0+ +while (top >=0 ) { + state = cc->mthp_bitmap_stack[top--];
OK so this is the recursive bit...
Oh man this function so needs a comment describing what it does, seriously.
I think honestly for sake of my own sanity I'm going to hold off reviewing the rest of this until there's something describing the algorithm, in detail here, above the function.
Quoted 60 lines
mm/khugepaged.c +44
-1+ order = state.order +KHUGEPAGED_MIN_MTHP_ORDER ; + offset = state.offset; + num_chunks =1 << (state.order); +/* Skip mTHP orders that are not enabled */ +if (!test_bit (order, &enabled_orders)) + goto next_order; + +/* copy the relavant section to a new bitmap */ +bitmap_shift_right (cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, +MTHP_BITMAP_SIZE ); + + bits_set =bitmap_weight (cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none -1 ) + >> (HPAGE_PMD_ORDER - state.order); + +/* Check if the region is "almost full" based on the threshold */ +if (bits_set > threshold_bits || is_pmd_only + ||test_bit (order, &huge_anon_orders_always)) { + ret =collapse_huge_page (mm, address, referenced, unmapped, + cc, mmap_locked, order, + offset *KHUGEPAGED_MIN_MTHP_NR ); +if (ret ==SCAN_SUCCEED ) { + collapsed += (1 << order); +continue ; + } + } + +next_order: +if (state.order >0 ) { + next_order = state.order -1 ; + mid_offset = offset + (num_chunks /2 ); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, offset }; + } + } +return collapsed; +} + static int collapse_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked,@@ -1307,31 +1406,60 @@ static int collapse_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; +int i;int result =SCAN_FAIL , referenced =0 ;int none_or_zero =0 , shared =0 ;struct page *page =NULL ;struct folio *folio =NULL ;unsigned long _address; +unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +bool is_pmd_only;bool writable = false; - +int chunk_none_count =0 ;
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1430
Quoted 10 lines
mm/khugepaged.c +4
-0+int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ); +unsigned long tva_flags = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;VM_BUG_ON (address & ~HPAGE_PMD_MASK ); result =find_pmd_or_thp_or_none (mm, address, &pmd);if (result !=SCAN_SUCCEED ) goto out; +bitmap_zero (cc->mthp_bitmap,MAX_MTHP_BITMAP_SIZE ); +bitmap_zero (cc->mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );
Having this 'temp' thing on the stack for everyone is just horrid.
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1440
Quoted 10 lines
mm/khugepaged.c +8
-0memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); + +if (cc->is_khugepaged) + enabled_orders =thp_vma_allowable_orders (vma, vma->vm_flags, + tva_flags,THP_ORDERS_ALL_ANON ); +else + enabled_orders =BIT (HPAGE_PMD_ORDER ); + + is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));
This is horrid, can we have a function broken out to do this please?
In general if you keep open coding stuff, just write a static function for it, the compiler is smart enough to inline.
Quoted 11 lines
mm/khugepaged.c +3
-2+ pte =pte_offset_map_lock (mm, pmd, address, &ptl);if (!pte) { result =SCAN_PMD_NULL ; goto out; } -for (_address = address, _pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, _address +=PAGE_SIZE ) { +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +/*
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1459
Quoted 10 lines
mm/khugepaged.c +10
-0+* we are reading in KHUGEPAGED_MIN_MTHP_NR page chunks. if +* there are pages in this chunk keep track of it in the bitmap +* for mTHP collapsing. +*/ +if (i %KHUGEPAGED_MIN_MTHP_NR ==0 ) { +if (i >0 && chunk_none_count <= scaled_none) +bitmap_set (cc->mthp_bitmap, + (i -1 ) /KHUGEPAGED_MIN_MTHP_NR ,1 ); + chunk_none_count =0 ; + }
This whole thing is really confusing and you are not explaining the algoritm here at all.
This requires a comment, and really this bit should be separated out please.
Quoted 32 lines
mm/khugepaged.c +7
-2+ + _pte = pte + i; + _address = address + i *PAGE_SIZE ;pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { ++unmapped;@@ -1354,10 +1482,11 @@ static int collapse_scan_pmd(struct mm_struct *mm, } }if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++chunk_none_count; ++none_or_zero;if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + (!cc->is_khugepaged || !is_pmd_only || + none_or_zero <= khugepaged_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -1453,6 +1582,7 @@ static int collapse_scan_pmd(struct mm_struct *mm, address))) referenced++; } +if (!writable) { result =SCAN_PAGE_RO ; } else if (cc->is_khugepaged &&@@ -1465,10 +1595,12 @@ static int collapse_scan_pmd(struct mm_struct *mm, out_unmap:pte_unmap_unlock (pte, ptl);if (result ==SCAN_SUCCEED ) {
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1603
Quoted 10 lines
mm/khugepaged.c +6
-4- result =collapse_huge_page (mm, address, referenced, - unmapped, cc); -/* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + result =collapse_scan_bitmap (mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); +if (result >0 ) + result =SCAN_SUCCEED ; +else + result =SCAN_FAIL ;
We're reusing result as both an enum value and as a storage for unmber colapsed PTE entries?
Can we just use a new local variable? Thanks
Quoted 6 lines
mm/khugepaged.c +0
-1} out: trace_mm_khugepaged_scan_pmd(mm, folio, writable, referenced, --2.50.1
I will review the bitmap/chunk stuff in more detail once the algorithm is commented.
Cheers, Lorenzo
Re: [PATCH v10 06/13] khugepaged: add mTHP support
Quoted 43 lines · Lorenzo Stoakes
Quoted 28 lines · Nico Pache
Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in
KHUGEPAGED_MIN_MTHP_ORDERchunks via a bitmap. Each bit represents a utilized region of orderKHUGEPAGED_MIN_MTHP_ORDERptes. If mTHPs are enabled we remove the restriction ofmax_ptes_noneduring the scan phase so we don't bailout early and miss potential mTHP candidates.A new function
collapse_scan_bitmapis used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion.max_ptes_nonewill be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "
max_ptes_none> HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.Quoted 7 lines
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently
madv_collapseis not supported and will only attempt PMD collapse.Yes I think this has to remain the case unfortunately as we override sysfs-specified orders for
MADV_COLLAPSEand there's no sensible way to determine what order we ought to be using.Quoted 1 line
Quoted 4 lines
Quoted 1 line
Signed-off-by Nico Pache <[email protected]>You've gone from small incremental changes to a huge one here... for the sake of reviewer sanity at least, any chance of breaking this up?
I had this as two patches (one for the bitmap and one for implementing it), but I was asked to squash them :/
Quoted 6 lines
Quoted 6 lines
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)
Quoted 12 lines
include/linux/khugepaged.h line 4
Quoted 9 lines
include/linux/khugepaged.h +1
-0@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2I guess this makes sense as by definition 2 pages is least it could possibly be.
Order, so 4 pages, 16kB mTHP
Quoted 4 lines
include/linux/khugepaged.h line 5
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER)Surely
KHUGEPAGED_MIN_NR_MTHP_PTESwould be more meaningful?
Sure!
Quoted 4 lines
include/linux/khugepaged.h line 6
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))This is confusing - size of what?
We need it like this due to ppc64 (and maybe others?), it used to be based on PMD_ORDER, but some arches fail to compile due to the PMD size only being known at boot time.
This compiles to 9 on arches that have 512 ptes.
so 1 << (9 - 2) == 128
Quoted 3 lines
If it's number of bits surely this should be
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER?
This would only be 7? We need a 128 bit bitmap
Quoted 11 lines
This seems to be more so 'the maximum value that could contain the bits right?
I think this is just wrong though, see below at
DECLARE_BITMAP()stuff.include/linux/khugepaged.h line 7
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define MTHP_BITMAP_SIZE (1 << (HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER))Hard to know how this relates to
MAX_MTHP_BITMAP_SIZE?I guess this is the current bitmap size indicating all that is possible, but if these are all #define's what is this accomplishing?
One for compile time one for runtime. Kind of annoying but it was the easiest solution given the architecture limitations.
Quoted 3 lines
For all - please do not do (1 << xxx)! This can lead to sign-extension bugs at least in theory, use
_BITUL(...), it's neater too.
ack, thanks!
Quoted 26 lines
NIT but the whitespace is all screwed up here.
KHUGEPAGED_MIN_MTHP_ORDERandKHUGEPAGED_MIN_MTHP_NRQuoted 21 lines
include/linux/khugepaged.h +0
-0#include <linux/mm.h> mm/khugepaged.c +5
-0@@ -94,6 +94,11 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);static struct kmem_cache *mm_slot_cache__ro_after_init ; +struct scan_bit_state { +u8 order; +u16 offset; +}; +struct collapse_control {bool is_khugepaged;@@ -102,6 +107,18 @@ struct collapse_control {
Quoted 8 lines
mm/khugepaged.c struct collapse_control { · line 113
Quoted 6 lines
mm/khugepaged.c +4
-0/* nodemask for allocation fallback */ nodemask_t alloc_nmask; + +/* +* bitmap used to collapse mTHP sizes. +* 1bit = order KHUGEPAGED_MIN_MTHP_ORDER mTHP I'm not sure what this '1bit = xxx' comment means?
A single bit represents 1 << MIN_MTHP_ORDER (4) pages. Ill express that better
Quoted 5 lines
mm/khugepaged.c struct collapse_control { · line 115
Quoted 3 lines
mm/khugepaged.c +2
-0+*/ +DECLARE_BITMAP (mthp_bitmap,MAX_MTHP_BITMAP_SIZE );Hmm this seems wrong.
Should be a bitmap with 128 bits (for 4k page size). Not sure what's wrong here.
Quoted 14 lines
DECLARE_BITMAP(..., val)is expessed as:#define
DECLARE_BITMAP(name,bits)\
unsigned long name[BITS_TO_LONGS(bits)]So the 2nd param should be number of bits.
But
MAX_MTHP_BITMAP_SIZEis:(1 << (
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER))So typically:
Quoted 8 lines
(1 << (9 - 2)) = 128
And
BITS_TO_LONGSis defined as:
__KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))So essentially this will be 128 / 8 on a 64-bit system so 16 bytes to store... 7 bits?
I think you mean 64. 8 would be BYTES_PER_TYPE
Quoted 2 lines
Unless I'm missing something here?
Hmm, unless the DECLARE_BITMAP is being used incorrectly in multiple places, DECLARE_BITMAP(..., # of bits) is how this is intended to be used.
I think it's an array of unsigned longs, so each part of the name[] is already 64 bits. hence the divide.
Quoted 6 lines
mm/khugepaged.c struct collapse_control { · line 116
Quoted 2 lines
mm/khugepaged.c +1
-0+DECLARE_BITMAP (mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Same comment as above obviously. But also this is kind of horrible, why are we putting a copy of this entire bitmap on the stack every time we declare a cc?
The temp one is used as a scratch pad, Baolin also finds this useful in his file mTHP collapse useful for another use as well.
In general khugepaged always uses the same CC, so it doesn't not having to constantly allocate this.
Quoted 9 lines
mm/khugepaged.c struct collapse_control { · line 122
Quoted 7 lines
mm/khugepaged.c +5
-0+struct scan_bit_state mthp_bitmap_stack[MAX_MTHP_BITMAP_SIZE ]; +}; + +struct collapse_control khugepaged_collapse_control = { + .is_khugepaged = true, };Why are we moving this here?
Because if not it doesn't compile.
Quoted 40 lines
Quoted 23 lines
mm/khugepaged.c +2
-5/** @@ -854,10 +871,6 @@ static void khugepaged_alloc_sleep(void)remove_wait_queue (&khugepaged_wait, &wait); } -struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; -static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -1136,17 +1149,19 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,static int collapse_huge_page (struct mm_struct *mm,unsigned long address,int referenced,int unmapped, -struct collapse_control *cc) +struct collapse_control *cc,bool *mmap_locked, +unsigned int order,unsigned long offset) {LIST_HEAD (compound_pagelist);mm/khugepaged.c static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, · line 1164
Quoted 10 lines
mm/khugepaged.c +2
-1pmd_t *pmd, _pmd; -pte_t *pte; +pte_t *pte =NULL , mthp_pte;pgtable_t pgtable;struct folio *folio;spinlock_t *pmd_ptl, *pte_ptl;int result =SCAN_FAIL ;struct vm_area_struct *vma;struct mmu_notifier_range range; +unsigned long _address = address + offset *PAGE_SIZE ;This name is really horrible. please name it sensibly.
It feels like address ought to be consistently the base of the THP or mTHP we wish to collapse, and if we need something PMD aligned for some reason we should rename _that_ to e.g.
pmd_address.
Quoted 8 lines
Orrr it could be
mthp_address...Perhaps we could just figure that out here and pass only the address... aligning to PMD boundary shouldn't be hard/costly.
But it may indicate we need further refactorisation so we don't need to paper over cracks + pass around a PMD address to do things when that may not be where the (m)THP range begins.
Ok i'll rename them, but we still need to know the PMD address as we rely on it for a few key operations.
Can we leave _address and rename address to pmd_address?
Quoted 21 lines
Quoted 21 lines
mm/khugepaged.c +6
-2VM_BUG_ON (address & ~HPAGE_PMD_MASK );@@ -1155,16 +1170,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* The allocation can take potentially a long time if it involves * sync compaction, and we do not need to hold the mmap_lock during * that. We will recheck the vma after taking it again in write mode. +* If collapsing mTHPs we may have already released the read_lock. */ -mmap_read_unlock (mm); +if (*mmap_locked) { +mmap_read_unlock (mm); + *mmap_locked = false; + } - result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out_nolock;
Quoted 9 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1186
Quoted 5 lines
mm/khugepaged.c +2
-2mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + *mmap_locked = true; + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));I mean this is kind of going back to previous commits, but it's really ugly to pass a BIT(xxx) here, is that really necessary? Can't we just pass in the order?
Yes and no... currently we only ever pass the bit of the current order so we could get away with it, but to generalize it we want the ability to pass a bitmap of the available orders. Like in the case of future madvise_collapse support, we would need to pass a bitmap of possible orders.
Quoted 111 lines
It's also inconsistent with other calls like
e.g.__collapse_huge_page_swapin()below which passes the order.Same goes obv. for all such invocations.
Quoted 31 lines
mm/khugepaged.c +4
-4if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1182,13 +1201,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced,HPAGE_PMD_ORDER ); + result =__collapse_huge_page_swapin (mm, vma, _address, pmd, + referenced, order);if (result !=SCAN_SUCCEED ) goto out_nolock; }mmap_read_unlock (mm); + *mmap_locked = false;/* * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM @@ -1198,8 +1218,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1233
Quoted 7 lines
mm/khugepaged.c +2
-2@@ -1210,11 +1229,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_lock_write (vma->anon_vma); -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, address, - address +HPAGE_PMD_SIZE ); +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, _address, + _address + (PAGE_SIZE << order));This
_addressis horrible. That really does have to change.mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1237
Quoted 4 lines
mm/khugepaged.c +1
-0mmu_notifier_invalidate_range_start (&range); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ +Odd whitespace...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1251
Quoted 9 lines
mm/khugepaged.c +1
-1/* * This removes any huge TLB entry from the CPU so we won't allow * huge and small TLB entries for the same virtual address to @@ -1228,19 +1248,16 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,mmu_notifier_invalidate_range_end (&range);tlb_remove_table_sync_one (); - pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl); + pte =pte_offset_map_lock (mm, &_pmd, _address, &pte_ptl);I see we already have a 'convention' of _ prefix on the pmd param, but two wrongs don't make a right...
Quoted 4 lines
mm/khugepaged.c +0
-3if (pte) { - result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist, -HPAGE_PMD_ORDER );mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1243
Quoted 10 lines
mm/khugepaged.c +2
-2+ result =__collapse_huge_page_isolate (vma, _address, pte, cc, + &compound_pagelist, order);spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ; }if (unlikely (result !=SCAN_SUCCEED )) { -if (pte) -pte_unmap (pte);Why are we removing this?
Quoted 5 lines
mm/khugepaged.c +0
-0spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));/* @@ -1255,17 +1272,17 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, }mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1279
Quoted 10 lines
mm/khugepaged.c +4
-3/* -* All pages are isolated and locked so anon_vma rmap -* can't run anymore. +* For PMD collapse all pages are isolated and locked so anon_vma +* rmap can't run anymore */ -anon_vma_unlock_write (vma->anon_vma); +if (order ==HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Hmm this is introducing a horrible new way for things to go wrong. And there's now a whole host of terrible error paths that can go wrong very easily around rmap locks and yeah, no way we cannot do it this way.
rmap locks are VERY sensitive and the ordering of the locking matters a great deal (see top of
mm/rmap.c). So we have to be SO careful here.I suggest you simply have a boolean '
anon_vma_locked' or something like this, and get rid of these horrible additional code paths and the second order ==HPAGE_PMD_ORDERcheck.
Quoted 5 lines
We'll track whether or not the lock is held and thereby needs releasing that way instead.
Also, and very importantly - are you 100% sure you can't possibly have a deadlock or issue beyond this point if you don't release the rmap lock?
I double checked, this was added as a fix to an issue Hugh reported. The gap between these callers is rather small, and I see no way that it could skip the lock/unlock cycle.
Quoted 45 lines
This is veeeery important, as there can be implicit assumptions around whether or not one can acquire these locks and you basically have to audit ALL code over which this lock is held.
I'm speaking from hard experience here having bumped into this in various attempts at work relating to this stuff...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1285
Quoted 10 lines
mm/khugepaged.c +3
-4result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, - vma, address, pte_ptl, - &compound_pagelist,HPAGE_PMD_ORDER ); -pte_unmap (pte); + vma, _address, pte_ptl, + &compound_pagelist, order);if (unlikely (result !=SCAN_SUCCEED )) - goto out_up_write; + goto out_unlock_anon_vma;See above...
Quoted 24 lines
mm/khugepaged.c +3
-14/* * The smp_wmb() inside __folio_mark_uptodate() ensures the @@ -1273,33 +1290,115 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* write. */ __folio_mark_uptodate (folio); - pgtable =pmd_pgtable (_pmd); - - _pmd =folio_mk_pmd (folio, vma->vm_page_prot); - _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); - -spin_lock (pmd_ptl); -BUG_ON (!pmd_none (*pmd)); -folio_add_new_anon_rmap (folio, vma, address,RMAP_EXCLUSIVE ); -folio_add_lru_vma (folio, vma); -pgtable_trans_huge_deposit (mm, pmd, pgtable); -set_pmd_at (mm, address, pmd, _pmd); -update_mmu_cache_pmd (vma, address, pmd); -deferred_split_folio (folio, false); -spin_unlock (pmd_ptl); +if (order ==HPAGE_PMD_ORDER ) { + pgtable =pmd_pgtable (_pmd); + _pmd =folio_mk_pmd (folio, vma->vm_page_prot);
Quoted 8 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1299
Quoted 4 lines
mm/khugepaged.c +4
-0+ _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));I know you're refactoring this, but be good to change this to a
WARN_ON_ONCE(),BUG_ON()is verboten unless it's absolutely definitely going to be a kernel nuclear event, so worth changing things up as we go.
Yeah i keep seeing those warning in checkpatch, so Ill go ahead and edit it.
Quoted 11 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1308
Quoted 10 lines
mm/khugepaged.c +9
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, address, pmd, _pmd); +update_mmu_cache_pmd (vma, address, pmd); +deferred_split_folio (folio, false); +spin_unlock (pmd_ptl); + }else {/* mTHP collapse */ + mthp_pte =mk_pte (&folio->page, vma->vm_page_prot);
Quoted 6 lines
I guess it's a rule that each THP or mTHP range spanned must span one and only one folio.
Not sure
&folio->pagehas a future though.Maybe better to use
folio_page(folio, 0)?
Ok sounds good I'll use that.
Quoted 9 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1312
Quoted 5 lines
mm/khugepaged.c +4
-0+ mthp_pte =maybe_mkwrite (pte_mkdirty (mthp_pte), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));having said the above, this is trictly introducing a new
BUG_ON()which is a no-no, please make it aWARN_ON_ONCE().
Quoted 6 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1313
Quoted 1 line
mm/khugepaged.c +1
-0+folio_ref_add (folio, (1 << order) -1 );Again no 1 << x please.
Do we do something similar somewhere else for mthp ref counting? Can we share code somehow?
Yeah but IIRC its only like 2 or 3 places that do something like this... most callers to folio_add_* do things in slightly different manners. Maybe something to look into for the future, but I think it will be difficult to generalize it.
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1316
Quoted 4 lines
mm/khugepaged.c +3
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, _address, pte, mthp_pte, (1 << order));Please avoid 1 << order, and I think at this point since you reference it a bunch of times, just store a local var like
nr_pagesor sth?
yeah not a bad idea!
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1319
Quoted 4 lines
mm/khugepaged.c +3
-0+update_mmu_cache_range (NULL , vma, _address, pte, (1 << order)); + +smp_wmb ();/* make pte visible before pmd */ Can you give some detail as to why this will work here and why it is necessary?
Other parts of the kernel do it when setting ptes before updating the PMD. I'm not sure if it's necessary, but better safe than sorry.
Quoted 4 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1320
Quoted 2 lines
mm/khugepaged.c +1
-0+pmd_populate (mm, pmd,pmd_pgtable (_pmd));If we're updating PTE entriess why do we need to assign the PMD entry?
We removed the PMD entry for GUP_fast reasons, then we reinstall the PMD entry after the mTHP is in place. Same as for PMD collapse.
Quoted 18 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1322
Quoted 3 lines
mm/khugepaged.c +2
-0+spin_unlock (pmd_ptl); + }This deeply, badly needs to be refactored into something that both shares code and separates out these two operations.
This function is disgustingly long as it is, and that's not your fault, but let's try to make things better as we go.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1329
Quoted 7 lines
mm/khugepaged.c +3
-0folio =NULL ; result =SCAN_SUCCEED ; +out_unlock_anon_vma: +if (order !=HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);
Quoted 8 lines
Obviously again as above, we need to simplify this and get rid of this whole bit.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1332
Quoted 3 lines
mm/khugepaged.c +2
-0out_up_write: +if (pte) +pte_unmap (pte);OK I guess you moved this from above down here? Is this a valid place to do this?
Yes if not we were potentially unmapping a pte early.
Quoted 24 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1335
Quoted 4 lines
mm/khugepaged.c +1
-0mmap_write_unlock (mm); out_nolock: + *mmap_locked = false;This is kind of horrible, we now have pretty mad logic around who sets
mmap_lockedand where.Can we just do this at the call sites so we avoid that?
I mean anything we do with this is hideous, but that'd be less confusing It hink.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1342
Quoted 7 lines
mm/khugepaged.c +1
-0if (folio)folio_put (folio);trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result);return result; } +/* Recursive function to consume the bitmap */ Err... please don't? Kernel stack is a seriously finite resource, we do not want recursion at all.
Quoted 8 lines
But I'm not actually seeing any recursion here? Am I missing something?
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1345
Quoted 3 lines
mm/khugepaged.c +3
-0+static int collapse_scan_bitmap (struct mm_struct *mm,unsigned long address, +int referenced,int unmapped,struct collapse_control *cc, +bool *mmap_locked,unsigned long enabled_orders)This is a complicated and confusing function, it requires a comment describing how it works.
Ok will do!
Quoted 9 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1351
Quoted 7 lines
mm/khugepaged.c +6
-0+{ +u8 order, next_order; +u16 offset, mid_offset; +int num_chunks; +int bits_set, threshold_bits; +int top =-1 ;Err why do we start at -1 then immediately increment it?
You are correct, it was probably a leftover bit from my development phase. Seems I can just set it to 0 to begin with.
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1355
Quoted 5 lines
mm/khugepaged.c +4
-0+int collapsed =0 ; +int ret; +struct scan_bit_state state; +bool is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));Extraneous outer parens.
ack
Quoted 11 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1358
Quoted 4 lines
mm/khugepaged.c +3
-0+ + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + {HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ,0 };This is the same as
cc->mthp_bitmap_stack[0] = ...; top = 1;No?
no it would be bitmap_stack[0] = ...
then top goes to -1 (at state =... ), and if we add more items (next_order) to the stack it would go top = 1 (adds one for each half of the split)
Quoted 4 lines
This is really horrible. Can we just have a helper function for this please?
Seems kinda excessive for 4 lines and one caller.
Quoted 15 lines
Like:
static int mthp_push_stack(struct collapse_control *cc, int index, u8 order, u16 offset) { struct scan_bit_state *state = &cc->mthp_bitmap_stack[index]; VM_WARN_ON(index >= MAX_MTHP_BITMAP_SIZE); state->order = order; state->offset = offset; return index + 1; }
This would not work in its current state because its ++index in the current implementation. I would need to refactor, but the general idea still stands
Quoted 11 lines
And can invoke via:
top = mthp_push_stack(cc, top, order, offset);Or pass index as a pointer possibly also.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1361
Quoted 3 lines
mm/khugepaged.c +3
-0+ +while (top >=0 ) { + state = cc->mthp_bitmap_stack[top--];
Quoted 7 lines
OK so this is the recursive bit...
Oh man this function so needs a comment describing what it does, seriously.
I think honestly for sake of my own sanity I'm going to hold off reviewing the rest of this until there's something describing the algorithm, in detail here, above the function.
It's basically binary recursion with a stack structure, that checks regions of the bitmap in descending order (ie order 9, order 8, ...) if we go to the next order we add two items to the stack (left and right half). I will add a comment describing it at the top of the function.
Quoted 65 lines
Quoted 65 lines
mm/khugepaged.c +46
-1+ order = state.order +KHUGEPAGED_MIN_MTHP_ORDER ; + offset = state.offset; + num_chunks =1 << (state.order); +/* Skip mTHP orders that are not enabled */ +if (!test_bit (order, &enabled_orders)) + goto next_order; + +/* copy the relavant section to a new bitmap */ +bitmap_shift_right (cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, +MTHP_BITMAP_SIZE ); + + bits_set =bitmap_weight (cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none -1 ) + >> (HPAGE_PMD_ORDER - state.order); + +/* Check if the region is "almost full" based on the threshold */ +if (bits_set > threshold_bits || is_pmd_only + ||test_bit (order, &huge_anon_orders_always)) { + ret =collapse_huge_page (mm, address, referenced, unmapped, + cc, mmap_locked, order, + offset *KHUGEPAGED_MIN_MTHP_NR ); +if (ret ==SCAN_SUCCEED ) { + collapsed += (1 << order); +continue ; + } + } + +next_order: +if (state.order >0 ) { + next_order = state.order -1 ; + mid_offset = offset + (num_chunks /2 ); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, offset }; + } + } +return collapsed; +} + static int collapse_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked,@@ -1307,31 +1406,60 @@ static int collapse_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; +int i;int result =SCAN_FAIL , referenced =0 ;int none_or_zero =0 , shared =0 ;struct page *page =NULL ;struct folio *folio =NULL ;unsigned long _address; +unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +bool is_pmd_only;bool writable = false; - +int chunk_none_count =0 ; +int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ); +unsigned long tva_flags = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;VM_BUG_ON (address & ~HPAGE_PMD_MASK );
Quoted 8 lines
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1430
Quoted 6 lines
mm/khugepaged.c +2
-0result =find_pmd_or_thp_or_none (mm, address, &pmd);if (result !=SCAN_SUCCEED ) goto out; +bitmap_zero (cc->mthp_bitmap,MAX_MTHP_BITMAP_SIZE ); +bitmap_zero (cc->mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Having this 'temp' thing on the stack for everyone is just horrid.
As I mention above this serves a very good purpose, and is also expanded in another series by Baolin to serve another similar purpose too.
Quoted 14 lines
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1440
Quoted 11 lines
mm/khugepaged.c +8
-0memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); + +if (cc->is_khugepaged) + enabled_orders =thp_vma_allowable_orders (vma, vma->vm_flags, + tva_flags,THP_ORDERS_ALL_ANON ); +else + enabled_orders =BIT (HPAGE_PMD_ORDER ); + + is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));This is horrid, can we have a function broken out to do this please?
Quoted 2 lines
In general if you keep open coding stuff, just write a static function for it, the compiler is smart enough to inline.
ok, we do this is a few places so perhaps its the best approach.
Quoted 23 lines
Quoted 12 lines
mm/khugepaged.c +3
-2+ pte =pte_offset_map_lock (mm, pmd, address, &ptl);if (!pte) { result =SCAN_PMD_NULL ; goto out; } -for (_address = address, _pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, _address +=PAGE_SIZE ) { +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +/* mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1459
Quoted 10 lines
mm/khugepaged.c +10
-0+* we are reading in KHUGEPAGED_MIN_MTHP_NR page chunks. if +* there are pages in this chunk keep track of it in the bitmap +* for mTHP collapsing. +*/ +if (i %KHUGEPAGED_MIN_MTHP_NR ==0 ) { +if (i >0 && chunk_none_count <= scaled_none) +bitmap_set (cc->mthp_bitmap, + (i -1 ) /KHUGEPAGED_MIN_MTHP_NR ,1 ); + chunk_none_count =0 ; + }
Quoted 4 lines
This whole thing is really confusing and you are not explaining the algoritm here at all.
This requires a comment, and really this bit should be separated out please.
This used to be its own commit, but multiple people wanted it squashed... ugh. Which should we go with?
Quoted 56 lines
Quoted 33 lines
mm/khugepaged.c +7
-2+ + _pte = pte + i; + _address = address + i *PAGE_SIZE ;pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { ++unmapped;@@ -1354,10 +1482,11 @@ static int collapse_scan_pmd(struct mm_struct *mm, } }if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++chunk_none_count; ++none_or_zero;if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + (!cc->is_khugepaged || !is_pmd_only || + none_or_zero <= khugepaged_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -1453,6 +1582,7 @@ static int collapse_scan_pmd(struct mm_struct *mm, address))) referenced++; } +if (!writable) { result =SCAN_PAGE_RO ; } else if (cc->is_khugepaged &&@@ -1465,10 +1595,12 @@ static int collapse_scan_pmd(struct mm_struct *mm, out_unmap:pte_unmap_unlock (pte, ptl);if (result ==SCAN_SUCCEED ) {mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1603
Quoted 10 lines
mm/khugepaged.c +6
-4- result =collapse_huge_page (mm, address, referenced, - unmapped, cc); -/* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + result =collapse_scan_bitmap (mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); +if (result >0 ) + result =SCAN_SUCCEED ; +else + result =SCAN_FAIL ;We're reusing result as both an enum value and as a storage for unmber colapsed PTE entries?
Can we just use a new local variable? Thanks
Quoted 6 lines
mm/khugepaged.c +0
-1} out: trace_mm_khugepaged_scan_pmd(mm, folio, writable, referenced, --2.50.1
Quoted 2 lines
I will review the bitmap/chunk stuff in more detail once the algorithm is commented.
ok thanks for the review.
Quoted 3 lines
Cheers, Lorenzo
Re: [PATCH v10 06/13] khugepaged: add mTHP support
You've not responded to a ton of comments, I'm guesing I should assume in those cases you're acking the comments implicitly? :) Do let me know.
Quoted 45 lines · Nico Pache
Quoted 43 lines · Lorenzo Stoakes
Quoted 28 lines · Nico Pache
Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in
KHUGEPAGED_MIN_MTHP_ORDERchunks via a bitmap. Each bit represents a utilized region of orderKHUGEPAGED_MIN_MTHP_ORDERptes. If mTHPs are enabled we remove the restriction ofmax_ptes_noneduring the scan phase so we don't bailout early and miss potential mTHP candidates.A new function
collapse_scan_bitmapis used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion.max_ptes_nonewill be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "
max_ptes_none> HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.Quoted 7 lines
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently
madv_collapseis not supported and will only attempt PMD collapse.Yes I think this has to remain the case unfortunately as we override sysfs-specified orders for
MADV_COLLAPSEand there's no sensible way to determine what order we ought to be using.Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Nico Pache <[email protected]>You've gone from small incremental changes to a huge one here... for the sake of reviewer sanity at least, any chance of breaking this up?
I had this as two patches (one for the bitmap and one for implementing it), but I was asked to squash them :/
Yeah it makes sense to combine those two, but maybe there's a better way of splitting things out.
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)Quoted 12 lines
include/linux/khugepaged.h line 4
Quoted 9 lines
include/linux/khugepaged.h +1
-0@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2I guess this makes sense as by definition 2 pages is least it could possibly be.
Quoted 1 line
Order, so 4 pages, 16kB mTHP
Right, misread that! :) sorry. I guess then in fact there is no such thing as an order-1 mTHP? I did wonder how useful that'd be so makes more sense
Quoted 5 lines
Quoted 4 lines
include/linux/khugepaged.h line 5
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER)Surely
KHUGEPAGED_MIN_NR_MTHP_PTESwould be more meaningful?Sure!
Thanks!
Quoted 7 lines
Quoted 4 lines
include/linux/khugepaged.h line 6
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))This is confusing - size of what?
We need it like this due to ppc64 (and maybe others?), it used to be based on
PMD_ORDER, but some arches fail to compile due to the PMD size only being known at boot time.
That really sucks. Do please put this context in a comment though (see below for more discussion on this).
Quoted 3 lines
This compiles to 9 on arches that have 512 ptes.
so 1 << (9 - 2) == 128
What I'm saying is - what is this expressed in? There's no information on that here.
So from what you say I can see it's the number of bits required in the bitmap, because we're being smart and only bothering to mark entries at a granularity of minimum mTHP size.
OK so the idea is this is the number of PTE entries per entry in the bitmap.
This is KEY missing context. We really need to spell things out here, the THP code is... confusing :) to put it politely, so we need to be especially careful to be mega clear.
So please please PLEASE add a comment explaining all this. And clearly state that the unit here is bits.
Quoted 4 lines
Quoted 3 lines
If it's number of bits surely this should be
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER?This would only be 7? We need a 128 bit bitmap
Again missing context is you're storing bits per minimum mTHP as per above.
Quoted 5 lines
Quoted 5 lines
This seems to be more so 'the maximum value that could contain the bits right?
I think this is just wrong though, see below at
DECLARE_BITMAP()stuff.
Quoted 8 lines
Quoted 6 lines
include/linux/khugepaged.h line 7
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define MTHP_BITMAP_SIZE (1 << (HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER))Hard to know how this relates to
MAX_MTHP_BITMAP_SIZE?I guess this is the current bitmap size indicating all that is possible, but if these are all #define's what is this accomplishing?
One for compile time one for runtime. Kind of annoying but it was the easiest solution given the architecture limitations.
OK, this context is fantastic + important for understanding, so let's put it in a comment :)
Quoted 4 lines
Quoted 3 lines
For all - please do not do (1 << xxx)! This can lead to sign-extension bugs at least in theory, use
_BITUL(...), it's neater too.ack, thanks!
OK sorry, based on David's feedback on this - just use 1UL << xxx here instead.
(An aside that isn't really relevant now but - also I was wrong to suggest _BITUL() anyway BIT() is The Way (TM), for some reason I had it in my head that the former was better :P)
Quoted 34 lines
Quoted 26 lines
NIT but the whitespace is all screwed up here.
KHUGEPAGED_MIN_MTHP_ORDERandKHUGEPAGED_MIN_MTHP_NRQuoted 21 lines
include/linux/khugepaged.h +0
-0#include <linux/mm.h> mm/khugepaged.c +5
-0@@ -94,6 +94,11 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);static struct kmem_cache *mm_slot_cache__ro_after_init ; +struct scan_bit_state { +u8 order; +u16 offset; +}; +struct collapse_control {bool is_khugepaged;@@ -102,6 +107,18 @@ struct collapse_control {Quoted 8 lines
mm/khugepaged.c struct collapse_control { · line 113
Quoted 6 lines
mm/khugepaged.c +4
-0/* nodemask for allocation fallback */ nodemask_t alloc_nmask; + +/* +* bitmap used to collapse mTHP sizes. +* 1bit = order KHUGEPAGED_MIN_MTHP_ORDER mTHP I'm not sure what this '1bit = xxx' comment means?
Quoted 1 line
A single bit represents 1 <<
MIN_MTHP_ORDER(4) pages. Ill express that better
Yeah again, this is the missing context that would have helped me feel a lot less confused here :P
Quoted 30 lines
Quoted 5 lines
mm/khugepaged.c struct collapse_control { · line 115
Quoted 3 lines
mm/khugepaged.c +2
-0+*/ +DECLARE_BITMAP (mthp_bitmap,MAX_MTHP_BITMAP_SIZE );Hmm this seems wrong.
Should be a bitmap with 128 bits (for 4k page size). Not sure what's wrong here.
Quoted 14 lines
DECLARE_BITMAP(..., val)is expessed as:#define
DECLARE_BITMAP(name,bits)\
unsigned long name[BITS_TO_LONGS(bits)]So the 2nd param should be number of bits.
But
MAX_MTHP_BITMAP_SIZEis:(1 << (
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER))So typically:
Quoted 8 lines
(1 << (9 - 2)) = 128
And
BITS_TO_LONGSis defined as:
__KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))So essentially this will be 128 / 8 on a 64-bit system so 16 bytes to store... 7 bits?
I think you mean 64. 8 would be
BYTES_PER_TYPEQuoted 1 line
Quoted 7 lines
Quoted 1 line
Unless I'm missing something here?
Hmm, unless the
DECLARE_BITMAPis being used incorrectly in multiple places,DECLARE_BITMAP(..., # of bits)is how this is intended to be used.I think it's an array of unsigned longs, so each part of the name[] is already 64 bits. hence the divide.
Yup again this is due to the missing context above.
Quoted 8 lines
Quoted 6 lines
mm/khugepaged.c struct collapse_control { · line 116
Quoted 2 lines
mm/khugepaged.c +1
-0+DECLARE_BITMAP (mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Same comment as above obviously. But also this is kind of horrible, why are we putting a copy of this entire bitmap on the stack every time we declare a cc?
The temp one is used as a scratch pad, Baolin also finds this useful in his file mTHP collapse useful for another use as well.
Saying 'scratch pad' is really just saying 'temporary' using different words :) this isn't hugely helpful.
If we _need_ this we should give this a better name, and I also don't know why we need this on the stack for all collapse_control users. Can't you just have your 'scratch pad' local to the function that needs it or passed as a pointer?
I'm sure it'd be useful to somebody to stick various temporary things in vm_area_struct for instance, but it'd be hugely egregious to do so...
Quoted 3 lines
In general khugepaged always uses the same CC, so it doesn't not having to constantly allocate this.
You're putting a LOT of data on the stack, and kernel stacks are very delicate, I'm quite concerned about this.
BEFORE:
struct collapse_control {
bool is_khugepaged; /* 0 1 */
/* XXX 3 bytes hole, try to pack */
u32 node_load[64]; /* 4 256 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */
nodemask_t alloc_nmask; /* 264 8 */
/* size: 272, cachelines: 5, members: 3 */
/* sum members: 265, holes: 2, sum holes: 7 */
/* last cacheline: 16 bytes */
};272 bytes.
AFTER:
struct collapse_control {
bool is_khugepaged; /* 0 1 */
/* XXX 3 bytes hole, try to pack */
u32 node_load[64]; /* 4 256 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */
nodemask_t alloc_nmask; /* 264 8 */
long unsigned int mthp_bitmap[2]; /* 272 16 */
long unsigned int mthp_bitmap_temp[2]; /* 288 16 */
struct scan_bit_state mthp_bitmap_stack[128]; /* 304 512 */
/* size: 816, cachelines: 13, members: 6 */
/* sum members: 809, holes: 2, sum holes: 7 */
/* last cacheline: 48 bytes */
};816 bytes.
So you're roughly tripling the size of this struct and making this a thing in all callstacks that reference collapse_control.
Kernel stack is a _super_ finite resource.
I mean I can't really review your stack implementation thing until you've added a comment to help me understand what you are doing there (sorry but it's just too fiddly for me to first principles it), but I wonder if we truly need to be doing this?
I wonder if we can just put this into somewhere allocated...
Quoted 10 lines
Quoted 9 lines
mm/khugepaged.c struct collapse_control { · line 122
Quoted 7 lines
mm/khugepaged.c +5
-0+struct scan_bit_state mthp_bitmap_stack[MAX_MTHP_BITMAP_SIZE ]; +}; + +struct collapse_control khugepaged_collapse_control = { + .is_khugepaged = true, };Why are we moving this here?
Because if not it doesn't compile.
A reason as to why this is the case would be helpful :)
But fair enough!
Quoted 45 lines
Quoted 45 lines
Quoted 23 lines
mm/khugepaged.c +2
-5/** @@ -854,10 +871,6 @@ static void khugepaged_alloc_sleep(void)remove_wait_queue (&khugepaged_wait, &wait); } -struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; -static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -1136,17 +1149,19 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,static int collapse_huge_page (struct mm_struct *mm,unsigned long address,int referenced,int unmapped, -struct collapse_control *cc) +struct collapse_control *cc,bool *mmap_locked, +unsigned int order,unsigned long offset) {LIST_HEAD (compound_pagelist);mm/khugepaged.c static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, · line 1164
Quoted 10 lines
mm/khugepaged.c +2
-1pmd_t *pmd, _pmd; -pte_t *pte; +pte_t *pte =NULL , mthp_pte;pgtable_t pgtable;struct folio *folio;spinlock_t *pmd_ptl, *pte_ptl;int result =SCAN_FAIL ;struct vm_area_struct *vma;struct mmu_notifier_range range; +unsigned long _address = address + offset *PAGE_SIZE ;This name is really horrible. please name it sensibly.
It feels like address ought to be consistently the base of the THP or mTHP we wish to collapse, and if we need something PMD aligned for some reason we should rename _that_ to e.g.
pmd_address.Orrr it could be
mthp_address...Perhaps we could just figure that out here and pass only the address... aligning to PMD boundary shouldn't be hard/costly.
Quoted 6 lines
Quoted 3 lines
But it may indicate we need further refactorisation so we don't need to paper over cracks + pass around a PMD address to do things when that may not be where the (m)THP range begins.
Ok i'll rename them, but we still need to know the PMD address as we rely on it for a few key operations.
Can we leave_addressand rename address topmd_address?
No, we absolutely cannot leave _address as '_address', that's terrible and I'm just not going to live with that.
I know it's trivial, but it's just such a horrid naming convention.
Quoted 30 lines
Quoted 21 lines
Quoted 21 lines
mm/khugepaged.c +6
-2VM_BUG_ON (address & ~HPAGE_PMD_MASK );@@ -1155,16 +1170,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* The allocation can take potentially a long time if it involves * sync compaction, and we do not need to hold the mmap_lock during * that. We will recheck the vma after taking it again in write mode. +* If collapsing mTHPs we may have already released the read_lock. */ -mmap_read_unlock (mm); +if (*mmap_locked) { +mmap_read_unlock (mm); + *mmap_locked = false; + } - result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out_nolock;Quoted 9 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1186
Quoted 5 lines
mm/khugepaged.c +2
-2mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + *mmap_locked = true; + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));I mean this is kind of going back to previous commits, but it's really ugly to pass a BIT(xxx) here, is that really necessary? Can't we just pass in the order?
Quoted 5 lines
Yes and no... currently we only ever pass the bit of the current order so we could get away with it, but to generalize it we want the ability to pass a bitmap of the available orders. Like in the case of future
madvise_collapsesupport, we would need to pass a bitmap of possible orders.
Can we just change that when we need it?
'Future proofing' for an possible future implementation detail is just not how we should implement things.
Right now we don't do that, so let's just pass the order. If in future we want to change it we can.
Quoted 78 lines
Quoted 78 lines
It's also inconsistent with other calls like
e.g.__collapse_huge_page_swapin()below which passes the order.Same goes obv. for all such invocations.
Quoted 31 lines
mm/khugepaged.c +4
-4if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1182,13 +1201,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced,HPAGE_PMD_ORDER ); + result =__collapse_huge_page_swapin (mm, vma, _address, pmd, + referenced, order);if (result !=SCAN_SUCCEED ) goto out_nolock; }mmap_read_unlock (mm); + *mmap_locked = false;/* * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM @@ -1198,8 +1218,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1233
Quoted 7 lines
mm/khugepaged.c +2
-2@@ -1210,11 +1229,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_lock_write (vma->anon_vma); -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, address, - address +HPAGE_PMD_SIZE ); +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, _address, + _address + (PAGE_SIZE << order));This
_addressis horrible. That really does have to change.mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1237
Quoted 4 lines
mm/khugepaged.c +1
-0mmu_notifier_invalidate_range_start (&range); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ +Odd whitespace...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1251
Quoted 9 lines
mm/khugepaged.c +1
-1/* * This removes any huge TLB entry from the CPU so we won't allow * huge and small TLB entries for the same virtual address to @@ -1228,19 +1248,16 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,mmu_notifier_invalidate_range_end (&range);tlb_remove_table_sync_one (); - pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl); + pte =pte_offset_map_lock (mm, &_pmd, _address, &pte_ptl);I see we already have a 'convention' of _ prefix on the pmd param, but two wrongs don't make a right...
Quoted 11 lines
mm/khugepaged.c +2
-3if (pte) { - result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist, -HPAGE_PMD_ORDER ); + result =__collapse_huge_page_isolate (vma, _address, pte, cc, + &compound_pagelist, order);spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ; }
Quoted 5 lines
Quoted 5 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1243
Quoted 3 lines
mm/khugepaged.c +0
-2if (unlikely (result !=SCAN_SUCCEED )) { -if (pte) -pte_unmap (pte);Why are we removing this?
You're missing some of the comments, I'm guesing for most of the smaller stuff you're just implicitly acking them but this one was a question :)
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
mm/khugepaged.c +0
-0spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));/* @@ -1255,17 +1272,17 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, }mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1279
Quoted 10 lines
mm/khugepaged.c +4
-3/* -* All pages are isolated and locked so anon_vma rmap -* can't run anymore. +* For PMD collapse all pages are isolated and locked so anon_vma +* rmap can't run anymore */ -anon_vma_unlock_write (vma->anon_vma); +if (order ==HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Hmm this is introducing a horrible new way for things to go wrong. And there's now a whole host of terrible error paths that can go wrong very easily around rmap locks and yeah, no way we cannot do it this way.
rmap locks are VERY sensitive and the ordering of the locking matters a great deal (see top of
mm/rmap.c). So we have to be SO careful here.
Quoted 6 lines
Quoted 6 lines
I suggest you simply have a boolean '
anon_vma_locked' or something like this, and get rid of these horrible additional code paths and the second order ==HPAGE_PMD_ORDERcheck.We'll track whether or not the lock is held and thereby needs releasing that way instead.
You've not responded to this suggestion re: refactoring here.
I'm really not a fan of us arbitrarily messing with rmap locks like this, and we should very carefully keep track of whether we have/have not released them.
Again rmap locking is a dangerous area, I've got personal experience of this (see top of mm/rmap.c for an indication of complexity here as well as
https://kernel.org/doc/html/latest/mm/process_addrs.html).
Quoted 7 lines
Quoted 3 lines
Also, and very importantly - are you 100% sure you can't possibly have a deadlock or issue beyond this point if you don't release the rmap lock?
I double checked, this was added as a fix to an issue Hugh reported.
The gap between these callers is rather small, and I see no way that it could skip the lock/unlock cycle.
We're going to need more than this to be confident, you need to clearly justify why we won't encounter issues this way.
Quoted 54 lines
Quoted 45 lines
This is veeeery important, as there can be implicit assumptions around whether or not one can acquire these locks and you basically have to audit ALL code over which this lock is held.
I'm speaking from hard experience here having bumped into this in various attempts at work relating to this stuff...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1285
Quoted 10 lines
mm/khugepaged.c +3
-4result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, - vma, address, pte_ptl, - &compound_pagelist,HPAGE_PMD_ORDER ); -pte_unmap (pte); + vma, _address, pte_ptl, + &compound_pagelist, order);if (unlikely (result !=SCAN_SUCCEED )) - goto out_up_write; + goto out_unlock_anon_vma;See above...
Quoted 24 lines
mm/khugepaged.c +3
-14/* * The smp_wmb() inside __folio_mark_uptodate() ensures the @@ -1273,33 +1290,115 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* write. */ __folio_mark_uptodate (folio); - pgtable =pmd_pgtable (_pmd); - - _pmd =folio_mk_pmd (folio, vma->vm_page_prot); - _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); - -spin_lock (pmd_ptl); -BUG_ON (!pmd_none (*pmd)); -folio_add_new_anon_rmap (folio, vma, address,RMAP_EXCLUSIVE ); -folio_add_lru_vma (folio, vma); -pgtable_trans_huge_deposit (mm, pmd, pgtable); -set_pmd_at (mm, address, pmd, _pmd); -update_mmu_cache_pmd (vma, address, pmd); -deferred_split_folio (folio, false); -spin_unlock (pmd_ptl); +if (order ==HPAGE_PMD_ORDER ) { + pgtable =pmd_pgtable (_pmd); + _pmd =folio_mk_pmd (folio, vma->vm_page_prot);Quoted 8 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1299
Quoted 4 lines
mm/khugepaged.c +4
-0+ _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));I know you're refactoring this, but be good to change this to a
WARN_ON_ONCE(),BUG_ON()is verboten unless it's absolutely definitely going to be a kernel nuclear event, so worth changing things up as we go.
Quoted 1 line
Yeah i keep seeing those warning in checkpatch, so Ill go ahead and edit it.
Thanks!
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
mm/khugepaged.c +9
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, address, pmd, _pmd); +update_mmu_cache_pmd (vma, address, pmd); +deferred_split_folio (folio, false); +spin_unlock (pmd_ptl); + }else {/* mTHP collapse */ + mthp_pte =mk_pte (&folio->page, vma->vm_page_prot);
Quoted 7 lines
Quoted 6 lines
I guess it's a rule that each THP or mTHP range spanned must span one and only one folio.
Not sure
&folio->pagehas a future though.Maybe better to use
folio_page(folio, 0)?Ok sounds good I'll use that.
Thanks!
Quoted 8 lines
Quoted 8 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1312
Quoted 5 lines
mm/khugepaged.c +4
-0+ mthp_pte =maybe_mkwrite (pte_mkdirty (mthp_pte), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));having said the above, this is trictly introducing a new
BUG_ON()which is a no-no, please make it aWARN_ON_ONCE().
This one is more important, please do do this.
Quoted 4 lines
Quoted 4 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1313
Quoted 2 lines
mm/khugepaged.c +1
-0+folio_ref_add (folio, (1 << order) -1 );Again no 1 << x please.
(as per David feedback elsewhere, 1UL << instead)
Quoted 8 lines
Quoted 3 lines
Do we do something similar somewhere else for mthp ref counting? Can we share code somehow?
Yeah but IIRC its only like 2 or 3 places that do something like this... most callers to folio_add_* do things in slightly different manners. Maybe something to look into for the future, but I think it will be difficult to generalize it.
OK.
Quoted 9 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1316
Quoted 4 lines
mm/khugepaged.c +3
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, _address, pte, mthp_pte, (1 << order));Please avoid 1 << order, and I think at this point since you reference it a bunch of times, just store a local var like
nr_pagesor sth?yeah not a bad idea!
Thanks!
Quoted 10 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1319
Quoted 4 lines
mm/khugepaged.c +3
-0+update_mmu_cache_range (NULL , vma, _address, pte, (1 << order)); + +smp_wmb ();/* make pte visible before pmd */ Can you give some detail as to why this will work here and why it is necessary?
Other parts of the kernel do it when setting ptes before updating the PMD. I'm not sure if it's necessary, but better safe than sorry.
Unfortunately this is a _totally_ unacceptable justification. We can't put in barriers based on 'better safe than sorry'. You need to analysis this and determine whether or not it's necessary.
So the comment in pmd_install() seems to give an indication:
if (likely(pmd_none(*pmd))) { /* Has another populated it ? */
mm_inc_nr_ptes(mm);
/*
* Ensure all pte setup (eg. pte page lock and page clearing) are
* visible before the pte is made visible to other CPUs by being
* put into page tables.
*
* The other side of the story is the pointer chasing in the page
* table walking code (when walking the page table without locking;
* ie. most of the time). Fortunately, these data accesses consist
* of a chain of data-dependent loads, meaning most CPUs (alpha
* being the notable exception) will already guarantee loads are
* seen in-order. See the alpha page table accessors for the
* smp_rmb() barriers in page table walking code.
*/
smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */ pmd_populate(mm, pmd, *pte);
*pte = NULL;
}
Quoted 7 lines
Quoted 4 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1320
Quoted 2 lines
mm/khugepaged.c +1
-0+pmd_populate (mm, pmd,pmd_pgtable (_pmd));If we're updating PTE entriess why do we need to assign the PMD entry?
We removed the PMD entry for
GUP_fastreasons, then we reinstall the PMD entry after the mTHP is in place. Same as for PMD collapse.
OK a comment to this effect would be useful.
Quoted 21 lines
Quoted 21 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1322
Quoted 3 lines
mm/khugepaged.c +2
-0+spin_unlock (pmd_ptl); + }This deeply, badly needs to be refactored into something that both shares code and separates out these two operations.
This function is disgustingly long as it is, and that's not your fault, but let's try to make things better as we go.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1329
Quoted 7 lines
mm/khugepaged.c +3
-0folio =NULL ; result =SCAN_SUCCEED ; +out_unlock_anon_vma: +if (order !=HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Obviously again as above, we need to simplify this and get rid of this whole bit.
Quoted 32 lines
Quoted 5 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1332
Quoted 3 lines
mm/khugepaged.c +2
-0out_up_write: +if (pte) +pte_unmap (pte);OK I guess you moved this from above down here? Is this a valid place to do this?
Yes if not we were potentially unmapping a pte early.
Quoted 26 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1335
Quoted 4 lines
mm/khugepaged.c +1
-0mmap_write_unlock (mm); out_nolock: + *mmap_locked = false;This is kind of horrible, we now have pretty mad logic around who sets
mmap_lockedand where.Can we just do this at the call sites so we avoid that?
I mean anything we do with this is hideous, but that'd be less confusing It hink.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1342
Quoted 7 lines
mm/khugepaged.c +1
-0if (folio)folio_put (folio);trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result);return result; } +/* Recursive function to consume the bitmap */ Err... please don't? Kernel stack is a seriously finite resource, we do not want recursion at all.
But I'm not actually seeing any recursion here? Am I missing something?
Yup this was before I realised it was an iterative implementation :)
Though we are putting load on the stack anyway...
Quoted 7 lines
Quoted 6 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1345
Quoted 3 lines
mm/khugepaged.c +3
-0+static int collapse_scan_bitmap (struct mm_struct *mm,unsigned long address, +int referenced,int unmapped,struct collapse_control *cc, +bool *mmap_locked,unsigned long enabled_orders)This is a complicated and confusing function, it requires a comment describing how it works.
Ok will do!
This is _VERY_ key - I need this I think to be able to sanely review this code.
Quoted 11 lines
Quoted 9 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1351
Quoted 7 lines
mm/khugepaged.c +6
-0+{ +u8 order, next_order; +u16 offset, mid_offset; +int num_chunks; +int bits_set, threshold_bits; +int top =-1 ;Err why do we start at -1 then immediately increment it?
You are correct, it was probably a leftover bit from my development phase. Seems I can just set it to 0 to begin with.
Thanks!
Quoted 8 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1355
Quoted 5 lines
mm/khugepaged.c +4
-0+int collapsed =0 ; +int ret; +struct scan_bit_state state; +bool is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));Extraneous outer parens.
ack
Thanks!
Quoted 10 lines
Quoted 10 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1358
Quoted 4 lines
mm/khugepaged.c +3
-0+ + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + {HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ,0 };This is the same as
cc->mthp_bitmap_stack[0] = ...; top = 1;
Quoted 7 lines
Quoted 1 line
No?
no it would be bitmap_stack[0] = ...
then top goes to -1 (at state =... ), and if we add more items (next_order) to the stack it would go top = 1 (adds one for each half of the split)
OK, again going to need that comment to really grok all this... :)
Quoted 6 lines
Quoted 4 lines
This is really horrible. Can we just have a helper function for this please?
Seems kinda excessive for 4 lines and one caller.
It's code that's very horrible to understand using a very unusual way of initialising a struct within the kernel.
And having code be understandable when the compiler can inline for us does indeed justify this even in the instance of 4 lines and one caller :)
Quoted 13 lines
Quoted 13 lines
Like:
static int mthp_push_stack(struct collapse_control *cc, int index, u8 order, u16 offset) { struct scan_bit_state *state = &cc->mthp_bitmap_stack[index]; VM_WARN_ON(index >= MAX_MTHP_BITMAP_SIZE); state->order = order; state->offset = offset;
Quoted 6 lines
Quoted 2 lines
return index + 1; }This would not work in its current state because its ++index in the current implementation. I would need to refactor, but the general idea still stands
OK thanks.
Something _like_ this would make things a great deal clearer.
Quoted 19 lines
Quoted 11 lines
And can invoke via:
top = mthp_push_stack(cc, top, order, offset);Or pass index as a pointer possibly also.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1361
Quoted 3 lines
mm/khugepaged.c +3
-0+ +while (top >=0 ) { + state = cc->mthp_bitmap_stack[top--];Quoted 7 lines
OK so this is the recursive bit...
Oh man this function so needs a comment describing what it does, seriously.
I think honestly for sake of my own sanity I'm going to hold off reviewing the rest of this until there's something describing the algorithm, in detail here, above the function.
Quoted 5 lines
It's basically binary recursion with a stack structure, that checks regions of the bitmap in descending order (ie order 9, order 8, ...) if we go to the next order we add two items to the stack (left and right half). I will add a comment describing it at the top of the function.
Right, to reiterate - this needs a _big_ comment.
I'm sorry to split the review of this patch in two on this, because once you do that I'm going to inevitably do the deferred review on the algorithm, but I just feel this is the only sensible way I can determine whether what you intend to do here makes sense and is correctly implemented.
Kinda need that 'what you intend to do' bit. So it should be an expansive and detailed comment explicitly explaining the algorithm and why you're doing it.
Thanks!
Quoted 69 lines
Quoted 69 lines
Quoted 69 lines
mm/khugepaged.c +46
-1+ order = state.order +KHUGEPAGED_MIN_MTHP_ORDER ; + offset = state.offset; + num_chunks =1 << (state.order); +/* Skip mTHP orders that are not enabled */ +if (!test_bit (order, &enabled_orders)) + goto next_order; + +/* copy the relavant section to a new bitmap */ +bitmap_shift_right (cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, +MTHP_BITMAP_SIZE ); + + bits_set =bitmap_weight (cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none -1 ) + >> (HPAGE_PMD_ORDER - state.order); + +/* Check if the region is "almost full" based on the threshold */ +if (bits_set > threshold_bits || is_pmd_only + ||test_bit (order, &huge_anon_orders_always)) { + ret =collapse_huge_page (mm, address, referenced, unmapped, + cc, mmap_locked, order, + offset *KHUGEPAGED_MIN_MTHP_NR ); +if (ret ==SCAN_SUCCEED ) { + collapsed += (1 << order); +continue ; + } + } + +next_order: +if (state.order >0 ) { + next_order = state.order -1 ; + mid_offset = offset + (num_chunks /2 ); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, offset }; + } + } +return collapsed; +} + static int collapse_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked,@@ -1307,31 +1406,60 @@ static int collapse_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; +int i;int result =SCAN_FAIL , referenced =0 ;int none_or_zero =0 , shared =0 ;struct page *page =NULL ;struct folio *folio =NULL ;unsigned long _address; +unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +bool is_pmd_only;bool writable = false; - +int chunk_none_count =0 ; +int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ); +unsigned long tva_flags = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;VM_BUG_ON (address & ~HPAGE_PMD_MASK ); result =find_pmd_or_thp_or_none (mm, address, &pmd);if (result !=SCAN_SUCCEED ) goto out;
Quoted 7 lines
Quoted 4 lines
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1430
Quoted 2 lines
mm/khugepaged.c +2
-0+bitmap_zero (cc->mthp_bitmap,MAX_MTHP_BITMAP_SIZE ); +bitmap_zero (cc->mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Having this 'temp' thing on the stack for everyone is just horrid.
As I mention above this serves a very good purpose, and is also expanded in another series by Baolin to serve another similar purpose too.
Yeah, I'm not hugely convinced :) responded there.
I can probably give more useful feedback on this once the big comment is added!
Quoted 40 lines
Quoted 14 lines
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1440
Quoted 11 lines
mm/khugepaged.c +8
-0memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); + +if (cc->is_khugepaged) + enabled_orders =thp_vma_allowable_orders (vma, vma->vm_flags, + tva_flags,THP_ORDERS_ALL_ANON ); +else + enabled_orders =BIT (HPAGE_PMD_ORDER ); + + is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));This is horrid, can we have a function broken out to do this please?
Quoted 2 lines
In general if you keep open coding stuff, just write a static function for it, the compiler is smart enough to inline.
ok, we do this is a few places so perhaps its the best approach.
Quoted 23 lines
Quoted 22 lines
mm/khugepaged.c +13
-2+ pte =pte_offset_map_lock (mm, pmd, address, &ptl);if (!pte) { result =SCAN_PMD_NULL ; goto out; } -for (_address = address, _pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, _address +=PAGE_SIZE ) { +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +/* +* we are reading in KHUGEPAGED_MIN_MTHP_NR page chunks. if +* there are pages in this chunk keep track of it in the bitmap +* for mTHP collapsing. +*/ +if (i %KHUGEPAGED_MIN_MTHP_NR ==0 ) { +if (i >0 && chunk_none_count <= scaled_none) +bitmap_set (cc->mthp_bitmap, + (i -1 ) /KHUGEPAGED_MIN_MTHP_NR ,1 ); + chunk_none_count =0 ; + }
Quoted 54 lines
Quoted 4 lines
This whole thing is really confusing and you are not explaining the algoritm here at all.
This requires a comment, and really this bit should be separated out please.
This used to be its own commit, but multiple people wanted it squashed... ugh. Which should we go with?
Quoted 48 lines
Quoted 33 lines
mm/khugepaged.c +7
-2+ + _pte = pte + i; + _address = address + i *PAGE_SIZE ;pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { ++unmapped;@@ -1354,10 +1482,11 @@ static int collapse_scan_pmd(struct mm_struct *mm, } }if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++chunk_none_count; ++none_or_zero;if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + (!cc->is_khugepaged || !is_pmd_only || + none_or_zero <= khugepaged_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -1453,6 +1582,7 @@ static int collapse_scan_pmd(struct mm_struct *mm, address))) referenced++; } +if (!writable) { result =SCAN_PAGE_RO ; } else if (cc->is_khugepaged &&@@ -1465,10 +1595,12 @@ static int collapse_scan_pmd(struct mm_struct *mm, out_unmap:pte_unmap_unlock (pte, ptl);if (result ==SCAN_SUCCEED ) {mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1603
Quoted 10 lines
mm/khugepaged.c +6
-4- result =collapse_huge_page (mm, address, referenced, - unmapped, cc); -/* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + result =collapse_scan_bitmap (mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); +if (result >0 ) + result =SCAN_SUCCEED ; +else + result =SCAN_FAIL ;We're reusing result as both an enum value and as a storage for unmber colapsed PTE entries?
Can we just use a new local variable? Thanks
Again you've skipped a ton of review comments here. You need to respond to everything.
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
mm/khugepaged.c +0
-1} out: trace_mm_khugepaged_scan_pmd(mm, folio, writable, referenced, --2.50.1
I will review the bitmap/chunk stuff in more detail once the algorithm is commented.
ok thanks for the review.
No problem! :)
Re: [PATCH v10 06/13] khugepaged: add mTHP support
Quoted 3 lines · Lorenzo Stoakes
You've not responded to a ton of comments, I'm guesing I should assume in those cases you're acking the comments implicitly? :) Do let me know.
Yes! If they are obvious nits, then implicit ack.
Sorry I wrote my response over a couple of days while making changes based on the review and trying to figure out a solution to the "creep" issue, and forgot to thoroughly review all the comments before sending this out. I'll make sure to go through it again.
Quoted 49 lines · Nico Pache
Quoted 47 lines
Quoted 45 lines · Lorenzo Stoakes
Quoted 28 lines · Nico Pache
Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in
KHUGEPAGED_MIN_MTHP_ORDERchunks via a bitmap. Each bit represents a utilized region of orderKHUGEPAGED_MIN_MTHP_ORDERptes. If mTHPs are enabled we remove the restriction ofmax_ptes_noneduring the scan phase so we don't bailout early and miss potential mTHP candidates.A new function
collapse_scan_bitmapis used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion.max_ptes_nonewill be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "
max_ptes_none> HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.Quoted 7 lines
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently
madv_collapseis not supported and will only attempt PMD collapse.Yes I think this has to remain the case unfortunately as we override sysfs-specified orders for
MADV_COLLAPSEand there's no sensible way to determine what order we ought to be using.Quoted 2 lines
Signed-off-by Nico Pache <[email protected]>
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
You've gone from small incremental changes to a huge one here... for the sake of reviewer sanity at least, any chance of breaking this up?
I had this as two patches (one for the bitmap and one for implementing it), but I was asked to squash them :/
Yeah it makes sense to combine those two, but maybe there's a better way of splitting things out.
I'll give it a shot but most of this patch belongs together in my eyes.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)include/linux/khugepaged.h +0
-0include/linux/khugepaged.h line 4
Quoted 4 lines
include/linux/khugepaged.h +1
-0@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2I guess this makes sense as by definition 2 pages is least it could possibly be.
Quoted 4 lines
Quoted 1 line
Order, so 4 pages, 16kB mTHP
Right, misread that! :) sorry. I guess then in fact there is no such thing as an order-1 mTHP? I did wonder how useful that'd be so makes more sense
Anon memory does not support order 1 THPs, im not sure if there are any plans do so either.
Quoted 12 lines
Quoted 6 lines
Quoted 5 lines
include/linux/khugepaged.h line 5
Quoted 3 lines
include/linux/khugepaged.h +1
-0+#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER)Surely
KHUGEPAGED_MIN_NR_MTHP_PTESwould be more meaningful?Sure!
Thanks!
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This is confusing - size of what?
We need it like this due to ppc64 (and maybe others?), it used to be based on
PMD_ORDER, but some arches fail to compile due to the PMD size only being known at boot time.That really sucks. Do please put this context in a comment though (see below for more discussion on this).
Ok, will do. although with my most recent changes we might just be able to get rid of this all together.
Quoted 7 lines
Quoted 4 lines
This compiles to 9 on arches that have 512 ptes.
so 1 << (9 - 2) == 128What I'm saying is - what is this expressed in? There's no information on that here.
In my next implementation its probably going to be gone, but what it indicates is the bitmap size (128), where each bit equals a range of 4 pages. 128 *4 = 512.
Quoted 7 lines
So from what you say I can see it's the number of bits required in the bitmap, because we're being smart and only bothering to mark entries at a granularity of minimum mTHP size.
OK so the idea is this is the number of PTE entries per entry in the bitmap.
Yeah it was an optimization that I was going for, but on second thought it's probably pretty useless and causes some inaccuracies in the max_ptes_none scaling and collapsing.
Quoted 4 lines
This is KEY missing context. We really need to spell things out here, the THP code is... confusing :) to put it politely, so we need to be especially careful to be mega clear.
Yeah its rather confusing, sorry, but I think my newer implementation that gets rid of this is actually much cleaner.
Quoted 3 lines
So please please PLEASE add a comment explaining all this. And clearly state that the unit here is bits.
ack, I'll be more clear with my newer implementation, and it should be easier to understand anyways.
Quoted 18 lines
Quoted 6 lines
Quoted 5 lines
If it's number of bits surely this should be
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER?This would only be 7? We need a 128 bit bitmap
Again missing context is you're storing bits per minimum mTHP as per above.
Quoted 9 lines
Quoted 9 lines
This seems to be more so 'the maximum value that could contain the bits right?
I think this is just wrong though, see below at
DECLARE_BITMAP()stuff.include/linux/khugepaged.h line 7
Quoted 1 line
include/linux/khugepaged.h +1
-0+#define MTHP_BITMAP_SIZE (1 << (HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER))Hard to know how this relates to
MAX_MTHP_BITMAP_SIZE?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I guess this is the current bitmap size indicating all that is possible, but if these are all #define's what is this accomplishing?
One for compile time one for runtime. Kind of annoying but it was the easiest solution given the architecture limitations.
OK, this context is fantastic + important for understanding, so let's put it in a comment :)
Or just get rid of it ;)
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
For all - please do not do (1 << xxx)! This can lead to sign-extension bugs at least in theory, use
_BITUL(...), it's neater too.ack, thanks!
OK sorry, based on David's feedback on this - just use 1UL << xxx here instead.
ack
Quoted 87 lines
(An aside that isn't really relevant now but - also I was wrong to suggest
_BITUL()anywayBIT()is The Way (TM), for some reason I had it in my head that the former was better :P)Quoted 34 lines
Quoted 26 lines
NIT but the whitespace is all screwed up here.
KHUGEPAGED_MIN_MTHP_ORDERandKHUGEPAGED_MIN_MTHP_NRQuoted 21 lines
include/linux/khugepaged.h +0
-0#include <linux/mm.h> mm/khugepaged.c +5
-0@@ -94,6 +94,11 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);static struct kmem_cache *mm_slot_cache__ro_after_init ; +struct scan_bit_state { +u8 order; +u16 offset; +}; +struct collapse_control {bool is_khugepaged;@@ -102,6 +107,18 @@ struct collapse_control {Quoted 8 lines
mm/khugepaged.c struct collapse_control { · line 113
Quoted 6 lines
mm/khugepaged.c +4
-0/* nodemask for allocation fallback */ nodemask_t alloc_nmask; + +/* +* bitmap used to collapse mTHP sizes. +* 1bit = order KHUGEPAGED_MIN_MTHP_ORDER mTHP I'm not sure what this '1bit = xxx' comment means?
Quoted 1 line
A single bit represents 1 <<
MIN_MTHP_ORDER(4) pages. Ill express that betterYeah again, this is the missing context that would have helped me feel a lot less confused here :P
Quoted 30 lines
Quoted 5 lines
mm/khugepaged.c struct collapse_control { · line 115
Quoted 3 lines
mm/khugepaged.c +2
-0+*/ +DECLARE_BITMAP (mthp_bitmap,MAX_MTHP_BITMAP_SIZE );Hmm this seems wrong.
Should be a bitmap with 128 bits (for 4k page size). Not sure what's wrong here.
Quoted 14 lines
DECLARE_BITMAP(..., val)is expessed as:#define
DECLARE_BITMAP(name,bits)\
unsigned long name[BITS_TO_LONGS(bits)]So the 2nd param should be number of bits.
But
MAX_MTHP_BITMAP_SIZEis:(1 << (
ilog2(MAX_PTRS_PER_PTE)-KHUGEPAGED_MIN_MTHP_ORDER))So typically:
Quoted 8 lines
(1 << (9 - 2)) = 128
And
BITS_TO_LONGSis defined as:
__KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))So essentially this will be 128 / 8 on a 64-bit system so 16 bytes to store... 7 bits?
I think you mean 64. 8 would be
BYTES_PER_TYPEQuoted 1 line
Quoted 7 lines
Quoted 1 line
Unless I'm missing something here?
Hmm, unless the
DECLARE_BITMAPis being used incorrectly in multiple places,DECLARE_BITMAP(..., # of bits)is how this is intended to be used.I think it's an array of unsigned longs, so each part of the name[] is already 64 bits. hence the divide.
Yup again this is due to the missing context above.
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
mm/khugepaged.c +1
-0+DECLARE_BITMAP (mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Same comment as above obviously. But also this is kind of horrible, why are we putting a copy of this entire bitmap on the stack every time we declare a cc?
The temp one is used as a scratch pad, Baolin also finds this useful in his file mTHP collapse useful for another use as well.
Saying 'scratch pad' is really just saying 'temporary' using different words :) this isn't hugely helpful.
When doing the "recursion", there is no easy way to read the bitmap between bits N0-N1, so we must manipulate it to extract that data; however, we still want to preserve the original state of the bitmap. The temp_bitmap is used to do these bit manipulations while not destroying the original state. I will see if there's a better way to read arbitrary ranges of the bitmap so we dont need to have two copies, but IIRC there is not-- I might have to expand the bitmap API to include this.
Quoted 5 lines
If we _need_ this we should give this a better name, and I also don't know why we need this on the stack for all
collapse_controlusers. Can't you just have your 'scratch pad' local to the function that needs it or passed as a pointer?
If I cant find a solution to the problem above I will look into moving it to the local stack.
Quoted 3 lines
I'm sure it'd be useful to somebody to stick various temporary things in
vm_area_structfor instance, but it'd be hugely egregious to do so...
The reason I didnt see this as a huge issue, is there is only one global CC that is reused for all khugepaged runs. It's a one time allocation and does not scale.
Quoted 128 lines
Quoted 4 lines
In general khugepaged always uses the same CC, so it doesn't not having to constantly allocate this.
You're putting a LOT of data on the stack, and kernel stacks are very delicate, I'm quite concerned about this.
BEFORE:
struct collapse_control { bool is_khugepaged; /* 0 1 */ /* XXX 3 bytes hole, try to pack */ u32 node_load[64]; /* 4 256 */ /* XXX 4 bytes hole, try to pack */ /* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */ nodemask_t alloc_nmask; /* 264 8 */ /* size: 272, cachelines: 5, members: 3 */ /* sum members: 265, holes: 2, sum holes: 7 */ /* last cacheline: 16 bytes */ };272 bytes.
AFTER:
struct collapse_control { bool is_khugepaged; /* 0 1 */ /* XXX 3 bytes hole, try to pack */ u32 node_load[64]; /* 4 256 */ /* XXX 4 bytes hole, try to pack */ /* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */ nodemask_t alloc_nmask; /* 264 8 */ long unsigned int mthp_bitmap[2]; /* 272 16 */ long unsigned int mthp_bitmap_temp[2]; /* 288 16 */ struct scan_bit_state mthp_bitmap_stack[128]; /* 304 512 */ /* size: 816, cachelines: 13, members: 6 */ /* sum members: 809, holes: 2, sum holes: 7 */ /* last cacheline: 48 bytes */ };816 bytes.
So you're roughly tripling the size of this struct and making this a thing in all callstacks that reference
collapse_control.Kernel stack is a _super_ finite resource.
I mean I can't really review your stack implementation thing until you've added a comment to help me understand what you are doing there (sorry but it's just too fiddly for me to first principles it), but I wonder if we truly need to be doing this?
I wonder if we can just put this into somewhere allocated...
Quoted 10 lines
Quoted 9 lines
mm/khugepaged.c struct collapse_control { · line 122
Quoted 7 lines
mm/khugepaged.c +5
-0+struct scan_bit_state mthp_bitmap_stack[MAX_MTHP_BITMAP_SIZE ]; +}; + +struct collapse_control khugepaged_collapse_control = { + .is_khugepaged = true, };Why are we moving this here?
Because if not it doesn't compile.
A reason as to why this is the case would be helpful :)
But fair enough!
Quoted 48 lines
Quoted 48 lines
Quoted 23 lines
mm/khugepaged.c +2
-5/** @@ -854,10 +871,6 @@ static void khugepaged_alloc_sleep(void)remove_wait_queue (&khugepaged_wait, &wait); } -struct collapse_control khugepaged_collapse_control = { - .is_khugepaged = true, -}; -static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;@@ -1136,17 +1149,19 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,static int collapse_huge_page (struct mm_struct *mm,unsigned long address,int referenced,int unmapped, -struct collapse_control *cc) +struct collapse_control *cc,bool *mmap_locked, +unsigned int order,unsigned long offset) {LIST_HEAD (compound_pagelist);mm/khugepaged.c static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, · line 1164
Quoted 10 lines
mm/khugepaged.c +2
-1pmd_t *pmd, _pmd; -pte_t *pte; +pte_t *pte =NULL , mthp_pte;pgtable_t pgtable;struct folio *folio;spinlock_t *pmd_ptl, *pte_ptl;int result =SCAN_FAIL ;struct vm_area_struct *vma;struct mmu_notifier_range range; +unsigned long _address = address + offset *PAGE_SIZE ;This name is really horrible. please name it sensibly.
It feels like address ought to be consistently the base of the THP or mTHP we wish to collapse, and if we need something PMD aligned for some reason we should rename _that_ to e.g.
pmd_address.Orrr it could be
mthp_address...Perhaps we could just figure that out here and pass only the address... aligning to PMD boundary shouldn't be hard/costly.
But it may indicate we need further refactorisation so we don't need to paper over cracks + pass around a PMD address to do things when that may not be where the (m)THP range begins.
Quoted 8 lines
Quoted 3 lines
Ok i'll rename them, but we still need to know the PMD address as we rely on it for a few key operations.
Can we leave_addressand rename address topmd_address?No, we absolutely cannot leave
_addressas '_address', that's terrible and I'm just not going to live with that.I know it's trivial, but it's just such a horrid naming convention.
Ok Ill rename it to something else!
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 17 lines
mm/khugepaged.c +5
-1VM_BUG_ON (address & ~HPAGE_PMD_MASK );@@ -1155,16 +1170,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* The allocation can take potentially a long time if it involves * sync compaction, and we do not need to hold the mmap_lock during * that. We will recheck the vma after taking it again in write mode. +* If collapsing mTHPs we may have already released the read_lock. */ -mmap_read_unlock (mm); +if (*mmap_locked) { +mmap_read_unlock (mm); + *mmap_locked = false; + }mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1186
Quoted 10 lines
mm/khugepaged.c +3
-3- result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out_nolock;mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + *mmap_locked = true; + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));I mean this is kind of going back to previous commits, but it's really ugly to pass a BIT(xxx) here, is that really necessary? Can't we just pass in the order?
Quoted 7 lines
Quoted 5 lines
Yes and no... currently we only ever pass the bit of the current order so we could get away with it, but to generalize it we want the ability to pass a bitmap of the available orders. Like in the case of future
madvise_collapsesupport, we would need to pass a bitmap of possible orders.Can we just change that when we need it?
Sure, it was more so that I generalized it for madv_collapse and khugepaged, but then we dropped madv_collapse support, and I never undid the change.
Quoted 6 lines
'Future proofing' for an possible future implementation detail is just not how we should implement things.
Right now we don't do that, so let's just pass the order. If in future we want to change it we can.
ack
Quoted 79 lines
Quoted 79 lines
Quoted 79 lines
It's also inconsistent with other calls like
e.g.__collapse_huge_page_swapin()below which passes the order.Same goes obv. for all such invocations.
Quoted 31 lines
mm/khugepaged.c +4
-4if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm); goto out_nolock;@@ -1182,13 +1201,14 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ - result =__collapse_huge_page_swapin (mm, vma, address, pmd, - referenced,HPAGE_PMD_ORDER ); + result =__collapse_huge_page_swapin (mm, vma, _address, pmd, + referenced, order);if (result !=SCAN_SUCCEED ) goto out_nolock; }mmap_read_unlock (mm); + *mmap_locked = false;/* * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM @@ -1198,8 +1218,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));if (result !=SCAN_SUCCEED ) goto out_up_write;/* check if the pmd is still valid */ mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1233
Quoted 7 lines
mm/khugepaged.c +2
-2@@ -1210,11 +1229,12 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_lock_write (vma->anon_vma); -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, address, - address +HPAGE_PMD_SIZE ); +mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, _address, + _address + (PAGE_SIZE << order));This
_addressis horrible. That really does have to change.mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1237
Quoted 4 lines
mm/khugepaged.c +1
-0mmu_notifier_invalidate_range_start (&range); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ +Odd whitespace...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1251
Quoted 9 lines
mm/khugepaged.c +1
-1/* * This removes any huge TLB entry from the CPU so we won't allow * huge and small TLB entries for the same virtual address to @@ -1228,19 +1248,16 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,mmu_notifier_invalidate_range_end (&range);tlb_remove_table_sync_one (); - pte =pte_offset_map_lock (mm, &_pmd, address, &pte_ptl); + pte =pte_offset_map_lock (mm, &_pmd, _address, &pte_ptl);I see we already have a 'convention' of _ prefix on the pmd param, but two wrongs don't make a right...
Quoted 11 lines
mm/khugepaged.c +2
-3if (pte) { - result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist, -HPAGE_PMD_ORDER ); + result =__collapse_huge_page_isolate (vma, _address, pte, cc, + &compound_pagelist, order);spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ; }
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1243
Quoted 3 lines
mm/khugepaged.c +0
-2if (unlikely (result !=SCAN_SUCCEED )) { -if (pte) -pte_unmap (pte);Why are we removing this?
You're missing some of the comments, I'm guesing for most of the smaller stuff you're just implicitly acking them but this one was a question :)
Ah, for some of your reviews you comment, then later when you see another piece you say something like "ah ok this is why", for a lot of these cases I might have skipped replying because it seems that you figured it out.
For this one we moved the pte_unmap further down, because if not, we could be prematurely unmapping a PTE, which is later referenced during the mTHP collapse, which was not the case when we only did PMD collapse.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 7 lines
mm/khugepaged.c +0
-0spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));/* @@ -1255,17 +1272,17 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, }mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1279
Quoted 10 lines
mm/khugepaged.c +4
-3/* -* All pages are isolated and locked so anon_vma rmap -* can't run anymore. +* For PMD collapse all pages are isolated and locked so anon_vma +* rmap can't run anymore */ -anon_vma_unlock_write (vma->anon_vma); +if (order ==HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Hmm this is introducing a horrible new way for things to go wrong. And there's now a whole host of terrible error paths that can go wrong very easily around rmap locks and yeah, no way we cannot do it this way.
rmap locks are VERY sensitive and the ordering of the locking matters a great deal (see top of
mm/rmap.c). So we have to be SO careful here.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
I suggest you simply have a boolean '
anon_vma_locked' or something like this, and get rid of these horrible additional code paths and the second order ==HPAGE_PMD_ORDERcheck.We'll track whether or not the lock is held and thereby needs releasing that way instead.
You've not responded to this suggestion re: refactoring here.
I think if anything this refactoring adds complexity... we simply postpone the unlock we normally do, to later if its mTHP collapse, because we havent fully isolated all the pages (like we do in the PMD case).
We arent doing anything extremely different other than hold the already held lock for a little longer. We would still need these order == HPAGE_PMD_ORDER checks, but just be flipping a flag that says its locked, essentially adding more complexity for the same outcome.
Quoted 4 lines
I'm really not a fan of us arbitrarily messing with rmap locks like this, and we should very carefully keep track of whether we have/have not released them.
Its important to note, the code was already doing this, as I stated above, we simply hold it for longer because we haven't isolated all the pages.
Quoted 8 lines
Again rmap locking is a dangerous area, I've got personal experience of this (see top of
mm/rmap.cfor an indication of complexity here as well as
https://kernel.org/doc/html/latest/mm/process_addrs.html).Quoted 3 lines
Quoted 3 lines
Also, and very importantly - are you 100% sure you can't possibly have a deadlock or issue beyond this point if you don't release the rmap lock?
Quoted 7 lines
Quoted 4 lines
I double checked, this was added as a fix to an issue Hugh reported.
The gap between these callers is rather small, and I see no way that it could skip the lock/unlock cycle.
We're going to need more than this to be confident, you need to clearly justify why we won't encounter issues this way.
// WE ARE HOLDING THE LOCK ALREADY
if (order == HPAGE_PMD_ORDER) //ONLY RELEASE IF ITS PMD
anon_vma_unlock_write(vma->anon_vma);
result = __collapse_huge_page_copy(pte, folio, pmd, _pmd,
vma, _address, pte_ptl,
order, &compound_pagelist);
if (unlikely(result != SCAN_SUCCEED))
goto out_unlock_anon_vma; //LETS RELEASE THE LOCK IF WE FAIL
if (order == HPAGE_PMD_ORDER) {
...
// NO EXIT PATHS
} else { /* mTHP collapse */
...
// NO EXIT PATHS
}
folio = NULL;
result = SCAN_SUCCEED;
out_unlock_anon_vma:
if (order != HPAGE_PMD_ORDER)
anon_vma_unlock_write(vma->anon_vma); //RELEASE IF IT WASN'TRELEASED EARLIER
Not sure if this is the verification you are looking for, but I really dont see how this could go wrong, or how this is any different than what the khugepaged code is already doing in a number of places.
Quoted 89 lines
Quoted 55 lines
Quoted 46 lines
This is veeeery important, as there can be implicit assumptions around whether or not one can acquire these locks and you basically have to audit ALL code over which this lock is held.
I'm speaking from hard experience here having bumped into this in various attempts at work relating to this stuff...
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1285
Quoted 10 lines
mm/khugepaged.c +3
-4result =__collapse_huge_page_copy (pte, folio, pmd, _pmd, - vma, address, pte_ptl, - &compound_pagelist,HPAGE_PMD_ORDER ); -pte_unmap (pte); + vma, _address, pte_ptl, + &compound_pagelist, order);if (unlikely (result !=SCAN_SUCCEED )) - goto out_up_write; + goto out_unlock_anon_vma;See above...
Quoted 24 lines
mm/khugepaged.c +3
-14/* * The smp_wmb() inside __folio_mark_uptodate() ensures the @@ -1273,33 +1290,115 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* write. */ __folio_mark_uptodate (folio); - pgtable =pmd_pgtable (_pmd); - - _pmd =folio_mk_pmd (folio, vma->vm_page_prot); - _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); - -spin_lock (pmd_ptl); -BUG_ON (!pmd_none (*pmd)); -folio_add_new_anon_rmap (folio, vma, address,RMAP_EXCLUSIVE ); -folio_add_lru_vma (folio, vma); -pgtable_trans_huge_deposit (mm, pmd, pgtable); -set_pmd_at (mm, address, pmd, _pmd); -update_mmu_cache_pmd (vma, address, pmd); -deferred_split_folio (folio, false); -spin_unlock (pmd_ptl); +if (order ==HPAGE_PMD_ORDER ) { + pgtable =pmd_pgtable (_pmd); + _pmd =folio_mk_pmd (folio, vma->vm_page_prot);Quoted 8 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1299
Quoted 4 lines
mm/khugepaged.c +4
-0+ _pmd =maybe_pmd_mkwrite (pmd_mkdirty (_pmd), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));I know you're refactoring this, but be good to change this to a
WARN_ON_ONCE(),BUG_ON()is verboten unless it's absolutely definitely going to be a kernel nuclear event, so worth changing things up as we go.Quoted 1 line
Yeah i keep seeing those warning in checkpatch, so Ill go ahead and edit it.
Thanks!
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
mm/khugepaged.c +9
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +set_pmd_at (mm, address, pmd, _pmd); +update_mmu_cache_pmd (vma, address, pmd); +deferred_split_folio (folio, false); +spin_unlock (pmd_ptl); + }else {/* mTHP collapse */ + mthp_pte =mk_pte (&folio->page, vma->vm_page_prot);Quoted 7 lines
Quoted 6 lines
I guess it's a rule that each THP or mTHP range spanned must span one and only one folio.
Not sure
&folio->pagehas a future though.Maybe better to use
folio_page(folio, 0)?Ok sounds good I'll use that.
Thanks!
Quoted 8 lines
Quoted 8 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1312
Quoted 5 lines
mm/khugepaged.c +4
-0+ mthp_pte =maybe_mkwrite (pte_mkdirty (mthp_pte), vma); + +spin_lock (pmd_ptl); +BUG_ON (!pmd_none (*pmd));having said the above, this is trictly introducing a new
BUG_ON()which is a no-no, please make it aWARN_ON_ONCE().
Quoted 8 lines
This one is more important, please do do this.
Quoted 4 lines
Quoted 4 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1313
Quoted 2 lines
mm/khugepaged.c +1
-0+folio_ref_add (folio, (1 << order) -1 );Again no 1 << x please.
(as per David feedback elsewhere, 1UL << instead)
ack
Quoted 52 lines
Quoted 9 lines
Quoted 4 lines
Do we do something similar somewhere else for mthp ref counting? Can we share code somehow?
Yeah but IIRC its only like 2 or 3 places that do something like this... most callers to folio_add_* do things in slightly different manners. Maybe something to look into for the future, but I think it will be difficult to generalize it.
OK.
Quoted 9 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1316
Quoted 4 lines
mm/khugepaged.c +3
-0+folio_add_new_anon_rmap (folio, vma, _address,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, _address, pte, mthp_pte, (1 << order));Please avoid 1 << order, and I think at this point since you reference it a bunch of times, just store a local var like
nr_pagesor sth?yeah not a bad idea!
Thanks!
Quoted 10 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1319
Quoted 4 lines
mm/khugepaged.c +3
-0+update_mmu_cache_range (NULL , vma, _address, pte, (1 << order)); + +smp_wmb ();/* make pte visible before pmd */ Can you give some detail as to why this will work here and why it is necessary?
Other parts of the kernel do it when setting ptes before updating the PMD. I'm not sure if it's necessary, but better safe than sorry.
Unfortunately this is a _totally_ unacceptable justification. We can't put in barriers based on 'better safe than sorry'. You need to analysis this and determine whether or not it's necessary.
So the comment in
pmd_install()seems to give an indication:if (likely(pmd_none(*pmd))) { /* Has another populated it ? */
mm_inc_nr_ptes(mm);
/*
* Ensure all pte setup (eg. pte page lock and page clearing) are
* visible before the pte is made visible to other CPUs by being
* put into page tables.
*
* The other side of the story is the pointer chasing in the page
* table walking code (when walking the page table without locking;
* ie. most of the time). Fortunately, these data accesses consist
* of a chain of data-dependent loads, meaning most CPUs (alpha
Quoted 8 lines
* being the notable exception) will already guarantee loads are
* seen in-order. See the alpha page table accessors for the
*smp_rmb()barriers in page table walking code. */
smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */pmd_populate(mm, pmd, *pte);
*pte = NULL;
}
So this seems to indicate we do need it for some rare edge cases/arches, and that's why it exists in other callers too.
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1320
Quoted 3 lines
mm/khugepaged.c +1
-0+pmd_populate (mm, pmd,pmd_pgtable (_pmd));If we're updating PTE entriess why do we need to assign the PMD entry?
We removed the PMD entry for
GUP_fastreasons, then we reinstall the PMD entry after the mTHP is in place. Same as for PMD collapse.OK a comment to this effect would be useful.
Ok ill add a small comment. feel free to take a look at the failure handling around __collapse_huge_page_isolate to see another example of this. We remove the pmd (pmdp_collapse_flush), then if the isolation fails, we repopulate the original PMD. In the mTHP case, because we are not installing a new PMD, we still remove the PMD (for gup-fast reasons), modify the PTE entries within that PMD, then reinstall the original PMD. This prevent GUP-fast from accessing these PTEs/PMD mid change.
Quoted 52 lines
Quoted 52 lines
Quoted 19 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1322
Quoted 4 lines
mm/khugepaged.c +2
-0+spin_unlock (pmd_ptl); + }This deeply, badly needs to be refactored into something that both shares code and separates out these two operations.
This function is disgustingly long as it is, and that's not your fault, but let's try to make things better as we go.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1329
Quoted 7 lines
mm/khugepaged.c +3
-0folio =NULL ; result =SCAN_SUCCEED ; +out_unlock_anon_vma: +if (order !=HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Quoted 8 lines
Obviously again as above, we need to simplify this and get rid of this whole bit.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1332
Quoted 3 lines
mm/khugepaged.c +2
-0out_up_write: +if (pte) +pte_unmap (pte);OK I guess you moved this from above down here? Is this a valid place to do this?
Yes if not we were potentially unmapping a pte early.
Quoted 24 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1335
Quoted 4 lines
mm/khugepaged.c +1
-0mmap_write_unlock (mm); out_nolock: + *mmap_locked = false;This is kind of horrible, we now have pretty mad logic around who sets
mmap_lockedand where.Can we just do this at the call sites so we avoid that?
I mean anything we do with this is hideous, but that'd be less confusing It hink.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1342
Quoted 7 lines
mm/khugepaged.c +1
-0if (folio)folio_put (folio);trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result);return result; } +/* Recursive function to consume the bitmap */ Err... please don't? Kernel stack is a seriously finite resource, we do not want recursion at all.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
But I'm not actually seeing any recursion here? Am I missing something?
Yup this was before I realised it was an iterative implementation :)
Though we are putting load on the stack anyway...
Ill see if I can shrink this load!
Quoted 11 lines
Quoted 8 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1345
Quoted 4 lines
mm/khugepaged.c +3
-0+static int collapse_scan_bitmap (struct mm_struct *mm,unsigned long address, +int referenced,int unmapped,struct collapse_control *cc, +bool *mmap_locked,unsigned long enabled_orders)This is a complicated and confusing function, it requires a comment describing how it works.
Ok will do!
This is _VERY_ key - I need this I think to be able to sanely review this code.
ack, im also simplifying it a bit in the next iteration.
Quoted 48 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
mm/khugepaged.c +6
-0+{ +u8 order, next_order; +u16 offset, mid_offset; +int num_chunks; +int bits_set, threshold_bits; +int top =-1 ;Quoted 3 lines
Quoted 1 line
Err why do we start at -1 then immediately increment it?
You are correct, it was probably a leftover bit from my development phase. Seems I can just set it to 0 to begin with.
Thanks!
Quoted 8 lines
Quoted 7 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1355
Quoted 5 lines
mm/khugepaged.c +4
-0+int collapsed =0 ; +int ret; +struct scan_bit_state state; +bool is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));Extraneous outer parens.
ack
Thanks!
Quoted 10 lines
Quoted 10 lines
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1358
Quoted 4 lines
mm/khugepaged.c +3
-0+ + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + {HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ,0 };This is the same as
cc->mthp_bitmap_stack[0] = ...; top = 1;Quoted 7 lines
Quoted 1 line
No?
no it would be bitmap_stack[0] = ...
then top goes to -1 (at state =... ), and if we add more items (next_order) to the stack it would go top = 1 (adds one for each half of the split)OK, again going to need that comment to really grok all this... :)
Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
This is really horrible. Can we just have a helper function for this please?
Seems kinda excessive for 4 lines and one caller.
It's code that's very horrible to understand using a very unusual way of initialising a struct within the kernel.
ok, ill see if i can clean it up.
Quoted 3 lines
And having code be understandable when the compiler can inline for us does indeed justify this even in the instance of 4 lines and one caller :)
ack
Quoted 50 lines
Quoted 14 lines
Quoted 14 lines
Like:
static int mthp_push_stack(struct collapse_control *cc, int index, u8 order, u16 offset) { struct scan_bit_state *state = &cc->mthp_bitmap_stack[index]; VM_WARN_ON(index >= MAX_MTHP_BITMAP_SIZE); state->order = order; state->offset = offset;Quoted 6 lines
Quoted 2 lines
return index + 1; }This would not work in its current state because its ++index in the current implementation. I would need to refactor, but the general idea still stands
OK thanks.
Something _like_ this would make things a great deal clearer.
Quoted 19 lines
Quoted 11 lines
And can invoke via:
top = mthp_push_stack(cc, top, order, offset);Or pass index as a pointer possibly also.
mm/khugepaged.c static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1361
Quoted 3 lines
mm/khugepaged.c +3
-0+ +while (top >=0 ) { + state = cc->mthp_bitmap_stack[top--];Quoted 7 lines
OK so this is the recursive bit...
Oh man this function so needs a comment describing what it does, seriously.
I think honestly for sake of my own sanity I'm going to hold off reviewing the rest of this until there's something describing the algorithm, in detail here, above the function.
Quoted 5 lines
It's basically binary recursion with a stack structure, that checks regions of the bitmap in descending order (ie order 9, order 8, ...) if we go to the next order we add two items to the stack (left and right half). I will add a comment describing it at the top of the function.
Quoted 7 lines
Right, to reiterate - this needs a _big_ comment.
I'm sorry to split the review of this patch in two on this, because once you do that I'm going to inevitably do the deferred review on the algorithm, but I just feel this is the only sensible way I can determine whether what you intend to do here makes sense and is correctly implemented.
Its all good! This pushed me to come up with a better solution, by both removing the compression into 128 bits, and also come up with a solution to the creep (although its not the 0/511 that you and David were talking about).
Quoted 80 lines
Kinda need that 'what you intend to do' bit. So it should be an expansive and detailed comment explicitly explaining the algorithm and why you're doing it.
Thanks!
Quoted 73 lines
Quoted 73 lines
Quoted 61 lines
mm/khugepaged.c +44
-1+ order = state.order +KHUGEPAGED_MIN_MTHP_ORDER ; + offset = state.offset; + num_chunks =1 << (state.order); +/* Skip mTHP orders that are not enabled */ +if (!test_bit (order, &enabled_orders)) + goto next_order; + +/* copy the relavant section to a new bitmap */ +bitmap_shift_right (cc->mthp_bitmap_temp, cc->mthp_bitmap, offset, +MTHP_BITMAP_SIZE ); + + bits_set =bitmap_weight (cc->mthp_bitmap_temp, num_chunks); + threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none -1 ) + >> (HPAGE_PMD_ORDER - state.order); + +/* Check if the region is "almost full" based on the threshold */ +if (bits_set > threshold_bits || is_pmd_only + ||test_bit (order, &huge_anon_orders_always)) { + ret =collapse_huge_page (mm, address, referenced, unmapped, + cc, mmap_locked, order, + offset *KHUGEPAGED_MIN_MTHP_NR ); +if (ret ==SCAN_SUCCEED ) { + collapsed += (1 << order); +continue ; + } + } + +next_order: +if (state.order >0 ) { + next_order = state.order -1 ; + mid_offset = offset + (num_chunks /2 ); + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, mid_offset }; + cc->mthp_bitmap_stack[++top] = (struct scan_bit_state ) + { next_order, offset }; + } + } +return collapsed; +} + static int collapse_scan_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked,@@ -1307,31 +1406,60 @@ static int collapse_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; +int i;int result =SCAN_FAIL , referenced =0 ;int none_or_zero =0 , shared =0 ;struct page *page =NULL ;struct folio *folio =NULL ;unsigned long _address; +unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +bool is_pmd_only;bool writable = false; - +int chunk_none_count =0 ;mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1430
Quoted 10 lines
mm/khugepaged.c +4
-0+int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ); +unsigned long tva_flags = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;VM_BUG_ON (address & ~HPAGE_PMD_MASK ); result =find_pmd_or_thp_or_none (mm, address, &pmd);if (result !=SCAN_SUCCEED ) goto out; +bitmap_zero (cc->mthp_bitmap,MAX_MTHP_BITMAP_SIZE ); +bitmap_zero (cc->mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Having this 'temp' thing on the stack for everyone is just horrid.
Quoted 8 lines
Quoted 3 lines
As I mention above this serves a very good purpose, and is also expanded in another series by Baolin to serve another similar purpose too.
Yeah, I'm not hugely convinced :) responded there.
I can probably give more useful feedback on this once the big comment is added!
Ok! hopefully I find a solution so I can remove the temp one.
Quoted 94 lines
Quoted 94 lines
Quoted 15 lines
mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1440
Quoted 12 lines
mm/khugepaged.c +8
-0memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); + +if (cc->is_khugepaged) + enabled_orders =thp_vma_allowable_orders (vma, vma->vm_flags, + tva_flags,THP_ORDERS_ALL_ANON ); +else + enabled_orders =BIT (HPAGE_PMD_ORDER ); + + is_pmd_only = (enabled_orders == (1 <<HPAGE_PMD_ORDER ));This is horrid, can we have a function broken out to do this please?
Quoted 2 lines
In general if you keep open coding stuff, just write a static function for it, the compiler is smart enough to inline.
ok, we do this is a few places so perhaps its the best approach.
Quoted 23 lines
Quoted 12 lines
mm/khugepaged.c +3
-2+ pte =pte_offset_map_lock (mm, pmd, address, &ptl);if (!pte) { result =SCAN_PMD_NULL ; goto out; } -for (_address = address, _pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, _address +=PAGE_SIZE ) { +for (i =0 ; i <HPAGE_PMD_NR ; i++) { +/* mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1459
Quoted 10 lines
mm/khugepaged.c +10
-0+* we are reading in KHUGEPAGED_MIN_MTHP_NR page chunks. if +* there are pages in this chunk keep track of it in the bitmap +* for mTHP collapsing. +*/ +if (i %KHUGEPAGED_MIN_MTHP_NR ==0 ) { +if (i >0 && chunk_none_count <= scaled_none) +bitmap_set (cc->mthp_bitmap, + (i -1 ) /KHUGEPAGED_MIN_MTHP_NR ,1 ); + chunk_none_count =0 ; + }Quoted 4 lines
This whole thing is really confusing and you are not explaining the algoritm here at all.
This requires a comment, and really this bit should be separated out please.
This used to be its own commit, but multiple people wanted it squashed... ugh. Which should we go with?
Quoted 47 lines
Quoted 33 lines
mm/khugepaged.c +7
-2+ + _pte = pte + i; + _address = address + i *PAGE_SIZE ;pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { ++unmapped;@@ -1354,10 +1482,11 @@ static int collapse_scan_pmd(struct mm_struct *mm, } }if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++chunk_none_count; ++none_or_zero;if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + (!cc->is_khugepaged || !is_pmd_only || + none_or_zero <= khugepaged_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -1453,6 +1582,7 @@ static int collapse_scan_pmd(struct mm_struct *mm, address))) referenced++; } +if (!writable) { result =SCAN_PAGE_RO ; } else if (cc->is_khugepaged &&@@ -1465,10 +1595,12 @@ static int collapse_scan_pmd(struct mm_struct *mm, out_unmap:pte_unmap_unlock (pte, ptl);if (result ==SCAN_SUCCEED ) {mm/khugepaged.c static int collapse_scan_pmd(struct mm_struct *mm, · line 1603
Quoted 10 lines
mm/khugepaged.c +6
-4- result =collapse_huge_page (mm, address, referenced, - unmapped, cc); -/* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + result =collapse_scan_bitmap (mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); +if (result >0 ) + result =SCAN_SUCCEED ; +else + result =SCAN_FAIL ;We're reusing result as both an enum value and as a storage for unmber colapsed PTE entries?
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Can we just use a new local variable? Thanks
Again you've skipped a ton of review comments here. You need to respond to everything.
Hmm we could, although it would serve no purpose in my eyes... nr_collapsed = collapse_scan_bitmap() if(nr_collapsed > 0)
result = SCAN_SUCCEED.
To me that is basically the same thing with extra steps. But maybe its better for a code readiblity standpoint.
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
mm/khugepaged.c +0
-1} out: trace_mm_khugepaged_scan_pmd(mm, folio, writable, referenced, --2.50.1
Quoted 3 lines
Quoted 2 lines
I will review the bitmap/chunk stuff in more detail once the algorithm is commented.
ok thanks for the review.
No problem! :)
Re: [PATCH v10 06/13] khugepaged: add mTHP support
Quoted 7 lines · Nico Pache
Quoted 3 lines · Lorenzo Stoakes
You've not responded to a ton of comments, I'm guesing I should assume in those cases you're acking the comments implicitly? :) Do let me know.
Yes! If they are obvious nits, then implicit ack.
OK that's fine, thanks!
Quoted 5 lines
Sorry I wrote my response over a couple of days while making changes based on the review and trying to figure out a solution to the "creep" issue, and forgot to thoroughly review all the comments before sending this out. I'll make sure to go through it again.
Thanks!
Quoted 51 lines
Quoted 51 lines · Nico Pache
Quoted 49 lines
Quoted 47 lines · Lorenzo Stoakes
Quoted 28 lines · Nico Pache
Introduce the ability for khugepaged to collapse to different mTHP sizes. While scanning PMD ranges for potential collapse candidates, keep track of pages in
KHUGEPAGED_MIN_MTHP_ORDERchunks via a bitmap. Each bit represents a utilized region of orderKHUGEPAGED_MIN_MTHP_ORDERptes. If mTHPs are enabled we remove the restriction ofmax_ptes_noneduring the scan phase so we don't bailout early and miss potential mTHP candidates.A new function
collapse_scan_bitmapis used to perform binary recursion on the bitmap and determine the best eligible order for the collapse. A stack struct is used instead of traditional recursion.max_ptes_nonewill be scaled by the attempted collapse order to determine how "full" an order must be before being considered for collapse.Once we determine what mTHP sizes fits best in that PMD range a collapse is attempted. A minimum collapse order of 2 is used as this is the lowest order supported by anon memory.
For orders configured with "always", we perform greedy collapsing to that order without considering bit density.
If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. This is because adding new entries can lead to new none pages, and these may lead to constant promotion into a higher order (m)THP. A similar issue can occur with "
max_ptes_none> HPAGE_PMD_NR/2" due to the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.Quoted 7 lines
For non-PMD collapse we must leave the anon VMA write locked until after we collapse the mTHP-- in the PMD case all the pages are isolated, but in the non-PMD case this is not true, and we must keep the lock to prevent changes to the VMA from occurring.
Currently
madv_collapseis not supported and will only attempt PMD collapse.Yes I think this has to remain the case unfortunately as we override sysfs-specified orders for
MADV_COLLAPSEand there's no sensible way to determine what order we ought to be using.Quoted 2 lines
Signed-off-by Nico Pache <[email protected]>You've gone from small incremental changes to a huge one here... for the sake of reviewer sanity at least, any chance of breaking this up?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I had this as two patches (one for the bitmap and one for implementing it), but I was asked to squash them :/
Yeah it makes sense to combine those two, but maybe there's a better way of splitting things out.
I'll give it a shot but most of this patch belongs together in my eyes.
OK, let's see what's most sensible.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
--- include/linux/khugepaged.h | 4 + mm/khugepaged.c | 236 +++++++++++++++++++++++++++++-------- 2 files changed, 188 insertions(+), 52 deletions(-)include/linux/khugepaged.h +0
-0include/linux/khugepaged.h line 4
Quoted 4 lines
include/linux/khugepaged.h +1
-0@@ -1,6 +1,10 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _LINUX_KHUGEPAGED_H#define _LINUX_KHUGEPAGED_H +#define KHUGEPAGED_MIN_MTHP_ORDER 2I guess this makes sense as by definition 2 pages is least it could possibly be.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Order, so 4 pages, 16kB mTHP
Right, misread that! :) sorry. I guess then in fact there is no such thing as an order-1 mTHP? I did wonder how useful that'd be so makes more sense
Anon memory does not support order 1 THPs, im not sure if there are any plans do so either.
Yeah right, this was me being dumb!
Quoted 26 lines
Quoted 12 lines
Quoted 6 lines
Quoted 5 lines
include/linux/khugepaged.h line 5
Quoted 3 lines
include/linux/khugepaged.h +1
-0+#define KHUGEPAGED_MIN_MTHP_NR (1 << KHUGEPAGED_MIN_MTHP_ORDER)Surely
KHUGEPAGED_MIN_NR_MTHP_PTESwould be more meaningful?Sure!
Thanks!
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
include/linux/khugepaged.h +1
-0+#define MAX_MTHP_BITMAP_SIZE (1 << (ilog2(MAX_PTRS_PER_PTE) - KHUGEPAGED_MIN_MTHP_ORDER))Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This is confusing - size of what?
We need it like this due to ppc64 (and maybe others?), it used to be based on
PMD_ORDER, but some arches fail to compile due to the PMD size only being known at boot time.That really sucks. Do please put this context in a comment though (see below for more discussion on this).
Ok, will do. although with my most recent changes we might just be able to get rid of this all together.
Quoted 5 lines
Quoted 4 lines
This compiles to 9 on arches that have 512 ptes.
so 1 << (9 - 2) == 128
Quoted 6 lines
Quoted 2 lines
What I'm saying is - what is this expressed in? There's no information on that here.
In my next implementation its probably going to be gone, but what it indicates is the bitmap size (128), where each bit equals a range of 4 pages. 128 *4 = 512.
OK thanks. Let's see what the new implementation is like.
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
include/linux/khugepaged.h line 7
Quoted 2 lines
include/linux/khugepaged.h +1
-0+DECLARE_BITMAP (mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Same comment as above obviously. But also this is kind of horrible, why are we putting a copy of this entire bitmap on the stack every time we declare a cc?
The temp one is used as a scratch pad, Baolin also finds this useful in his file mTHP collapse useful for another use as well.
Saying 'scratch pad' is really just saying 'temporary' using different words :) this isn't hugely helpful.
Quoted 8 lines
When doing the "recursion", there is no easy way to read the bitmap between bits N0-N1, so we must manipulate it to extract that data; however, we still want to preserve the original state of the bitmap. The
temp_bitmapis used to do these bit manipulations while not destroying the original state. I will see if there's a better way to read arbitrary ranges of the bitmap so we dont need to have two copies, but IIRC there is not-- I might have to expand the bitmap API to include this.
It at the very least needs a better name.
Quoted 7 lines
Quoted 5 lines
If we _need_ this we should give this a better name, and I also don't know why we need this on the stack for all
collapse_controlusers. Can't you just have your 'scratch pad' local to the function that needs it or passed as a pointer?If I cant find a solution to the problem above I will look into moving it to the local stack.
Yeah, maybe not necessary after I realised that in fact we don't put cc on the stack... :)
It does definitely need a better name, however...
Quoted 7 lines
Quoted 3 lines
I'm sure it'd be useful to somebody to stick various temporary things in
vm_area_structfor instance, but it'd be hugely egregious to do so...The reason I didnt see this as a huge issue, is there is only one global CC that is reused for all khugepaged runs. It's a one time allocation and does not scale.
Yeah, see below, I realised cc isn't on the stack so. Fine :)
Quoted 56 lines
Quoted 56 lines
Quoted 4 lines
In general khugepaged always uses the same CC, so it doesn't not having to constantly allocate this.
You're putting a LOT of data on the stack, and kernel stacks are very delicate, I'm quite concerned about this.
BEFORE:
struct collapse_control { bool is_khugepaged; /* 0 1 */ /* XXX 3 bytes hole, try to pack */ u32 node_load[64]; /* 4 256 */ /* XXX 4 bytes hole, try to pack */ /* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */ nodemask_t alloc_nmask; /* 264 8 */ /* size: 272, cachelines: 5, members: 3 */ /* sum members: 265, holes: 2, sum holes: 7 */ /* last cacheline: 16 bytes */ };272 bytes.
AFTER:
struct collapse_control { bool is_khugepaged; /* 0 1 */ /* XXX 3 bytes hole, try to pack */ u32 node_load[64]; /* 4 256 */ /* XXX 4 bytes hole, try to pack */ /* --- cacheline 4 boundary (256 bytes) was 8 bytes ago --- */ nodemask_t alloc_nmask; /* 264 8 */ long unsigned int mthp_bitmap[2]; /* 272 16 */ long unsigned int mthp_bitmap_temp[2]; /* 288 16 */ struct scan_bit_state mthp_bitmap_stack[128]; /* 304 512 */ /* size: 816, cachelines: 13, members: 6 */ /* sum members: 809, holes: 2, sum holes: 7 */ /* last cacheline: 48 bytes */ };816 bytes.
So you're roughly tripling the size of this struct and making this a thing in all callstacks that reference
collapse_control.
Quoted 8 lines
Quoted 8 lines
Kernel stack is a _super_ finite resource.
I mean I can't really review your stack implementation thing until you've added a comment to help me understand what you are doing there (sorry but it's just too fiddly for me to first principles it), but I wonder if we truly need to be doing this?
I wonder if we can just put this into somewhere allocated...
OK hang on... HANG ON :) have I been dumb here?
Sorry... this isn't on the stack is it?
It just looks like, from the function signatures _that it should be_... or at least as per how we usually implement xxx_control structs.
OK that's really good. So you can ignore this bit... Probably we're fine with the size being larger.
I see we kmalloc() a cc in madvise_collapse() (fine) or have it static in khugepaged_collapse_control.
So ok. Never mind :)
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 25 lines
Quoted 8 lines
include/linux/khugepaged.h +2
-1static int collapse_huge_page (struct mm_struct *mm,unsigned long address,int referenced,int unmapped, -struct collapse_control *cc) +struct collapse_control *cc,bool *mmap_locked, +unsigned int order,unsigned long offset) {LIST_HEAD (compound_pagelist);include/linux/khugepaged.h
Quoted 10 lines
include/linux/khugepaged.h +2
-1pmd_t *pmd, _pmd; -pte_t *pte; +pte_t *pte =NULL , mthp_pte;pgtable_t pgtable;struct folio *folio;spinlock_t *pmd_ptl, *pte_ptl;int result =SCAN_FAIL ;struct vm_area_struct *vma;struct mmu_notifier_range range; +unsigned long _address = address + offset *PAGE_SIZE ;This name is really horrible. please name it sensibly.
It feels like address ought to be consistently the base of the THP or mTHP we wish to collapse, and if we need something PMD aligned for some reason we should rename _that_ to e.g.
pmd_address.Quoted 8 lines
Orrr it could be
mthp_address...Perhaps we could just figure that out here and pass only the address... aligning to PMD boundary shouldn't be hard/costly.
But it may indicate we need further refactorisation so we don't need to paper over cracks + pass around a PMD address to do things when that may not be where the (m)THP range begins.
Ok i'll rename them, but we still need to know the PMD address as we rely on it for a few key operations.
Can we leave_addressand rename address topmd_address?
Quoted 6 lines
Quoted 4 lines
No, we absolutely cannot leave
_addressas '_address', that's terrible and I'm just not going to live with that.I know it's trivial, but it's just such a horrid naming convention.
Ok Ill rename it to something else!
Thanks!
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 22 lines
Quoted 22 lines
VM_BUG_ON(address & ~HPAGE_PMD_MASK);include/linux/khugepaged.h +6
-2@@ -1155,16 +1170,20 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* The allocation can take potentially a long time if it involves * sync compaction, and we do not need to hold the mmap_lock during * that. We will recheck the vma after taking it again in write mode. +* If collapsing mTHPs we may have already released the read_lock. */ -mmap_read_unlock (mm); +if (*mmap_locked) { +mmap_read_unlock (mm); + *mmap_locked = false; + } - result =alloc_charge_folio (&folio, mm, cc,HPAGE_PMD_ORDER ); + result =alloc_charge_folio (&folio, mm, cc, order);if (result !=SCAN_SUCCEED ) goto out_nolock;Quoted 9 lines
include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1186
Quoted 5 lines
include/linux/khugepaged.h +2
-2mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, address, true, &vma, cc, -BIT (HPAGE_PMD_ORDER )); + *mmap_locked = true; + result =hugepage_vma_revalidate (mm, address, true, &vma, cc,BIT (order));I mean this is kind of going back to previous commits, but it's really ugly to pass a BIT(xxx) here, is that really necessary? Can't we just pass in the order?
Yes and no... currently we only ever pass the bit of the current order so we could get away with it, but to generalize it we want the ability to pass a bitmap of the available orders. Like in the case of future
madvise_collapsesupport, we would need to pass a bitmap of possible orders.
Quoted 5 lines
Quoted 1 line
Can we just change that when we need it?
Sure, it was more so that I generalized it for
madv_collapseand khugepaged, but then we droppedmadv_collapsesupport, and I never undid the change.
Thanks, I think best to undo it until we need it.
Quoted 8 lines
Quoted 6 lines
'Future proofing' for an possible future implementation detail is just not how we should implement things.
Right now we don't do that, so let's just pass the order. If in future we want to change it we can.
ack
Thanks!
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
include/linux/khugepaged.h +0
-3if (pte) { - result =__collapse_huge_page_isolate (vma, address, pte, cc, - &compound_pagelist, -HPAGE_PMD_ORDER );include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
Quoted 10 lines
include/linux/khugepaged.h +2
-2+ result =__collapse_huge_page_isolate (vma, _address, pte, cc, + &compound_pagelist, order);spin_unlock (pte_ptl); }else { result =SCAN_PMD_NULL ; }if (unlikely (result !=SCAN_SUCCEED )) { -if (pte) -pte_unmap (pte);Why are we removing this?
Quoted 7 lines
Quoted 2 lines
You're missing some of the comments, I'm guesing for most of the smaller stuff you're just implicitly acking them but this one was a question :)
Ah, for some of your reviews you comment, then later when you see another piece you say something like "ah ok this is why", for a lot of these cases I might have skipped replying because it seems that you figured it out.
Sure it's fine, I'm just sort of trying to draw your attention to stuff that needs some discussion :)
There's a lot here so it's easy to lose track from both sides (I may very well be repeating myself, apologies if so!)
Quoted 4 lines
For this one we moved the
pte_unmapfurther down, because if not, we could be prematurely unmapping a PTE, which is later referenced during the mTHP collapse, which was not the case when we only did PMD collapse.
Ack!
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 7 lines
spin_lock(pmd_ptl); BUG_ON(!pmd_none(*pmd)); /*include/linux/khugepaged.h +0
-0@@ -1255,17 +1272,17 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, }include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1279
Quoted 10 lines
include/linux/khugepaged.h +4
-3/* -* All pages are isolated and locked so anon_vma rmap -* can't run anymore. +* For PMD collapse all pages are isolated and locked so anon_vma +* rmap can't run anymore */ -anon_vma_unlock_write (vma->anon_vma); +if (order ==HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Hmm this is introducing a horrible new way for things to go wrong. And there's now a whole host of terrible error paths that can go wrong very easily around rmap locks and yeah, no way we cannot do it this way.
rmap locks are VERY sensitive and the ordering of the locking matters a great deal (see top of
mm/rmap.c). So we have to be SO careful here.I suggest you simply have a boolean '
anon_vma_locked' or something like this, and get rid of these horrible additional code paths and the second order ==HPAGE_PMD_ORDERcheck.
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
We'll track whether or not the lock is held and thereby needs releasing that way instead.
You've not responded to this suggestion re: refactoring here.
I think if anything this refactoring adds complexity... we simply
I don't really think it does.
We have an open-coded condition that you just have to happen to know when reading the code determines whether a lock is held or released.
Or, we could just have a boolean that explicitly encodes this.
Quoted 7 lines
postpone the unlock we normally do, to later if its mTHP collapse, because we havent fully isolated all the pages (like we do in the PMD case).
We arent doing anything extremely different other than hold the already held lock for a little longer. We would still need these order ==HPAGE_PMD_ORDERchecks, but just be flipping a flag that says its locked, essentially adding more complexity for the same outcome.
Yeah, I don't agree sorry.
I really do not like this, we should have the boolean.
Quoted 47 lines
Quoted 4 lines
I'm really not a fan of us arbitrarily messing with rmap locks like this, and we should very carefully keep track of whether we have/have not released them.
Its important to note, the code was already doing this, as I stated above, we simply hold it for longer because we haven't isolated all the pages.
Quoted 8 lines
Again rmap locking is a dangerous area, I've got personal experience of this (see top of
mm/rmap.cfor an indication of complexity here as well as
https://kernel.org/doc/html/latest/mm/process_addrs.html).Quoted 3 lines
Quoted 3 lines
Also, and very importantly - are you 100% sure you can't possibly have a deadlock or issue beyond this point if you don't release the rmap lock?
Quoted 7 lines
Quoted 4 lines
I double checked, this was added as a fix to an issue Hugh reported.
The gap between these callers is rather small, and I see no way that it could skip the lock/unlock cycle.
We're going to need more than this to be confident, you need to clearly justify why we won't encounter issues this way.
// WE ARE HOLDING THE LOCK ALREADY
if (order ==HPAGE_PMD_ORDER) //ONLY RELEASE IF ITS PMD
anon_vma_unlock_write(vma->anon_vma);result = __collapse_huge_page_copy(pte, folio, pmd,
_pmd,
vma,_address,pte_ptl,
order,&compound_pagelist);
if (unlikely(result != SCAN_SUCCEED))
gotoout_unlock_anon_vma; //LETS RELEASE THE LOCK IF WE FAILif (order == HPAGE_PMD_ORDER) { ... // NO EXIT PATHS } else { /* mTHP collapse */ ... // NO EXIT PATHS } folio = NULL; result = SCAN_SUCCEED; out_unlock_anon_vma: if (order != HPAGE_PMD_ORDER) anon_vma_unlock_write(vma->anon_vma); //RELEASE IF IT WASN'T
Quoted 1 line
RELEASED EARLIER
Thanks for this, but I think this only makes my point re: a booelan: now you have to just realise at each point in the code that the open-coded condition is what determines whether the lock is held or not.
A boolean that does this is a _lot_ clearer.
Quoted 4 lines
Not sure if this is the verification you are looking for, but I really dont see how this could go wrong, or how this is any different than what the khugepaged code is already doing in a number of places.
Existing khugepaged code is not a good guideline for future development :) the code is not always entirely delightful there or in huge_memory.c either...
Quoted 38 lines
Quoted 29 lines
Quoted 11 lines
Quoted 8 lines
include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1282
Quoted 5 lines
include/linux/khugepaged.h +3
-0+update_mmu_cache_range (NULL , vma, _address, pte, (1 << order)); + +smp_wmb ();/* make pte visible before pmd */ Can you give some detail as to why this will work here and why it is necessary?
Other parts of the kernel do it when setting ptes before updating the PMD. I'm not sure if it's necessary, but better safe than sorry.
Unfortunately this is a _totally_ unacceptable justification. We can't put in barriers based on 'better safe than sorry'. You need to analysis this and determine whether or not it's necessary.
So the comment in
pmd_install()seems to give an indication:if (likely(pmd_none(*pmd))) { /* Has another populated it ? */
mm_inc_nr_ptes(mm);
/*
* Ensure all pte setup (eg. pte page lock and page clearing) are
* visible before the pte is made visible to other CPUs by being
* put into page tables.
*
* The other side of the story is the pointer chasing in the page
* table walking code (when walking the page table without locking;
* ie. most of the time). Fortunately, these data accesses consist
* of a chain of data-dependent loads, meaning most CPUs (alphaQuoted 8 lines
* being the notable exception) will already guarantee loads are
* seen in-order. See the alpha page table accessors for the
*smp_rmb()barriers in page table walking code. */
smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */pmd_populate(mm, pmd, *pte);
*pte = NULL;
}
Quoted 2 lines
So this seems to indicate we do need it for some rare edge cases/arches, and that's why it exists in other callers too.
Yup, so let's be explicit and say something like 'see pmd_install()' or something.
Open-coded 'just have to know' stuff is just something we want to avoid in general.
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1283
Quoted 3 lines
include/linux/khugepaged.h +1
-0+pmd_populate (mm, pmd,pmd_pgtable (_pmd));If we're updating PTE entriess why do we need to assign the PMD entry?
We removed the PMD entry for
GUP_fastreasons, then we reinstall the PMD entry after the mTHP is in place. Same as for PMD collapse.OK a comment to this effect would be useful.
Quoted 8 lines
Ok ill add a small comment. feel free to take a look at the failure handling around
__collapse_huge_page_isolateto see another example of this. We remove the pmd (pmdp_collapse_flush), then if the isolation fails, we repopulate the original PMD. In the mTHP case, because we are not installing a new PMD, we still remove the PMD (for gup-fast reasons), modify the PTE entries within that PMD, then reinstall the original PMD. This prevent GUP-fast from accessing these PTEs/PMD mid change.
Thanks. We need this kind of context in the code.
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 19 lines
include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · line 1285
Quoted 4 lines
include/linux/khugepaged.h +2
-0+spin_unlock (pmd_ptl); + }This deeply, badly needs to be refactored into something that both shares code and separates out these two operations.
This function is disgustingly long as it is, and that's not your fault, but let's try to make things better as we go.
include/linux/khugepaged.h static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
Quoted 7 lines
include/linux/khugepaged.h +3
-0folio =NULL ; result =SCAN_SUCCEED ; +out_unlock_anon_vma: +if (order !=HPAGE_PMD_ORDER ) +anon_vma_unlock_write (vma->anon_vma);Quoted 8 lines
Obviously again as above, we need to simplify this and get rid of this whole bit.
include/linux/khugepaged.h
Quoted 3 lines
out_up_write:include/linux/khugepaged.h +2
-0+if (pte) +pte_unmap (pte);OK I guess you moved this from above down here? Is this a valid place to do this?
Yes if not we were potentially unmapping a pte early.
Quoted 24 lines
Quoted 4 lines
mmap_write_unlock(mm);
out_nolock:
+*mmap_locked= false;This is kind of horrible, we now have pretty mad logic around who sets
mmap_lockedand where.Can we just do this at the call sites so we avoid that?
I mean anything we do with this is hideous, but that'd be less confusing It hink.
Quoted 7 lines
if (folio) folio_put(folio); trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result); return result;}
+/* Recursive function to consume the bitmap */
Err... please don't? Kernel stack is a seriously finite resource, we do not want recursion at all.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
But I'm not actually seeing any recursion here? Am I missing something?
Yup this was before I realised it was an iterative implementation :)
Though we are putting load on the stack anyway...
Ill see if I can shrink this load!
Thanks!
Things are less problematic with cc not on the stack (my mistake that, sorry), but obviously anything that could help reduce the stack usage here is good anyway.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
include/linux/khugepaged.h +3
-0+static int collapse_scan_bitmap (struct mm_struct *mm,unsigned long address, +int referenced,int unmapped,struct collapse_control *cc, +bool *mmap_locked,unsigned long enabled_orders)
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
This is a complicated and confusing function, it requires a comment describing how it works.
Ok will do!
This is _VERY_ key - I need this I think to be able to sanely review this code.
ack, im also simplifying it a bit in the next iteration.
Thanks, much appreciated :)
I do wonder if this could be separate, but perhaps not.
Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 24 lines
Quoted 24 lines
static int collapse_scan_pmd(struct
mm_struct*mm,
structvm_area_struct*vma, unsigned long address, bool*mmap_locked,include/linux/khugepaged.h +6
-1@@ -1307,31 +1406,60 @@ static int collapse_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; +int i;int result =SCAN_FAIL , referenced =0 ;int none_or_zero =0 , shared =0 ;struct page *page =NULL ;struct folio *folio =NULL ;unsigned long _address; +unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ; +bool is_pmd_only;bool writable = false; - +int chunk_none_count =0 ; +int scaled_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER -KHUGEPAGED_MIN_MTHP_ORDER ); +unsigned long tva_flags = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ;VM_BUG_ON (address & ~HPAGE_PMD_MASK );Quoted 8 lines
include/linux/khugepaged.h static int collapse_scan_pmd(struct mm_struct *mm, · line 1430
Quoted 6 lines
include/linux/khugepaged.h +2
-0result =find_pmd_or_thp_or_none (mm, address, &pmd);if (result !=SCAN_SUCCEED ) goto out; +bitmap_zero (cc->mthp_bitmap,MAX_MTHP_BITMAP_SIZE ); +bitmap_zero (cc->mthp_bitmap_temp,MAX_MTHP_BITMAP_SIZE );Having this 'temp' thing on the stack for everyone is just horrid.
As I mention above this serves a very good purpose, and is also expanded in another series by Baolin to serve another similar purpose too.
Quoted 6 lines
Quoted 4 lines
Yeah, I'm not hugely convinced :) responded there.
I can probably give more useful feedback on this once the big comment is added!
Ok! hopefully I find a solution so I can remove the temp one.
Thanks!
Again sorry to separate this out (+ force more review back-and-forth), it's just the only way I can sensibly review this I think!
Quoted 50 lines
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 32 lines
include/linux/khugepaged.h +7
-2+ + _pte = pte + i; + _address = address + i *PAGE_SIZE ;pte_t pteval =ptep_get (_pte);if (is_swap_pte (pteval)) { ++unmapped;@@ -1354,10 +1482,11 @@ static int collapse_scan_pmd(struct mm_struct *mm, } }if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) { + ++chunk_none_count; ++none_or_zero;if (!userfaultfd_armed (vma) && - (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + (!cc->is_khugepaged || !is_pmd_only || + none_or_zero <= khugepaged_max_ptes_none)) {continue ; }else { result =SCAN_EXCEED_NONE_PTE ;@@ -1453,6 +1582,7 @@ static int collapse_scan_pmd(struct mm_struct *mm, address))) referenced++; } +if (!writable) { result =SCAN_PAGE_RO ; } else if (cc->is_khugepaged &&@@ -1465,10 +1595,12 @@ static int collapse_scan_pmd(struct mm_struct *mm, out_unmap:pte_unmap_unlock (pte, ptl);if (result ==SCAN_SUCCEED ) {include/linux/khugepaged.h static int collapse_scan_pmd(struct mm_struct *mm, · line 1603
Quoted 10 lines
include/linux/khugepaged.h +6
-4- result =collapse_huge_page (mm, address, referenced, - unmapped, cc); -/* collapse_huge_page will return with the mmap_lock released */ - *mmap_locked = false; + result =collapse_scan_bitmap (mm, address, referenced, unmapped, cc, + mmap_locked, enabled_orders); +if (result >0 ) + result =SCAN_SUCCEED ; +else + result =SCAN_FAIL ;We're reusing result as both an enum value and as a storage for unmber colapsed PTE entries?
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Can we just use a new local variable? Thanks
Again you've skipped a ton of review comments here. You need to respond to everything.
Quoted 7 lines
Hmm we could, although it would serve no purpose in my eyes...
nr_collapsed=collapse_scan_bitmap()if(nr_collapsed > 0)
result =SCAN_SUCCEED.To me that is basically the same thing with extra steps. But maybe its better for a code readiblity standpoint.
Code readability is a very good purpose to serve, unless we discover perf issues that suggest we should behave otherwise, this should be our _primary_ aim in my view.
Even more so the case in THP code which is, often, well let's just say 'not beautiful' :)
Thanks for responding so thoroughly, we'll get this over the line I promise!
Cheers, Lorenzo
PATCH v10 7/13 khugepaged: skip collapsing mTHP to smaller orders
In reply to Nico Pachekhugepaged may try to collapse a mTHP to a smaller mTHP, resulting in some pages being unmapped. Skip these cases until we have a way to check if its ok to collapse to a smaller mTHP size (like in the case of a partially mapped folio).
This patch is inspired by Dev Jain's work on khugepaged mTHP support [1].
[1] https://lore.kernel.org/lkml/[email protected]/
--- mm/khugepaged.c | 9 +++++++++ 1 file changed, 9 insertions(+)
mm/khugepaged.c +9 -0
@@ -611,6 +611,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); +/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/ +if (order !=HPAGE_PMD_ORDER &&folio_order (folio) >= order) { + result =SCAN_PTE_MAPPED_HUGEPAGE ; + goto out; + } +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared;
Signature · 1 line
2.50.1
Re: [PATCH v10 07/13] khugepaged: skip collapsing mTHP to smaller orders
Quoted 7 lines · Nico Pache
khugepaged may try to collapse a mTHP to a smaller mTHP, resulting in some pages being unmapped. Skip these cases until we have a way to check if its ok to collapse to a smaller mTHP size (like in the case of a partially mapped folio).
This patch is inspired by Dev Jain's work on khugepaged mTHP support [1].
Quoted 7 lines
[1] https://lore.kernel.org/lkml/[email protected]/
Reviewed-by Baolin Wang <[email protected]>Co-developed-by Dev Jain <[email protected]>Signed-off-by Dev Jain <[email protected]>Signed-off-by Nico Pache <[email protected]>
Other than comment below, LGTM so:
Quoted 9 lines
--- mm/khugepaged.c | 9 +++++++++ 1 file changed, 9 insertions(+)mm/khugepaged.c +0
-0
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 617
Quoted 7 lines
mm/khugepaged.c +4
-0@@ -611,6 +611,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio); +/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/
Not a fan of adding todo's in code, they have a habit of being left forever. I'd maybe put a more written out comment something similar to the commit message.
Quoted 11 lines
mm/khugepaged.c +5
-1+if (order !=HPAGE_PMD_ORDER &&folio_order (folio) >= order) { + result =SCAN_PTE_MAPPED_HUGEPAGE ; + goto out; + } +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; --2.50.1
Dev Jain
reviewerRe: [PATCH v10 07/13] khugepaged: skip collapsing mTHP to smaller orders
Quoted 14 lines · Lorenzo Stoakes
Quoted 14 lines
--- mm/khugepaged.c | 9 +++++++++ 1 file changed, 9 insertions(+)mm/khugepaged.c +0
-0@@ -611,6 +611,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio);
Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 617
Quoted 4 lines
mm/khugepaged.c +4
-0+/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/ Not a fan of adding todo's in code, they have a habit of being left forever. I'd maybe put a more written out comment something similar to the commit message.
I had suggested to add in https://lore.kernel.org/all/[email protected]/
from the get go, but then we decided to leave it for later. So rest assured this TODO won't be left forever : )
Quoted 12 lines
Quoted 12 lines
mm/khugepaged.c +5
-1+if (order !=HPAGE_PMD_ORDER &&folio_order (folio) >= order) { + result =SCAN_PTE_MAPPED_HUGEPAGE ; + goto out; + } +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; --2.50.1
Re: [PATCH v10 07/13] khugepaged: skip collapsing mTHP to smaller orders
Quoted 23 lines · Dev Jain
Quoted 14 lines · Lorenzo Stoakes
Quoted 14 lines
--- mm/khugepaged.c | 9 +++++++++ 1 file changed, 9 insertions(+)mm/khugepaged.c +0
-0@@ -611,6 +611,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio =page_folio (page);VM_BUG_ON_FOLIO (!folio_test_anon (folio), folio);Quoted 6 lines
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 617
Quoted 4 lines
mm/khugepaged.c +4
-0+/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/ Not a fan of adding todo's in code, they have a habit of being left forever. I'd maybe put a more written out comment something similar to the commit message.
Quoted 3 lines
I had suggested to add in https://lore.kernel.org/all/[email protected]/
from the get go, but then we decided to leave it for later. So rest assured this TODO won't be left forever : )
:)
I think it's better, despite Steven's sage words :P, to put something a little more meaningful here like 'currently we don't blah blah because blah blah' without the TODO ;)
But obviously this isn't a substantially pressing issue... :>)
Cheers, Lorenzo
Steven Rostedt
maintainerRe: [PATCH v10 07/13] khugepaged: skip collapsing mTHP to smaller orders
In reply to Lorenzo StoakesQuoted 6 lines · On Thu, 21 Aug 2025 13:05:42 +0100 Lorenzo Stoakes…
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 617
Quoted 4 lines
mm/khugepaged.c +4
-0+/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/ Not a fan of adding todo's in code, they have a habit of being left forever.
It's a way to make the developer more depressed by reminding them that they will never be able to complete their TODO list :-p
Personally, I enjoy the torture of these comments in the code ;-)
-- Steve
Re: [PATCH v10 07/13] khugepaged: skip collapsing mTHP to smaller orders
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Thu, 21 Aug 2025 13:05:42 +0100 Lorenzo Stoakes…
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 617
Quoted 4 lines
mm/khugepaged.c +4
-0+/* +* TODO: In some cases of partially-mapped folios, we'd actually +* want to collapse. +*/ Not a fan of adding todo's in code, they have a habit of being left forever.
Quoted 4 lines
It's a way to make the developer more depressed by reminding them that they will never be able to complete their TODO list :-p
Personally, I enjoy the torture of these comments in the code ;-)
Well a person must at least _somewhat_ enjoy torture if they decide to do kernel development (let alone maintainership :P) so this very much figures ;)
PATCH v10 8/13 khugepaged: avoid unnecessary mTHP collapse attempts
In reply to Nico PacheThere are cases where, if an attempted collapse fails, all subsequent orders are guaranteed to also fail. Avoid these collapse attempts by bailing out early.
--- mm/khugepaged.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
mm/khugepaged.c +30 -1
@@ -1389,10 +1389,39 @@ static int collapse_scan_bitmap(struct mm_struct *mm, unsigned long address, ret =collapse_huge_page (mm, address, referenced, unmapped, cc, mmap_locked, order, offset *KHUGEPAGED_MIN_MTHP_NR ); -if (ret ==SCAN_SUCCEED ) { + +/* +* Analyze failure reason to determine next action: +* - goto next_order: try smaller orders in same region +* - continue: try other regions at same order +* - break: stop all attempts (system-wide failure) +*/ +switch (ret) { +/* Cases were we should continue to the next region */ +case SCAN_SUCCEED : collapsed += (1 << order); +case SCAN_PAGE_RO : +case SCAN_PTE_MAPPED_HUGEPAGE :continue ; +/* Cases were lower orders might still succeed */ +case SCAN_LACK_REFERENCED_PAGE : +case SCAN_EXCEED_NONE_PTE : +case SCAN_EXCEED_SWAP_PTE : +case SCAN_EXCEED_SHARED_PTE : +case SCAN_PAGE_LOCK : +case SCAN_PAGE_COUNT : +case SCAN_PAGE_LRU : +case SCAN_PAGE_NULL : +case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_NON_PRESENT : +case SCAN_PTE_UFFD_WP : +case SCAN_ALLOC_HUGE_PAGE_FAIL : + goto next_order; +/* All other cases should stop collapse attempts */ +default : +break ; } +break ; } next_order:
Signature · 1 line
2.50.1
kernel test robot
build botRe: [PATCH v10 08/13] khugepaged: avoid unnecessary mTHP collapse attempts
kernel test robot noticed the following build warnings:
Hi Nico,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[cannot apply to lwn/docs-next linus/master v6.17-rc2 next-20250819] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nico-Pache/khugepaged-rename-hpage_collapse_-to-collapse_/20250819-222051 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250819134205.622806-9-npache%40redhat.com patch subject: [PATCH v10 08/13] khugepaged: avoid unnecessary mTHP collapse attempts config: i386-randconfig-003-20250820 (https://download.01.org/0day-ci/archive/20250820/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250820/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> mm/khugepaged.c:1403:4: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
1403 | case SCAN_PAGE_RO:
| ^
mm/khugepaged.c:1403:4: note: insert '__attribute__((fallthrough));' to silence this warning
1403 | case SCAN_PAGE_RO:
| ^
| __attribute__((fallthrough));
mm/khugepaged.c:1403:4: note: insert 'break;' to avoid fall-through
1403 | case SCAN_PAGE_RO:
| ^
| break; 1 warning generated.
vim +1403 mm/khugepaged.c
1350
1351 /* Recursive function to consume the bitmap */
1352 static int collapse_scan_bitmap(struct mm_struct *mm, unsigned long address,
1353 int referenced, int unmapped, struct collapse_control *cc,
1354 bool *mmap_locked, unsigned long enabled_orders)
1355 {
1356 u8 order, next_order;
1357 u16 offset, mid_offset;
1358 int num_chunks;
1359 int bits_set, threshold_bits;
1360 int top = -1;
1361 int collapsed = 0;
1362 int ret;
1363 struct scan_bit_state state;
1364 bool is_pmd_only = (enabled_orders == (1 << HPAGE_PMD_ORDER));
1365
1366 cc->mthp_bitmap_stack[++top] = (struct scan_bit_state)
1367 { HPAGE_PMD_ORDER - KHUGEPAGED_MIN_MTHP_ORDER, 0 };
1368
1369 while (top >= 0) {
1370 state = cc->mthp_bitmap_stack[top--];
1371 order = state.order + KHUGEPAGED_MIN_MTHP_ORDER;
1372 offset = state.offset;
1373 num_chunks = 1 << (state.order);
1374 /* Skip mTHP orders that are not enabled */
1375 if (!test_bit(order, &enabled_orders))
1376 goto next_order;
1377
1378 /* copy the relavant section to a new bitmap */
1379 bitmap_shift_right(cc->mthp_bitmap_temp, cc->mthp_bitmap, offset,
1380 MTHP_BITMAP_SIZE);
1381
1382 bits_set = bitmap_weight(cc->mthp_bitmap_temp, num_chunks);
1383 threshold_bits = (HPAGE_PMD_NR - khugepaged_max_ptes_none - 1)
1384 >> (HPAGE_PMD_ORDER - state.order);
1385
1386 /* Check if the region is "almost full" based on the threshold */
1387 if (bits_set > threshold_bits || is_pmd_only
1388 || test_bit(order, &huge_anon_orders_always)) {
1389 ret = collapse_huge_page(mm, address, referenced, unmapped,
1390 cc, mmap_locked, order,
1391 offset * KHUGEPAGED_MIN_MTHP_NR);
1392
1393 /*
1394 * Analyze failure reason to determine next action:
1395 * - goto next_order: try smaller orders in same region
1396 * - continue: try other regions at same order
1397 * - break: stop all attempts (system-wide failure)
1398 */
1399 switch (ret) {
1400 /* Cases were we should continue to the next region */
1401 case SCAN_SUCCEED:
1402 collapsed += (1 << order);
> 1403 case SCAN_PAGE_RO:
1404 case SCAN_PTE_MAPPED_HUGEPAGE:
1405 continue;
1406 /* Cases were lower orders might still succeed */
1407 case SCAN_LACK_REFERENCED_PAGE:
1408 case SCAN_EXCEED_NONE_PTE:
1409 case SCAN_EXCEED_SWAP_PTE:
1410 case SCAN_EXCEED_SHARED_PTE:
1411 case SCAN_PAGE_LOCK:
1412 case SCAN_PAGE_COUNT:
1413 case SCAN_PAGE_LRU:
1414 case SCAN_PAGE_NULL:
1415 case SCAN_DEL_PAGE_LRU:
1416 case SCAN_PTE_NON_PRESENT:
1417 case SCAN_PTE_UFFD_WP:
1418 case SCAN_ALLOC_HUGE_PAGE_FAIL:
1419 goto next_order;
1420 /* All other cases should stop collapse attempts */
1421 default:
1422 break;
1423 }
1424 break;
1425 }
1426
1427 next_order:
1428 if (state.order > 0) {
1429 next_order = state.order - 1;
1430 mid_offset = offset + (num_chunks / 2);
1431 cc->mthp_bitmap_stack[++top] = (struct scan_bit_state)
1432 { next_order, mid_offset };
1433 cc->mthp_bitmap_stack[++top] = (struct scan_bit_state)
1434 { next_order, offset };
1435 }
1436 }
1437 return collapsed;
1438 }
1439
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v10 08/13] khugepaged: avoid unnecessary mTHP collapse attempts
In reply to Nico PacheQuoted 19 lines · Nico Pache
There are cases where, if an attempted collapse fails, all subsequent orders are guaranteed to also fail. Avoid these collapse attempts by bailing out early.
Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)mm/khugepaged.c +1
-1@@ -1389,10 +1389,39 @@ static int collapse_scan_bitmap(struct mm_struct *mm, unsigned long address, ret =collapse_huge_page (mm, address, referenced, unmapped, cc, mmap_locked, order, offset *KHUGEPAGED_MIN_MTHP_NR ); -if (ret ==SCAN_SUCCEED ) { +
mm/khugepaged.c static int collapse_scan_bitmap(struct mm_struct *mm, unsigned long address, · line 1402
Quoted 10 lines
mm/khugepaged.c +9
-0+/* +* Analyze failure reason to determine next action: +* - goto next_order: try smaller orders in same region +* - continue: try other regions at same order +* - break: stop all attempts (system-wide failure) +*/ +switch (ret) { +/* Cases were we should continue to the next region */ +case SCAN_SUCCEED : collapsed += (1 << order);
Yeah as bot noticed (and clang locally)
This needs a break or fallthrough.
Quoted 10 lines
mm/khugepaged.c +9
-0+case SCAN_PAGE_RO : +case SCAN_PTE_MAPPED_HUGEPAGE :continue ; +/* Cases were lower orders might still succeed */ +case SCAN_LACK_REFERENCED_PAGE : +case SCAN_EXCEED_NONE_PTE : +case SCAN_EXCEED_SWAP_PTE : +case SCAN_EXCEED_SHARED_PTE : +case SCAN_PAGE_LOCK : +case SCAN_PAGE_COUNT :
mm/khugepaged.c static int collapse_scan_bitmap(struct mm_struct *mm, unsigned long address, · line 1422
Quoted 10 lines
mm/khugepaged.c +10
-0+case SCAN_PAGE_LRU : +case SCAN_PAGE_NULL : +case SCAN_DEL_PAGE_LRU : +case SCAN_PTE_NON_PRESENT : +case SCAN_PTE_UFFD_WP : +case SCAN_ALLOC_HUGE_PAGE_FAIL : + goto next_order; +/* All other cases should stop collapse attempts */ +default : +break ;
Wouldn't it be better to not have this so the compiler asserts that you have all cases listed here?
Quoted 8 lines
mm/khugepaged.c +1
-1} +break ; } next_order: --2.50.1
PATCH v10 9/13 khugepaged: enable collapsing mTHPs even when PMD THPs are disabled
In reply to Nico PacheFrom: Baolin Wang <[email protected]>
We have now allowed mTHP collapse, but thp_vma_allowable_order() still only checks if the PMD-sized mTHP is allowed to collapse. This prevents scanning and collapsing of 64K mTHP when only 64K mTHP is enabled. Thus, we should modify the checks to allow all large orders of anonymous mTHP.
--- mm/khugepaged.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
mm/khugepaged.c +9 -2
@@ -491,7 +491,11 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, {if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) &&hugepage_pmd_enabled ()) { -if (thp_vma_allowable_order (vma, vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) +unsigned long orders =vma_is_anonymous (vma) ? +THP_ORDERS_ALL_ANON :BIT (PMD_ORDER ); + +if (thp_vma_allowable_orders (vma, vm_flags,TVA_KHUGEPAGED , + orders))__khugepaged_enter (vma->vm_mm); } }
@@ -2671,6 +2675,8 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,vma_iter_init (&vmi, mm, khugepaged_scan.address);for_each_vma (vmi, vma) { +unsigned long orders =vma_is_anonymous (vma) ? +THP_ORDERS_ALL_ANON :BIT (PMD_ORDER );unsigned long hstart, hend;cond_resched ();
@@ -2678,7 +2684,8 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { +if (!thp_vma_allowable_orders (vma, vma->vm_flags, +TVA_KHUGEPAGED , orders)) { skip: progress++;continue ;
Signature · 1 line
2.50.1
Re: [PATCH v10 09/13] khugepaged: enable collapsing mTHPs even when PMD THPs are disabled
Quoted 19 lines · Nico Pache
From: Baolin Wang <[email protected]>
We have now allowed mTHP collapse, but
thp_vma_allowable_order()still only checks if the PMD-sized mTHP is allowed to collapse. This prevents scanning and collapsing of 64K mTHP when only 64K mTHP is enabled. Thus, we should modify the checks to allow all large orders of anonymous mTHP.Signed-off-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c void khugepaged_enter_vma(struct vm_area_struct *vma, · line 495
Quoted 6 lines
mm/khugepaged.c +2
-1@@ -491,7 +491,11 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, {if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) &&hugepage_pmd_enabled ()) { -if (thp_vma_allowable_order (vma, vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) +unsigned long orders =vma_is_anonymous (vma) ? +THP_ORDERS_ALL_ANON :BIT (PMD_ORDER );
We need some explanation here please, a comment explaining what's going on here would go a long way.
mm/khugepaged.c static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, · line 2679
Quoted 12 lines
mm/khugepaged.c +5
-0+ +if (thp_vma_allowable_orders (vma, vm_flags,TVA_KHUGEPAGED , + orders))__khugepaged_enter (vma->vm_mm); } }@@ -2671,6 +2675,8 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,vma_iter_init (&vmi, mm, khugepaged_scan.address);for_each_vma (vmi, vma) { +unsigned long orders =vma_is_anonymous (vma) ? +THP_ORDERS_ALL_ANON :BIT (PMD_ORDER );
Can we have this as a separate helper function please? As you're now open-coding this in two places.
In fact, you can put the comment I mention above there and have that document what's happening here.
Quoted 16 lines
mm/khugepaged.c +2
-2unsigned long hstart, hend;cond_resched ();@@ -2678,7 +2684,8 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result, progress++;break ; } -if (!thp_vma_allowable_order (vma, vma->vm_flags,TVA_KHUGEPAGED ,PMD_ORDER )) { +if (!thp_vma_allowable_orders (vma, vma->vm_flags, +TVA_KHUGEPAGED , orders)) { skip: progress++;continue ; --2.50.1
PATCH v10 10/13 khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
In reply to Nico PacheFrom: Baolin Wang <[email protected]>
When only non-PMD-sized mTHP is enabled (such as only 64K mTHP enabled), we should also allow kicking khugepaged to attempt scanning and collapsing 64K mTHP. Modify hugepage_pmd_enabled() to support mTHP collapse, and while we are at it, rename it to make the function name more clear.
--- mm/khugepaged.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
mm/khugepaged.c +10 -10
@@ -430,7 +430,7 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); } -static bool hugepage_pmd_enabled (void ) +static bool hugepage_enabled (void ) {/* * We cover the anon, shmem and the file-backed case here; file-backed
@@ -442,11 +442,11 @@ static bool hugepage_pmd_enabled(void)if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) &&hugepage_global_enabled ())return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_always)) +if (READ_ONCE (huge_anon_orders_always))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_madvise)) +if (READ_ONCE (huge_anon_orders_madvise))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_inherit) && +if (READ_ONCE (huge_anon_orders_inherit) &&hugepage_global_enabled ())return true;if (IS_ENABLED (CONFIG_SHMEM ) &&shmem_hpage_pmd_enabled ())
@@ -490,7 +490,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) {if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) && -hugepage_pmd_enabled ()) { +hugepage_enabled ()) {unsigned long orders =vma_is_anonymous (vma) ?THP_ORDERS_ALL_ANON :BIT (PMD_ORDER );
@@ -2762,7 +2762,7 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,static int khugepaged_has_work (void ) { -return !list_empty (&khugepaged_scan.mm_head) &&hugepage_pmd_enabled (); +return !list_empty (&khugepaged_scan.mm_head) &&hugepage_enabled (); }static int khugepaged_wait_event (void )
@@ -2835,7 +2835,7 @@ static void khugepaged_wait_work(void)return ; } -if (hugepage_pmd_enabled ()) +if (hugepage_enabled ())wait_event_freezable (khugepaged_wait,khugepaged_wait_event ()); }
@@ -2866,7 +2866,7 @@ static void set_recommended_min_free_kbytes(void)int nr_zones =0 ;unsigned long recommended_min; -if (!hugepage_pmd_enabled ()) { +if (!hugepage_enabled ()) {calculate_min_free_kbytes (); goto update_wmarks; }
@@ -2916,7 +2916,7 @@ int start_stop_khugepaged(void)int err =0 ;mutex_lock (&khugepaged_mutex); -if (hugepage_pmd_enabled ()) { +if (hugepage_enabled ()) {if (!khugepaged_thread) khugepaged_thread =kthread_run (khugepaged,NULL ,"khugepaged" );
@@ -2942,7 +2942,7 @@ int start_stop_khugepaged(void)void khugepaged_min_free_kbytes_update (void ) {mutex_lock (&khugepaged_mutex); -if (hugepage_pmd_enabled () && khugepaged_thread) +if (hugepage_enabled () && khugepaged_thread)set_recommended_min_free_kbytes ();mutex_unlock (&khugepaged_mutex); }
Signature · 1 line
2.50.1
Re: [PATCH v10 10/13] khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
Quoted 3 lines · Nico Pache
From: Baolin Wang <[email protected]>
When only non-PMD-sized mTHP is enabled (such as only 64K mTHP enabled),
I don't think this example is very useful, probably just remove it.
Also 'non-PMD-sized mTHP' implies there is such a thing as PMD-sized mTP :)
Quoted 1 line
we should also allow kicking khugepaged to attempt scanning and collapsing
What is kicking? I think this should be rephrased to something like 'we should also allow khugepaged to attempt scanning...'
Quoted 1 line
64K mTHP. Modify
hugepage_pmd_enabled()to support mTHP collapse, and
64K mTHP -> "of mTHP ranges". Put the 'Modify...' bit in a new paragraph to be clear.
Quoted 1 line
while we are at it, rename it to make the function name more clear.
To make this clearer let me suggest:
In order for khugepaged to operate when only mTHP sizes are specified in sysfs, we must modify the predicate function that determines whether it ought to run to do so.
This function is currently called hugepage_pmd_enabled(), this patch renames it to hugepage_enabled() and updates the logic to check to determine whether any valid orders may exist which would justify khugepaged running.
mm/khugepaged.c static bool hugepage_pmd_enabled(void)
Quoted 22 lines
Signed-off-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)mm/khugepaged.c +1
-1@@ -430,7 +430,7 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); } -static bool hugepage_pmd_enabled (void ) +static bool hugepage_enabled (void ) {/* * We cover the anon, shmem and the file-backed case here; file-backed @@ -442,11 +442,11 @@ static bool hugepage_pmd_enabled(void)
The comment above this still references PMD-sized, please make sure to update comments when you change the described behaviour, as it is now incorrect:
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
* Anon pmd-sized hugepages are determined by the pmd-size control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the global shmem_huge is set to SHMEM_HUGE_DENY. */
Please correct this.
mm/khugepaged.c static bool hugepage_pmd_enabled(void) · line 450
Quoted 12 lines
mm/khugepaged.c +3
-3if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) &&hugepage_global_enabled ())return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_always)) +if (READ_ONCE (huge_anon_orders_always))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_madvise)) +if (READ_ONCE (huge_anon_orders_madvise))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_inherit) && +if (READ_ONCE (huge_anon_orders_inherit) &&hugepage_global_enabled ())
I guess READ_ONCE() is probably sufficient here as memory ordering isn't important here, right?
Quoted 59 lines
mm/khugepaged.c +6
-7return true;if (IS_ENABLED (CONFIG_SHMEM ) &&shmem_hpage_pmd_enabled ())@@ -490,7 +490,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) {if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) && -hugepage_pmd_enabled ()) { +hugepage_enabled ()) {unsigned long orders =vma_is_anonymous (vma) ?THP_ORDERS_ALL_ANON :BIT (PMD_ORDER );@@ -2762,7 +2762,7 @@ static unsigned int collapse_scan_mm_slot(unsigned int pages, int *result,static int khugepaged_has_work (void ) { -return !list_empty (&khugepaged_scan.mm_head) &&hugepage_pmd_enabled (); +return !list_empty (&khugepaged_scan.mm_head) &&hugepage_enabled (); }static int khugepaged_wait_event (void )@@ -2835,7 +2835,7 @@ static void khugepaged_wait_work(void)return ; } -if (hugepage_pmd_enabled ()) +if (hugepage_enabled ())wait_event_freezable (khugepaged_wait,khugepaged_wait_event ()); }@@ -2866,7 +2866,7 @@ static void set_recommended_min_free_kbytes(void)int nr_zones =0 ;unsigned long recommended_min; -if (!hugepage_pmd_enabled ()) { +if (!hugepage_enabled ()) {calculate_min_free_kbytes (); goto update_wmarks; }@@ -2916,7 +2916,7 @@ int start_stop_khugepaged(void)int err =0 ;mutex_lock (&khugepaged_mutex); -if (hugepage_pmd_enabled ()) { +if (hugepage_enabled ()) {if (!khugepaged_thread) khugepaged_thread =kthread_run (khugepaged,NULL ,"khugepaged" );@@ -2942,7 +2942,7 @@ int start_stop_khugepaged(void)void khugepaged_min_free_kbytes_update (void ) {mutex_lock (&khugepaged_mutex); -if (hugepage_pmd_enabled () && khugepaged_thread) +if (hugepage_enabled () && khugepaged_thread)set_recommended_min_free_kbytes ();mutex_unlock (&khugepaged_mutex); } --2.50.1
Re: [PATCH v10 10/13] khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
Oh and on this one I forgot to say - this seems to be where we're effectively turning the whole thing on, right?
I think it's really worth underlining that in the commit message :)
Baolin Wang
reviewerRe: [PATCH v10 10/13] khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
In reply to Lorenzo StoakesQuoted 23 lines · Lorenzo Stoakes
Quoted 3 lines · Nico Pache
From: Baolin Wang <[email protected]>
When only non-PMD-sized mTHP is enabled (such as only 64K mTHP enabled),
I don't think this example is very useful, probably just remove it.
Also 'non-PMD-sized mTHP' implies there is such a thing as PMD-sized mTP :)
Quoted 1 line
we should also allow kicking khugepaged to attempt scanning and collapsing
What is kicking? I think this should be rephrased to something like 'we should also allow khugepaged to attempt scanning...'
Quoted 1 line
64K mTHP. Modify
hugepage_pmd_enabled()to support mTHP collapse, and64K mTHP -> "of mTHP ranges". Put the 'Modify...' bit in a new paragraph to be clear.
Quoted 1 line
while we are at it, rename it to make the function name more clear.
To make this clearer let me suggest:
Quoted 8 lines
In order for khugepaged to operate when only mTHP sizes are specified in sysfs, we must modify the predicate function that determines whether it ought to run to do so.
This function is currently called
hugepage_pmd_enabled(), this patch renames it tohugepage_enabled()and updates the logic to check to determine whether any valid orders may exist which would justify khugepaged running.
Thanks. This looks good to me.
Quoted 33 lines
mm/khugepaged.c static bool hugepage_pmd_enabled(void)
Quoted 21 lines
Signed-off-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)mm/khugepaged.c +1
-1@@ -430,7 +430,7 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm)mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); } -static bool hugepage_pmd_enabled (void ) +static bool hugepage_enabled (void ) {/* * We cover the anon, shmem and the file-backed case here; file-backed @@ -442,11 +442,11 @@ static bool hugepage_pmd_enabled(void)The comment above this still references PMD-sized, please make sure to update comments when you change the described behaviour, as it is now incorrect:
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
* Anon pmd-sized hugepages are determined by the pmd-size control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the globalshmem_hugeis set toSHMEM_HUGE_DENY. */
Quoted 1 line
Please correct this.
Sure. How about:
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
* Anon hugepages are determined by its per-size mTHP control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the global shmem_huge is set to SHMEM_HUGE_DENY. */
Quoted 7 lines
Quoted 7 lines
mm/khugepaged.c +1
-2if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) &&hugepage_global_enabled ())return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_always)) +if (READ_ONCE (huge_anon_orders_always))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_madvise))
Quoted 8 lines
mm/khugepaged.c static bool hugepage_pmd_enabled(void) · line 450
Quoted 5 lines
mm/khugepaged.c +2
-1+if (READ_ONCE (huge_anon_orders_madvise))return true; -if (test_bit (PMD_ORDER , &huge_anon_orders_inherit) && +if (READ_ONCE (huge_anon_orders_inherit) &&hugepage_global_enabled ())I guess
READ_ONCE()is probably sufficient here as memory ordering isn't important here, right?
Yes, I think so.
Dev Jain
reviewerRe: [PATCH v10 10/13] khugepaged: kick khugepaged for enabling none-PMD-sized mTHPs
Quoted 75 lines · Baolin Wang
Quoted 23 lines · Lorenzo Stoakes
Quoted 3 lines · Nico Pache
From: Baolin Wang <[email protected]>
When only non-PMD-sized mTHP is enabled (such as only 64K mTHP enabled),
I don't think this example is very useful, probably just remove it.
Also 'non-PMD-sized mTHP' implies there is such a thing as PMD-sized mTP :)
Quoted 1 line
we should also allow kicking khugepaged to attempt scanning and collapsing
What is kicking? I think this should be rephrased to something like 'we should also allow khugepaged to attempt scanning...'
Quoted 1 line
64K mTHP. Modify
hugepage_pmd_enabled()to support mTHP collapse, and64K mTHP -> "of mTHP ranges". Put the 'Modify...' bit in a new paragraph to be clear.
Quoted 1 line
while we are at it, rename it to make the function name more clear.
To make this clearer let me suggest:
Quoted 8 lines
In order for khugepaged to operate when only mTHP sizes are specified in sysfs, we must modify the predicate function that determines whether it ought to run to do so.
This function is currently called
hugepage_pmd_enabled(), this patch renames it tohugepage_enabled()and updates the logic to check to determine whether any valid orders may exist which would justify khugepaged running.Thanks. This looks good to me.
Quoted 34 lines
mm/khugepaged.c static bool hugepage_pmd_enabled(void)
Quoted 22 lines
Signed-off-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>---
mm/khugepaged.c| 20 ++++++++++----------1 file changed, 10 insertions(+), 10 deletions(-)mm/khugepaged.c +1
-1@@ -430,7 +430,7 @@ static inline int collapse_test_exit_or_disable(struct mm_struct *mm) mm_flags_test(MMF_DISABLE_THP_COMPLETELY, mm); } -static bool hugepage_pmd_enabled (void ) +static bool hugepage_enabled (void ) { /** We cover the anon, shmem and the file-backed case here; file-backed
mm/khugepaged.c +0
-0@@ -442,11 +442,11 @@ static bool hugepage_pmd_enabled(void)The comment above this still references PMD-sized, please make sure to update comments when you change the described behaviour, as it is now incorrect:
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
* Anon pmd-sized hugepages are determined by the pmd-size control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the globalshmem_hugeis set toSHMEM_HUGE_DENY. */Quoted 1 line
Please correct this.
Sure. How about:
Quoted 7 lines
/*
* We cover the anon, shmem and the file-backed case here; file-backed
* hugepages, when configured in, are determined by the global control.
* Anon hugepages are determined by its per-size mTHP control.
* Shmem pmd-sized hugepages are also determined by its pmd-size control,
* except when the globalshmem_hugeis set toSHMEM_HUGE_DENY. */
Looks good, had done something similar in my version.
Quoted 18 lines
Quoted 14 lines
Quoted 13 lines
if (
IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)&&hugepage_global_enabled())
return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_always)) + if (READ_ONCE(huge_anon_orders_always)) return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_madvise)) + if (READ_ONCE(huge_anon_orders_madvise)) return true;
- if (test_bit(PMD_ORDER, &huge_anon_orders_inherit)&& + if (READ_ONCE(huge_anon_orders_inherit)&&hugepage_global_enabled())Quoted 2 lines
I guess
READ_ONCE()is probably sufficient here as memory ordering isn't important here, right?Yes, I think so.
PATCH v10 11/13 khugepaged: improve tracepoints for mTHP orders
In reply to Nico PacheAdd the order to the tracepoints to give better insight into what order is being operated at for khugepaged.
--- include/trace/events/huge_memory.h | 34 +++++++++++++++++++----------- mm/khugepaged.c | 10 +++++---- 2 files changed, 28 insertions(+), 16 deletions(-)
include/trace/events/huge_memory.h +22 -12
@@ -92,34 +92,37 @@ TRACE_EVENT(mm_khugepaged_scan_pmd,TRACE_EVENT (mm_collapse_huge_page, - TP_PROTO(struct mm_struct *mm, int isolated, int status), + TP_PROTO(struct mm_struct *mm, int isolated, int status, unsigned int order), -TP_ARGS (mm, isolated, status), +TP_ARGS (mm, isolated, status, order),TP_STRUCT__entry ( __field(struct mm_struct *, mm) __field(int, isolated) __field(int, status) + __field(unsigned int, order) ),TP_fast_assign ( __entry->mm = mm; __entry->isolated = isolated; __entry->status = status; + __entry->order = order; ), -TP_printk ("mm=%p, isolated=%d, status=%s" , +TP_printk ("mm=%p, isolated=%d, status=%s order=%u" , __entry->mm, __entry->isolated, -__print_symbolic (__entry->status,SCAN_STATUS )) +__print_symbolic (__entry->status,SCAN_STATUS ), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_isolate, TP_PROTO(struct folio *folio, int none_or_zero, - int referenced, bool writable, int status), + int referenced, bool writable, int status, unsigned int order), - TP_ARGS(folio, none_or_zero, referenced, writable, status), + TP_ARGS(folio, none_or_zero, referenced, writable, status, order), TP_STRUCT__entry( __field(unsigned long, pfn)
@@ -127,6 +130,7 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __field(int, referenced) __field(bool, writable) __field(int, status) + __field(unsigned int, order) ), TP_fast_assign(
@@ -135,27 +139,31 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __entry->referenced = referenced; __entry->writable = writable; __entry->status = status; + __entry->order = order; ), -TP_printk ("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s" , +TP_printk ("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s order=%u" , __entry->pfn, __entry->none_or_zero, __entry->referenced, __entry->writable, -__print_symbolic (__entry->status,SCAN_STATUS )) +__print_symbolic (__entry->status,SCAN_STATUS ), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_swapin, - TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret), + TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret, + unsigned int order), - TP_ARGS(mm, swapped_in, referenced, ret), + TP_ARGS(mm, swapped_in, referenced, ret, order), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(int, swapped_in) __field(int, referenced) __field(int, ret) + __field(unsigned int, order) ), TP_fast_assign(
@@ -163,13 +171,15 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, __entry->swapped_in = swapped_in; __entry->referenced = referenced; __entry->ret = ret; + __entry->order = order; ), -TP_printk ("mm=%p, swapped_in=%d, referenced=%d, ret=%d" , +TP_printk ("mm=%p, swapped_in=%d, referenced=%d, ret=%d, order=%u" , __entry->mm, __entry->swapped_in, __entry->referenced, - __entry->ret) + __entry->ret, + __entry->order) ); TRACE_EVENT(mm_khugepaged_scan_file,
mm/khugepaged.c +6 -4
@@ -721,13 +721,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }else { result =SCAN_SUCCEED ;trace_mm_collapse_huge_page_isolate (folio, none_or_zero, - referenced, writable, result); + referenced, writable, result, + order);return result; } out:release_pte_pages (pte, _pte, compound_pagelist);trace_mm_collapse_huge_page_isolate (folio, none_or_zero, - referenced, writable, result); + referenced, writable, result, order);return result; }
@@ -1123,7 +1124,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, result =SCAN_SUCCEED ; out: -trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced, result); +trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced, result, + order);return result; }
@@ -1348,7 +1350,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, *mmap_locked = false;if (folio)folio_put (folio); -trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result); +trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result, order);return result; }
Signature · 1 line
2.50.1
Re: [PATCH v10 11/13] khugepaged: improve tracepoints for mTHP orders
Quoted 2 lines · Nico Pache
Add the order to the tracepoints to give better insight into what order is being operated at for khugepaged.
NIT: Would be good to list the tracepoints you changed here.
Quoted 4 lines
Reviewed-by Baolin Wang <[email protected]>Signed-off-by Nico Pache <[email protected]>
LGTM to me, so:
Quoted 158 lines
--- include/trace/events/huge_memory.h | 34 +++++++++++++++++++----------- mm/khugepaged.c | 10 +++++---- 2 files changed, 28 insertions(+), 16 deletions(-)include/trace/events/huge_memory.h +22
-12@@ -92,34 +92,37 @@ TRACE_EVENT(mm_khugepaged_scan_pmd,TRACE_EVENT (mm_collapse_huge_page, - TP_PROTO(struct mm_struct *mm, int isolated, int status), + TP_PROTO(struct mm_struct *mm, int isolated, int status, unsigned int order), -TP_ARGS (mm, isolated, status), +TP_ARGS (mm, isolated, status, order),TP_STRUCT__entry ( __field(struct mm_struct *, mm) __field(int, isolated) __field(int, status) + __field(unsigned int, order) ),TP_fast_assign ( __entry->mm = mm; __entry->isolated = isolated; __entry->status = status; + __entry->order = order; ), -TP_printk ("mm=%p, isolated=%d, status=%s" , +TP_printk ("mm=%p, isolated=%d, status=%s order=%u" , __entry->mm, __entry->isolated, -__print_symbolic (__entry->status,SCAN_STATUS )) +__print_symbolic (__entry->status,SCAN_STATUS ), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_isolate, TP_PROTO(struct folio *folio, int none_or_zero, - int referenced, bool writable, int status), + int referenced, bool writable, int status, unsigned int order), - TP_ARGS(folio, none_or_zero, referenced, writable, status), + TP_ARGS(folio, none_or_zero, referenced, writable, status, order), TP_STRUCT__entry( __field(unsigned long, pfn)@@ -127,6 +130,7 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __field(int, referenced) __field(bool, writable) __field(int, status) + __field(unsigned int, order) ), TP_fast_assign(@@ -135,27 +139,31 @@ TRACE_EVENT(mm_collapse_huge_page_isolate, __entry->referenced = referenced; __entry->writable = writable; __entry->status = status; + __entry->order = order; ), -TP_printk ("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s" , +TP_printk ("scan_pfn=0x%lx, none_or_zero=%d, referenced=%d, writable=%d, status=%s order=%u" , __entry->pfn, __entry->none_or_zero, __entry->referenced, __entry->writable, -__print_symbolic (__entry->status,SCAN_STATUS )) +__print_symbolic (__entry->status,SCAN_STATUS ), + __entry->order) ); TRACE_EVENT(mm_collapse_huge_page_swapin, - TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret), + TP_PROTO(struct mm_struct *mm, int swapped_in, int referenced, int ret, + unsigned int order), - TP_ARGS(mm, swapped_in, referenced, ret), + TP_ARGS(mm, swapped_in, referenced, ret, order), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(int, swapped_in) __field(int, referenced) __field(int, ret) + __field(unsigned int, order) ), TP_fast_assign(@@ -163,13 +171,15 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, __entry->swapped_in = swapped_in; __entry->referenced = referenced; __entry->ret = ret; + __entry->order = order; ), -TP_printk ("mm=%p, swapped_in=%d, referenced=%d, ret=%d" , +TP_printk ("mm=%p, swapped_in=%d, referenced=%d, ret=%d, order=%u" , __entry->mm, __entry->swapped_in, __entry->referenced, - __entry->ret) + __entry->ret, + __entry->order) ); TRACE_EVENT(mm_khugepaged_scan_file,mm/khugepaged.c +6
-5@@ -721,13 +721,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }else { result =SCAN_SUCCEED ;trace_mm_collapse_huge_page_isolate (folio, none_or_zero, - referenced, writable, result); + referenced, writable, result, + order);return result; } out:release_pte_pages (pte, _pte, compound_pagelist);trace_mm_collapse_huge_page_isolate (folio, none_or_zero, - referenced, writable, result); + referenced, writable, result, order);return result; }@@ -1123,7 +1124,8 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, result =SCAN_SUCCEED ; out: -trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced, result); +trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced, result, + order);return result; }@@ -1348,7 +1350,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, *mmap_locked = false;if (folio)folio_put (folio); -trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result); +trace_mm_collapse_huge_page (mm, result ==SCAN_SUCCEED , result, order);return result; } --2.50.1
PATCH v10 12/13 khugepaged: add per-order mTHP khugepaged stats
In reply to Nico PacheWith mTHP support inplace, let add the per-order mTHP stats for exceeding NONE, SWAP, and SHARED.
--- Documentation/admin-guide/mm/transhuge.rst | 17 +++++++++++++++++ include/linux/huge_mm.h | 3 +++ mm/huge_memory.c | 7 +++++++ mm/khugepaged.c | 16 +++++++++++++--- 4 files changed, 40 insertions(+), 3 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +17 -0
@@ -705,6 +705,23 @@ nr_anon_partially_mapped an anonymous THP as "partially mapped" and count it here, even though it is not actually partially mapped anymore. +collapse_exceed_swap_pte + The number of anonymous THP which contain at least one swap PTE. + Currently khugepaged does not support collapsing mTHP regions that + contain a swap PTE. + +collapse_exceed_none_pte + The number of anonymous THP which have exceeded the none PTE threshold. + With mTHP collapse, a bitmap is used to gather the state of a PMD region + and is then recursively checked from largest to smallest order against + the scaled max_ptes_none count. This counter indicates that the next + enabled order will be checked. + +collapse_exceed_shared_pte + The number of anonymous THP which contain at least one shared PTE. + Currently khugepaged does not support collapsing mTHP regions that + contain a shared PTE. + As the system ages, allocating huge pages may be expensive as the system uses memory compaction to copy data around memory to free a huge page for use. There are some counters in ``/proc/vmstat`` to help
include/linux/huge_mm.h +3 -0
@@ -144,6 +144,9 @@ enum mthp_stat_item {MTHP_STAT_SPLIT_DEFERRED ,MTHP_STAT_NR_ANON ,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED , +MTHP_STAT_COLLAPSE_EXCEED_SWAP , +MTHP_STAT_COLLAPSE_EXCEED_NONE , +MTHP_STAT_COLLAPSE_EXCEED_SHARED , __MTHP_STAT_COUNT };
mm/huge_memory.c +7 -0
@@ -639,6 +639,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);DEFINE_MTHP_STAT_ATTR (split_deferred,MTHP_STAT_SPLIT_DEFERRED );DEFINE_MTHP_STAT_ATTR (nr_anon,MTHP_STAT_NR_ANON );DEFINE_MTHP_STAT_ATTR (nr_anon_partially_mapped,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_swap_pte,MTHP_STAT_COLLAPSE_EXCEED_SWAP ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_none_pte,MTHP_STAT_COLLAPSE_EXCEED_NONE ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_shared_pte,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); +static struct attribute *anon_stats_attrs[] = { &anon_fault_alloc_attr.attr,
@@ -655,6 +659,9 @@ static struct attribute *anon_stats_attrs[] = { &split_deferred_attr.attr, &nr_anon_attr.attr, &nr_anon_partially_mapped_attr.attr, + &collapse_exceed_swap_pte_attr.attr, + &collapse_exceed_none_pte_attr.attr, + &collapse_exceed_shared_pte_attr.attr, NULL, };
mm/khugepaged.c +13 -3
@@ -594,7 +594,9 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,continue ; }else { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_NONE ); goto out; } }
@@ -633,10 +635,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,* shared may cause a future higher order collapse on a * rescan of the same range. */ -if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared)) { +if (order !=HPAGE_PMD_ORDER ) { + result =SCAN_EXCEED_SHARED_PTE ; +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); + goto out; + } + +if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); goto out; } }
@@ -1084,6 +1093,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,* range. */ if (order !=HPAGE_PMD_ORDER ) { +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SWAP );pte_unmap (pte);mmap_read_unlock (mm); result =SCAN_EXCEED_SWAP_PTE ;
Signature · 1 line
2.50.1
Re: [PATCH v10 12/13] khugepaged: add per-order mTHP khugepaged stats
Quoted 3 lines · Nico Pache
With mTHP support inplace, let add the per-order mTHP stats for exceeding NONE, SWAP, and SHARED.
This is really not enough of a commit message. Exceeding what, where, why, how? What does 'exceeding' mean here, etc. etc. More words please :)
Quoted 13 lines
Signed-off-by Nico Pache <[email protected]>--- Documentation/admin-guide/mm/transhuge.rst | 17 +++++++++++++++++ include/linux/huge_mm.h | 3 +++ mm/huge_memory.c | 7 +++++++ mm/khugepaged.c | 16 +++++++++++++--- 4 files changed, 40 insertions(+), 3 deletions(-)Documentation/admin-guide/mm/transhuge.rst +0
-0
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 709
Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0@@ -705,6 +705,23 @@ nr_anon_partially_mapped an anonymous THP as "partially mapped" and count it here, even though it is not actually partially mapped anymore. +collapse_exceed_swap_pte + The number of anonymous THP which contain at least one swap PTE.
The number of anonymous THP what? Pages? Let's be specific.
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 711
Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a swap PTE.
Wait what? So we have a counter for something that's unsupported? That seems not so useful?
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 714
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_none_pte + The number of anonymous THP which have exceeded the none PTE threshold.
THP pages. What's the 'none PTE threshold'? Do you mean /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none ?
Let's spell that out please, this is far too vague.
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 718
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0+ With mTHP collapse, a bitmap is used to gather the state of a PMD region + and is then recursively checked from largest to smallest order against + the scaled max_ptes_none count. This counter indicates that the next + enabled order will be checked.
I think you really need to expand upon this as this is confusing and vague.
I also don't think saying 'recursive' here really benefits anything, Just saying that we try to collapse the largest mTHP size we can in each instance, and then give a more 'words-y' explanation as to how max_ptes_none is (in effect) converted to a ratio of a PMD, and then that ratio is applied to the mTHP sizes.
You can then go on to say that this counter measures the number of occasions in which this occurred.
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 721
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_shared_pte + The number of anonymous THP which contain at least one shared PTE.
anonymous THP pages right? :)
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 723
Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a shared PTE.
Again I don't really understand the purpose of creating a counter for something we don't support.
Let's add it when we support it.
I also in this case and the exceed swap case don't understand what you mean by exceed here, you need to spell this out clearly.
Perhaps the context missing here is that you _also_ count THP events in these counters.
But again, given we have THP_... counters for the stats mTHP doesn't do yet, I'd say adding these is pointless.
Quoted 9 lines
Documentation/admin-guide/mm/transhuge.rst +1
-0+ As the system ages, allocating huge pages may be expensive as the system uses memory compaction to copy data around memory to free a huge page for use. There are some counters in ``/proc/vmstat`` to helpinclude/linux/huge_mm.h +0
-0
include/linux/huge_mm.h enum mthp_stat_item { · line 149
Quoted 6 lines
include/linux/huge_mm.h +3
-0@@ -144,6 +144,9 @@ enum mthp_stat_item {MTHP_STAT_SPLIT_DEFERRED ,MTHP_STAT_NR_ANON ,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED , +MTHP_STAT_COLLAPSE_EXCEED_SWAP , +MTHP_STAT_COLLAPSE_EXCEED_NONE , + MTHP_STAT_COLLAPSE_EXCEED_SHARED,
Wh do we put 'collapse' here but not in the THP equivalents?
Quoted 33 lines
include/linux/huge_mm.h +0
-0__MTHP_STAT_COUNT };mm/huge_memory.c +7
-0@@ -639,6 +639,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);DEFINE_MTHP_STAT_ATTR (split_deferred,MTHP_STAT_SPLIT_DEFERRED );DEFINE_MTHP_STAT_ATTR (nr_anon,MTHP_STAT_NR_ANON );DEFINE_MTHP_STAT_ATTR (nr_anon_partially_mapped,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_swap_pte,MTHP_STAT_COLLAPSE_EXCEED_SWAP ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_none_pte,MTHP_STAT_COLLAPSE_EXCEED_NONE ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_shared_pte,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); +static struct attribute *anon_stats_attrs[] = { &anon_fault_alloc_attr.attr,@@ -655,6 +659,9 @@ static struct attribute *anon_stats_attrs[] = { &split_deferred_attr.attr, &nr_anon_attr.attr, &nr_anon_partially_mapped_attr.attr, + &collapse_exceed_swap_pte_attr.attr, + &collapse_exceed_none_pte_attr.attr, + &collapse_exceed_shared_pte_attr.attr, NULL, };mm/khugepaged.c +0
-0
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 597
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -594,7 +594,9 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,continue ; }else { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE );
Hm so wait you were miscounting statistics in patch 10/13 when you turned all this one? That's not good.
This should be in place _first_ before enabling the feature.
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 638
Quoted 13 lines
mm/khugepaged.c +4
-2+if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_NONE ); goto out; } }@@ -633,10 +635,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,* shared may cause a future higher order collapse on a * rescan of the same range. */ -if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared)) { +if (order !=HPAGE_PMD_ORDER ) {
Hm wait what? I dont understand what's going on here? You're no longer actually doing any check except order != HPAGE_PMD_ORDER?... am I missnig something?
Again why we are bothering to maintain a counter that doesn't mean anything I don't know? I may be misinterpreting somehow however.
Quoted 14 lines
mm/khugepaged.c +8
-0+ result =SCAN_EXCEED_SHARED_PTE ; +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); + goto out; + } + +if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); goto out; } }
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1096
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -1084,6 +1093,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,* range. */ if (order !=HPAGE_PMD_ORDER ) { +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SWAP );
This again seems surely to not be testing for what it claims to be tracking? I may again be missing context here.
Quoted 6 lines
mm/khugepaged.c +0
-1pte_unmap (pte);mmap_read_unlock (mm); result =SCAN_EXCEED_SWAP_PTE ; --2.50.1
Re: [PATCH v10 12/13] khugepaged: add per-order mTHP khugepaged stats
Quoted 8 lines · Lorenzo Stoakes
Quoted 3 lines · Nico Pache
With mTHP support inplace, let add the per-order mTHP stats for exceeding NONE, SWAP, and SHARED.
This is really not enough of a commit message. Exceeding what, where, why, how? What does 'exceeding' mean here, etc. etc. More words please :)
Ok I will add more in the next version
Quoted 17 lines
Quoted 17 lines
Signed-off-by Nico Pache <[email protected]>--- Documentation/admin-guide/mm/transhuge.rst | 17 +++++++++++++++++ include/linux/huge_mm.h | 3 +++ mm/huge_memory.c | 7 +++++++ mm/khugepaged.c | 16 +++++++++++++--- 4 files changed, 40 insertions(+), 3 deletions(-)Documentation/admin-guide/mm/transhuge.rst +0
-0@@ -705,6 +705,23 @@ nr_anon_partially_mapped an anonymous THP as "partially mapped" and count it here, even though it is not actually partially mapped anymore.
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 709
Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+collapse_exceed_swap_pte + The number of anonymous THP which contain at least one swap PTE.The number of anonymous THP what? Pages? Let's be specific.
ack
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 711
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a swap PTE.Wait what? So we have a counter for something that's unsupported? That seems not so useful?
The current implementation does not support swapped out or shared pages. However these counters allow us to monitor when a mTHP collapse fails due to exceeding the threshold (ie 0, hitting any swapped out or shared page)
Quoted 7 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 714
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_none_pte + The number of anonymous THP which have exceeded the none PTE threshold.THP pages. What's the 'none PTE threshold'? Do you mean
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none?
ack, I will expand these descriptions
Quoted 10 lines
Let's spell that out please, this is far too vague.
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 718
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0+ With mTHP collapse, a bitmap is used to gather the state of a PMD region + and is then recursively checked from largest to smallest order against + the scaled max_ptes_none count. This counter indicates that the next + enabled order will be checked.I think you really need to expand upon this as this is confusing and vague.
Quoted 8 lines
I also don't think saying 'recursive' here really benefits anything, Just saying that we try to collapse the largest mTHP size we can in each instance, and then give a more 'words-y' explanation as to how
max_ptes_noneis (in effect) converted to a ratio of a PMD, and then that ratio is applied to the mTHP sizes.You can then go on to say that this counter measures the number of occasions in which this occurred.
ack I will clean it up
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 721
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_shared_pte + The number of anonymous THP which contain at least one shared PTE.anonymous THP pages right? :)
regions?
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 723
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a shared PTE.Again I don't really understand the purpose of creating a counter for something we don't support.
see above
Quoted 21 lines
Let's add it when we support it.
I also in this case and the exceed swap case don't understand what you mean by exceed here, you need to spell this out clearly.
Perhaps the context missing here is that you _also_ count THP events in these counters.
But again, given we have
THP_... counters for the stats mTHP doesn't do yet, I'd say adding these is pointless.Quoted 9 lines
Documentation/admin-guide/mm/transhuge.rst +1
-0+ As the system ages, allocating huge pages may be expensive as the system uses memory compaction to copy data around memory to free a huge page for use. There are some counters in ``/proc/vmstat`` to helpinclude/linux/huge_mm.h +0
-0@@ -144,6 +144,9 @@ enum mthp_stat_item {
Quoted 8 lines
include/linux/huge_mm.h enum mthp_stat_item { · line 149
Quoted 6 lines
include/linux/huge_mm.h +3
-0MTHP_STAT_SPLIT_DEFERRED ,MTHP_STAT_NR_ANON ,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED , +MTHP_STAT_COLLAPSE_EXCEED_SWAP , +MTHP_STAT_COLLAPSE_EXCEED_NONE , + MTHP_STAT_COLLAPSE_EXCEED_SHARED,Wh do we put 'collapse' here but not in the THP equivalents?
to indicate they come from the collapse functionality. I can shorten it by removing COLLAPSE if youd like
Quoted 42 lines
Quoted 34 lines
include/linux/huge_mm.h +0
-0__MTHP_STAT_COUNT };mm/huge_memory.c +7
-0@@ -639,6 +639,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);DEFINE_MTHP_STAT_ATTR (split_deferred,MTHP_STAT_SPLIT_DEFERRED );DEFINE_MTHP_STAT_ATTR (nr_anon,MTHP_STAT_NR_ANON );DEFINE_MTHP_STAT_ATTR (nr_anon_partially_mapped,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_swap_pte,MTHP_STAT_COLLAPSE_EXCEED_SWAP ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_none_pte,MTHP_STAT_COLLAPSE_EXCEED_NONE ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_shared_pte,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); +static struct attribute *anon_stats_attrs[] = { &anon_fault_alloc_attr.attr,@@ -655,6 +659,9 @@ static struct attribute *anon_stats_attrs[] = { &split_deferred_attr.attr, &nr_anon_attr.attr, &nr_anon_partially_mapped_attr.attr, + &collapse_exceed_swap_pte_attr.attr, + &collapse_exceed_none_pte_attr.attr, + &collapse_exceed_shared_pte_attr.attr, NULL, };mm/khugepaged.c +0
-0mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 597
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -594,7 +594,9 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,continue ; }else { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE );Hm so wait you were miscounting statistics in patch 10/13 when you turned all this one? That's not good.
Quoted 1 line
This should be in place _first_ before enabling the feature.
Ok I can move them around.
Quoted 35 lines
Quoted 8 lines
mm/khugepaged.c +3
-0+if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_NONE ); goto out; } }mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 638
Quoted 6 lines
mm/khugepaged.c +1
-2@@ -633,10 +635,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,* shared may cause a future higher order collapse on a * rescan of the same range. */ -if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared)) { +if (order !=HPAGE_PMD_ORDER ) {Hm wait what? I dont understand what's going on here? You're no longer actually doing any check except order !=
HPAGE_PMD_ORDER?... am I missnig something?Again why we are bothering to maintain a counter that doesn't mean anything I don't know? I may be misinterpreting somehow however.
Quoted 13 lines
mm/khugepaged.c +8
-0+ result =SCAN_EXCEED_SHARED_PTE ; +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); + goto out; + } + +if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); goto out; } }
Quoted 8 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1096
Quoted 5 lines
mm/khugepaged.c +1
-0@@ -1084,6 +1093,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,* range. */ if (order !=HPAGE_PMD_ORDER ) { +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SWAP );This again seems surely to not be testing for what it claims to be tracking? I may again be missing context here.
We are bailing out of the mTHP collapse due to it having a SWAP page. In turn exceeding our threshold of 0.
Cheers,
-- Nico
Quoted 8 lines
Quoted 7 lines
mm/khugepaged.c +0
-1pte_unmap (pte);mmap_read_unlock (mm); result =SCAN_EXCEED_SWAP_PTE ; --2.50.1
Re: [PATCH v10 12/13] khugepaged: add per-order mTHP khugepaged stats
One small nitty note - be super helpful if you could add a newline before/after your reply or soething like this as
Quoted 3 lines
Quoted 1 line
blah blah blah
blah
Quoted 1 line
blah blah blah
Is harder to read than:
Quoted 5 lines
Quoted 1 line
blah blah blah
blah
Quoted 1 line
blah blah blah
Thanks :)
Quoted 11 lines · Nico Pache
Quoted 8 lines · Lorenzo Stoakes
Quoted 3 lines · Nico Pache
With mTHP support inplace, let add the per-order mTHP stats for exceeding NONE, SWAP, and SHARED.
This is really not enough of a commit message. Exceeding what, where, why, how? What does 'exceeding' mean here, etc. etc. More words please :)
Ok I will add more in the next version
Thanks
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Signed-off-by Nico Pache <[email protected]>--- Documentation/admin-guide/mm/transhuge.rst | 17 +++++++++++++++++ include/linux/huge_mm.h | 3 +++ mm/huge_memory.c | 7 +++++++ mm/khugepaged.c | 16 +++++++++++++--- 4 files changed, 40 insertions(+), 3 deletions(-)Documentation/admin-guide/mm/transhuge.rst +0
-0@@ -705,6 +705,23 @@ nr_anon_partially_mapped an anonymous THP as "partially mapped" and count it here, even though it is not actually partially mapped anymore.
Quoted 5 lines
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 709
Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+collapse_exceed_swap_pte + The number of anonymous THP which contain at least one swap PTE.The number of anonymous THP what? Pages? Let's be specific.
ack
Thanks
Quoted 10 lines
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 711
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a swap PTE.Wait what? So we have a counter for something that's unsupported? That seems not so useful?
The current implementation does not support swapped out or shared pages. However these counters allow us to monitor when a mTHP collapse fails due to exceeding the threshold (ie 0, hitting any swapped out or shared page)
So the collapse counters are not measuring collapses? That seems a bit confusing. Or actually is this implied in the 'exceed' bit? Because that'd make sense actually.
But let's obviously document this carefully.
Quoted 8 lines
Quoted 7 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 714
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_none_pte + The number of anonymous THP which have exceeded the none PTE threshold.THP pages. What's the 'none PTE threshold'? Do you mean
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none?ack, I will expand these descriptions
Thanks.
Quoted 16 lines
Quoted 16 lines
Let's spell that out please, this is far too vague.
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 718
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0+ With mTHP collapse, a bitmap is used to gather the state of a PMD region + and is then recursively checked from largest to smallest order against + the scaled max_ptes_none count. This counter indicates that the next + enabled order will be checked.I think you really need to expand upon this as this is confusing and vague.
I also don't think saying 'recursive' here really benefits anything, Just saying that we try to collapse the largest mTHP size we can in each instance, and then give a more 'words-y' explanation as to how
max_ptes_noneis (in effect) converted to a ratio of a PMD, and then that ratio is applied to the mTHP sizes.
Quoted 3 lines
Quoted 2 lines
You can then go on to say that this counter measures the number of occasions in which this occurred.
ack I will clean it up
Thanks
Quoted 7 lines
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 721
Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +3
-0+ +collapse_exceed_shared_pte + The number of anonymous THP which contain at least one shared PTE.anonymous THP pages right? :)
regions?
I don't understand what regions would mean here?
So what are you actually measuring? The number of anonymous THP mappings? If so I think 'mappings' is probably better. Or 'The number of anonymous THP page table ranges...' perhaps?
Quoted 8 lines
Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst nr_anon_partially_mapped · line 723
Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +2
-0+ Currently khugepaged does not support collapsing mTHP regions that + contain a shared PTE.Again I don't really understand the purpose of creating a counter for something we don't support.
see above
Ack
Quoted 30 lines
Quoted 21 lines
Let's add it when we support it.
I also in this case and the exceed swap case don't understand what you mean by exceed here, you need to spell this out clearly.
Perhaps the context missing here is that you _also_ count THP events in these counters.
But again, given we have
THP_... counters for the stats mTHP doesn't do yet, I'd say adding these is pointless.Quoted 9 lines
Documentation/admin-guide/mm/transhuge.rst +1
-0+ As the system ages, allocating huge pages may be expensive as the system uses memory compaction to copy data around memory to free a huge page for use. There are some counters in ``/proc/vmstat`` to helpinclude/linux/huge_mm.h +0
-0@@ -144,6 +144,9 @@ enum mthp_stat_item {Quoted 8 lines
include/linux/huge_mm.h enum mthp_stat_item { · line 149
Quoted 6 lines
include/linux/huge_mm.h +3
-0MTHP_STAT_SPLIT_DEFERRED ,MTHP_STAT_NR_ANON ,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED , +MTHP_STAT_COLLAPSE_EXCEED_SWAP , +MTHP_STAT_COLLAPSE_EXCEED_NONE , + MTHP_STAT_COLLAPSE_EXCEED_SHARED,Wh do we put 'collapse' here but not in the THP equivalents?
Quoted 2 lines
to indicate they come from the collapse functionality. I can shorten it by removing COLLAPSE if youd like
Hmm, if this is actually giving information then fine to keep.
Quoted 42 lines
Quoted 42 lines
Quoted 34 lines
include/linux/huge_mm.h +0
-0__MTHP_STAT_COUNT };mm/huge_memory.c +7
-0@@ -639,6 +639,10 @@ DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);DEFINE_MTHP_STAT_ATTR (split_deferred,MTHP_STAT_SPLIT_DEFERRED );DEFINE_MTHP_STAT_ATTR (nr_anon,MTHP_STAT_NR_ANON );DEFINE_MTHP_STAT_ATTR (nr_anon_partially_mapped,MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_swap_pte,MTHP_STAT_COLLAPSE_EXCEED_SWAP ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_none_pte,MTHP_STAT_COLLAPSE_EXCEED_NONE ); +DEFINE_MTHP_STAT_ATTR (collapse_exceed_shared_pte,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); +static struct attribute *anon_stats_attrs[] = { &anon_fault_alloc_attr.attr,@@ -655,6 +659,9 @@ static struct attribute *anon_stats_attrs[] = { &split_deferred_attr.attr, &nr_anon_attr.attr, &nr_anon_partially_mapped_attr.attr, + &collapse_exceed_swap_pte_attr.attr, + &collapse_exceed_none_pte_attr.attr, + &collapse_exceed_shared_pte_attr.attr, NULL, };mm/khugepaged.c +0
-0mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 597
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -594,7 +594,9 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,continue ; }else { result =SCAN_EXCEED_NONE_PTE ; -count_vm_event (THP_SCAN_EXCEED_NONE_PTE );Hm so wait you were miscounting statistics in patch 10/13 when you turned all this one? That's not good.
Quoted 2 lines
Quoted 1 line
This should be in place _first_ before enabling the feature.
Ok I can move them around.
Thanks
Quoted 19 lines
Quoted 19 lines
Quoted 8 lines
mm/khugepaged.c +3
-0+if (order ==HPAGE_PMD_ORDER ) +count_vm_event (THP_SCAN_EXCEED_NONE_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_NONE ); goto out; } }mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 638
Quoted 6 lines
mm/khugepaged.c +1
-2@@ -633,10 +635,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,* shared may cause a future higher order collapse on a * rescan of the same range. */ -if (order !=HPAGE_PMD_ORDER || (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared)) { +if (order !=HPAGE_PMD_ORDER ) {Hm wait what? I dont understand what's going on here? You're no longer actually doing any check except order !=
HPAGE_PMD_ORDER?... am I missnig something?
Quoted 2 lines
Quoted 2 lines
Again why we are bothering to maintain a counter that doesn't mean anything I don't know? I may be misinterpreting somehow however.
I guess answered by rest.
Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
mm/khugepaged.c +8
-0+ result =SCAN_EXCEED_SHARED_PTE ; +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); + goto out; + } + +if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { result =SCAN_EXCEED_SHARED_PTE ;count_vm_event (THP_SCAN_EXCEED_SHARED_PTE ); +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SHARED ); goto out; } }Quoted 8 lines
mm/khugepaged.c static int __collapse_huge_page_swapin(struct mm_struct *mm, · line 1096
Quoted 5 lines
mm/khugepaged.c +1
-0@@ -1084,6 +1093,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm,* range. */ if (order !=HPAGE_PMD_ORDER ) { +count_mthp_stat (order,MTHP_STAT_COLLAPSE_EXCEED_SWAP );This again seems surely to not be testing for what it claims to be tracking? I may again be missing context here.
Quoted 2 lines
We are bailing out of the mTHP collapse due to it having a SWAP page. In turn exceeding our threshold of 0.
OK.
Quoted 12 lines
Cheers,
-- NicoQuoted 8 lines
Quoted 7 lines
mm/khugepaged.c +0
-1pte_unmap (pte);mmap_read_unlock (mm); result =SCAN_EXCEED_SWAP_PTE ; --2.50.1
PATCH v10 13/13 Documentation: mm: update the admin guide for mTHP collapse
In reply to Nico PacheNow that we can collapse to mTHPs lets update the admin guide to reflect these changes and provide proper guidence on how to utilize it.
--- Documentation/admin-guide/mm/transhuge.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +13 -6
@@ -63,7 +63,7 @@ often. THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task's address space. Unless THP is completely disabled, there is ``khugepaged`` daemon that scans memory and -collapses sequences of basic pages into PMD-sized huge pages. +collapses sequences of basic pages into huge pages. The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>` interface and using madvise(2) and prctl(2) system calls.
@@ -149,6 +149,18 @@ hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation. +khugepaged uses max_ptes_none scaled to the order of the enabled mTHP size +to determine collapses. When using mTHPs it's recommended to set +max_ptes_none low-- ideally less than HPAGE_PMD_NR / 2 (255 on 4k page +size). This will prevent undesired "creep" behavior that leads to +continuously collapsing to the largest mTHP size; when we collapse, we are +bringing in new non-zero pages that will, on a subsequent scan, cause the +max_ptes_none check of the +1 order to always be satisfied. By limiting +this to less than half the current order, we make sure we don't cause this +feedback loop. max_ptes_shared and max_ptes_swap have no effect when +collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out +pages. + It's also possible to limit defrag efforts in the VM to generate anonymous hugepages in case they're not immediately free to madvise regions or to never try to defrag memory and simply fallback to regular
@@ -264,11 +276,6 @@ support the following arguments:: Khugepaged controls ------------------- -.. note:: - khugepaged currently only searches for opportunities to collapse to - PMD-sized THP and no attempt is made to collapse to other THP - sizes. - khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's
Signature · 1 line
2.50.1
Re: [PATCH v10 13/13] Documentation: mm: update the admin guide for mTHP collapse
Quoted 14 lines · Nico Pache
Now that we can collapse to mTHPs lets update the admin guide to reflect these changes and provide proper guidence on how to utilize it.
Reviewed-by Bagas Sanjaya <[email protected]>Signed-off-by Nico Pache <[email protected]>--- Documentation/admin-guide/mm/transhuge.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)Documentation/admin-guide/mm/transhuge.rst +0
-0
Documentation/admin-guide/mm/transhuge.rst often. · line 66
Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +1
-1@@ -63,7 +63,7 @@ often. THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task's address space. Unless THP is completely disabled, there is ``khugepaged`` daemon that scans memory and -collapses sequences of basic pages into PMD-sized huge pages. +collapses sequences of basic pages into huge pages.
Maybe worth saying 'of either PMD size or mTHP sizes, if the system is configured to do so.' to really spell it out.
Documentation/admin-guide/mm/transhuge.rst hugepage sizes have enabled="never". If enabling multiple hugepage · line 155
Quoted 11 lines
Documentation/admin-guide/mm/transhuge.rst +4
-0The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>` interface and using madvise(2) and prctl(2) system calls.@@ -149,6 +149,18 @@ hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation. +khugepaged uses max_ptes_none scaled to the order of the enabled mTHP size +to determine collapses. When using mTHPs it's recommended to set +max_ptes_none low-- ideally less than HPAGE_PMD_NR / 2 (255 on 4k page +size). This will prevent undesired "creep" behavior that leads to
Woah wait what??
OK sorry but this is crazy - I've sent a reply to the 00/13 patch as this is really concerning, this suggests to me that this sysctl is completely broken for this purpose.
I think the series has to be altered to introduce an mthp-specific ratio, anwyay let's discuss that on the 00/13 reply thread please.
Documentation/admin-guide/mm/transhuge.rst hugepage sizes have enabled="never". If enabling multiple hugepage · line 162
Quoted 7 lines
Documentation/admin-guide/mm/transhuge.rst +7
-0+continuously collapsing to the largest mTHP size; when we collapse, we are +bringing in new non-zero pages that will, on a subsequent scan, cause the +max_ptes_none check of the +1 order to always be satisfied. By limiting +this to less than half the current order, we make sure we don't cause this +feedback loop. max_ptes_shared and max_ptes_swap have no effect when +collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out +pages.
In general, though I actually want you fundamentally change your approach so you don't need to document this at all, this whole sentence is far too dense, it needs paragraphs and some text around it so it's easier to read, right now it's really quite difficult.
Quoted 17 lines
Documentation/admin-guide/mm/transhuge.rst +1
-6+ It's also possible to limit defrag efforts in the VM to generate anonymous hugepages in case they're not immediately free to madvise regions or to never try to defrag memory and simply fallback to regular@@ -264,11 +276,6 @@ support the following arguments:: Khugepaged controls ------------------- -.. note:: - khugepaged currently only searches for opportunities to collapse to - PMD-sized THP and no attempt is made to collapse to other THP - sizes. - khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's --
Quoted 2 lines
2.50.1
Thanks, Lorenzo
Andrew Morton
maintainerQuoted 9 lines · On Tue, 19 Aug 2025 07:41:52 -0600 Nico Pache <[email protected]> wrote:
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
...
- I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit)
Could this be turned into something in tools/testing/selftests/mm/?
Quoted 1 line
V10 Changes:
I'll add this to mm-new, thanks. I'll suppress the usual emails.
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Tue, 19 Aug 2025 07:41:52 -0600 Nico Pache <[email protected]> wrote:
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
Quoted 8 lines
Quoted 6 lines
...
- I created a test script that I used to push khugepaged to its limits while monitoring a number of stats and tracepoints. The code is available here[1] (Run in legacy mode for these changes and set mthp sizes to inherit)
Could this be turned into something in
tools/testing/selftests/mm/?
Yep! I was actually working on some selftests for this before I hit a weird bug during some of my testing, which took precedence over that.
I was planning on sending a separate series for the testing. One of the pain points was that selftests helpers were set up for PMDs, but I think Baolin just cleaned that up in his khugepaged mTHP shmem series (still need to review). So it should be a lot easier to implement now.
Quoted 4 lines
Quoted 2 lines
V10 Changes:
I'll add this to mm-new, thanks. I'll suppress the usual emails.
Thank you :)
-- Nico
Quoted 1 line
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses max_ptes_none scaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to set max_ptes_none low-- ideally less than HPAGE_PMD_NR / 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause the max_ptes_none check of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop. max_ptes_shared and max_ptes_swap have no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.
This seems to me to suggest that using /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none as some means of establishing a 'ratio' to do this calculation is fundamentally flawed.
So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratio
Or something like this?
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
Dev Jain
reviewerQuoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/
The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/
ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/
This should help you with the context.
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Thanks and I"ll have a look, but this series is unmergeable with a broken default in /sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratio sorry.
We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Thanks, Lorenzo
Quoted 56 lines · Lorenzo Stoakes
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/Quoted 9 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
The first thing that comes to mind is that we can pin max_ptes_none to 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.
ie)
if order!=HPAGE_PMD_ORDER && khugepaged_max_ptes_none > 255
temp_max_ptes_none = 255;
Quoted 3 lines
Thanks, Lorenzo
Quoted 67 lines · Nico Pache
Quoted 56 lines · Lorenzo Stoakes
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/Quoted 9 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 7 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;
Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 3 lines
Quoted 3 lines
Thanks, Lorenzo
Quoted 65 lines · Nico Pache
Quoted 64 lines · Nico Pache
Quoted 61 lines · Lorenzo Stoakes
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/Quoted 9 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.
No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
Quoted 7 lines
Quoted 4 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
And given you're now proposing changing the default for even THP pages with a cap or perhaps having mTHP being used silently change the cap - that is clearly _far_ worse from a tuning standpoint.
With a new tunable you can just set a sensible default and people don't even necessarily have to think about it.
Quoted 70 lines · Lorenzo Stoakes
Quoted 68 lines · Nico Pache
Quoted 67 lines · Nico Pache
Quoted 64 lines · Lorenzo Stoakes
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/Quoted 9 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
We wouldn't be changing it for PMD collapse, just for the new behavior. At 511, no mTHP collapses would ever occur anyways, unless you have 2MB disabled and other mTHP sizes enabled. Technically at 511 only the highest enabled order always gets collapsed.
Ive also argued in the past that 511 is a terrible default for anything other than thp.enabled=always, but that's a whole other can of worms we dont need to discuss now.
with this cap of 255, the PMD scan/collapse would work as intended, then in mTHP collapses we would never introduce this undesired behavior. We've discussed before that this would be a hard problem to solve without introducing some expensive way of tracking what has already been through a collapse, and that doesnt even consider what happens if things change or are unmapped, and rescanning that section would be helpful. So having a strictly enforced limit of 255 actually seems like a good idea to me, as it completely avoids the undesired behavior and does not require the admins to be aware of such an issue.
Another thought similar to what (IIRC) Dev has mentioned before, if we have max_ptes_none > 255 then we only consider collapses to the largest enabled order, that way no creep to the largest enabled order would occur in the first place, and we would get there straight away.
To me one of these two solutions seem sane in the context of what we are dealing with.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 6 lines
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
I agree that the current tunable wasn't designed for this, but we tried to come up with something that leverages the tunable we have to avoid new tunables and added complexity.
Quoted 3 lines
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
What happens if max_ptes_none = 0 and the ratio is 50% - 1 pte (ideally the max number)? seems like we would be saying we want no new none pages, but also to allow new none pages. To me that seems equally broken and more confusing than just taking a scale of the current number (now with a cap).
-- Nico
Quoted 8 lines
And given you're now proposing changing the default for even THP pages with a cap or perhaps having mTHP being used silently change the cap - that is clearly _far_ worse from a tuning standpoint.
With a new tunable you can just set a sensible default and people don't even necessarily have to think about it.
Quoted 59 lines · Nico Pache
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
We wouldn't be changing it for PMD collapse, just for the new behavior. At 511, no mTHP collapses would ever occur anyways, unless you have 2MB disabled and other mTHP sizes enabled. Technically at 511 only the highest enabled order always gets collapsed.
Ive also argued in the past that 511 is a terrible default for anything other than thp.enabled=always, but that's a whole other can of worms we dont need to discuss now.
with this cap of 255, the PMD scan/collapse would work as intended, then in mTHP collapses we would never introduce this undesired behavior. We've discussed before that this would be a hard problem to solve without introducing some expensive way of tracking what has already been through a collapse, and that doesnt even consider what happens if things change or are unmapped, and rescanning that section would be helpful. So having a strictly enforced limit of 255 actually seems like a good idea to me, as it completely avoids the undesired behavior and does not require the admins to be aware of such an issue.
Another thought similar to what (IIRC) Dev has mentioned before, if we have
max_ptes_none> 255 then we only consider collapses to the largest enabled order, that way no creep to the largest enabled order would occur in the first place, and we would get there straight away.To me one of these two solutions seem sane in the context of what we are dealing with.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 6 lines
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
I agree that the current tunable wasn't designed for this, but we tried to come up with something that leverages the tunable we have to avoid new tunables and added complexity.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
What happens if
max_ptes_none= 0 and the ratio is 50% - 1 pte (ideally the max number)? seems like we would be saying we want no new none pages, but also to allow new none pages. To me that seems equally broken and more confusing than just taking a scale of the current number (now with a cap).
The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
I spoke to David off-list and he gave some insight into this and perhaps some reasonable means of avoiding an additional tunable.
I don't want to rehash what he said as I think it's more productive for him to reply when he has time but broadly I think how we handle this needs careful consideration.
To me it's clear that some sense of ratio is just immediately very very confusing, but then again this interface is already confusing, as with much of THP.
Anyway I'll let David respond here so we don't loop around before he has a chance to add his input.
Cheers, Lorenzo
Quoted 81 lines · Lorenzo Stoakes
Quoted 59 lines · Nico Pache
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
We wouldn't be changing it for PMD collapse, just for the new behavior. At 511, no mTHP collapses would ever occur anyways, unless you have 2MB disabled and other mTHP sizes enabled. Technically at 511 only the highest enabled order always gets collapsed.
Ive also argued in the past that 511 is a terrible default for anything other than thp.enabled=always, but that's a whole other can of worms we dont need to discuss now.
with this cap of 255, the PMD scan/collapse would work as intended, then in mTHP collapses we would never introduce this undesired behavior. We've discussed before that this would be a hard problem to solve without introducing some expensive way of tracking what has already been through a collapse, and that doesnt even consider what happens if things change or are unmapped, and rescanning that section would be helpful. So having a strictly enforced limit of 255 actually seems like a good idea to me, as it completely avoids the undesired behavior and does not require the admins to be aware of such an issue.
Another thought similar to what (IIRC) Dev has mentioned before, if we have
max_ptes_none> 255 then we only consider collapses to the largest enabled order, that way no creep to the largest enabled order would occur in the first place, and we would get there straight away.To me one of these two solutions seem sane in the context of what we are dealing with.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 6 lines
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
I agree that the current tunable wasn't designed for this, but we tried to come up with something that leverages the tunable we have to avoid new tunables and added complexity.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
What happens if
max_ptes_none= 0 and the ratio is 50% - 1 pte (ideally the max number)? seems like we would be saying we want no new none pages, but also to allow new none pages. To me that seems equally broken and more confusing than just taking a scale of the current number (now with a cap).The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
I spoke to David off-list and he gave some insight into this and perhaps some reasonable means of avoiding an additional tunable.
I don't want to rehash what he said as I think it's more productive for him to reply when he has time but broadly I think how we handle this needs careful consideration.
Quoted 6 lines
To me it's clear that some sense of ratio is just immediately very very confusing, but then again this interface is already confusing, as with much of THP.
Anyway I'll let David respond here so we don't loop around before he has a chance to add his input.
I've been summoned.
As raised in the past, I would initially only support specific values here like
0 : Never collapse with any pte_none/zeropage
511 (HPAGE_PMD_NR - 1) / default : Always collapse, ignoring pte_none/zeropage
Once could also easily support the value 255 (HPAGE_PMD_NR / 2- 1), but not sure if we have to add that for now.
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Signature · 3 lines
Cheers David / dhildenb
Quoted 12 lines
The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
I spoke to David off-list and he gave some insight into this and perhaps some reasonable means of avoiding an additional tunable.
I don't want to rehash what he said as I think it's more productive for him to reply when he has time but broadly I think how we handle this needs careful consideration.
Quoted 9 lines
To me it's clear that some sense of ratio is just immediately very very confusing, but then again this interface is already confusing, as with much of THP.
Anyway I'll let David respond here so we don't loop around before he has a chance to add his input.
Cheers, Lorenzo
[Resending because Thunderbird decided to use the wrong smtp server]
I've been summoned.
As raised in the past, I would initially only support specific values here like
0 : Never collapse with any pte_none/zeropage
511 (HPAGE_PMD_NR - 1) / default : Always collapse, ignoring pte_none/zeropage
Once could also easily support the value 255 (HPAGE_PMD_NR / 2- 1), but not sure if we have to add that for now.
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Signature · 3 lines
Cheers David / dhildenb
Quoted 19 lines · David Hildenbrand
Quoted 19 lines
The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
I spoke to David off-list and he gave some insight into this and perhaps some reasonable means of avoiding an additional tunable.
I don't want to rehash what he said as I think it's more productive for him to reply when he has time but broadly I think how we handle this needs careful consideration.
To me it's clear that some sense of ratio is just immediately very very confusing, but then again this interface is already confusing, as with much of THP.
Anyway I'll let David respond here so we don't loop around before he has a chance to add his input.
Quoted 6 lines
Quoted 2 lines
Cheers, Lorenzo
[Resending because Thunderbird decided to use the wrong smtp server]
I've been summoned.
Welcome :)
Quoted 6 lines
As raised in the past, I would initially only support specific values here like
0 : Never collapse with any pte_none/zeropage
511 (HPAGE_PMD_NR- 1) / default : Always collapse, ignoring pte_none/zeropage
OK so if had effectively an off/on (I guess we have to keep this as it is for legay purposes) and is forced to one or other of these values then fine (as long as we don't have uAPI worries).
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.
Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
Quoted 4 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 6 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Quoted 6 lines
--
CheersDavid / dhildenb
Cheers, Lorenzo
Quoted 9 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 5 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 10 lines
Quoted 5 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Signature · 3 lines
Cheers David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
Quoted 5 lines
Quoted 2 lines
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Yeah I'm fine with 0/511.
Quoted 7 lines
Quoted 6 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
Quoted 5 lines
I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
Feels like we should be looking at this in reverse? What's the largest, then next largest, then etc.?
Surely this is the sensible way of doing it?
Quoted 3 lines
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Yes.
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 5 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
Ack
Quoted 4 lines
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Or we could make it an error to set anything but 0, 511, but on the other hand that's likely to break userspace so yeah probably not.
Maybe have a warning saying 'this is no longer supported and will be ignored' then set the value to 0 for anything but 511 or 0.
Then can remove the warning later.
By having 0/511 we can really simplify the 'scaling' logic too which would be fantastic! :)
Cheers, Lorenzo
Dev Jain
reviewerQuoted 19 lines · Lorenzo Stoakes
Quoted 10 lines · David Hildenbrand
Quoted 8 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Yeah I'm fine with 0/511.
Quoted 6 lines
Quoted 4 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
I think the creep is unavoidable for certain values.
Quoted 7 lines
Quoted 3 lines
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
Feels like we should be looking at this in reverse? What's the largest, then next largest, then etc.?
Surely this is the sensible way of doing it?
What David means to say is, for example, suppose all orders are enabled, and we fail to collapse for order-9, then order-8, then order-7, and so on, *only* because the distribution of ptes did not obey the scaled max_ptes_none. Let order-4 collapse succeed.
Next time, khugepaged comes and tries for order-9, fails, then order-8, fails and so on. Then it checks for order-5, and it comes under the scaled max_ptes_none constraint only because the previous cycle's order-4 collapse changed the ptes' distribution.
Quoted 28 lines
Quoted 3 lines
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Yes.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 4 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");Quoted 6 lines
Quoted 3 lines
Quoted 1 line
for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
Ack
Quoted 3 lines
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Or we could make it an error to set anything but 0, 511, but on the other hand that's likely to break userspace so yeah probably not.
Quoted 7 lines
Maybe have a warning saying 'this is no longer supported and will be ignored' then set the value to 0 for anything but 511 or 0.
Then can remove the warning later.
By having 0/511 we can really simplify the 'scaling' logic too which would be fantastic! :)
FWIW here was my implementation of this thing, for ease of everyone:
https://lore.kernel.org/all/[email protected]/
Quoted 2 lines
Cheers, Lorenzo
Quoted 27 lines · Dev Jain
Quoted 25 lines · Lorenzo Stoakes
Quoted 10 lines · David Hildenbrand
Quoted 8 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Yeah I'm fine with 0/511.
Quoted 9 lines
Quoted 4 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
Feels like we should be looking at this in reverse? What's the largest, then next largest, then etc.?
Quoted 6 lines
Quoted 1 line
Surely this is the sensible way of doing it?
What David means to say is, for example, suppose all orders are enabled, and we fail to collapse for order-9, then order-8, then order-7, and so on, *only* because the distribution of ptes did not obey the scaled
max_ptes_none. Let order-4 collapse succeed.
Ah so it is the overhead of this that's the problem?
All roads lead to David's suggestion imo.
Quoted 5 lines
Quoted 2 lines
By having 0/511 we can really simplify the 'scaling' logic too which would be fantastic! :)
FWIW here was my implementation of this thing, for ease of everyone:
https://lore.kernel.org/all/[email protected]/
That's fine, but I really think we should just replace all this stuff with a boolean, and change the interface to max_ptes to set boolean if 511, or clear if 0.
Cheers, Lorenzo
Baolin Wang
reviewer(Sorry for chiming in late)
Quoted 18 lines · David Hildenbrand
Quoted 9 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 5 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
Quoted 6 lines
I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
In other words, for the scenario you described, although there are only 2 PTEs present in a PMD, it would still get collapsed into a PMD-sized THP. In reality, what we probably need is just an order-2 mTHP collapse.
If 'khugepaged_max_ptes_none' is set to 255, I think this would achieve the desired result: when there are only 2 PTEs present in a PMD, an order-2 mTHP collapse would be successed, but it wouldn’t creep up to an order-3 mTHP collapse. That’s because:
When attempting an order-3 mTHP collapse, 'threshold_bits' = 1, while 'bits_set' = 1 (means only 1 chunk is present), so 'bits_set > threshold_bits' is false, then an order-3 mTHP collapse wouldn’t be attempted. No?
So I have some concerns that if we only allow setting 0 or 511, it may not meet the goal we have for mTHP collapsing.
Quoted 20 lines
Quoted 8 lines
Quoted 3 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Dev Jain
reviewerQuoted 21 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 18 lines · David Hildenbrand
Quoted 9 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 5 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
Quoted 8 lines
Quoted 6 lines
I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Quoted 29 lines
In other words, for the scenario you described, although there are only 2 PTEs present in a PMD, it would still get collapsed into a PMD-sized THP. In reality, what we probably need is just an order-2 mTHP collapse.
If '
khugepaged_max_ptes_none' is set to 255, I think this would achieve the desired result: when there are only 2 PTEs present in a PMD, an order-2 mTHP collapse would be successed, but it wouldn’t creep up to an order-3 mTHP collapse. That’s because:
When attempting an order-3 mTHP collapse, 'threshold_bits' = 1, while 'bits_set' = 1 (means only 1 chunk is present), so 'bits_set>threshold_bits' is false, then an order-3 mTHP collapse wouldn’t be attempted. No?So I have some concerns that if we only allow setting 0 or 511, it may not meet the goal we have for mTHP collapsing.
Quoted 20 lines
Quoted 8 lines
Quoted 3 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Baolin Wang
reviewerQuoted 25 lines · Dev Jain
Quoted 23 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 20 lines · David Hildenbrand
Quoted 9 lines
Quoted 2 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you might add it to the docs, but people are going to be mightily confused, esp if it's a calculated value.
I don't see any other way around having a separate tunable if we don't just have something VERY simple like on/off.
Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 5 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and happy to stand corrected if this is somehow inherent, but reallly feels that way).
I think the creep is unavoidable for certain values.
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a order-2 folio, then and order-3 ... until you reached PMD order.
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
So we might also need to consider whether 255 is a reasonable configuration for mTHP collapse.
Quoted 28 lines
Quoted 28 lines
In other words, for the scenario you described, although there are only 2 PTEs present in a PMD, it would still get collapsed into a PMD- sized THP. In reality, what we probably need is just an order-2 mTHP collapse.
If '
khugepaged_max_ptes_none' is set to 255, I think this would achieve the desired result: when there are only 2 PTEs present in a PMD, an order-2 mTHP collapse would be successed, but it wouldn’t creep up to an order-3 mTHP collapse. That’s because:
When attempting an order-3 mTHP collapse, 'threshold_bits' = 1, while 'bits_set' = 1 (means only 1 chunk is present), so 'bits_set>threshold_bits' is false, then an order-3 mTHP collapse wouldn’t be attempted. No?So I have some concerns that if we only allow setting 0 or 511, it may not meet the goal we have for mTHP collapsing.
Quoted 20 lines
Quoted 8 lines
Quoted 3 lines
Because, as raised in the past, I'm afraid nobody on this earth has a clue how to set this parameter to values different to 0 (don't waste memory with khugepaged) and 511 (page fault behavior).
Yup
Quoted 2 lines
Quoted 7 lines
Quoted 4 lines
If any other value is set, essentially
pr_warn("Unsupported 'max_ptes_none' value for mTHP collapse");for now and just disable it.
Hmm but under what circumstances? I would just say unsupported value not mention mTHP or people who don't use mTHP might find that confusing.
Well, we can check whether any mTHP size is enabled while the value is set to something unexpected. We can then even print the problematic sizes if we have to.
We could also just just say that if the value is set to something else than 511 (which is the default), it will be treated as being "0" when collapsing mthp, instead of doing any scaling.
Quoted 49 lines · Baolin Wang
Quoted 46 lines · Dev Jain
Quoted 44 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 28 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
Quoted 8 lines
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
Quoted 8 lines
Quoted 3 lines
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
How do these users control allocation at fault time where this parameter is completely ignored?
Signature · 3 lines
Cheers David / dhildenb
Baolin Wang
reviewerQuoted 54 lines · David Hildenbrand
Quoted 53 lines · Baolin Wang
Quoted 49 lines · Dev Jain
Quoted 36 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 33 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.Quoted 8 lines
Quoted 3 lines
So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Quoted 6 lines
Quoted 4 lines
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the 'max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.
The comment over khugepaged_max_ptes_none gives a hint:
/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */
In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some MADV_DONTNEED and wants to control the usage of THPs itself using MADV_COLLAPSE.
It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would MADV_DONTNEED (resulting in
a page table with at least one pte_none entry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.
(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.
But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of pte_nones around the fault area and install an mTHP.
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.
Signature · 3 lines
Cheers David / dhildenb
Quoted 105 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.The comment over
khugepaged_max_ptes_nonegives a hint:/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some
MADV_DONTNEEDand wants to control the usage of THPs itself usingMADV_COLLAPSE.It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would
MADV_DONTNEED(resulting in
a page table with at least onepte_noneentry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of
pte_nonesaround the fault area and install an mTHP.
Quoted 9 lines
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.
For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than HPAGE_PMD_NR -1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory. I have seen in our prod workloads where the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with max_ptes_none = 511, while still still keeping the benefits of THPs like lower TLB misses.
I do agree that the value of max_ptes_none is magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I use max_ptes_none = 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.
Quoted 118 lines · Usama Arif
Quoted 105 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.The comment over
khugepaged_max_ptes_nonegives a hint:/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some
MADV_DONTNEEDand wants to control the usage of THPs itself usingMADV_COLLAPSE.It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would
MADV_DONTNEED(resulting in
a page table with at least onepte_noneentry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of
pte_nonesaround the fault area and install an mTHP.Quoted 9 lines
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.
Quoted 3 lines
For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.
You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.
Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and max_ptes_none includes zero pages.
Quoted 4 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.
To which value would you set it? Just 510? 0?
Signature · 3 lines
Cheers David / dhildenb
Quoted 123 lines · David Hildenbrand
Quoted 118 lines · Usama Arif
Quoted 105 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.The comment over
khugepaged_max_ptes_nonegives a hint:/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some
MADV_DONTNEEDand wants to control the usage of THPs itself usingMADV_COLLAPSE.It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would
MADV_DONTNEED(resulting in
a page table with at least onepte_noneentry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of
pte_nonesaround the fault area and install an mTHP.Quoted 9 lines
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.Quoted 3 lines
For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.
Quoted 9 lines
You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on max_ptes_none to prevent one of these things thrashing the effect of the other.
Quoted 10 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 4 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
In terms of wider rollout across the fleet, we are going to target 0 (or a very very small value) when moving from THP=madvise to always. Mainly because it is the least likely to cause a memory regression as THP shrinker will deal with page faults faulting in mostly zero-filled pages and khugepaged wont collapse pages that are dominated by 4K zero-filled chunks.
Baolin Wang
reviewerQuoted 112 lines · Usama Arif
Quoted 109 lines · David Hildenbrand
Quoted 108 lines · Usama Arif
Quoted 105 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.The comment over
khugepaged_max_ptes_nonegives a hint:/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some
MADV_DONTNEEDand wants to control the usage of THPs itself usingMADV_COLLAPSE.It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would
MADV_DONTNEED(resulting in
a page table with at least onepte_noneentry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of
pte_nonesaround the fault area and install an mTHP.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.
Thanks David for your explanation. I see your point now.
Quoted 33 lines
Quoted 4 lines
Quoted 3 lines
For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.Quoted 9 lines
You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.Quoted 10 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 4 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
Quoted 4 lines
In terms of wider rollout across the fleet, we are going to target 0 (or a very very small value) when moving from THP=madvise to always. Mainly because it is the least likely to cause a memory regression as THP shrinker will deal with page faults faulting in mostly zero-filled pages and khugepaged wont collapse pages that are dominated by 4K zero-filled chunks.
Thanks for sharing this. We're also investigating what max_ptes_none should be set to in order to use the THP shrinker properly, and currently, our customers always set max_ptes_none to its default value: 511, which is not good.
If 0 is better, it seems like there isn't much conflict with the values expected by mTHP collapse (0 and 511). Sounds good to me.
Quoted 130 lines · Usama Arif
Quoted 126 lines · David Hildenbrand
Quoted 118 lines · Usama Arif
Quoted 105 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Baolin Wang
Quoted 42 lines · Dev Jain
Quoted 40 lines · Baolin Wang
(Sorry for chiming in late)
Quoted 36 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Once could also easily support the value 255 (
HPAGE_PMD_NR/ 2- 1), but not sure
if we have to add that for now.Yeah not so sure about this, this is a 'just have to know' too, and yes you
might add it to the docs, but people are going to be mightily confused, esp if
it's a calculated value.Quoted 3 lines
I don't see any other way around having a separate tunable if we don't just have
something VERY simple like on/off.Yeah, not advocating that we add support for other values than 0/511, really.
Quoted 8 lines
Also the mentioned issue sounds like something that needs to be fixed elsewhere
honestly in the algorithm used to figure out mTHP ranges (I may be wrong - and
happy to stand corrected if this is somehow inherent, but reallly feels that
way).I think the creep is unavoidable for certain values.
If you have the first two pages of a PMD area populated, and you allow for at least half of the #PTEs to be non/zero, you'd collapse first a
order-2 folio, then and order-3 ... until you reached PMD order.So for now we really should just support 0 / 511 to say "don't collapse if there are holes" vs. "always collapse if there is at least one pte used".
Quoted 7 lines
Quoted 4 lines
If we only allow setting 0 or 511, as Nico mentioned before, "At 511, no mTHP collapses would ever occur anyway, unless you have 2MB disabled and other mTHP sizes enabled. Technically, at 511, only the highest enabled order would ever be collapsed."
I didn't understand this statement. At 511, mTHP collapses will occur if khugepaged cannot get a PMD folio. Our goal is to collapse to the highest order folio.
Yes, I’m not saying that it’s incorrect behavior when set to 511. What I mean is, as in the example I gave below, users may only want to allow a large order collapse when the number of present PTEs reaches half of the large folio, in order to avoid RSS bloat.
Quoted 5 lines
Quoted 2 lines
How do these users control allocation at fault time where this parameter is completely ignored?
Sorry, I did not get your point. Why does the '
max_pte_none' need to control allocation at fault time? Could you be more specific? Thanks.The comment over
khugepaged_max_ptes_nonegives a hint:/*
* default collapse hugepages if there is at least one pte mapped like
* it would have happened if the vma was large enough during page
* fault.
*
* Note that these are only respected if collapse was initiated by khugepaged. */In the common case (for anything that really cares about RSS bloat) you will just a get a THP during page fault and consequently RSS bloat.
As raised in my other reply, the only documented reason to set max_ptes_none=0 seems to be when an application later (after once possibly getting a THP already during page faults) did some
MADV_DONTNEEDand wants to control the usage of THPs itself usingMADV_COLLAPSE.It's a questionable use case, that already got more problematic with mTHP and page table reclaim.
Let me explain:
Before mTHP, if someone would
MADV_DONTNEED(resulting in
a page table with at least onepte_noneentry), there would have been no way we would get memory over-allocated afterwards with max_ptes_none=0.(1) Page faults would spot "there is a page table" and just fallback to order-0 pages.
(2) khugepaged was told to not collapse through max_ptes_none=0.But now:
(A) With mTHP during page-faults, we can just end up over-allocating memory in such an area again: page faults will simply spot a bunch of
pte_nonesaround the fault area and install an mTHP.Quoted 9 lines
(B) With page table reclaim (when zapping all PTEs in a table at once), we will reclaim the page table. The next page fault will just try installing a PMD THP again, because there is no PTE table anymore.
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.Quoted 3 lines
For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.
I believe with mTHP support in khugepaged, the max_ptes_none value in the shrinker must also leverage the 'order' scaling to properly prevent thrashing.
I've been testing a patch for this that I might include in the V11.
Quoted 23 lines
Quoted 11 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 4 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
In terms of wider rollout across the fleet, we are going to target 0 (or a very very small value) when moving from THP=madvise to always. Mainly because it is the least likely to cause a memory regression as THP shrinker will deal with page faults faulting in mostly zero-filled pages and khugepaged wont collapse pages that are dominated by 4K zero-filled chunks.
Quoted 18 lines · Nico Pache
Quoted 17 lines · Usama Arif
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.I believe with mTHP support in khugepaged, the
max_ptes_nonevalue in the shrinker must also leverage the 'order' scaling to properly prevent thrashing.
No please do not extend this 'scalling' stuff somewhere else, it's really horrid.
We have to find an alternative to that, it's extremely confusing in what is already extremely confusing THP code.
As I said before, if we can't have a boolean we need another interface, which makes most sense to be a ratio or in practice, a percentage sysctl.
Speaking with David off-list, maybe the answer - if we must have this - is to add a new percentage interface and have this in lock-step with the existing max_ptes_none interface. One updates the other, but internally we're just using the percentage value.
Quoted 18 lines
I've been testing a patch for this that I might include in the V11.
Quoted 17 lines
Quoted 11 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 4 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
(->Usama) It's a pity that such workloads exist. But then the percentage solution should work.
Quoted 6 lines
Quoted 6 lines
In terms of wider rollout across the fleet, we are going to target 0 (or a very very small value) when moving from THP=madvise to always. Mainly because it is the least likely to cause a memory regression as THP shrinker will deal with page faults faulting in mostly zero-filled pages and khugepaged wont collapse pages that are dominated by 4K zero-filled chunks.
(->Usama) Interesting though that you've decided against doing this fleetwide... I wonder then again whether we truly need non-boolean values.
But the fact workloads might theoretically exist where it's useful does make me think we have to have this, sadly.
Cheers, Lorenzo
Quoted 32 lines · Lorenzo Stoakes
Quoted 18 lines · Nico Pache
Quoted 17 lines · Usama Arif
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.I believe with mTHP support in khugepaged, the
max_ptes_nonevalue in the shrinker must also leverage the 'order' scaling to properly prevent thrashing.No please do not extend this 'scalling' stuff somewhere else, it's really horrid.
We have to find an alternative to that, it's extremely confusing in what is already extremely confusing THP code.
Quoted 7 lines
As I said before, if we can't have a boolean we need another interface, which makes most sense to be a ratio or in practice, a percentage sysctl.
Speaking with David off-list, maybe the answer - if we must have this - is to add a new percentage interface and have this in lock-step with the existing
max_ptes_noneinterface. One updates the other, but internally we're just using the percentage value.
Yes, I'll try hacking something up and sending it as an RFC.
Quoted 7 lines
Quoted 7 lines
I've been testing a patch for this that I might include in the V11.
Quoted 5 lines
Quoted 5 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
Sorry, I missed Usama's reply. Thanks Usama!
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
(->Usama) It's a pity that such workloads exist. But then the percentage solution should work.
Good. So if there is no strong case for > 255, that's already valuable for mTHP.
Signature · 3 lines
Cheers David / dhildenb
Quoted 66 lines · David Hildenbrand
Quoted 32 lines · Lorenzo Stoakes
Quoted 18 lines · Nico Pache
Quoted 17 lines · Usama Arif
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.I believe with mTHP support in khugepaged, the
max_ptes_nonevalue in the shrinker must also leverage the 'order' scaling to properly prevent thrashing.No please do not extend this 'scalling' stuff somewhere else, it's really horrid.
We have to find an alternative to that, it's extremely confusing in what is already extremely confusing THP code.
Quoted 7 lines
As I said before, if we can't have a boolean we need another interface, which makes most sense to be a ratio or in practice, a percentage sysctl.
Speaking with David off-list, maybe the answer - if we must have this - is to add a new percentage interface and have this in lock-step with the existing
max_ptes_noneinterface. One updates the other, but internally we're just using the percentage value.Yes, I'll try hacking something up and sending it as an RFC.
Quoted 7 lines
Quoted 7 lines
I've been testing a patch for this that I might include in the V11.
Quoted 5 lines
Quoted 5 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
Sorry, I missed Usama's reply. Thanks Usama!
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
Quoted 4 lines
Quoted 1 line
(->Usama) It's a pity that such workloads exist. But then the percentage solution should work.
Good. So if there is no strong case for > 255, that's already valuable for mTHP.
tbh the default value of 511 is horrible. I have thought about sending a patch to change it to 0 as default in upstream for sometime, but it might mean that people who upgrade their kernel might suddenly see their memory not getting hugified and it could be confusing for them?
Quoted 69 lines · Usama Arif
Quoted 66 lines · David Hildenbrand
Quoted 32 lines · Lorenzo Stoakes
Quoted 18 lines · Nico Pache
Quoted 17 lines · Usama Arif
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
So I question the utility of
max_ptes_none. If you can't tame page faults, then there is only limited sense in taming khugepaged. I think there is vale in setting max_ptes_none=0 for some corner cases, but I am yet to learn why max_ptes_none=123 would make any sense.For PMD mapped THPs with THP shrinker, this has changed. You can basically tame pagefaults, as when you encounter memory pressure, the shrinker kicks in if the value is less than
HPAGE_PMD_NR-1 (i.e. 511 for x86), and will break down those hugepages and free up zero-filled memory.You are not really taming page faults, though, you are undoing what page faults might have messed up :)
I have seen in our prod workloads where
Quoted 3 lines
the memory usage and THP usage can spike (usually when the workload starts), but with memory pressure, the memory usage is lower compared to with
max_ptes_none= 511, while still still keeping the benefits of THPs like lower TLB misses.Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Thanks for raising that: I think the current behavior is in place such that you don't bounce back-and-forth between khugepaged collapse and shrinker-split.
Yes, both collapse and shrinker split hinge on
max_ptes_noneto prevent one of these things thrashing the effect of the other.I believe with mTHP support in khugepaged, the
max_ptes_nonevalue in the shrinker must also leverage the 'order' scaling to properly prevent thrashing.No please do not extend this 'scalling' stuff somewhere else, it's really horrid.
We have to find an alternative to that, it's extremely confusing in what is already extremely confusing THP code.
Quoted 7 lines
As I said before, if we can't have a boolean we need another interface, which makes most sense to be a ratio or in practice, a percentage sysctl.
Speaking with David off-list, maybe the answer - if we must have this - is to add a new percentage interface and have this in lock-step with the existing
max_ptes_noneinterface. One updates the other, but internally we're just using the percentage value.Yes, I'll try hacking something up and sending it as an RFC.
Quoted 7 lines
Quoted 7 lines
I've been testing a patch for this that I might include in the V11.
Quoted 5 lines
Quoted 5 lines
There are likely other ways to achieve that, when we have in mind that the thp shrinker will install zero pages and
max_ptes_noneincludes zero pages.Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
I do agree that the value of
max_ptes_noneis magical and different workloads can react very differently to it. The relationship is definitely not linear. i.e. if I usemax_ptes_none= 256, it does not mean that the memory regression of using THP=always vs THP=madvise is halved.To which value would you set it? Just 510? 0?
Sorry, I missed Usama's reply. Thanks Usama!
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
There are some very large workloads in the meta fleet that I experimented with and found that having a small value works out. I experimented with 0, 51 (10%) and 256 (50%). 51 was found to be an optimal comprimise in terms of application metrics improving, having an acceptable amount of memory regression and improved system level metrics (lower TLB misses, lower page faults). I am sure there was a better value out there for these workloads, but not possible to experiment with every value.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
(->Usama) It's a pity that such workloads exist. But then the percentage solution should work.
Good. So if there is no strong case for > 255, that's already valuable for mTHP.
tbh the default value of 511 is horrible. I have thought about sending a patch to change it to 0 as default in upstream for sometime, but it might mean that people who upgrade their kernel might suddenly see their memory not getting hugified and it could be confusing for them?
511 is just what a page fault would have done, so I think it makes perfect sense. More than anything else, actually.
It's just not optimal in many cases.
Signature · 3 lines
Cheers David / dhildenb
Quoted 71 lines · Lorenzo Stoakes
Quoted 59 lines · Nico Pache
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
We wouldn't be changing it for PMD collapse, just for the new behavior. At 511, no mTHP collapses would ever occur anyways, unless you have 2MB disabled and other mTHP sizes enabled. Technically at 511 only the highest enabled order always gets collapsed.
Ive also argued in the past that 511 is a terrible default for anything other than thp.enabled=always, but that's a whole other can of worms we dont need to discuss now.
with this cap of 255, the PMD scan/collapse would work as intended, then in mTHP collapses we would never introduce this undesired behavior. We've discussed before that this would be a hard problem to solve without introducing some expensive way of tracking what has already been through a collapse, and that doesnt even consider what happens if things change or are unmapped, and rescanning that section would be helpful. So having a strictly enforced limit of 255 actually seems like a good idea to me, as it completely avoids the undesired behavior and does not require the admins to be aware of such an issue.
Another thought similar to what (IIRC) Dev has mentioned before, if we have
max_ptes_none> 255 then we only consider collapses to the largest enabled order, that way no creep to the largest enabled order would occur in the first place, and we would get there straight away.To me one of these two solutions seem sane in the context of what we are dealing with.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 6 lines
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
I agree that the current tunable wasn't designed for this, but we tried to come up with something that leverages the tunable we have to avoid new tunables and added complexity.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
What happens if
max_ptes_none= 0 and the ratio is 50% - 1 pte (ideally the max number)? seems like we would be saying we want no new none pages, but also to allow new none pages. To me that seems equally broken and more confusing than just taking a scale of the current number (now with a cap).
Quoted 3 lines
The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
Ok I've put a lot of thought and time into this and came up with a solution.
Here is what I currently have tested and would like to proposing:
- Expand bitmap to HPAGE_PMD_NR (512)*, this increases the accuracy of the max_pte_none handling, and removes a lot of inaccuracies caused by the compression into 128 bits that was being done. This also makes the code a lot easier to understand.
- When attempting mTHP level collapses cap max_ptes_none to 255 to prevent the creep issue
Ive tested this and found this performs better than my previous version, allows for more granular control via max_ptes_none, and prevents the creep issue without any admin knowledge needed.
I think this is a good middle ground between completely disabling the fine tune control, and doing a better job at mitigating misconfiguration.
**Baolin actually also expands the bitmap to 512 in his khugepaged collapse file mTHP support patchset
Does this sound reasonable to you?
-- Nico
Quoted 17 lines
I spoke to David off-list and he gave some insight into this and perhaps some reasonable means of avoiding an additional tunable.
I don't want to rehash what he said as I think it's more productive for him to reply when he has time but broadly I think how we handle this needs careful consideration.
To me it's clear that some sense of ratio is just immediately very very confusing, but then again this interface is already confusing, as with much of THP.
Anyway I'll let David respond here so we don't loop around before he has a chance to add his input.
Cheers, Lorenzo
Quoted 76 lines · Nico Pache
Quoted 71 lines · Lorenzo Stoakes
Quoted 59 lines · Nico Pache
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks and I"ll have a look, but this series is unmergeable with a broken default in
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratiosorry.We need to have a new tunable as far as I can tell. I also find the use of this PMD-specific value as an arbitrary way of expressing a ratio pretty gross.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
The first thing that comes to mind is that we can pin
max_ptes_noneto 255 if it exceeds 255. It's worth noting that the issue occurs only for adjacently enabled mTHP sizes.No! Presumably the default of 511 (for PMDs with 512 entries) is set for a reason, arbitrarily changing this to suit a specific case seems crazy no?
We wouldn't be changing it for PMD collapse, just for the new behavior. At 511, no mTHP collapses would ever occur anyways, unless you have 2MB disabled and other mTHP sizes enabled. Technically at 511 only the highest enabled order always gets collapsed.
Ive also argued in the past that 511 is a terrible default for anything other than thp.enabled=always, but that's a whole other can of worms we dont need to discuss now.
with this cap of 255, the PMD scan/collapse would work as intended, then in mTHP collapses we would never introduce this undesired behavior. We've discussed before that this would be a hard problem to solve without introducing some expensive way of tracking what has already been through a collapse, and that doesnt even consider what happens if things change or are unmapped, and rescanning that section would be helpful. So having a strictly enforced limit of 255 actually seems like a good idea to me, as it completely avoids the undesired behavior and does not require the admins to be aware of such an issue.
Another thought similar to what (IIRC) Dev has mentioned before, if we have
max_ptes_none> 255 then we only consider collapses to the largest enabled order, that way no creep to the largest enabled order would occur in the first place, and we would get there straight away.To me one of these two solutions seem sane in the context of what we are dealing with.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
ie)
if order!=HPAGE_PMD_ORDER &&khugepaged_max_ptes_none> 255temp_max_ptes_none = 255;Oh and my second point, introducing a new tunable to control mTHP collapse may become exceedingly complex from a tuning and code management standpoint.
Quoted 6 lines
Umm right now you hve a ratio expressed in PTES per mTHP * ((PTEs per PMD) / PMD) 'except please don't set to the usual default when using mTHP' and it's currently default-broken.
I'm really not sure how that is simpler than a seprate tunable that can be expressed as a ratio (e.g. percentage) that actually makes some kind of sense?
I agree that the current tunable wasn't designed for this, but we tried to come up with something that leverages the tunable we have to avoid new tunables and added complexity.
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
And we can make anything workable from a code management point of view by refactoring/developing appropriately.
What happens if
max_ptes_none= 0 and the ratio is 50% - 1 pte (ideally the max number)? seems like we would be saying we want no new none pages, but also to allow new none pages. To me that seems equally broken and more confusing than just taking a scale of the current number (now with a cap).Quoted 3 lines
The one thing we absolutely cannot have is a default that causes this 'creeping' behaviour. This feels like shipping something that is broken and alluding to it in the documentation.
Quoted 8 lines
Ok I've put a lot of thought and time into this and came up with a solution.
Here is what I currently have tested and would like to proposing:
- Expand bitmap to
HPAGE_PMD_NR(512)*, this increases the accuracy of themax_pte_nonehandling, and removes a lot of inaccuracies caused by the compression into 128 bits that was being done. This also makes the code a lot easier to understand.
That sounds good to me. Should make the code easier as well.
Quoted 3 lines
- When attempting mTHP level collapses cap
max_ptes_noneto 255 to prevent the creep issue
I guess the documentation would then state something like
* When collapsing smaller THPs, "max_ptes_none" is scaled proportional to the THP size.
* When collapsing smaller THPs, "max_ptes_none" may be internally capped at 255 if it exceeds 255 but is not set to the default (511).
Not 100% a fan of all of that, but maybe the only option when wanting to avoid other toggles.
The only alternative would really be respecting only 0/511 for mTHP, and not doing any scaling. That would obviously make the documentation easier and would allow us to revisit that later. The documentation would be:
* When collapsing smaller THPs, "max_ptes_none" may be interpreted as "0" when set to a value different to the default (511). This behavior might change in the future.
Quoted 4 lines
Ive tested this and found this performs better than my previous version, allows for more granular control via
max_ptes_none, and prevents the creep issue without any admin knowledge needed.
How would this interact with the shrinker once extended to mTHP? Would your RFC patch be sufficient for that or would we actually also want to cap? I haven't; fully thought this through yet. I'd assume we would not want to cap here. Which makes the doc weird as well, lol.
Signature · 3 lines
Cheers David / dhildenb
Quoted 44 lines · Dev Jain
Quoted 27 lines · Lorenzo Stoakes
OK so I noticed in patch 13/13 (!) where you change the documentation that you essentially state that the whole method used to determine the ratio of PTEs to collapse to mTHP is broken:
khugepaged uses
max_ptes_nonescaled to the order of the enabled mTHP size to determine collapses. When using mTHPs it's recommended to setmax_ptes_nonelow-- ideally less thanHPAGE_PMD_NR/ 2 (255 on 4k page size). This will prevent undesired "creep" behavior that leads to continuously collapsing to the largest mTHP size; when we collapse, we are bringing in new non-zero pages that will, on a subsequent scan, cause themax_ptes_nonecheck of the +1 order to always be satisfied. By limiting this to less than half the current order, we make sure we don't cause this feedback
loop.max_ptes_sharedandmax_ptes_swaphave no effect when collapsing to a mTHP, and mTHP collapse will fail on shared or swapped out pages.This seems to me to suggest that using
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_noneas some means of establishing a 'ratio' to do this calculation is fundamentally flawed.So surely we ought to introduce a new sysfs tunable for this? Perhaps
/sys/kernel/mm/transparent_hugepage/khugepaged/mthp_max_ptes_none_ratioOr something like this?
Quoted 8 lines
It's already questionable that we are taking a value that is expressed essentially in terms of PTE entries per PMD and then use it implicitly to determine the ratio for mTHP, but to then say 'oh but the default value is known-broken' is just a blocker for the series in my opinion.
This really has to be done a different way I think.
Cheers, Lorenzo
FWIW this was my version of the documentation patch:
https://lore.kernel.org/all/[email protected]/The discussion about the creep problem started here:
https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
and the discussion continuing here:
https://lore.kernel.org/all/[email protected]/ending with a summary I gave here:
https://lore.kernel.org/all/[email protected]/This should help you with the context.
Thanks for hunting this down, the context should be referenced in the change log so we can find it easier in the future (and now). Or at least in the cover letter.
The way the change log in the cover letter is written makes it exceedingly long. Could you switch to listing the changes from v9 and links to v1-8 (+RFCs if there are any)? Well, I guess include v10 changes and v1-9 urls..
At the length it is now, it's most likely a tl;dr for most. If you're starting to review this at v10, then you'd probably appreciate not rehashing discussions and if you're going from v9 then you already have an idea of what v10 should have changed.
Said another way, the changelog is more useful with context and context is difficult to find without a lore link.
I am having issues tracking down the contexts of many items of what has been generated here and it'll only get worse as time moves on. We do our best to keep change logs with the necessary details, but having bread crumbs to follow is extremely helpful for review and in the long run.
Thanks,
Liam
Quoted 24 lines · Nico Pache
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
To achieve this we generalize the khugepaged functions to no longer depend on
PMD_ORDER. Then during the PMD scan, we use a bitmap to track chunks of pages (defined byKHUGEPAGED_MTHP_MIN_ORDER) that are utilized. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction onmax_ptes_noneis removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained.max_ptes_nonewill be scaled by the attempted collapse order to determine how full a mTHP must be to be eligible for the collapse to occur. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. It is now also possible to collapse to mTHPs without requiring the PMD THP size to be enabled.With the default max_ptes_none=511, the code should keep its most of its original behavior. When enabling multiple adjacent (m)THP sizes we need to set max_ptes_none<=255. With
max_ptes_none> HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. This is due the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.
Quoted 7 lines
Patch 1: Refactor/rename hpage_collapse Patch 2: Some refactoring to combine madvise_collapse and khugepaged Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-8: The mTHP patches Patch 9-10: Allow khugepaged to operate without PMD enabled Patch 11-12: Tracing/stats Patch 13: Documentation
Would it be feasible to start with simply not supporting the max_pte_none parameter in the first version, just like we won't support max_pte_swapped/max_pte_shared in the first version?
That gives us more time to think about how to use/modify the old interface.
For example, I could envision a ratio-based interface, or as discussed with Lorenzo a simple boolean. We could make the existing max_ptes* interface backwards compatible then.
That also gives us the opportunity to think about the creep problem separately.
I'm sure initial mTHP collapse will be valuable even without support for that weird set of parameters.
Would there be implementation-wise a problem?
But let me think further about the creep problem ... :/
Signature · 3 lines
Cheers David / dhildenb
Quoted 46 lines · David Hildenbrand
Quoted 24 lines · Nico Pache
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
To achieve this we generalize the khugepaged functions to no longer depend on
PMD_ORDER. Then during the PMD scan, we use a bitmap to track chunks of pages (defined byKHUGEPAGED_MTHP_MIN_ORDER) that are utilized. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction onmax_ptes_noneis removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained.max_ptes_nonewill be scaled by the attempted collapse order to determine how full a mTHP must be to be eligible for the collapse to occur. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. It is now also possible to collapse to mTHPs without requiring the PMD THP size to be enabled.With the default max_ptes_none=511, the code should keep its most of its original behavior. When enabling multiple adjacent (m)THP sizes we need to set max_ptes_none<=255. With
max_ptes_none> HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. This is due the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.Quoted 7 lines
Patch 1: Refactor/rename hpage_collapse Patch 2: Some refactoring to combine madvise_collapse and khugepaged Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-8: The mTHP patches Patch 9-10: Allow khugepaged to operate without PMD enabled Patch 11-12: Tracing/stats Patch 13: DocumentationWould it be feasible to start with simply not supporting the
max_pte_noneparameter in the first version, just like we won't support max_pte_swapped/max_pte_shared in the first version?That gives us more time to think about how to use/modify the old interface.
For example, I could envision a ratio-based interface, or as discussed with Lorenzo a simple boolean. We could make the existing max_ptes* interface backwards compatible then.
That also gives us the opportunity to think about the creep problem separately.
Quoted 6 lines
I'm sure initial mTHP collapse will be valuable even without support for that weird set of parameters.
Would there be implementation-wise a problem?
But let me think further about the creep problem ... :/
FWIW, I just looked around and there is documented usage of setting max_ptes_none to 0 [1, 2, 3].
In essence, I think it can make sense to set it to 0 when an application wants to manage THP on its own (MADV_COLLAPSE), and avoid khugepaged interfering. Now, using a system-wide toggle for such a use case is rather questionable, but it's all we have.
I did not find anything only recommending to set values different to 0 or 511 -- so far.
So *likely* focusing on 0 vs. 511 initially would cover most use cases out there. Ignoring the parameter initially (require all to be !none) could of course also work.
[1] https://www.mongodb.com/docs/manual/administration/tcmalloc-performance/
[2] https://google.github.io/tcmalloc/tuning.html
[3] https://support.yugabyte.com/hc/en-us/articles/36558155921165-Mitigating-Excessive-RSS-Memory-Usage-Due-to-THP-Transparent-Huge-Pages
Signature · 3 lines
Cheers David / dhildenb
Dev Jain
reviewerQuoted 24 lines · Nico Pache
The following series provides khugepaged with the capability to collapse anonymous memory regions to mTHPs.
To achieve this we generalize the khugepaged functions to no longer depend on
PMD_ORDER. Then during the PMD scan, we use a bitmap to track chunks of pages (defined byKHUGEPAGED_MTHP_MIN_ORDER) that are utilized. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction onmax_ptes_noneis removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained.max_ptes_nonewill be scaled by the attempted collapse order to determine how full a mTHP must be to be eligible for the collapse to occur. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we don't perform the collapse. It is now also possible to collapse to mTHPs without requiring the PMD THP size to be enabled.With the default max_ptes_none=511, the code should keep its most of its original behavior. When enabling multiple adjacent (m)THP sizes we need to set max_ptes_none<=255. With
max_ptes_none> HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. This is due the fact that a collapse will introduce at least 2x the number of pages, and on a future scan will satisfy the promotion condition once again.
Quoted 9 lines
Patch 1: Refactor/rename hpage_collapse Patch 2: Some refactoring to combine madvise_collapse and khugepaged Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-8: The mTHP patches Patch 9-10: Allow khugepaged to operate without PMD enabled Patch 11-12: Tracing/stats Patch 13: Documentation
For the next version, it will be really great if you can mention the lore links referencing important ideas guiding the evolution of the algorithm - say, a policy decision we make. (I frequently do that albeit I think I over-do it :)) I am asking because I am completely lost on the current discussion going on around the max_ptes_* scaling (been busy with other stuff).