mm: khugepaged cleanups and mTHP prerequisites
Subsystems (7)
v3 cover mm: khugepaged cleanups and mTHP prerequisites
The following series contains cleanups and prerequisites for my work on khugepaged mTHP support [1]. These have been separated out to ease review.
The first patch in the series refactors the page fault folio to pte mapping and follows a similar convention as defined by map_anon_folio_pmd_(no)pf(). This not only cleans up the current implementation of do_anonymous_page(), but will allow for reuse later in the khugepaged mTHP implementation.
The second patch adds a small is_pmd_order() helper to check if an order is the PMD order. This check is open-coded in a number of places. This patch aims to clean this up and will be used more in the khugepaged mTHP work. The third patch also adds a small DEFINE for (HPAGE_PMD_NR - 1) which is used often across the khugepaged code.
The fourth and fifth patch come from the khugepaged mTHP patchset [1]. These two patches include the rename of function prefixes, and the unification of khugepaged and madvise_collapse via a new collapse_single_pmd function.
Patch 1: refactor do_anonymous_page into map_anon_folio_pte_(no)pf Patch 2: add is_pmd_order helper Patch 3: Add define for (HPAGE_PMD_NR - 1) Patch 4: Refactor/rename hpage_collapse Patch 5: Refactoring to combine madvise_collapse and khugepaged
Testing:
- Built for x86_64, aarch64, ppc64le, and s390x
- ran all arches on test suites provided by the kernel-tests project
- selftests mm
V3 Changes:
- Patch1: leverage folio_order rather than passing nr_pages into map_anon_folio_pte_pf [2]
- Patch 2: fixed conflict with is_pmd_order
- Patch 3: Change COLLAPSE_MAX_PTES_LIMIT to KHUGEPAGED_MAX_PTES_LIMIT [3]
- Patch 4: conflict resolution with other khugepaged patches
- Patch 5: Drop lock_dropped unnecessary change, Dropped Lorenzos RB, !triggered_wb in if statement rather than triggered_wb, remove unnecessary mmap_locked flipping [4]
V2 - https://lore.kernel.org/all/[email protected]/
V1 - https://lore.kernel.org/lkml/[email protected]/
A big thanks to everyone that has reviewed, tested, and participated in the development process.
[1] - https://lore.kernel.org/all/[email protected]/
[2] - https://lore.kernel.org/all/[email protected]/
[3] - https://lore.kernel.org/all/[email protected]/
[4] - https://lore.kernel.org/all/[email protected]/
Nico Pache (5):
mm: consolidate anonymous folio PTE mapping into helpers
mm: introduce is_pmd_order helper
mm/khugepaged: define KHUGEPAGED_MAX_PTES_LIMIT as HPAGE_PMD_NR - 1 mm/khugepaged: rename hpage_collapse_* to collapse_* mm/khugepaged: unify khugepaged and madv_collapse with
collapse_single_pmd()
include/linux/huge_mm.h | 5 ++ include/linux/mm.h | 4 + mm/huge_memory.c | 2 +- mm/khugepaged.c | 187 +++++++++++++++++++++------------------- mm/memory.c | 62 ++++++++----- mm/mempolicy.c | 2 +- mm/mremap.c | 2 +- mm/page_alloc.c | 4 +- mm/shmem.c | 3 +- 9 files changed, 153 insertions(+), 118 deletions(-)
Signature · 1 line
2.53.0
PATCH v3 1/5 mm: consolidate anonymous folio PTE mapping into helpers
The anonymous page fault handler in do_anonymous_page() open-codes the sequence to map a newly allocated anonymous folio at the PTE level:
- construct the PTE entry
- add rmap
- add to LRU
- set the PTEs
- update the MMU cache.
Introduce a two helpers to consolidate this duplicated logic, mirroring the existing map_anon_folio_pmd_nopf() pattern for PMD-level mappings:
map_anon_folio_pte_nopf(): constructs the PTE entry, takes folio references, adds anon rmap and LRU. This function also handles the uffd_wp that can occur in the pf variant.
map_anon_folio_pte_pf(): extends the nopf variant to handle MM_ANONPAGES counter updates, and mTHP fault allocation statistics for the page fault path.
The zero-page read path in do_anonymous_page() is also untangled from the shared setpte label, since it does not allocate a folio and should not share the same mapping sequence as the write path. Make nr_pages = 1 rather than relying on the variable. This makes it more clear that we are operating on the zero page only.
This refactoring will also help reduce code duplication between mm/memory.c and mm/khugepaged.c, and provides a clean API for PTE-level anonymous folio mapping that can be reused by future callers.
--- include/linux/mm.h | 4 ++++ mm/memory.c | 60 +++++++++++++++++++++++++++++++--------------- 2 files changed, 45 insertions(+), 19 deletions(-)
include/linux/mm.h +4 -0
@@ -4903,4 +4903,8 @@ static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps)void snapshot_page (struct page_snapshot *ps,const struct page *page); +void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr, +bool uffd_wp); +#endif /* _LINUX_MM_H */
mm/memory.c +41 -19
84 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH mm-unstable v3 1/5] mm: consolidate anonymous folio PTE mapping into helpers
Quoted 7 lines · Nico Pache
The anonymous page fault handler in
do_anonymous_page()open-codes the sequence to map a newly allocated anonymous folio at the PTE level:
- construct the PTE entry
- add rmap
- add to LRU
- set the PTEs
- update the MMU cache.
Yikes yeah this all needs work. Thanks for looking at this!
Quoted 2 lines
Introduce a two helpers to consolidate this duplicated logic, mirroring the
NIT: 'Introduce a two helpers' -> "introduce two helpers'
Quoted 9 lines
existing
map_anon_folio_pmd_nopf()pattern for PMD-level mappings:
map_anon_folio_pte_nopf(): constructs the PTE entry, takes folio references, adds anon rmap and LRU. This function also handles theuffd_wpthat can occur in the pf variant.
map_anon_folio_pte_pf(): extends the nopf variant to handleMM_ANONPAGEScounter updates, and mTHP fault allocation statistics for the page fault path.
MEGA nit, not sure why you're not just putting this in a bullet list, just weird to see not-code indented here :)
Quoted 10 lines
The zero-page read path in
do_anonymous_page()is also untangled from the shared setpte label, since it does not allocate a folio and should not share the same mapping sequence as the write path. Makenr_pages= 1 rather than relying on the variable. This makes it more clear that we are operating on the zero page only.This refactoring will also help reduce code duplication between
mm/memory.candmm/khugepaged.c, and provides a clean API for PTE-level anonymous folio mapping that can be reused by future callers.
Maybe worth mentioning subseqent patches that will use what you set up here?
Also you split things out into _nopf() and _pf() variants, it might be worth saying exactly why you're doing that or what you are preparing to do?
Quoted 5 lines
Reviewed-by Dev Jain <[email protected]>Reviewed-by Lance Yang <[email protected]>Signed-off-by Nico Pache <[email protected]>
There's nits above and below, but overall the logic looks good, so with nits addressed/reasonably responded to:
Quoted 10 lines
--- include/linux/mm.h | 4 ++++ mm/memory.c | 60 +++++++++++++++++++++++++++++++--------------- 2 files changed, 45 insertions(+), 19 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps) · line 4908
Quoted 6 lines
include/linux/mm.h +3
-0@@ -4903,4 +4903,8 @@ static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps)void snapshot_page (struct page_snapshot *ps,const struct page *page); +void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr, +bool uffd_wp);
How I hate how uffd infiltrates all our code like this.
Not your fault :)
Quoted 7 lines
include/linux/mm.h +1
-0+#endif /* _LINUX_MM_H */ mm/memory.c +0
-0
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5204
Quoted 8 lines
mm/memory.c +5
-0@@ -5197,6 +5197,37 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)return folio_prealloc (vma->vm_mm, vma, vmf->address, true); } +void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr, +bool uffd_wp) +{ +unsigned int nr_pages =folio_nr_pages (folio);
const would be good
Quoted 10 lines
mm/memory.c +10
-0+pte_t entry =folio_mk_pte (folio, vma->vm_page_prot); + + entry =pte_sw_mkyoung (entry); + +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry), vma); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); + +folio_ref_add (folio, nr_pages -1 );
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5224
Quoted 10 lines
mm/memory.c +10
-0+folio_add_new_anon_rmap (folio, vma, addr,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, addr, pte, entry, nr_pages); +update_mmu_cache_range (NULL , vma, addr, pte, nr_pages); +} + +static void map_anon_folio_pte_pf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr,bool uffd_wp) +{ +unsigned int order =folio_order (folio);
const would be good here also!
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5227
Quoted 3 lines
mm/memory.c +3
-0+ +map_anon_folio_pte_nopf (folio, pte, vma, addr, uffd_wp); +add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 << order);
Is 1 << order strictly right here? This field is a long value, so 1L << order maybe? I get nervous about these shifts...
Note that folio_large_nr_pages() uses 1L << order so that does seem preferable.
Quoted 7 lines
mm/memory.c +3
-0+count_mthp_stat (order,MTHP_STAT_ANON_FAULT_ALLOC ); +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5279
Quoted 7 lines
mm/memory.c +3
-1@@ -5243,7 +5274,14 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING ); } - goto setpte; +if (vmf_orig_pte_uffd_wp (vmf)) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, addr, vmf->pte, entry);
How I _despise_ how uffd is implemented in mm. Feels like open coded nonsense spills out everywhere.
Not your fault obviously :)
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5283
Quoted 4 lines
mm/memory.c +4
-0+ +/* No need to invalidate - it was non-present before */ +update_mmu_cache_range (vmf, vma, addr, vmf->pte, +/*nr_pages=*/ 1 );
Is there any point in passing vmf here given you pass NULL above, and it appears that nobody actually uses this? I guess it doesn't matter but seeing this immediately made we question why you set it in one, and not the other?
Maybe I'm mistaken and some arch uses it? Don't think so though.
Also can't we then just use update_mmu_cache() which is the single-page wrapper of this AFAICT? That'd make it even simpler.
Having done this, is there any reason to keep the annoying and confusing initial assignment of nr_pages = 1 at declaration time?
It seems that nr_pages is unconditionally assigned before it's used anywhere now at line 5298:
nr_pages = folio_nr_pages(folio); addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); ...
It's kinda weird to use nr_pages again after you go out of your way to avoid it using folio_nr_pages() in map_anon_folio_pte_nopf() and folio_order() in map_anon_folio_pte_pf().
But yeah, ok we align the address and it's yucky maybe leave for now (but we can definitely stop defaulting nr_pages to 1 :)
Quoted 25 lines
mm/memory.c +1
-10+ goto unlock; }/* Allocate our own private page. */ @@ -5267,11 +5305,6 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =folio_mk_pte (folio, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry), vma); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release;@@ -5293,19 +5326,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)folio_put (folio);return handle_userfault (vmf,VM_UFFD_MISSING ); } - -folio_ref_add (folio, nr_pages -1 ); -add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); -count_mthp_stat (folio_order (folio),MTHP_STAT_ANON_FAULT_ALLOC ); -folio_add_new_anon_rmap (folio, vma, addr,RMAP_EXCLUSIVE );
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5330
Quoted 10 lines
mm/memory.c +2
-8-folio_add_lru_vma (folio, vma); -setpte: -if (vmf_orig_pte_uffd_wp (vmf)) - entry =pte_mkuffd_wp (entry); -set_ptes (vma->vm_mm, addr, vmf->pte, entry, nr_pages); - -/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, addr, vmf->pte, nr_pages); +map_anon_folio_pte_pf (folio, vmf->pte, vma, addr, +vmf_orig_pte_uffd_wp (vmf));
So we're going from:
entry = folio_mk_pte(...)
entry = pte_sw_mkyoung(...)
if (write)
entry = pte_mkwrite(also dirty...)folio_ref_add(nr_pages - 1)
add_mm_counter(... MM_ANON_PAGES, nr_pages) count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC) folio_add_new_anon_rmap(.., RMAP_EXCLUSIVE) folio_add_lru_vma(folio, vma)
if (vmf_orig_pte_uffd_wp(vmf))
entry = pte_mkuffd_wp(entry)set_ptes(mm, addr, vmf->pte, entry, nr_pages) update_mmu_cache_range(vmf, vma, addr, vmf->pte, nr_pages)
To:
<map_anon_folio_pte_nopf>
entry = folio_mk_pte(...)
entry = pte_sw_mkyoung(...)
if (write)
entry = pte_mkwrite(also dirty...)
if (vmf_orig_pte_uffd_wp(vmf) <passed in via uffd_wp>) <-- reordered
entry = pte_mkuffd_wp(entry)folio_ref_add(nr_pages - 1)folio_add_new_anon_rmap(.., RMAP_EXCLUSIVE) folio_add_lru_vma(folio, vma)set_ptes(mm, addr, pte, entry, nr_pages) update_mmu_cache_range(NULL, vma, addr, pte, nr_pages)
<map_anon_folio_pte_pf>
add_mm_counter(... MM_ANON_PAGES, nr_pages) <-- reordered count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC) <-- reodrdered
But the reorderings seem fine, and it is achieving the same thing.
All the parameters being passed seem correct too.
Quoted 6 lines
mm/memory.c +0
-1unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl); --2.53.0
Cheers, Lorenzo
Re: [PATCH mm-unstable v3 1/5] mm: consolidate anonymous folio PTE mapping into helpers
Quoted 10 lines · Lorenzo Stoakes (Oracle)
Quoted 7 lines · Nico Pache
The anonymous page fault handler in
do_anonymous_page()open-codes the sequence to map a newly allocated anonymous folio at the PTE level:
- construct the PTE entry
- add rmap
- add to LRU
- set the PTEs
- update the MMU cache.Yikes yeah this all needs work. Thanks for looking at this!
np! I believe it looks much cleaner now, and it has the added benefit of cleaning up some of the mTHP patchset.
I also believe you suggested this so I'll add your SB when I add your RB tag :)
Quoted 5 lines
Quoted 3 lines
Introduce a two helpers to consolidate this duplicated logic, mirroring the
NIT: 'Introduce a two helpers' -> "introduce two helpers'
ack!
Quoted 7 lines
Quoted 7 lines
existing
map_anon_folio_pmd_nopf()pattern for PMD-level mappings:
map_anon_folio_pte_nopf(): constructs the PTE entry, takes folio references, adds anon rmap and LRU. This function also handles theuffd_wpthat can occur in the pf variant.
Quoted 6 lines
Quoted 3 lines
map_anon_folio_pte_pf(): extends the nopf variant to handleMM_ANONPAGEScounter updates, and mTHP fault allocation statistics for the page fault path.MEGA nit, not sure why you're not just putting this in a bullet list, just weird to see not-code indented here :)
Ill drop the indentation!
Quoted 8 lines
Quoted 8 lines
The zero-page read path in
do_anonymous_page()is also untangled from the shared setpte label, since it does not allocate a folio and should not share the same mapping sequence as the write path. Makenr_pages= 1 rather than relying on the variable. This makes it more clear that we are operating on the zero page only.
Quoted 6 lines
Quoted 3 lines
This refactoring will also help reduce code duplication between
mm/memory.candmm/khugepaged.c, and provides a clean API for PTE-level anonymous folio mapping that can be reused by future callers.Maybe worth mentioning subseqent patches that will use what you set up here?
ok ill add something like "... that will be used when adding mTHP support to khugepaged, and may find future reuse by other callers." Speaking of which I tried to leverage this elsewhere and I believe that will take extra focus and time, but may be doable.
Quoted 3 lines
Also you split things out into
_nopf()and_pf()variants, it might be worth saying exactly why you're doing that or what you are preparing to do?
ok ill expand on this in the "bullet list" you reference above.
Quoted 11 lines
Quoted 6 lines
Reviewed-by Dev Jain <[email protected]>Reviewed-by Lance Yang <[email protected]>Signed-off-by Nico Pache <[email protected]>There's nits above and below, but overall the logic looks good, so with nits addressed/reasonably responded to:
Reviewed-by Lorenzo Stoakes (Oracle) <[email protected]>
Thanks :) Ill take care of those
Quoted 54 lines
Quoted 11 lines
--- include/linux/mm.h | 4 ++++ mm/memory.c | 60 +++++++++++++++++++++++++++++++--------------- 2 files changed, 45 insertions(+), 19 deletions(-)include/linux/mm.h +0
-0include/linux/mm.h static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps) · line 4908
Quoted 6 lines
include/linux/mm.h +3
-0@@ -4903,4 +4903,8 @@ static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps)void snapshot_page (struct page_snapshot *ps,const struct page *page); +void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr, +bool uffd_wp);How I hate how uffd infiltrates all our code like this.
Not your fault :)
Quoted 7 lines
include/linux/mm.h +1
-0+#endif /* _LINUX_MM_H */ mm/memory.c +0
-0mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5204
Quoted 8 lines
mm/memory.c +5
-0@@ -5197,6 +5197,37 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)return folio_prealloc (vma->vm_mm, vma, vmf->address, true); } +void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr, +bool uffd_wp) +{ +unsigned int nr_pages =folio_nr_pages (folio);const would be good
Quoted 14 lines
mm/memory.c +14
-0+pte_t entry =folio_mk_pte (folio, vma->vm_page_prot); + + entry =pte_sw_mkyoung (entry); + +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry), vma); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); + +folio_ref_add (folio, nr_pages -1 ); +folio_add_new_anon_rmap (folio, vma, addr,RMAP_EXCLUSIVE ); +folio_add_lru_vma (folio, vma); +set_ptes (vma->vm_mm, addr, pte, entry, nr_pages); +update_mmu_cache_range (NULL , vma, addr, pte, nr_pages);
Quoted 8 lines
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5224
Quoted 6 lines
mm/memory.c +6
-0+} + +static void map_anon_folio_pte_pf (struct folio *folio,pte_t *pte, +struct vm_area_struct *vma,unsigned long addr,bool uffd_wp) +{ +unsigned int order =folio_order (folio);const would be good here also!
ack on the const's
Quoted 10 lines
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 5227
Quoted 4 lines
mm/memory.c +3
-0+ +map_anon_folio_pte_nopf (folio, pte, vma, addr, uffd_wp); +add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 << order);Is 1 << order strictly right here? This field is a long value, so 1L << order maybe? I get nervous about these shifts...
Note that
folio_large_nr_pages()uses 1L << order so that does seem preferable.
Ok sounds good thanks!
Quoted 43 lines
Quoted 8 lines
mm/memory.c +3
-0+count_mthp_stat (order,MTHP_STAT_ANON_FAULT_ALLOC ); +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5279
Quoted 7 lines
mm/memory.c +3
-1@@ -5243,7 +5274,14 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING ); } - goto setpte; +if (vmf_orig_pte_uffd_wp (vmf)) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, addr, vmf->pte, entry);How I _despise_ how uffd is implemented in mm. Feels like open coded nonsense spills out everywhere.
Not your fault obviously :)
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5283
Quoted 4 lines
mm/memory.c +4
-0+ +/* No need to invalidate - it was non-present before */ +update_mmu_cache_range (vmf, vma, addr, vmf->pte, +/*nr_pages=*/ 1 );Is there any point in passing vmf here given you pass NULL above, and it appears that nobody actually uses this? I guess it doesn't matter but seeing this immediately made we question why you set it in one, and not the other?
Maybe I'm mistaken and some arch uses it? Don't think so though.
Also can't we then just use
update_mmu_cache()which is the single-page wrapper of this AFAICT? That'd make it even simpler.Having done this, is there any reason to keep the annoying and confusing initial assignment of
nr_pages= 1 at declaration time?It seems that
nr_pagesis unconditionally assigned before it's used anywhere now at line 5298:
Quoted 7 lines
nr_pages = folio_nr_pages(folio); addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); ...It's kinda weird to use
nr_pagesagain after you go out of your way to avoid it usingfolio_nr_pages()inmap_anon_folio_pte_nopf()andfolio_order()inmap_anon_folio_pte_pf().
Yes I believe so, thank you that is cleaner! In the past I had nr_pages as a variable but David suggested being explicit with the "1" so it would be obvious its a single page... update_mmu_cache() should indicate the same thing.
Quoted 71 lines
But yeah, ok we align the address and it's yucky maybe leave for now (but we can definitely stop defaulting
nr_pagesto 1 :)Quoted 25 lines
mm/memory.c +1
-10+ goto unlock; }/* Allocate our own private page. */ @@ -5267,11 +5305,6 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =folio_mk_pte (folio, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry), vma); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release;@@ -5293,19 +5326,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)folio_put (folio);return handle_userfault (vmf,VM_UFFD_MISSING ); } - -folio_ref_add (folio, nr_pages -1 ); -add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); -count_mthp_stat (folio_order (folio),MTHP_STAT_ANON_FAULT_ALLOC ); -folio_add_new_anon_rmap (folio, vma, addr,RMAP_EXCLUSIVE );mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 5330
Quoted 10 lines
mm/memory.c +2
-8-folio_add_lru_vma (folio, vma); -setpte: -if (vmf_orig_pte_uffd_wp (vmf)) - entry =pte_mkuffd_wp (entry); -set_ptes (vma->vm_mm, addr, vmf->pte, entry, nr_pages); - -/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, addr, vmf->pte, nr_pages); +map_anon_folio_pte_pf (folio, vmf->pte, vma, addr, +vmf_orig_pte_uffd_wp (vmf));So we're going from:
entry =
folio_mk_pte(...)
entry =pte_sw_mkyoung(...)
if (write)
entry =pte_mkwrite(also dirty...)folio_ref_add(nr_pages - 1)
add_mm_counter(...MM_ANON_PAGES,nr_pages)count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC)folio_add_new_anon_rmap(.., RMAP_EXCLUSIVE)folio_add_lru_vma(folio, vma)
if (vmf_orig_pte_uffd_wp(vmf))
entry =pte_mkuffd_wp(entry)set_ptes(mm, addr, vmf->pte, entry, nr_pages)update_mmu_cache_range(vmf, vma, addr, vmf->pte, nr_pages)To:
<
map_anon_folio_pte_nopf>
entry =folio_mk_pte(...)
entry =pte_sw_mkyoung(...)
if (write)
entry =pte_mkwrite(also dirty...)
if (vmf_orig_pte_uffd_wp(vmf)<passed in viauffd_wp>) <-- reordered
entry =pte_mkuffd_wp(entry)folio_ref_add(nr_pages - 1)folio_add_new_anon_rmap(.., RMAP_EXCLUSIVE)folio_add_lru_vma(folio, vma)set_ptes(mm, addr, pte, entry, nr_pages)update_mmu_cache_range(NULL, vma, addr, pte, nr_pages)
Quoted 7 lines
<
map_anon_folio_pte_pf>
add_mm_counter(...MM_ANON_PAGES,nr_pages) <-- reorderedcount_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC)<-- reodrderedBut the reorderings seem fine, and it is achieving the same thing.
All the parameters being passed seem correct too.
Thanks for the review and verifying the logic :) I was particularly scared of changing the page fault handler, so im glad multiple people have confirmed this seems fine.
Cheers,
-- Nico
Quoted 10 lines
Quoted 7 lines
mm/memory.c +0
-1unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl); --2.53.0
Cheers, Lorenzo
PATCH v3 2/5 mm: introduce is_pmd_order helper
In reply to Nico PacheIn order to add mTHP support to khugepaged, we will often be checking if a given order is (or is not) a PMD order. Some places in the kernel already use this check, so lets create a simple helper function to keep the code clean and readable.
--- include/linux/huge_mm.h | 5 +++++ mm/huge_memory.c | 2 +- mm/khugepaged.c | 6 +++--- mm/memory.c | 2 +- mm/mempolicy.c | 2 +- mm/page_alloc.c | 4 ++-- mm/shmem.c | 3 +-- 7 files changed, 14 insertions(+), 10 deletions(-)
include/linux/huge_mm.h +5 -0
@@ -771,6 +771,11 @@ static inline bool pmd_is_huge(pmd_t pmd) }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool is_pmd_order (unsigned int order) +{ +return order ==HPAGE_PMD_ORDER ; +} +static inline int split_folio_to_list_to_order (struct folio *folio,struct list_head *list,int new_order) {
mm/huge_memory.c +1 -1
@@ -4122,7 +4122,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,i_mmap_unlock_read (mapping); out:xas_destroy (&xas); -if (old_order ==HPAGE_PMD_ORDER ) +if (is_pmd_order (old_order))count_vm_event (!ret ?THP_SPLIT_PAGE :THP_SPLIT_PAGE_FAILED );count_mthp_stat (old_order, !ret ?MTHP_STAT_SPLIT :MTHP_STAT_SPLIT_FAILED );return ret;
mm/khugepaged.c +3 -3
@@ -1532,7 +1532,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsignif (IS_ERR (folio))return SCAN_PAGE_NULL ; -if (folio_order (folio) !=HPAGE_PMD_ORDER ) { +if (!is_pmd_order (folio_order (folio))) { result =SCAN_PAGE_COMPOUND ; goto drop_folio; }
@@ -2015,7 +2015,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,* we locked the first folio, then a THP might be there already. * This will be discovered on the first iteration. */ -if (folio_order (folio) ==HPAGE_PMD_ORDER ) { +if (is_pmd_order (folio_order (folio))) { result =SCAN_PTE_MAPPED_HUGEPAGE ; goto out_unlock; }
@@ -2343,7 +2343,7 @@ static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm,continue ; } -if (folio_order (folio) ==HPAGE_PMD_ORDER ) { +if (is_pmd_order (folio_order (folio))) { result =SCAN_PTE_MAPPED_HUGEPAGE ;/* * PMD-sized THP implies that we can only try
mm/memory.c +1 -1
@@ -5479,7 +5479,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *paif (!thp_vma_suitable_order (vma, haddr,PMD_ORDER ))return ret; -if (folio_order (folio) !=HPAGE_PMD_ORDER ) +if (!is_pmd_order (folio_order (folio)))return ret; page = &folio->page;
mm/mempolicy.c +1 -1
@@ -2449,7 +2449,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) &&/* filter "hugepage" allocation, unless from alloc_pages() */ - order ==HPAGE_PMD_ORDER && ilx !=NO_INTERLEAVE_INDEX ) { +is_pmd_order (order) && ilx !=NO_INTERLEAVE_INDEX ) {/* * For hugepage allocation and non-interleave policy which * allows the current node (or other explicitly preferred
mm/page_alloc.c +2 -2
@@ -652,7 +652,7 @@ static inline unsigned int order_to_pindex(int migratetype, int order)#ifdef CONFIG_TRANSPARENT_HUGEPAGE bool movable;if (order >PAGE_ALLOC_COSTLY_ORDER ) { -VM_BUG_ON (order !=HPAGE_PMD_ORDER ); +VM_BUG_ON (!is_pmd_order (order)); movable = migratetype ==MIGRATE_MOVABLE ;
@@ -684,7 +684,7 @@ static inline bool pcp_allowed_order(unsigned int order)if (order <=PAGE_ALLOC_COSTLY_ORDER )return true;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (order ==HPAGE_PMD_ORDER ) +if (is_pmd_order (order))return true;#endif return false;
mm/shmem.c +1 -2
@@ -5532,8 +5532,7 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,spin_unlock (&huge_shmem_orders_lock); }else if (sysfs_streq (buf,"inherit" )) {/* Do not override huge allocation policy with non-PMD sized mTHP */ -if (shmem_huge ==SHMEM_HUGE_FORCE && - order !=HPAGE_PMD_ORDER ) +if (shmem_huge ==SHMEM_HUGE_FORCE && !is_pmd_order (order))return -EINVAL ;spin_lock (&huge_shmem_orders_lock);
Signature · 1 line
2.53.0
PATCH v3 3/5 mm/khugepaged: define KHUGEPAGED_MAX_PTES_LIMIT as HPAGE_PMD_NR - 1
In reply to Nico PacheThe value (HPAGE_PMD_NR - 1) is used often in the khugepaged code to signify the limit of the max_ptes_* values. Add a define for this to increase code readability and reuse.
--- mm/khugepaged.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
mm/khugepaged.c +5 -4
@@ -89,6 +89,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);* * Note that these are only respected if collapse was initiated by khugepaged. */ +#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1)unsigned int khugepaged_max_ptes_none__read_mostly ;static unsigned int khugepaged_max_ptes_swap__read_mostly ;static unsigned int khugepaged_max_ptes_shared__read_mostly ;
@@ -259,7 +260,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj,unsigned long max_ptes_none; err =kstrtoul (buf,10 , &max_ptes_none); -if (err || max_ptes_none >HPAGE_PMD_NR -1 ) +if (err || max_ptes_none >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_none = max_ptes_none;
@@ -284,7 +285,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj,unsigned long max_ptes_swap; err =kstrtoul (buf,10 , &max_ptes_swap); -if (err || max_ptes_swap >HPAGE_PMD_NR -1 ) +if (err || max_ptes_swap >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_swap = max_ptes_swap;
@@ -310,7 +311,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj,unsigned long max_ptes_shared; err =kstrtoul (buf,10 , &max_ptes_shared); -if (err || max_ptes_shared >HPAGE_PMD_NR -1 ) +if (err || max_ptes_shared >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_shared = max_ptes_shared;
@@ -382,7 +383,7 @@ int __init khugepaged_init(void)return -ENOMEM ; khugepaged_pages_to_scan =HPAGE_PMD_NR *8 ; - khugepaged_max_ptes_none =HPAGE_PMD_NR -1 ; + khugepaged_max_ptes_none =KHUGEPAGED_MAX_PTES_LIMIT ; khugepaged_max_ptes_swap =HPAGE_PMD_NR /8 ; khugepaged_max_ptes_shared =HPAGE_PMD_NR /2 ;
Signature · 1 line
2.53.0
Re: [PATCH mm-unstable v3 3/5] mm/khugepaged: define KHUGEPAGED_MAX_PTES_LIMIT as HPAGE_PMD_NR - 1
Quoted 9 lines · Nico Pache
The value (
HPAGE_PMD_NR- 1) is used often in the khugepaged code to signify the limit of the max_ptes_* values. Add a define for this to increase code readability and reuse.Acked-by Pedro Falcato <[email protected]>Reviewed-by Baolin Wang <[email protected]>Reviewed-by Zi Yan <[email protected]>Signed-off-by Nico Pache <[email protected]>
Hm didn't I suggest this? Or actually I can't remember :P
Anyway LGTM, so:
Quoted 55 lines
--- mm/khugepaged.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/khugepaged.c +5
-5@@ -89,6 +89,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);* * Note that these are only respected if collapse was initiated by khugepaged. */ +#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1)unsigned int khugepaged_max_ptes_none__read_mostly ;static unsigned int khugepaged_max_ptes_swap__read_mostly ;static unsigned int khugepaged_max_ptes_shared__read_mostly ;@@ -259,7 +260,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj,unsigned long max_ptes_none; err =kstrtoul (buf,10 , &max_ptes_none); -if (err || max_ptes_none >HPAGE_PMD_NR -1 ) +if (err || max_ptes_none >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_none = max_ptes_none;@@ -284,7 +285,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj,unsigned long max_ptes_swap; err =kstrtoul (buf,10 , &max_ptes_swap); -if (err || max_ptes_swap >HPAGE_PMD_NR -1 ) +if (err || max_ptes_swap >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_swap = max_ptes_swap;@@ -310,7 +311,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj,unsigned long max_ptes_shared; err =kstrtoul (buf,10 , &max_ptes_shared); -if (err || max_ptes_shared >HPAGE_PMD_NR -1 ) +if (err || max_ptes_shared >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_shared = max_ptes_shared;@@ -382,7 +383,7 @@ int __init khugepaged_init(void)return -ENOMEM ; khugepaged_pages_to_scan =HPAGE_PMD_NR *8 ; - khugepaged_max_ptes_none =HPAGE_PMD_NR -1 ; + khugepaged_max_ptes_none =KHUGEPAGED_MAX_PTES_LIMIT ; khugepaged_max_ptes_swap =HPAGE_PMD_NR /8 ; khugepaged_max_ptes_shared =HPAGE_PMD_NR /2 ; --2.53.0
Re: [PATCH mm-unstable v3 3/5] mm/khugepaged: define KHUGEPAGED_MAX_PTES_LIMIT as HPAGE_PMD_NR - 1
Quoted 5 lines · Lorenzo Stoakes (Oracle)
Quoted 4 lines · Nico Pache
The value (
HPAGE_PMD_NR- 1) is used often in the khugepaged code to signify the limit of the max_ptes_* values. Add a define for this to increase code readability and reuse.
Quoted 7 lines
Quoted 5 lines
Acked-by Pedro Falcato <[email protected]>Reviewed-by Baolin Wang <[email protected]>Reviewed-by Zi Yan <[email protected]>Signed-off-by Nico Pache <[email protected]>Hm didn't I suggest this? Or actually I can't remember :P
I think you suggested most of these patches ;P Ill make sure to add your SB tag on the followups!
Quoted 4 lines
Anyway LGTM, so:
Reviewed-by Lorenzo Stoakes (Oracle) <[email protected]>
Thank you!
Quoted 57 lines
Quoted 56 lines
--- mm/khugepaged.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/khugepaged.c +5
-5@@ -89,6 +89,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);* * Note that these are only respected if collapse was initiated by khugepaged. */ +#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1)unsigned int khugepaged_max_ptes_none__read_mostly ;static unsigned int khugepaged_max_ptes_swap__read_mostly ;static unsigned int khugepaged_max_ptes_shared__read_mostly ;@@ -259,7 +260,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj,unsigned long max_ptes_none; err =kstrtoul (buf,10 , &max_ptes_none); -if (err || max_ptes_none >HPAGE_PMD_NR -1 ) +if (err || max_ptes_none >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_none = max_ptes_none;@@ -284,7 +285,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj,unsigned long max_ptes_swap; err =kstrtoul (buf,10 , &max_ptes_swap); -if (err || max_ptes_swap >HPAGE_PMD_NR -1 ) +if (err || max_ptes_swap >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_swap = max_ptes_swap;@@ -310,7 +311,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj,unsigned long max_ptes_shared; err =kstrtoul (buf,10 , &max_ptes_shared); -if (err || max_ptes_shared >HPAGE_PMD_NR -1 ) +if (err || max_ptes_shared >KHUGEPAGED_MAX_PTES_LIMIT )return -EINVAL ; khugepaged_max_ptes_shared = max_ptes_shared;@@ -382,7 +383,7 @@ int __init khugepaged_init(void)return -ENOMEM ; khugepaged_pages_to_scan =HPAGE_PMD_NR *8 ; - khugepaged_max_ptes_none =HPAGE_PMD_NR -1 ; + khugepaged_max_ptes_none =KHUGEPAGED_MAX_PTES_LIMIT ; khugepaged_max_ptes_swap =HPAGE_PMD_NR /8 ; khugepaged_max_ptes_shared =HPAGE_PMD_NR /2 ; --2.53.0
PATCH v3 4/5 mm/khugepaged: rename hpage_collapse_* to collapse_*
In reply to Nico PacheThe hpage_collapse functions describe functions used by madvise_collapse and khugepaged. remove the unnecessary hpage prefix to shorten the function name.
--- mm/khugepaged.c | 60 ++++++++++++++++++++++++------------------------- mm/mremap.c | 2 +- 2 files changed, 30 insertions(+), 32 deletions(-)
mm/khugepaged.c +29 -31
@@ -395,14 +395,14 @@ void __init khugepaged_destroy(void)kmem_cache_destroy (mm_slot_cache); } -static inline int hpage_collapse_test_exit (struct mm_struct *mm) +static inline int collapse_test_exit (struct mm_struct *mm) {return atomic_read (&mm->mm_users) ==0 ; } -static inline int hpage_collapse_test_exit_or_disable (struct mm_struct *mm) +static inline int collapse_test_exit_or_disable (struct mm_struct *mm) { -return hpage_collapse_test_exit (mm) || +return collapse_test_exit (mm) ||mm_flags_test (MMF_DISABLE_THP_COMPLETELY , mm); }
@@ -436,7 +436,7 @@ void __khugepaged_enter(struct mm_struct *mm)int wakeup;/* __khugepaged_exit() must not run from under us */ -VM_BUG_ON_MM (hpage_collapse_test_exit (mm), mm); +VM_BUG_ON_MM (collapse_test_exit (mm), mm);if (unlikely (mm_flags_test_and_set (MMF_VM_HUGEPAGE , mm)))return ;
@@ -490,7 +490,7 @@ void __khugepaged_exit(struct mm_struct *mm) }else if (slot) {/* * This is required to serialize against -* hpage_collapse_test_exit() (which is guaranteed to run +* collapse_test_exit() (which is guaranteed to run * under mmap sem read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock.
@@ -585,7 +585,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } -/* See hpage_collapse_scan_pmd(). */ +/* See collapse_scan_pmd(). */ if (folio_maybe_mapped_shared (folio)) { ++shared; if (cc->is_khugepaged &&
@@ -836,7 +836,7 @@ static struct collapse_control khugepaged_collapse_control = { .is_khugepaged = true, }; -static bool hpage_collapse_scan_abort (int nid,struct collapse_control *cc) +static bool collapse_scan_abort (int nid,struct collapse_control *cc) {int i;
@@ -871,7 +871,7 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void) }#ifdef CONFIG_NUMA -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {int nid, target_node =0 , max_value =0 ;
@@ -890,7 +890,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) return target_node; }#else -static int hpage_collapse_find_target_node (struct collapse_control *cc) +static int collapse_find_target_node (struct collapse_control *cc) {return 0 ; }
@@ -909,7 +909,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned lenum tva_type type = cc->is_khugepaged ?TVA_KHUGEPAGED :TVA_FORCED_COLLAPSE ; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm)))return SCAN_ANY_PROCESS ; *vmap = vma =find_vma (mm, address);
@@ -980,7 +980,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm,/* * Bring missing pages in from swap, to complete THP collapse. -* Only done if hpage_collapse_scan_pmd believes it is worthwhile. +* Only done if khugepaged_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released.
@@ -1066,7 +1066,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru {gfp_t gfp = (cc->is_khugepaged ?alloc_hugepage_khugepaged_gfpmask () :GFP_TRANSHUGE ); -int node =hpage_collapse_find_target_node (cc); +int node =collapse_find_target_node (cc);struct folio *folio; folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask);
@@ -1244,7 +1244,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long areturn result; } -static enum scan_result hpage_collapse_scan_pmd (struct mm_struct *mm, +static enum scan_result collapse_scan_pmd (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long start_addr,bool *mmap_locked,struct collapse_control *cc) {
@@ -1365,7 +1365,7 @@ static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,* hit record. */ node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ; goto out_unmap; }
@@ -1431,7 +1431,7 @@ static void collect_mm_slot(struct mm_slot *slot)lockdep_assert_held (&khugepaged_mm_lock); -if (hpage_collapse_test_exit (mm)) { +if (collapse_test_exit (mm)) {/* free mm_slot */ hash_del (&slot->hash);list_del (&slot->mm_node);
@@ -1786,7 +1786,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)if (find_pmd_or_thp_or_none (mm, addr, &pmd) !=SCAN_SUCCEED )continue ; -if (hpage_collapse_test_exit (mm)) +if (collapse_test_exit (mm))continue ;if (!file_backed_vma_is_retractable (vma))
@@ -2302,7 +2302,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,return result; } -static enum scan_result hpage_collapse_scan_file (struct mm_struct *mm, +static enum scan_result collapse_scan_file (struct mm_struct *mm,unsigned long addr,struct file *file,pgoff_t start,struct collapse_control *cc) {
@@ -2355,7 +2355,7 @@ static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, } node =folio_nid (folio); -if (hpage_collapse_scan_abort (node, cc)) { +if (collapse_scan_abort (node, cc)) { result =SCAN_SCAN_ABORT ;folio_put (folio);break ;
@@ -2409,7 +2409,7 @@ static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm,return result; } -static void khugepaged_scan_mm_slot(unsigned int progress_max, +static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) __acquires(&khugepaged_mm_lock)
@@ -2443,7 +2443,7 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max, goto breakouterloop_mmap_lock; cc->progress++; -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop;vma_iter_init (&vmi, mm, khugepaged_scan.address);
@@ -2451,7 +2451,7 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max,unsigned long hstart, hend;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) { +if (unlikely (collapse_test_exit_or_disable (mm))) { cc->progress++;break ; }
@@ -2473,7 +2473,7 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max,bool mmap_locked = true;cond_resched (); -if (unlikely (hpage_collapse_test_exit_or_disable (mm))) +if (unlikely (collapse_test_exit_or_disable (mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart ||
@@ -2486,12 +2486,12 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max,mmap_read_unlock (mm); mmap_locked = false; - *result =hpage_collapse_scan_file (mm, + *result =collapse_scan_file (mm, khugepaged_scan.address, file, pgoff, cc);fput (file);if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) {mmap_read_lock (mm); -if (hpage_collapse_test_exit_or_disable (mm)) +if (collapse_test_exit_or_disable (mm)) goto breakouterloop; *result =try_collapse_pte_mapped_thp (mm, khugepaged_scan.address, false);
@@ -2500,7 +2500,7 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max,mmap_read_unlock (mm); } }else { - *result =hpage_collapse_scan_pmd (mm, vma, + *result =collapse_scan_pmd (mm, vma, khugepaged_scan.address, &mmap_locked, cc); }
@@ -2532,7 +2532,7 @@ static void khugepaged_scan_mm_slot(unsigned int progress_max,* Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm, or THP got disabled. */ -if (hpage_collapse_test_exit_or_disable (mm) || !vma) { +if (collapse_test_exit_or_disable (mm) || !vma) {/* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find
@@ -2585,7 +2585,7 @@ static void khugepaged_do_scan(struct collapse_control *cc) pass_through_head++;if (khugepaged_has_work () && pass_through_head <2 ) -khugepaged_scan_mm_slot (progress_max, &result, cc); +collapse_scan_mm_slot (progress_max, &result, cc);else cc->progress = progress_max;spin_unlock (&khugepaged_mm_lock);
@@ -2830,8 +2830,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,mmap_read_unlock (mm); mmap_locked = false; *lock_dropped = true; - result =hpage_collapse_scan_file (mm, addr, file, pgoff, - cc); + result =collapse_scan_file (mm, addr, file, pgoff, cc);if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&mapping_can_writeback (file->f_mapping)) {
@@ -2845,8 +2844,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, }fput (file); }else { - result =hpage_collapse_scan_pmd (mm, vma, addr, - &mmap_locked, cc); + result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); }if (!mmap_locked) *lock_dropped = true;
mm/mremap.c +1 -1
@@ -244,7 +244,7 @@ static int move_ptes(struct pagetable_move_control *pmc, goto out; }/* -* Now new_pte is none, so hpage_collapse_scan_file() path can not find +* Now new_pte is none, so collapse_scan_file() path can not find * this by traversing file->f_mapping, so there is no concurrency with * retract_page_tables(). In addition, we already hold the exclusive * mmap_lock, so this new_pte page is stable, so there is no need to get
Signature · 1 line
2.53.0
PATCH v3 5/5 mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
In reply to Nico PacheThe khugepaged daemon and madvise_collapse have two different implementations that do almost the same thing. Create collapse_single_pmd to increase code reuse and create an entry point to these two users.
Refactor madvise_collapse and collapse_scan_mm_slot to use the new collapse_single_pmd function. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged tests collapse_test_exit_or_disable before calling collapse_pte_mapped_thp, but we weren't doing it in the madvise_collapse case. By unifying these two callers madvise_collapse now also performs this check. We also modify the return value to be SCAN_ANY_PROCESS which properly indicates that this process is no longer valid to operate on.
By moving the madvise_collapse writeback-retry logic into the helper function we can also avoid having to revalidate the VMA.
We also guard the khugepaged_pages_collapsed variable to ensure its only incremented for khugepaged.
--- mm/khugepaged.c | 120 +++++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 57 deletions(-)
mm/khugepaged.c +63 -57
@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; + +if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback, +* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry; + } +fput (file); + +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); +if (collapse_test_exit_or_disable (mm)) + result =SCAN_ANY_PROCESS ; +else + result =try_collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + } +end: +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +return result; +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)
@@ -2479,34 +2538,9 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =try_collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (!mmap_locked)
@@ -2806,9 +2840,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {enum scan_result result =SCAN_FAIL ; -bool triggered_wb = false; -retry:if (!mmap_locked) {cond_resched ();mmap_read_lock (mm);
@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE :
Signature · 1 line
2.53.0
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
On Wed, Mar 11, 2026 at 03:13:15PM -0600, Nico Pache wrote:
[..]
Quoted 36 lines
mm/khugepaged.c +1
-22@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2871
Quoted 10 lines
mm/khugepaged.c +1
-6break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :
It looks we won't have this case after refactor?
Current code flow is like this:
result = collapse_single_pmd()
result = collapse_scan_file()
result = collapse_file()
if (result == SCAN_PTE_MAPPED_HUGEPAGE) { --- (1)
result = SCAN_ANY_PROCESS;
or
result = try_collapse_pte_mapped_thp();
}Only collapse_scan_file() and collapse_file() may return SCAN_PTE_MAPPED_HUGEPAGE, and then handled in (1). After this, result is set to another value to indicate whether we collapse it or not.
So I am afraid we don't expect to see SCAN_PTE_MAPPED_HUGEPAGE here. Do I miss something?
Quoted 6 lines
mm/khugepaged.c +0
-0case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE :Signature · 1 line
2.53.0
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
Quoted 63 lines · Wei Yang
On Wed, Mar 11, 2026 at 03:13:15PM -0600, Nico Pache wrote:
[..]Quoted 36 lines
mm/khugepaged.c +1
-22@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2871
Quoted 10 lines
mm/khugepaged.c +1
-6break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :It looks we won't have this case after refactor?
Current code flow is like this:
result =
collapse_single_pmd()result = collapse_scan_file() result = collapse_file() if (result == SCAN_PTE_MAPPED_HUGEPAGE) { --- (1) result = SCAN_ANY_PROCESS; or result = try_collapse_pte_mapped_thp(); }
Quoted 6 lines
Only
collapse_scan_file()andcollapse_file()may returnSCAN_PTE_MAPPED_HUGEPAGE, and then handled in (1). After this, result is set to another value to indicate whether we collapse it or not.So I am afraid we don't expect to see
SCAN_PTE_MAPPED_HUGEPAGEhere. Do I miss something?
No your assessment is correct, should I remove it from the list? I've been quite confused about requests to list all the available ENUMs, does that mean we want all the enums that are reachable or all the enums that are available as a result? Im guessing the former based on your comment.
Cheers,
-- Nico
Quoted 8 lines
Quoted 7 lines
mm/khugepaged.c +0
-0case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE :Signature · 1 line
2.53.0
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
Quoted 70 lines · Nico Pache
Quoted 67 lines · Wei Yang
On Wed, Mar 11, 2026 at 03:13:15PM -0600, Nico Pache wrote:
[..]Quoted 36 lines
mm/khugepaged.c +1
-22@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2871
Quoted 10 lines
mm/khugepaged.c +1
-6break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :It looks we won't have this case after refactor?
Current code flow is like this:
result =
collapse_single_pmd()result = collapse_scan_file() result = collapse_file() if (result == SCAN_PTE_MAPPED_HUGEPAGE) { --- (1) result = SCAN_ANY_PROCESS; or result = try_collapse_pte_mapped_thp(); }Only
collapse_scan_file()andcollapse_file()may returnSCAN_PTE_MAPPED_HUGEPAGE, and then handled in (1). After this, result is set to another value to indicate whether we collapse it or not.
Quoted 8 lines
Quoted 2 lines
So I am afraid we don't expect to see
SCAN_PTE_MAPPED_HUGEPAGEhere. Do I miss something?No your assessment is correct, should I remove it from the list? I've been quite confused about requests to list all the available ENUMs, does that mean we want all the enums that are reachable or all the enums that are available as a result? Im guessing the former based on your comment.
If it is me, I would remove it :-) Otherwise, I will think collapse_single_pmd() may return SCAN_PTE_MAPPED_HUGEPAGE.
But, yeah, this is really trivial.
Quoted 11 lines
Cheers,
-- NicoQuoted 8 lines
Quoted 7 lines
mm/khugepaged.c +0
-0case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE :Signature · 1 line
2.53.0
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
In reply to Nico PacheQuoted 142 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.Signed-off-by Nico Pache <[email protected]>--- mm/khugepaged.c | 120 +++++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 57 deletions(-)mm/khugepaged.c +61
-29@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; + +if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback, +* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry; + } +fput (file); + +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); +if (collapse_test_exit_or_disable (mm)) + result =SCAN_ANY_PROCESS ; +else + result =try_collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + } +end: +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +return result; +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)@@ -2479,34 +2538,9 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =try_collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (!mmap_locked)@@ -2806,9 +2840,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {enum scan_result result =SCAN_FAIL ; -bool triggered_wb = false; -retry:if (!mmap_locked) {cond_resched ();mmap_read_lock (mm);
mm/khugepaged.c int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · line 2832
Quoted 10 lines
mm/khugepaged.c +0
-6@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true;
Okay, we can get rid of this because collapse_single_pmd() will never drop the lock, to retake it, returning with "mmap_locked == true".
If it dropped the lock, even if it relocked, it will always return with the lock dropped and "mmap_locked == false".
The "if (!mmap_locked)" will properly set "*lock_dropped = true;".
Signature · 3 lines
Cheers, David
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
In reply to Nico PacheQuoted 13 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.
Quoted 8 lines
By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.Signed-off-by Nico Pache <[email protected]>---
LGTM.
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
In reply to Nico PacheQuoted 61 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.Signed-off-by Nico Pache <[email protected]>---
mm/khugepaged.c| 120 +++++++++++++++++++++++++-----------------------1 file changed, 63 insertions(+), 57 deletions(-)mm/khugepaged.c +29
-0@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd (unsigned long addr, +struct vm_area_struct *vma,bool *mmap_locked, +struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; + +if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback,
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2450
Quoted 10 lines
mm/khugepaged.c +10
-0+* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry;
While the old retry path did go back through hugepage_vma_revalidate(), the retry itself is not relying on the original VMA remaining unchanged IIUC.
After dropping mmap_lock, the code still holds a reference to the file, so no lifetime issue should arise here :)
So, LGTM!
Cheers,
Lance
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
In reply to Nico PacheQuoted 3 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.
Ah this is nice :) Thanks!
Quoted 10 lines
Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.
Quoted 7 lines
By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.Signed-off-by Nico Pache <[email protected]>
The logic all seems correct to me, just a bunch of nits below really. This is a really nice refactoring! :)
With them addressed:
Cheers, Lorenzo
Quoted 9 lines
--- mm/khugepaged.c | 120 +++++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 57 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2417
Quoted 9 lines
mm/khugepaged.c +6
-0@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *mmap_locked,
mmap_locked seems mildly pointless here, and it's a semi-code smell to pass 'is locked' flags I think.
You never read this, but the parameter implies somebody might pass in mmaplocked == false, but you know it's always true here.
Anyway I think it makes more sense to pass in lock_dropped and get rid of mmap_locked in madvise_collapse() and just pass in lock_dropped directly (setting it false if anon).
Also obviously update collapse_scan_mm_slot() to use lock_dropped instead just inverted.
That's clearer I think since it makes it a verb rather than a noun and the function is dictating whether or not the lock is dropped, it also implies the lock is held on entry.
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2425
Quoted 8 lines
mm/khugepaged.c +8
-0+ struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; +
Maybe move the mmap_assert_locked() from madvise_collapse() to here? Then we assert it in both cases.
Quoted 15 lines
mm/khugepaged.c +15
-0+if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback,
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2450
Quoted 10 lines
mm/khugepaged.c +10
-0+* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + +filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry;
Thinking through this logic I do agree that we don't need to revalidate here, which should be quite a nice win, I just don't know why we previously assumed we'd have to... or maybe it was just because it became too spaghetti to goto around it somehow??
Quoted 24 lines
mm/khugepaged.c +20
-0+ } +fput (file); + +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); +if (collapse_test_exit_or_disable (mm)) + result =SCAN_ANY_PROCESS ; +else + result =try_collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + } +end: +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +return result; +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)
mm/khugepaged.c static void collapse_scan_mm_slot(unsigned int progress_max, · line 2540
Quoted 3 lines
mm/khugepaged.c +0
-0@@ -2479,34 +2538,9 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend);
Nice-to-have, but could we convert these VM_BUG_ON()'s to VM_WARN_ON_ONCE()'s while we're passing?
Quoted 95 lines
mm/khugepaged.c +4
-58-if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =try_collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (!mmap_locked)@@ -2806,9 +2840,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {enum scan_result result =SCAN_FAIL ; -bool triggered_wb = false; -retry:if (!mmap_locked) {cond_resched ();mmap_read_lock (mm);@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE : --2.53.0
Re: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
Quoted 6 lines · Lorenzo Stoakes (Oracle)
Quoted 3 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.Ah this is nice :) Thanks!
Thanks :) hopefully more khugepaged cleanups to come after these series' land.
Quoted 17 lines
Quoted 17 lines
Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.
Quoted 8 lines
Quoted 1 line
Signed-off-by Nico Pache <[email protected]>The logic all seems correct to me, just a bunch of nits below really. This is a really nice refactoring! :)
With them addressed:
Reviewed-by Lorenzo Stoakes (Oracle) <[email protected]>
Thanks I will address those!
Quoted 32 lines
Cheers, Lorenzo
Quoted 9 lines
--- mm/khugepaged.c | 120 +++++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 57 deletions(-)mm/khugepaged.c +0
-0mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2417
Quoted 9 lines
mm/khugepaged.c +6
-0@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *mmap_locked,
mmap_lockedseems mildly pointless here, and it's a semi-code smell to pass 'is locked' flags I think.You never read this, but the parameter implies somebody might pass in mmaplocked == false, but you know it's always true here.
Anyway I think it makes more sense to pass in
lock_droppedand get rid ofmmap_lockedinmadvise_collapse()and just pass inlock_droppeddirectly (setting it false if anon).
Quoted 6 lines
Also obviously update
collapse_scan_mm_slot()to uselock_droppedinstead just inverted.That's clearer I think since it makes it a verb rather than a noun and the function is dictating whether or not the lock is dropped, it also implies the lock is held on entry.
Ok I will give this a shot!
Quoted 12 lines
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2425
Quoted 9 lines
mm/khugepaged.c +8
-0+ struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; +Maybe move the
mmap_assert_locked()frommadvise_collapse()to here? Then we assert it in both cases.
ack, Sounds like a good idea!
Quoted 23 lines
Quoted 23 lines
mm/khugepaged.c +22
-0+if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback, +* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; +
Quoted 8 lines
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2450
Quoted 3 lines
mm/khugepaged.c +3
-0+filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry;Thinking through this logic I do agree that we don't need to revalidate here, which should be quite a nice win, I just don't know why we previously assumed we'd have to... or maybe it was just because it became too spaghetti to goto around it somehow??
I believe the latter, the retry went at the top of the loop, and the revalidation was already being done.
Quoted 23 lines
Quoted 23 lines
mm/khugepaged.c +20
-0+ } +fput (file); + +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); +if (collapse_test_exit_or_disable (mm)) + result =SCAN_ANY_PROCESS ; +else + result =try_collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + } +end: +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +return result; +} +static void collapse_scan_mm_slot (unsigned int progress_max,enum scan_result *result,struct collapse_control *cc)
Quoted 8 lines
mm/khugepaged.c static void collapse_scan_mm_slot(unsigned int progress_max, · line 2540
Quoted 5 lines
mm/khugepaged.c +0
-0__releases (&khugepaged_mm_lock)@@ -2479,34 +2538,9 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend);Nice-to-have, but could we convert these VM_BUG_ON()'s to VM_WARN_ON_ONCE()'s while we're passing?
Yeah sure, I have a question about these, because they do concern me (perhaps out of ignorance). does a WARN_ON_ONCE stop the daemon? I would be concerned about a rogue khugepaged instance going through and messing with page tables when it fails some assertion. Could this not lead to serious memory/file corruptions?
Thanks for the reviews!
Cheers,
-- Nico
Quoted 97 lines
Quoted 96 lines
mm/khugepaged.c +4
-58-if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =try_collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (!mmap_locked)@@ -2806,9 +2840,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {enum scan_result result =SCAN_FAIL ; -bool triggered_wb = false; -retry:if (!mmap_locked) {cond_resched ();mmap_read_lock (mm);@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE : --2.53.0
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH mm-unstable v3 5/5] mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()
Quoted 94 lines · Nico Pache
Quoted 6 lines · Lorenzo Stoakes (Oracle)
Quoted 3 lines · Nico Pache
The khugepaged daemon and
madvise_collapsehave two different implementations that do almost the same thing. Createcollapse_single_pmdto increase code reuse and create an entry point to these two users.Ah this is nice :) Thanks!
Thanks :) hopefully more khugepaged cleanups to come after these series' land.
Quoted 17 lines
Quoted 17 lines
Refactor
madvise_collapseandcollapse_scan_mm_slotto use the newcollapse_single_pmdfunction. This introduces a minor behavioral change that is most likely an undiscovered bug. The current implementation of khugepaged testscollapse_test_exit_or_disablebefore callingcollapse_pte_mapped_thp, but we weren't doing it in themadvise_collapsecase. By unifying these two callersmadvise_collapsenow also performs this check. We also modify the return value to beSCAN_ANY_PROCESSwhich properly indicates that this process is no longer valid to operate on.By moving the
madvise_collapsewriteback-retry logic into the helper function we can also avoid having to revalidate the VMA.We also guard the
khugepaged_pages_collapsedvariable to ensure its only incremented for khugepaged.Quoted 8 lines
Quoted 1 line
Signed-off-by Nico Pache <[email protected]>The logic all seems correct to me, just a bunch of nits below really. This is a really nice refactoring! :)
With them addressed:
Reviewed-by Lorenzo Stoakes (Oracle) <[email protected]>Thanks I will address those!
Quoted 32 lines
Cheers, Lorenzo
Quoted 9 lines
--- mm/khugepaged.c | 120 +++++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 57 deletions(-)mm/khugepaged.c +0
-0mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2417
Quoted 9 lines
mm/khugepaged.c +6
-0@@ -2409,6 +2409,65 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } +/* +* Try to collapse a single PMD starting at a PMD aligned addr, and return +* the results. +*/ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *mmap_locked,
mmap_lockedseems mildly pointless here, and it's a semi-code smell to pass 'is locked' flags I think.You never read this, but the parameter implies somebody might pass in mmaplocked == false, but you know it's always true here.
Anyway I think it makes more sense to pass in
lock_droppedand get rid ofmmap_lockedinmadvise_collapse()and just pass inlock_droppeddirectly (setting it false if anon).Quoted 6 lines
Also obviously update
collapse_scan_mm_slot()to uselock_droppedinstead just inverted.That's clearer I think since it makes it a verb rather than a noun and the function is dictating whether or not the lock is dropped, it also implies the lock is held on entry.
Ok I will give this a shot!
Quoted 12 lines
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2425
Quoted 9 lines
mm/khugepaged.c +8
-0+ struct collapse_control *cc) +{ +struct mm_struct *mm = vma->vm_mm; +bool triggered_wb = false; +enum scan_result result; +struct file *file; +pgoff_t pgoff; +Maybe move the
mmap_assert_locked()frommadvise_collapse()to here? Then we assert it in both cases.
Quoted 1 line
ack, Sounds like a good idea!
Thanks + for above! :)
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
mm/khugepaged.c +22
-0+if (vma_is_anonymous (vma)) { + result =collapse_scan_pmd (mm, vma, addr, mmap_locked, cc); + goto end; + } + + file =get_file (vma->vm_file); + pgoff =linear_page_index (vma, addr); + +mmap_read_unlock (mm); + *mmap_locked = false; +retry: + result =collapse_scan_file (mm, addr, file, pgoff, cc); + +/* +* For MADV_COLLAPSE, when encountering dirty pages, try to writeback, +* then retry the collapse one time. +*/ +if (!cc->is_khugepaged && result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && + !triggered_wb &&mapping_can_writeback (file->f_mapping)) { +const loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; +const loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; +Quoted 8 lines
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2450
Quoted 3 lines
mm/khugepaged.c +3
-0+filemap_write_and_wait_range (file->f_mapping, lstart, lend); + triggered_wb = true; + goto retry;Thinking through this logic I do agree that we don't need to revalidate here, which should be quite a nice win, I just don't know why we previously assumed we'd have to... or maybe it was just because it became too spaghetti to goto around it somehow??
Quoted 2 lines
I believe the latter, the retry went at the top of the loop, and the revalidation was already being done.
Yeah makes sense!
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
mm/khugepaged.c +20
-0+ } +fput (file); + +if (result ==SCAN_PTE_MAPPED_HUGEPAGE ) { +mmap_read_lock (mm); +if (collapse_test_exit_or_disable (mm)) + result =SCAN_ANY_PROCESS ; +else + result =try_collapse_pte_mapped_thp (mm, addr, + !cc->is_khugepaged); +if (result ==SCAN_PMD_MAPPED ) + result =SCAN_SUCCEED ; +mmap_read_unlock (mm); + } +end: +if (cc->is_khugepaged && result ==SCAN_SUCCEED ) + ++khugepaged_pages_collapsed; +return result; +} +static void collapse_scan_mm_slot (unsigned int progress_max,enum scan_result *result,struct collapse_control *cc)Quoted 8 lines
mm/khugepaged.c static void collapse_scan_mm_slot(unsigned int progress_max, · line 2540
Quoted 5 lines
mm/khugepaged.c +0
-0__releases (&khugepaged_mm_lock)@@ -2479,34 +2538,9 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address < hstart || khugepaged_scan.address +HPAGE_PMD_SIZE > hend);Nice-to-have, but could we convert these VM_BUG_ON()'s to VM_WARN_ON_ONCE()'s while we're passing?
Quoted 5 lines
Yeah sure, I have a question about these, because they do concern me (perhaps out of ignorance). does a
WARN_ON_ONCEstop the daemon? I would be concerned about a rogue khugepaged instance going through and messing with page tables when it fails some assertion. Could this not lead to serious memory/file corruptions?
It won't, but all of this is in CONFIG_DEBUG_VM anyway, so it's a kernel bug if it happens in a release kernel, this is just for visibility when debugging and those systems should be e.g. VMS etc.
The fact this is VM_xxx and hasn't apparently fired before suggests we're good.
Quoted 104 lines
Thanks for the reviews!
Cheers,
-- NicoQuoted 97 lines
Quoted 96 lines
mm/khugepaged.c +4
-58-if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, - khugepaged_scan.address); - -mmap_read_unlock (mm); - mmap_locked = false; - *result =collapse_scan_file (mm, - khugepaged_scan.address, file, pgoff, cc); -fput (file); -if (*result ==SCAN_PTE_MAPPED_HUGEPAGE ) { -mmap_read_lock (mm); -if (collapse_test_exit_or_disable (mm)) - goto breakouterloop; - *result =try_collapse_pte_mapped_thp (mm, - khugepaged_scan.address, false); -if (*result ==SCAN_PMD_MAPPED ) - *result =SCAN_SUCCEED ; -mmap_read_unlock (mm); - } - }else { - *result =collapse_scan_pmd (mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - -if (*result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed; + *result =collapse_single_pmd (khugepaged_scan.address, + vma, &mmap_locked, cc);/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (!mmap_locked)@@ -2806,9 +2840,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {enum scan_result result =SCAN_FAIL ; -bool triggered_wb = false; -retry:if (!mmap_locked) {cond_resched ();mmap_read_lock (mm);@@ -2823,46 +2855,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); }mmap_assert_locked (mm); -if (!vma_is_anonymous (vma)) { -struct file *file =get_file (vma->vm_file); -pgoff_t pgoff =linear_page_index (vma, addr); -mmap_read_unlock (mm); - mmap_locked = false; - *lock_dropped = true; - result =collapse_scan_file (mm, addr, file, pgoff, cc); - -if (result ==SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && -mapping_can_writeback (file->f_mapping)) { -loff_t lstart = (loff_t )pgoff <<PAGE_SHIFT ; -loff_t lend = lstart +HPAGE_PMD_SIZE -1 ; + result =collapse_single_pmd (addr, vma, &mmap_locked, cc); -filemap_write_and_wait_range (file->f_mapping, lstart, lend); - triggered_wb = true; -fput (file); - goto retry; - } -fput (file); - }else { - result =collapse_scan_pmd (mm, vma, addr, &mmap_locked, cc); - }if (!mmap_locked) *lock_dropped = true; -handle_result:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED : ++thps;break ; -case SCAN_PTE_MAPPED_HUGEPAGE : -BUG_ON (mmap_locked); -mmap_read_lock (mm); - result =try_collapse_pte_mapped_thp (mm, addr, true); -mmap_read_unlock (mm); - goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_NO_PTE_TABLE : +case SCAN_PTE_MAPPED_HUGEPAGE :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_LACK_REFERENCED_PAGE : --2.53.0
Cheers, Lorenzo
Andrew Morton
maintainerQuoted 2 lines · On Wed, 11 Mar 2026 15:13:10 -0600 Nico Pache <[email protected]> wrote:
The following series contains cleanups and prerequisites for my work on khugepaged mTHP support [1]. These have been separated out to ease review.
Thanks, I'll add[1] these to mm.git's mm-new branch for testing. Later (a few days) I'll hopefully move them into the mm-unstable branch, where they will get linux-next exposure.
[1]: I'll suppress the usual added-to-mm emails. What a lot of cc's.