Optimizations for khugepaged
Subsystems (5)
Dev Jain
reviewerv2 cover Optimizations for khugepaged
This patch series uses PTE batching to optimize two functions, and also addresses a race condition.
---
@David I did not make the clear_full_ptes clean up since that will require some more work than I originally thought :)
v1->v2:
- Use for loop instead of do-while loop (Lorenzo)
- Remove folio_test_large check since the subpage-check condition will imply that (Baolin)
- Combine patch 1 and 2 into this series, add new patch 3
Dev Jain (3):
khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE
batching
khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE
batching
khugepaged: Reduce race probability between migration and khugepaged
mm/khugepaged.c | 77 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 57 insertions(+), 20 deletions(-)
Signature · 1 line
2.30.2
Dev Jain
reviewerPATCH v2 1/3 khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
Use PTE batching to optimize __collapse_huge_page_copy_succeeded().
On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.
For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().
No issues were observed with mm-selftests.
--- mm/khugepaged.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)
mm/khugepaged.c +21 -6
@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;struct folio *src, *tmp; -pte_t *_pte;pte_t pteval; +pte_t *_pte; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { +for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte += nr_ptes, + address += nr_ptes *PAGE_SIZE ) { + nr_ptes =1 ; pteval =ptep_get (_pte);if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 );
@@ -719,21 +722,33 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +struct page *src_page =pte_page (pteval); src =page_folio (src_page);if (!folio_test_large (src))release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (folio_test_large (src)) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, +/* full = */ false); +folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } }
Signature · 1 line
2.30.2
Re: [PATCH 1/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
Quoted 61 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)mm/khugepaged.c +16
-3@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;struct folio *src, *tmp; -pte_t *_pte;pte_t pteval; +pte_t *_pte; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { +for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte += nr_ptes, + address += nr_ptes *PAGE_SIZE ) { + nr_ptes =1 ; pteval =ptep_get (_pte);if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 );@@ -719,21 +722,33 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +struct page *src_page =pte_page (pteval); src =page_folio (src_page);if (!folio_test_large (src))release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (folio_test_large (src)) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 10 lines
mm/khugepaged.c +2
-2/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, +/* full = */ false);
Can you add this patch to your series if nobody objects and use clear_ptes() instead?
From 95e20ab0ff62bbbdcd89898c9d76fdc1ea961257 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Wed, 25 Jun 2025 12:55:20 +0200
Subject: [PATCH] mm: add get_and_clear_ptes() and clear_ptes()
Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.
Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next.
Should we make these inline functions instead and add separate docs? Probably not worth it for now.
--- include/linux/pgtable.h | 6 ++++++ mm/mremap.c | 2 +- mm/rmap.c | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-)
include/linux/pgtable.h +6 -0
@@ -736,6 +736,9 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, }#endif +#define get_and_clear_ptes (_mm, _addr, _ptep, _nr) \ + get_and_clear_full_ptes(_mm, _addr, _ptep, _nr, 0) +#ifndef clear_full_ptes/** * clear_full_ptes - Clear present PTEs that map consecutive pages of the same
@@ -768,6 +771,9 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, }#endif +#define clear_ptes (_mm, _addr, _ptep, _nr) \ + clear_full_ptes(_mm, _addr, _ptep, _nr, 0) +/* * If two threads concurrently fault at the same page, the thread that * won the race updates the PTE and its local TLB/Cache. The other thread
mm/mremap.c +1 -1
@@ -322,7 +322,7 @@ static int move_ptes(struct pagetable_move_control *pmc, old_pte, max_nr_ptes); force_flush = true; } - pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr_ptes,0 ); + pte =get_and_clear_ptes (mm, old_addr, old_ptep, nr_ptes); pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);
mm/rmap.c +1 -1
@@ -2031,7 +2031,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,flush_cache_range (vma, address, end_addr);/* Nuke the page table entry. */ - pteval =get_and_clear_full_ptes (mm, address, pvmw.pte, nr_pages,0 ); + pteval =get_and_clear_ptes (mm, address, pvmw.pte, nr_pages);/* * We clear the PTE but do not flush so potentially * a remote CPU could still be writing to the folio.
Signature · 7 lines
2.49.0 -- Cheers, David / dhildenb
Dev Jain
reviewerRe: [PATCH 1/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
Quoted 68 lines · David Hildenbrand
Quoted 67 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>---
mm/khugepaged.c| 27 +++++++++++++++++++++------1 file changed, 21 insertions(+), 6 deletions(-)mm/khugepaged.c +6
-2@@ -700,12 +700,15 @@ static void__collapse_huge_page_copy_succeeded (pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE; struct folio *src, *tmp; - pte_t *_pte; pte_t pteval; + pte_t *_pte; + int nr_ptes; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; - _pte++, address += PAGE_SIZE) { + for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes, + address += nr_ptes * PAGE_SIZE) { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);mm/khugepaged.c +10
-0@@ -719,21 +722,33 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { + const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int max_nr_ptes; + struct page *src_page = pte_page(pteval); src = page_folio(src_page); if (!folio_test_large(src)) release_pte_folio(src); + + max_nr_ptes = (end - address) >> PAGE_SHIFT; + if (folio_test_large(src)) + nr_ptes = folio_pte_batch(src, address, _pte, + pteval, max_nr_ptes, + flags, NULL, NULL, NULL); + /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */
Quoted 8 lines
mm/khugepaged.c static void · line 748
Quoted 5 lines
mm/khugepaged.c +2
-2spin_lock(ptl); - ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src, src_page, vma); + clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, + /* full = */ false);Can you add this patch to your series if nobody objects and use
clear_ptes()instead?
Thanks! Stupid me thought, because of arm64-specific impl of these, it
will be churn to do this, I am indeed lazy :)
Quoted 80 lines
From
95e20ab0ff62bbbdcd89898c9d76fdc1ea961257Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Wed, 25 Jun 2025 12:55:20 +0200
Subject: [PATCH] mm: addget_and_clear_ptes()andclear_ptes()Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.
Use
get_and_clear_ptes()in existing code,clear_ptes()users will be added next.Should we make these inline functions instead and add separate docs? Probably not worth it for now.
Signed-off-by David Hildenbrand <[email protected]>---
include/linux/pgtable.h | 6 ++++++
mm/mremap.c | 2 +-
mm/rmap.c | 2 +-3 files changed, 8 insertions(+), 2 deletions(-)include/linux/pgtable.h +3
-0@@ -736,6 +736,9 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, } #endif +#define get_and_clear_ptes (_mm, _addr, _ptep, _nr) \ + get_and_clear_full_ptes(_mm, _addr, _ptep, _nr, 0) + #ifndef clear_full_ptes /***
clear_full_ptes- Clear present PTEs that map consecutive pages of the sameinclude/linux/pgtable.h +3
-0@@ -768,6 +771,9 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, } #endif +#define clear_ptes(_mm, _addr, _ptep, _nr) \ + clear_full_ptes(_mm, _addr, _ptep, _nr, 0) + /* * If two threads concurrently fault at the same page, the thread that* won the race updates the PTE and its local TLB/Cache. The other thread
mm/mremap.c +1
-1@@ -322,7 +322,7 @@ static int move_ptes(struct pagetable_move_control *pmc, old_pte, max_nr_ptes); force_flush = true; } - pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0); + pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes); pte = move_pte(pte, old_addr, new_addr);pte =
move_soft_dirty_pte(pte);mm/rmap.c +1
-1@@ -2031,7 +2031,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, flush_cache_range(vma, address, end_addr); /* Nuke the page table entry. */ - pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0); + pteval = get_and_clear_ptes(mm, address, pvmw.pte, nr_pages);/*
* We clear the PTE but do not flush so potentially
* a remote CPU could still be writing to the folio.
Re: [PATCH 1/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
In reply to Dev JainYou forgot the v2 here :) this breaks b4 shazam...
I managed to do this on the cover letter (but not patches) of a series before. So you're in good company... ;)
Quoted 9 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.
Quoted 6 lines
For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>
Overall looking way way better! Just some nits below.
Quoted 13 lines
--- mm/khugepaged.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)mm/khugepaged.c +1
-0@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 710
Quoted 10 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte;pte_t pteval; +pte_t *_pte; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { +for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte += nr_ptes, + address += nr_ptes *PAGE_SIZE ) {
Thanks this is much better.
Quoted 12 lines
mm/khugepaged.c +4
-0+ nr_ptes =1 ; pteval =ptep_get (_pte);if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 );@@ -719,21 +722,33 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +struct page *src_page =pte_page (pteval);
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 738
Quoted 10 lines
mm/khugepaged.c +6
-0src =page_folio (src_page);if (!folio_test_large (src))release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (folio_test_large (src)) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL );
Nit, but we only use max_nr_ptes here so could declare and set here, e.g.:
if (folio_test_large(src)) {
int max_nr_ptes = (end - address) >> PAGE_SHIFT;
nr_ptes = folio_pte_batch(src, address, _pte,
pteval, max_nr_ptes,
flags, NULL, NULL, NULL);
}BTW I think David raised it, but is there a way to wrap folio_pte_batch() to not have to NULL, NULL, NULL here? :)
oh and if we do this, we can also combine this line with above so:
if (folio_test_large(src)) {
int max_nr_ptes = (end - address) >> PAGE_SHIFT;
nr_ptes = folio_pte_batch(src, address, _pte,
pteval, max_nr_ptes,
flags, NULL, NULL, NULL);
} else {
release_pte_folio(src);
}Which is neater.
Quoted 22 lines
mm/khugepaged.c +6
-4+/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, +/* full = */ false); +folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } } --2.30.2
Re: [PATCH 1/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
Quoted 10 lines
if (folio_test_large(src)) { int max_nr_ptes = (end - address) >> PAGE_SHIFT; nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); }BTW I think David raised it, but is there a way to wrap
folio_pte_batch()to not have to NULL, NULL, NULL here? :)
I'm working on a cleanup for all that. So stay tuned :)
Signature · 3 lines
Cheers, David / dhildenb
Dev Jain
reviewerRe: [PATCH 1/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching
In reply to Lorenzo StoakesQuoted 96 lines · Lorenzo Stoakes
You forgot the v2 here :) this breaks b4 shazam...
I managed to do this on the cover letter (but not patches) of a series before. So you're in good company... ;)
Quoted 9 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.Quoted 6 lines
For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>Overall looking way way better! Just some nits below.
Quoted 13 lines
--- mm/khugepaged.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-)mm/khugepaged.c +1
-0@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 710
Quoted 10 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte;pte_t pteval; +pte_t *_pte; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { +for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte += nr_ptes, + address += nr_ptes *PAGE_SIZE ) {Thanks this is much better.
Quoted 12 lines
mm/khugepaged.c +4
-0+ nr_ptes =1 ; pteval =ptep_get (_pte);if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {add_mm_counter (vma->vm_mm,MM_ANONPAGES ,1 );@@ -719,21 +722,33 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +struct page *src_page =pte_page (pteval);mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 738
Quoted 10 lines
mm/khugepaged.c +6
-0src =page_folio (src_page);if (!folio_test_large (src))release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (folio_test_large (src)) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL );Nit, but we only use
max_nr_pteshere so could declare and set here, e.g.:if (folio_test_large(src)) { int max_nr_ptes = (end - address) >> PAGE_SHIFT; nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); }BTW I think David raised it, but is there a way to wrap
folio_pte_batch()to not have to NULL, NULL, NULL here? :)oh and if we do this, we can also combine this line with above so:
if (folio_test_large(src)) { int max_nr_ptes = (end - address) >> PAGE_SHIFT; nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); } else { release_pte_folio(src); }
Quoted 1 line
Which is neater.
Okay.
Quoted 23 lines
Quoted 23 lines
mm/khugepaged.c +6
-4+/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, +/* full = */ false); +folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } } --2.30.2
Dev Jain
reviewerPATCH v2 2/3 khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
In reply to Dev JainUse PTE batching to optimize collapse_pte_mapped_thp().
On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.
For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().
Note that we do not need to make a change to the check
"if (folio_page(folio, i) != page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i + nr_batch_ptes - 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.
No issues were observed with mm-selftests.
--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)
mm/khugepaged.c +26 -12
@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);
@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte); + nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/*
@@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort; + mapped_folio =page_folio (page); + nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }/* step 4: remove empty page table */
@@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte)
Signature · 1 line
2.30.2
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 68 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +10
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte); + nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/*
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 7 lines
mm/khugepaged.c +2
-0@@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;
Isn't this asserting that folio == mapped_folio here? We're saying page is the ith page of folio, so why do we need to look up mapped_folio?
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 2 lines
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);
You're assigning this twice.
Quoted 31 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }/* step 4: remove empty page table */
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {
I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Quoted 11 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --2.30.2
V
Dev Jain
reviewerRe: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 63 lines · Lorenzo Stoakes
Quoted 62 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +8
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte);
Quoted 15 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 13 lines
mm/khugepaged.c +4
-0+ nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* @@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Quoted 3 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 2 lines
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 30 lines
Quoted 30 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }
Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 7 lines
mm/khugepaged.c +1
-1/* step 4: remove empty page table */ @@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because nr_ptes is an existing variable and I need a new variable to make the jump at the end of the PTE batch.
Quoted 13 lines
Quoted 10 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --Quoted 2 lines
2.30.2
V
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 81 lines · Dev Jain
Quoted 63 lines · Lorenzo Stoakes
Quoted 62 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +8
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte);Quoted 15 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 13 lines
mm/khugepaged.c +4
-0+ nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* @@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?
Quoted 2 lines
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Right but I'm saying mapped_folio == folio right? You're literally asserting it here? So there's no need to assign mapped_folio at all, just reference folio no?
Quoted 47 lines
Quoted 4 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 3 lines
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 30 lines
Quoted 30 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 7 lines
mm/khugepaged.c +1
-1/* step 4: remove empty page table */ @@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Quoted 2 lines
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.
I thought you eliminated nr_ptes as a variable here? Where else is it used?
Oh how this code needs refactoring...
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --Quoted 2 lines
2.30.2
V
Dev Jain
reviewerRe: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 80 lines · Lorenzo Stoakes
Quoted 79 lines · Dev Jain
Quoted 78 lines · Lorenzo Stoakes
Quoted 68 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +10
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte); + nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 7 lines
mm/khugepaged.c +2
-0@@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?
Quoted 4 lines
Quoted 2 lines
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Right but I'm saying
mapped_folio== folio right? You're literally asserting it here? So there's no need to assignmapped_folioat all, just reference folio no?
Ah I get you now, we can do the PTE batching directly on the folio, thanks.
Quoted 61 lines
Quoted 43 lines
Quoted 3 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 2 lines
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 29 lines
Quoted 29 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 7 lines
mm/khugepaged.c +1
-1/* step 4: remove empty page table */ @@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Quoted 2 lines
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --2.30.2
V
Dev Jain
reviewerRe: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
In reply to Lorenzo StoakesQuoted 127 lines · Lorenzo Stoakes
Quoted 79 lines · Dev Jain
Quoted 63 lines · Lorenzo Stoakes
Quoted 62 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +8
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte);Quoted 15 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 13 lines
mm/khugepaged.c +4
-0+ nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* @@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?Quoted 2 lines
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Right but I'm saying
mapped_folio== folio right? You're literally asserting it here? So there's no need to assignmapped_folioat all, just reference folio no?Quoted 42 lines
Quoted 2 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 1 line
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 38 lines
Quoted 31 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }/* step 4: remove empty page table */ mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 5 lines
mm/khugepaged.c +1
-1@@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Quoted 5 lines
Quoted 2 lines
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain nr_ptes, then the two variables will be nr_ptes and nr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call them nr_batch_ptes and nr_mapped_ptes.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --Quoted 2 lines
2.30.2
V
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 131 lines · Dev Jain
Quoted 129 lines · Lorenzo Stoakes
Quoted 79 lines · Dev Jain
Quoted 63 lines · Lorenzo Stoakes
Quoted 62 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +8
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte);Quoted 15 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 13 lines
mm/khugepaged.c +4
-0+ nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* @@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?Quoted 2 lines
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Right but I'm saying
mapped_folio== folio right? You're literally asserting it here? So there's no need to assignmapped_folioat all, just reference folio no?Quoted 44 lines
Quoted 2 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 1 line
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 29 lines
Quoted 29 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 7 lines
mm/khugepaged.c +1
-1/* step 4: remove empty page table */ @@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.
Quoted 8 lines
Quoted 3 lines
I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain
nr_ptes, then the two variables will benr_ptesandnr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call themnr_batch_ptesandnr_mapped_ptes.
Sigh, this is still awful. But probably just existing awfulness. This whole thing needs a tent thrown over it and fumigation... but again not your fault :)
I mean fine, this is fine then.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
mm/khugepaged.c +2
-3flush_tlb_mm (mm); -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); } unlock:if (start_pte) --Quoted 2 lines
2.30.2
V
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 141 lines · Lorenzo Stoakes
Quoted 131 lines · Dev Jain
Quoted 129 lines · Lorenzo Stoakes
Quoted 79 lines · Dev Jain
Quoted 63 lines · Lorenzo Stoakes
Quoted 62 lines · Dev Jain
Use PTE batching to optimize
collapse_pte_mapped_thp().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().Note that we do not need to make a change to the check
"if (folio_page(folio, i)!= page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i +nr_batch_ptes- 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-)mm/khugepaged.c +8
-3@@ -1499,15 +1499,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,int collapse_pte_mapped_thp (struct mm_struct *mm,unsigned long addr,bool install_pmd) { +int nr_mapped_ptes =0 , nr_batch_ptes, result =SCAN_FAIL ;struct mmu_notifier_range range;bool notified = false;unsigned long haddr = addr &HPAGE_PMD_MASK ; +unsigned long end = haddr +HPAGE_PMD_SIZE ;struct vm_area_struct *vma =vma_lookup (mm, haddr);struct folio *folio;pte_t *start_pte, *pte;pmd_t *pmd, pgt_pmd;spinlock_t *pml =NULL , *ptl; -int nr_ptes =0 , result =SCAN_FAIL ;int i;mmap_assert_locked (mm);@@ -1621,11 +1622,17 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort;/* step 2: clear page table and adjust rmap */ -for (i =0 , addr = haddr, pte = start_pte; - i <HPAGE_PMD_NR ; i++, addr +=PAGE_SIZE , pte++) { +for (i =0 , addr = haddr, pte = start_pte; i <HPAGE_PMD_NR ; + i += nr_batch_ptes, addr += nr_batch_ptes *PAGE_SIZE , + pte += nr_batch_ptes) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_batch_ptes = (end - addr) >>PAGE_SHIFT ; +struct folio *mapped_folio;struct page *page;pte_t ptent =ptep_get (pte);Quoted 15 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1652
Quoted 13 lines
mm/khugepaged.c +4
-0+ nr_batch_ptes =1 ; +if (pte_none (ptent))continue ;/* @@ -1639,26 +1646,33 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page =vm_normal_page (vma, addr, ptent); + mapped_folio =page_folio (page); +if (folio_page (folio, i) != page) goto abort;Isn't this asserting that folio ==
mapped_foliohere? We're saying page is the ith page of folio, so why do we need to look upmapped_folio?Quoted 2 lines
We need to check for all PTEs whether they map the right page or not. This may get disturbed due to mremap and stuff.
Right but I'm saying
mapped_folio== folio right? You're literally asserting it here? So there's no need to assignmapped_folioat all, just reference folio no?Quoted 44 lines
Quoted 2 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1653
Quoted 1 line
mm/khugepaged.c +1
-0+ mapped_folio =page_folio (page);You're assigning this twice.
Forgot to remove, thanks.
Quoted 29 lines
Quoted 29 lines
mm/khugepaged.c +10
-6+ nr_batch_ptes =folio_pte_batch (mapped_folio, addr, pte, ptent, + max_nr_batch_ptes, flags, +NULL ,NULL ,NULL ); +/* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ -ptep_clear (mm, addr, pte); -folio_remove_rmap_pte (folio, page, vma); - nr_ptes++; +clear_full_ptes (mm, addr, pte, nr_batch_ptes,/* full = */ false); +folio_remove_rmap_ptes (folio, page, nr_batch_ptes, vma); + nr_mapped_ptes += nr_batch_ptes; }if (!pml)spin_unlock (ptl);/* step 3: set proper refcount and mm_counters. */ -if (nr_ptes) { -folio_ref_sub (folio, nr_ptes); -add_mm_counter (mm,mm_counter_file (folio), -nr_ptes); +if (nr_mapped_ptes) { +folio_ref_sub (folio, nr_mapped_ptes); +add_mm_counter (mm,mm_counter_file (folio), -nr_mapped_ptes); }Quoted 9 lines
mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1708
Quoted 7 lines
mm/khugepaged.c +1
-1/* step 4: remove empty page table */ @@ -1691,10 +1705,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: -if (nr_ptes) { +if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.Quoted 8 lines
Quoted 3 lines
I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain
nr_ptes, then the two variables will benr_ptesandnr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call themnr_batch_ptesandnr_mapped_ptes.
Quoted 4 lines
Sigh, this is still awful. But probably just existing awfulness. This whole thing needs a tent thrown over it and fumigation... but again not your fault :)
I mean fine, this is fine then.
Probably best to be pragmatic here: as long as the educated reader understands the code, all good. I hope I'll never have to explain it to my daughters.
I know, I'm a perfectionist myself ;)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 15 lines · David Hildenbrand
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- if (nr_ptes) { + if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain
nr_ptes, then the two variables will benr_ptesandnr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call themnr_batch_ptesandnr_mapped_ptes.
Quoted 8 lines
Quoted 4 lines
Sigh, this is still awful. But probably just existing awfulness. This whole thing needs a tent thrown over it and fumigation... but again not your fault :)
I mean fine, this is fine then.
Probably best to be pragmatic here: as long as the educated reader understands the code, all good. I hope I'll never have to explain it to my daughters.
:)
To be crystal clear here - Dev - I am not asking you to refactor the existing mess, and obviously I concede your point on the issue raised here.
Did you need to respin other things here? If not I can relook and give tags.
Quoted 2 lines
I know, I'm a perfectionist myself ;)
I mean if I wanted perfection I'd have run screaming from the kernel right at the start :P
But in general on the refactoring point - I think this code is hard to understand even for the informed, the endless nesting and nesting and levels of 'you just have to know what's happening' make it really easy to miss details here.
What I'm saying is - we can improve on this, and really all I'm doing is todo++. Possibly even dave_todo++?? ;) Or could even be $<name here>_todo++ :P
I find it impossible to not mention this when it seems apparent to me... force of habit :)
Cheers, Lorenzo
Dev Jain
reviewerRe: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 21 lines · Lorenzo Stoakes
Quoted 13 lines · David Hildenbrand
Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- if (nr_ptes) { + if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain
nr_ptes, then the two variables will benr_ptesandnr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call themnr_batch_ptesandnr_mapped_ptes.Quoted 7 lines
Quoted 4 lines
Sigh, this is still awful. But probably just existing awfulness. This whole thing needs a tent thrown over it and fumigation... but again not your fault :)
I mean fine, this is fine then.
Probably best to be pragmatic here: as long as the educated reader understands the code, all good. I hope I'll never have to explain it to my daughters.
Quoted 6 lines
:)
To be crystal clear here - Dev - I am not asking you to refactor the existing mess, and obviously I concede your point on the issue raised here.
Did you need to respin other things here? If not I can relook and give tags.
Yes, I will have to respin because of folio_pte_batch() changes from David and have to add a simplification patch to this series authored by David, I was just doing that and was reading the email thread to see if I didn't miss any review comment and then I noticed I forgot to reply to the above point.
Quoted 10 lines
Quoted 2 lines
I know, I'm a perfectionist myself ;)
I mean if I wanted perfection I'd have run screaming from the kernel right at the start :P
But in general on the refactoring point - I think this code is hard to understand even for the informed, the endless nesting and nesting and levels of 'you just have to know what's happening' make it really easy to miss details here.
Quoted 5 lines
What I'm saying is - we can improve on this, and really all I'm doing is todo++. Possibly even dave_todo++?? ;) Or could even be $<name here>_todo++ :P
I find it impossible to not mention this when it seems apparent to me... force of habit :)
I personally suck at refactoring code - it is something I have been trying to get better at. Now that I understand khugepaged a code a lot better, I have been thinking for a long time to clean this file up, maybe someday I will gather the courage to do that : )
Quoted 2 lines
Cheers, Lorenzo
Re: [PATCH v2 2/3] khugepaged: Optimize collapse_pte_mapped_thp() for large folios by PTE batching
Quoted 28 lines · Dev Jain
Quoted 26 lines · Lorenzo Stoakes
Quoted 13 lines · David Hildenbrand
Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-1- if (nr_ptes) { + if (nr_mapped_ptes) {I know it's ironic coming from me :P but I'm not sure why we need to churn this up by renaming?
Because
nr_ptesis an existing variable and I need a new variable to make the jump at the end of the PTE batch.I thought you eliminated
nr_ptesas a variable here? Where else is it used?Oh how this code needs refactoring...
If we retain
nr_ptes, then the two variables will benr_ptesandnr_mapped_ptes, which is confusing since the former is plain and the latter has a _mapped_ thingy in it, so instead now we call themnr_batch_ptesandnr_mapped_ptes.Quoted 7 lines
Quoted 4 lines
Sigh, this is still awful. But probably just existing awfulness. This whole thing needs a tent thrown over it and fumigation... but again not your fault :)
I mean fine, this is fine then.
Probably best to be pragmatic here: as long as the educated reader understands the code, all good. I hope I'll never have to explain it to my daughters.
:)
To be crystal clear here - Dev - I am not asking you to refactor the existing mess, and obviously I concede your point on the issue raised here.
Quoted 7 lines
Quoted 1 line
Did you need to respin other things here? If not I can relook and give tags.
Yes, I will have to respin because of
folio_pte_batch()changes from David and have to add a simplification patch to this series authored by David, I was just doing that and was reading the email thread to see if I didn't miss any review comment and then I noticed I forgot to reply to the above point.
Cool thanks :)
Will have a look on respin!
Quoted 15 lines
Quoted 15 lines
Quoted 4 lines
I know, I'm a perfectionist myself ;)
I mean if I wanted perfection I'd have run screaming from the kernel right at the start :P
But in general on the refactoring point - I think this code is hard to understand even for the informed, the endless nesting and nesting and levels of 'you just have to know what's happening' make it really easy to miss details here.
What I'm saying is - we can improve on this, and really all I'm doing is todo++. Possibly even dave_todo++?? ;) Or could even be $<name here>_todo++ :P
Quoted 7 lines
Quoted 2 lines
I find it impossible to not mention this when it seems apparent to me... force of habit :)
I personally suck at refactoring code - it is something I have been trying to get better at. Now that I understand khugepaged a code a lot better, I have been thinking for a long time to clean this file up, maybe someday I will gather the courage to do that : )
:))
Quoted 3 lines
Quoted 3 lines
Cheers, Lorenzo
Dev Jain
reviewerPATCH v2 3/3 khugepaged: Reduce race probability between migration and khugepaged
In reply to Dev JainSuppose a folio is under migration, and khugepaged is also trying to collapse it. collapse_pte_mapped_thp() will retrieve the folio from the page cache via filemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping -> folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
--- mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
mm/khugepaged.c +10 -2
@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,
@@ -956,6 +957,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))
@@ -1518,9 +1521,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/*
@@ -2745,6 +2751,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to
@@ -2834,6 +2841,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO :
Signature · 1 line
2.30.2
Re: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 10 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Is there a Fixes, Closes, etc.? Do we need something in stable?
Quoted 61 lines
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +10
-3@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -956,6 +957,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1518,9 +1521,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/* @@ -2745,6 +2751,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2834,6 +2841,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO : --2.30.2
Dev Jain
reviewerRe: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.
Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 2 lines
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
For some previous context, you may look at:
https://lore.kernel.org/all/[email protected]/
and
https://lore.kernel.org/all/[email protected]/
Quoted 61 lines
Quoted 61 lines
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)mm/khugepaged.c +10
-3@@ -31,6 +31,7 @@ enum scan_result {SCAN_FAIL ,SCAN_SUCCEED ,SCAN_PMD_NULL , +SCAN_PMD_MIGRATION ,SCAN_PMD_NONE ,SCAN_PMD_MAPPED , SCAN_EXCEED_NONE_PTE,@@ -956,6 +957,8 @@ static inline int check_pmd_state(pmd_t *pmd)if (pmd_none (pmde))return SCAN_PMD_NONE ; +if (is_pmd_migration_entry (pmde)) +return SCAN_PMD_MIGRATION ;if (!pmd_present (pmde))return SCAN_PMD_NULL ;if (pmd_trans_huge (pmde))@@ -1518,9 +1521,12 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, !range_in_vma(vma, haddr, haddr + HPAGE_PMD_SIZE))return SCAN_VMA_CHECK ; -/* Fast check before locking page if already PMD-mapped */ +/* +* Fast check before locking folio if already PMD-mapped, or if the +* folio is under migration +*/ result =find_pmd_or_thp_or_none (mm, haddr, &pmd); -if (result ==SCAN_PMD_MAPPED ) +if (result ==SCAN_PMD_MAPPED || result ==SCAN_PMD_MIGRATION )return result;/* @@ -2745,6 +2751,7 @@ static int madvise_collapse_errno(enum scan_result r)case SCAN_PAGE_LRU :case SCAN_DEL_PAGE_LRU :case SCAN_PAGE_FILLED : +case SCAN_PMD_MIGRATION :return -EAGAIN ;/* * Other: Trying again likely not to succeed / error intrinsic to @@ -2834,6 +2841,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, goto handle_result;/* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL : +case SCAN_PMD_MIGRATION :case SCAN_PTE_NON_PRESENT :case SCAN_PTE_UFFD_WP :case SCAN_PAGE_RO : --2.30.2
Re: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 18 lines · Dev Jain
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Does it rely on previous patches? If not probably best to send this one separately :)
Dev Jain
reviewerRe: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 16 lines · Lorenzo Stoakes
Quoted 15 lines · Dev Jain
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Does it rely on previous patches? If not probably best to send this one separately :)
To prevent rebasing headaches for others (if any) I thought to send all together. I'll send it separately if still that is the preference.
Re: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 23 lines · Dev Jain
Quoted 21 lines · Lorenzo Stoakes
Quoted 20 lines · Dev Jain
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Quoted 7 lines
Quoted 2 lines
Does it rely on previous patches? If not probably best to send this one separately :)
To prevent rebasing headaches for others (if any) I thought to send all together. I'll send it separately if still that is the preference.
Oh actually would it be a pain to rebase given the previous 2 patches? Maybe leave it then. And I can actually finally review it... :)
Dev Jain
reviewerRe: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 23 lines · Lorenzo Stoakes
Quoted 22 lines · Dev Jain
Quoted 21 lines · Lorenzo Stoakes
Quoted 20 lines · Dev Jain
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Does it rely on previous patches? If not probably best to send this one separately :)
To prevent rebasing headaches for others (if any) I thought to send all together. I'll send it separately if still that is the preference.
Oh actually would it be a pain to rebase given the previous 2 patches? Maybe
Didn't take the time to actually check that, more of a "it *may* be a pain so let's send this together".
Quoted 1 line
leave it then. And I can actually finally review it... :)
Re: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
In reply to Lorenzo StoakesQuoted 27 lines · Lorenzo Stoakes
Quoted 26 lines · Dev Jain
Quoted 23 lines · Lorenzo Stoakes
Quoted 12 lines · Dev Jain
Quoted 11 lines · Lorenzo Stoakes
Quoted 10 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Quoted 8 lines
Quoted 1 line
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Does it rely on previous patches? If not probably best to send this one separately :)
Quoted 7 lines
Quoted 4 lines
To prevent rebasing headaches for others (if any) I thought to send all together. I'll send it separately if still that is the preference.
Oh actually would it be a pain to rebase given the previous 2 patches? Maybe leave it then. And I can actually finally review it... :)
At the same time you'd have to update cover letter to mention and it's really not good practice to include unrelated patches to a series.
So probably best overall to send separately but _wait_ for this series to be merged first :>) then you can based it on mm-new without problems.
Dev Jain
reviewerRe: [PATCH v2 3/3] khugepaged: Reduce race probability between migration and khugepaged
Quoted 30 lines · Lorenzo Stoakes
Quoted 29 lines · Lorenzo Stoakes
Quoted 28 lines · Dev Jain
Quoted 16 lines · Lorenzo Stoakes
Quoted 15 lines · Dev Jain
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · Dev Jain
Suppose a folio is under migration, and khugepaged is also trying to collapse it.
collapse_pte_mapped_thp()will retrieve the folio from the page cache viafilemap_lock_folio(), thus taking a reference on the folio and sleeping on the folio lock, since the lock is held by the migration path. Migration will then fail in__folio_migrate_mapping->folio_ref_freeze. Reduce the probability of such a race happening (leading to migration failure) by bailing out if we detect a PMD is marked with a migration entry.Quoted 2 lines
Quoted 1 line
This fixes the migration-shared-anon-thp testcase failure on Apple M3.
Hm is this related to the series at all? Seems somewhat unrelated?
Not related.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Is there a Fixes, Closes, etc.? Do we need something in stable?
We don't need anything. This is an "expected race" in the sense that both migration and khugepaged collapse are best effort algorithms. I am just seeing a test failure on my system because my system hits the race more often. So this patch reduces the window for the race.
Does it rely on previous patches? If not probably best to send this one separately :)
To prevent rebasing headaches for others (if any) I thought to send all together. I'll send it separately if still that is the preference.
Quoted 7 lines
Quoted 2 lines
Oh actually would it be a pain to rebase given the previous 2 patches? Maybe leave it then. And I can actually finally review it... :)
At the same time you'd have to update cover letter to mention and it's really not good practice to include unrelated patches to a series.
So probably best overall to send separately but _wait_ for this series to be merged first :>) then you can based it on mm-new without problems.
Sure.
Quoted 2 lines · Dev Jain
This patch series uses PTE batching to optimize two functions, and also addresses a race condition.
Kinda succinct, be nice to expand a little. I know it can end up being a little duplicative, but I think more information is always better than less (as long as not truly overwrought)
Quoted 10 lines
---
@David I did not make theclear_full_ptesclean up since that will require some more work than I originally thought :)
v1->v2:
- Use for loop instead of do-while loop (Lorenzo)
- Removefolio_test_largecheck since the subpage-check condition will imply that (Baolin)
- Combine patch 1 and 2 into this series, add new patch 3
One small nit, it'd be really handy to include links to previous series to help us find them :P
As this one combines a couple earlier (which is nice!) but handy just for comparison's sake :>)
Quoted 14 lines
Dev Jain (3):
khugepaged: Optimize__collapse_huge_page_copy_succeeded()by PTE
batching
khugepaged: Optimizecollapse_pte_mapped_thp()for large folios by PTE
batching
khugepaged: Reduce race probability between migration and khugepagedmm/khugepaged.c | 77 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 57 insertions(+), 20 deletions(-)--
2.30.2