khugepaged: Optimize __collapse_huge_page_copy_succeeded() for large folios by PTE batching
Subsystems (2)
Dev Jain
reviewerPATCH 1/1 khugepaged: Optimize __collapse_huge_page_copy_succeeded() for large folios by PTE batching
Use PTE batching to optimize __collapse_huge_page_copy_succeeded().
On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.
For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().
No issues were observed with mm-selftests.
--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +23 -8
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +bool is_large; +struct page *src_page =pte_page (pteval); src =page_folio (src_page); -if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (is_large && max_nr_ptes !=1 ) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, false); +folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } - } + }while (_pte += nr_ptes, address += nr_ptes *PAGE_SIZE , address != end);list_for_each_entry_safe (src, tmp, compound_pagelist, lru) {list_del (&src->lru);
Signature · 1 line
2.30.2
Quoted 42 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)mm/khugepaged.c +5
-3@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 726
Quoted 6 lines
mm/khugepaged.c +3
-0@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +bool is_large;
folio_test_large() should be cheap, no need for the temporary variable (the compiler will likely optimize this either way).
Quoted 4 lines
mm/khugepaged.c +1
-0+struct page *src_page =pte_page (pteval); src =page_folio (src_page);
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 739
Quoted 10 lines
mm/khugepaged.c +8
-1-if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src); + + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (is_large && max_nr_ptes !=1 ) + nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL );
Starting to wonder if we want a simplified, non-inlined version of folio_pte_batch() in mm/util.c (e.g., without the 3 NULL parameters), renaming existing folio_pte_batch to __folio_pte_batch() and only using it where required (performance like in fork/zap, or because the other parameters are relevant).
Let me see if I find time for a quick patch later. Have to look at what other similar code needs.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 746
Quoted 7 lines
mm/khugepaged.c +1
-0+/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl);
Existing code: The PTL locking should just be moved outside of the loop.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 3 lines
mm/khugepaged.c +1
-2-ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, false);
Starting to wonder if we want a shortcut
#define clear_ptes(__mm, __addr, __pte, __nr_ptes) \ clear_full_ptes(__mm, __addr, __pte, __nr_ptes, false)
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 756
Quoted 11 lines
mm/khugepaged.c +4
-2+folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } - } + }while (_pte += nr_ptes, address += nr_ptes *PAGE_SIZE , address != end);list_for_each_entry_safe (src, tmp, compound_pagelist, lru) {list_del (&src->lru);
I think this should just work.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Dev Jain
+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* insidefolio_remove_rmap_pte(). */
spin_lock(ptl);Existing code: The PTL locking should just be moved outside of the loop.
Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dev Jain
+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* insidefolio_remove_rmap_pte(). */
spin_lock(ptl);
Quoted 5 lines
Quoted 1 line
Existing code: The PTL locking should just be moved outside of the loop.
Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?
It's a very weird thing to not hold the PTL while walking page tables, and then only grabbing it for clearing entries just to make selected functions happy ...
I mostly spotted the release_pte_folio(), which I think should be fine with a spinlock held. I missed the free_folio_and_swap_cache(), not sure if that is problematic.
Interestingly, release_pte_folio() does a
a) node_stat_mod_folio
b) folio_unlock
c) folio_putback_lru
... and folio_putback_lru() is documented to "lru_lock must not be held, interrupts must be enabled". Hmmmm. I suspect that doc is wrong.
So yeah, maybe better keep that weird looking locking like it is :)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dev Jain
+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* insidefolio_remove_rmap_pte(). */
spin_lock(ptl);Quoted 5 lines
Quoted 1 line
Existing code: The PTL locking should just be moved outside of the loop.
Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?
It's a very weird thing to not hold the PTL while walking page tables, and then only grabbing it for clearing entries just to make selected functions happy ...
I mostly spotted the
release_pte_folio(), which I think should be fine with a spinlock held. I missed thefree_folio_and_swap_cache(), not sure if that is problematic.
Quoted 8 lines
Interestingly,
release_pte_folio()does aa)
node_stat_mod_folio
b)folio_unlock
c)folio_putback_lru... and
folio_putback_lru()is documented to "lru_lockmust not be held, interrupts must be enabled". Hmmmm. I suspect that doc is wrong.
It's late ... doc is correct. Was reading "must be held".
So yeah, let's leave it as is.
Signature · 3 lines
Cheers, David / dhildenb
Dev Jain
reviewerQuoted 45 lines · David Hildenbrand
Quoted 44 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>---
mm/khugepaged.c| 31 +++++++++++++++++++++++--------1 file changed, 23 insertions(+), 8 deletions(-)mm/khugepaged.c +5
-2@@ -700,12 +700,14 @@ static void__collapse_huge_page_copy_succeeded (pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE; struct folio *src, *tmp; - pte_t *_pte; + pte_t *_pte = pte; pte_t pteval; + int nr_ptes; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; - _pte++, address += PAGE_SIZE) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);mm/khugepaged.c +0
-0@@ -719,23 +721,36 @@ static void__collapse_huge_page_copy_succeeded (pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval);
Quoted 8 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 726
Quoted 6 lines
mm/khugepaged.c +3
-0} } else { + const fpb_t flags = FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; + int max_nr_ptes; + bool is_large;
folio_test_large()should be cheap, no need for the temporary variable (the compiler will likely optimize this either way).
Okay.
Quoted 17 lines
Quoted 4 lines
mm/khugepaged.c +1
-0+ struct page *src_page = pte_page(pteval); src = page_folio(src_page);mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 739
Quoted 10 lines
mm/khugepaged.c +8
-1- if (!folio_test_large(src)) + is_large = folio_test_large(src); + if (!is_large) release_pte_folio(src); + + max_nr_ptes = (end - address) >> PAGE_SHIFT; + if (is_large && max_nr_ptes != 1) + nr_ptes = folio_pte_batch(src, address, _pte, + pteval, max_nr_ptes, + flags, NULL, NULL, NULL);Starting to wonder if we want a simplified, non-inlined version of
folio_pte_batch()inmm/util.c(e.g., without the 3 NULL parameters), renaming existingfolio_pte_batchto__folio_pte_batch()and only using it where required (performance like in fork/zap, or because the other parameters are relevant).
Quoted 1 line
Let me see if I find time for a quick patch later. Have to look at what other similar code needs.
Perhaps that version can also have the default fpb_flags ignoring dirty and soft-dirty, since that is what
most code will do. So the wrapper can pass which flags to remove.
Quoted 11 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 746
Quoted 8 lines
mm/khugepaged.c +1
-0+ /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock(ptl);Existing code: The PTL locking should just be moved outside of the loop.
Quoted 8 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 3 lines
mm/khugepaged.c +1
-2- ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src, src_page, vma); + clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);Starting to wonder if we want a shortcut
#define
clear_ptes(__mm, __addr, __pte, __nr_ptes)\
clear_full_ptes(__mm,__addr,__pte,__nr_ptes, false)
Thanks for the suggestion! I will definitely do this cleanup as part of this
series. It is very confusing and if someone does not know, it becomes hard
to find the batched version of ptep_clear, because this does not follow the
convention we have right now - ptep_set_wrprotect -> wrprotect_ptes and
so on, the "full" comes out of nowhere.
Quoted 15 lines
Quoted 11 lines
mm/khugepaged.c +4
-2+ folio_remove_rmap_ptes(src, src_page, nr_ptes, vma); spin_unlock(ptl); - free_folio_and_swap_cache(src); + free_swap_cache(src); + folio_put_refs(src, nr_ptes); } - } + } while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end); list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {Quoted 1 line
list_del(&src->lru);I think this should just work.
Quoted 24 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;
I assume we always enter here with aligned address...
Quoted 14 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723
Quoted 3 lines
mm/khugepaged.c +0
-0@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else {
Existing code but hate this level of indentation.
The code before was (barely) sort of ok-ish, but now it's realyl out of hand.
On the other hand, I look at __collapse_huge_page_isolate() and want to cry so I guess this maybe is something that needs addressing outside of this patch.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733
Quoted 11 lines
mm/khugepaged.c +6
-1+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +bool is_large; +struct page *src_page =pte_page (pteval); src =page_folio (src_page); -if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src);
Hm, in this case right, release_pte_folio() does a folio_unlock().
Where does a large folio get unlocked?
I mean this must have been existing code because I don't see where this happens previously either.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736
Quoted 3 lines
mm/khugepaged.c +3
-0+ + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (is_large && max_nr_ptes !=1 )
Is it really that harmful if max_nr_ptes == 1? Doesn't folio_pte_batch() figure it out?
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740
Quoted 4 lines
mm/khugepaged.c +4
-0+ nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +
It'd be nice(r) if this was:
if (folio_test_large(src))
nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); else release_pte_folio(src);
But even that is horrid because of the asymmetry.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 9 lines
mm/khugepaged.c +1
-2/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, false);
Be nice to use 'Liam's convention' of sticking /* full = */ false) on the end here so we know what the false refers to.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748
Quoted 1 line
mm/khugepaged.c +1
-0+folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);
Kinda neat that folio_remove_map_pte() is jus ta define onto this with nr_ptes == 1 :)
Quoted 11 lines
mm/khugepaged.c +3
-3spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } - } + }while (_pte += nr_ptes, address += nr_ptes *PAGE_SIZE , address != end);list_for_each_entry_safe (src, tmp, compound_pagelist, lru) {list_del (&src->lru); --
Quoted 2 lines
2.30.2
I can't see much wrong with this though, just 'yuck' at existing code really :)
Quoted 72 lines · Lorenzo Stoakes
Quoted 24 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)mm/khugepaged.c +0
-0mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703
Quoted 4 lines
mm/khugepaged.c +1
-0@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;I assume we always enter here with aligned address...
Quoted 14 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723
Quoted 3 lines
mm/khugepaged.c +0
-0@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else {Existing code but hate this level of indentation.
The code before was (barely) sort of ok-ish, but now it's realyl out of hand.
On the other hand, I look at
__collapse_huge_page_isolate()and want to cry so I guess this maybe is something that needs addressing outside of this patch.mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733
Quoted 11 lines
mm/khugepaged.c +6
-1+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +bool is_large; +struct page *src_page =pte_page (pteval); src =page_folio (src_page); -if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src);Hm, in this case right,
release_pte_folio()does afolio_unlock().
Quoted 1 line
Where does a large folio get unlocked?
Through the "compound_pagelist" below. ... this code is so ugly.
"large_folio_list" ...
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 1 line · Lorenzo Stoakes
Where does a large folio get unlocked?
Through the "
compound_pagelist" below. ... this code is so ugly."
large_folio_list" ...
Yeah I did wonder if that was what it was... yuck indeed!
Dev Jain
reviewerQuoted 20 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)
Quoted 10 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703
Quoted 9 lines
mm/khugepaged.c +1
-0@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;I assume we always enter here with aligned address...
Yes.
Quoted 20 lines
Quoted 15 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723
Quoted 3 lines
mm/khugepaged.c +0
-0@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else {Existing code but hate this level of indentation.
Quoted 4 lines
The code before was (barely) sort of ok-ish, but now it's realyl out of hand.
On the other hand, I look at
__collapse_huge_page_isolate()and want to cry so I guess this maybe is something that needs addressing outside of this patch.
Trust me I have already cried a lot before while doing the khugepaged mTHP stuff :)
Quoted 17 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733
Quoted 13 lines
mm/khugepaged.c +6
-1+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +int max_nr_ptes; +bool is_large; +struct page *src_page =pte_page (pteval); src =page_folio (src_page); -if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src);Hm, in this case right,
release_pte_folio()does afolio_unlock().Where does a large folio get unlocked?
Quoted 8 lines
I mean this must have been existing code because I don't see where this happens previously either.
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736
Quoted 3 lines
mm/khugepaged.c +3
-0+ + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (is_large && max_nr_ptes !=1 )Is it really that harmful if
max_nr_ptes== 1? Doesn'tfolio_pte_batch()figure it out?
Yup it will figure that out, was just following the pattern of zap_present_ptes and copy_present_ptes. Will drop this.
Quoted 16 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740
Quoted 5 lines
mm/khugepaged.c +4
-0+ nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +It'd be nice(r) if this was:
if (
folio_test_large(src))nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); else release_pte_folio(src);But even that is horrid because of the asymmetry.
Quoted 11 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 9 lines
mm/khugepaged.c +1
-2/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, false);Be nice to use 'Liam's convention' of sticking
/* full = */ false)on the end here so we know what the false refers to.
Sounds good, although in the other mail David mentioned a way to elide this so I will prefer that.
Quoted 20 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748
Quoted 2 lines
mm/khugepaged.c +1
-0+folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);Kinda neat that
folio_remove_map_pte()is jus ta define onto this withnr_ptes== 1 :)Quoted 11 lines
mm/khugepaged.c +3
-3spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } - } + }while (_pte += nr_ptes, address += nr_ptes *PAGE_SIZE , address != end);list_for_each_entry_safe (src, tmp, compound_pagelist, lru) {list_del (&src->lru); --Quoted 2 lines
2.30.2
I can't see much wrong with this though, just 'yuck' at existing code really :)
Quoted 33 lines · Dev Jain
Quoted 20 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain
Use PTE batching to optimize
__collapse_huge_page_copy_succeeded().On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling
ptep_clear()for every pte will cause a TLB flush for every contpte block. Instead,clear_full_ptes()does acontpte_try_unfold_partial()which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.For all arches, there should be a benefit due to batching atomic operations on mapcounts due to
folio_remove_rmap_ptes().No issues were observed with mm-selftests.
Signed-off-by Dev Jain <[email protected]>--- mm/khugepaged.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)Quoted 10 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703
Quoted 9 lines
mm/khugepaged.c +1
-0@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end = address + HPAGE_PMD_SIZE;I assume we always enter here with aligned address...
Quoted 1 line
Yes.
OK cool would be weird otherwise :)
Quoted 21 lines
Quoted 21 lines
Quoted 16 lines
mm/khugepaged.c +4
-3struct folio *src, *tmp; -pte_t *_pte; +pte_t *_pte = pte;pte_t pteval; +int nr_ptes; -for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; - _pte++, address +=PAGE_SIZE ) { + do { + nr_ptes = 1; pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723
Quoted 3 lines
mm/khugepaged.c +0
-0@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,ksm_might_unmap_zero_page (vma->vm_mm, pteval); } }else {Existing code but hate this level of indentation.
Quoted 6 lines
Quoted 4 lines
The code before was (barely) sort of ok-ish, but now it's realyl out of hand.
On the other hand, I look at
__collapse_huge_page_isolate()and want to cry so I guess this maybe is something that needs addressing outside of this patch.Trust me I have already cried a lot before while doing the khugepaged mTHP stuff :)
Seems we all cry together about this code ;)
Quoted 21 lines
Quoted 21 lines
Quoted 4 lines
mm/khugepaged.c +1
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733
Quoted 10 lines
mm/khugepaged.c +5
-1+int max_nr_ptes; +bool is_large; +struct page *src_page =pte_page (pteval); src =page_folio (src_page); -if (!folio_test_large (src)) + is_large =folio_test_large (src); +if (!is_large)release_pte_folio (src);Hm, in this case right,
release_pte_folio()does afolio_unlock().Where does a large folio get unlocked?
I mean this must have been existing code because I don't see where this happens previously either.
Quoted 8 lines
Quoted 5 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736
Quoted 3 lines
mm/khugepaged.c +3
-0+ + max_nr_ptes = (end - address) >>PAGE_SHIFT ; +if (is_large && max_nr_ptes !=1 )Is it really that harmful if
max_nr_ptes== 1? Doesn'tfolio_pte_batch()figure it out?Yup it will figure that out, was just following the pattern of
zap_present_ptesandcopy_present_ptes. Will drop this.
Thanks
Quoted 29 lines
Quoted 17 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740
Quoted 6 lines
mm/khugepaged.c +4
-0+ nr_ptes =folio_pte_batch (src, address, _pte, + pteval, max_nr_ptes, + flags,NULL ,NULL ,NULL ); +It'd be nice(r) if this was:
if (
folio_test_large(src))nr_ptes = folio_pte_batch(src, address, _pte, pteval, max_nr_ptes, flags, NULL, NULL, NULL); else release_pte_folio(src);But even that is horrid because of the asymmetry.
Quoted 11 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747
Quoted 9 lines
mm/khugepaged.c +1
-2/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock (ptl); -ptep_clear (vma->vm_mm, address, _pte); -folio_remove_rmap_pte (src, src_page, vma); +clear_full_ptes (vma->vm_mm, address, _pte, nr_ptes, false);Be nice to use 'Liam's convention' of sticking
/* full = */ false)on the end here so we know what the false refers to.
Quoted 2 lines
Sounds good, although in the other mail David mentioned a way to elide this so I will prefer that.
OK
Quoted 21 lines
Quoted 21 lines
mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748
Quoted 3 lines
mm/khugepaged.c +1
-0+folio_remove_rmap_ptes (src, src_page, nr_ptes, vma);Kinda neat that
folio_remove_map_pte()is jus ta define onto this withnr_ptes== 1 :)Quoted 11 lines
mm/khugepaged.c +3
-3spin_unlock (ptl); -free_folio_and_swap_cache (src); +free_swap_cache (src); +folio_put_refs (src, nr_ptes); } - } + }while (_pte += nr_ptes, address += nr_ptes *PAGE_SIZE , address != end);list_for_each_entry_safe (src, tmp, compound_pagelist, lru) {list_del (&src->lru); --Quoted 2 lines
2.30.2
I can't see much wrong with this though, just 'yuck' at existing code really :)