Skip to contentSkip to conversation

khugepaged: Optimize __collapse_huge_page_copy_succeeded() for large folios by PTE batching

PATCH Superseded Dev Jain 3 participants 11 messages Jun 18, 2025 – Jun 19, 2025

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Dev Jain

author reviewer
#0 lore raw

PATCH 1/1 khugepaged: Optimize __collapse_huge_page_copy_succeeded() for large folios by PTE batching

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
 mm/khugepaged.c | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +23 -8
index d45d08b521f6..649ccb2670f8 100644
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 						spinlock_t *ptl,
 						struct list_head *compound_pagelist)
 {
+	unsigned long end = address + HPAGE_PMD_SIZE;
 	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
 	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
 		pteval = ptep_get(_pte);
 		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
 			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
 			}
 		} else {
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
+			int max_nr_ptes;
+			bool is_large;
+
 			struct page *src_page = pte_page(pteval);
 
 			src = page_folio(src_page);
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
 				release_pte_folio(src);
+
+			max_nr_ptes = (end - address) >> PAGE_SHIFT;
+			if (is_large && max_nr_ptes != 1)
+				nr_ptes = folio_pte_batch(src, address, _pte,
+							  pteval, max_nr_ptes,
+							  flags, NULL, NULL, NULL);
+
 			/*
 			 * ptl mostly unnecessary, but preempt has to
 			 * be disabled to update the per-cpu stats
 			 * inside folio_remove_rmap_pte().
 			 */
 			spin_lock(ptl);
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);
 			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
 		}
-	}
+	} while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end);
 
 	list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
 		list_del(&src->lru);
Signature · 1 line
2.30.2
#1 lore raw
Quoted 42 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
  mm/khugepaged.c | 31 +++++++++++++++++++++++--------
  1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +5 -3
index d45d08b521f6..649ccb2670f8 100644
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  						spinlock_t *ptl,
  						struct list_head *compound_pagelist)
  {
+	unsigned long end = address + HPAGE_PMD_SIZE;
  	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
  	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
  		pteval = ptep_get(_pte);
  		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
  			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 726

Quoted 6 lines
mm/khugepaged.c +3 -0
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
  			}
  		} else {
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
+			int max_nr_ptes;
+			bool is_large;

folio_test_large() should be cheap, no need for the temporary variable (the compiler will likely optimize this either way).

Quoted 4 lines
mm/khugepaged.c +1 -0
+
  			struct page *src_page = pte_page(pteval);
 
  			src = page_folio(src_page);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 739

Quoted 10 lines
mm/khugepaged.c +8 -1
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
  				release_pte_folio(src);
+
+			max_nr_ptes = (end - address) >> PAGE_SHIFT;
+			if (is_large && max_nr_ptes != 1)
+				nr_ptes = folio_pte_batch(src, address, _pte,
+							  pteval, max_nr_ptes,
+							  flags, NULL, NULL, NULL);

Starting to wonder if we want a simplified, non-inlined version of folio_pte_batch() in mm/util.c (e.g., without the 3 NULL parameters), renaming existing folio_pte_batch to __folio_pte_batch() and only using it where required (performance like in fork/zap, or because the other parameters are relevant).

Let me see if I find time for a quick patch later. Have to look at what other similar code needs.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 746

Quoted 7 lines
mm/khugepaged.c +1 -0
+
  			/*
  			 * ptl mostly unnecessary, but preempt has to
  			 * be disabled to update the per-cpu stats
  			 * inside folio_remove_rmap_pte().
  			 */
  			spin_lock(ptl);

Existing code: The PTL locking should just be moved outside of the loop.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747

Quoted 3 lines
mm/khugepaged.c +1 -2
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);

Starting to wonder if we want a shortcut

#define clear_ptes(__mm, __addr, __pte, __nr_ptes) \
	clear_full_ptes(__mm, __addr, __pte, __nr_ptes, false)

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 756

Quoted 11 lines
mm/khugepaged.c +4 -2
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);
  			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
  		}
-	}
+	} while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end);
 
  	list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
  		list_del(&src->lru);

I think this should just work.

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Dev Jain

+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* inside folio_remove_rmap_pte(). */
spin_lock(ptl);

Existing code: The PTL locking should just be moved outside of the loop.

Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?

#3 lore raw
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dev Jain

+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* inside folio_remove_rmap_pte(). */
spin_lock(ptl);

Quoted 5 lines
Quoted 1 line

Existing code: The PTL locking should just be moved outside of the loop.

Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?

It's a very weird thing to not hold the PTL while walking page tables, and then only grabbing it for clearing entries just to make selected functions happy ...

I mostly spotted the release_pte_folio(), which I think should be fine with a spinlock held. I missed the free_folio_and_swap_cache(), not sure if that is problematic.

Interestingly, release_pte_folio() does a

a) node_stat_mod_folio
b) folio_unlock
c) folio_putback_lru

... and folio_putback_lru() is documented to "lru_lock must not be held, interrupts must be enabled". Hmmmm. I suspect that doc is wrong.

So yeah, maybe better keep that weird looking locking like it is :)

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw
Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dev Jain

+
/*
* ptl mostly unnecessary, but preempt has to
* be disabled to update the per-cpu stats
* inside folio_remove_rmap_pte(). */
spin_lock(ptl);

Quoted 5 lines
Quoted 1 line

Existing code: The PTL locking should just be moved outside of the loop.

Do we really want to hold the PTL for the duration of the loop? Are we sure it's safe to do so? Are there any locks taken in other functions that might sleep that'd mean holding a spinlock would be a problem?

It's a very weird thing to not hold the PTL while walking page tables, and then only grabbing it for clearing entries just to make selected functions happy ...

I mostly spotted the release_pte_folio(), which I think should be fine with a spinlock held. I missed the free_folio_and_swap_cache(), not sure if that is problematic.

Quoted 8 lines

Interestingly, release_pte_folio() does a

a) node_stat_mod_folio
b) folio_unlock
c) folio_putback_lru

... and folio_putback_lru() is documented to "lru_lock must not be held, interrupts must be enabled". Hmmmm. I suspect that doc is wrong.

It's late ... doc is correct. Was reading "must be held".

So yeah, let's leave it as is.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#5 lore raw
In reply to David Hildenbrand
Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
  mm/khugepaged.c | 31 +++++++++++++++++++++++--------

  1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +5 -2
index d45d08b521f6..649ccb2670f8 100644
@@ -700,12 +700,14 @@ static void 
 __collapse_huge_page_copy_succeeded(pte_t *pte,
                           spinlock_t *ptl,
                           struct list_head *compound_pagelist)
   {
+    unsigned long end = address + HPAGE_PMD_SIZE;
       struct folio *src, *tmp;
-    pte_t *_pte;
+    pte_t *_pte = pte;
       pte_t pteval;
+    int nr_ptes;
   -    for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-         _pte++, address += PAGE_SIZE) {
+    do {
+        nr_ptes = 1;
           pteval = ptep_get(_pte);
           if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {

              add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c +0 -0
@@ -719,23 +721,36 @@ static void 
 __collapse_huge_page_copy_succeeded(pte_t *pte,
                   ksm_might_unmap_zero_page(vma->vm_mm, pteval);
Quoted 8 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 726

Quoted 6 lines
mm/khugepaged.c +3 -0
               }
           } else {
+            const fpb_t flags = FPB_IGNORE_DIRTY | 
 FPB_IGNORE_SOFT_DIRTY;
+            int max_nr_ptes;
+            bool is_large;

folio_test_large() should be cheap, no need for the temporary variable (the compiler will likely optimize this either way).

Okay.

Quoted 17 lines
Quoted 4 lines
mm/khugepaged.c +1 -0
+
               struct page *src_page = pte_page(pteval);
                 src = page_folio(src_page);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 739

Quoted 10 lines
mm/khugepaged.c +8 -1
-            if (!folio_test_large(src))
+            is_large = folio_test_large(src);
+            if (!is_large)
                   release_pte_folio(src);
+
+            max_nr_ptes = (end - address) >> PAGE_SHIFT;
+            if (is_large && max_nr_ptes != 1)
+                nr_ptes = folio_pte_batch(src, address, _pte,
+                              pteval, max_nr_ptes,
+                              flags, NULL, NULL, NULL);

Starting to wonder if we want a simplified, non-inlined version of folio_pte_batch() in mm/util.c (e.g., without the 3 NULL parameters), renaming existing folio_pte_batch to __folio_pte_batch() and only using it where required (performance like in fork/zap, or because the other parameters are relevant).

Quoted 1 line

Let me see if I find time for a quick patch later. Have to look at what other similar code needs.

Perhaps that version can also have the default fpb_flags ignoring dirty and soft-dirty, since that is what

most code will do. So the wrapper can pass which flags to remove.

Quoted 11 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 746

Quoted 8 lines
mm/khugepaged.c +1 -0
+
               /*
                * ptl mostly unnecessary, but preempt has to
                * be disabled to update the per-cpu stats
                * inside folio_remove_rmap_pte().
                */
               spin_lock(ptl);

Existing code: The PTL locking should just be moved outside of the loop.

Quoted 8 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747

Quoted 3 lines
mm/khugepaged.c +1 -2
-            ptep_clear(vma->vm_mm, address, _pte);
-            folio_remove_rmap_pte(src, src_page, vma);
+            clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);

Starting to wonder if we want a shortcut

#define clear_ptes(__mm, __addr, __pte, __nr_ptes) \
    clear_full_ptes(__mm, __addr, __pte, __nr_ptes, false)

Thanks for the suggestion! I will definitely do this cleanup as part of this

series. It is very confusing and if someone does not know, it becomes hard

to find the batched version of ptep_clear, because this does not follow the

convention we have right now - ptep_set_wrprotect -> wrprotect_ptes and

so on, the "full" comes out of nowhere.

Quoted 15 lines
Quoted 11 lines
mm/khugepaged.c +4 -2
+            folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);
               spin_unlock(ptl);
-            free_folio_and_swap_cache(src);
+            free_swap_cache(src);
+            folio_put_refs(src, nr_ptes);
           }
-    }
+    } while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, 
 address != end);
         list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
Quoted 1 line

          list_del(&src->lru);

I think this should just work.

#6 lore raw
In reply to Dev Jain
Quoted 24 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
 mm/khugepaged.c | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +0 -0
index d45d08b521f6..649ccb2670f8 100644

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703

Quoted 4 lines
mm/khugepaged.c +1 -0
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 						spinlock_t *ptl,
 						struct list_head *compound_pagelist)
 {
+	unsigned long end = address + HPAGE_PMD_SIZE;

I assume we always enter here with aligned address...

Quoted 14 lines
mm/khugepaged.c +4 -3
 	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
 	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
 		pteval = ptep_get(_pte);
 		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
 			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723

Quoted 3 lines
mm/khugepaged.c +0 -0
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
 			}
 		} else {

Existing code but hate this level of indentation.

The code before was (barely) sort of ok-ish, but now it's realyl out of hand.

On the other hand, I look at __collapse_huge_page_isolate() and want to cry so I guess this maybe is something that needs addressing outside of this patch.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733

Quoted 11 lines
mm/khugepaged.c +6 -1
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
+			int max_nr_ptes;
+			bool is_large;
+
 			struct page *src_page = pte_page(pteval);
 
 			src = page_folio(src_page);
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
 				release_pte_folio(src);

Hm, in this case right, release_pte_folio() does a folio_unlock().

Where does a large folio get unlocked?

I mean this must have been existing code because I don't see where this happens previously either.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736

Quoted 3 lines
mm/khugepaged.c +3 -0
+
+			max_nr_ptes = (end - address) >> PAGE_SHIFT;
+			if (is_large && max_nr_ptes != 1)

Is it really that harmful if max_nr_ptes == 1? Doesn't folio_pte_batch() figure it out?

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740

Quoted 4 lines
mm/khugepaged.c +4 -0
+				nr_ptes = folio_pte_batch(src, address, _pte,
+							  pteval, max_nr_ptes,
+							  flags, NULL, NULL, NULL);
+

It'd be nice(r) if this was:

if (folio_test_large(src))

	nr_ptes = folio_pte_batch(src, address, _pte,
		pteval, max_nr_ptes,
		flags, NULL, NULL, NULL);
else
	release_pte_folio(src);

But even that is horrid because of the asymmetry.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747

Quoted 9 lines
mm/khugepaged.c +1 -2
 			/*
 			 * ptl mostly unnecessary, but preempt has to
 			 * be disabled to update the per-cpu stats
 			 * inside folio_remove_rmap_pte().
 			 */
 			spin_lock(ptl);
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);

Be nice to use 'Liam's convention' of sticking /* full = */ false) on the end here so we know what the false refers to.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748

Quoted 1 line
mm/khugepaged.c +1 -0
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);

Kinda neat that folio_remove_map_pte() is jus ta define onto this with nr_ptes == 1 :)

Quoted 11 lines
mm/khugepaged.c +3 -3
 			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
 		}
-	}
+	} while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end);
 
 	list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
 		list_del(&src->lru);
--
Quoted 2 lines

2.30.2

I can't see much wrong with this though, just 'yuck' at existing code really :)

#7 lore raw
Quoted 72 lines · Lorenzo Stoakes
Quoted 24 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
  mm/khugepaged.c | 31 +++++++++++++++++++++++--------
  1 file changed, 23 insertions(+), 8 deletions(-)
mm/khugepaged.c +0 -0
index d45d08b521f6..649ccb2670f8 100644

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703

Quoted 4 lines
mm/khugepaged.c +1 -0
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  						spinlock_t *ptl,
  						struct list_head *compound_pagelist)
  {
+	unsigned long end = address + HPAGE_PMD_SIZE;

I assume we always enter here with aligned address...

Quoted 14 lines
mm/khugepaged.c +4 -3
  	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
  	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
  		pteval = ptep_get(_pte);
  		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
  			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723

Quoted 3 lines
mm/khugepaged.c +0 -0
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
  			}
  		} else {

Existing code but hate this level of indentation.

The code before was (barely) sort of ok-ish, but now it's realyl out of hand.

On the other hand, I look at __collapse_huge_page_isolate() and want to cry so I guess this maybe is something that needs addressing outside of this patch.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733

Quoted 11 lines
mm/khugepaged.c +6 -1
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
+			int max_nr_ptes;
+			bool is_large;
+
  			struct page *src_page = pte_page(pteval);
 
  			src = page_folio(src_page);
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
  				release_pte_folio(src);

Hm, in this case right, release_pte_folio() does a folio_unlock().

Quoted 1 line

Where does a large folio get unlocked?

Through the "compound_pagelist" below. ... this code is so ugly.

"large_folio_list" ...

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
Quoted 7 lines · David Hildenbrand
Quoted 1 line · Lorenzo Stoakes

Where does a large folio get unlocked?

Through the "compound_pagelist" below. ... this code is so ugly.

"large_folio_list" ...

Yeah I did wonder if that was what it was... yuck indeed!

author reviewer
#9 lore raw
In reply to Lorenzo Stoakes
Quoted 20 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
  mm/khugepaged.c | 31 +++++++++++++++++++++++--------
  1 file changed, 23 insertions(+), 8 deletions(-)
Quoted 10 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703

Quoted 9 lines
mm/khugepaged.c +1 -0
index d45d08b521f6..649ccb2670f8 100644
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  						spinlock_t *ptl,
  						struct list_head *compound_pagelist)
  {
+	unsigned long end = address + HPAGE_PMD_SIZE;

I assume we always enter here with aligned address...

Yes.

Quoted 20 lines
Quoted 15 lines
mm/khugepaged.c +4 -3
  	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
  	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
  		pteval = ptep_get(_pte);
  		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
  			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723

Quoted 3 lines
mm/khugepaged.c +0 -0
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
  			}
  		} else {

Existing code but hate this level of indentation.

Quoted 4 lines

The code before was (barely) sort of ok-ish, but now it's realyl out of hand.

On the other hand, I look at __collapse_huge_page_isolate() and want to cry so I guess this maybe is something that needs addressing outside of this patch.

Trust me I have already cried a lot before while doing the khugepaged mTHP stuff :)

Quoted 17 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733

Quoted 13 lines
mm/khugepaged.c +6 -1
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
+			int max_nr_ptes;
+			bool is_large;
+
  			struct page *src_page = pte_page(pteval);
 
  			src = page_folio(src_page);
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
  				release_pte_folio(src);

Hm, in this case right, release_pte_folio() does a folio_unlock().

Where does a large folio get unlocked?

Quoted 8 lines

I mean this must have been existing code because I don't see where this happens previously either.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736

Quoted 3 lines
mm/khugepaged.c +3 -0
+
+			max_nr_ptes = (end - address) >> PAGE_SHIFT;
+			if (is_large && max_nr_ptes != 1)

Is it really that harmful if max_nr_ptes == 1? Doesn't folio_pte_batch() figure it out?

Yup it will figure that out, was just following the pattern of zap_present_ptes and copy_present_ptes. Will drop this.

Quoted 16 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740

Quoted 5 lines
mm/khugepaged.c +4 -0
+				nr_ptes = folio_pte_batch(src, address, _pte,
+							  pteval, max_nr_ptes,
+							  flags, NULL, NULL, NULL);
+

It'd be nice(r) if this was:

if (folio_test_large(src))

	nr_ptes = folio_pte_batch(src, address, _pte,
		pteval, max_nr_ptes,
		flags, NULL, NULL, NULL);
else
	release_pte_folio(src);

But even that is horrid because of the asymmetry.

Quoted 11 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747

Quoted 9 lines
mm/khugepaged.c +1 -2
  			/*
  			 * ptl mostly unnecessary, but preempt has to
  			 * be disabled to update the per-cpu stats
  			 * inside folio_remove_rmap_pte().
  			 */
  			spin_lock(ptl);
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);

Be nice to use 'Liam's convention' of sticking /* full = */ false) on the end here so we know what the false refers to.

Sounds good, although in the other mail David mentioned a way to elide this so I will prefer that.

Quoted 20 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748

Quoted 2 lines
mm/khugepaged.c +1 -0
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);

Kinda neat that folio_remove_map_pte() is jus ta define onto this with nr_ptes == 1 :)

Quoted 11 lines
mm/khugepaged.c +3 -3
  			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
  		}
-	}
+	} while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end);
 
  	list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
  		list_del(&src->lru);
--
Quoted 2 lines

2.30.2

I can't see much wrong with this though, just 'yuck' at existing code really :)

#10 lore raw
Quoted 33 lines · Dev Jain
Quoted 20 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_full_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes().

No issues were observed with mm-selftests.

---
  mm/khugepaged.c | 31 +++++++++++++++++++++++--------
  1 file changed, 23 insertions(+), 8 deletions(-)
Quoted 10 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 703

Quoted 9 lines
mm/khugepaged.c +1 -0
index d45d08b521f6..649ccb2670f8 100644
@@ -700,12 +700,14 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  						spinlock_t *ptl,
  						struct list_head *compound_pagelist)
  {
+	unsigned long end = address + HPAGE_PMD_SIZE;

I assume we always enter here with aligned address...

Quoted 1 line

Yes.

OK cool would be weird otherwise :)

Quoted 21 lines
Quoted 21 lines
Quoted 16 lines
mm/khugepaged.c +4 -3
  	struct folio *src, *tmp;
-	pte_t *_pte;
+	pte_t *_pte = pte;
  	pte_t pteval;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	do {
+		nr_ptes = 1;
  		pteval = ptep_get(_pte);
  		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
  			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 723

Quoted 3 lines
mm/khugepaged.c +0 -0
@@ -719,23 +721,36 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  				ksm_might_unmap_zero_page(vma->vm_mm, pteval);
  			}
  		} else {

Existing code but hate this level of indentation.

Quoted 6 lines
Quoted 4 lines

The code before was (barely) sort of ok-ish, but now it's realyl out of hand.

On the other hand, I look at __collapse_huge_page_isolate() and want to cry so I guess this maybe is something that needs addressing outside of this patch.

Trust me I have already cried a lot before while doing the khugepaged mTHP stuff :)

Seems we all cry together about this code ;)

Quoted 21 lines
Quoted 21 lines
Quoted 4 lines
mm/khugepaged.c +1 -0
+			const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 733

Quoted 10 lines
mm/khugepaged.c +5 -1
+			int max_nr_ptes;
+			bool is_large;
+
  			struct page *src_page = pte_page(pteval);
 
  			src = page_folio(src_page);
-			if (!folio_test_large(src))
+			is_large = folio_test_large(src);
+			if (!is_large)
  				release_pte_folio(src);

Hm, in this case right, release_pte_folio() does a folio_unlock().

Where does a large folio get unlocked?

I mean this must have been existing code because I don't see where this happens previously either.

Quoted 8 lines
Quoted 5 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736

Quoted 3 lines
mm/khugepaged.c +3 -0
+
+			max_nr_ptes = (end - address) >> PAGE_SHIFT;
+			if (is_large && max_nr_ptes != 1)

Is it really that harmful if max_nr_ptes == 1? Doesn't folio_pte_batch() figure it out?

Yup it will figure that out, was just following the pattern of zap_present_ptes and copy_present_ptes. Will drop this.

Thanks

Quoted 29 lines
Quoted 17 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 740

Quoted 6 lines
mm/khugepaged.c +4 -0
+				nr_ptes = folio_pte_batch(src, address, _pte,
+							  pteval, max_nr_ptes,
+							  flags, NULL, NULL, NULL);
+

It'd be nice(r) if this was:

if (folio_test_large(src))

	nr_ptes = folio_pte_batch(src, address, _pte,
		pteval, max_nr_ptes,
		flags, NULL, NULL, NULL);
else
	release_pte_folio(src);

But even that is horrid because of the asymmetry.

Quoted 11 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 747

Quoted 9 lines
mm/khugepaged.c +1 -2
  			/*
  			 * ptl mostly unnecessary, but preempt has to
  			 * be disabled to update the per-cpu stats
  			 * inside folio_remove_rmap_pte().
  			 */
  			spin_lock(ptl);
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_full_ptes(vma->vm_mm, address, _pte, nr_ptes, false);

Be nice to use 'Liam's convention' of sticking /* full = */ false) on the end here so we know what the false refers to.

Quoted 2 lines

Sounds good, although in the other mail David mentioned a way to elide this so I will prefer that.

OK

Quoted 21 lines
Quoted 21 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 748

Quoted 3 lines
mm/khugepaged.c +1 -0
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);

Kinda neat that folio_remove_map_pte() is jus ta define onto this with nr_ptes == 1 :)

Quoted 11 lines
mm/khugepaged.c +3 -3
  			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
  		}
-	}
+	} while (_pte += nr_ptes, address += nr_ptes * PAGE_SIZE, address != end);
 
  	list_for_each_entry_safe(src, tmp, compound_pagelist, lru) {
  		list_del(&src->lru);
--
Quoted 2 lines

2.30.2

I can't see much wrong with this though, just 'yuck' at existing code really :)