Skip to contentSkip to conversation

Optimizations for khugepaged

PATCH Superseded Dev Jain 4 participants 12 messages Jul 22, 2025 – Jul 23, 2025

Subsystems (6)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Dev Jain

author reviewer
#0 lore raw

v3 cover Optimizations for khugepaged

If the underlying folio mapped by the ptes is large, we can process those ptes in a batch using folio_pte_batch().

For arm64 specifically, this results in a 16x reduction in the number of ptep_get() calls, since on a contig block, ptep_get() on arm64 will iterate through all 16 entries to collect a/d bits. Next, ptep_clear() will cause a TLBI for every contig block in the range via contpte_try_unfold(). Instead, use clear_ptes() to only do the TLBI at the first and last contig block of the range.

For split folios, there will be no pte batching; the batch size returned by folio_pte_batch() will be 1. For pagetable split folios, the ptes will still point to the same large folio; for arm64, this results in the optimization described above, and for other arches, a minor improvement is expected due to a reduction in the number of function calls and batching atomic operations.

---
Rebased on today's mm-new.

v2->v3:
- Drop patch 3 (was merged separately)
- Add patch 1 (David)
- Coding style change, drop mapped_folio (Lorenzo)

v1->v2:
- Use for loop instead of do-while loop (Lorenzo)
- Remove folio_test_large check since the subpage-check condition will imply that (Baolin)
- Combine patch 1 and 2 into this series, add new patch 3

David Hildenbrand (1):
mm: add get_and_clear_ptes() and clear_ptes()

Dev Jain (2):
khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE
batching
khugepaged: Optimize collapse_pte_mapped_thp() by PTE batching

 arch/arm64/mm/mmu.c     |  2 +-
 include/linux/pgtable.h |  6 +++++
 mm/khugepaged.c         | 57 +++++++++++++++++++++++++++--------------
 mm/mremap.c             |  2 +-
 mm/rmap.c               |  2 +-
 5 files changed, 47 insertions(+), 22 deletions(-)
Signature · 1 line
2.30.2
author reviewer
#1 lore raw

PATCH v3 1/3 mm: add get_and_clear_ptes() and clear_ptes()

From: David Hildenbrand <[email protected]>

From: David Hildenbrand <[email protected]>

Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next.

Should we make these inline functions instead and add separate docs? Probably not worth it for now.

---
 arch/arm64/mm/mmu.c     | 2 +-
 include/linux/pgtable.h | 6 ++++++
 mm/mremap.c             | 2 +-
 mm/rmap.c               | 2 +-
 4 files changed, 9 insertions(+), 3 deletions(-)
arch/arm64/mm/mmu.c +1 -1
index abd9725796e9..20a89ab97dc5 100644
@@ -1528,7 +1528,7 @@ early_initcall(prevent_bootmem_remove_init);
 pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t *ptep, unsigned int nr)
 {
-	pte_t pte = get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full = */ 0);
+	pte_t pte = get_and_clear_ptes(vma->vm_mm, addr, ptep, nr);
 
 	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) {
 		/*
include/linux/pgtable.h +6 -0
index e3b99920be05..e45986b54277 100644
@@ -736,6 +736,9 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
 }
 #endif
 
+#define get_and_clear_ptes(_mm, _addr, _ptep, _nr) \
+	get_and_clear_full_ptes(_mm, _addr, _ptep, _nr, 0)
+
 #ifndef clear_full_ptes
 /**
  * clear_full_ptes - Clear present PTEs that map consecutive pages of the same
@@ -768,6 +771,9 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
 }
 #endif
 
+#define clear_ptes(_mm, _addr, _ptep, _nr) \
+	clear_full_ptes(_mm, _addr, _ptep, _nr, 0)
+
 /*
  * If two threads concurrently fault at the same page, the thread that
  * won the race updates the PTE and its local TLB/Cache. The other thread
mm/mremap.c +1 -1
index ac39845e9718..677a4d744df9 100644
@@ -280,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
 							 old_pte, max_nr_ptes);
 			force_flush = true;
 		}
-		pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0);
+		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
 		pte = move_pte(pte, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
 
mm/rmap.c +1 -1
index f93ce27132ab..568198e9efc2 100644
@@ -2036,7 +2036,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			flush_cache_range(vma, address, end_addr);
 
 			/* Nuke the page table entry. */
-			pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0);
+			pteval = get_and_clear_ptes(mm, address, pvmw.pte, nr_pages);
 			/*
 			 * We clear the PTE but do not flush so potentially
 			 * a remote CPU could still be writing to the folio.
Signature · 1 line
2.30.2
maintainer
#2 lore raw

Re: [PATCH v3 1/3] mm: add get_and_clear_ptes() and clear_ptes()

Quoted 8 lines · On Tue, 22 Jul 2025 20:35:57 +0530 Dev Jain <[email protected]> wrote:

Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next.

Should we make these inline functions instead and add separate docs?

inlined functions would be preferable.

Quoted 1 line

Probably not worth it for now.

Well, as David has called for a v4 I suggest you make that change.

Thanks, I'll queue the series in mm-new for a bit of testing.

author reviewer
#3 lore raw

Re: [PATCH v3 1/3] mm: add get_and_clear_ptes() and clear_ptes()

Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Tue, 22 Jul 2025 20:35:57 +0530 Dev Jain <[email protected]> wrote:

Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.

Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next.

Quoted 7 lines
Quoted 1 line

Should we make these inline functions instead and add separate docs?

inlined functions would be preferable.

Quoted 1 line

Probably not worth it for now.

Well, as David has called for a v4 I suggest you make that change.

Thanks, I'll queue the series in mm-new for a bit of testing.

Thanks. I'll post v4 later in the day.

reviewer Reviewed-by on v3 1/3
#4 lore raw

Re: [PATCH v3 1/3] mm: add get_and_clear_ptes() and clear_ptes()

In reply to Dev Jain
Quoted 8 lines · Dev Jain

From: David Hildenbrand <[email protected]>

From: David Hildenbrand <[email protected]>

Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM.

Quoted 8 lines

Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next.

Should we make these inline functions instead and add separate docs? Probably not worth it for now.

LGTM.

Quoted 70 lines
---
  arch/arm64/mm/mmu.c     | 2 +-
  include/linux/pgtable.h | 6 ++++++
  mm/mremap.c             | 2 +-
  mm/rmap.c               | 2 +-
  4 files changed, 9 insertions(+), 3 deletions(-)
arch/arm64/mm/mmu.c +1 -1
index abd9725796e9..20a89ab97dc5 100644
@@ -1528,7 +1528,7 @@ early_initcall(prevent_bootmem_remove_init);
  pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
  			     pte_t *ptep, unsigned int nr)
  {
-	pte_t pte = get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full = */ 0);
+	pte_t pte = get_and_clear_ptes(vma->vm_mm, addr, ptep, nr);
 
  	if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) {
  		/*
include/linux/pgtable.h +6 -0
index e3b99920be05..e45986b54277 100644
@@ -736,6 +736,9 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
  }
  #endif
 
+#define get_and_clear_ptes(_mm, _addr, _ptep, _nr) \
+	get_and_clear_full_ptes(_mm, _addr, _ptep, _nr, 0)
+
  #ifndef clear_full_ptes
  /**
   * clear_full_ptes - Clear present PTEs that map consecutive pages of the same
@@ -768,6 +771,9 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
  }
  #endif
 
+#define clear_ptes(_mm, _addr, _ptep, _nr) \
+	clear_full_ptes(_mm, _addr, _ptep, _nr, 0)
+
  /*
   * If two threads concurrently fault at the same page, the thread that
   * won the race updates the PTE and its local TLB/Cache. The other thread
mm/mremap.c +1 -1
index ac39845e9718..677a4d744df9 100644
@@ -280,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
  							 old_pte, max_nr_ptes);
  			force_flush = true;
  		}
-		pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0);
+		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
  		pte = move_pte(pte, old_addr, new_addr);
  		pte = move_soft_dirty_pte(pte);
 
mm/rmap.c +1 -1
index f93ce27132ab..568198e9efc2 100644
@@ -2036,7 +2036,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
  			flush_cache_range(vma, address, end_addr);
 
  			/* Nuke the page table entry. */
-			pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0);
+			pteval = get_and_clear_ptes(mm, address, pvmw.pte, nr_pages);
  			/*
  			 * We clear the PTE but do not flush so potentially
  			 * a remote CPU could still be writing to the folio.
author reviewer
#5 lore raw

PATCH v3 2/3 khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching

In reply to Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

---
 mm/khugepaged.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
mm/khugepaged.c +18 -7
index a55fb1dcd224..63517ef7eafb 100644
@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 						spinlock_t *ptl,
 						struct list_head *compound_pagelist)
 {
+	unsigned long end = address + HPAGE_PMD_SIZE;
 	struct folio *src, *tmp;
-	pte_t *_pte;
 	pte_t pteval;
+	pte_t *_pte;
+	int nr_ptes;
 
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes,
+	     address += nr_ptes * PAGE_SIZE) {
+		nr_ptes = 1;
 		pteval = ptep_get(_pte);
 		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
 			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
@@ -722,18 +725,26 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
 			struct page *src_page = pte_page(pteval);
 
 			src = page_folio(src_page);
-			if (!folio_test_large(src))
+
+			if (folio_test_large(src)) {
+				int max_nr_ptes = (end - address) >> PAGE_SHIFT;
+
+				nr_ptes = folio_pte_batch(src, _pte, pteval, max_nr_ptes);
+			} else {
 				release_pte_folio(src);
+			}
+
 			/*
 			 * ptl mostly unnecessary, but preempt has to
 			 * be disabled to update the per-cpu stats
 			 * inside folio_remove_rmap_pte().
 			 */
 			spin_lock(ptl);
-			ptep_clear(vma->vm_mm, address, _pte);
-			folio_remove_rmap_pte(src, src_page, vma);
+			clear_ptes(vma->vm_mm, address, _pte, nr_ptes);
+			folio_remove_rmap_ptes(src, src_page, nr_ptes, vma);
 			spin_unlock(ptl);
-			free_folio_and_swap_cache(src);
+			free_swap_cache(src);
+			folio_put_refs(src, nr_ptes);
 		}
 	}
 
Signature · 1 line
2.30.2
Acked-by on v3 2/3
#6 lore raw

Re: [PATCH v3 2/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching

Quoted 7 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

I suggest not talking so much about arm specifics.

Simply say that batching reduced the number of TLB flushes, especially on architectures that support cont-pte optimizations.

Quoted 3 lines

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

I would rephrase that to "Independent of that, batching PTE unmapping has known performance benfits (i.e., less refcount and mapcount updates)".

Quoted 11 lines
---
  mm/khugepaged.c | 25 ++++++++++++++++++-------
  1 file changed, 18 insertions(+), 7 deletions(-)
mm/khugepaged.c +0 -0
index a55fb1dcd224..63517ef7eafb 100644

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 707

Quoted 9 lines
mm/khugepaged.c +3 -1
@@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  						spinlock_t *ptl,
  						struct list_head *compound_pagelist)
  {
+	unsigned long end = address + HPAGE_PMD_SIZE;
  	struct folio *src, *tmp;
-	pte_t *_pte;
  	pte_t pteval;
+	pte_t *_pte;
+	int nr_ptes;

Nit: I guess we should switch to "unsigned int" here now for consistency with folio_pte_batch().

Quoted 10 lines
mm/khugepaged.c +3 -2
-	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-	     _pte++, address += PAGE_SIZE) {
+	for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes,
+	     address += nr_ptes * PAGE_SIZE) {
+		nr_ptes = 1;
  		pteval = ptep_get(_pte);
  		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
  			add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 730

Quoted 7 lines
mm/khugepaged.c +3 -1
@@ -722,18 +725,26 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte,
  			struct page *src_page = pte_page(pteval);
 
  			src = page_folio(src_page);
-			if (!folio_test_large(src))
+
+			if (folio_test_large(src)) {
+				int max_nr_ptes = (end - address) >> PAGE_SHIFT;

Dito.

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736

Quoted 6 lines
mm/khugepaged.c +5 -0
+
+				nr_ptes = folio_pte_batch(src, _pte, pteval, max_nr_ptes);
+			} else {
  				release_pte_folio(src);
+			}
+
Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#7 lore raw

Re: [PATCH v3 2/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching

Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

Quoted 3 lines

I suggest not talking so much about arm specifics.

Simply say that batching reduced the number of TLB flushes, especially on architectures that support cont-pte optimizations.

Makes sense.

Quoted 6 lines
Quoted 4 lines

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

I would rephrase that to "Independent of that, batching PTE unmapping has known performance benfits (i.e., less refcount and mapcount updates)".

Thanks.

Quoted 16 lines
Quoted 16 lines

---
  mm/khugepaged.c | 25 ++++++++++++++++++-------

  1 file changed, 18 insertions(+), 7 deletions(-)
mm/khugepaged.c +0 -0
index a55fb1dcd224..63517ef7eafb 100644
@@ -700,12 +700,15 @@ static void 
 __collapse_huge_page_copy_succeeded(pte_t *pte,
                           spinlock_t *ptl,
                           struct list_head *compound_pagelist)
   {
Quoted 8 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 707

Quoted 6 lines
mm/khugepaged.c +3 -1
+    unsigned long end = address + HPAGE_PMD_SIZE;
       struct folio *src, *tmp;
-    pte_t *_pte;
       pte_t pteval;
+    pte_t *_pte;
+    int nr_ptes;

Nit: I guess we should switch to "unsigned int" here now for consistency with folio_pte_batch().

Okay.

Quoted 20 lines
Quoted 10 lines
mm/khugepaged.c +3 -1
   -    for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
-         _pte++, address += PAGE_SIZE) {
+    for (_pte = pte; _pte < pte + HPAGE_PMD_NR; _pte += nr_ptes,
+         address += nr_ptes * PAGE_SIZE) {
+        nr_ptes = 1;
           pteval = ptep_get(_pte);
           if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {

              add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);

mm/khugepaged.c static void · line 730

Quoted 7 lines
mm/khugepaged.c +3 -1
@@ -722,18 +725,26 @@ static void 
 __collapse_huge_page_copy_succeeded(pte_t *pte,
               struct page *src_page = pte_page(pteval);
                 src = page_folio(src_page);
-            if (!folio_test_large(src))
+
+            if (folio_test_large(src)) {
+                int max_nr_ptes = (end - address) >> PAGE_SHIFT;

Dito.

Quoted 9 lines

mm/khugepaged.c static void __collapse_huge_page_copy_succeeded(pte_t *pte, · line 736

Quoted 7 lines
mm/khugepaged.c +5 -0
+
+                nr_ptes = folio_pte_batch(src, _pte, pteval, 
 max_nr_ptes);
+            } else {
                   release_pte_folio(src);
+            }
+

Thanks.

reviewer Reviewed-by on v3 2/3
#8 lore raw

Re: [PATCH v3 2/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching

In reply to Dev Jain
Quoted 8 lines · Dev Jain

Use PTE batching to optimize __collapse_huge_page_copy_succeeded().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

Quoted 4 lines

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

With David's comments addressed, LGTM.

author reviewer
#9 lore raw

PATCH v3 3/3 khugepaged: Optimize collapse_pte_mapped_thp() by PTE batching

In reply to Dev Jain

Use PTE batching to optimize collapse_pte_mapped_thp().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

Note that we do not need to make a change to the check
"if (folio_page(folio, i) != page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i + nr_batch_ptes - 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.

---
 mm/khugepaged.c | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)
mm/khugepaged.c +20 -12
index 63517ef7eafb..1ff0c7dd2be4 100644
@@ -1503,15 +1503,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
 int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 			    bool install_pmd)
 {
+	int nr_mapped_ptes = 0, nr_batch_ptes, result = SCAN_FAIL;
 	struct mmu_notifier_range range;
 	bool notified = false;
 	unsigned long haddr = addr & HPAGE_PMD_MASK;
+	unsigned long end = haddr + HPAGE_PMD_SIZE;
 	struct vm_area_struct *vma = vma_lookup(mm, haddr);
 	struct folio *folio;
 	pte_t *start_pte, *pte;
 	pmd_t *pmd, pgt_pmd;
 	spinlock_t *pml = NULL, *ptl;
-	int nr_ptes = 0, result = SCAN_FAIL;
 	int i;
 
 	mmap_assert_locked(mm);
@@ -1625,11 +1626,15 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 		goto abort;
 
 	/* step 2: clear page table and adjust rmap */
-	for (i = 0, addr = haddr, pte = start_pte;
-	     i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) {
+	for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR;
+	     i += nr_batch_ptes, addr += nr_batch_ptes * PAGE_SIZE,
+	     pte += nr_batch_ptes) {
+		int max_nr_batch_ptes = (end - addr) >> PAGE_SHIFT;
 		struct page *page;
 		pte_t ptent = ptep_get(pte);
 
+		nr_batch_ptes = 1;
+
 		if (pte_none(ptent))
 			continue;
 		/*
@@ -1643,26 +1648,29 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 			goto abort;
 		}
 		page = vm_normal_page(vma, addr, ptent);
+
 		if (folio_page(folio, i) != page)
 			goto abort;
 
+		nr_batch_ptes = folio_pte_batch(folio, pte, ptent, max_nr_batch_ptes);
+
 		/*
 		 * Must clear entry, or a racing truncate may re-remove it.
 		 * TLB flush can be left until pmdp_collapse_flush() does it.
 		 * PTE dirty? Shmem page is already dirty; file is read-only.
 		 */
-		ptep_clear(mm, addr, pte);
-		folio_remove_rmap_pte(folio, page, vma);
-		nr_ptes++;
+		clear_ptes(mm, addr, pte, nr_batch_ptes);
+		folio_remove_rmap_ptes(folio, page, nr_batch_ptes, vma);
+		nr_mapped_ptes += nr_batch_ptes;
 	}
 
 	if (!pml)
 		spin_unlock(ptl);
 
 	/* step 3: set proper refcount and mm_counters. */
-	if (nr_ptes) {
-		folio_ref_sub(folio, nr_ptes);
-		add_mm_counter(mm, mm_counter_file(folio), -nr_ptes);
+	if (nr_mapped_ptes) {
+		folio_ref_sub(folio, nr_mapped_ptes);
+		add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes);
 	}
 
 	/* step 4: remove empty page table */
@@ -1695,10 +1703,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 			: SCAN_SUCCEED;
 	goto drop_folio;
 abort:
-	if (nr_ptes) {
+	if (nr_mapped_ptes) {
 		flush_tlb_mm(mm);
-		folio_ref_sub(folio, nr_ptes);
-		add_mm_counter(mm, mm_counter_file(folio), -nr_ptes);
+		folio_ref_sub(folio, nr_mapped_ptes);
+		add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes);
 	}
 unlock:
 	if (start_pte)
Signature · 1 line
2.30.2
Acked-by on v3 3/3
#10 lore raw

Re: [PATCH v3 3/3] khugepaged: Optimize collapse_pte_mapped_thp() by PTE batching

Quoted 10 lines · Dev Jain

Use PTE batching to optimize collapse_pte_mapped_thp().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

Please simplify that (and make it less arm specific) like suggested for patch #2.

PTE batching has known benefits on all architectures :)

Quoted 6 lines

Note that we do not need to make a change to the check
"if (folio_page(folio, i) != page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i + nr_batch_ptes - 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.

Yeah, that must stay to make the comment from "step 1" happy.

Quoted 46 lines
---
  mm/khugepaged.c | 32 ++++++++++++++++++++------------
  1 file changed, 20 insertions(+), 12 deletions(-)
mm/khugepaged.c +8 -3
index 63517ef7eafb..1ff0c7dd2be4 100644
@@ -1503,15 +1503,16 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
  int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  			    bool install_pmd)
  {
+	int nr_mapped_ptes = 0, nr_batch_ptes, result = SCAN_FAIL;
  	struct mmu_notifier_range range;
  	bool notified = false;
  	unsigned long haddr = addr & HPAGE_PMD_MASK;
+	unsigned long end = haddr + HPAGE_PMD_SIZE;
  	struct vm_area_struct *vma = vma_lookup(mm, haddr);
  	struct folio *folio;
  	pte_t *start_pte, *pte;
  	pmd_t *pmd, pgt_pmd;
  	spinlock_t *pml = NULL, *ptl;
-	int nr_ptes = 0, result = SCAN_FAIL;
  	int i;
 
  	mmap_assert_locked(mm);
@@ -1625,11 +1626,15 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  		goto abort;
 
  	/* step 2: clear page table and adjust rmap */
-	for (i = 0, addr = haddr, pte = start_pte;
-	     i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) {
+	for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR;
+	     i += nr_batch_ptes, addr += nr_batch_ptes * PAGE_SIZE,
+	     pte += nr_batch_ptes) {
+		int max_nr_batch_ptes = (end - addr) >> PAGE_SHIFT;>   		struct page *page;
  		pte_t ptent = ptep_get(pte);
 
+		nr_batch_ptes = 1;
+
  		if (pte_none(ptent))
  			continue;
  		/*

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1655

Quoted 8 lines
mm/khugepaged.c +2 -0
@@ -1643,26 +1648,29 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  			goto abort;
  		}
  		page = vm_normal_page(vma, addr, ptent);
+
  		if (folio_page(folio, i) != page)
  			goto abort;
 
+		nr_batch_ptes = folio_pte_batch(folio, pte, ptent, max_nr_batch_ptes);

Same comment regarding matching types for folio_pte_batch(), now that we changed that -- unsigned int.

Quoted 28 lines
mm/khugepaged.c +7 -6
+
  		/*
  		 * Must clear entry, or a racing truncate may re-remove it.
  		 * TLB flush can be left until pmdp_collapse_flush() does it.
  		 * PTE dirty? Shmem page is already dirty; file is read-only.
  		 */
-		ptep_clear(mm, addr, pte);
-		folio_remove_rmap_pte(folio, page, vma);
-		nr_ptes++;
+		clear_ptes(mm, addr, pte, nr_batch_ptes);
+		folio_remove_rmap_ptes(folio, page, nr_batch_ptes, vma);
+		nr_mapped_ptes += nr_batch_ptes;
  	}
 
  	if (!pml)
  		spin_unlock(ptl);
 
  	/* step 3: set proper refcount and mm_counters. */
-	if (nr_ptes) {
-		folio_ref_sub(folio, nr_ptes);
-		add_mm_counter(mm, mm_counter_file(folio), -nr_ptes);
+	if (nr_mapped_ptes) {
+		folio_ref_sub(folio, nr_mapped_ptes);
+		add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes);
  	}
 
  	/* step 4: remove empty page table */

mm/khugepaged.c int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · line 1709

Quoted 10 lines
mm/khugepaged.c +3 -3
@@ -1695,10 +1703,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
  			: SCAN_SUCCEED;
  	goto drop_folio;
  abort:
-	if (nr_ptes) {
+	if (nr_mapped_ptes) {
  		flush_tlb_mm(mm);
-		folio_ref_sub(folio, nr_ptes);
-		add_mm_counter(mm, mm_counter_file(folio), -nr_ptes);
+		folio_ref_sub(folio, nr_mapped_ptes);
+		add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes);

Doing the TLB flush and adjusting the refcount after dropping the PTL ... interesting. Well, nothing surprises me in khugpaged code anymore.

Signature · 3 lines
Cheers,

David / dhildenb
reviewer Reviewed-by on v3 3/3
#11 lore raw

Re: [PATCH v3 3/3] khugepaged: Optimize collapse_pte_mapped_thp() by PTE batching

In reply to Dev Jain
Quoted 11 lines · Dev Jain

Use PTE batching to optimize collapse_pte_mapped_thp().

On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at.

For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls.

Quoted 7 lines

Note that we do not need to make a change to the check
"if (folio_page(folio, i) != page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i + nr_batch_ptes - 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages.

With David's comments addressed, LGTM. Thanks.