Skip to contentSkip to conversation

mm/khugepaged: refactor and merge PTE scanning logic

PATCH Superseded Lance Yang 4 participants 18 messages Oct 2, 2025 – Oct 5, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lance Yang

author reviewer
#0 lore raw

cover mm/khugepaged: refactor and merge PTE scanning logic

Hi all,

This series cleans up the almost-duplicated PTE scanning logic in the collapse path.

The first one is a preparatory step that refactors both loops to use a single if-else-if-else-if chain for checking disjoint PTEs.

The second one then extracts the common logic into a shared helper.

Thanks,
Lance

Lance Yang (2):
mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain mm/khugepaged: merge PTE scanning logic into a new helper

 mm/khugepaged.c | 171 +++++++++++++++++++++++++++++-------------------
 1 file changed, 104 insertions(+), 67 deletions(-)
Signature · 1 line
2.49.0
author reviewer
#1 lore raw

PATCH 1/2 mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain

From: Lance Yang <[email protected]>

As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.

This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.

Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.

---
 mm/khugepaged.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
mm/khugepaged.c +4 -8
index f4f57ba69d72..808523f92c7b 100644
@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
 	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
 	     _pte++, addr += PAGE_SIZE) {
 		pte_t pteval = ptep_get(_pte);
-		if (pte_none(pteval) || (pte_present(pteval) &&
-				is_zero_pfn(pte_pfn(pteval)))) {
+		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
 			++none_or_zero;
 			if (!userfaultfd_armed(vma) &&
 			    (!cc->is_khugepaged ||
@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
 				count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
 				goto out;
 			}
-		}
-		if (!pte_present(pteval)) {
+		} else if (!pte_present(pteval)) {
 			result = SCAN_PTE_NON_PRESENT;
 			goto out;
-		}
-		if (pte_uffd_wp(pteval)) {
+		} else if (pte_uffd_wp(pteval)) {
 			result = SCAN_PTE_UFFD_WP;
 			goto out;
 		}
@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
 				count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
 				goto out_unmap;
 			}
-		}
-		if (pte_uffd_wp(pteval)) {
+		} else if (pte_uffd_wp(pteval)) {
 			/*
 			 * Don't collapse the page if any of the small
 			 * PTEs are armed with uffd write protection.
Signature · 1 line
2.49.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain

Quoted 10 lines · Lance Yang

From: Lance Yang <[email protected]>

As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.

This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.

Quoted 7 lines

Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.

Signature · 2 lines
Wei Yang
Help you, Help me
reviewer Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain

In reply to Lance Yang
Quoted 26 lines · Lance Yang

From: Lance Yang <[email protected]>

As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.

This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.

Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.

---
  mm/khugepaged.c | 12 ++++--------
  1 file changed, 4 insertions(+), 8 deletions(-)
mm/khugepaged.c +0 -0
index f4f57ba69d72..808523f92c7b 100644

mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 551

Quoted 6 lines
mm/khugepaged.c +1 -2
@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
  	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
  	     _pte++, addr += PAGE_SIZE) {
  		pte_t pteval = ptep_get(_pte);
-		if (pte_none(pteval) || (pte_present(pteval) &&
-				is_zero_pfn(pte_pfn(pteval)))) {
+		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {

Should have mentioned in the description that pte_present() is not required here, so removing it.

Quoted 28 lines
mm/khugepaged.c +3 -6
  			++none_or_zero;
  			if (!userfaultfd_armed(vma) &&
  			    (!cc->is_khugepaged ||
@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
  				count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
  				goto out;
  			}
-		}
-		if (!pte_present(pteval)) {
+		} else if (!pte_present(pteval)) {
  			result = SCAN_PTE_NON_PRESENT;
  			goto out;
-		}
-		if (pte_uffd_wp(pteval)) {
+		} else if (pte_uffd_wp(pteval)) {
  			result = SCAN_PTE_UFFD_WP;
  			goto out;
  		}
@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
  				count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
  				goto out_unmap;
  			}
-		}
-		if (pte_uffd_wp(pteval)) {
+		} else if (pte_uffd_wp(pteval)) {
  			/*
  			 * Don't collapse the page if any of the small
  			 * PTEs are armed with uffd write protection.
author reviewer
#4 lore raw

Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain

Quoted 23 lines · Dev Jain
Quoted 21 lines · Lance Yang

From: Lance Yang <[email protected]>

As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.

This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.

Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.

---
  mm/khugepaged.c | 12 ++++--------

  1 file changed, 4 insertions(+), 8 deletions(-)
Quoted 15 lines

mm/khugepaged.c static int __collapse_huge_page_isolate(struct · line 552

Quoted 12 lines
mm/khugepaged.c +1 -2
index f4f57ba69d72..808523f92c7b 100644
@@ -548,8 +548,7 @@ static int __collapse_huge_page_isolate(struct 
 vm_area_struct *vma,
       for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
            _pte++, addr += PAGE_SIZE) {
           pte_t pteval = ptep_get(_pte);
-        if (pte_none(pteval) || (pte_present(pteval) &&
-                is_zero_pfn(pte_pfn(pteval)))) {
+        if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {

Should have mentioned in the description that pte_present() is not required here, so removing it.

Yep, got it.

Quoted 2 lines

Cheers,
Lance

Quoted 31 lines
Quoted 31 lines
mm/khugepaged.c +0 -0
               ++none_or_zero;

              if (!userfaultfd_armed(vma) &&
                  (!cc->is_khugepaged ||

mm/khugepaged.c +2 -4
@@ -560,12 +559,10 @@ static int __collapse_huge_page_isolate(struct 
 vm_area_struct *vma,
                   count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
                   goto out;
               }
-        }
-        if (!pte_present(pteval)) {
+        } else if (!pte_present(pteval)) {
               result = SCAN_PTE_NON_PRESENT;
               goto out;
-        }
-        if (pte_uffd_wp(pteval)) {
+        } else if (pte_uffd_wp(pteval)) {
               result = SCAN_PTE_UFFD_WP;
               goto out;

          }

mm/khugepaged.c +1 -2
@@ -1316,8 +1313,7 @@ static int hpage_collapse_scan_pmd(struct 
 mm_struct *mm,
                   count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
                   goto out_unmap;
               }
-        }
-        if (pte_uffd_wp(pteval)) {
+        } else if (pte_uffd_wp(pteval)) {
               /*
                * Don't collapse the page if any of the small

               * PTEs are armed with uffd write protection.

reviewer
#5 lore raw

Re: [PATCH mm-new 1/2] mm/khugepaged: optimize PTE scanning with if-else-if-else-if chain

In reply to Lance Yang
Quoted 14 lines · Lance Yang

From: Lance Yang <[email protected]>

As pointed out by Dev, the PTE checks for disjoint conditions in the scanning loops can be optimized. is_swap_pte, (pte_none && is_zero_pfn), and pte_uffd_wp are mutually exclusive.

This patch refactors the loops in both __collapse_huge_page_isolate() and hpage_collapse_scan_pmd() to use a continuous if-else-if-else-if chain instead of separate if blocks.

Also, this is a preparatory step to make it easier to merge the almost-duplicated scanning logic in these two functions, as suggested by David.

Quoted 7 lines
---
 mm/khugepaged.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

Best Regards,
Yan, Zi

author reviewer
#6 lore raw

PATCH 2/2 mm/khugepaged: merge PTE scanning logic into a new helper

In reply to Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---
 mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------
 1 file changed, 104 insertions(+), 63 deletions(-)
mm/khugepaged.c +104 -63
index 808523f92c7b..2a897cfb1d03 100644
@@ -61,6 +61,12 @@ enum scan_result {
 	SCAN_PAGE_FILLED,
 };
 
+enum pte_check_result {
+	PTE_CHECK_SUCCEED,
+	PTE_CHECK_CONTINUE,
+	PTE_CHECK_FAIL,
+};
+
 #define CREATE_TRACE_POINTS
 #include <trace/events/huge_memory.h>
 
@@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
 	}
 }
 
+/*
+ * thp_collapse_check_pte - Check if a PTE is suitable for THP collapse
+ * @pte:           PTE to check
+ * @vma:           VMA the PTE belongs to
+ * @cc:            Collapse control settings
+ * @scan_swap_pte: Allow scanning of swap PTEs if true
+ * @none_or_zero:  Counter for none/zero PTEs (must be non-NULL)
+ * @unmapped:      Counter for swap PTEs (must be non-NULL if scan_swap_pte
+ *                 is true)
+ * @scan_result:   Used to return the failure reason (SCAN_*) on a
+ *                 PTE_CHECK_FAIL return. Must be non-NULL
+ *
+ * Returns:
+ *   PTE_CHECK_SUCCEED  - Valid PTE, proceed with collapse
+ *   PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning
+ *   PTE_CHECK_FAIL     - Abort collapse scan
+ */
+static inline int thp_collapse_check_pte(pte_t pte, struct vm_area_struct *vma,
+		struct collapse_control *cc, bool scan_swap_pte,
+		int *none_or_zero, int *unmapped, int *scan_result)
+{
+	VM_BUG_ON(!none_or_zero || !scan_result);
+	VM_BUG_ON(scan_swap_pte && !unmapped);
+
+	if (pte_none(pte) || is_zero_pfn(pte_pfn(pte))) {
+		(*none_or_zero)++;
+		if (!userfaultfd_armed(vma) &&
+		    (!cc->is_khugepaged ||
+		     *none_or_zero <= khugepaged_max_ptes_none)) {
+			return PTE_CHECK_CONTINUE;
+		} else {
+			*scan_result = SCAN_EXCEED_NONE_PTE;
+			count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
+			return PTE_CHECK_FAIL;
+		}
+	} else if (!pte_present(pte)) {
+		if (!scan_swap_pte) {
+			*scan_result = SCAN_PTE_NON_PRESENT;
+			return PTE_CHECK_FAIL;
+		}
+
+		if (non_swap_entry(pte_to_swp_entry(pte))) {
+			*scan_result = SCAN_PTE_NON_PRESENT;
+			return PTE_CHECK_FAIL;
+		}
+
+		(*unmapped)++;
+		if (!cc->is_khugepaged ||
+		    *unmapped <= khugepaged_max_ptes_swap) {
+			/*
+			 * Always be strict with uffd-wp
+			 * enabled swap entries. Please see
+			 * comment below for pte_uffd_wp().
+			 */
+			if (pte_swp_uffd_wp(pte)) {
+				*scan_result = SCAN_PTE_UFFD_WP;
+				return PTE_CHECK_FAIL;
+			}
+			return PTE_CHECK_CONTINUE;
+		} else {
+			*scan_result = SCAN_EXCEED_SWAP_PTE;
+			count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
+			return PTE_CHECK_FAIL;
+		}
+	} else if (pte_uffd_wp(pte)) {
+		/*
+		 * Don't collapse the page if any of the small
+		 * PTEs are armed with uffd write protection.
+		 * Here we can also mark the new huge pmd as
+		 * write protected if any of the small ones is
+		 * marked but that could bring unknown
+		 * userfault messages that falls outside of
+		 * the registered range.  So, just be simple.
+		 */
+		*scan_result = SCAN_PTE_UFFD_WP;
+		return PTE_CHECK_FAIL;
+	}
+
+	return PTE_CHECK_SUCCEED;
+}
+
 static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
 					unsigned long start_addr,
 					pte_t *pte,
@@ -544,28 +631,20 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
 	unsigned long addr = start_addr;
 	pte_t *_pte;
 	int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
+	int pte_check_res;
 
 	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
 	     _pte++, addr += PAGE_SIZE) {
 		pte_t pteval = ptep_get(_pte);
-		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
-			++none_or_zero;
-			if (!userfaultfd_armed(vma) &&
-			    (!cc->is_khugepaged ||
-			     none_or_zero <= khugepaged_max_ptes_none)) {
-				continue;
-			} else {
-				result = SCAN_EXCEED_NONE_PTE;
-				count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-				goto out;
-			}
-		} else if (!pte_present(pteval)) {
-			result = SCAN_PTE_NON_PRESENT;
-			goto out;
-		} else if (pte_uffd_wp(pteval)) {
-			result = SCAN_PTE_UFFD_WP;
+		pte_check_res = thp_collapse_check_pte(
+			pteval, vma, cc, false, /* scan_swap_pte = false */
+			&none_or_zero, NULL, &result);
+
+		if (pte_check_res == PTE_CHECK_CONTINUE)
+			continue;
+		else if (pte_check_res == PTE_CHECK_FAIL)
 			goto out;
-		}
+
 		page = vm_normal_page(vma, addr, pteval);
 		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
 			result = SCAN_PAGE_NULL;
@@ -1260,6 +1339,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
 	unsigned long addr;
 	spinlock_t *ptl;
 	int node = NUMA_NO_NODE, unmapped = 0;
+	int pte_check_res;
 
 	VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK);
 
@@ -1278,54 +1358,15 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
 	for (addr = start_addr, _pte = pte; _pte < pte + HPAGE_PMD_NR;
 	     _pte++, addr += PAGE_SIZE) {
 		pte_t pteval = ptep_get(_pte);
-		if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
-			++none_or_zero;
-			if (!userfaultfd_armed(vma) &&
-			    (!cc->is_khugepaged ||
-			     none_or_zero <= khugepaged_max_ptes_none)) {
-				continue;
-			} else {
-				result = SCAN_EXCEED_NONE_PTE;
-				count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-				goto out_unmap;
-			}
-		} else if (!pte_present(pteval)) {
-			if (non_swap_entry(pte_to_swp_entry(pteval))) {
-				result = SCAN_PTE_NON_PRESENT;
-				goto out_unmap;
-			}
 
-			++unmapped;
-			if (!cc->is_khugepaged ||
-			    unmapped <= khugepaged_max_ptes_swap) {
-				/*
-				 * Always be strict with uffd-wp
-				 * enabled swap entries.  Please see
-				 * comment below for pte_uffd_wp().
-				 */
-				if (pte_swp_uffd_wp(pteval)) {
-					result = SCAN_PTE_UFFD_WP;
-					goto out_unmap;
-				}
-				continue;
-			} else {
-				result = SCAN_EXCEED_SWAP_PTE;
-				count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
-				goto out_unmap;
-			}
-		} else if (pte_uffd_wp(pteval)) {
-			/*
-			 * Don't collapse the page if any of the small
-			 * PTEs are armed with uffd write protection.
-			 * Here we can also mark the new huge pmd as
-			 * write protected if any of the small ones is
-			 * marked but that could bring unknown
-			 * userfault messages that falls outside of
-			 * the registered range.  So, just be simple.
-			 */
-			result = SCAN_PTE_UFFD_WP;
+		pte_check_res = thp_collapse_check_pte(
+			pteval, vma, cc, true, /* scan_swap_pte = true */
+			&none_or_zero, &unmapped, &result);
+
+		if (pte_check_res == PTE_CHECK_CONTINUE)
+			continue;
+		else if (pte_check_res == PTE_CHECK_FAIL)
 			goto out_unmap;
-		}
 
 		page = vm_normal_page(vma, addr, pteval);
 		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
Signature · 1 line
2.49.0
reviewer
#7 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 7 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

author reviewer
#8 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

Quoted 6 lines
Quoted 3 lines

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))?

Ah, indeed :)

Quoted 2 lines

In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

So you meant something like this:

mm/khugepaged.c +1 -5
index 2a897cfb1d03..ef87d7fe3d50 100644
@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
                        goto out_unmap;
                }
                folio = page_folio(page);
-
-               if (!folio_test_anon(folio)) {
-                       result = SCAN_PAGE_ANON;
-                       goto out_unmap;
-               }
+               VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
 
                /*
                 * We treat a single page as shared if any part of the THP
reviewer
#9 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 32 lines · Lance Yang
Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

Quoted 6 lines
Quoted 3 lines

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))?

Ah, indeed :)

Quoted 2 lines

In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

So you meant something like this:

mm/khugepaged.c +0 -0
index 2a897cfb1d03..ef87d7fe3d50 100644

mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1378

Quoted 10 lines
mm/khugepaged.c +1 -5
@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(struct 
 mm_struct *mm,
                         goto out_unmap;
                 }
                 folio = page_folio(page);
-
-               if (!folio_test_anon(folio)) {
-                       result = SCAN_PAGE_ANON;
-                       goto out_unmap;
-               }
+               VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);

Yes, I would suggest a different patch making this change, then in the

last patch you can abstract away till the shared statistics line since that

much code will become common between scan_pmd and _isolate.

Quoted 5 lines
mm/khugepaged.c +0 -0
                 /*
                  * We treat a single page as shared if any part of the THP
author reviewer
#10 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 47 lines · Dev Jain
Quoted 32 lines · Lance Yang
Quoted 10 lines · Dev Jain
Quoted 8 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

Quoted 6 lines
Quoted 3 lines

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for an anonymous vma, is it even possible to hit if (! folio_test_anon(folio))?

Ah, indeed :)

Quoted 2 lines

In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

So you meant something like this:

mm/khugepaged.c +0 -0
index 2a897cfb1d03..ef87d7fe3d50 100644

mm/khugepaged.c static int hpage_collapse_scan_pmd(struct · line 1378

Quoted 10 lines
mm/khugepaged.c +1 -5
@@ -1374,11 +1374,7 @@ static int hpage_collapse_scan_pmd(struct 
 mm_struct *mm,
                         goto out_unmap;
                 }
                 folio = page_folio(page);
-
-               if (!folio_test_anon(folio)) {
-                       result = SCAN_PAGE_ANON;
-                       goto out_unmap;
-               }
+               VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);

Yes, I would suggest a different patch making this change, then in the

Quoted 3 lines

last patch you can abstract away till the shared statistics line since that

much code will become common between scan_pmd and _isolate.

Nice. Makes sense to me!

I'll rework the series into three patches then:

1) The if-else-if-else-if optimization
2) Convert the !folio_test_anon() check to a VM_BUG_ON_FOLIO()
3) The final merge of the scanning logic into a new helper

Thanks,
Lance

Quoted 7 lines
Quoted 7 lines
mm/khugepaged.c +0 -0
                 /*
                  * We treat a single page as shared if any part of the THP
#11 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

In reply to Dev Jain
Quoted 7 lines · Dev Jain
Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 8 lines
Quoted 6 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for

This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.

Quoted 3 lines

an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.

My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?

Signature · 2 lines
Wei Yang
Help you, Help me
reviewer
#12 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 17 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 7 lines
Quoted 6 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for

This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.

Quoted 8 lines
Quoted 3 lines

an anonymous vma, is it even possible to hit if (!folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.

My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?

Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this BUG_ON is under CONFIG_DEBUG_VM anways. But if you want to change this to WARN then you can do it at both places.

Quoted 1 line
author reviewer
#13 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 22 lines · Dev Jain
Quoted 20 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 7 lines
Quoted 6 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for

This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.

Quoted 3 lines

an anonymous vma, is it even possible to hit if (! folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

Quoted 8 lines
Quoted 5 lines

But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.

My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?

Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this BUG_ON is under CONFIG_DEBUG_VM anways. But

Yeah, VM_BUG_ON_FOLIO() is under CONFIG_DEBUG_VM, so it won't affect production kernels.

Quoted 1 line

if you want to change this to WARN then you can do it at both places.

It should flag such an impossible condition there during development. So, I'd prefer to stick with VM_BUG_ON_FOLIO().

@Wei please let me know if you feel strongly otherwise :)

reviewer
#14 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 26 lines · Lance Yang
Quoted 25 lines · Dev Jain
Quoted 23 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 7 lines
Quoted 6 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for

This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.

Quoted 3 lines

an anonymous vma, is it even possible to hit if (! folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?

Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this BUG_ON is under CONFIG_DEBUG_VM anways. But

Yeah, VM_BUG_ON_FOLIO() is under CONFIG_DEBUG_VM, so it won't affect production kernels.

Many distros enable it by default. For mm, we are moving away from using BUG_ON or VM_BUG_ON. No need to crash the system if it is possible to handle it gracefully.

Quoted 7 lines
Quoted 2 lines

if you want to change this to WARN then you can do it at both places.

It should flag such an impossible condition there during development. So, I'd prefer to stick with VM_BUG_ON_FOLIO().

@Wei please let me know if you feel strongly otherwise :)

--
Best Regards,
Yan, Zi

author reviewer
#15 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 34 lines · Zi Yan
Quoted 32 lines · Lance Yang
Quoted 30 lines · Dev Jain
Quoted 17 lines · Wei Yang
Quoted 6 lines · Dev Jain
Quoted 5 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

Quoted 7 lines
Quoted 6 lines

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---

In hpage_collapse_scan_pmd(), we enter with mmap lock held, so for

This is true for the first loop, but we will unlock/lock mmap and revalidate vma before isolation.

Quoted 8 lines
Quoted 3 lines

an anonymous vma, is it even possible to hit if (! folio_test_anon(folio))? In which case we can replace this with VM_BUG_ON_FOLIO and abstract away till the folio_maybe_mapped_shared() block?

But it looks still valid, since hugepage_vma_revalidate() will check the vma is still anonymous vma after grab the mmap lock again.

My concern is would VM_BUG_ON_FOLIO() be too heavy? How about warn on and return?

Frankly I do not have much opinion on the BUG_ON/WARN_ON debate since I haven't properly understood that, but this BUG_ON is under CONFIG_DEBUG_VM anways. But

Quoted 6 lines
Quoted 2 lines

Yeah, VM_BUG_ON_FOLIO() is under CONFIG_DEBUG_VM, so it won't affect production kernels.

Many distros enable it by default. For mm, we are moving away from using BUG_ON or VM_BUG_ON. No need to crash the system if it is possible to handle it gracefully.

Ah, good to know that, thanks!

Quoted 13 lines
Quoted 8 lines
Quoted 3 lines

if you want to change this to WARN then you can do it at both places.

It should flag such an impossible condition there during development. So, I'd prefer to stick with VM_BUG_ON_FOLIO().

@Wei please let me know if you feel strongly otherwise :)

--
Best Regards,
Yan, Zi

reviewer
#16 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

In reply to Lance Yang
Quoted 48 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---
 mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------
 1 file changed, 104 insertions(+), 63 deletions(-)
mm/khugepaged.c +19 -0
index 808523f92c7b..2a897cfb1d03 100644
@@ -61,6 +61,12 @@ enum scan_result {
 	SCAN_PAGE_FILLED,
 };
 
+enum pte_check_result {
+	PTE_CHECK_SUCCEED,
+	PTE_CHECK_CONTINUE,
+	PTE_CHECK_FAIL,
+};
+
 #define CREATE_TRACE_POINTS
 #include <trace/events/huge_memory.h>
 
@@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
 	}
 }
 
+/*
+ * thp_collapse_check_pte - Check if a PTE is suitable for THP collapse
+ * @pte:           PTE to check
+ * @vma:           VMA the PTE belongs to
+ * @cc:            Collapse control settings
+ * @scan_swap_pte: Allow scanning of swap PTEs if true
+ * @none_or_zero:  Counter for none/zero PTEs (must be non-NULL)
+ * @unmapped:      Counter for swap PTEs (must be non-NULL if scan_swap_pte
+ *                 is true)
+ * @scan_result:   Used to return the failure reason (SCAN_*) on a
+ *                 PTE_CHECK_FAIL return. Must be non-NULL
+ *
+ * Returns:

mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte, · line 564

Quoted 10 lines
mm/khugepaged.c +10 -0
+ *   PTE_CHECK_SUCCEED  - Valid PTE, proceed with collapse
+ *   PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning
+ *   PTE_CHECK_FAIL     - Abort collapse scan
+ */
+static inline int thp_collapse_check_pte(pte_t pte, struct vm_area_struct *vma,
+		struct collapse_control *cc, bool scan_swap_pte,
+		int *none_or_zero, int *unmapped, int *scan_result)
+{
+	VM_BUG_ON(!none_or_zero || !scan_result);
+	VM_BUG_ON(scan_swap_pte && !unmapped);

Please use VM_WARN_ONCE_ON and just return. Or put all stats in a struct, pass and check the struct pointer instead.

Best Regards,
Yan, Zi

author reviewer
#17 lore raw

Re: [PATCH mm-new 2/2] mm/khugepaged: merge PTE scanning logic into a new helper

Quoted 55 lines · Zi Yan
Quoted 53 lines · Lance Yang

From: Lance Yang <[email protected]>

As David suggested, the PTE scanning logic in hpage_collapse_scan_pmd() and __collapse_huge_page_isolate() was almost duplicated.

This patch cleans things up by moving all the common PTE checking logic into a new shared helper, thp_collapse_check_pte().

---
  mm/khugepaged.c | 167 ++++++++++++++++++++++++++++++------------------
  1 file changed, 104 insertions(+), 63 deletions(-)
mm/khugepaged.c +24 -0
index 808523f92c7b..2a897cfb1d03 100644
@@ -61,6 +61,12 @@ enum scan_result {
  	SCAN_PAGE_FILLED,
  };
 
+enum pte_check_result {
+	PTE_CHECK_SUCCEED,
+	PTE_CHECK_CONTINUE,
+	PTE_CHECK_FAIL,
+};
+
  #define CREATE_TRACE_POINTS
  #include <trace/events/huge_memory.h>
 
@@ -533,6 +539,87 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte,
  	}
  }
 
+/*
+ * thp_collapse_check_pte - Check if a PTE is suitable for THP collapse
+ * @pte:           PTE to check
+ * @vma:           VMA the PTE belongs to
+ * @cc:            Collapse control settings
+ * @scan_swap_pte: Allow scanning of swap PTEs if true
+ * @none_or_zero:  Counter for none/zero PTEs (must be non-NULL)
+ * @unmapped:      Counter for swap PTEs (must be non-NULL if scan_swap_pte
+ *                 is true)
+ * @scan_result:   Used to return the failure reason (SCAN_*) on a
+ *                 PTE_CHECK_FAIL return. Must be non-NULL
+ *
+ * Returns:
+ *   PTE_CHECK_SUCCEED  - Valid PTE, proceed with collapse
+ *   PTE_CHECK_CONTINUE - Skip this none/zero PTE but continue scanning
+ *   PTE_CHECK_FAIL     - Abort collapse scan
+ */
+static inline int thp_collapse_check_pte(pte_t pte, struct vm_area_struct *vma,
Quoted 8 lines

mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte, · line 564

Quoted 5 lines
mm/khugepaged.c +5 -0
+		struct collapse_control *cc, bool scan_swap_pte,
+		int *none_or_zero, int *unmapped, int *scan_result)
+{
+	VM_BUG_ON(!none_or_zero || !scan_result);
+	VM_BUG_ON(scan_swap_pte && !unmapped);

Please use VM_WARN_ONCE_ON and just return. Or put all stats in a struct, pass and check the struct pointer instead.

Yes, will use VM_WARN_ONCE_ON instead of VM_BUG_ON, and just return PTE_CHECK_FAIL.

Thanks,
Lance

Quoted 4 lines

Best Regards,
Yan, Zi