Skip to contentSkip to conversation

mm: split underused anonymous mTHP folios

PATCH In review Joanne Koong 7 participants 22 messages Nov 3 – Nov 12

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v4 cover mm: split underused anonymous mTHP folios

PMD-sized THPs that are mostly zero-filled are reclaimed under memory pressure by the deferred split shrinker, but this is not done for mTHP folios. At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M provides the contpte benefits while the PMD size there (512M) is too big to use. However, 2M THP=always causes memory regressions unless the unused portions of those folios can be broken down and reclaimed.

This series lets the underused shrinker handle anonymous mTHP folios. It has four patches:

Patch 1 makes thp_underused() folio-size-aware. It scales khugepaged/max_ptes_none to the folio size, so every folio tolerates the same fraction of zero-filled pages.

Patch 2 stops queueing folios for underused splitting when khugepaged/max_ptes_none is at its default maximum, where the shrinker would never split them anyway. This applies to PMD-sized folios too, which are unconditionally queued today.

Patch 3 keeps thp_underused_split_page counting only PMD-sized folios and adds a per-order split_underused counter, matching how thp_split_page and thp_deferred_split_page are handled.

Patch 4 queues anonymous mTHP folios on the deferred split list, with a hard-coded 2M minimum folio size as suggested by Barry. PMD-sized folios are exempt from the minimum, since on s390 the PMD size is 1M.

At the default khugepaged/max_ptes_none, no folio is queued for underused splitting. Once it's lowered, mTHP folios of at least 2M are queued along with PMD-sized folios.

Changelog
---------
v3: https://lore.kernel.org/linux-mm/[email protected]/
Changes since v3:
* Scale khugepaged/max_ptes_none with truncating division instead of rounding up, which is exact given the strict > comparison (Sashiko, Lance)
* New patch 3: keep thp_underused_split_page PMD-only and add a per-order split_underused counter (Kiryl)
* Fix typo in patch 4's commit message (Kiryl)

v2: https://lore.kernel.org/linux-mm/[email protected]/
Changes since v2:
* Go back to scaling khugepaged/max_ptes_none per folio size (David)
* Keep the check for the sysctl maximum on the raw, unscaled value (Sashiko, from v1). Round the scaled value up so a folio is never split when fewer than the configured fraction of its pages are zero-filled.
* Only queue mTHP folios of at least 2M, a hard-coded minimum (Barry, David)

v1: https://lore.kernel.org/linux-mm/[email protected]/
Changes since v1:
* Drop the per-order scaling of khugepaged_max_ptes_none. Keep it absolute, matching collapse (David, Johannes)
* New patch 2 (suggested by Johannes): don't queue folios that can never be underused, which both bounds what patch 3 adds and stops queuing PMD folios under default settings (Johannes, David, Barry)
* Fix the thp_underused() early exit to scale to the folio's own size

Joanne Koong (4):
mm/huge_memory: scale thp_underused() to the folio size mm/huge_memory: don't queue folios when underused splitting is
inactive
mm/huge_memory: add a per-order counter for underused splits
mm: add anonymous mTHP folios to the deferred split list

 Documentation/admin-guide/mm/transhuge.rst |  6 ++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 66 ++++++++++++++++++----
 mm/memory.c                                |  2 +
 4 files changed, 65 insertions(+), 10 deletions(-)

base-commit: 74d73ffcffe03804f97e9fb82301dadf75671282

Signature · 1 line
2.52.0
#1 lore raw

PATCH v4 1/4 mm/huge_memory: scale thp_underused() to the folio size

thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed.

Today it only ever runs on PMD-sized folios, so it assumes that the folio has HPAGE_PMD_NR pages and compares the zero-filled page count directly against khugepaged_max_ptes_none. A later patch will queue anonymous mTHP folios onto the deferred split list as well, at which point that assumption will no longer hold.

Make thp_underused() folio-size-aware, and scale khugepaged_max_ptes_none proportionally to the number of pages in the folio (eg every folio tolerates the same fraction of zero-filled pages the user configured for a PMD-sized folio).

khugepaged deliberately doesn't scale intermediate values when collapsing to mTHP orders (to avoid the memory footprint creeping up through repeated collapses), but if/when collapse later gains support for intermediate values, a folio that khugepaged collapses won't then be split right back by the shrinker (and vice versa) since both collapse and split would use the same threshold for a given folio size.

There is no functional change for PMD-sized folios, as for those, nr_pages is HPAGE_PMD_NR.

---
 mm/huge_memory.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
mm/huge_memory.c +14 -6
index 14599d0fff01..3720c12bdb0d 100644
@@ -4625,25 +4625,33 @@ static unsigned long deferred_split_count(struct shrinker *shrink,
 
 static bool thp_underused(struct folio *folio)
 {
-	int num_zero_pages = 0, num_filled_pages = 0;
-	int i;
+	const unsigned long nr_pages = folio_nr_pages(folio);
+	unsigned long num_zero_pages = 0, num_filled_pages = 0;
+	unsigned long max_ptes_none = khugepaged_max_ptes_none;
+	unsigned long i;
 
-	if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1)
+	if (max_ptes_none == HPAGE_PMD_NR - 1)
 		return false;
 
 	if (folio_contain_hwpoisoned_page(folio))
 		return false;
 
-	for (i = 0; i < folio_nr_pages(folio); i++) {
+	/*
+	 * khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR.
+	 * Scale it to this folio's size.
+	 */
+	max_ptes_none = max_ptes_none * nr_pages / HPAGE_PMD_NR;
+
+	for (i = 0; i < nr_pages; i++) {
 		if (pages_identical(folio_page(folio, i), ZERO_PAGE(0))) {
-			if (++num_zero_pages > khugepaged_max_ptes_none)
+			if (++num_zero_pages > max_ptes_none)
 				return true;
 		} else {
 			/*
 			 * Another path for early exit once the number
 			 * of non-zero filled pages exceeds threshold.
 			 */
-			if (++num_filled_pages >= HPAGE_PMD_NR - khugepaged_max_ptes_none)
+			if (++num_filled_pages >= nr_pages - max_ptes_none)
 				return false;
 		}
 	}
Signature · 1 line
2.52.0
reviewer Reviewed-by on v4 1/4
#2 lore raw

Re: [PATCH v4 1/4] mm/huge_memory: scale thp_underused() to the folio size

Quoted 22 lines · Joanne Koong

thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed.

Today it only ever runs on PMD-sized folios, so it assumes that the folio has HPAGE_PMD_NR pages and compares the zero-filled page count directly against khugepaged_max_ptes_none. A later patch will queue anonymous mTHP folios onto the deferred split list as well, at which point that assumption will no longer hold.

Make thp_underused() folio-size-aware, and scale khugepaged_max_ptes_none proportionally to the number of pages in the folio (eg every folio tolerates the same fraction of zero-filled pages the user configured for a PMD-sized folio).

khugepaged deliberately doesn't scale intermediate values when collapsing to mTHP orders (to avoid the memory footprint creeping up through repeated collapses), but if/when collapse later gains support for intermediate values, a folio that khugepaged collapses won't then be split right back by the shrinker (and vice versa) since both collapse and split would use the same threshold for a given folio size.

Quoted 5 lines

There is no functional change for PMD-sized folios, as for those, nr_pages is HPAGE_PMD_NR.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer Acked-by on v4 1/4
#3 lore raw

Re: [PATCH v4 1/4] mm/huge_memory: scale thp_underused() to the folio size

In reply to Joanne Koong
Quoted 47 lines · Joanne Koong

thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed.

Today it only ever runs on PMD-sized folios, so it assumes that the folio has HPAGE_PMD_NR pages and compares the zero-filled page count directly against khugepaged_max_ptes_none. A later patch will queue anonymous mTHP folios onto the deferred split list as well, at which point that assumption will no longer hold.

Make thp_underused() folio-size-aware, and scale khugepaged_max_ptes_none proportionally to the number of pages in the folio (eg every folio tolerates the same fraction of zero-filled pages the user configured for a PMD-sized folio).

khugepaged deliberately doesn't scale intermediate values when collapsing to mTHP orders (to avoid the memory footprint creeping up through repeated collapses), but if/when collapse later gains support for intermediate values, a folio that khugepaged collapses won't then be split right back by the shrinker (and vice versa) since both collapse and split would use the same threshold for a given folio size.

There is no functional change for PMD-sized folios, as for those, nr_pages is HPAGE_PMD_NR.

---
 mm/huge_memory.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
mm/huge_memory.c +4 -3
index 14599d0fff01..3720c12bdb0d 100644
@@ -4625,25 +4625,33 @@ static unsigned long deferred_split_count(struct shrinker *shrink,
 
 static bool thp_underused(struct folio *folio)
 {
-	int num_zero_pages = 0, num_filled_pages = 0;
-	int i;
+	const unsigned long nr_pages = folio_nr_pages(folio);
+	unsigned long num_zero_pages = 0, num_filled_pages = 0;
+	unsigned long max_ptes_none = khugepaged_max_ptes_none;
+	unsigned long i;
 
-	if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1)

mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4641

Quoted 10 lines
mm/huge_memory.c +4 -1
+	if (max_ptes_none == HPAGE_PMD_NR - 1)
 		return false;
 
 	if (folio_contain_hwpoisoned_page(folio))
 		return false;
 
-	for (i = 0; i < folio_nr_pages(folio); i++) {
+	/*
+	 * khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR.
+	 * Scale it to this folio's size.

It is better to document max_ptes_none is different from what khugepaged does, especially khugepaged_max_ptes_none is mentioned above.

Quoted 8 lines
mm/huge_memory.c +5 -1
+	 */
+	max_ptes_none = max_ptes_none * nr_pages / HPAGE_PMD_NR;
+
+	for (i = 0; i < nr_pages; i++) {
 		if (pages_identical(folio_page(folio, i), ZERO_PAGE(0))) {
-			if (++num_zero_pages > khugepaged_max_ptes_none)
+			if (++num_zero_pages > max_ptes_none)
 				return true;

mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4657

Quoted 10 lines
mm/huge_memory.c +1 -1
 		} else {
 			/*
 			 * Another path for early exit once the number
 			 * of non-zero filled pages exceeds threshold.
 			 */
-			if (++num_filled_pages >= HPAGE_PMD_NR - khugepaged_max_ptes_none)
+			if (++num_filled_pages >= nr_pages - max_ptes_none)
 				return false;
 		}
 	}

Otherwise, LGTM.

Signature · 2 lines
Best Regards,
Yan, Zi
#4 lore raw

Re: [PATCH v4 1/4] mm/huge_memory: scale thp_underused() to the folio size

Quoted 23 lines · Zi Yan
Quoted 23 lines
mm/huge_memory.c +5 -3
index 14599d0fff01..3720c12bdb0d 100644
@@ -4625,25 +4625,33 @@ static unsigned long deferred_split_count(struct shrinker *shrink,
 
 static bool thp_underused(struct folio *folio)
 {
-     int num_zero_pages = 0, num_filled_pages = 0;
-     int i;
+     const unsigned long nr_pages = folio_nr_pages(folio);
+     unsigned long num_zero_pages = 0, num_filled_pages = 0;
+     unsigned long max_ptes_none = khugepaged_max_ptes_none;
+     unsigned long i;
 
-     if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1)
+     if (max_ptes_none == HPAGE_PMD_NR - 1)
              return false;
 
      if (folio_contain_hwpoisoned_page(folio))
              return false;
 
Quoted 7 lines

mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4641

Quoted 4 lines
mm/huge_memory.c +3 -1
-     for (i = 0; i < folio_nr_pages(folio); i++) {
+     /*
+      * khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR.
+      * Scale it to this folio's size.

It is better to document max_ptes_none is different from what khugepaged does, especially khugepaged_max_ptes_none is mentioned above.

I will add a comment about this to v5.

Thanks,
Joanne

reviewer Reviewed-by on v4 1/4
#5 lore raw

Re: [PATCH v4 1/4] mm/huge_memory: scale thp_underused() to the folio size

In reply to Joanne Koong
Quoted 22 lines · Joanne Koong

thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed.

Today it only ever runs on PMD-sized folios, so it assumes that the folio has HPAGE_PMD_NR pages and compares the zero-filled page count directly against khugepaged_max_ptes_none. A later patch will queue anonymous mTHP folios onto the deferred split list as well, at which point that assumption will no longer hold.

Make thp_underused() folio-size-aware, and scale khugepaged_max_ptes_none proportionally to the number of pages in the folio (eg every folio tolerates the same fraction of zero-filled pages the user configured for a PMD-sized folio).

khugepaged deliberately doesn't scale intermediate values when collapsing to mTHP orders (to avoid the memory footprint creeping up through repeated collapses), but if/when collapse later gains support for intermediate values, a folio that khugepaged collapses won't then be split right back by the shrinker (and vice versa) since both collapse and split would use the same threshold for a given folio size.

Quoted 6 lines

There is no functional change for PMD-sized folios, as for those, nr_pages is HPAGE_PMD_NR.

---

Make sense to me.

#6 lore raw

PATCH v4 2/4 mm/huge_memory: don't queue folios when underused splitting is inactive

In reply to Joanne Koong

Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.

With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.

Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into thp_underused_never() so that deferred_split_folio() and thp_underused() share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.

This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.

Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.

---
 mm/huge_memory.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
mm/huge_memory.c +21 -3
index 3720c12bdb0d..903bbe972011 100644
@@ -4564,6 +4564,15 @@ bool __folio_unqueue_deferred_split(struct folio *folio)
 	return unqueued;	/* useful for debug warnings */
 }
 
+/*
+ * At the sysctl maximum the user tolerates any number of zero-filled pages,
+ * so no folio is ever underused.
+ */
+static bool thp_underused_never(unsigned long max_ptes_none)
+{
+	return max_ptes_none == HPAGE_PMD_NR - 1;
+}
+
 /* partially_mapped=false won't clear PG_partially_mapped folio flag */
 void deferred_split_folio(struct folio *folio, bool partially_mapped)
 {
@@ -4579,8 +4588,17 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
 	if (folio_order(folio) <= 1)
 		return;
 
-	if (!partially_mapped && !split_underused_thp)
-		return;
+	if (!partially_mapped) {
+		if (!split_underused_thp)
+			return;
+		/*
+		 * Nothing will ever split this folio for being underused, so
+		 * keep it off the queue entirely rather than paying for the
+		 * list_lru lock here and a shrinker scan later.
+		 */
+		if (thp_underused_never(khugepaged_max_ptes_none))
+			return;
+	}
 
 	/*
 	 * Exclude swapcache: originally to avoid a corrupt deferred split
@@ -4630,7 +4648,7 @@ static bool thp_underused(struct folio *folio)
 	unsigned long max_ptes_none = khugepaged_max_ptes_none;
 	unsigned long i;
 
-	if (max_ptes_none == HPAGE_PMD_NR - 1)
+	if (thp_underused_never(max_ptes_none))
 		return false;
 
 	if (folio_contain_hwpoisoned_page(folio))
Signature · 1 line
2.52.0
reviewer Acked-by on v4 2/4
#7 lore raw

Re: [PATCH v4 2/4] mm/huge_memory: don't queue folios when underused splitting is inactive

Quoted 29 lines · Joanne Koong

Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.

With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.

Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into thp_underused_never() so that deferred_split_folio() and thp_underused() share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.

This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.

Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.

Quoted 4 lines

---

This sounds reasonable. In production, as far as I know from speaking with folks in conferences, no one changes the value during application runtime (atleast this is true in meta). Its usually only changed before the workload starts (or just at boot time).

Quoted 52 lines
 mm/huge_memory.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
mm/huge_memory.c +21 -3
index 3720c12bdb0d..903bbe972011 100644
@@ -4564,6 +4564,15 @@ bool __folio_unqueue_deferred_split(struct folio *folio)
 	return unqueued;	/* useful for debug warnings */
 }
 
+/*
+ * At the sysctl maximum the user tolerates any number of zero-filled pages,
+ * so no folio is ever underused.
+ */
+static bool thp_underused_never(unsigned long max_ptes_none)
+{
+	return max_ptes_none == HPAGE_PMD_NR - 1;
+}
+
 /* partially_mapped=false won't clear PG_partially_mapped folio flag */
 void deferred_split_folio(struct folio *folio, bool partially_mapped)
 {
@@ -4579,8 +4588,17 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
 	if (folio_order(folio) <= 1)
 		return;
 
-	if (!partially_mapped && !split_underused_thp)
-		return;
+	if (!partially_mapped) {
+		if (!split_underused_thp)
+			return;
+		/*
+		 * Nothing will ever split this folio for being underused, so
+		 * keep it off the queue entirely rather than paying for the
+		 * list_lru lock here and a shrinker scan later.
+		 */
+		if (thp_underused_never(khugepaged_max_ptes_none))
+			return;
+	}
 
 	/*
 	 * Exclude swapcache: originally to avoid a corrupt deferred split
@@ -4630,7 +4648,7 @@ static bool thp_underused(struct folio *folio)
 	unsigned long max_ptes_none = khugepaged_max_ptes_none;
 	unsigned long i;
 
-	if (max_ptes_none == HPAGE_PMD_NR - 1)
+	if (thp_underused_never(max_ptes_none))
 		return false;
 
 	if (folio_contain_hwpoisoned_page(folio))
reviewer Reviewed-by on v4 2/4
#8 lore raw

Re: [PATCH v4 2/4] mm/huge_memory: don't queue folios when underused splitting is inactive

In reply to Joanne Koong
Quoted 29 lines · Joanne Koong

Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.

With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.

Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into thp_underused_never() so that deferred_split_folio() and thp_underused() share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.

This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.

Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.

Quoted 3 lines

I believe I've already:

Don't lose it in case of repost :)

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer Reviewed-by on v4 2/4
#9 lore raw

Re: [PATCH v4 2/4] mm/huge_memory: don't queue folios when underused splitting is inactive

In reply to Joanne Koong
Quoted 29 lines · Joanne Koong

Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.

With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.

Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into thp_underused_never() so that deferred_split_folio() and thp_underused() share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.

This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.

Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.

Quoted 4 lines

---

#10 lore raw

PATCH v4 3/4 mm/huge_memory: add a per-order counter for underused splits

In reply to Joanne Koong

thp_underused_split_page counts every folio that the deferred split shrinker splits for being underused. Today that is only ever a PMD-sized folio, but a later patch queues anonymous mTHP folios for underused splitting too, at which point this counter would start counting mTHP splits as well.

thp_underused_split_page could then end up larger than thp_split_page, and there would be no per-order count of underused splits. That doesn't match how the other THP split stats work. In /proc/vmstat, thp_split_page and thp_deferred_split_page only count PMD-sized folios. Splits of every size, PMD included, are counted per order in the split and split_deferred stats under hugepages-<size>/stats/.

Do the same for underused splits. Only count PMD-sized folios in thp_underused_split_page, and add a per-order split_underused stat that counts underused splits of every size. The folio's order has to be read before splitting it, since the folio is no longer large once the split succeeds.

There is no functional change to thp_underused_split_page yet, as only PMD-sized folios are split for being underused today.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 ++++++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 10 ++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +6 -0
index b187d618452f..6f2d92a2a556 100644
@@ -719,6 +719,12 @@ split_deferred
         it would free up some memory. Pages on split queue are going to
         be split under memory pressure, if splitting is possible.
 
+split_underused
+	is incremented when a huge page on the split queue was split
+	because it was underused. This happens when the number of zero-filled
+	pages in the huge page is above a certain threshold scaled from
+	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none.
+
 nr_anon
        the number of anonymous THP we have in the whole system. These THPs
        might be currently entirely mapped or have partially unmapped/unused
include/linux/huge_mm.h +1 -0
index c745f7ad2298..421217aca4ed 100644
@@ -142,6 +142,7 @@ enum mthp_stat_item {
 	MTHP_STAT_SPLIT,
 	MTHP_STAT_SPLIT_FAILED,
 	MTHP_STAT_SPLIT_DEFERRED,
+	MTHP_STAT_SPLIT_UNDERUSED,
 	MTHP_STAT_NR_ANON,
 	MTHP_STAT_NR_ANON_PARTIALLY_MAPPED,
 	MTHP_STAT_COLLAPSE_EXCEED_SWAP,
mm/huge_memory.c +8 -2
index 903bbe972011..8f90fb609e57 100644
@@ -822,6 +822,7 @@ DEFINE_MTHP_STAT_ATTR(shmem_fallback_charge, MTHP_STAT_SHMEM_FALLBACK_CHARGE);
 DEFINE_MTHP_STAT_ATTR(split, MTHP_STAT_SPLIT);
 DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);
 DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED);
+DEFINE_MTHP_STAT_ATTR(split_underused, MTHP_STAT_SPLIT_UNDERUSED);
 DEFINE_MTHP_STAT_ATTR(nr_anon, MTHP_STAT_NR_ANON);
 DEFINE_MTHP_STAT_ATTR(nr_anon_partially_mapped, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED);
 DEFINE_MTHP_STAT_ATTR(collapse_exceed_swap_pte, MTHP_STAT_COLLAPSE_EXCEED_SWAP);
@@ -842,6 +843,7 @@ static struct attribute *anon_stats_attrs[] = {
 	&swpout_fallback_attr.attr,
 #endif
 	&split_deferred_attr.attr,
+	&split_underused_attr.attr,
 	&nr_anon_attr.attr,
 	&nr_anon_partially_mapped_attr.attr,
 	&collapse_exceed_swap_pte_attr.attr,
@@ -4703,6 +4705,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 					    deferred_split_isolate, &dispose);
 
 	list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
+		const unsigned int order = folio_order(folio);
 		bool did_split = false;
 		bool underused = false;
 
@@ -4724,8 +4727,11 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 			goto requeue;
 		if (!split_folio(folio)) {
 			did_split = true;
-			if (underused)
-				count_vm_event(THP_UNDERUSED_SPLIT_PAGE);
+			if (underused) {
+				if (is_pmd_order(order))
+					count_vm_event(THP_UNDERUSED_SPLIT_PAGE);
+				count_mthp_stat(order, MTHP_STAT_SPLIT_UNDERUSED);
+			}
 			split++;
 		}
 		folio_unlock(folio);
Signature · 1 line
2.52.0
reviewer Acked-by on v4 3/4
#11 lore raw

Re: [PATCH v4 3/4] mm/huge_memory: add a per-order counter for underused splits

Quoted 35 lines · Joanne Koong

thp_underused_split_page counts every folio that the deferred split shrinker splits for being underused. Today that is only ever a PMD-sized folio, but a later patch queues anonymous mTHP folios for underused splitting too, at which point this counter would start counting mTHP splits as well.

thp_underused_split_page could then end up larger than thp_split_page, and there would be no per-order count of underused splits. That doesn't match how the other THP split stats work. In /proc/vmstat, thp_split_page and thp_deferred_split_page only count PMD-sized folios. Splits of every size, PMD included, are counted per order in the split and split_deferred stats under hugepages-<size>/stats/.

Do the same for underused splits. Only count PMD-sized folios in thp_underused_split_page, and add a per-order split_underused stat that counts underused splits of every size. The folio's order has to be read before splitting it, since the folio is no longer large once the split succeeds.

There is no functional change to thp_underused_split_page yet, as only PMD-sized folios are split for being underused today.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 ++++++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 10 ++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b187d618452f..6f2d92a2a556 100644

Documentation/admin-guide/mm/transhuge.rst split_deferred · line 721

Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -719,6 +719,12 @@ split_deferred
         it would free up some memory. Pages on split queue are going to
         be split under memory pressure, if splitting is possible.
 

In this same commit, would you mind also doing s/THP/PMD mapped THP/ in thp_underused_split_page section in /proc/vmstat section?

Apart from that, LGTM

Quoted 63 lines
Documentation/admin-guide/mm/transhuge.rst +6 -0
+split_underused
+	is incremented when a huge page on the split queue was split
+	because it was underused. This happens when the number of zero-filled
+	pages in the huge page is above a certain threshold scaled from
+	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none.
+
 nr_anon
        the number of anonymous THP we have in the whole system. These THPs
        might be currently entirely mapped or have partially unmapped/unused
include/linux/huge_mm.h +1 -0
index c745f7ad2298..421217aca4ed 100644
@@ -142,6 +142,7 @@ enum mthp_stat_item {
 	MTHP_STAT_SPLIT,
 	MTHP_STAT_SPLIT_FAILED,
 	MTHP_STAT_SPLIT_DEFERRED,
+	MTHP_STAT_SPLIT_UNDERUSED,
 	MTHP_STAT_NR_ANON,
 	MTHP_STAT_NR_ANON_PARTIALLY_MAPPED,
 	MTHP_STAT_COLLAPSE_EXCEED_SWAP,
mm/huge_memory.c +8 -2
index 903bbe972011..8f90fb609e57 100644
@@ -822,6 +822,7 @@ DEFINE_MTHP_STAT_ATTR(shmem_fallback_charge, MTHP_STAT_SHMEM_FALLBACK_CHARGE);
 DEFINE_MTHP_STAT_ATTR(split, MTHP_STAT_SPLIT);
 DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);
 DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED);
+DEFINE_MTHP_STAT_ATTR(split_underused, MTHP_STAT_SPLIT_UNDERUSED);
 DEFINE_MTHP_STAT_ATTR(nr_anon, MTHP_STAT_NR_ANON);
 DEFINE_MTHP_STAT_ATTR(nr_anon_partially_mapped, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED);
 DEFINE_MTHP_STAT_ATTR(collapse_exceed_swap_pte, MTHP_STAT_COLLAPSE_EXCEED_SWAP);
@@ -842,6 +843,7 @@ static struct attribute *anon_stats_attrs[] = {
 	&swpout_fallback_attr.attr,
 #endif
 	&split_deferred_attr.attr,
+	&split_underused_attr.attr,
 	&nr_anon_attr.attr,
 	&nr_anon_partially_mapped_attr.attr,
 	&collapse_exceed_swap_pte_attr.attr,
@@ -4703,6 +4705,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 					    deferred_split_isolate, &dispose);
 
 	list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
+		const unsigned int order = folio_order(folio);
 		bool did_split = false;
 		bool underused = false;
 
@@ -4724,8 +4727,11 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 			goto requeue;
 		if (!split_folio(folio)) {
 			did_split = true;
-			if (underused)
-				count_vm_event(THP_UNDERUSED_SPLIT_PAGE);
+			if (underused) {
+				if (is_pmd_order(order))
+					count_vm_event(THP_UNDERUSED_SPLIT_PAGE);
+				count_mthp_stat(order, MTHP_STAT_SPLIT_UNDERUSED);
+			}
 			split++;
 		}
 		folio_unlock(folio);
reviewer Reviewed-by
#12 lore raw

Re: [PATCH v4 3/4] mm/huge_memory: add a per-order counter for underused splits

Quoted 33 lines · Usama Arif
Quoted 30 lines · Joanne Koong

thp_underused_split_page counts every folio that the deferred split shrinker splits for being underused. Today that is only ever a PMD-sized folio, but a later patch queues anonymous mTHP folios for underused splitting too, at which point this counter would start counting mTHP splits as well.

thp_underused_split_page could then end up larger than thp_split_page, and there would be no per-order count of underused splits. That doesn't match how the other THP split stats work. In /proc/vmstat, thp_split_page and thp_deferred_split_page only count PMD-sized folios. Splits of every size, PMD included, are counted per order in the split and split_deferred stats under hugepages-<size>/stats/.

Do the same for underused splits. Only count PMD-sized folios in thp_underused_split_page, and add a per-order split_underused stat that counts underused splits of every size. The folio's order has to be read before splitting it, since the folio is no longer large once the split succeeds.

There is no functional change to thp_underused_split_page yet, as only PMD-sized folios are split for being underused today.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 ++++++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 10 ++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Quoted 11 lines

Documentation/admin-guide/mm/transhuge.rst split_deferred · line 721

Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b187d618452f..6f2d92a2a556 100644
@@ -719,6 +719,12 @@ split_deferred
         it would free up some memory. Pages on split queue are going to
         be split under memory pressure, if splitting is possible.
 

In this same commit, would you mind also doing s/THP/PMD mapped THP/ in thp_underused_split_page section in /proc/vmstat section?

PMD-size, not PMD-mapped. But yes, it would be nice to update.

With that updated, feel free to use my

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#13 lore raw

Re: [PATCH v4 3/4] mm/huge_memory: add a per-order counter for underused splits

Quoted 47 lines · Kiryl Shutsemau
Quoted 33 lines · Usama Arif
Quoted 30 lines · Joanne Koong

thp_underused_split_page counts every folio that the deferred split shrinker splits for being underused. Today that is only ever a PMD-sized folio, but a later patch queues anonymous mTHP folios for underused splitting too, at which point this counter would start counting mTHP splits as well.

thp_underused_split_page could then end up larger than thp_split_page, and there would be no per-order count of underused splits. That doesn't match how the other THP split stats work. In /proc/vmstat, thp_split_page and thp_deferred_split_page only count PMD-sized folios. Splits of every size, PMD included, are counted per order in the split and split_deferred stats under hugepages-<size>/stats/.

Do the same for underused splits. Only count PMD-sized folios in thp_underused_split_page, and add a per-order split_underused stat that counts underused splits of every size. The folio's order has to be read before splitting it, since the folio is no longer large once the split succeeds.

There is no functional change to thp_underused_split_page yet, as only PMD-sized folios are split for being underused today.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 ++++++
 include/linux/huge_mm.h                    |  1 +
 mm/huge_memory.c                           | 10 ++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Quoted 11 lines

Documentation/admin-guide/mm/transhuge.rst split_deferred · line 721

Quoted 8 lines
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b187d618452f..6f2d92a2a556 100644
@@ -719,6 +719,12 @@ split_deferred
         it would free up some memory. Pages on split queue are going to
         be split under memory pressure, if splitting is possible.
 

In this same commit, would you mind also doing s/THP/PMD mapped THP/ in thp_underused_split_page section in /proc/vmstat section?

Quoted 6 lines

PMD-size, not PMD-mapped. But yes, it would be nice to update.

With that updated, feel free to use my

I will make this update for v5. Thanks for reviewing this series, Usama and Kiryl.

Quoted 2 lines

--
Kiryl Shutsemau / Kirill A. Shutemov

reviewer Reviewed-by on v4 3/4
#14 lore raw

Re: [PATCH v4 3/4] mm/huge_memory: add a per-order counter for underused splits

In reply to Joanne Koong
Quoted 19 lines · Joanne Koong

thp_underused_split_page counts every folio that the deferred split shrinker splits for being underused. Today that is only ever a PMD-sized folio, but a later patch queues anonymous mTHP folios for underused splitting too, at which point this counter would start counting mTHP splits as well.

thp_underused_split_page could then end up larger than thp_split_page, and there would be no per-order count of underused splits. That doesn't match how the other THP split stats work. In /proc/vmstat, thp_split_page and thp_deferred_split_page only count PMD-sized folios. Splits of every size, PMD included, are counted per order in the split and split_deferred stats under hugepages-<size>/stats/.

Do the same for underused splits. Only count PMD-sized folios in thp_underused_split_page, and add a per-order split_underused stat that counts underused splits of every size. The folio's order has to be read before splitting it, since the folio is no longer large once the split succeeds.

Quoted 6 lines

There is no functional change to thp_underused_split_page yet, as only PMD-sized folios are split for being underused today.

---

LGTM.

#15 lore raw

PATCH v4 4/4 mm: add anonymous mTHP folios to the deferred split list

In reply to Joanne Koong

Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure.

At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed.

Add anonymous mTHP folios to the deferred split list from map_anon_folio_pte_nopf(), mirroring what map_anon_folio_pmd_nopf() already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed.

Only queue mTHP folios of at least 2M for underused splitting. With thp_underused() scaling khugepaged/max_ptes_none to the folio size, any folio of order 2 or above could in principle be underused, but for small folios it is unlikely to be worth it. An order-2 folio on 4k base pages can free at most four pages and on systems that only enable small mTHP sizes (eg Android's order-2-only configuration) zero-filled subpages tend to be few and short-lived.

The minimum is hard-coded rather than a knob, since admins have no good way of choosing it, and is a size rather than an order since what a split recovers depends on the folio's size in bytes, and the same order means very different sizes across different base page sizes. It can be lowered later if data shows smaller sizes are worth it. PMD-sized folios are exempt from the minimum in order to maintain backwards compatibility since on some architectures (s390, with a 1M PMD) the PMD size can be below 2M.

To minimize overhead on the common order-0 fault path, the deferred_split_folio() call is guarded by an inline folio_test_large() check.

---
 mm/huge_memory.c | 14 ++++++++++++++
 mm/memory.c      |  2 ++
 2 files changed, 16 insertions(+)
mm/huge_memory.c +14 -0
index 8f90fb609e57..cda3c886ec53 100644
@@ -4575,6 +4575,13 @@ static bool thp_underused_never(unsigned long max_ptes_none)
 	return max_ptes_none == HPAGE_PMD_NR - 1;
 }
 
+/*
+ * Smallest mTHP folio that is queued for underused splitting. Kept
+ * conservative, as small folios are unlikely to free enough memory to be
+ * worth queueing and scanning.
+ */
+#define THP_UNDERUSED_MIN_SIZE	SZ_2M
+
 /* partially_mapped=false won't clear PG_partially_mapped folio flag */
 void deferred_split_folio(struct folio *folio, bool partially_mapped)
 {
@@ -4600,6 +4607,13 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
 		 */
 		if (thp_underused_never(khugepaged_max_ptes_none))
 			return;
+		/*
+		 * Keep queueing PMD-sized folios as before, even where the PMD
+		 * size is below the minimum (eg 1M on s390).
+		 */
+		if (folio_size(folio) < THP_UNDERUSED_MIN_SIZE &&
+		    !is_pmd_order(folio_order(folio)))
+			return;
 	}
 
 	/*
mm/memory.c +2 -0
index 0c9ed04528d8..44376321180b 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
 	folio_add_lru_vma(folio, vma);
 	set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
 	update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+	if (folio_test_large(folio))
+		deferred_split_folio(folio, false);
 }
 
 static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte,
Signature · 1 line
2.52.0
reviewer Reviewed-by on v4 4/4
#16 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

Quoted 36 lines · Joanne Koong

Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure.

At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed.

Add anonymous mTHP folios to the deferred split list from map_anon_folio_pte_nopf(), mirroring what map_anon_folio_pmd_nopf() already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed.

Only queue mTHP folios of at least 2M for underused splitting. With thp_underused() scaling khugepaged/max_ptes_none to the folio size, any folio of order 2 or above could in principle be underused, but for small folios it is unlikely to be worth it. An order-2 folio on 4k base pages can free at most four pages and on systems that only enable small mTHP sizes (eg Android's order-2-only configuration) zero-filled subpages tend to be few and short-lived.

The minimum is hard-coded rather than a knob, since admins have no good way of choosing it, and is a size rather than an order since what a split recovers depends on the folio's size in bytes, and the same order means very different sizes across different base page sizes. It can be lowered later if data shows smaller sizes are worth it. PMD-sized folios are exempt from the minimum in order to maintain backwards compatibility since on some architectures (s390, with a 1M PMD) the PMD size can be below 2M.

Quoted 7 lines

To minimize overhead on the common order-0 fault path, the deferred_split_folio() call is guarded by an inline folio_test_large() check.

---

I believe we discussed this before and agreed that a hardcoded 2MB would be a better starting point.

#17 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

In reply to Joanne Koong
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

commit 82b7b7063b986d802fe52e75e17211e58f3f420c
Author: Joanne Koong <[email protected]>

mm: add anonymous mTHP folios to the deferred split list

This commit queues newly mapped anonymous mTHP folios of at least 2M onto the deferred split list in map_anon_folio_pte_nopf() so that underused mTHP folios can be split by the deferred split shrinker under memory pressure.

mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5482

Quoted 11 lines
mm/memory.c +2 -0
index 0c9ed04528d88..44376321180b5 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
 	folio_add_lru_vma(folio, vma);
 	set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
 	update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+	if (folio_test_large(folio))
+		deferred_split_folio(folio, false);
 }

[Severity: High]
When deferred_split_scan() splits an underused PTE-mapped mTHP folio queued here with PG_partially_mapped left clear, doesn't unmap_folio() trigger a false-positive deferred_split_folio(folio, true) transition during the split?

In deferred_split_scan(), the folio is first removed from the local dispose list while PG_partially_mapped is still false:

mm/huge_memory.c:deferred_split_scan()

	list_del_init(&folio->_deferred_list);

	if (!folio_test_partially_mapped(folio)) {
		if (folio_test_mlocked(folio))
			goto next;
		underused = thp_underused(folio);
		if (!underused)
			goto next;
	}
	if (!folio_trylock(folio))
		goto requeue;
	if (!split_folio(folio)) {

split_folio() then calls __folio_split() -> __folio_freeze_split_anon(), which unmaps the folio before freezing its refcount:

mm/huge_memory.c:__folio_freeze_split_anon()

	if (folio_mapped(folio)) {
		anon_vma = folio_get_anon_vma(folio);
		if (!anon_vma)
			return -EBUSY;
		anon_vma_lock_write(anon_vma);
		ret = unmap_folio(folio);
		if (ret)
			goto out_unlock;
	}

	local_irq_disable();

	if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
		ret = -EAGAIN;
		goto out_no_split;
	}

Unlike a PMD-mapped THP (where split_huge_pmd_locked() with freeze=true removes the entire PMD rmap in one folio_remove_rmap_pmd() call), an mTHP folio is PTE-mapped, so unmap_folio() -> try_to_migrate() -> try_to_migrate_one() removes its PTEs one subpage at a time via folio_remove_rmap_pte().

On the first subpage unmapped while the remaining subpages are still mapped, __folio_remove_rmap() sees partially_mapped == true and !folio_test_partially_mapped(folio) == true:

mm/rmap.c:__folio_remove_rmap()

	if (partially_mapped && folio_test_anon(folio) &&
	    !folio_test_partially_mapped(folio) &&
	    !folio_is_device_private(folio))
		deferred_split_folio(folio, true);

Because folio->_deferred_list was emptied by list_del_init() in deferred_split_scan() and the folio refcount is not frozen yet, deferred_split_folio(folio, true) sets PG_partially_mapped, increments MTHP_STAT_SPLIT_DEFERRED and MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, and re-adds folio->_deferred_list onto deferred_split_lru.

Could this cause three problems?

1. Even when folio_ref_freeze() succeeds and __folio_freeze_split_anon() subsequently unqueues the folio and resets PG_partially_mapped, every underused mTHP split will also increment MTHP_STAT_SPLIT_DEFERRED in addition to MTHP_STAT_SPLIT_UNDERUSED.

2. If folio_ref_freeze() fails (for example due to a concurrent GUP pin, or because a second CPU running deferred_split_scan() isolated the re-queued folio via folio_try_get() while the remaining PTEs were still being unmapped), __folio_freeze_split_anon() jumps to out_no_split and remaps all PTEs via remap_anon_folio() without unqueueing the folio or clearing PG_partially_mapped. That leaves a fully mapped folio permanently marked PG_partially_mapped with MTHP_STAT_NR_ANON_PARTIALLY_MAPPED leaked.

3. If a second CPU running deferred_split_scan() isolates the re-queued folio onto its local dispose list while unmap_folio() is still in progress, that extra refcount causes folio_ref_freeze() on the first CPU to fail. When the first CPU returns to deferred_split_scan(), it sees !did_split && folio_test_partially_mapped(folio) == true and calls list_lru_add_irq() on folio->_deferred_list while the folio is on the second CPU's unlocked dispose list, corrupting the list.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#18 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

Quoted 4 lines · On Wed, Nov 4, 2026 at 1:21 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

Hi Sashiko,

I'll try to address these points below.

Quoted 10 lines

commit 82b7b7063b986d802fe52e75e17211e58f3f420c
Author: Joanne Koong <[email protected]>

mm: add anonymous mTHP folios to the deferred split list

This commit queues newly mapped anonymous mTHP folios of at least 2M onto the deferred split list in map_anon_folio_pte_nopf() so that underused mTHP folios can be split by the deferred split shrinker under memory pressure.

Quoted 16 lines

mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5482

Quoted 11 lines
mm/memory.c +2 -0
index 0c9ed04528d88..44376321180b5 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
      folio_add_lru_vma(folio, vma);
      set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
      update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+     if (folio_test_large(folio))
+             deferred_split_folio(folio, false);
 }

[Severity: High]
When deferred_split_scan() splits an underused PTE-mapped mTHP folio queued here with PG_partially_mapped left clear, doesn't unmap_folio() trigger a false-positive deferred_split_folio(folio, true) transition during the split?

Yes, but I don't think this is specific to this series. Splitting any PTE-mapped anon large folio that isn't already marked partially mapped goes through the try_to_migrate_one() -> folio_remove_rmap_pte() path you describe below, where __folio_remove_rmap() requeues it as partially mapped mid-split. That already happens today (eg underused splits of PMD THPs whose PMD has been split). This series makes it more frequent since underused mTHP splits now take that path too, but this issue is pre-existing.

Quoted 68 lines

In deferred_split_scan(), the folio is first removed from the local dispose list while PG_partially_mapped is still false:

mm/huge_memory.c:deferred_split_scan()

        list_del_init(&folio->_deferred_list);

        if (!folio_test_partially_mapped(folio)) {
                if (folio_test_mlocked(folio))
                        goto next;
                underused = thp_underused(folio);
                if (!underused)
                        goto next;
        }
        if (!folio_trylock(folio))
                goto requeue;
        if (!split_folio(folio)) {

split_folio() then calls __folio_split() -> __folio_freeze_split_anon(), which unmaps the folio before freezing its refcount:

mm/huge_memory.c:__folio_freeze_split_anon()

        if (folio_mapped(folio)) {
                anon_vma = folio_get_anon_vma(folio);
                if (!anon_vma)
                        return -EBUSY;
                anon_vma_lock_write(anon_vma);
                ret = unmap_folio(folio);
                if (ret)
                        goto out_unlock;
        }

        local_irq_disable();

        if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
                ret = -EAGAIN;
                goto out_no_split;
        }

Unlike a PMD-mapped THP (where split_huge_pmd_locked() with freeze=true removes the entire PMD rmap in one folio_remove_rmap_pmd() call), an mTHP folio is PTE-mapped, so unmap_folio() -> try_to_migrate() -> try_to_migrate_one() removes its PTEs one subpage at a time via folio_remove_rmap_pte().

On the first subpage unmapped while the remaining subpages are still mapped, __folio_remove_rmap() sees partially_mapped == true and !folio_test_partially_mapped(folio) == true:

mm/rmap.c:__folio_remove_rmap()

        if (partially_mapped && folio_test_anon(folio) &&
            !folio_test_partially_mapped(folio) &&
            !folio_is_device_private(folio))
                deferred_split_folio(folio, true);

Because folio->_deferred_list was emptied by list_del_init() in deferred_split_scan() and the folio refcount is not frozen yet, deferred_split_folio(folio, true) sets PG_partially_mapped, increments MTHP_STAT_SPLIT_DEFERRED and MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, and re-adds folio->_deferred_list onto deferred_split_lru.

Could this cause three problems?

1. Even when folio_ref_freeze() succeeds and __folio_freeze_split_anon() subsequently unqueues the folio and resets PG_partially_mapped, every underused mTHP split will also increment MTHP_STAT_SPLIT_DEFERRED in addition to MTHP_STAT_SPLIT_UNDERUSED.

Quoted 7 lines

2. If folio_ref_freeze() fails (for example due to a concurrent GUP pin, or because a second CPU running deferred_split_scan() isolated the re-queued folio via folio_try_get() while the remaining PTEs were still being unmapped), __folio_freeze_split_anon() jumps to out_no_split and remaps all PTEs via remap_anon_folio() without unqueueing the folio or clearing PG_partially_mapped. That leaves a fully mapped folio permanently marked PG_partially_mapped with MTHP_STAT_NR_ANON_PARTIALLY_MAPPED leaked.

This (and #1 above) are pre-existing issues that are already possible for underused splits of PTE-mapped PMD THPs.
I'll look into this and submit a fix for this separately.

Quoted 8 lines

3. If a second CPU running deferred_split_scan() isolates the re-queued folio onto its local dispose list while unmap_folio() is still in progress, that extra refcount causes folio_ref_freeze() on the first CPU to fail. When the first CPU returns to deferred_split_scan(), it sees !did_split && folio_test_partially_mapped(folio) == true and calls list_lru_add_irq() on folio->_deferred_list while the folio is on the second CPU's unlocked dispose list, corrupting the list.

I don't think this can corrupt the list. __list_lru_add() only adds the entry if the "if list_empty(item)" check passes, so while the folio is on the other CPU's dispose list, list_lru_add_irq() returns false and leaves it alone.

Thanks,
Joanne

Quoted 3 lines

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4

#19 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

Quoted 37 lines · Joanne Koong
Quoted 4 lines · On Wed, Nov 4, 2026 at 1:21 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

Hi Sashiko,

I'll try to address these points below.

Quoted 10 lines

commit 82b7b7063b986d802fe52e75e17211e58f3f420c
Author: Joanne Koong <[email protected]>

mm: add anonymous mTHP folios to the deferred split list

This commit queues newly mapped anonymous mTHP folios of at least 2M onto the deferred split list in map_anon_folio_pte_nopf() so that underused mTHP folios can be split by the deferred split shrinker under memory pressure.

Quoted 16 lines

mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5482

Quoted 11 lines
mm/memory.c +2 -0
index 0c9ed04528d88..44376321180b5 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
      folio_add_lru_vma(folio, vma);
      set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
      update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+     if (folio_test_large(folio))
+             deferred_split_folio(folio, false);
 }

[Severity: High]
When deferred_split_scan() splits an underused PTE-mapped mTHP folio queued here with PG_partially_mapped left clear, doesn't unmap_folio() trigger a false-positive deferred_split_folio(folio, true) transition during the split?

Quoted 8 lines

Yes, but I don't think this is specific to this series. Splitting any PTE-mapped anon large folio that isn't already marked partially mapped goes through the try_to_migrate_one() -> folio_remove_rmap_pte() path you describe below, where __folio_remove_rmap() requeues it as partially mapped mid-split. That already happens today (eg underused splits of PMD THPs whose PMD has been split). This series makes it more frequent since underused mTHP splits now take that path too, but this issue is pre-existing.

I think it is a bit of understatement.

For mTHP it is every underused split: the folio is always PTE-mapped and try_to_migrate_one() drops one PTE at a time. Each underused split bumps split_deferred for that size next to the new split_underused, and a split that fails at folio_ref_freeze() leaves a fully mapped folio flagged partially mapped on the queue, which the next scan splits without calling thp_underused().

Can the fix go in before this series? I think it is hard requirement. Let's not brush if off as pre-existing.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#20 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

Quoted 57 lines · Kiryl Shutsemau
Quoted 37 lines · Joanne Koong
Quoted 4 lines · On Wed, Nov 4, 2026 at 1:21 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

Hi Sashiko,

I'll try to address these points below.

Quoted 10 lines

commit 82b7b7063b986d802fe52e75e17211e58f3f420c
Author: Joanne Koong <[email protected]>

mm: add anonymous mTHP folios to the deferred split list

This commit queues newly mapped anonymous mTHP folios of at least 2M onto the deferred split list in map_anon_folio_pte_nopf() so that underused mTHP folios can be split by the deferred split shrinker under memory pressure.

Quoted 16 lines

mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5482

Quoted 11 lines
mm/memory.c +2 -0
index 0c9ed04528d88..44376321180b5 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
      folio_add_lru_vma(folio, vma);
      set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
      update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+     if (folio_test_large(folio))
+             deferred_split_folio(folio, false);
 }

[Severity: High]
When deferred_split_scan() splits an underused PTE-mapped mTHP folio queued here with PG_partially_mapped left clear, doesn't unmap_folio() trigger a false-positive deferred_split_folio(folio, true) transition during the split?

Quoted 8 lines

Yes, but I don't think this is specific to this series. Splitting any PTE-mapped anon large folio that isn't already marked partially mapped goes through the try_to_migrate_one() -> folio_remove_rmap_pte() path you describe below, where __folio_remove_rmap() requeues it as partially mapped mid-split. That already happens today (eg underused splits of PMD THPs whose PMD has been split). This series makes it more frequent since underused mTHP splits now take that path too, but this issue is pre-existing.

I think it is a bit of understatement.

For mTHP it is every underused split: the folio is always PTE-mapped and try_to_migrate_one() drops one PTE at a time. Each underused split bumps split_deferred for that size next to the new split_underused, and a split that fails at folio_ref_freeze() leaves a fully mapped folio flagged partially mapped on the queue, which the next scan splits without calling thp_underused().

Quoted 3 lines

Can the fix go in before this series? I think it is hard requirement. Let's not brush if off as pre-existing.

Sounds good, I am looking into this and will submit a fix separately.

Thanks,
Joanne

#21 lore raw

Re: [PATCH v4 4/4] mm: add anonymous mTHP folios to the deferred split list

Quoted 59 lines · Joanne Koong
Quoted 57 lines · Kiryl Shutsemau
Quoted 37 lines · Joanne Koong
Quoted 4 lines · On Wed, Nov 4, 2026 at 1:21 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: try_to_migrate_one() triggers deferred_split_folio() on underused mTHP --

Hi Sashiko,

I'll try to address these points below.

Quoted 10 lines

commit 82b7b7063b986d802fe52e75e17211e58f3f420c
Author: Joanne Koong <[email protected]>

mm: add anonymous mTHP folios to the deferred split list

This commit queues newly mapped anonymous mTHP folios of at least 2M onto the deferred split list in map_anon_folio_pte_nopf() so that underused mTHP folios can be split by the deferred split shrinker under memory pressure.

Quoted 16 lines

mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5482

Quoted 11 lines
mm/memory.c +2 -0
index 0c9ed04528d88..44376321180b5 100644
@@ -5477,6 +5477,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
      folio_add_lru_vma(folio, vma);
      set_ptes(vma->vm_mm, addr, pte, entry, nr_pages);
      update_mmu_cache_range(NULL, vma, addr, pte, nr_pages);
+     if (folio_test_large(folio))
+             deferred_split_folio(folio, false);
 }

[Severity: High]
When deferred_split_scan() splits an underused PTE-mapped mTHP folio queued here with PG_partially_mapped left clear, doesn't unmap_folio() trigger a false-positive deferred_split_folio(folio, true) transition during the split?

Quoted 8 lines

Yes, but I don't think this is specific to this series. Splitting any PTE-mapped anon large folio that isn't already marked partially mapped goes through the try_to_migrate_one() -> folio_remove_rmap_pte() path you describe below, where __folio_remove_rmap() requeues it as partially mapped mid-split. That already happens today (eg underused splits of PMD THPs whose PMD has been split). This series makes it more frequent since underused mTHP splits now take that path too, but this issue is pre-existing.

I think it is a bit of understatement.

For mTHP it is every underused split: the folio is always PTE-mapped and try_to_migrate_one() drops one PTE at a time. Each underused split bumps split_deferred for that size next to the new split_underused, and a split that fails at folio_ref_freeze() leaves a fully mapped folio flagged partially mapped on the queue, which the next scan splits without calling thp_underused().

Quoted 5 lines
Quoted 3 lines

Can the fix go in before this series? I think it is hard requirement. Let's not brush if off as pre-existing.

Sounds good, I am looking into this and will submit a fix separately.

This is the link to the submitted fix:
https://lore.kernel.org/linux-mm/[email protected]/

I'll wait to submit v5 of this series until I've received feedback on that fix.

Thanks,
Joanne