mm: split underused anonymous mTHP folios
Subsystems (3)
v3 cover mm: split underused anonymous mTHP folios
PMD-sized THPs that are mostly zero-filled are reclaimed under memory pressure by the deferred split shrinker, but this is not done for mTHP folios. At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M provides the contpte benefits while the PMD size there (512M) is too big to use. However, 2M THP=always causes memory regressions unless the unused portions of those folios can be broken down and reclaimed.
This series lets the underused shrinker handle anonymous mTHP folios. It has three patches:
Patch 1 makes thp_underused() folio-size-aware. It scales khugepaged/max_ptes_none to the folio size, so every folio tolerates the same fraction of zero-filled pages.
Patch 2 stops queueing folios for underused splitting when khugepaged/max_ptes_none is at its default maximum, where the shrinker would never split them anyway. This applies to PMD-sized folios too, which are unconditionally queued today.
Patch 3 queues anonymous mTHP folios on the deferred split list, with a hard-coded 2M minimum folio size as suggested by Barry. PMD-sized folios are exempt from the minimum, since on s390 the PMD size is 1M.
At the default khugepaged/max_ptes_none, no folio is queued for underused splitting. Once it's lowered, mTHP folios of at least 2M are queued along with PMD-sized folios.
Changelog
---------
v2: https://lore.kernel.org/linux-mm/[email protected]/
Changes since v2:
* Go back to scaling khugepaged/max_ptes_none per folio size (David)
* Keep the check for the sysctl maximum on the raw, unscaled value (Sashiko, from v1). Round the scaled value up so a folio is never split when fewer than the configured fraction of its pages are zero-filled.
* Only queue mTHP folios of at least 2M, a hard-coded minimum (Barry, David)
v1: https://lore.kernel.org/linux-mm/[email protected]/
Changes since v1:
* Drop the per-order scaling of khugepaged_max_ptes_none. Keep it absolute, matching collapse (David, Johannes)
* New patch 2 (suggested by Johannes): don't queue folios that can never be underused, which both bounds what patch 3 adds and stops queuing PMD folios under default settings (Johannes, David, Barry)
* Fix the thp_underused() early exit to scale to the folio's own size
Joanne Koong (3):
mm/huge_memory: scale thp_underused() to the folio size mm/huge_memory: don't queue folios when underused splitting is
inactive
mm: add anonymous mTHP folios to the deferred split list
mm/huge_memory.c | 58 +++++++++++++++++++++++++++++++++++++++++------- mm/memory.c | 2 ++ 2 files changed, 52 insertions(+), 8 deletions(-)
base-commit: 3fdcce698abd9193752ee89fb1a440cdceb1b74a
Signature · 1 line
2.52.0
PATCH v3 1/3 mm/huge_memory: scale thp_underused() to the folio size
thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed.
Today it only ever runs on PMD-sized folios, so it assumes that the folio has HPAGE_PMD_NR pages and compares the zero-filled page count directly against khugepaged_max_ptes_none. A later patch will queue anonymous mTHP folios onto the deferred split list as well, at which point that assumption will no longer hold.
Make thp_underused() folio-size-aware, and scale khugepaged_max_ptes_none proportionally to the number of pages in the folio (eg every folio tolerates the same fraction of zero-filled pages the user configured for a PMD-sized folio).
khugepaged deliberately doesn't scale intermediate values when collapsing to mTHP orders (to avoid the memory footprint creeping up through repeated collapses), but if/when collapse later gains support for intermediate values, a folio that khugepaged collapses won't then be split right back by the shrinker (and vice versa) since both collapse and split would use the same threshold for a given folio size.
There is no functional change for PMD-sized folios, as for those, nr_pages is HPAGE_PMD_NR.
--- mm/huge_memory.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
mm/huge_memory.c +16 -6
@@ -4613,25 +4613,35 @@ static unsigned long deferred_split_count(struct shrinker *shrink,static bool thp_underused (struct folio *folio) { -int num_zero_pages =0 , num_filled_pages =0 ; -int i; +const unsigned long nr_pages =folio_nr_pages (folio); +unsigned long num_zero_pages =0 , num_filled_pages =0 ; +unsigned long max_ptes_none = khugepaged_max_ptes_none; +unsigned long i; -if (khugepaged_max_ptes_none ==HPAGE_PMD_NR -1 ) +if (max_ptes_none ==HPAGE_PMD_NR -1 )return false;if (folio_contain_hwpoisoned_page (folio))return false; -for (i =0 ; i <folio_nr_pages (folio); i++) { +/* +* khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR. +* Scale it to this folio's size (rounding up so that the folio is never +* split when fewer than the configured fraction of its pages are +* zero-filled). +*/ + max_ptes_none =DIV_ROUND_UP (max_ptes_none * nr_pages,HPAGE_PMD_NR ); + +for (i =0 ; i < nr_pages; i++) {if (pages_identical (folio_page (folio, i),ZERO_PAGE (0 ))) { -if (++num_zero_pages > khugepaged_max_ptes_none) +if (++num_zero_pages > max_ptes_none)return true; }else {/* * Another path for early exit once the number * of non-zero filled pages exceeds threshold. */ -if (++num_filled_pages >=HPAGE_PMD_NR - khugepaged_max_ptes_none) +if (++num_filled_pages >= nr_pages - max_ptes_none)return false; } }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v3 1/3] mm/huge_memory: scale thp_underused() to the folio size
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] mm/huge_memory: wrong rounding in thp_underused() max_ptes_none scaling --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm/huge_memory: wrong rounding in thp_underused() max_ptes_none scaling --
commit 56eb0cfccba9e5e4276f2a8250815399987bc848
Author: Joanne Koong <[email protected]>
mm/huge_memory: scale thp_underused() to the folio size
This commit updates thp_underused() to scale khugepaged_max_ptes_none proportionally to the number of pages in the folio rather than assuming the folio has HPAGE_PMD_NR pages.
Quoted 26 lines
mm/huge_memory.c +8
-4@@ -4613,25 +4613,35 @@ static unsigned long deferred_split_count(struct shrinker *shrink,static bool thp_underused (struct folio *folio) { -int num_zero_pages =0 , num_filled_pages =0 ; -int i; +const unsigned long nr_pages =folio_nr_pages (folio); +unsigned long num_zero_pages =0 , num_filled_pages =0 ; +unsigned long max_ptes_none = khugepaged_max_ptes_none; +unsigned long i; -if (khugepaged_max_ptes_none ==HPAGE_PMD_NR -1 ) +if (max_ptes_none ==HPAGE_PMD_NR -1 )return false;if (folio_contain_hwpoisoned_page (folio))return false; -for (i =0 ; i <folio_nr_pages (folio); i++) { +/* +* khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR. +* Scale it to this folio's size (rounding up so that the folio is never
mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4638
Quoted 10 lines
mm/huge_memory.c +7
-1+* split when fewer than the configured fraction of its pages are +* zero-filled). +*/ + max_ptes_none =DIV_ROUND_UP (max_ptes_none * nr_pages,HPAGE_PMD_NR ); + +for (i =0 ; i < nr_pages; i++) {if (pages_identical (folio_page (folio, i),ZERO_PAGE (0 ))) { -if (++num_zero_pages > khugepaged_max_ptes_none) +if (++num_zero_pages > max_ptes_none)return true;
[Severity: Medium]
Should max_ptes_none in thp_underused() be scaled with truncating integer division rather than DIV_ROUND_UP()?
Because the loop checks a strict inequality:
if (++num_zero_pages > max_ptes_none) return true;
max_ptes_none is the maximum number of zero-filled pages tolerated without splitting, not the minimum count that triggers a split. Since num_zero_pages is an integer, the condition that the zero-filled fraction exceeds the configured PMD fraction (num_zero_pages / nr_pages > khugepaged_max_ptes_none / HPAGE_PMD_NR) is equivalent to:
num_zero_pages > (khugepaged_max_ptes_none * nr_pages) / HPAGE_PMD_NR
using truncating division. With truncating division, any folio with at most the configured fraction of zero-filled pages already satisfies num_zero_pages <= max_ptes_none and is not split.
Rounding max_ptes_none up with DIV_ROUND_UP() alongside the strict > check requires floor((khugepaged_max_ptes_none * nr_pages) / HPAGE_PMD_NR) + 2 zero pages to trigger a split whenever the product is not an exact multiple of HPAGE_PMD_NR, allowing sub-PMD folios that exceed the configured zero-page fraction to escape splitting in deferred_split_scan() -> thp_underused().
In addition, for any sub-PMD folio (nr_pages < HPAGE_PMD_NR), whenever khugepaged_max_ptes_none is in the range
[HPAGE_PMD_NR - HPAGE_PMD_NR / nr_pages + 1, HPAGE_PMD_NR - 2] (for example, 481..510 for an order-4 mTHP on 4K base pages where HPAGE_PMD_NR is 512), DIV_ROUND_UP(max_ptes_none * nr_pages, HPAGE_PMD_NR) evaluates to nr_pages. In that case, ++num_zero_pages > max_ptes_none can never be true during the nr_pages iterations, so even a 100% zero-filled folio scans all nr_pages subpages with pages_identical() and returns false, dropping the folio from deferred_split_lru without splitting.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Lance Yang
reviewerRe: [PATCH v3 1/3] mm/huge_memory: scale thp_underused() to the folio size
Quoted 45 lines · On Sat, Oct 24, 2026 at 11:01:50PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium]mm/huge_memory: wrong rounding inthp_underused()max_ptes_nonescaling --commit
56eb0cfccba9e5e4276f2a8250815399987bc848
Author: Joanne Koong <[email protected]>
mm/huge_memory: scalethp_underused()to the folio sizeThis commit updates
thp_underused()to scalekhugepaged_max_ptes_noneproportionally to the number of pages in the folio rather than assuming the folio hasHPAGE_PMD_NRpages.Quoted 32 lines
mm/huge_memory.c +14
-4@@ -4613,25 +4613,35 @@ static unsigned long deferred_split_count(struct shrinker *shrink,static bool thp_underused (struct folio *folio) { -int num_zero_pages =0 , num_filled_pages =0 ; -int i; +const unsigned long nr_pages =folio_nr_pages (folio); +unsigned long num_zero_pages =0 , num_filled_pages =0 ; +unsigned long max_ptes_none = khugepaged_max_ptes_none; +unsigned long i; -if (khugepaged_max_ptes_none ==HPAGE_PMD_NR -1 ) +if (max_ptes_none ==HPAGE_PMD_NR -1 )return false;if (folio_contain_hwpoisoned_page (folio))return false; -for (i =0 ; i <folio_nr_pages (folio); i++) { +/* +* khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR. +* Scale it to this folio's size (rounding up so that the folio is never +* split when fewer than the configured fraction of its pages are +* zero-filled). +*/ + max_ptes_none =DIV_ROUND_UP (max_ptes_none * nr_pages,HPAGE_PMD_NR ); + +for (i =0 ; i < nr_pages; i++) {
Quoted 8 lines
mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4638
Quoted 4 lines
mm/huge_memory.c +1
-1if (pages_identical (folio_page (folio, i),ZERO_PAGE (0 ))) { -if (++num_zero_pages > khugepaged_max_ptes_none) +if (++num_zero_pages > max_ptes_none)return true;[Severity: Medium]
Shouldmax_ptes_noneinthp_underused()be scaled with truncating integer division rather thanDIV_ROUND_UP()?
Agreed, we already use a strict > comparison, so truncating division is enough here.
[...]
Cheers, Lance
Re: [PATCH v3 1/3] mm/huge_memory: scale thp_underused() to the folio size
Quoted 57 lines · Lance Yang
Quoted 45 lines · On Sat, Oct 24, 2026 at 11:01:50PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium]mm/huge_memory: wrong rounding inthp_underused()max_ptes_nonescaling --commit
56eb0cfccba9e5e4276f2a8250815399987bc848
Author: Joanne Koong <[email protected]>
mm/huge_memory: scalethp_underused()to the folio sizeThis commit updates
thp_underused()to scalekhugepaged_max_ptes_noneproportionally to the number of pages in the folio rather than assuming the folio hasHPAGE_PMD_NRpages.Quoted 32 lines
mm/huge_memory.c +14
-4@@ -4613,25 +4613,35 @@ static unsigned long deferred_split_count(struct shrinker *shrink,static bool thp_underused (struct folio *folio) { -int num_zero_pages =0 , num_filled_pages =0 ; -int i; +const unsigned long nr_pages =folio_nr_pages (folio); +unsigned long num_zero_pages =0 , num_filled_pages =0 ; +unsigned long max_ptes_none = khugepaged_max_ptes_none; +unsigned long i; -if (khugepaged_max_ptes_none ==HPAGE_PMD_NR -1 ) +if (max_ptes_none ==HPAGE_PMD_NR -1 )return false;if (folio_contain_hwpoisoned_page (folio))return false; -for (i =0 ; i <folio_nr_pages (folio); i++) { +/* +* khugepaged_max_ptes_none is a number of pages out of HPAGE_PMD_NR. +* Scale it to this folio's size (rounding up so that the folio is never +* split when fewer than the configured fraction of its pages are +* zero-filled). +*/ + max_ptes_none =DIV_ROUND_UP (max_ptes_none * nr_pages,HPAGE_PMD_NR ); + +for (i =0 ; i < nr_pages; i++) {Quoted 8 lines
mm/huge_memory.c static unsigned long deferred_split_count(struct shrinker *shrink, · line 4638
Quoted 4 lines
mm/huge_memory.c +1
-1if (pages_identical (folio_page (folio, i),ZERO_PAGE (0 ))) { -if (++num_zero_pages > khugepaged_max_ptes_none) +if (++num_zero_pages > max_ptes_none)return true;[Severity: Medium]
Shouldmax_ptes_noneinthp_underused()be scaled with truncating integer division rather thanDIV_ROUND_UP()?
Quoted 3 lines
Agreed, we already use a strict > comparison, so truncating division is enough here.
Agreed, thanks. I'll switch back to truncating division in v4.
Thanks,
Joanne
PATCH v3 2/3 mm/huge_memory: don't queue folios when underused splitting is inactive
In reply to Joanne KoongEvery anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.
With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.
Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into thp_underused_never() so that deferred_split_folio() and thp_underused() share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.
This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.
Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.
--- mm/huge_memory.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
mm/huge_memory.c +21 -3
@@ -4552,6 +4552,15 @@ bool __folio_unqueue_deferred_split(struct folio *folio)return unqueued;/* useful for debug warnings */ } +/* +* At the sysctl maximum the user tolerates any number of zero-filled pages, +* so no folio is ever underused. +*/ +static bool thp_underused_never (unsigned long max_ptes_none) +{ +return max_ptes_none ==HPAGE_PMD_NR -1 ; +} +/* partially_mapped=false won't clear PG_partially_mapped folio flag */ void deferred_split_folio (struct folio *folio,bool partially_mapped) {
@@ -4567,8 +4576,17 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)if (folio_order (folio) <=1 )return ; -if (!partially_mapped && !split_underused_thp) -return ; +if (!partially_mapped) { +if (!split_underused_thp) +return ; +/* +* Nothing will ever split this folio for being underused, so +* keep it off the queue entirely rather than paying for the +* list_lru lock here and a shrinker scan later. +*/ +if (thp_underused_never (khugepaged_max_ptes_none)) +return ; + }/* * Exclude swapcache: originally to avoid a corrupt deferred split
@@ -4618,7 +4636,7 @@ static bool thp_underused(struct folio *folio)unsigned long max_ptes_none = khugepaged_max_ptes_none;unsigned long i; -if (max_ptes_none ==HPAGE_PMD_NR -1 ) +if (thp_underused_never (max_ptes_none))return false;if (folio_contain_hwpoisoned_page (folio))
Signature · 1 line
2.52.0
Re: [PATCH v3 2/3] mm/huge_memory: don't queue folios when underused splitting is inactive
Quoted 29 lines · Joanne Koong
Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled.
With the default khugepaged/max_ptes_none this is wasted work. The default is
HPAGE_PMD_NR- 1, which tellsthp_underused()that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take thelist_lrulock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned.Skip the queueing when khugepaged/max_ptes_none is at its maximum, and factor that check out into
thp_underused_never()so thatdeferred_split_folio()andthp_underused()share it. Since the default is the maximum, out of the box no folio is queued for underused splitting at all, PMD-sized ones included. Partially mapped folios are still queued as before.This also matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it keeps systems running the default from paying for that on every anonymous fault.
Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch.
Quoted 3 lines
Suggested-by Johannes Weiner <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Joanne Koong <[email protected]>
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
PATCH v3 3/3 mm: add anonymous mTHP folios to the deferred split list
In reply to Joanne KoongUnlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure.
At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed.
Add anonymous mTHP folios to the deferred split list from map_anon_folio_pte_nopf(), mirroring what map_anon_folio_pmd_nopf() already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed.
Only queue mTHP folios of at least 2M for underused splitting. With thp_underused() scaling khugepaged/max_ptes_none to the folio size, any folio of order 2 or above could in principle be underused, but for small folios it is unlikely to be worth it. An order-2 folio on 4k base pages can free at most four pages and on systems that only enable small mTHP sizes (eg Android's order-2-only configuration) zero-filled subpages tend to be few and short-lived.
The minimum is hard-coded rather than a knob, since admins have no good way of choosing it, and is a size rather than an order since what a split recovers depend on the folio's size in bytes, and the same order means very different sizes across different base page sizes. It can be lowered later if data shows smaller sizes are worth it. PMD-sized folios are exempt from the minimum in order to maintain backwards compatibility since on some architectures (s390, with a 1M PMD) the PMD size can be below 2M.
To minimize overhead on the common order-0 fault path, the deferred_split_folio() call is guarded by an inline folio_test_large() check.
--- mm/huge_memory.c | 14 ++++++++++++++ mm/memory.c | 2 ++ 2 files changed, 16 insertions(+)
mm/huge_memory.c +14 -0
@@ -4561,6 +4561,13 @@ static bool thp_underused_never(unsigned long max_ptes_none)return max_ptes_none ==HPAGE_PMD_NR -1 ; } +/* +* Smallest mTHP folio that is queued for underused splitting. Kept +* conservative, as small folios are unlikely to free enough memory to be +* worth queueing and scanning. +*/ +#define THP_UNDERUSED_MIN_SIZE SZ_2M +/* partially_mapped=false won't clear PG_partially_mapped folio flag */ void deferred_split_folio (struct folio *folio,bool partially_mapped) {
@@ -4586,6 +4593,13 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)*/ if (thp_underused_never (khugepaged_max_ptes_none))return ; +/* +* Keep queueing PMD-sized folios as before, even where the PMD +* size is below the minimum (eg 1M on s390). +*/ +if (folio_size (folio) <THP_UNDERUSED_MIN_SIZE && + !is_pmd_order (folio_order (folio))) +return ; }/*
mm/memory.c +2 -0
@@ -5501,6 +5501,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,folio_add_lru_vma (folio, vma);set_ptes (vma->vm_mm, addr, pte, entry, nr_pages);update_mmu_cache_range (NULL , vma, addr, pte, nr_pages); +if (folio_test_large (folio)) +deferred_split_folio (folio, false); } static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte,
Signature · 1 line
2.52.0
Kiryl Shutsemau
reviewerRe: [PATCH v3 3/3] mm: add anonymous mTHP folios to the deferred split list
Quoted 26 lines · Joanne Koong
Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure.
At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed.
Add anonymous mTHP folios to the deferred split list from
map_anon_folio_pte_nopf(), mirroring whatmap_anon_folio_pmd_nopf()already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed.Only queue mTHP folios of at least 2M for underused splitting. With
thp_underused()scaling khugepaged/max_ptes_none to the folio size, any folio of order 2 or above could in principle be underused, but for small folios it is unlikely to be worth it. An order-2 folio on 4k base pages can free at most four pages and on systems that only enable small mTHP sizes (eg Android's order-2-only configuration) zero-filled subpages tend to be few and short-lived.
Quoted 3 lines
The minimum is hard-coded rather than a knob, since admins have no good way of choosing it, and is a size rather than an order since what a split recovers depend on the folio's size in bytes, and the same order
s/depend/depends/
Quoted 5 lines
means very different sizes across different base page sizes. It can be lowered later if data shows smaller sizes are worth it. PMD-sized folios are exempt from the minimum in order to maintain backwards compatibility since on some architectures (s390, with a 1M PMD) the PMD size can be below 2M.
...
mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5506
Quoted 11 lines
mm/memory.c +2
-0@@ -5501,6 +5501,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,folio_add_lru_vma (folio, vma);set_ptes (vma->vm_mm, addr, pte, entry, nr_pages);update_mmu_cache_range (NULL , vma, addr, pte, nr_pages); +if (folio_test_large (folio)) +deferred_split_folio (folio, false); }
With this, thp_underused_split_page starts counting mTHP splits. thp_split_page and thp_deferred_split_page count PMD-sized folios only, and mTHPs get per-order stats instead. thp_underused_split_page can now be larger than thp_split_page, and the per-order split counter does not tell underused splits from the rest.
Should thp_underused_split_page stay PMD-only, with a per-order counter for underused splits?
This is messy. I am not sure excluding non-PMD splits from the counter is the right call, but it seems to be the most consistent solution.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v3 3/3] mm: add anonymous mTHP folios to the deferred split list
Quoted 28 lines · Kiryl Shutsemau
Quoted 26 lines · Joanne Koong
Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure.
At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed.
Add anonymous mTHP folios to the deferred split list from
map_anon_folio_pte_nopf(), mirroring whatmap_anon_folio_pmd_nopf()already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed.Only queue mTHP folios of at least 2M for underused splitting. With
thp_underused()scaling khugepaged/max_ptes_none to the folio size, any folio of order 2 or above could in principle be underused, but for small folios it is unlikely to be worth it. An order-2 folio on 4k base pages can free at most four pages and on systems that only enable small mTHP sizes (eg Android's order-2-only configuration) zero-filled subpages tend to be few and short-lived.
Quoted 4 lines
Quoted 3 lines
The minimum is hard-coded rather than a knob, since admins have no good way of choosing it, and is a size rather than an order since what a split recovers depend on the folio's size in bytes, and the same order
Hi Kiryl,
Thanks for taking a look at this patch.
Quoted 1 line
s/depend/depends/
I will fix this in v4.
Quoted 27 lines
Quoted 6 lines
means very different sizes across different base page sizes. It can be lowered later if data shows smaller sizes are worth it. PMD-sized folios are exempt from the minimum in order to maintain backwards compatibility since on some architectures (s390, with a 1M PMD) the PMD size can be below 2M.
...
mm/memory.c void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, · line 5506
Quoted 11 lines
mm/memory.c +2
-0@@ -5501,6 +5501,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,folio_add_lru_vma (folio, vma);set_ptes (vma->vm_mm, addr, pte, entry, nr_pages);update_mmu_cache_range (NULL , vma, addr, pte, nr_pages); +if (folio_test_large (folio)) +deferred_split_folio (folio, false); }With this,
thp_underused_split_pagestarts counting mTHP splits.thp_split_pageandthp_deferred_split_pagecount PMD-sized folios only, and mTHPs get per-order stats instead.thp_underused_split_pagecan now be larger thanthp_split_page, and the per-order split counter does not tell underused splits from the rest.
Quoted 5 lines
Should
thp_underused_split_pagestay PMD-only, with a per-order counter for underused splits?This is messy. I am not sure excluding non-PMD splits from the counter is the right call, but it seems to be the most consistent solution.
This is a great point. Your suggestion makes sense to me, to keep it consistent with how we handle thp_split_page and thp_deferred_split_page.
I'll make this change for v4, unless anyone else thinks there is a cleaner way of accounting this.
Thanks,
Joanne