Skip to contentSkip to conversation

mm: split underused THPs

PATCH In review Usama Arif 7 participants 33 messages Aug 30, 2024 – Sep 19, 2025

Subsystems (13)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v5 cover mm: split underused THPs

The current upstream default policy for THP is always. However, Meta uses madvise in production as the current THP=always policy vastly overprovisions THPs in sparsely accessed memory areas, resulting in excessive memory pressure and premature OOM killing.
Using madvise + relying on khugepaged has certain drawbacks over THP=always. Using madvise hints mean THPs aren't "transparent" and require userspace changes. Waiting for khugepaged to scan memory and collapse pages into THP can be slow and unpredictable in terms of performance (i.e. you dont know when the collapse will happen), while production environments require predictable performance. If there is enough memory available, its better for both performance and predictability to have a THP from fault time, i.e. THP=always rather than wait for khugepaged to collapse it, and deal with sparsely populated THPs when the system is running out of memory.

This patch-series is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in or collapsed by khugepaged, the THP is added to a list. Whenever memory reclaim happens, the kernel runs the deferred_split shrinker which goes through the list and checks if the THP was underused, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold, the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory. This method avoids the downside of wasting memory in areas where THP is sparsely filled when THP is always enabled, while still providing the upside THPs like reduced TLB misses without having to use madvise.

Meta production workloads that were CPU bound (>99% CPU utilzation) were tested with THP shrinker. The results after 2 hours are as follows:

                            | THP=madvise |  THP=always   | THP=always
                            |             |               | + shrinker series
                            |             |               | + max_ptes_none=409
-----------------------------------------------------------------------------
Performance improvement     |      -      |    +1.8%      |     +1.7%
(over THP=madvise)          |             |               |

-----------------------------------------------------------------------------
Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%)
-----------------------------------------------------------------------------
max_ptes_none=409 means that any THP that has more than 409 out of 512 (80%) zero filled filled pages will be split.

To test out the patches, the below commands without the shrinker will invoke OOM killer immediately and kill stress, but will not fail with the shrinker:

echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none mkdir /sys/fs/cgroup/test
echo $$ > /sys/fs/cgroup/test/cgroup.procs
echo 20M > /sys/fs/cgroup/test/memory.max
echo 0 > /sys/fs/cgroup/test/memory.swap.max

# allocate twice memory.max for each stress worker and touch 40/512 of
# each THP, i.e. vm-stride 50K.
# With the shrinker, max_ptes_none of 470 and below won't invoke OOM
# killer.
# Without the shrinker, OOM killer is invoked immediately irrespective
# of max_ptes_none value and kills stress.
stress --vm 1 --vm-bytes 40M --vm-stride 50K

---
v4 -> v5:
- rebase on top of latest mm-unstable. This includes Barrys' anon mTHP accounting series. All merge conflicts should be resolved with the patches on mailing list. This also means all places where partially_mapped flag is set is first tested for it (Barry).
- uint64_t to unsigned long for rss_anon function in selftest.
- convert PageMlocked to folio_test_mlocked in try_to_map_unused_to_zeropage.

v3 -> v4:
- do not clear partially_mapped flag on hugeTLB folios (Yu Zhao).
- fix condition for calling deferred_folio_split in partially mapped case and count for partially mapped vm events (Barry Song).
- use non-atomic versions of set/clear partially_mapped flags (David Hildenbrand)
- use PG_partially_mapped = PG_reclaim (Matthew Wilcox)
- delete folio from lru list and folio_batch_add "new_folio" instead of folio in __split_huge_page. (Kairui Song)
- fix deadlock in deferred_split_scan by not doing folio_put while holding split_queue_lock (Hugh Dickins)
- underutilized to underused and thp_low_util_shrinker to shrink_underused (Hugh Dickins)

v2 -> v3:
- Use my_zero_pfn instead of page_to_pfn(ZERO_PAGE(..)) (Johannes)
- Use flags argument instead of bools in remove_migration_ptes (Johannes)
- Use a new flag in folio->_flags_1 instead of folio->_partially_mapped (David Hildenbrand).
- Split out the last patch of v2 into 3, one for introducing the flag, one for splitting underutilized THPs on _deferred_list and one for adding sysfs entry to disable splitting (David Hildenbrand).

v1 -> v2:
- Turn page checks and operations to folio versions in __split_huge_page. This means patches 1 and 2 from v1 are no longer needed. (David Hildenbrand)
- Map to shared zeropage in all cases if the base page is zero-filled. The uffd selftest was removed.
(David Hildenbrand).
- rename 'dirty' to 'contains_data' in try_to_map_unused_to_zeropage (Rik van Riel).
- Use unsigned long instead of uint64_t (kernel test robot).

Alexander Zhu (1):
mm: selftest to verify zero-filled pages are mapped to zeropage

Usama Arif (3):
mm: Introduce a pageflag for partially mapped folios
mm: split underused THPs
mm: add sysfs entry to disable splitting underused THPs

Yu Zhao (2):
mm: free zapped tail pages when splitting isolated thp
mm: remap unused subpages to shared zeropage when splitting isolated
thp

 Documentation/admin-guide/mm/transhuge.rst    |  16 ++
 include/linux/huge_mm.h                       |   4 +-
 include/linux/khugepaged.h                    |   1 +
 include/linux/page-flags.h                    |  13 +-
 include/linux/rmap.h                          |   7 +-
 include/linux/vm_event_item.h                 |   1 +
 mm/huge_memory.c                              | 163 ++++++++++++++++--
 mm/khugepaged.c                               |   3 +-
 mm/memcontrol.c                               |   3 +-
 mm/migrate.c                                  |  75 ++++++--
 mm/migrate_device.c                           |   4 +-
 mm/page_alloc.c                               |   5 +-
 mm/rmap.c                                     |   5 +-
 mm/vmscan.c                                   |   3 +-
 mm/vmstat.c                                   |   1 +
 .../selftests/mm/split_huge_page_test.c       |  71 ++++++++
 tools/testing/selftests/mm/vm_util.c          |  22 +++
 tools/testing/selftests/mm/vm_util.h          |   1 +
 18 files changed, 358 insertions(+), 40 deletions(-)
Signature · 1 line
2.43.5
author
#1 lore raw

PATCH v5 1/6 mm: free zapped tail pages when splitting isolated thp

From: Yu Zhao <[email protected]>

If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.

---
 mm/huge_memory.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
mm/huge_memory.c +28 -0
index 15418ffdd377..0c48806ccb9a 100644
@@ -3170,7 +3170,9 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 	unsigned int new_nr = 1 << new_order;
 	int order = folio_order(folio);
 	unsigned int nr = 1 << order;
+	struct folio_batch free_folios;
 
+	folio_batch_init(&free_folios);
 	/* complete memcg works before add pages to LRU */
 	split_page_memcg(head, order, new_order);
 
@@ -3254,6 +3256,27 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 		if (subpage == page)
 			continue;
 		folio_unlock(new_folio);
+		/*
+		 * If a folio has only two references left, one inherited
+		 * from the isolation of its head and the other from
+		 * lru_add_page_tail() which we are about to drop, it means this
+		 * folio was concurrently zapped. Then we can safely free it
+		 * and save page reclaim or migration the trouble of trying it.
+		 */
+		if (list && folio_ref_freeze(new_folio, 2)) {
+			VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new_folio), new_folio);
+			VM_WARN_ON_ONCE_FOLIO(folio_test_large(new_folio), new_folio);
+			VM_WARN_ON_ONCE_FOLIO(folio_mapped(new_folio), new_folio);
+
+			folio_clear_active(new_folio);
+			folio_clear_unevictable(new_folio);
+			list_del(&new_folio->lru);
+			if (!folio_batch_add(&free_folios, new_folio)) {
+				mem_cgroup_uncharge_folios(&free_folios);
+				free_unref_folios(&free_folios);
+			}
+			continue;
+		}
 
 		/*
 		 * Subpages may be freed if there wasn't any mapping
@@ -3264,6 +3287,11 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 		 */
 		free_page_and_swap_cache(subpage);
 	}
+
+	if (free_folios.nr) {
+		mem_cgroup_uncharge_folios(&free_folios);
+		free_unref_folios(&free_folios);
+	}
 }
 
 /* Racy check whether the huge page can be split */
Signature · 1 line
2.43.5
#2 lore raw

Re: [PATCH v5 1/6] mm: free zapped tail pages when splitting isolated thp

Quoted 8 lines · Usama Arif

From: Yu Zhao <[email protected]>

If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.

Quoted 4 lines

I'm sorry, but I think this patch (just this 1/6) needs to be dropped:
it is only an optimization, and unless a persuasive performance case can be made to extend it, it ought to go (perhaps revisited later).

The problem I kept hitting was that all my work, requiring compaction and reclaim, got (killably) stuck in or repeatedly calling reclaim_throttle():
because nr_isolated_anon had grown high - and remained high even when the load had all been killed.

Bisection led to the 2/6 (remap to shared zeropage), but I'd say this 1/6 is the one to blame. I was intending to send this patch to "fix" it:

mm/huge_memory.c +2 -0
@@ -3295,6 +3295,8 @@ static void __split_huge_page(struct pag
 			folio_clear_active(new_folio);
 			folio_clear_unevictable(new_folio);
 			list_del(&new_folio->lru);
+			node_stat_sub_folio(folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(folio));
 			if (!folio_batch_add(&free_folios, new_folio)) {
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);

And that ran nicely, until I terminated the run and did grep nr_isolated /proc/sys/vm/stat_refresh /proc/vmstat at the end: stat_refresh kindly left a pr_warn in dmesg to say nr_isolated_anon -334013737

My patch is not good enough. IIUC, some split_huge_pagers (reclaim?) know how many pages they isolated and decremented the stats by, and increment by that same number at the end; whereas other split_huge_pagers (migration?) decrement one by one as they go through the list afterwards.

I've run out of time (I'm about to take a break): I gave up researching who needs what, and was already feeling this optimization does too much second guessing of what's needed (and its array of VM_WARN_ON_ONCE_FOLIOs rather admits to that).

And I don't think it's as simple as moving the node_stat_sub_folio() into 2/6 where the zero pte is substituted: that would probably handle the vast majority of cases, but aren't there others which pass the folio_ref_freeze(new_folio, 2) test - the title's zapped tail pages, or racily truncated now that the folio has been unlocked, for example?

Hugh

author
#3 lore raw

Re: [PATCH v5 1/6] mm: free zapped tail pages when splitting isolated thp

Quoted 10 lines · Hugh Dickins
Quoted 8 lines · Usama Arif

From: Yu Zhao <[email protected]>

If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.

Quoted 9 lines
Quoted 4 lines

I'm sorry, but I think this patch (just this 1/6) needs to be dropped:
it is only an optimization, and unless a persuasive performance case can be made to extend it, it ought to go (perhaps revisited later).

I am ok for patch 1 only to be dropped. Patches 2-6 are not dependent on it.

Its an optimization and underused shrinker doesn't depend on it. Its possible that folio->new_folio below might fix it? But if it doesn't, I can retry later on to make this work and resend it only if it alone shows a significant performance improvement.

Thanks a lot for debugging this! and sorry it caused an issue.

Quoted 11 lines

The problem I kept hitting was that all my work, requiring compaction and reclaim, got (killably) stuck in or repeatedly calling reclaim_throttle():
because nr_isolated_anon had grown high - and remained high even when the load had all been killed.

Bisection led to the 2/6 (remap to shared zeropage), but I'd say this 1/6 is the one to blame. I was intending to send this patch to "fix" it:

mm/huge_memory.c static void __split_huge_page(struct pag · line 3299

Quoted 5 lines
mm/huge_memory.c +2 -0
@@ -3295,6 +3295,8 @@ static void __split_huge_page(struct pag
 			folio_clear_active(new_folio);
 			folio_clear_unevictable(new_folio);
 			list_del(&new_folio->lru);
+			node_stat_sub_folio(folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(folio));

Maybe this should have been below? (Notice the folio->new_folio)

+ node_stat_sub_folio(new_folio, NR_ISOLATED_ANON +
+ folio_is_file_lru(new_folio));

Quoted 26 lines
mm/huge_memory.c +0 -0
 			if (!folio_batch_add(&free_folios, new_folio)) {
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);

And that ran nicely, until I terminated the run and did grep nr_isolated /proc/sys/vm/stat_refresh /proc/vmstat at the end: stat_refresh kindly left a pr_warn in dmesg to say nr_isolated_anon -334013737

My patch is not good enough. IIUC, some split_huge_pagers (reclaim?) know how many pages they isolated and decremented the stats by, and increment by that same number at the end; whereas other split_huge_pagers (migration?) decrement one by one as they go through the list afterwards.

I've run out of time (I'm about to take a break): I gave up researching who needs what, and was already feeling this optimization does too much second guessing of what's needed (and its array of VM_WARN_ON_ONCE_FOLIOs rather admits to that).

And I don't think it's as simple as moving the node_stat_sub_folio() into 2/6 where the zero pte is substituted: that would probably handle the vast majority of cases, but aren't there others which pass the folio_ref_freeze(new_folio, 2) test - the title's zapped tail pages, or racily truncated now that the folio has been unlocked, for example?

Hugh

#4 lore raw

Re: [PATCH v5 1/6] mm: free zapped tail pages when splitting isolated thp

Quoted 50 lines · Usama Arif
Quoted 10 lines · Hugh Dickins
Quoted 8 lines · Usama Arif

From: Yu Zhao <[email protected]>

If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.

Quoted 9 lines
Quoted 4 lines

I'm sorry, but I think this patch (just this 1/6) needs to be dropped:
it is only an optimization, and unless a persuasive performance case can be made to extend it, it ought to go (perhaps revisited later).

I am ok for patch 1 only to be dropped. Patches 2-6 are not dependent on it.

Its an optimization and underused shrinker doesn't depend on it. Its possible that folio->new_folio below might fix it? But if it doesn't, I can retry later on to make this work and resend it only if it alone shows a significant performance improvement.

Thanks a lot for debugging this! and sorry it caused an issue.

Quoted 11 lines

The problem I kept hitting was that all my work, requiring compaction and reclaim, got (killably) stuck in or repeatedly calling reclaim_throttle():
because nr_isolated_anon had grown high - and remained high even when the load had all been killed.

Bisection led to the 2/6 (remap to shared zeropage), but I'd say this 1/6 is the one to blame. I was intending to send this patch to "fix" it:

mm/huge_memory.c static void __split_huge_page(struct pag · line 3299

Quoted 5 lines
mm/huge_memory.c +2 -0
@@ -3295,6 +3295,8 @@ static void __split_huge_page(struct pag
 			folio_clear_active(new_folio);
 			folio_clear_unevictable(new_folio);
 			list_del(&new_folio->lru);
+			node_stat_sub_folio(folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(folio));

Maybe this should have been below? (Notice the folio->new_folio)

Review

Quoted 2 lines
(quoted diff) +2 -0
+			node_stat_sub_folio(new_folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(new_folio));

Yes, how very stupid of me (I'm well aware of the earlier bugfixes here, and ought not to have done a blind copy and paste of those lines): thanks.

However... it makes no difference. I gave yours a run, expecting a much smaller negative number, but actually it works out much the same:
because, of course, by this point in the code "folio" is left pointing to a small folio, and is almost equivalent to new_folio for the node_stat calculations.

(I say "almost" because I guess there's a chance that the page at folio got reused as part of a larger folio meanwhile, which would throw the stats off (if they weren't off already).)

So, even with your fix to my fix, this code doesn't work. Whereas a revert of this 1/6 does work: nr_isolated_anon and nr_isolated_file come to 0 when the system is at rest, as expected (and as silence from vmstat_refresh confirms - /proc/vmstat itself presents negative stats as 0, in order to hide transient oddities).

Hugh

Quoted 27 lines
Quoted 27 lines
mm/huge_memory.c +0 -0
 			if (!folio_batch_add(&free_folios, new_folio)) {
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);

And that ran nicely, until I terminated the run and did grep nr_isolated /proc/sys/vm/stat_refresh /proc/vmstat at the end: stat_refresh kindly left a pr_warn in dmesg to say nr_isolated_anon -334013737

My patch is not good enough. IIUC, some split_huge_pagers (reclaim?) know how many pages they isolated and decremented the stats by, and increment by that same number at the end; whereas other split_huge_pagers (migration?) decrement one by one as they go through the list afterwards.

I've run out of time (I'm about to take a break): I gave up researching who needs what, and was already feeling this optimization does too much second guessing of what's needed (and its array of VM_WARN_ON_ONCE_FOLIOs rather admits to that).

And I don't think it's as simple as moving the node_stat_sub_folio() into 2/6 where the zero pte is substituted: that would probably handle the vast majority of cases, but aren't there others which pass the folio_ref_freeze(new_folio, 2) test - the title's zapped tail pages, or racily truncated now that the folio has been unlocked, for example?

Hugh

author
#5 lore raw

Re: [PATCH v5 1/6] mm: free zapped tail pages when splitting isolated thp

Quoted 67 lines · Hugh Dickins
Quoted 50 lines · Usama Arif
Quoted 10 lines · Hugh Dickins
Quoted 8 lines · Usama Arif

From: Yu Zhao <[email protected]>

If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it.

Quoted 9 lines
Quoted 4 lines

I'm sorry, but I think this patch (just this 1/6) needs to be dropped:
it is only an optimization, and unless a persuasive performance case can be made to extend it, it ought to go (perhaps revisited later).

I am ok for patch 1 only to be dropped. Patches 2-6 are not dependent on it.

Its an optimization and underused shrinker doesn't depend on it. Its possible that folio->new_folio below might fix it? But if it doesn't, I can retry later on to make this work and resend it only if it alone shows a significant performance improvement.

Thanks a lot for debugging this! and sorry it caused an issue.

Quoted 11 lines

The problem I kept hitting was that all my work, requiring compaction and reclaim, got (killably) stuck in or repeatedly calling reclaim_throttle():
because nr_isolated_anon had grown high - and remained high even when the load had all been killed.

Bisection led to the 2/6 (remap to shared zeropage), but I'd say this 1/6 is the one to blame. I was intending to send this patch to "fix" it:

mm/huge_memory.c static void __split_huge_page(struct pag · line 3299

Quoted 5 lines
mm/huge_memory.c +2 -0
@@ -3295,6 +3295,8 @@ static void __split_huge_page(struct pag
 			folio_clear_active(new_folio);
 			folio_clear_unevictable(new_folio);
 			list_del(&new_folio->lru);
+			node_stat_sub_folio(folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(folio));

Maybe this should have been below? (Notice the folio->new_folio)

Review

Quoted 2 lines
(quoted diff) +2 -0
+			node_stat_sub_folio(new_folio, NR_ISOLATED_ANON +
+						folio_is_file_lru(new_folio));

Yes, how very stupid of me (I'm well aware of the earlier bugfixes here, and ought not to have done a blind copy and paste of those lines): thanks.

However... it makes no difference. I gave yours a run, expecting a much smaller negative number, but actually it works out much the same:
because, of course, by this point in the code "folio" is left pointing to a small folio, and is almost equivalent to new_folio for the node_stat calculations.

(I say "almost" because I guess there's a chance that the page at folio got reused as part of a larger folio meanwhile, which would throw the stats off (if they weren't off already).)

Quoted 7 lines

So, even with your fix to my fix, this code doesn't work. Whereas a revert of this 1/6 does work: nr_isolated_anon and nr_isolated_file come to 0 when the system is at rest, as expected (and as silence from vmstat_refresh confirms - /proc/vmstat itself presents negative stats as 0, in order to hide transient oddities).

Hugh

Thanks for trying. I was hoping you had mTHPs enabled and then the folio -> new_folio change would have fixed it.

Happy for patch 1 only to be dropped. I can try to figure it out later and send if I can actually show any performance numbers for the fixed version on real world cases.

Thanks,
Usama

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
mm/huge_memory.c +0 -0
 			if (!folio_batch_add(&free_folios, new_folio)) {
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);

And that ran nicely, until I terminated the run and did grep nr_isolated /proc/sys/vm/stat_refresh /proc/vmstat at the end: stat_refresh kindly left a pr_warn in dmesg to say nr_isolated_anon -334013737

My patch is not good enough. IIUC, some split_huge_pagers (reclaim?) know how many pages they isolated and decremented the stats by, and increment by that same number at the end; whereas other split_huge_pagers (migration?) decrement one by one as they go through the list afterwards.

I've run out of time (I'm about to take a break): I gave up researching who needs what, and was already feeling this optimization does too much second guessing of what's needed (and its array of VM_WARN_ON_ONCE_FOLIOs rather admits to that).

And I don't think it's as simple as moving the node_stat_sub_folio() into 2/6 where the zero pte is substituted: that would probably handle the vast majority of cases, but aren't there others which pass the folio_ref_freeze(new_folio, 2) test - the title's zapped tail pages, or racily truncated now that the folio has been unlocked, for example?

Hugh

author
#6 lore raw

PATCH v5 2/6 mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |  4 +--
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
 		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
 	return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
 	int i = 0;
 
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
 	if (!folio_test_anon(folio))
 		return;
 	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
 		i += folio_nr_pages(folio);
 		if (i >= nr)
 			break;
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 
 	if (nr_dropped)
 		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
 	/*
 	 * set page to its compound_head when split to non order-0 pages, so
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
 		if (mapping)
 			xas_unlock(&xas);
 		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
 		ret = -EAGAIN;
 	}
 
mm/migrate.c +63 -9

124 lines not on this page · raw message

mm/migrate_device.c +2 -2
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 			continue;
 
 		folio = page_folio(page);
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
 		folio_unlock(folio);
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			dst = src;
 		}
 
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 		folio_unlock(src);
 
 		if (folio_is_zone_device(src))
Signature · 1 line
2.43.5
reviewer
#7 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 88 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |  4 +--
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
 		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
 	return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
 	int i = 0;
 
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
 	if (!folio_test_anon(folio))
 		return;
 	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
 		i += folio_nr_pages(folio);
 		if (i >= nr)
 			break;
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 
 	if (nr_dropped)
 		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
 	/*
 	 * set page to its compound_head when split to non order-0 pages, so
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
 		if (mapping)
 			xas_unlock(&xas);
 		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
 		ret = -EAGAIN;
 	}
 
mm/migrate.c +0 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
 	return true;
 }
 

mm/migrate.c bool isolate_folio_to_list(struct folio *folio, struct list_head *list) · line 216

Quoted 10 lines
mm/migrate.c +10 -0
+static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
+					  struct folio *folio,
+					  unsigned long idx)
+{
+	struct page *page = folio_page(folio, idx);
+	bool contains_data;
+	pte_t newpte;
+	void *addr;
+
+	VM_BUG_ON_PAGE(PageCompound(page), page);

This should be:

mm/migrate.c +2 -1
index e950fd62607f..7ffdbe078aa7 100644
@@ -206,7 +206,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
        pte_t newpte;
        void *addr;
 
-       VM_BUG_ON_PAGE(PageCompound(page), page);
+       if (PageCompound(page))
+               return false;
        VM_BUG_ON_PAGE(!PageAnon(page), page);
        VM_BUG_ON_PAGE(!PageLocked(page), page);
        VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);

Otherwise, splitting anonymous large folios to non order-0 ones just triggers this BUG_ON.

Quoted 141 lines
mm/migrate.c +53 -9
+	VM_BUG_ON_PAGE(!PageAnon(page), page);
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+	if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags & VM_LOCKED) ||
+	    mm_forbids_zeropage(pvmw->vma->vm_mm))
+		return false;
+
+	/*
+	 * The pmd entry mapping the old thp was flushed and the pte mapping
+	 * this subpage has been non present. If the subpage is only zero-filled
+	 * then map it to the shared zeropage.
+	 */
+	addr = kmap_local_page(page);
+	contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+	kunmap_local(addr);
+
+	if (contains_data)
+		return false;
+
+	newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+					pvmw->vma->vm_page_prot));
+	set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
+
+	dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+	return true;
+}
+
+struct rmap_walk_arg {
+	struct folio *folio;
+	bool map_unused_to_zeropage;
+};
+
 /*
  * Restore a potential migration pte to a working pte entry
  */
 static bool remove_migration_pte(struct folio *folio,
-		struct vm_area_struct *vma, unsigned long addr, void *old)
+		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
-	DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+	struct rmap_walk_arg *rmap_walk_arg = arg;
+	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		rmap_t rmap_flags = RMAP_NONE;
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio *folio,
 			continue;
 		}
 #endif
+		if (rmap_walk_arg->map_unused_to_zeropage &&
+		    try_to_map_unused_to_zeropage(&pvmw, folio, idx))
+			continue;
 
 		folio_get(folio);
 		pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio *folio,
  * Get rid of all migration entries and replace them by
  * references to the indicated page.
  */
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 {
+	struct rmap_walk_arg rmap_walk_arg = {
+		.folio = src,
+		.map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
+	};
+
 	struct rmap_walk_control rwc = {
 		.rmap_one = remove_migration_pte,
-		.arg = src,
+		.arg = &rmap_walk_arg,
 	};
 
-	if (locked)
+	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
+
+	if (flags & RMP_LOCKED)
 		rmap_walk_locked(dst, &rwc);
 	else
 		rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space *mapping, struct folio *folio)
 	 * At this point we know that the migration attempt cannot
 	 * be successful.
 	 */
-	remove_migration_ptes(folio, folio, false);
+	remove_migration_ptes(folio, folio, 0);
 
 	rc = mapping->a_ops->writepage(&folio->page, &wbc);
 
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio *src,
 				   struct list_head *ret)
 {
 	if (page_was_mapped)
-		remove_migration_ptes(src, src, false);
+		remove_migration_ptes(src, src, 0);
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
 		put_anon_vma(anon_vma);
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		lru_add_drain();
 
 	if (old_page_state & PAGE_WAS_MAPPED)
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 
 out_unlock_both:
 	folio_unlock(dst);
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 
 	if (page_was_mapped)
 		remove_migration_ptes(src,
-			rc == MIGRATEPAGE_SUCCESS ? dst : src, false);
+			rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 
 unlock_put_anon:
 	folio_unlock(dst);
mm/migrate_device.c +2 -2
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 			continue;
 
 		folio = page_folio(page);
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
 		folio_unlock(folio);
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			dst = src;
 		}
 
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 		folio_unlock(src);
 
 		if (folio_is_zone_device(src))
Signature · 1 line
2.43.5

Best Regards,
Yan, Zi

author
#8 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 118 lines · Zi Yan
Quoted 88 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |  4 +--
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
 		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
 	return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
 	int i = 0;
 
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
 	if (!folio_test_anon(folio))
 		return;
 	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
 		i += folio_nr_pages(folio);
 		if (i >= nr)
 			break;
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 
 	if (nr_dropped)
 		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
 	/*
 	 * set page to its compound_head when split to non order-0 pages, so
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
 		if (mapping)
 			xas_unlock(&xas);
 		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
 		ret = -EAGAIN;
 	}
 
mm/migrate.c +0 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
 	return true;
 }
 

mm/migrate.c bool isolate_folio_to_list(struct folio *folio, struct list_head *list) · line 216

Quoted 10 lines
mm/migrate.c +10 -0
+static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
+					  struct folio *folio,
+					  unsigned long idx)
+{
+	struct page *page = folio_page(folio, idx);
+	bool contains_data;
+	pte_t newpte;
+	void *addr;
+
+	VM_BUG_ON_PAGE(PageCompound(page), page);

This should be:

mm/migrate.c +2 -1
index e950fd62607f..7ffdbe078aa7 100644
@@ -206,7 +206,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
        pte_t newpte;
        void *addr;
 
-       VM_BUG_ON_PAGE(PageCompound(page), page);
+       if (PageCompound(page))
+               return false;
        VM_BUG_ON_PAGE(!PageAnon(page), page);
        VM_BUG_ON_PAGE(!PageLocked(page), page);
        VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
Quoted 3 lines

Otherwise, splitting anonymous large folios to non order-0 ones just triggers this BUG_ON.

That makes sense, would you like to send the fix?

Adding Yu Zhao to "To" incase he has any objections.

Thanks,
Usama

Quoted 144 lines
Quoted 141 lines
mm/migrate.c +53 -9
+	VM_BUG_ON_PAGE(!PageAnon(page), page);
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+	if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags & VM_LOCKED) ||
+	    mm_forbids_zeropage(pvmw->vma->vm_mm))
+		return false;
+
+	/*
+	 * The pmd entry mapping the old thp was flushed and the pte mapping
+	 * this subpage has been non present. If the subpage is only zero-filled
+	 * then map it to the shared zeropage.
+	 */
+	addr = kmap_local_page(page);
+	contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+	kunmap_local(addr);
+
+	if (contains_data)
+		return false;
+
+	newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+					pvmw->vma->vm_page_prot));
+	set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
+
+	dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+	return true;
+}
+
+struct rmap_walk_arg {
+	struct folio *folio;
+	bool map_unused_to_zeropage;
+};
+
 /*
  * Restore a potential migration pte to a working pte entry
  */
 static bool remove_migration_pte(struct folio *folio,
-		struct vm_area_struct *vma, unsigned long addr, void *old)
+		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
-	DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+	struct rmap_walk_arg *rmap_walk_arg = arg;
+	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		rmap_t rmap_flags = RMAP_NONE;
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio *folio,
 			continue;
 		}
 #endif
+		if (rmap_walk_arg->map_unused_to_zeropage &&
+		    try_to_map_unused_to_zeropage(&pvmw, folio, idx))
+			continue;
 
 		folio_get(folio);
 		pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio *folio,
  * Get rid of all migration entries and replace them by
  * references to the indicated page.
  */
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 {
+	struct rmap_walk_arg rmap_walk_arg = {
+		.folio = src,
+		.map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
+	};
+
 	struct rmap_walk_control rwc = {
 		.rmap_one = remove_migration_pte,
-		.arg = src,
+		.arg = &rmap_walk_arg,
 	};
 
-	if (locked)
+	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
+
+	if (flags & RMP_LOCKED)
 		rmap_walk_locked(dst, &rwc);
 	else
 		rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space *mapping, struct folio *folio)
 	 * At this point we know that the migration attempt cannot
 	 * be successful.
 	 */
-	remove_migration_ptes(folio, folio, false);
+	remove_migration_ptes(folio, folio, 0);
 
 	rc = mapping->a_ops->writepage(&folio->page, &wbc);
 
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio *src,
 				   struct list_head *ret)
 {
 	if (page_was_mapped)
-		remove_migration_ptes(src, src, false);
+		remove_migration_ptes(src, src, 0);
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
 		put_anon_vma(anon_vma);
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 		lru_add_drain();
 
 	if (old_page_state & PAGE_WAS_MAPPED)
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 
 out_unlock_both:
 	folio_unlock(dst);
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 
 	if (page_was_mapped)
 		remove_migration_ptes(src,
-			rc == MIGRATEPAGE_SUCCESS ? dst : src, false);
+			rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 
 unlock_put_anon:
 	folio_unlock(dst);
mm/migrate_device.c +2 -2
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 			continue;
 
 		folio = page_folio(page);
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
 		folio_unlock(folio);
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			dst = src;
 		}
 
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 		folio_unlock(src);
 
 		if (folio_is_zone_device(src))
Signature · 1 line
2.43.5

Best Regards,
Yan, Zi

reviewer
#9 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 119 lines · Usama Arif
Quoted 118 lines · Zi Yan
Quoted 88 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |  4 +--
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
 		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
 	return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
 	int i = 0;
 
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
 	if (!folio_test_anon(folio))
 		return;
 	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
 		i += folio_nr_pages(folio);
 		if (i >= nr)
 			break;
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 
 	if (nr_dropped)
 		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
 	/*
 	 * set page to its compound_head when split to non order-0 pages, so
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
 		if (mapping)
 			xas_unlock(&xas);
 		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
 		ret = -EAGAIN;
 	}
 
mm/migrate.c +0 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
 	return true;
 }
 

mm/migrate.c bool isolate_folio_to_list(struct folio *folio, struct list_head *list) · line 216

Quoted 10 lines
mm/migrate.c +10 -0
+static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
+					  struct folio *folio,
+					  unsigned long idx)
+{
+	struct page *page = folio_page(folio, idx);
+	bool contains_data;
+	pte_t newpte;
+	void *addr;
+
+	VM_BUG_ON_PAGE(PageCompound(page), page);

This should be:

mm/migrate.c +2 -1
index e950fd62607f..7ffdbe078aa7 100644
@@ -206,7 +206,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
        pte_t newpte;
        void *addr;
 
-       VM_BUG_ON_PAGE(PageCompound(page), page);
+       if (PageCompound(page))
+               return false;
        VM_BUG_ON_PAGE(!PageAnon(page), page);
        VM_BUG_ON_PAGE(!PageLocked(page), page);
        VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
Quoted 8 lines
Quoted 3 lines

Otherwise, splitting anonymous large folios to non order-0 ones just triggers this BUG_ON.

That makes sense, would you like to send the fix?

Adding Yu Zhao to "To" incase he has any objections.

Sure, will do.

Best Regards,
Yan, Zi

#10 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to Usama Arif
Quoted 120 lines · Usama Arif
Quoted 118 lines · Zi Yan
Quoted 88 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will be introduced in later patches, where underutilized THPs are split, and the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++------
 mm/migrate_device.c  |  4 +--
 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
 int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
                   struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
+enum rmp_flags {
+    RMP_LOCKED              = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
 
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +4 -4
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
     return false;
 }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int flags)
 {
     int i = 0;
 
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
     if (!folio_test_anon(folio))
             return;
     for (;;) {
-            remove_migration_ptes(folio, folio, true);
+            remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
             i += folio_nr_pages(folio);
             if (i >= nr)
                     break;
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 
     if (nr_dropped)
             shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ? RMP_USE_SHARED_ZEROPAGE : 0);
 
     /*
      * set page to its compound_head when split to non order-0 pages, so
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
             if (mapping)
                     xas_unlock(&xas);
             local_irq_enable();
-            remap_page(folio, folio_nr_pages(folio));
+            remap_page(folio, folio_nr_pages(folio), 0);
             ret = -EAGAIN;
     }
 
mm/migrate.c +0 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
     return true;
 }
 

mm/migrate.c bool isolate_folio_to_list(struct folio *folio, struct list_head *list) · line 216

Quoted 10 lines
mm/migrate.c +10 -0
+static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
+                                      struct folio *folio,
+                                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);

This should be:

mm/migrate.c +2 -1
index e950fd62607f..7ffdbe078aa7 100644
@@ -206,7 +206,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
        pte_t newpte;
        void *addr;
 
-       VM_BUG_ON_PAGE(PageCompound(page), page);
+       if (PageCompound(page))
+               return false;
        VM_BUG_ON_PAGE(!PageAnon(page), page);
        VM_BUG_ON_PAGE(!PageLocked(page), page);
        VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
Quoted 7 lines
Quoted 3 lines

Otherwise, splitting anonymous large folios to non order-0 ones just triggers this BUG_ON.

That makes sense, would you like to send the fix?

Adding Yu Zhao to "To" incase he has any objections.

LGTM. Thanks!

#11 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to Usama Arif
Quoted 273 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
 include/linux/rmap.h |  7 ++++-
 mm/huge_memory.c     |  8 ++---
 mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++---- --
 mm/migrate_device.c  |  4 +--

 4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +0 -0
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
  int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
  		      struct vm_area_struct *vma);


-void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);

include/linux/rmap.h +6 -0
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int

flags);

 /*
  * rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +0 -0
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
  	return false;
  }

mm/huge_memory.c +1 -1
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int

flags)
 {
  int i = 0;

mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
  	if (!folio_test_anon(folio))
  		return;
  	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
  		i += folio_nr_pages(folio);

  if (i >= nr)
  break;

mm/huge_memory.c +0 -0
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,


  if (nr_dropped) shmem_uncharge(folio->mapping->host, nr_dropped);

mm/huge_memory.c +1 -1
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ?

RMP_USE_SHARED_ZEROPAGE : 0);

  /*
  * set page to its compound_head when split to non order-0 pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
  		if (mapping)
  			xas_unlock(&xas);
  		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
  		ret = -EAGAIN;
  	}

mm/migrate.c +0 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
  	return true;
  }


+static bool try_to_map_unused_to_zeropage(struct
page_vma_mapped_walk *pvmw,

mm/migrate.c +14 -0
+					  struct folio *folio,
+					  unsigned long idx)
+{
+	struct page *page = folio_page(folio, idx);
+	bool contains_data;
+	pte_t newpte;
+	void *addr;
+
+	VM_BUG_ON_PAGE(PageCompound(page), page);
+	VM_BUG_ON_PAGE(!PageAnon(page), page);
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+	if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||

mm/migrate.c +5 -0
+	    mm_forbids_zeropage(pvmw->vma->vm_mm))
+		return false;
+
+	/*
+	 * The pmd entry mapping the old thp was flushed and the pte

mapping

mm/migrate.c +1 -0
+	 * this subpage has been non present. If the subpage is only

zero-filled

mm/migrate.c +12 -0
+	 * then map it to the shared zeropage.
+	 */
+	addr = kmap_local_page(page);
+	contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+	kunmap_local(addr);
+
+	if (contains_data)
+		return false;
+
+	newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+					pvmw->vma->vm_page_prot));
+	set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -0
+
+	dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+	return true;
+}
+
+struct rmap_walk_arg {
+	struct folio *folio;
+	bool map_unused_to_zeropage;
+};
+

 /*
  * Restore a potential migration pte to a working pte entry */
 static bool remove_migration_pte(struct folio *folio,

mm/migrate.c +0 -1
-		struct vm_area_struct *vma, unsigned long addr, void

*old)

mm/migrate.c +1 -0
+		struct vm_area_struct *vma, unsigned long addr, void

*arg)
 {

mm/migrate.c +0 -1
-	DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC |

PVMW_MIGRATION);

mm/migrate.c +2 -0
+	struct rmap_walk_arg *rmap_walk_arg = arg;
+	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr,

PVMW_SYNC | PVMW_MIGRATION);

  while (page_vma_mapped_walk(&pvmw)) {
  rmap_t rmap_flags = RMAP_NONE;

mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
  			continue;
  		}
  #endif
+		if (rmap_walk_arg->map_unused_to_zeropage &&
+		    try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+			continue;


  folio_get(folio);
  pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));

mm/migrate.c +7 -2
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
   * Get rid of all migration entries and replace them by
   * references to the indicated page.
   */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
  {
+	struct rmap_walk_arg rmap_walk_arg = {
+		.folio = src,
+		.map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+	};
+
  	struct rmap_walk_control rwc = {
  		.rmap_one = remove_migration_pte,
-		.arg = src,
+		.arg = &rmap_walk_arg,
  	};

mm/migrate.c +1 -1
-	if (locked)
+	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +2 -0
+
+	if (flags & RMP_LOCKED)

  rmap_walk_locked(dst, &rwc);
  else
  rmap_walk(dst, &rwc);

mm/migrate.c +1 -1
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
  	 * At this point we know that the migration attempt cannot
  	 * be successful.
  	 */
-	remove_migration_ptes(folio, folio, false);
+	remove_migration_ptes(folio, folio, 0);


  rc = mapping->a_ops->writepage(&folio->page, &wbc);

mm/migrate.c +1 -1
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
  				   struct list_head *ret)
  {
  	if (page_was_mapped)
-		remove_migration_ptes(src, src, false);
+		remove_migration_ptes(src, src, 0);
  	/* Drop an anon_vma reference if we took one */
  	if (anon_vma)

  put_anon_vma(anon_vma);

mm/migrate.c +0 -0
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
  		lru_add_drain();


  if (old_page_state & PAGE_WAS_MAPPED)

mm/migrate.c +1 -1
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);


 out_unlock_both:
  folio_unlock(dst);

mm/migrate.c +0 -0
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,


  if (page_was_mapped)
  remove_migration_ptes(src,

mm/migrate.c +0 -1
-			rc == MIGRATEPAGE_SUCCESS ? dst : src,

false);

mm/migrate.c +1 -0
+			rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);


 unlock_put_anon:
  folio_unlock(dst);

mm/migrate_device.c +0 -0
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
  			continue;


  folio = page_folio(page);

mm/migrate_device.c +1 -1
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);


  src_pfns[i] = 0;
  folio_unlock(folio);

mm/migrate_device.c +0 -0
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
  			dst = src;
  		}

mm/migrate_device.c +1 -1
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
Quoted 3 lines

  folio_unlock(src);

  if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

In Android, the tombstone is as follows:

---
Build fingerprint:
'alps/vext_k6993v1_64/k6993v1_64:16/BP2A.250605.031.A3/mp1cs1ofp41:user debug/dev-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2025-08-12 04:58:28.507086720+0800
Process uptime: 0s
Cmdline: /system/bin/audioserver
pid: 8217, tid: 8882, name: binder:8217_4 >>> /system/bin/audioserver <<<
uid: 1041
tagged_addr_ctrl: 000000000007fff3 (PR_TAGGED_ADDR_ENABLE, PR_MTE_TCF_SYNC, mask 0xfffe)
signal 11 (SIGSEGV), code 9 (SEGV_MTESERR), fault addr 0x0a00007055220000
Cause: [MTE]: Buffer Overflow, 14016 bytes into a 23070-byte allocation at 0x705521c940

    x0  0a0000705521c940  x1  0300006f75210ab0  x2  00000000000022a5 
x3  0a0000705521ffc0
    x4  0300006f75212de5  x5  0a000070552222f5  x6  0000000000005a1e 
x7  0000000000000000
    x8  339c000005a1e11e  x9  00000000f041339c  x10 000000009dd48904 
x11 000000000000ffff
    x12 0000000022b70889  x13 000000004cc0b2ff  x14 0000000000000000 
x15 0000000000000010
    x16 00000071cc5d8fc0  x17 00000071cc54e040  x18 0000006ef7bd4000 
x19 0300006f7520d430
    x20 00000071cc5e0340  x21 0000000000005a1e  x22 0a0000705521c940 
x23 00000000000059b5
    x24 00000000000000b1  x25 0300006f75212e4e  x26 caa20000059b511d 
x27 000000000000001d
    x28 0300006f75212e30  x29 0000006f1fe385f0
    lr  00000071cc54200c  sp  0000006f1fe385c0  pc  00000071cc54e158 

pst 0000000020001000

26 total frames
backtrace:

      #00 pc 000000000006c158 
/apex/com.android.runtime/lib64/bionic/libc.so
(__memcpy_aarch64_simd+280) (BuildId: 1e819f3e369d59be98bee38a8fbd0322)
      #01 pc 0000000000060008 

/apex/com.android.runtime/lib64/bionic/libc.so (scudo::Allocator<scudo::AndroidNormalConfig, &scudo_malloc_postinit>::reallocate(void*, unsigned long, unsigned long)+696) (BuildId: 1e819f3e369d59be98bee38a8fbd0322)

      #02 pc 000000000005fccc 
/apex/com.android.runtime/lib64/bionic/libc.so (scudo_realloc+44)
(BuildId: 1e819f3e369d59be98bee38a8fbd0322)
      #03 pc 000000000005c2cc 
/apex/com.android.runtime/lib64/bionic/libc.so (LimitRealloc(void*,
unsigned long)+124) (BuildId: 1e819f3e369d59be98bee38a8fbd0322)
      #04 pc 0000000000059a90 
/apex/com.android.runtime/lib64/bionic/libc.so (realloc+160) (BuildId:
1e819f3e369d59be98bee38a8fbd0322)
      #05 pc 0000000000011a74  /system/lib64/libutils.so
(android::SharedBuffer::editResize(unsigned long) const+68) (BuildId:
7aa2d71e030a290c8dd28236ba0a838f)
      #06 pc 0000000000011ba8  /system/lib64/libutils.so
(android::String8::real_append(char const*, unsigned long)+88)
(BuildId: 7aa2d71e030a290c8dd28236ba0a838f)
      #07 pc 000000000007b880 
/system/lib64/libaudiopolicycomponents.so
(android::DeviceDescriptor::dump(android::String8*, int, bool)
const+208) (BuildId: 553fefffdca2f3a5dde634e123bd2c81)
      #08 pc 000000000008094c 

/system/lib64/libaudiopolicycomponents.so (android::DeviceVector::dump(android::String8*, android::String8 const&, int, bool) const+636) (BuildId:
553fefffdca2f3a5dde634e123bd2c81)

      #09 pc 0000000000092ed4 
/system/lib64/libaudiopolicycomponents.so
(android::IOProfile::dump(android::String8*, int) const+980) (BuildId:
553fefffdca2f3a5dde634e123bd2c81)
      #10 pc 000000000008bd7c 
/system/lib64/libaudiopolicycomponents.so
(android::HwModule::dump(android::String8*, int) const+1148) (BuildId:
553fefffdca2f3a5dde634e123bd2c81)
      #11 pc 000000000009044c 
/system/lib64/libaudiopolicycomponents.so
(android::HwModuleCollection::dump(android::String8*) const+508)
(BuildId: 553fefffdca2f3a5dde634e123bd2c81)
      #12 pc 0000000000090134 
/system/lib64/libaudiopolicymanagerdefault.so
(android::AudioPolicyManager::dump(android::String8*) const+3908)
(BuildId: fdba879fc1a0c470759bfeb3d594ab81)
      #13 pc 0000000000092e40 
/system/lib64/libaudiopolicymanagerdefault.so
(android::AudioPolicyManager::dump(int)+80) (BuildId:
fdba879fc1a0c470759bfeb3d594ab81)
      #14 pc 000000000022b218  /system/bin/audioserver
(android::AudioPolicyService::dump(int,
android::Vector<android::String16> const&)+392) (BuildId:
1988c27ce74b125f598a07a93367cfdd)
      #15 pc 000000000022c8cc  /system/bin/audioserver (non-virtual
thunk to android::AudioPolicyService::dump(int,
android::Vector<android::String16> const&)+12) (BuildId:
1988c27ce74b125f598a07a93367cfdd)
      #16 pc 00000000000883f4  /system/lib64/libbinder.so
(android::BBinder::onTransact(unsigned int, android::Parcel const&,
android::Parcel*, unsigned int)+340) (BuildId:
4ace0dcb0135b71ba70b7aaee457d26f)
      #17 pc 000000000003fadc  /system/lib64/audiopolicy-aidl-cpp.so
(android::media::BnAudioPolicyService::onTransact(unsigned int,
android::Parcel const&, android::Parcel*, unsigned int)+19884)
(BuildId: ae185d80e4e54668275f262317dc2d7d)
      #18 pc 000000000022adc4  /system/bin/audioserver
(android::AudioPolicyService::onTransact(unsigned int, android::Parcel
const&, android::Parcel*, unsigned int)+1076) (BuildId:
1988c27ce74b125f598a07a93367cfdd)
      #19 pc 0000000000048adc  /system/lib64/libbinder.so
(android::IPCThreadState::executeCommand(int)+748) (BuildId:
4ace0dcb0135b71ba70b7aaee457d26f)
      #20 pc 0000000000051788  /system/lib64/libbinder.so
(android::IPCThreadState::joinThreadPool(bool)+296) (BuildId:
4ace0dcb0135b71ba70b7aaee457d26f)
      #21 pc 000000000007e528  /system/lib64/libbinder.so
(android::PoolThread::threadLoop()+24) (BuildId:
4ace0dcb0135b71ba70b7aaee457d26f)
      #22 pc 0000000000019268  /system/lib64/libutils.so
(android::Thread::_threadLoop(void*)+248) (BuildId:
7aa2d71e030a290c8dd28236ba0a838f)
      #23 pc 000000000001b994  /system/lib64/libutils.so
(libutil_thread_trampoline(void*)
(.__uniq.226528677032898775202282855395389835431)+20) (BuildId:
7aa2d71e030a290c8dd28236ba0a838f)
      #24 pc 0000000000083c8c 
/apex/com.android.runtime/lib64/bionic/libc.so
(__pthread_start(void*)+236) (BuildId:
1e819f3e369d59be98bee38a8fbd0322)
      #25 pc 00000000000761a0 

/apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 1e819f3e369d59be98bee38a8fbd0322)

Memory tags around the fault address (0xa00007055220000), one tag per 16 bytes:

      0x705521f800: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521f900: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521fa00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521fb00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521fc00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521fd00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521fe00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
      0x705521ff00: a  a  a  a  a  a  a  a  a  a  a  a  a  a  a  a
    =>0x7055220000:[0] 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220100: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220200: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220300: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220400: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220500: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220600: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0x7055220700: 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

---

Whenever the memory pressure is high, it will happen to any process with MTE enabled.

Any suggestion is appreciated.

Thanks,
Qun-wei

#12 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 280 lines · Qun-wei Lin (林群崴)
Quoted 275 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
  include/linux/rmap.h |  7 ++++-
  mm/huge_memory.c     |  8 ++---
  mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----
--
  mm/migrate_device.c  |  4 +--
  4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
  int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
  		      struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+	RMP_LOCKED		= 1 << 0,
+	RMP_USE_SHARED_ZEROPAGE	= 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

 /*
  * rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -1
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
  	return false;
  }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
  {

  int i = 0;

mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
  	if (!folio_test_anon(folio))
  		return;
  	for (;;) {
-		remove_migration_ptes(folio, folio, true);
+		remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
  		i += folio_nr_pages(folio);

  if (i >= nr)
  break;

mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
 
  	if (nr_dropped)
  		shmem_uncharge(folio->mapping->host, nr_dropped);
-	remap_page(folio, nr);
+	remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
 

  /*
  * set page to its compound_head when split to non order-0
pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
  		if (mapping)
  			xas_unlock(&xas);
  		local_irq_enable();
-		remap_page(folio, folio_nr_pages(folio));
+		remap_page(folio, folio_nr_pages(folio), 0);
  		ret = -EAGAIN;
  	}
mm/migrate.c +15 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
  	return true;
  }
 
+static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+					  struct folio *folio,
+					  unsigned long idx)
+{
+	struct page *page = folio_page(folio, idx);
+	bool contains_data;
+	pte_t newpte;
+	void *addr;
+
+	VM_BUG_ON_PAGE(PageCompound(page), page);
+	VM_BUG_ON_PAGE(!PageAnon(page), page);
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+	if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||

mm/migrate.c +5 -0
+	    mm_forbids_zeropage(pvmw->vma->vm_mm))
+		return false;
+
+	/*
+	 * The pmd entry mapping the old thp was flushed and the pte

mapping

mm/migrate.c +1 -0
+	 * this subpage has been non present. If the subpage is only

zero-filled

mm/migrate.c +12 -0
+	 * then map it to the shared zeropage.
+	 */
+	addr = kmap_local_page(page);
+	contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+	kunmap_local(addr);
+
+	if (contains_data)
+		return false;
+
+	newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+					pvmw->vma->vm_page_prot));
+	set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -1
+
+	dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+	return true;
+}
+
+struct rmap_walk_arg {
+	struct folio *folio;
+	bool map_unused_to_zeropage;
+};
+
  /*
   * Restore a potential migration pte to a working pte entry
   */
  static bool remove_migration_pte(struct folio *folio,
-		struct vm_area_struct *vma, unsigned long addr, void

*old)

mm/migrate.c +1 -0
+		struct vm_area_struct *vma, unsigned long addr, void

*arg)
 {

mm/migrate.c +0 -1
-	DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC |

PVMW_MIGRATION);

mm/migrate.c +2 -0
+	struct rmap_walk_arg *rmap_walk_arg = arg;
+	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr,

PVMW_SYNC | PVMW_MIGRATION);

  while (page_vma_mapped_walk(&pvmw)) {
  rmap_t rmap_flags = RMAP_NONE;

mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
  			continue;
  		}
  #endif
+		if (rmap_walk_arg->map_unused_to_zeropage &&
+		    try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+			continue;
 

  folio_get(folio);
  pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));

mm/migrate.c +8 -3
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
   * Get rid of all migration entries and replace them by
   * references to the indicated page.
   */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
  {
+	struct rmap_walk_arg rmap_walk_arg = {
+		.folio = src,
+		.map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+	};
+
  	struct rmap_walk_control rwc = {
  		.rmap_one = remove_migration_pte,
-		.arg = src,
+		.arg = &rmap_walk_arg,
  	};
 
-	if (locked)
+	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +4 -2
+
+	if (flags & RMP_LOCKED)
  		rmap_walk_locked(dst, &rwc);
  	else
  		rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
  	 * At this point we know that the migration attempt cannot
  	 * be successful.
  	 */
-	remove_migration_ptes(folio, folio, false);
+	remove_migration_ptes(folio, folio, 0);
 
  	rc = mapping->a_ops->writepage(&folio->page, &wbc);
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
  				   struct list_head *ret)
  {
  	if (page_was_mapped)
-		remove_migration_ptes(src, src, false);
+		remove_migration_ptes(src, src, 0);
  	/* Drop an anon_vma reference if we took one */
  	if (anon_vma)

  put_anon_vma(anon_vma);

mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
  		lru_add_drain();
 
  	if (old_page_state & PAGE_WAS_MAPPED)
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
 
  out_unlock_both:

  folio_unlock(dst);

mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
 
  	if (page_was_mapped)
  		remove_migration_ptes(src,
-			rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+			rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 

 unlock_put_anon:
  folio_unlock(dst);

mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
  			continue;
 
  		folio = page_folio(page);
-		remove_migration_ptes(folio, folio, false);
+		remove_migration_ptes(folio, folio, 0);
 
  		src_pfns[i] = 0;

  folio_unlock(folio);

mm/migrate_device.c +1 -1
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
  			dst = src;
  		}
 
-		remove_migration_ptes(src, dst, false);
+		remove_migration_ptes(src, dst, 0);
  		folio_unlock(src);
 
Quoted 1 line

  if (folio_is_zone_device(src))

Hi,

Quoted 6 lines

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

Also, I wonder how KSM and the shared zeropage works in general with that, because I would expect similar issues when we de-duplicate memory?

Signature · 3 lines
Cheers

David / dhildenb
author
#13 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 260 lines · David Hildenbrand
Quoted 259 lines · Qun-wei Lin (林群崴)
Quoted 248 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
  include/linux/rmap.h |  7 ++++-
  mm/huge_memory.c     |  8 ++---
  mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----

Signature · 2 lines
  mm/migrate_device.c  |  4 +--
  4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -0
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
   int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                 struct vm_area_struct *vma);
   -void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED        = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE    = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

    /*
   * rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -0
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
       return false;
   }
   -static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
   {

      int i = 0;
  @@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio, unsigned long nr)
      if (!folio_test_anon(folio))
          return;
      for (;;) {
-        remove_migration_ptes(folio, folio, true);
+        remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
          i += folio_nr_pages(folio);
          if (i >= nr)
              break;

mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
         if (nr_dropped)
           shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
         /*
        * set page to its compound_head when split to non order-0

pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
           if (mapping)
               xas_unlock(&xas);
           local_irq_enable();
-        remap_page(folio, folio_nr_pages(folio));
+        remap_page(folio, folio_nr_pages(folio), 0);
           ret = -EAGAIN;
       }

  diff --git a/mm/migrate.c b/mm/migrate.c
index 6f9c62c746be..d039863e014b 100644

mm/migrate.c +0 -0
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
       return true;
   }

  +static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
+                      struct folio *folio, +                      unsigned long idx)

mm/migrate.c +12 -0
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+        return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte mapping
+     * this subpage has been non present. If the subpage is only zero-filled
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);

mm/migrate.c +7 -0
+
+    if (contains_data)
+        return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -0
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+

  /*
   * Restore a potential migration pte to a working pte entry */
  static bool remove_migration_pte(struct folio *folio, -        struct vm_area_struct *vma, unsigned long addr, void *old)
+        struct vm_area_struct *vma, unsigned long addr, void *arg)
  {
-    DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
        while (page_vma_mapped_walk(&pvmw)) {
          rmap_t rmap_flags = RMAP_NONE;

mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
               continue;
           }
   #endif
+        if (rmap_walk_arg->map_unused_to_zeropage &&
+            try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+            continue;
             folio_get(folio);

          pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));

mm/migrate.c +8 -2
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
    * Get rid of all migration entries and replace them by
    * references to the indicated page.
    */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
   {
+    struct rmap_walk_arg rmap_walk_arg = {
+        .folio = src,
+        .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
       struct rmap_walk_control rwc = {
           .rmap_one = remove_migration_pte,
-        .arg = src,
+        .arg = &rmap_walk_arg,
       };
   -    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);
+
+    if (flags & RMP_LOCKED)
          rmap_walk_locked(dst, &rwc);
      else
          rmap_walk(dst, &rwc);

mm/migrate.c +1 -1
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
        * At this point we know that the migration attempt cannot
        * be successful.
        */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
         rc = mapping->a_ops->writepage(&folio->page, &wbc);

  @@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio *src,
                     struct list_head *ret)
  {
      if (page_was_mapped)
-        remove_migration_ptes(src, src, false);
+        remove_migration_ptes(src, src, 0);
      /* Drop an anon_vma reference if we took one */
      if (anon_vma)
          put_anon_vma(anon_vma);

mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
           lru_add_drain();
         if (old_page_state & PAGE_WAS_MAPPED)
-        remove_migration_ptes(src, dst, false);
+        remove_migration_ptes(src, dst, 0);

    out_unlock_both:
      folio_unlock(dst);

mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
         if (page_was_mapped)
           remove_migration_ptes(src,
-            rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+            rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);

    unlock_put_anon:
      folio_unlock(dst);

mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
               continue;
             folio = page_folio(page);
-        remove_migration_ptes(folio, folio, false);
+        remove_migration_ptes(folio, folio, 0);
             src_pfns[i] = 0;
           folio_unlock(folio);

mm/migrate_device.c void migrate_device_finalize(unsigned long · line 846

Quoted 7 lines
mm/migrate_device.c +1 -0
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
               dst = src;
           }
   -        remove_migration_ptes(src, dst, false);
+        remove_migration_ptes(src, dst, 0);
           folio_unlock(src);
             if (folio_is_zone_device(src))

Hi,

Quoted 9 lines
Quoted 6 lines

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

I believe MTE is all or nothing? i.e. all the memory is tagged when enabled, but will let the arm folks confirm.

Yeah unfortunately I think that might be the only way. We cant change the pointers and I dont think there is a way to mark the memory as "untagged". If we cant remap to zeropage, then there is no point of shrinker. I am guessing instead of checking at runtime whether mte is enabled when remapping to shared zeropage, we need to ifndef the shrinker if CONFIG_ARM64_MTE is enabled?

Quoted 2 lines

Also, I wonder how KSM and the shared zeropage works in general with that, because I would expect similar issues when we de-duplicate memory?

Yeah thats a very good point!

Also the initial report mentioned mTHP instead of THP, but I dont think that matters.

#14 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 267 lines · Usama Arif
Quoted 264 lines · David Hildenbrand
Quoted 263 lines · Qun-wei Lin (林群崴)
Quoted 248 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
   include/linux/rmap.h |  7 ++++-
   mm/huge_memory.c     |  8 ++---
   mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----
Signature · 2 lines
   mm/migrate_device.c  |  4 +--
   4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -0
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
   int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                 struct vm_area_struct *vma);
   -void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED        = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE    = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);
     /*
    * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +1 -0
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
       return false;
   }
   -static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
   {
       int i = 0;
   @@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,

unsigned long nr)
      if (!folio_test_anon(folio))
          return;
      for (;;) {
-        remove_migration_ptes(folio, folio, true); +        remove_migration_ptes(folio, folio, RMP_LOCKED | flags);
          i += folio_nr_pages(folio);
          if (i >= nr)
              break;

mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
         if (nr_dropped)
           shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
         /*
        * set page to its compound_head when split to non order-0

pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
           if (mapping)
               xas_unlock(&xas);
           local_irq_enable();
-        remap_page(folio, folio_nr_pages(folio));
+        remap_page(folio, folio_nr_pages(folio), 0);
           ret = -EAGAIN;
       }

  diff --git a/mm/migrate.c b/mm/migrate.c
index 6f9c62c746be..d039863e014b 100644

mm/migrate.c +14 -0
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
       return true;
   }
   +static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+                      struct folio *folio,
+                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+        return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte mapping
+     * this subpage has been non present. If the subpage is only zero-filled
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);

mm/migrate.c +7 -0
+
+    if (contains_data)
+        return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -1
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+
   /*
    * Restore a potential migration pte to a working pte entry
    */
   static bool remove_migration_pte(struct folio *folio,
-        struct vm_area_struct *vma, unsigned long addr, void

*old)
+        struct vm_area_struct *vma, unsigned long addr, void *arg)
  {
-    DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
        while (page_vma_mapped_walk(&pvmw)) {
          rmap_t rmap_flags = RMAP_NONE;

mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
               continue;
           }
   #endif
+        if (rmap_walk_arg->map_unused_to_zeropage &&
+            try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+            continue;
             folio_get(folio);

          pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));

mm/migrate.c +8 -2
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
    * Get rid of all migration entries and replace them by
    * references to the indicated page.
    */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
   {
+    struct rmap_walk_arg rmap_walk_arg = {
+        .folio = src,
+        .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
       struct rmap_walk_control rwc = {
           .rmap_one = remove_migration_pte,
-        .arg = src,
+        .arg = &rmap_walk_arg,
       };
   -    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +3 -1
+
+    if (flags & RMP_LOCKED)
           rmap_walk_locked(dst, &rwc);
       else
           rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
        * At this point we know that the migration attempt cannot
        * be successful.
        */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
         rc = mapping->a_ops->writepage(&folio->page, &wbc);
   @@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio

*src,
                     struct list_head *ret) {
      if (page_was_mapped)
-        remove_migration_ptes(src, src, false); +        remove_migration_ptes(src, src, 0);
      /* Drop an anon_vma reference if we took one */ if (anon_vma)
          put_anon_vma(anon_vma);

mm/migrate.c +2 -2
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
           lru_add_drain();
         if (old_page_state & PAGE_WAS_MAPPED)
-        remove_migration_ptes(src, dst, false);
+        remove_migration_ptes(src, dst, 0);
     out_unlock_both:
       folio_unlock(dst);
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
         if (page_was_mapped)
           remove_migration_ptes(src,
-            rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+            rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
     unlock_put_anon:
       folio_unlock(dst);
mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
               continue;
             folio = page_folio(page);
-        remove_migration_ptes(folio, folio, false);
+        remove_migration_ptes(folio, folio, 0);
             src_pfns[i] = 0;
           folio_unlock(folio);

mm/migrate_device.c void migrate_device_finalize(unsigned long · line 846

Quoted 7 lines
mm/migrate_device.c +1 -0
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
               dst = src;
           }
   -        remove_migration_ptes(src, dst, false);
+        remove_migration_ptes(src, dst, 0);
           folio_unlock(src);
             if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

I believe MTE is all or nothing? i.e. all the memory is tagged when enabled, but will let the arm folks confirm.

I think for a process it's only active when PROT_MTE was called.

In khugepaged, I recall that we use copy_mc_user_highpage()->copy_user_highpage() that translates on arm64 to its custom copy_highpage() where tags are copied as well.

So that's why khugepaged works.

KSM? Hmmm. Not sure how that is fenced off. We'd likely need some hidden page_mte_tagged() check somewhere.

Signature · 3 lines
Cheers

David / dhildenb
reviewer
#15 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to David Hildenbrand
Quoted 286 lines · David Hildenbrand
Quoted 284 lines · Qun-wei Lin (林群崴)
Quoted 275 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
  include/linux/rmap.h |  7 ++++-
  mm/huge_memory.c     |  8 ++---
  mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----
--
  mm/migrate_device.c  |  4 +--
  4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
  int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                   struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED              = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

/*
* rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -1
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
     return false;
  }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
  {
     int i = 0;
mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
     if (!folio_test_anon(folio))
             return;
     for (;;) {
-            remove_migration_ptes(folio, folio, true);
+            remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
             i += folio_nr_pages(folio);
             if (i >= nr)
                     break;
mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
 
     if (nr_dropped)
             shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
 

/*
* set page to its compound_head when split to non order-0
pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
             if (mapping)
                     xas_unlock(&xas);
             local_irq_enable();
-            remap_page(folio, folio_nr_pages(folio));
+            remap_page(folio, folio_nr_pages(folio), 0);
             ret = -EAGAIN;
     }
mm/migrate.c +15 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
     return true;
  }
 
+static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+                                      struct folio *folio,
+                                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||

mm/migrate.c +5 -0
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+            return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte

mapping
+ * this subpage has been non present. If the subpage is only
zero-filled

mm/migrate.c +12 -0
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);
+
+    if (contains_data)
+            return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -1
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+
  /*
   * Restore a potential migration pte to a working pte entry
   */
  static bool remove_migration_pte(struct folio *folio,
-            struct vm_area_struct *vma, unsigned long addr, void

*old)
+ struct vm_area_struct *vma, unsigned long addr, void
*arg)
{
- DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC |
PVMW_MIGRATION);

mm/migrate.c +2 -0
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr,

PVMW_SYNC | PVMW_MIGRATION);

     while (page_vma_mapped_walk(&pvmw)) {
             rmap_t rmap_flags = RMAP_NONE;
mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
                     continue;
             }
  #endif
+            if (rmap_walk_arg->map_unused_to_zeropage &&
+                try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+                    continue;
 
             folio_get(folio);
             pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
mm/migrate.c +8 -3
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
   * Get rid of all migration entries and replace them by
   * references to the indicated page.
   */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
  {
+    struct rmap_walk_arg rmap_walk_arg = {
+            .folio = src,
+            .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
     struct rmap_walk_control rwc = {
             .rmap_one = remove_migration_pte,
-            .arg = src,
+            .arg = &rmap_walk_arg,
     };
 
-    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +4 -2
+
+    if (flags & RMP_LOCKED)
             rmap_walk_locked(dst, &rwc);
     else
             rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
      * At this point we know that the migration attempt cannot
      * be successful.
      */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
 
     rc = mapping->a_ops->writepage(&folio->page, &wbc);
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
                                struct list_head *ret)
  {
     if (page_was_mapped)
-            remove_migration_ptes(src, src, false);
+            remove_migration_ptes(src, src, 0);
     /* Drop an anon_vma reference if we took one */
     if (anon_vma)
             put_anon_vma(anon_vma);
mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
             lru_add_drain();
 
     if (old_page_state & PAGE_WAS_MAPPED)
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
 
  out_unlock_both:
     folio_unlock(dst);
mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
 
     if (page_was_mapped)
             remove_migration_ptes(src,
-                    rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+                    rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 
  unlock_put_anon:
     folio_unlock(dst);
mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
                     continue;
 
             folio = page_folio(page);
-            remove_migration_ptes(folio, folio, false);
+            remove_migration_ptes(folio, folio, 0);
 
             src_pfns[i] = 0;
             folio_unlock(folio);
mm/migrate_device.c +1 -1
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
                     dst = src;
             }
 
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
             folio_unlock(src);
 
Quoted 1 line

if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

Quoted 5 lines
Quoted 2 lines

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

It seems RISC-V also has a similar feature (RISCV_ISA_SUPM) that uses the same prctl(PR_{GET,SET}_TAGGED_ADDR_CTRL) API.

config RISCV_ISA_SUPM
bool "Supm extension for userspace pointer masking" depends on 64BIT
default y
help
Add support for pointer masking in userspace (Supm) when the underlying hardware extension (Smnpm or Ssnpm) is detected at boot.

If this option is disabled, userspace will be unable to use the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.

I wonder if we should disable the THP shrinker for such architectures that define PR_SET_TAGGED_ADDR_CTRL (or PR_GET_TAGGED_ADDR_CTRL).

Cheers,
Lance

Quoted 10 lines

Also, I wonder how KSM and the shared zeropage works in general with that, because I would expect similar issues when we de-duplicate memory?

--
Cheers

David / dhildenb

reviewer
#16 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 305 lines · Lance Yang
Quoted 286 lines · David Hildenbrand
Quoted 284 lines · Qun-wei Lin (林群崴)
Quoted 275 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
  include/linux/rmap.h |  7 ++++-
  mm/huge_memory.c     |  8 ++---
  mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----
--
  mm/migrate_device.c  |  4 +--
  4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
  int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                   struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED              = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

/*
* rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -1
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
     return false;
  }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
  {
     int i = 0;
mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
     if (!folio_test_anon(folio))
             return;
     for (;;) {
-            remove_migration_ptes(folio, folio, true);
+            remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
             i += folio_nr_pages(folio);
             if (i >= nr)
                     break;
mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
 
     if (nr_dropped)
             shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
 

/*
* set page to its compound_head when split to non order-0
pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
             if (mapping)
                     xas_unlock(&xas);
             local_irq_enable();
-            remap_page(folio, folio_nr_pages(folio));
+            remap_page(folio, folio_nr_pages(folio), 0);
             ret = -EAGAIN;
     }
mm/migrate.c +15 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
     return true;
  }
 
+static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+                                      struct folio *folio,
+                                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||

mm/migrate.c +5 -0
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+            return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte

mapping
+ * this subpage has been non present. If the subpage is only
zero-filled

mm/migrate.c +12 -0
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);
+
+    if (contains_data)
+            return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -1
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+
  /*
   * Restore a potential migration pte to a working pte entry
   */
  static bool remove_migration_pte(struct folio *folio,
-            struct vm_area_struct *vma, unsigned long addr, void

*old)
+ struct vm_area_struct *vma, unsigned long addr, void
*arg)
{
- DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC |
PVMW_MIGRATION);

mm/migrate.c +2 -0
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr,

PVMW_SYNC | PVMW_MIGRATION);

     while (page_vma_mapped_walk(&pvmw)) {
             rmap_t rmap_flags = RMAP_NONE;
mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
                     continue;
             }
  #endif
+            if (rmap_walk_arg->map_unused_to_zeropage &&
+                try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+                    continue;
 
             folio_get(folio);
             pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
mm/migrate.c +8 -3
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
   * Get rid of all migration entries and replace them by
   * references to the indicated page.
   */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
  {
+    struct rmap_walk_arg rmap_walk_arg = {
+            .folio = src,
+            .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
     struct rmap_walk_control rwc = {
             .rmap_one = remove_migration_pte,
-            .arg = src,
+            .arg = &rmap_walk_arg,
     };
 
-    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +4 -2
+
+    if (flags & RMP_LOCKED)
             rmap_walk_locked(dst, &rwc);
     else
             rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
      * At this point we know that the migration attempt cannot
      * be successful.
      */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
 
     rc = mapping->a_ops->writepage(&folio->page, &wbc);
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
                                struct list_head *ret)
  {
     if (page_was_mapped)
-            remove_migration_ptes(src, src, false);
+            remove_migration_ptes(src, src, 0);
     /* Drop an anon_vma reference if we took one */
     if (anon_vma)
             put_anon_vma(anon_vma);
mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
             lru_add_drain();
 
     if (old_page_state & PAGE_WAS_MAPPED)
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
 
  out_unlock_both:
     folio_unlock(dst);
mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
 
     if (page_was_mapped)
             remove_migration_ptes(src,
-                    rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+                    rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 
  unlock_put_anon:
     folio_unlock(dst);
mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
                     continue;
 
             folio = page_folio(page);
-            remove_migration_ptes(folio, folio, false);
+            remove_migration_ptes(folio, folio, 0);
 
             src_pfns[i] = 0;
             folio_unlock(folio);
mm/migrate_device.c +1 -1
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
                     dst = src;
             }
 
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
             folio_unlock(src);
 
Quoted 1 line

if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

Quoted 5 lines
Quoted 2 lines

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

It seems RISC-V also has a similar feature (RISCV_ISA_SUPM) that uses the same prctl(PR_{GET,SET}_TAGGED_ADDR_CTRL) API.

config RISCV_ISA_SUPM
bool "Supm extension for userspace pointer masking" depends on 64BIT
default y
help
Add support for pointer masking in userspace (Supm) when the underlying hardware extension (Smnpm or Ssnpm) is detected at boot.

Quoted 5 lines

If this option is disabled, userspace will be unable to use the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.

I wonder if we should disable the THP shrinker for such architectures that define PR_SET_TAGGED_ADDR_CTRL (or PR_GET_TAGGED_ADDR_CTRL).

SET_TAGGED_ADDR_CTRL or GET_TAGGED_ADDR_CTRL

File: kernel/sys.c 114
#ifndef SET_TAGGED_ADDR_CTRL

# define SET_TAGGED_ADDR_CTRL(a) (-EINVAL)
#endif
#ifndef GET_TAGGED_ADDR_CTRL
# define GET_TAGGED_ADDR_CTRL() (-EINVAL)
#endif

Cheers,
Lance

Quoted 14 lines

Cheers,
Lance

Quoted 10 lines

Also, I wonder how KSM and the shared zeropage works in general with that, because I would expect similar issues when we de-duplicate memory?

--
Cheers

David / dhildenb

#17 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to Lance Yang
Quoted 304 lines · Lance Yang
Quoted 286 lines · David Hildenbrand
Quoted 284 lines · Qun-wei Lin (林群崴)
Quoted 275 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
   include/linux/rmap.h |  7 ++++-
   mm/huge_memory.c     |  8 ++---
   mm/migrate.c         | 72 ++++++++++++++++++++++++++++++++++++++----
--
   mm/migrate_device.c  |  4 +--
   4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
   int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                    struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED              = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

/*
* rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -1
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
      return false;
   }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
   {
      int i = 0;
mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
      if (!folio_test_anon(folio))
              return;
      for (;;) {
-            remove_migration_ptes(folio, folio, true);
+            remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
              i += folio_nr_pages(folio);
              if (i >= nr)
                      break;
mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
 
      if (nr_dropped)
              shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
 

/*
* set page to its compound_head when split to non order-0
pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
              if (mapping)
                      xas_unlock(&xas);
              local_irq_enable();
-            remap_page(folio, folio_nr_pages(folio));
+            remap_page(folio, folio_nr_pages(folio), 0);
              ret = -EAGAIN;
      }
mm/migrate.c +15 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
      return true;
   }
 
+static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+                                      struct folio *folio,
+                                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||

mm/migrate.c +5 -0
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+            return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte

mapping
+ * this subpage has been non present. If the subpage is only
zero-filled

mm/migrate.c +12 -0
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);
+
+    if (contains_data)
+            return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -1
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+
   /*
    * Restore a potential migration pte to a working pte entry
    */
   static bool remove_migration_pte(struct folio *folio,
-            struct vm_area_struct *vma, unsigned long addr, void

*old)
+ struct vm_area_struct *vma, unsigned long addr, void
*arg)
{
- DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC |
PVMW_MIGRATION);

mm/migrate.c +2 -0
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr,

PVMW_SYNC | PVMW_MIGRATION);

      while (page_vma_mapped_walk(&pvmw)) {
              rmap_t rmap_flags = RMAP_NONE;
mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
                      continue;
              }
   #endif
+            if (rmap_walk_arg->map_unused_to_zeropage &&
+                try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+                    continue;
 
              folio_get(folio);
              pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
mm/migrate.c +8 -3
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
    * Get rid of all migration entries and replace them by
    * references to the indicated page.
    */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
   {
+    struct rmap_walk_arg rmap_walk_arg = {
+            .folio = src,
+            .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
      struct rmap_walk_control rwc = {
              .rmap_one = remove_migration_pte,
-            .arg = src,
+            .arg = &rmap_walk_arg,
      };
 
-    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);

mm/migrate.c +4 -2
+
+    if (flags & RMP_LOCKED)
              rmap_walk_locked(dst, &rwc);
      else
              rmap_walk(dst, &rwc);
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
       * At this point we know that the migration attempt cannot
       * be successful.
       */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
 
      rc = mapping->a_ops->writepage(&folio->page, &wbc);
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
                                 struct list_head *ret)
   {
      if (page_was_mapped)
-            remove_migration_ptes(src, src, false);
+            remove_migration_ptes(src, src, 0);
      /* Drop an anon_vma reference if we took one */
      if (anon_vma)
              put_anon_vma(anon_vma);
mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
              lru_add_drain();
 
      if (old_page_state & PAGE_WAS_MAPPED)
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
 
   out_unlock_both:
      folio_unlock(dst);
mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
 
      if (page_was_mapped)
              remove_migration_ptes(src,
-                    rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+                    rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 
   unlock_put_anon:
      folio_unlock(dst);
mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
                      continue;
 
              folio = page_folio(page);
-            remove_migration_ptes(folio, folio, false);
+            remove_migration_ptes(folio, folio, 0);
 
              src_pfns[i] = 0;
              folio_unlock(folio);
mm/migrate_device.c +1 -1
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
                      dst = src;
              }
 
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
              folio_unlock(src);
 
Quoted 1 line

if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

Quoted 5 lines
Quoted 2 lines

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

It seems RISC-V also has a similar feature (RISCV_ISA_SUPM) that uses the same prctl(PR_{GET,SET}_TAGGED_ADDR_CTRL) API.

config RISCV_ISA_SUPM
bool "Supm extension for userspace pointer masking" depends on 64BIT
default y
help
Add support for pointer masking in userspace (Supm) when the underlying hardware extension (Smnpm or Ssnpm) is detected at boot.

Quoted 4 lines

If this option is disabled, userspace will be unable to use the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.

I wonder if we should disable the THP shrinker for such architectures that

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Signature · 3 lines
Cheers

David / dhildenb
reviewer
#18 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 306 lines · David Hildenbrand
Quoted 305 lines · Lance Yang
Quoted 287 lines · David Hildenbrand
Quoted 285 lines · Qun-wei Lin (林群崴)
Quoted 276 lines · Usama Arif

From: Yu Zhao <[email protected]>

Here being unused means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, the unused subpages can be mapped to the shared zeropage, hence saving
memory. This is particularly helpful when the internal fragmentation of a thp is high, i.e. it has many untouched subpages.

This is also a prerequisite for THP low utilization shrinker which will
be introduced in later patches, where underutilized THPs are split, and
the zero-filled pages are freed saving memory.

---
   include/linux/rmap.h |  7 ++++-
   mm/huge_memory.c     |  8 ++---
   mm/migrate.c         | 72 +++++++++++++++++++++++++++++++++++++
+----

Signature · 2 lines
   mm/migrate_device.c  |  4 +--
   4 files changed, 75 insertions(+), 16 deletions(-)
include/linux/rmap.h +6 -1
index 91b5935e8485..d5e93e44322e 100644
@@ -745,7 +745,12 @@ int folio_mkclean(struct folio *);
    int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages,
 pgoff_t pgoff,
                     struct vm_area_struct *vma);
 
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked);
+enum rmp_flags {
+    RMP_LOCKED              = 1 << 0,
+    RMP_USE_SHARED_ZEROPAGE = 1 << 1,
+};
+
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags);

   /*
    * rmap_walk_control: To control rmap traversing for specific needs

mm/huge_memory.c +1 -1
index 0c48806ccb9a..af60684e7c70 100644
@@ -3020,7 +3020,7 @@ bool unmap_huge_pmd_locked(struct
 vm_area_struct *vma, unsigned long addr,
       return false;
    }
 
-static void remap_page(struct folio *folio, unsigned long nr)
+static void remap_page(struct folio *folio, unsigned long nr, int
 flags)
    {

      int i = 0;

mm/huge_memory.c +1 -1
@@ -3028,7 +3028,7 @@ static void remap_page(struct folio *folio,
 unsigned long nr)
       if (!folio_test_anon(folio))
               return;
       for (;;) {
-            remove_migration_ptes(folio, folio, true);
+            remove_migration_ptes(folio, folio, RMP_LOCKED |
 flags);
               i += folio_nr_pages(folio);

              if (i >= nr)
                      break;

mm/huge_memory.c +1 -1
@@ -3240,7 +3240,7 @@ static void __split_huge_page(struct page
 *page, struct list_head *list,
 
       if (nr_dropped)
               shmem_uncharge(folio->mapping->host, nr_dropped);
-    remap_page(folio, nr);
+    remap_page(folio, nr, PageAnon(head) ?
 RMP_USE_SHARED_ZEROPAGE : 0);
 

      /*
       * set page to its compound_head when split to non order-0 pages, so

mm/huge_memory.c +1 -1
@@ -3542,7 +3542,7 @@ int split_huge_page_to_list_to_order(struct
 page *page, struct list_head *list,
               if (mapping)
                       xas_unlock(&xas);
               local_irq_enable();
-            remap_page(folio, folio_nr_pages(folio));
+            remap_page(folio, folio_nr_pages(folio), 0);
               ret = -EAGAIN;
       }
mm/migrate.c +15 -0
index 6f9c62c746be..d039863e014b 100644
@@ -204,13 +204,57 @@ bool isolate_folio_to_list(struct folio *folio,
 struct list_head *list)
       return true;
    }
 
+static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
+                                      struct folio *folio,
+                                      unsigned long idx)
+{
+    struct page *page = folio_page(folio, idx);
+    bool contains_data;
+    pte_t newpte;
+    void *addr;
+
+    VM_BUG_ON_PAGE(PageCompound(page), page);
+    VM_BUG_ON_PAGE(!PageAnon(page), page);
+    VM_BUG_ON_PAGE(!PageLocked(page), page);
+    VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page);
+
+    if (folio_test_mlocked(folio) || (pvmw->vma->vm_flags &

VM_LOCKED) ||
+        mm_forbids_zeropage(pvmw->vma->vm_mm))
+            return false;
+
+    /*
+     * The pmd entry mapping the old thp was flushed and the pte mapping
+     * this subpage has been non present. If the subpage is only zero-filled
+     * then map it to the shared zeropage.
+     */
+    addr = kmap_local_page(page);
+    contains_data = memchr_inv(addr, 0, PAGE_SIZE);
+    kunmap_local(addr);

mm/migrate.c +7 -0
+
+    if (contains_data)
+            return false;
+
+    newpte = pte_mkspecial(pfn_pte(my_zero_pfn(pvmw->address),
+                                    pvmw->vma->vm_page_prot));
+    set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte,

newpte);

mm/migrate.c +10 -0
+
+    dec_mm_counter(pvmw->vma->vm_mm, mm_counter(folio));
+    return true;
+}
+
+struct rmap_walk_arg {
+    struct folio *folio;
+    bool map_unused_to_zeropage;
+};
+

   /*
    * Restore a potential migration pte to a working pte entry */
   static bool remove_migration_pte(struct folio *folio, -            struct vm_area_struct *vma, unsigned long addr, void *old)
+            struct vm_area_struct *vma, unsigned long addr, void *arg)
   {
-    DEFINE_FOLIO_VMA_WALK(pvmw, old, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+    struct rmap_walk_arg *rmap_walk_arg = arg;
+    DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);

      while (page_vma_mapped_walk(&pvmw)) {
              rmap_t rmap_flags = RMAP_NONE;

mm/migrate.c +3 -0
@@ -234,6 +278,9 @@ static bool remove_migration_pte(struct folio
 *folio,
                       continue;
               }
    #endif
+            if (rmap_walk_arg->map_unused_to_zeropage &&
+                try_to_map_unused_to_zeropage(&pvmw, folio,
 idx))
+                    continue;
 

              folio_get(folio);
              pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));

mm/migrate.c +8 -3
@@ -312,14 +359,21 @@ static bool remove_migration_pte(struct folio
 *folio,
     * Get rid of all migration entries and replace them by
     * references to the indicated page.
     */
-void remove_migration_ptes(struct folio *src, struct folio *dst,
 bool locked)
+void remove_migration_ptes(struct folio *src, struct folio *dst, int
 flags)
    {
+    struct rmap_walk_arg rmap_walk_arg = {
+            .folio = src,
+            .map_unused_to_zeropage = flags &
 RMP_USE_SHARED_ZEROPAGE,
+    };
+
       struct rmap_walk_control rwc = {
               .rmap_one = remove_migration_pte,
-            .arg = src,
+            .arg = &rmap_walk_arg,
       };
 
-    if (locked)
+    VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src !=

dst), src);
+
+    if (flags & RMP_LOCKED)
              rmap_walk_locked(dst, &rwc);
      else
              rmap_walk(dst, &rwc);

mm/migrate.c +2 -2
@@ -934,7 +988,7 @@ static int writeout(struct address_space
 *mapping, struct folio *folio)
        * At this point we know that the migration attempt cannot
        * be successful.
        */
-    remove_migration_ptes(folio, folio, false);
+    remove_migration_ptes(folio, folio, 0);
 
       rc = mapping->a_ops->writepage(&folio->page, &wbc);
@@ -1098,7 +1152,7 @@ static void migrate_folio_undo_src(struct folio
 *src,
                                  struct list_head *ret)
    {
       if (page_was_mapped)
-            remove_migration_ptes(src, src, false);
+            remove_migration_ptes(src, src, 0);
       /* Drop an anon_vma reference if we took one */
       if (anon_vma)

              put_anon_vma(anon_vma);

mm/migrate.c +1 -1
@@ -1336,7 +1390,7 @@ static int migrate_folio_move(free_folio_t
 put_new_folio, unsigned long private,
               lru_add_drain();
 
       if (old_page_state & PAGE_WAS_MAPPED)
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
 

   out_unlock_both:
      folio_unlock(dst);

mm/migrate.c +1 -1
@@ -1474,7 +1528,7 @@ static int unmap_and_move_huge_page(new_folio_t
 get_new_folio,
 
       if (page_was_mapped)
               remove_migration_ptes(src,
-                    rc == MIGRATEPAGE_SUCCESS ? dst : src,
 false);
+                    rc == MIGRATEPAGE_SUCCESS ? dst : src, 0);
 

   unlock_put_anon:
      folio_unlock(dst);

mm/migrate_device.c +1 -1
index 8d687de88a03..9cf26592ac93 100644
@@ -424,7 +424,7 @@ static unsigned long
 migrate_device_unmap(unsigned long *src_pfns,
                       continue;
 
               folio = page_folio(page);
-            remove_migration_ptes(folio, folio, false);
+            remove_migration_ptes(folio, folio, 0);
 
               src_pfns[i] = 0;

              folio_unlock(folio);

mm/migrate_device.c +1 -1
@@ -840,7 +840,7 @@ void migrate_device_finalize(unsigned long
 *src_pfns,
                       dst = src;
               }
 
-            remove_migration_ptes(src, dst, false);
+            remove_migration_ptes(src, dst, 0);
               folio_unlock(src);
 
Quoted 1 line

              if (folio_is_zone_device(src))

Hi,

This patch has been in the mainline for some time, but we recently discovered an issue when both mTHP and MTE (Memory Tagging Extension) are enabled.

Quoted 5 lines
Quoted 2 lines

It seems that remapping to the same zeropage might causes MTE tag mismatches, since MTE tags are associated with physical addresses.

Does this only trigger when the VMA has mte enabled? Maybe we'll have to bail out if we detect that mte is enabled.

It seems RISC-V also has a similar feature (RISCV_ISA_SUPM) that uses the same prctl(PR_{GET,SET}_TAGGED_ADDR_CTRL) API.

config RISCV_ISA_SUPM
         bool "Supm extension for userspace pointer masking" depends on 64BIT
         default y
         help
           Add support for pointer masking in userspace (Supm) when the underlying hardware extension (Smnpm or Ssnpm) is detected at boot.

Quoted 6 lines
Quoted 4 lines

           If this option is disabled, userspace will be unable to use the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.

I wonder if we should disable the THP shrinker for such architectures that

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct folio *folio). By default, it would return false, and architectures like arm64 can override it.

Looking at the code, the PG_mte_tagged flag is not set for regular THP. The MTE status actually comes from the VM_MTE flag in the VMA that maps it.

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio)

{
	bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

	VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

	/*
	 * If the folio is tagged, ensure ordering with a likely subsequent
	 * read of the tags.
	 */
	if (ret)
		smp_rmb();
	return ret;
}

static inline bool page_mte_tagged(struct page *page)

{
	bool ret = test_bit(PG_mte_tagged, &page->flags.f);

	VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

	/*
	 * If the page is tagged, ensure ordering with a likely subsequent
	 * read of the tags.
	 */
	if (ret)
		smp_rmb();
	return ret;
}

contpte_set_ptes()

	__set_ptes()
		__set_ptes_anysz()
			__sync_cache_and_tags()
				mte_sync_tags()
					set_page_mte_tagged()

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.
Cheers,
Lance

#19 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 9 lines
Quoted 3 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like arm64
can override it.

If we add a new helper it should instead express the semantics that we cannot deduplicate.

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Quoted 2 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Quoted 3 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 31 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio)

{
	bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

	VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

	/*
	 * If the folio is tagged, ensure ordering with a likely subsequent
	 * read of the tags.
	 */
	if (ret)
		smp_rmb();
	return ret;
}

static inline bool page_mte_tagged(struct page *page)

{
	bool ret = test_bit(PG_mte_tagged, &page->flags.f);

	VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

	/*
	 * If the page is tagged, ensure ordering with a likely subsequent
	 * read of the tags.
	 */
	if (ret)
		smp_rmb();
	return ret;
}

contpte_set_ptes()

Quoted 8 lines
	__set_ptes()
		__set_ptes_anysz()
			__sync_cache_and_tags()
				mte_sync_tags()
					set_page_mte_tagged()

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
        for (i = 0; i < folio_nr_pages(folio); i++) {
                kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                        num_zero_pages++;
                        if (num_zero_pages > khugepaged_max_ptes_none) {
                                kunmap_local(kaddr);
mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
 
        if (PageCompound(page))
                return false;
+       if (!page_is_mergable(page))
+               return false;
        VM_BUG_ON_PAGE(!PageAnon(page), page);
        VM_BUG_ON_PAGE(!PageLocked(page), page);
        VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

For KSM, similarly just bail out early. But still wondering if this is already checked somehow for KSM.

Signature · 3 lines
Cheers

David / dhildenb
reviewer
#20 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 11 lines · David Hildenbrand
Quoted 9 lines
Quoted 3 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like arm64
can override it.

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 2 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 5 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 44 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
    bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

    VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

    /*
     * If the folio is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

static inline bool page_mte_tagged(struct page *page) {
    bool ret = test_bit(PG_mte_tagged, &page->flags.f);

    VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

    /*
     * If the page is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

contpte_set_ptes()
    __set_ptes()
        __set_ptes_anysz()
            __sync_cache_and_tags()
                mte_sync_tags()
                    set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 35 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
         for (i = 0; i < folio_nr_pages(folio); i++) {
                 kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                         num_zero_pages++;
                         if (num_zero_pages > khugepaged_max_ptes_none) {
                                 kunmap_local(kaddr);
mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
 
         if (PageCompound(page))
                 return false;
+       if (!page_is_mergable(page))
+               return false;
         VM_BUG_ON_PAGE(!PageAnon(page), page);
         VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

        VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 4 lines

For KSM, similarly just bail out early. But still wondering if this is already checked somehow for KSM.

+1 I'm looking for a machine to test it on.

reviewer
#21 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 127 lines · Lance Yang
Quoted 11 lines · David Hildenbrand
Quoted 9 lines
Quoted 3 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like arm64
can override it.

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 2 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 5 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 44 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
    bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

    VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

    /*
     * If the folio is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

static inline bool page_mte_tagged(struct page *page) {
    bool ret = test_bit(PG_mte_tagged, &page->flags.f);

    VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

    /*
     * If the page is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

contpte_set_ptes()
    __set_ptes()
        __set_ptes_anysz()
            __sync_cache_and_tags()
                mte_sync_tags()
                    set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 35 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
          for (i = 0; i < folio_nr_pages(folio); i++) {
                  kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                          num_zero_pages++;
                          if (num_zero_pages > khugepaged_max_ptes_none) {
                                  kunmap_local(kaddr);
mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
 
          if (PageCompound(page))
                  return false;
+       if (!page_is_mergable(page))
+               return false;
          VM_BUG_ON_PAGE(!PageAnon(page), page);
          VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

         VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 2 lines
Quoted 4 lines
Quoted 2 lines

For KSM, similarly just bail out early. But still wondering if this is already checked somehow for KSM.

+1 I'm looking for a machine to test it on.

Interestingly, it seems KSM is already skipping MTE-tagged pages. My test, running on a v6.8.0 kernel inside QEMU (with MTE enabled), shows no merging activity for those pages ...

reviewer
#22 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Hey David,

I believe I've found the exact reason why KSM skips MTE-tagged pages ;p

Quoted 130 lines · Lance Yang
Quoted 127 lines · David Hildenbrand
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like arm64
can override it.

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 2 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 5 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 44 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
    bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

    VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

    /*
     * If the folio is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

static inline bool page_mte_tagged(struct page *page) {
    bool ret = test_bit(PG_mte_tagged, &page->flags.f);

    VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

    /*
     * If the page is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

contpte_set_ptes()
    __set_ptes()
        __set_ptes_anysz()
            __sync_cache_and_tags()
                mte_sync_tags()
                    set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 35 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
          for (i = 0; i < folio_nr_pages(folio); i++) {
                  kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                          num_zero_pages++;
                          if (num_zero_pages > khugepaged_max_ptes_none) {
                                  kunmap_local(kaddr);
mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
 
          if (PageCompound(page))
                  return false;
+       if (!page_is_mergable(page))
+               return false;
          VM_BUG_ON_PAGE(!PageAnon(page), page);
          VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

         VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

For KSM, similarly just bail out early. But still wondering if this is already checked somehow for KSM.

+1 I'm looking for a machine to test it on.

Interestingly, it seems KSM is already skipping MTE-tagged pages. My test, running on a v6.8.0 kernel inside QEMU (with MTE enabled), shows no merging activity for those pages ...

KSM's call to pages_identical() ultimately leads to memcmp_pages(). The arm64 implementation of memcmp_pages() in arch/arm64/kernel/mte.c contains a specific check that prevents merging in this case.

try_to_merge_one_page()

	-> pages_identical()
		-> !memcmp_pages() Fails!
		-> replace_page()

int memcmp_pages(struct page *page1, struct page *page2)

{
	char *addr1, *addr2;
	int ret;

	addr1 = page_address(page1);
	addr2 = page_address(page2);
	ret = memcmp(addr1, addr2, PAGE_SIZE);

	if (!system_supports_mte() || ret)
		return ret;

	/*
	 * If the page content is identical but at least one of the pages is
	 * tagged, return non-zero to avoid KSM merging. If only one of the
	 * pages is tagged, __set_ptes() may zero or change the tags of the
	 * other page via mte_sync_tags().
	 */
	if (page_mte_tagged(page1) || page_mte_tagged(page2))
		return addr1 != addr2;

	return ret;
}

IIUC, if either page is MTE-tagged, memcmp_pages() intentionally returns a non-zero value, which in turn causes pages_identical() to return false.

Cheers,
Lance

reviewer
#23 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 143 lines · Lance Yang

Hey David,

I believe I've found the exact reason why KSM skips MTE-tagged pages ;p

Quoted 130 lines · Lance Yang
Quoted 127 lines · David Hildenbrand
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like arm64
can override it.

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 2 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 5 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 44 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
    bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

    VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

    /*
     * If the folio is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

static inline bool page_mte_tagged(struct page *page) {
    bool ret = test_bit(PG_mte_tagged, &page->flags.f);

    VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

    /*
     * If the page is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

contpte_set_ptes()
    __set_ptes()
        __set_ptes_anysz()
            __sync_cache_and_tags()
                mte_sync_tags()
                    set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 35 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
          for (i = 0; i < folio_nr_pages(folio); i++) {
                  kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                          num_zero_pages++;
                          if (num_zero_pages > khugepaged_max_ptes_none) {
                                  kunmap_local(kaddr);
mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct 
 page_vma_mapped_walk *pvmw,
 
          if (PageCompound(page))
                  return false;
+       if (!page_is_mergable(page))
+               return false;
          VM_BUG_ON_PAGE(!PageAnon(page), page);
          VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

         VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 2 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

For KSM, similarly just bail out early. But still wondering if this is already checked somehow for KSM.

+1 I'm looking for a machine to test it on.

Interestingly, it seems KSM is already skipping MTE-tagged pages. My test, running on a v6.8.0 kernel inside QEMU (with MTE enabled), shows no merging activity for those pages ...

Quoted 8 lines

KSM's call to pages_identical() ultimately leads to memcmp_pages(). The arm64 implementation of memcmp_pages() in arch/arm64/kernel/mte.c contains a specific check that prevents merging in this case.

try_to_merge_one_page()
    -> pages_identical()
        -> !memcmp_pages() Fails!
        -> replace_page()

Forgot to add:

memcmp_pages() is also called in other KSM paths, such as stable_tree_search(), stable_tree_insert(), and unstable_tree_search_insert(), effectively blocking MTE-tagged pages from entering either of KSM's trees.

Quoted 31 lines

int memcmp_pages(struct page *page1, struct page *page2) {
    char *addr1, *addr2;
    int ret;

    addr1 = page_address(page1);
    addr2 = page_address(page2);
    ret = memcmp(addr1, addr2, PAGE_SIZE);

    if (!system_supports_mte() || ret)
        return ret;

    /*
     * If the page content is identical but at least one of the pages is
     * tagged, return non-zero to avoid KSM merging. If only one of the
     * pages is tagged, __set_ptes() may zero or change the tags of the
     * other page via mte_sync_tags().
     */
    if (page_mte_tagged(page1) || page_mte_tagged(page2)) return addr1 != addr2;

    return ret;
}

IIUC, if either page is MTE-tagged, memcmp_pages() intentionally returns a non-zero value, which in turn causes pages_identical() to return false.

Cheers,
Lance

#24 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

In reply to Lance Yang
Quoted 185 lines · Lance Yang

Hey David,

I believe I've found the exact reason why KSM skips MTE-tagged pages ;p

Quoted 141 lines · Lance Yang
Quoted 138 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM
as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like
arm64
can override it.

Quoted 2 lines

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 3 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 6 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 45 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
    bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

    VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

    /*
     * If the folio is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

static inline bool page_mte_tagged(struct page *page) {
    bool ret = test_bit(PG_mte_tagged, &page->flags.f);

    VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

    /*
     * If the page is tagged, ensure ordering with a likely subsequent
     * read of the tags.
     */
    if (ret)
        smp_rmb();
    return ret;
}

contpte_set_ptes()
    __set_ptes()
        __set_ptes_anysz()
            __sync_cache_and_tags()
                mte_sync_tags()
                    set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 36 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
          for (i = 0; i < folio_nr_pages(folio); i++) {
                  kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                          num_zero_pages++;
                          if (num_zero_pages >
 khugepaged_max_ptes_none) {

                                 kunmap_local(kaddr);

mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
 
          if (PageCompound(page))
                  return false;
+       if (!page_is_mergable(page))
+               return false;
          VM_BUG_ON_PAGE(!PageAnon(page), page);
          VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

         VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 5 lines

For KSM, similarly just bail out early. But still wondering if this is already checked
somehow for KSM.

Quoted 5 lines
Quoted 1 line

+1 I'm looking for a machine to test it on.

Interestingly, it seems KSM is already skipping MTE-tagged pages. My test, running on a v6.8.0 kernel inside QEMU (with MTE enabled), shows no merging activity for those pages ...

KSM's call to pages_identical() ultimately leads to memcmp_pages(). The arm64 implementation of memcmp_pages() in arch/arm64/kernel/mte.c contains a specific check that prevents merging in this case.

try_to_merge_one_page()

	-> pages_identical()
		-> !memcmp_pages() Fails!
		-> replace_page()

int memcmp_pages(struct page *page1, struct page *page2)

{
	char *addr1, *addr2;
	int ret;

	addr1 = page_address(page1);
	addr2 = page_address(page2);
	ret = memcmp(addr1, addr2, PAGE_SIZE);

	if (!system_supports_mte() || ret)
		return ret;

	/*
	 * If the page content is identical but at least one of the pages is
	 * tagged, return non-zero to avoid KSM merging. If only one of the
	 * pages is tagged, __set_ptes() may zero or change the tags of the
	 * other page via mte_sync_tags().
	 */
	if (page_mte_tagged(page1) || page_mte_tagged(page2))
		return addr1 != addr2;

	return ret;
}
Quoted 2 lines

IIUC, if either page is MTE-tagged, memcmp_pages() intentionally returns a non-zero value, which in turn causes pages_identical() to return false.

Cool, so we should likely just use that then in the shrinker code. Can you send a fix?

Signature · 3 lines
Cheers

David / dhildenb
reviewer
#25 lore raw

Re: [PATCH v5 2/6] mm: remap unused subpages to shared zeropage when splitting isolated thp

Quoted 183 lines · David Hildenbrand
Quoted 182 lines · Lance Yang

Hey David,

I believe I've found the exact reason why KSM skips MTE-tagged pages ;p

Quoted 141 lines · Lance Yang
Quoted 138 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines

I think where possible we really only want to identify problematic (tagged) pages and skip them. And we should either look into fixing KSM
as well or finding out why KSM is not affected.

Yeah. Seems like we could introduce a new helper, folio_test_mte_tagged(struct
folio *folio). By default, it would return false, and architectures like
arm64
can override it.

Quoted 2 lines

If we add a new helper it should instead express the semantics that we cannot deduplicate.

Agreed.

Quoted 3 lines

For THP, I recall that only some pages might be tagged. So likely we want to check per page.

Yes, a per-page check would be simpler.

Quoted 6 lines
Quoted 3 lines

Looking at the code, the PG_mte_tagged flag is not set for regular THP.

I think it's supported for THP per page. Only for hugetlb we tag the whole thing through the head page instead of individual pages.

Right. That's exactly what I meant.

Quoted 45 lines
Quoted 4 lines

The MTE
status actually comes from the VM_MTE flag in the VMA that maps it.

During the rmap walk we could check the VMA flag, but there would be no way to just stop the THP shrinker scanning this page early.

Quoted 37 lines

static inline bool folio_test_hugetlb_mte_tagged(struct folio *folio) {
     bool ret = test_bit(PG_mte_tagged, &folio->flags.f);

     VM_WARN_ON_ONCE(!folio_test_hugetlb(folio));

     /*
      * If the folio is tagged, ensure ordering with a likely subsequent
      * read of the tags.
      */
     if (ret)
         smp_rmb();
     return ret;
}

static inline bool page_mte_tagged(struct page *page) {
     bool ret = test_bit(PG_mte_tagged, &page->flags.f);

     VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page)));

     /*
      * If the page is tagged, ensure ordering with a likely subsequent
      * read of the tags.
      */
     if (ret)
         smp_rmb();
     return ret;
}

contpte_set_ptes()
     __set_ptes()
         __set_ptes_anysz()
             __sync_cache_and_tags()
                 mte_sync_tags()
                     set_page_mte_tagged()

Quoted 8 lines
Quoted 2 lines

Then, having the THP shrinker skip any folios that are identified as MTE-tagged.

Likely we should just do something like (maybe we want better naming)

#ifndef page_is_mergable
#define page_is_mergable(page) (true)
#endif

Maybe something like page_is_optimizable()? Just a thought ;p

Quoted 36 lines

And for arm64 have it be

#define page_is_mergable(page) (!page_mte_tagged(page))

And then do

mm/huge_memory.c +2 -1
index 1f0813b956436..1cac9093918d6 100644
@@ -4251,7 +4251,8 @@ static bool thp_underused(struct folio *folio)
 
           for (i = 0; i < folio_nr_pages(folio); i++) {
                   kaddr = kmap_local_folio(folio, i * PAGE_SIZE);
-               if (!memchr_inv(kaddr, 0, PAGE_SIZE)) {
+               if (page_is_mergable(folio_page(folio, i)) &&
+                   !memchr_inv(kaddr, 0, PAGE_SIZE)) {
                           num_zero_pages++;

                          if (num_zero_pages > khugepaged_max_ptes_none) { kunmap_local(kaddr);

mm/migrate.c +2 -0
index 946253c398072..476a9a9091bd3 100644
@@ -306,6 +306,8 @@ static bool try_to_map_unused_to_zeropage(struct
 page_vma_mapped_walk *pvmw,
 
           if (PageCompound(page))
                   return false;
+       if (!page_is_mergable(page))
+               return false;
           VM_BUG_ON_PAGE(!PageAnon(page), page);
           VM_BUG_ON_PAGE(!PageLocked(page), page);
Quoted 1 line

          VM_BUG_ON_PAGE(pte_present(ptep_get(pvmw->pte)), page);

Looks good to me!

Quoted 5 lines

For KSM, similarly just bail out early. But still wondering if this is already checked
somehow for KSM.

Quoted 5 lines
Quoted 1 line

+1 I'm looking for a machine to test it on.

Interestingly, it seems KSM is already skipping MTE-tagged pages. My test, running on a v6.8.0 kernel inside QEMU (with MTE enabled), shows no merging activity for those pages ...

KSM's call to pages_identical() ultimately leads to memcmp_pages(). The arm64 implementation of memcmp_pages() in arch/arm64/kernel/mte.c contains a specific check that prevents merging in this case.

try_to_merge_one_page()
    -> pages_identical()
        -> !memcmp_pages() Fails!
        -> replace_page()

int memcmp_pages(struct page *page1, struct page *page2) {
    char *addr1, *addr2;
    int ret;

    addr1 = page_address(page1);
    addr2 = page_address(page2);
    ret = memcmp(addr1, addr2, PAGE_SIZE);

    if (!system_supports_mte() || ret)
        return ret;

    /*
     * If the page content is identical but at least one of the pages is
     * tagged, return non-zero to avoid KSM merging. If only one of the
     * pages is tagged, __set_ptes() may zero or change the tags of the
     * other page via mte_sync_tags().
     */
    if (page_mte_tagged(page1) || page_mte_tagged(page2)) return addr1 != addr2;

Quoted 7 lines
Quoted 5 lines

    return ret;
}

IIUC, if either page is MTE-tagged, memcmp_pages() intentionally returns a non-zero value, which in turn causes pages_identical() to return false.

Cool, so we should likely just use that then in the shrinker code. Can you send a fix?

Certainly! I'll get on that ;p

Cheers,
Lance

author
#26 lore raw

PATCH v5 3/6 mm: selftest to verify zero-filled pages are mapped to zeropage

In reply to Usama Arif

From: Alexander Zhu <[email protected]>

When a THP is split, any subpage that is zero-filled will be mapped to the shared zeropage, hence saving memory. Add selftest to verify this by allocating zero-filled THP and comparing RssAnon before and after split.

---
 .../selftests/mm/split_huge_page_test.c       | 71 +++++++++++++++++++
 tools/testing/selftests/mm/vm_util.c          | 22 ++++++
 tools/testing/selftests/mm/vm_util.h          |  1 +
 3 files changed, 94 insertions(+)
tools/testing/selftests/mm/split_huge_page_test.c +71 -0
index e5e8dafc9d94..eb6d1b9fc362 100644
@@ -84,6 +84,76 @@ static void write_debugfs(const char *fmt, ...)
 	write_file(SPLIT_DEBUGFS, input, ret + 1);
 }
 
+static char *allocate_zero_filled_hugepage(size_t len)
+{
+	char *result;
+	size_t i;
+
+	result = memalign(pmd_pagesize, len);
+	if (!result) {
+		printf("Fail to allocate memory\n");
+		exit(EXIT_FAILURE);
+	}
+
+	madvise(result, len, MADV_HUGEPAGE);
+
+	for (i = 0; i < len; i++)
+		result[i] = (char)0;
+
+	return result;
+}
+
+static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hpages, size_t len)
+{
+	unsigned long rss_anon_before, rss_anon_after;
+	size_t i;
+
+	if (!check_huge_anon(one_page, 4, pmd_pagesize)) {
+		printf("No THP is allocated\n");
+		exit(EXIT_FAILURE);
+	}
+
+	rss_anon_before = rss_anon();
+	if (!rss_anon_before) {
+		printf("No RssAnon is allocated before split\n");
+		exit(EXIT_FAILURE);
+	}
+
+	/* split all THPs */
+	write_debugfs(PID_FMT, getpid(), (uint64_t)one_page,
+		      (uint64_t)one_page + len, 0);
+
+	for (i = 0; i < len; i++)
+		if (one_page[i] != (char)0) {
+			printf("%ld byte corrupted\n", i);
+			exit(EXIT_FAILURE);
+		}
+
+	if (!check_huge_anon(one_page, 0, pmd_pagesize)) {
+		printf("Still AnonHugePages not split\n");
+		exit(EXIT_FAILURE);
+	}
+
+	rss_anon_after = rss_anon();
+	if (rss_anon_after >= rss_anon_before) {
+		printf("Incorrect RssAnon value. Before: %ld After: %ld\n",
+		       rss_anon_before, rss_anon_after);
+		exit(EXIT_FAILURE);
+	}
+}
+
+void split_pmd_zero_pages(void)
+{
+	char *one_page;
+	int nr_hpages = 4;
+	size_t len = nr_hpages * pmd_pagesize;
+
+	one_page = allocate_zero_filled_hugepage(len);
+	verify_rss_anon_split_huge_page_all_zeroes(one_page, nr_hpages, len);
+	printf("Split zero filled huge pages successful\n");
+	free(one_page);
+}
+
 void split_pmd_thp(void)
 {
 	char *one_page;
@@ -431,6 +501,7 @@ int main(int argc, char **argv)
 
 	fd_size = 2 * pmd_pagesize;
 
+	split_pmd_zero_pages();
 	split_pmd_thp();
 	split_pte_mapped_thp();
 	split_file_backed_thp();
tools/testing/selftests/mm/vm_util.c +22 -0
index 5a62530da3b5..d8d0cf04bb57 100644
@@ -12,6 +12,7 @@
 
 #define PMD_SIZE_FILE_PATH "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size"
 #define SMAP_FILE_PATH "/proc/self/smaps"
+#define STATUS_FILE_PATH "/proc/self/status"
 #define MAX_LINE_LENGTH 500
 
 unsigned int __page_size;
@@ -171,6 +172,27 @@ uint64_t read_pmd_pagesize(void)
 	return strtoul(buf, NULL, 10);
 }
 
+unsigned long rss_anon(void)
+{
+	unsigned long rss_anon = 0;
+	FILE *fp;
+	char buffer[MAX_LINE_LENGTH];
+
+	fp = fopen(STATUS_FILE_PATH, "r");
+	if (!fp)
+		ksft_exit_fail_msg("%s: Failed to open file %s\n", __func__, STATUS_FILE_PATH);
+
+	if (!check_for_pattern(fp, "RssAnon:", buffer, sizeof(buffer)))
+		goto err_out;
+
+	if (sscanf(buffer, "RssAnon:%10lu kB", &rss_anon) != 1)
+		ksft_exit_fail_msg("Reading status error\n");
+
+err_out:
+	fclose(fp);
+	return rss_anon;
+}
+
 bool __check_huge(void *addr, char *pattern, int nr_hpages,
 		  uint64_t hpage_size)
 {
tools/testing/selftests/mm/vm_util.h +1 -0
index 9007c420d52c..2eaed8209925 100644
@@ -39,6 +39,7 @@ unsigned long pagemap_get_pfn(int fd, char *start);
 void clear_softdirty(void);
 bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t len);
 uint64_t read_pmd_pagesize(void);
+unsigned long rss_anon(void);
 bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size);
 bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size);
 bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size);
Signature · 1 line
2.43.5
author
#27 lore raw

PATCH v5 4/6 mm: Introduce a pageflag for partially mapped folios

In reply to Usama Arif

Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list.

This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan.

Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios.

---
 include/linux/huge_mm.h    |  4 ++--
 include/linux/page-flags.h | 13 +++++++++++-
 mm/huge_memory.c           | 41 ++++++++++++++++++++++++++++----------
 mm/memcontrol.c            |  3 ++-
 mm/migrate.c               |  3 ++-
 mm/page_alloc.c            |  5 +++--
 mm/rmap.c                  |  5 +++--
 mm/vmscan.c                |  3 ++-
 8 files changed, 56 insertions(+), 21 deletions(-)
include/linux/huge_mm.h +2 -2
index 4da102b74a8c..0b0539f4ee1a 100644
@@ -333,7 +333,7 @@ static inline int split_huge_page(struct page *page)
 {
 	return split_huge_page_to_list_to_order(page, NULL, 0);
 }
-void deferred_split_folio(struct folio *folio);
+void deferred_split_folio(struct folio *folio, bool partially_mapped);
 
 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long address, bool freeze, struct folio *folio);
@@ -502,7 +502,7 @@ static inline int split_huge_page(struct page *page)
 {
 	return 0;
 }
-static inline void deferred_split_folio(struct folio *folio) {}
+static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
 #define split_huge_pmd(__vma, __pmd, __address)	\
 	do { } while (0)
 
include/linux/page-flags.h +12 -1
index 2175ebceb41c..1b3a76710487 100644
@@ -186,6 +186,7 @@ enum pageflags {
 	/* At least one page in this folio has the hwpoison flag set */
 	PG_has_hwpoisoned = PG_active,
 	PG_large_rmappable = PG_workingset, /* anon or file-backed */
+	PG_partially_mapped = PG_reclaim, /* was identified to be partially mapped */
 };
 
 #define PAGEFLAGS_MASK		((1UL << NR_PAGEFLAGS) - 1)
@@ -859,8 +860,18 @@ static inline void ClearPageCompound(struct page *page)
 	ClearPageHead(page);
 }
 FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE)
+FOLIO_TEST_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
+/*
+ * PG_partially_mapped is protected by deferred_split split_queue_lock,
+ * so its safe to use non-atomic set/clear.
+ */
+__FOLIO_SET_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
+__FOLIO_CLEAR_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
 #else
 FOLIO_FLAG_FALSE(large_rmappable)
+FOLIO_TEST_FLAG_FALSE(partially_mapped)
+__FOLIO_SET_FLAG_NOOP(partially_mapped)
+__FOLIO_CLEAR_FLAG_NOOP(partially_mapped)
 #endif
 
 #define PG_head_mask ((1UL << PG_head))
@@ -1171,7 +1182,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page)
  */
 #define PAGE_FLAGS_SECOND						\
 	(0xffUL /* order */		| 1UL << PG_has_hwpoisoned |	\
-	 1UL << PG_large_rmappable)
+	 1UL << PG_large_rmappable	| 1UL << PG_partially_mapped)
 
 #define PAGE_FLAGS_PRIVATE				\
 	(1UL << PG_private | 1UL << PG_private_2)
mm/huge_memory.c +30 -11

72 lines not on this page · raw message

mm/memcontrol.c +2 -1
index 087a8cb1a6d8..e66da58a365d 100644
@@ -4629,7 +4629,8 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug)
 	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
 	VM_BUG_ON_FOLIO(folio_order(folio) > 1 &&
 			!folio_test_hugetlb(folio) &&
-			!list_empty(&folio->_deferred_list), folio);
+			!list_empty(&folio->_deferred_list) &&
+			folio_test_partially_mapped(folio), folio);
 
 	/*
 	 * Nobody should be changing or seriously looking at
mm/migrate.c +2 -1
index d039863e014b..35cc9d35064b 100644
@@ -1766,7 +1766,8 @@ static int migrate_pages_batch(struct list_head *from,
 			 * use _deferred_list.
 			 */
 			if (nr_pages > 2 &&
-			   !list_empty(&folio->_deferred_list)) {
+			   !list_empty(&folio->_deferred_list) &&
+			   folio_test_partially_mapped(folio)) {
 				if (!try_split_folio(folio, split_folios, mode)) {
 					nr_failed++;
 					stats->nr_thp_failed += is_thp;
mm/page_alloc.c +3 -2
index c2ffccf9d213..a82c221b7c2e 100644
@@ -962,8 +962,9 @@ static int free_tail_page_prepare(struct page *head_page, struct page *page)
 		break;
 	case 2:
 		/* the second tail page: deferred_list overlaps ->mapping */
-		if (unlikely(!list_empty(&folio->_deferred_list))) {
-			bad_page(page, "on deferred list");
+		if (unlikely(!list_empty(&folio->_deferred_list) &&
+		    folio_test_partially_mapped(folio))) {
+			bad_page(page, "partially mapped folio on deferred list");
 			goto out;
 		}
 		break;
mm/rmap.c +3 -2
index 78529cf0fd66..a8797d1b3d49 100644
@@ -1579,8 +1579,9 @@ static __always_inline void __folio_remove_rmap(struct folio *folio,
 	 * Check partially_mapped first to ensure it is a large folio.
 	 */
 	if (partially_mapped && folio_test_anon(folio) &&
-	    list_empty(&folio->_deferred_list))
-		deferred_split_folio(folio);
+	    !folio_test_partially_mapped(folio))
+		deferred_split_folio(folio, true);
+
 	__folio_mod_stat(folio, -nr, -nr_pmdmapped);
 
 	/*
mm/vmscan.c +2 -1
index f27792e77a0f..4ca612f7e473 100644
@@ -1238,7 +1238,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 					 * Split partially mapped folios right away.
 					 * We can free the unmapped pages without IO.
 					 */
-					if (data_race(!list_empty(&folio->_deferred_list)) &&
+					if (data_race(!list_empty(&folio->_deferred_list) &&
+					    folio_test_partially_mapped(folio)) &&
 					    split_folio_to_list(folio, folio_list))
 						goto activate_locked;
 				}
Signature · 1 line
2.43.5
#28 lore raw

Re: [PATCH v5 4/6] mm: Introduce a pageflag for partially mapped folios

Quoted 65 lines · Usama Arif

Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list.

This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan.

Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios.

---
  include/linux/huge_mm.h    |  4 ++--
  include/linux/page-flags.h | 13 +++++++++++-
  mm/huge_memory.c           | 41 ++++++++++++++++++++++++++++----------
  mm/memcontrol.c            |  3 ++-
  mm/migrate.c               |  3 ++-
  mm/page_alloc.c            |  5 +++--
  mm/rmap.c                  |  5 +++--
  mm/vmscan.c                |  3 ++-
  8 files changed, 56 insertions(+), 21 deletions(-)
include/linux/huge_mm.h +2 -2
index 4da102b74a8c..0b0539f4ee1a 100644
@@ -333,7 +333,7 @@ static inline int split_huge_page(struct page *page)
  {
  	return split_huge_page_to_list_to_order(page, NULL, 0);
  }
-void deferred_split_folio(struct folio *folio);
+void deferred_split_folio(struct folio *folio, bool partially_mapped);
 
  void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
  		unsigned long address, bool freeze, struct folio *folio);
@@ -502,7 +502,7 @@ static inline int split_huge_page(struct page *page)
  {
  	return 0;
  }
-static inline void deferred_split_folio(struct folio *folio) {}
+static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
  #define split_huge_pmd(__vma, __pmd, __address)	\
  	do { } while (0)
 
include/linux/page-flags.h +1 -0
index 2175ebceb41c..1b3a76710487 100644
@@ -186,6 +186,7 @@ enum pageflags {
  	/* At least one page in this folio has the hwpoison flag set */
  	PG_has_hwpoisoned = PG_active,
  	PG_large_rmappable = PG_workingset, /* anon or file-backed */
+	PG_partially_mapped = PG_reclaim, /* was identified to be partially mapped */
  };
 
  #define PAGEFLAGS_MASK		((1UL << NR_PAGEFLAGS) - 1)

include/linux/page-flags.h static inline void ClearPageCompound(struct page *page) · line 866

Quoted 7 lines
include/linux/page-flags.h +4 -0
@@ -859,8 +860,18 @@ static inline void ClearPageCompound(struct page *page)
  	ClearPageHead(page);
  }
  FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE)
+FOLIO_TEST_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
+/*
+ * PG_partially_mapped is protected by deferred_split split_queue_lock,
+ * so its safe to use non-atomic set/clear.

Just stumbled over that. In my understanding, this assumption is wrong.

I don't think anything prevents other PF_ANY (PG_anon_exclusive, PG_PG_hwpoison) / PF_SECOND (PF_has_hwpoisoned) flags from getting modified concurrently I'm afraid.

Signature · 3 lines
Cheers,

David / dhildenb
author
#29 lore raw

Re: [PATCH v5 4/6] mm: Introduce a pageflag for partially mapped folios

Quoted 73 lines · David Hildenbrand
Quoted 62 lines · Usama Arif

Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list.

This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan.

Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios.

---
  include/linux/huge_mm.h    |  4 ++-- include/linux/page-flags.h | 13 +++++++++++- mm/huge_memory.c           | 41 ++++++++++++++++++++++++++++---------- mm/memcontrol.c            |  3 ++-
  mm/migrate.c               |  3 ++-
  mm/page_alloc.c            |  5 +++--
  mm/rmap.c                  |  5 +++--
  mm/vmscan.c                |  3 ++-

  8 files changed, 56 insertions(+), 21 deletions(-)
include/linux/huge_mm.h +2 -2
index 4da102b74a8c..0b0539f4ee1a 100644
@@ -333,7 +333,7 @@ static inline int split_huge_page(struct page *page)
   {
       return split_huge_page_to_list_to_order(page, NULL, 0);
   }
-void deferred_split_folio(struct folio *folio);
+void deferred_split_folio(struct folio *folio, bool partially_mapped);
     void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
           unsigned long address, bool freeze, struct folio *folio);
@@ -502,7 +502,7 @@ static inline int split_huge_page(struct page *page)
   {
       return 0;
   }
-static inline void deferred_split_folio(struct folio *folio) {}
+static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
   #define split_huge_pmd(__vma, __pmd, __address)    \
       do { } while (0)

  diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 2175ebceb41c..1b3a76710487 100644

include/linux/page-flags.h +0 -0
@@ -186,6 +186,7 @@ enum pageflags {
       /* At least one page in this folio has the hwpoison flag set */
       PG_has_hwpoisoned = PG_active,

      PG_large_rmappable = PG_workingset, /* anon or file-backed */
+    PG_partially_mapped = PG_reclaim, /* was identified to be partially mapped */ };
    #define PAGEFLAGS_MASK        ((1UL << NR_PAGEFLAGS) - 1)

include/linux/page-flags.h static inline void ClearPageCompound(struct page *page) · line 866

Quoted 7 lines
include/linux/page-flags.h +4 -0
@@ -859,8 +860,18 @@ static inline void ClearPageCompound(struct page *page)
       ClearPageHead(page);
   }
   FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE)
+FOLIO_TEST_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
+/*
+ * PG_partially_mapped is protected by deferred_split split_queue_lock,
+ * so its safe to use non-atomic set/clear.

Just stumbled over that. In my understanding, this assumption is wrong.

Quoted 2 lines

I don't think anything prevents other PF_ANY (PG_anon_exclusive, PG_PG_hwpoison) / PF_SECOND (PF_has_hwpoisoned) flags from getting modified concurrently I'm afraid.

Hi David,

Just to clear my understanding, what you are suggesting could happen in __folio_set/clear_partially_mapped is:
1) __folio_set/clear_partially_mapped reads the 2nd page flags (x) where one of the other 2nd page flags is lets say not set.
2) One of the other 2nd page flags are set atomically.
3) __folio_set/clear_partially_mapped writes x + changes to partially_mapped. However, the change in step 2 to one of the other 2nd page flag is lost.

Is that correct? But that would mean we shouldn't have any page flags (first or second page) as non atomic? although it would depend if they are being changed at the same time point. If you encountered a particular instance of PG_anon_exclusive or PF_has_hwpoisoned being changed at the same point as __folio_set/clear_partially_mapped, could you point to it?

I am happy to send a fix to change all set/clear_partially_mapped to atomic, but just want to understand this better.

Thanks!
Usama

#30 lore raw

Re: [PATCH v5 4/6] mm: Introduce a pageflag for partially mapped folios

Quoted 78 lines · Usama Arif
Quoted 73 lines · David Hildenbrand
Quoted 62 lines · Usama Arif

Currently folio->_deferred_list is used to keep track of partially_mapped folios that are going to be split under memory pressure. In the next patch, all THPs that are faulted in and collapsed by khugepaged are also going to be tracked using _deferred_list.

This patch introduces a pageflag to be able to distinguish between partially mapped folios and others in the deferred_list at split time in deferred_split_scan. Its needed as __folio_remove_rmap decrements _mapcount, _large_mapcount and _entire_mapcount, hence it won't be possible to distinguish between partially mapped folios and others in deferred_split_scan.

Eventhough it introduces an extra flag to track if the folio is partially mapped, there is no functional change intended with this patch and the flag is not useful in this patch itself, it will become useful in the next patch when _deferred_list has non partially mapped folios.

---
   include/linux/huge_mm.h    |  4 ++--
   include/linux/page-flags.h | 13 +++++++++++-
   mm/huge_memory.c           | 41 ++++++++++++++++++++++++++++----------
   mm/memcontrol.c            |  3 ++-
   mm/migrate.c               |  3 ++-
   mm/page_alloc.c            |  5 +++--
   mm/rmap.c                  |  5 +++--
   mm/vmscan.c                |  3 ++-
   8 files changed, 56 insertions(+), 21 deletions(-)
include/linux/huge_mm.h +2 -2
index 4da102b74a8c..0b0539f4ee1a 100644
@@ -333,7 +333,7 @@ static inline int split_huge_page(struct page *page)
   {
       return split_huge_page_to_list_to_order(page, NULL, 0);
   }
-void deferred_split_folio(struct folio *folio);
+void deferred_split_folio(struct folio *folio, bool partially_mapped);
     void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
           unsigned long address, bool freeze, struct folio *folio);
@@ -502,7 +502,7 @@ static inline int split_huge_page(struct page *page)
   {
       return 0;
   }
-static inline void deferred_split_folio(struct folio *folio) {}
+static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
   #define split_huge_pmd(__vma, __pmd, __address)    \
       do { } while (0)
   diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h

index 2175ebceb41c..1b3a76710487 100644

include/linux/page-flags.h +1 -0
@@ -186,6 +186,7 @@ enum pageflags {
       /* At least one page in this folio has the hwpoison flag set */
       PG_has_hwpoisoned = PG_active,
       PG_large_rmappable = PG_workingset, /* anon or file-backed */
+    PG_partially_mapped = PG_reclaim, /* was identified to be partially mapped */
   };
     #define PAGEFLAGS_MASK        ((1UL << NR_PAGEFLAGS) - 1)

include/linux/page-flags.h static inline void ClearPageCompound(struct page *page) · line 866

Quoted 7 lines
include/linux/page-flags.h +4 -0
@@ -859,8 +860,18 @@ static inline void ClearPageCompound(struct page *page)
       ClearPageHead(page);
   }
   FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE)
+FOLIO_TEST_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
+/*
+ * PG_partially_mapped is protected by deferred_split split_queue_lock,
+ * so its safe to use non-atomic set/clear.

Just stumbled over that. In my understanding, this assumption is wrong.

Quoted 2 lines

I don't think anything prevents other PF_ANY (PG_anon_exclusive, PG_PG_hwpoison) / PF_SECOND (PF_has_hwpoisoned) flags from getting modified concurrently I'm afraid.

Quoted 8 lines

Hi David,

Just to clear my understanding, what you are suggesting could happen in __folio_set/clear_partially_mapped is:
1) __folio_set/clear_partially_mapped reads the 2nd page flags (x) where one of the other 2nd page flags is lets say not set.
2) One of the other 2nd page flags are set atomically.
3) __folio_set/clear_partially_mapped writes x + changes to partially_mapped. However, the change in step 2 to one of the other 2nd page flag is lost.

Is that correct?

That matches my understanding.

But that would mean we shouldn't have any page flags (first or second page) as non atomic?

Yes. We may only use non-atomic variants as long as we can guarantee that nobody can concurrently operate on the flags, for example on the early folio allocation path or on the folio freeing path.

Observe how the other SECOND users are atomic, PG_anon_exclusive is atomic (except on two page freeing paths) and PF_hwpoison is atomic.

Quoted 3 lines

although it would depend if they are being
changed at the same time point. If you encountered a particular instance of PG_anon_exclusive or PF_has_hwpoisoned being changed at the same point as __folio_set/clear_partially_mapped, could you point to it?

Regarding PG_hwpoison, observe how memory_failure() performs the TestSetPageHWPoison() + folio_set_has_hwpoisoned() before unmapping the pages, without any locking. This can race with pretty much any operation that triggers unmapping.

With PG_anon_exclusive it's a bit more complicated, but it's probably sufficient if MADV_DONTNEED (setting deferred) races with concurrent swapout/mgration (clearing PG_anon_exclusive), whereby both operations are not performed under the same PT lock. This can happen after partial mremap(), or after fork() when only parts of the large folio were shared with the child.

Signature · 3 lines
Cheers,

David / dhildenb
author
#31 lore raw

PATCH v5 5/6 mm: split underused THPs

In reply to Usama Arif

This is an attempt to mitigate the issue of running out of memory when THP is always enabled. During runtime whenever a THP is being faulted in (__do_huge_pmd_anonymous_page) or collapsed by khugepaged (collapse_huge_page), the THP is added to _deferred_list. Whenever memory reclaim happens in linux, the kernel runs the deferred_split shrinker which goes through the _deferred_list.

If the folio was partially mapped, the shrinker attempts to split it. If the folio is not partially mapped, the shrinker checks if the THP was underused, i.e. how many of the base 4K pages of the entire THP were zero-filled. If this number goes above a certain threshold (decided by /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none), the shrinker will attempt to split that THP. Then at remap time, the pages that were zero-filled are mapped to the shared zeropage, hence saving memory.

---
 Documentation/admin-guide/mm/transhuge.rst |  6 +++
 include/linux/khugepaged.h                 |  1 +
 include/linux/vm_event_item.h              |  1 +
 mm/huge_memory.c                           | 60 +++++++++++++++++++++-
 mm/khugepaged.c                            |  3 +-
 mm/vmstat.c                                |  1 +
 6 files changed, 69 insertions(+), 3 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +6 -0
index 56a086900651..aca0cff852b8 100644
@@ -471,6 +471,12 @@ thp_deferred_split_page
 	splitting it would free up some memory. Pages on split queue are
 	going to be split under memory pressure.
 
+thp_underused_split_page
+	is incremented when a huge page on the split queue was split
+	because it was underused. A THP is underused if the number of
+	zero pages in the THP is above a certain threshold
+	(/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none).
+
 thp_split_pmd
 	is incremented every time a PMD split into table of PTEs.
 	This can happen, for instance, when application calls mprotect() or
include/linux/khugepaged.h +1 -0
index f68865e19b0b..30baae91b225 100644
@@ -4,6 +4,7 @@
 
 #include <linux/sched/coredump.h> /* MMF_VM_HUGEPAGE */
 
+extern unsigned int khugepaged_max_ptes_none __read_mostly;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 extern struct attribute_group khugepaged_attr_group;
 
include/linux/vm_event_item.h +1 -0
index aae5c7c5cfb4..aed952d04132 100644
@@ -105,6 +105,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		THP_SPLIT_PAGE,
 		THP_SPLIT_PAGE_FAILED,
 		THP_DEFERRED_SPLIT_PAGE,
+		THP_UNDERUSED_SPLIT_PAGE,
 		THP_SPLIT_PMD,
 		THP_SCAN_EXCEED_NONE_PTE,
 		THP_SCAN_EXCEED_SWAP_PTE,
mm/huge_memory.c +58 -2

83 lines not on this page · raw message

mm/khugepaged.c +2 -1
index 5bfb5594c604..bf1734e8e665 100644
@@ -85,7 +85,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);
  *
  * Note that these are only respected if collapse was initiated by khugepaged.
  */
-static unsigned int khugepaged_max_ptes_none __read_mostly;
+unsigned int khugepaged_max_ptes_none __read_mostly;
 static unsigned int khugepaged_max_ptes_swap __read_mostly;
 static unsigned int khugepaged_max_ptes_shared __read_mostly;
 
@@ -1237,6 +1237,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	pgtable_trans_huge_deposit(mm, pmd, pgtable);
 	set_pmd_at(mm, address, pmd, _pmd);
 	update_mmu_cache_pmd(vma, address, pmd);
+	deferred_split_folio(folio, false);
 	spin_unlock(pmd_ptl);
 
 	folio = NULL;
mm/vmstat.c +1 -0
index f41984dc856f..bb081ae4d0ae 100644
@@ -1385,6 +1385,7 @@ const char * const vmstat_text[] = {
 	"thp_split_page",
 	"thp_split_page_failed",
 	"thp_deferred_split_page",
+	"thp_underused_split_page",
 	"thp_split_pmd",
 	"thp_scan_exceed_none_pte",
 	"thp_scan_exceed_swap_pte",
Signature · 1 line
2.43.5
author
#32 lore raw

PATCH v5 6/6 mm: add sysfs entry to disable splitting underused THPs

In reply to Usama Arif

If disabled, THPs faulted in or collapsed will not be added to _deferred_list, and therefore won't be considered for splitting under memory pressure if underused.

---
 Documentation/admin-guide/mm/transhuge.rst | 10 +++++++++
 mm/huge_memory.c                           | 26 ++++++++++++++++++++++
 2 files changed, 36 insertions(+)
Documentation/admin-guide/mm/transhuge.rst +10 -0
index aca0cff852b8..cfdd16a52e39 100644
@@ -202,6 +202,16 @@ PMD-mappable transparent hugepage::
 
 	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 
+All THPs at fault and collapse time will be added to _deferred_list,
+and will therefore be split under memory presure if they are considered
+"underused". A THP is underused if the number of zero-filled pages in
+the THP is above max_ptes_none (see below). It is possible to disable
+this behaviour by writing 0 to shrink_underused, and enable it by writing
+1 to it::
+
+	echo 0 > /sys/kernel/mm/transparent_hugepage/shrink_underused
+	echo 1 > /sys/kernel/mm/transparent_hugepage/shrink_underused
+
 khugepaged will be automatically started when PMD-sized THP is enabled
 (either of the per-size anon control or the top-level control are set
 to "always" or "madvise"), and it'll be automatically shutdown when
mm/huge_memory.c +26 -0
index a97aeffc55d6..0993dfe9ae94 100644
@@ -74,6 +74,7 @@ static unsigned long deferred_split_count(struct shrinker *shrink,
 					  struct shrink_control *sc);
 static unsigned long deferred_split_scan(struct shrinker *shrink,
 					 struct shrink_control *sc);
+static bool split_underused_thp = true;
 
 static atomic_t huge_zero_refcount;
 struct folio *huge_zero_folio __read_mostly;
@@ -440,6 +441,27 @@ static ssize_t hpage_pmd_size_show(struct kobject *kobj,
 static struct kobj_attribute hpage_pmd_size_attr =
 	__ATTR_RO(hpage_pmd_size);
 
+static ssize_t split_underused_thp_show(struct kobject *kobj,
+			    struct kobj_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%d\n", split_underused_thp);
+}
+
+static ssize_t split_underused_thp_store(struct kobject *kobj,
+			     struct kobj_attribute *attr,
+			     const char *buf, size_t count)
+{
+	int err = kstrtobool(buf, &split_underused_thp);
+
+	if (err < 0)
+		return err;
+
+	return count;
+}
+
+static struct kobj_attribute split_underused_thp_attr = __ATTR(
+	shrink_underused, 0644, split_underused_thp_show, split_underused_thp_store);
+
 static struct attribute *hugepage_attr[] = {
 	&enabled_attr.attr,
 	&defrag_attr.attr,
@@ -448,6 +470,7 @@ static struct attribute *hugepage_attr[] = {
 #ifdef CONFIG_SHMEM
 	&shmem_enabled_attr.attr,
 #endif
+	&split_underused_thp_attr.attr,
 	NULL,
 };
 
@@ -3601,6 +3624,9 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
 	if (folio_order(folio) <= 1)
 		return;
 
+	if (!partially_mapped && !split_underused_thp)
+		return;
+
 	/*
 	 * The try_to_unmap() in page reclaim path might reach here too,
 	 * this may cause a race condition to corrupt deferred split queue.
Signature · 1 line
2.43.5