Skip to contentSkip to conversation

mm: skip mlocked THPs that are underused early in deferred_split_scan()

PATCH Applied Lance Yang 6 participants 13 messages Sep 8, 2025 – Sep 9, 2025

Subsystems (2)
lore
Share & tools
lore

Lance Yang

author reviewer
#0 lore raw

PATCH v2 1/1 mm: skip mlocked THPs that are underused early in deferred_split_scan()

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

---
v1 -> v2:
- Update subject and changelog (per David, thanks!)
- https://lore.kernel.org/linux-mm/[email protected]

 mm/huge_memory.c | 7 +++++++
 1 file changed, 7 insertions(+)
mm/huge_memory.c +7 -0
index 77f0c3417973..6d99893c18e8 100644
@@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 		bool underused = false;
 
 		if (!folio_test_partially_mapped(folio)) {
+			/*
+			 * See try_to_map_unused_to_zeropage(): we cannot
+			 * optimize zero-filled pages after splitting an
+			 * mlocked folio.
+			 */
+			if (folio_test_mlocked(folio))
+				goto next;
 			underused = thp_underused(folio);
 			if (!underused)
 				goto next;
Signature · 1 line
2.49.0
Acked-by on v2 1/1
#1 lore raw
Quoted 11 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

Feel free to drop that, I only tweaked you description :)

Signature · 3 lines
Cheers

David / dhildenb
author reviewer
#2 lore raw
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 6 lines
Quoted 4 lines

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

Feel free to drop that, I only tweaked you description :)

Well, I'd actually prefer to keep it. Thanks for setting me straight on try_to_map_unused_to_zeropage() ;p

Quoted 2 lines

Cheers,
Lance

#3 lore raw
In reply to Lance Yang
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#4 lore raw
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 7 lines

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

I assume if you're using mlock and at the same time enable KSM for a process/VMA, you're doing something wrong.

In contrast, THP is supposed to be transparent (yeah, I know ...).

Signature · 3 lines
Cheers

David / dhildenb
author reviewer
#9 lore raw
In reply to Kiryl Shutsemau
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 7 lines

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

Good catch!

Surprisingly, my test shows that KSM is indeed able to replace a zeroed mlocked THP with ZERO_PAGE(). And MADV_MERGEABLE allows a VMA to be marked as mergeable even if it's already locked(VM_LOCKED) ...

#5 lore raw
In reply to David Hildenbrand
Quoted 15 lines · David Hildenbrand
Quoted 14 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Quoted 6 lines
Quoted 1 line

Hm?

I assume if you're using mlock and at the same time enable KSM for a process/VMA, you're doing something wrong.

In contrast, THP is supposed to be transparent (yeah, I know ...).

Yeah, I guess it is user error.

Maybe we should make ksm_compatible() return false for VM_LOCKED? KSM breaks mlock() contract.

But it can be risky if someone already relies on this broken behaviour.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#6 lore raw
Quoted 21 lines · Kiryl Shutsemau
Quoted 20 lines · David Hildenbrand
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 7 lines

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

I assume if you're using mlock and at the same time enable KSM for a process/VMA, you're doing something wrong.

Quoted 6 lines
Quoted 1 line

In contrast, THP is supposed to be transparent (yeah, I know ...).

Yeah, I guess it is user error.

Maybe we should make ksm_compatible() return false for VM_LOCKED? KSM breaks mlock() contract.

I was thinking the same and falsely remembered that we would already be checking for that.

Quoted 2 lines

But it can be risky if someone already relies on this broken behaviour.

Could be.

Staring at QEMU, we have the following parameters:

mem-merge=on|off

Enables or disables memory merge support. This feature, when supported by the host, de-duplicates identical memory pages among VMs instances (enabled by default).

And

-overcommit mem-lock=on|off|on-fault

"Run qemu with hints about host resource overcommit. The default
is to assume that host overcommits all resources."

Now, I would assume that anybody who sets "-overcommit mem-lock=on" either

(a) Has KSM disabled on that machine.

(b) Sets mem-merge=off

as well. But QEMU would allow for configuring it.

Interestingly, mm_populate()->populate_vma_page_range() wants to break COW. [*]

But if the app later calls fork(), we still allow for cow-sharing pages with the child. (another case of "don't do it", like KSM I guess)

[*] it doesn't do it for mappings that start out R/O. I think we might end up with sharedzero pages in that case, but not sure if worth fixing.

Signature · 3 lines
Cheers

David / dhildenb
#7 lore raw
Quoted 53 lines · David Hildenbrand
Quoted 21 lines · Kiryl Shutsemau
Quoted 20 lines · David Hildenbrand
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 7 lines

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

I assume if you're using mlock and at the same time enable KSM for a process/VMA, you're doing something wrong.

Quoted 6 lines
Quoted 1 line

In contrast, THP is supposed to be transparent (yeah, I know ...).

Yeah, I guess it is user error.

Maybe we should make ksm_compatible() return false for VM_LOCKED? KSM breaks mlock() contract.

I was thinking the same and falsely remembered that we would already be checking for that.

Quoted 2 lines

But it can be risky if someone already relies on this broken behaviour.

Could be.

Staring at QEMU, we have the following parameters:

mem-merge=on|off

Enables or disables memory merge support. This feature, when supported by the host, de-duplicates identical memory pages among VMs instances (enabled by default).

And

-overcommit mem-lock=on|off|on-fault

"Run qemu with hints about host resource overcommit. The default
is to assume that host overcommits all resources."

Quoted 7 lines

Now, I would assume that anybody who sets "-overcommit mem-lock=on" either

(a) Has KSM disabled on that machine.

(b) Sets mem-merge=off

as well. But QEMU would allow for configuring it.

ksm_madvise(MADV_MERGEABLE) succeeds on !vma_ksm_compatible(), so it wouldn't be functional breakage, but may result in unexpected increase of memory consumption.

Quoted 5 lines

Interestingly, mm_populate()->populate_vma_page_range() wants to break COW. [*]

But if the app later calls fork(), we still allow for cow-sharing pages with the child. (another case of "don't do it", like KSM I guess)

CoW has bunch of these "don't do it". :P

Quoted 8 lines

[*] it doesn't do it for mappings that start out R/O. I think we might end up with sharedzero pages in that case, but not sure if worth fixing.

Signature · 3 lines
Cheers

David / dhildenb
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#8 lore raw
Quoted 54 lines · Kiryl Shutsemau
Quoted 53 lines · David Hildenbrand
Quoted 21 lines · Kiryl Shutsemau
Quoted 20 lines · David Hildenbrand
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 7 lines

It makes me think, does KSM follows the same logic as try_to_map_unused_to_zeropage()?

I cannot immediately find what prevents KSM from replacing zeroed mlocked folio with ZERO_PAGE().

Hm?

I assume if you're using mlock and at the same time enable KSM for a process/VMA, you're doing something wrong.

Quoted 6 lines
Quoted 1 line

In contrast, THP is supposed to be transparent (yeah, I know ...).

Yeah, I guess it is user error.

Maybe we should make ksm_compatible() return false for VM_LOCKED? KSM breaks mlock() contract.

I was thinking the same and falsely remembered that we would already be checking for that.

Quoted 2 lines

But it can be risky if someone already relies on this broken behaviour.

Could be.

Staring at QEMU, we have the following parameters:

mem-merge=on|off

Enables or disables memory merge support. This feature, when
supported by the host, de-duplicates identical memory pages among VMs instances (enabled by default).

And

-overcommit mem-lock=on|off|on-fault

"Run qemu with hints about host resource overcommit. The default
is to assume that host overcommits all resources."

Quoted 7 lines
Quoted 7 lines

Now, I would assume that anybody who sets "-overcommit mem-lock=on" either

(a) Has KSM disabled on that machine.

(b) Sets mem-merge=off

as well. But QEMU would allow for configuring it.

That's really interesting ;)

I guess it's likely that people are already relying on that behavior, even if it's flawed.

Quoted 4 lines

ksm_madvise(MADV_MERGEABLE) succeeds on !vma_ksm_compatible(), so it wouldn't be functional breakage, but may result in unexpected increase of memory consumption.

Hmm... it's hard to argue that nothing is broken.

An application losing the memory savings from KSM might not fit in memory at all and could be taken down by the OOM killer ;p

Quoted 18 lines
Quoted 6 lines

Interestingly, mm_populate()->populate_vma_page_range() wants to break COW. [*]

But if the app later calls fork(), we still allow for cow-sharing pages with the child. (another case of "don't do it", like KSM I guess)

CoW has bunch of these "don't do it". :P

Quoted 8 lines

[*] it doesn't do it for mappings that start out R/O. I think we might end up with sharedzero pages in that case, but not sure if worth fixing.

Signature · 3 lines
Cheers

David / dhildenb
Acked-by on v2 1/1
#10 lore raw
In reply to Lance Yang
Quoted 7 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 5 lines

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

Reviewed-by on v2 1/1
#11 lore raw
In reply to Lance Yang
Quoted 7 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

I see it's along this lovely code path: :)

deferred_split_scan()
(thp_underused() returns true)
-> split_folio()
-> split_folio_to_list()
-> split_huge_page_to_list_to_order()
-> __folio_split()
-> remap_page()
-> remove_migration_ptes()
-> remove_migration_pte()
-> try_to_map_unused_to_zeropage()
(oops mlock'd)

Quoted 5 lines

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

Anyway this does make a lot of sense! So:

Quoted 29 lines

---
v1 -> v2:
- Update subject and changelog (per David, thanks!)
- https://lore.kernel.org/linux-mm/[email protected]

 mm/huge_memory.c | 7 +++++++
 1 file changed, 7 insertions(+)
mm/huge_memory.c +7 -1
index 77f0c3417973..6d99893c18e8 100644
@@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
 		bool underused = false;
 
 		if (!folio_test_partially_mapped(folio)) {
+			/*
+			 * See try_to_map_unused_to_zeropage(): we cannot
+			 * optimize zero-filled pages after splitting an
+			 * mlocked folio.
+			 */
+			if (folio_test_mlocked(folio))
+				goto next;
 			underused = thp_underused(folio);
 			if (!underused)
 				goto next;
--

2.49.0

reviewer Reviewed-by on v2 1/1
#12 lore raw
In reply to Lance Yang
Quoted 7 lines · Lance Yang

From: Lance Yang <[email protected]>

When we stumble over a fully-mapped mlocked THP in the deferred shrinker, it does not make sense to try to detect whether it is underused, because try_to_map_unused_to_zeropage(), called while splitting the folio, will not actually replace any zeroed pages by the shared zeropage.

Quoted 6 lines

Splitting the folio in that case does not make any sense, so let's not even scan to check if the folio is underused.

---

LGTM.

Quoted 25 lines

v1 -> v2:
- Update subject and changelog (per David, thanks!)
- https://lore.kernel.org/linux-mm/[email protected]

  mm/huge_memory.c | 7 +++++++
  1 file changed, 7 insertions(+)
mm/huge_memory.c +7 -0
index 77f0c3417973..6d99893c18e8 100644
@@ -4183,6 +4183,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
  		bool underused = false;
 
  		if (!folio_test_partially_mapped(folio)) {
+			/*
+			 * See try_to_map_unused_to_zeropage(): we cannot
+			 * optimize zero-filled pages after splitting an
+			 * mlocked folio.
+			 */
+			if (folio_test_mlocked(folio))
+				goto next;
  			underused = thp_underused(folio);
  			if (!underused)
  				goto next;