mm: drop redundant lru_add_drain in anon folio reuse paths

PATCH In review Barry Song (Xiaomi) 6 participants 20 messages Jul 2 – Jul 8

Subsystems (2)

Memory management MM core

lore

Share & tools

lore

Barry Song (Xiaomi)

author

2026-07-02 00:00 UTC #0 lore raw

v3 cover mm: drop redundant lru_add_drain in anon folio reuse paths

We are doing a large number of redundant lru_add_drain() calls in both wp_can_reuse_anon_folio() and do_swap_page(), leading to LRU lock contention and unnecessary overhead.

In wp_can_reuse_anon_folio(), we can check the refcount against the lru_cache before deciding to drain. In do_swap_page(), the drain is now entirely redundant after Kairui's work to route SYNC I/O through the swapcache in the same way as ASYNC I/O.

Build the kernel within a 1 GB memcg using 20 threads with zRAM swap. The number of lru_add_drain() calls is reduced from 276,278 to 226,318, a reduction of about 18%.

Build the kernel within an 800 MB memcg using 20 threads with zRAM swap. The number of lru_add_drain() calls is reduced from 778,950 to 541,149, a reduction of 30.5%.

-v3:
* code improvements for patch1/4 and patch2/4 as suggested by David;
* improve changelog as suggested by Shakeel and David;
* collect the reviewed-by and acked-by tags;
* for patch4, remove all dependency for refcount regarding swapcache free in do_swap_page, as suggested by David

-v2:
https://lore.kernel.org/linux-mm/[email protected]/
* collect the reviewed-by and acked-by tags from Usama, Baoquan, Shakeel, Kairui, thanks!
* add patch4 to free swapcache for non-LRU folios, as suggested by Kairui, thanks!

-RFC:
https://lore.kernel.org/linux-mm/[email protected]/

Barry Song (Xiaomi) (4):
mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()
mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic
mm: entirely remove lru_add_drain in do_swap_page
mm: clarify the folio_free_swap() for do_swap_page()

 mm/memory.c | 49 ++++++++++++++++++-------------------------------
 1 file changed, 18 insertions(+), 31 deletions(-)

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author

2026-07-02 00:00 UTC #1 lore raw

PATCH v3 1/4 mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.

During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.

Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.

Reviewed-by Kairui Song <[email protected]>

Acked-by Shakeel Butt <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/memory.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

mm/memory.c +7 -3

index ff338c2abe92..87da78eb1abd 100644
@@ -4181,6 +4181,9 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 static bool wp_can_reuse_anon_folio(struct folio *folio,
 				    struct vm_area_struct *vma)
 {
+	const bool in_lru_cache = !folio_test_lru(folio);
+	const bool in_swapcache = folio_test_swapcache(folio);
+
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio))
 		return __wp_can_reuse_large_anon_folio(folio, vma);

@@ -4191,15 +4194,16 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	 *
 	 * KSM doesn't necessarily raise the folio refcount.
 	 */
-	if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
+	if (folio_test_ksm(folio) ||
+	    folio_ref_count(folio) > 1 + in_lru_cache + in_swapcache)
 		return false;
-	if (!folio_test_lru(folio))
+	if (in_lru_cache)
 		/*
 		 * We cannot easily detect+handle references from
 		 * remote LRU caches or references to LRU folios.
 		 */
 		lru_add_drain();
-	if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
+	if (folio_ref_count(folio) > 1 + in_swapcache)
 		return false;
 	if (!folio_trylock(folio))
 		return false;

Signature · 1 line

2.39.3 (Apple Git-146)

David Hildenbrand (Arm)

maintainer Acked-by on v3 1/4

2026-07-02 08:05 UTC #2 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Quoted 13 lines · Barry Song (Xiaomi)

There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.

Quoted 6 lines

Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---

The other folks should probably re-review this patch that changed quite a bit :)

Acked-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David

Barry Song

author

2026-07-06 08:18 UTC #3 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Quoted 21 lines · David Hildenbrand (Arm)

Quoted 13 lines · Barry Song (Xiaomi)
There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.
Quoted 6 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---

Quoted 4 lines

The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>

Thanks!

Hi Baoquan, Shakeel, and Kairui,

Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?

The concept hasn't changed since v2, but the code readability has improved quite a bit based on David's suggestions.

Best Regards
Barry

Baoquan He

2026-07-06 09:13 UTC #4 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Hi Barry,

Quoted 27 lines · Barry Song

Quoted 21 lines · David Hildenbrand (Arm)
Quoted 13 lines · Barry Song (Xiaomi)
There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.
Quoted 6 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
Quoted 4 lines
The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>

Quoted 7 lines

Thanks!
Hi Baoquan, Shakeel, and Kairui,
Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?

Is there change you need to do in a new version? If yes, please remove my tags, I will review v4. Oterwise, I will check this v3. Thanks.

AFAIK, usually, if patch is updated in a new version, any reviewing tags from the old version should be dropped unless the change is minor or trivial.

Thanks
Baoquan

Quoted 6 lines

The concept hasn't changed since v2, but the code readability has improved quite a bit based on David's suggestions.
Best Regards
Barry

Barry Song

author

2026-07-06 09:17 UTC #5 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Quoted 33 lines · Baoquan He

Hi Barry,
Quoted 29 lines · Barry Song
Quoted 21 lines · David Hildenbrand (Arm)
Quoted 13 lines · Barry Song (Xiaomi)
There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.
Quoted 6 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
Quoted 4 lines
The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>
Thanks!

Quoted 8 lines

Quoted 5 lines
Hi Baoquan, Shakeel, and Kairui,
Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?
Is there change you need to do in a new version? If yes, please remove my tags, I will review v4. Oterwise, I will check this v3. Thanks.

no change for v4. Please feel free to re-review v3.

Quoted 4 lines

AFAIK, usually, if patch is updated in a new version, any reviewing tags from the old version should be dropped unless the change is minor or trivial.

Thanks for the clarification. make sense to me.

Best Regards
Barry

Kairui Song

Reviewed-by

2026-07-07 15:27 UTC #6 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

In reply to Barry Song

Quoted 28 lines · Barry Song

Quoted 21 lines · David Hildenbrand (Arm)
Quoted 13 lines · Barry Song (Xiaomi)
There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.
Quoted 6 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
Quoted 4 lines
The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>

Quoted 7 lines

Thanks!
Hi Baoquan, Shakeel, and Kairui,
Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?

Sure, no problem.

The only complain from me, and a trivial one is that in_lru_cache = !folio_test_lru(folio) looks a bit odd. !folio_test_lru only means the folio is not on the lru. Could it be off the lru due to isolation or other reason? In this context it just indicates that we should try to flush lru cache and attempt release the ref. It's trivial, just hope people won't be confused by this. The code is correct and sound though, so:

Reviewed-by Kairui Song <[email protected]>

David Hildenbrand (Arm)

maintainer

2026-07-07 15:58 UTC #7 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Quoted 13 lines · Kairui Song

Quoted 12 lines · Barry Song
Quoted 5 lines · David Hildenbrand (Arm)
The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>
Thanks!
Hi Baoquan, Shakeel, and Kairui,

Quoted 8 lines

Quoted 3 lines
Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?
Sure, no problem.
The only complain from me, and a trivial one is that in_lru_cache = !folio_test_lru(folio) looks a bit odd.

I was debating with myself a bit, how to keep it short yet expressive.

"maybe_in_lru_cache" could work.

Signature · 3 lines

Cheers,

David

Barry Song (Xiaomi)

author

2026-07-08 14:57 UTC #8 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

Quoted 19 lines · David Hildenbrand (Arm)

Quoted 17 lines · Kairui Song
Quoted 15 lines · Barry Song
Quoted 5 lines · David Hildenbrand (Arm)
The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>
Thanks!
Hi Baoquan, Shakeel, and Kairui,
Would you like to re-review the patches so I can re-collect your tags? If not, are you okay with me removing your tags when I send v4?

Quoted 8 lines

Quoted 4 lines
Sure, no problem.
The only complain from me, and a trivial one is that in_lru_cache = !folio_test_lru(folio) looks a bit odd.
I was debating with myself a bit, how to keep it short yet expressive.
"maybe_in_lru_cache" could work.

yes. sounds very good to me. Thanks, Kairui and David.

Hi Andrew,

Would you like to squash the below change?

From 4ce3a8e95eaa58ca58291ac8c2986f00e0e4db12 Mon Sep 17 00:00:00 2001
From: "Barry Song (Xiaomi)" <[email protected]>
Date: Wed, 8 Jul 2026 22:46:43 +0800
Subject: [PATCH] mm: rename in_lru_cache to maybe_in_lru_cache

!folio_test_lru(folio) doesn't necessarily mean folio is in lru cache. so let's put "maybe" prefix.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/memory.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

mm/memory.c +3 -3

index 5689b7cff76c..631c170f812a 100644
@@ -4174,7 +4174,7 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 static bool wp_can_reuse_anon_folio(struct folio *folio,
 				    struct vm_area_struct *vma)
 {
-	const bool in_lru_cache = !folio_test_lru(folio);
+	const bool maybe_in_lru_cache = !folio_test_lru(folio);
 	const bool in_swapcache = folio_test_swapcache(folio);
 
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio))

@@ -4188,9 +4188,9 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	 * KSM doesn't necessarily raise the folio refcount.
 	 */
 	if (folio_test_ksm(folio) ||
-	    folio_ref_count(folio) > 1 + in_lru_cache + in_swapcache)
+	    folio_ref_count(folio) > 1 + maybe_in_lru_cache + in_swapcache)
 		return false;
-	if (in_lru_cache)
+	if (maybe_in_lru_cache)
 		/*
 		 * We cannot easily detect+handle references from
 		 * remote LRU caches or references to LRU folios.

Signature · 1 line

2.39.3 (Apple Git-146)

Quoted 5 lines

--
Cheers,
David

Shakeel Butt

2026-07-07 15:21 UTC #9 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

In reply to David Hildenbrand (Arm)

Quoted 20 lines · David Hildenbrand (Arm)

Quoted 13 lines · Barry Song (Xiaomi)
There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.
Quoted 6 lines
Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---

Quoted 4 lines

The other folks should probably re-review this patch that changed quite a bit :)
Acked-by David Hildenbrand (Arm) <[email protected]>

The code looks much better and Barry can keep my tag.

Baoquan He

Reviewed-by on v3 1/4

2026-07-07 12:55 UTC #10 lore raw

Re: [PATCH v3 1/4] mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()

In reply to Barry Song (Xiaomi)

Quoted 13 lines · Barry Song (Xiaomi)

There is a case where folio_ref_count(folio) == 3 and !folio_test_swapcache(folio). In that case, both folio_ref_count(folio) > 3 and
folio_ref_count(folio) > 1 + folio_test_swapcache(folio) evaluate false, causing an unnecessary local LRU drain.
During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains.
Fix this by checking against: 1 + in_swapcache + in_lrucache instead of hardcoding folio_ref_count(folio) > 3.

Quoted 8 lines

Suggested-by David Hildenbrand (Arm) <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/memory.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

Rechecked, LGTM,

Reviewed-by Baoquan He <[email protected]>

Quoted 38 lines

mm/memory.c +7 -3

index ff338c2abe92..87da78eb1abd 100644
@@ -4181,6 +4181,9 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio,
 static bool wp_can_reuse_anon_folio(struct folio *folio,
 				    struct vm_area_struct *vma)
 {
+	const bool in_lru_cache = !folio_test_lru(folio);
+	const bool in_swapcache = folio_test_swapcache(folio);
+
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio))
 		return __wp_can_reuse_large_anon_folio(folio, vma);

@@ -4191,15 +4194,16 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	 *
 	 * KSM doesn't necessarily raise the folio refcount.
 	 */
-	if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
+	if (folio_test_ksm(folio) ||
+	    folio_ref_count(folio) > 1 + in_lru_cache + in_swapcache)
 		return false;
-	if (!folio_test_lru(folio))
+	if (in_lru_cache)
 		/*
 		 * We cannot easily detect+handle references from
 		 * remote LRU caches or references to LRU folios.
 		 */
 		lru_add_drain();
-	if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
+	if (folio_ref_count(folio) > 1 + in_swapcache)
 		return false;
 	if (!folio_trylock(folio))
 		return false;

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author

2026-07-02 00:00 UTC #11 lore raw

PATCH v3 2/4 mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

In reply to Barry Song (Xiaomi)

The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly.

[1] https://lore.kernel.org/all/[email protected]/

Acked-by Usama Arif <[email protected]>

Reviewed-by Kairui Song <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Acked-by Shakeel Butt <[email protected]>

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/memory.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

mm/memory.c +3 -6

index 87da78eb1abd..71e9d394816b 100644
@@ -5046,13 +5046,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		pte = pte_mkuffd_wp(pte);
 
 	/*
-	 * Same logic as in do_wp_page(); however, optimize for pages that are
-	 * certainly not shared either because we just allocated them without
-	 * exposing them to the swapcache or because the swap entry indicates
-	 * exclusivity.
+	 * Similar logic as in do_wp_page(); however, optimize for pages that are
+	 * certainly not because the swap entry indicates exclusivity.
 	 */
-	if (!folio_test_ksm(folio) &&
-	    (exclusive || folio_ref_count(folio) == 1)) {
+	if (exclusive) {
 		if ((vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) &&
 		    !pte_needs_soft_dirty_wp(vma, pte)) {
 			pte = pte_mkwrite(pte, vma);

Signature · 1 line

2.39.3 (Apple Git-146)

David Hildenbrand (Arm)

maintainer Acked-by on v3 2/4

2026-07-02 08:08 UTC #12 lore raw

Re: [PATCH v3 2/4] mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

Quoted 8 lines · Barry Song (Xiaomi)

The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
Acked-by Usama Arif <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---

Acked-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David

David Hildenbrand (Arm)

maintainer

2026-07-02 08:14 UTC #13 lore raw

Re: [PATCH v3 2/4] mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

In reply to Barry Song (Xiaomi)

Quoted 7 lines · Barry Song (Xiaomi)

The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly.

Sashiko points out two minor things (one flagged as medium, lol, sure sure).

Here, you can clarify that folio_ref_count()==1 is true for freshly allocated pages (due to the KSM check) in which case exclusive=true already.

Quoted 17 lines

[1] https://lore.kernel.org/all/[email protected]/
Acked-by Usama Arif <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/memory.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
mm/memory.c +0 -0
index 87da78eb1abd..71e9d394816b 100644

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 5050

Quoted 9 lines

mm/memory.c +2 -4

@@ -5046,13 +5046,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		pte = pte_mkuffd_wp(pte);
 
 	/*
-	 * Same logic as in do_wp_page(); however, optimize for pages that are
-	 * certainly not shared either because we just allocated them without
-	 * exposing them to the swapcache or because the swap entry indicates
-	 * exclusivity.
+	 * Similar logic as in do_wp_page(); however, optimize for pages that are
+	 * certainly not because the swap entry indicates exclusivity.

s/not/not shared/

but likely you should just simplify to "... pages that are certainly exclusive."

Quoted 7 lines

mm/memory.c +1 -2

 	 */
-	if (!folio_test_ksm(folio) &&
-	    (exclusive || folio_ref_count(folio) == 1)) {
+	if (exclusive) {
 		if ((vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) &&
 		    !pte_needs_soft_dirty_wp(vma, pte)) {
 			pte = pte_mkwrite(pte, vma);

Signature · 3 lines

Cheers,

David

Barry Song

author

2026-07-06 08:14 UTC #14 lore raw

Re: [PATCH v3 2/4] mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

Quoted 10 lines · David Hildenbrand (Arm)

Quoted 7 lines · Barry Song (Xiaomi)
The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly.

Quoted 4 lines

Sashiko points out two minor things (one flagged as medium, lol, sure sure).
Here, you can clarify that folio_ref_count()==1 is true for freshly allocated pages (due to the KSM check) in which case exclusive=true already.

Thanks. Sashiko's comment below seems valid. While there's no need to change the code, the changelog should be updated to reflect it. I'll do that in v4.

"Is the claim that folio_ref_count() should never be 1 in this path accurate? When ksm_might_need_to_copy() allocates a fresh folio to break KSM sharing, the new folio bypasses the swap cache:
do_swap_page() {
...
if (!folio_test_ksm(folio)) {
...
if (folio != swapcache) {
/*
* We have a fresh page that is not exposed to the
* swapcache -> certainly exclusive.
*/
exclusive = true;
}
...
}
In this specific case, the folio refcount would still be exactly 1 when it reaches this path.
While the code change itself is safe because the KSM copy path explicitly sets exclusive = true earlier (satisfying the new if (exclusive) check without needing the refcount check), the rationale in the commit message seems to overlook this scenario."

Quoted 30 lines

Quoted 18 lines
[1] https://lore.kernel.org/all/[email protected]/
Acked-by Usama Arif <[email protected]>
Reviewed-by Kairui Song <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Acked-by Shakeel Butt <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/memory.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
mm/memory.c +0 -0
index 87da78eb1abd..71e9d394816b 100644
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 5050
Quoted 9 lines
mm/memory.c +2 -4
@@ -5046,13 +5046,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
              pte = pte_mkuffd_wp(pte);
 
      /*
-      * Same logic as in do_wp_page(); however, optimize for pages that are
-      * certainly not shared either because we just allocated them without
-      * exposing them to the swapcache or because the swap entry indicates
-      * exclusivity.
+      * Similar logic as in do_wp_page(); however, optimize for pages that are
+      * certainly not because the swap entry indicates exclusivity.
s/not/not shared/

Quoted 1 line

but likely you should just simplify to "... pages that are certainly exclusive."

Thanks. I will fix the doc in v4.

Best Regards
Barry

Barry Song

author

2026-07-07 11:08 UTC #15 lore raw

Re: [PATCH v3 2/4] mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

In reply to David Hildenbrand (Arm)

Quoted 10 lines · David Hildenbrand (Arm)

Quoted 7 lines · Barry Song (Xiaomi)
The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly.

Quoted 4 lines

Sashiko points out two minor things (one flagged as medium, lol, sure sure).
Here, you can clarify that folio_ref_count()==1 is true for freshly allocated pages (due to the KSM check) in which case exclusive=true already.

Hi Andrew,

I saw that you've queued this in mm-new. Thanks very much for that!

Would you mind adding the following sentences to the end of the changelog?

The ksm_might_need_to_copy() check may allocate a fresh folio with folio_ref_count() == 1. Along that path, exclusive has already been set to true, so the folio can still be reused correctly.

Best Regards
Barry

Andrew Morton

maintainer

2026-07-07 20:18 UTC #16 lore raw

Re: [PATCH v3 2/4] mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic

Quoted 7 lines · On Tue, 7 Jul 2026 19:08:13 +0800 Barry Song <[email protected]> wrote:

Quoted 4 lines
Sashiko points out two minor things (one flagged as medium, lol, sure sure).
Here, you can clarify that folio_ref_count()==1 is true for freshly allocated pages (due to the KSM check) in which case exclusive=true already.
Hi Andrew,

Quoted 7 lines

I saw that you've queued this in mm-new. Thanks very much for that!
Would you mind adding the following sentences to the end of the changelog?
The ksm_might_need_to_copy() check may allocate a fresh folio with folio_ref_count() == 1. Along that path, exclusive has already been set to true, so the folio can still be reused correctly.

Done, thanks.

Barry Song (Xiaomi)

author

2026-07-02 00:00 UTC #17 lore raw

PATCH v3 3/4 mm: entirely remove lru_add_drain in do_swap_page

In reply to Barry Song (Xiaomi)

We are doing a lot of redundant lru_add_drain() calls in do_swap_page(), especially for synchronous I/O devices. For example, the test program below currently ends up draining lru_cache 100% of the time:

int main(int argc, char *argv[])

{
        int i;
 #define SIZE 100*1024*1024
	while(1) {
		volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

		for (int i = 0; i < SIZE/sizeof(int); i++)
			p[i] =  i%64;
		madvise((void *)p, SIZE, MADV_PAGEOUT);
		for (int i = 0; i < SIZE/sizeof(int); i++)
			p[i] =  i%64;
		munmap(p, SIZE);
	}
	return 0;
}

Folio reuse now relies primarily on the exclusive hint, making lru_cache draining to drop the refcount in lru_cache largely irrelevant.
For a kernel build with a minimal configuration running in a 1 GB memcg, this patch skips more than 43,000 redundant local LRU drains.

Acked-by David Hildenbrand (Arm) <[email protected]>

Acked-by Shakeel Butt <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/memory.c | 10 ----------
 1 file changed, 10 deletions(-)

mm/memory.c +0 ~~-10~~

index 71e9d394816b..4665405ace5a 100644
@@ -4901,16 +4901,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	} else if (folio != swapcache)
 		page = folio_page(folio, 0);
 
-	/*
-	 * If we want to map a page that's in the swapcache writable, we
-	 * have to detect via the refcount if we're really the exclusive
-	 * owner. Try removing the extra reference from the local LRU
-	 * caches if required.
-	 */
-	if ((vmf->flags & FAULT_FLAG_WRITE) &&
-	    !folio_test_ksm(folio) && !folio_test_lru(folio))
-		lru_add_drain();
-
 	folio_throttle_swaprate(folio, GFP_KERNEL);
 
 	/*

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author

2026-07-02 00:00 UTC #18 lore raw

PATCH v3 4/4 mm: clarify the folio_free_swap() for do_swap_page()

In reply to Barry Song (Xiaomi)

Since commit 4b34f1d82c654 ("mm, swap: free the swap cache after folio is mapped"), we have relied on do_wp_page() to handle the non-exclusive case, where the folio may either be reused or require CoW.

As a result, using the refcount in do_swap_page() to decide when to free the swap cache is no longer necessary, since do_wp_page() can handle this more cleanly and consistently.

We can now simply use FAULT_FLAG_WRITE together with exclusivity to decide when to free the swap cache in do_swap_page().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/memory.c | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

mm/memory.c +8 ~~-12~~

index 4665405ace5a..a29cd38ad547 100644
@@ -4516,7 +4516,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
 static inline bool should_try_to_free_swap(struct swap_info_struct *si,
 					   struct folio *folio,
 					   struct vm_area_struct *vma,
-					   unsigned int extra_refs,
+					   bool exclusive,
 					   unsigned int fault_flags)
 {
 	if (!folio_test_swapcache(folio))

@@ -4532,14 +4532,12 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
 	if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
 	    folio_test_mlocked(folio))
 		return true;
+
 	/*
-	 * If we want to map a page that's in the swapcache writable, we
-	 * have to detect via the refcount if we're really the exclusive
-	 * user. Try freeing the swapcache to get rid of the swapcache
-	 * reference only in case it's likely that we'll be the exclusive user.
+	 * Free the swapcache only if we are the exclusive user and
+	 * this is a write fault.
 	 */
-	return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) &&
-		folio_ref_count(folio) == (extra_refs + folio_nr_pages(folio));
+	return (fault_flags & FAULT_FLAG_WRITE) && exclusive;
 }
 
 static vm_fault_t pte_marker_clear(struct vm_fault *vmf)

@@ -5043,10 +5041,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		if ((vma->vm_flags & VM_WRITE) && !userfaultfd_pte_wp(vma, pte) &&
 		    !pte_needs_soft_dirty_wp(vma, pte)) {
 			pte = pte_mkwrite(pte, vma);
-			if (vmf->flags & FAULT_FLAG_WRITE) {
+			if (vmf->flags & FAULT_FLAG_WRITE)
 				pte = pte_mkdirty(pte);
-				vmf->flags &= ~FAULT_FLAG_WRITE;
-			}
 		}
 		rmap_flags |= RMAP_EXCLUSIVE;
 	}

@@ -5086,7 +5082,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	 * Do it after mapping, so raced page faults will likely see the folio
 	 * in swap cache and wait on the folio lock.
 	 */
-	if (should_try_to_free_swap(si, folio, vma, nr_pages, vmf->flags))
+	if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags))
 		folio_free_swap(folio);
 
 	folio_unlock(folio);

@@ -5103,7 +5099,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		folio_put(swapcache);
 	}
 
-	if (vmf->flags & FAULT_FLAG_WRITE) {
+	if ((vmf->flags & FAULT_FLAG_WRITE) && !pte_write(pte)) {
 		ret |= do_wp_page(vmf);
 		if (ret & VM_FAULT_ERROR)
 			ret &= VM_FAULT_ERROR;

Signature · 1 line

2.39.3 (Apple Git-146)

David Hildenbrand (Arm)

maintainer Acked-by on v3 4/4

2026-07-02 08:10 UTC #19 lore raw

Re: [PATCH v3 4/4] mm: clarify the folio_free_swap() for do_swap_page()

Quoted 9 lines · Barry Song (Xiaomi)

Since commit 4b34f1d82c654 ("mm, swap: free the swap cache after folio is mapped"), we have relied on do_wp_page() to handle the non-exclusive case, where the folio may either be reused or require CoW.
As a result, using the refcount in do_swap_page() to decide when to free the swap cache is no longer necessary, since do_wp_page() can handle this more cleanly and consistently.

Quoted 6 lines

We can now simply use FAULT_FLAG_WRITE together with exclusivity to decide when to free the swap cache in do_swap_page().
Suggested-by David Hildenbrand (Arm) <[email protected]>
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---

Acked-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David