FAILED: patch "[PATCH] mm/gup: check ref_count instead of lru before migration" failed to apply to 6.6-stable tree

PATCH [email protected] 2 participants 2 messages Sep 21, 2025 – Sep 24, 2025

Subsystems (6)

Stable branch Memory management MM core Memory mapping +2 more subsystems

lore

Share & tools

lore

[email protected]

author maintainer

2025-09-21 12:51 UTC #0 lore raw

The patch below does not apply to the 6.6-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.6.y
git checkout FETCH_HEAD
git cherry-pick -x 98c6d259319ecf6e8d027abd3f14b81324b8c0ad
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2025092111-hedging-brunch-bdeb@gregkh' --subject-prefix 'PATCH 6.6.y' HEAD^..

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 98c6d259319ecf6e8d027abd3f14b81324b8c0ad Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Mon, 8 Sep 2025 15:15:03 -0700
Subject: [PATCH] mm/gup: check ref_count instead of lru before migration

Patch series "mm: better GUP pin lru_add_drain_all()", v2.

Series of lru_add_drain_all()-related patches, arising from recent mm/gup migration report from Will Deacon.

This patch (of 5):

Will Deacon reports:-

When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration.

It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.

In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table.

Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated.

This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.

Hugh Dickins adds:-

!folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.

5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.

And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.

Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.

Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation: but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all(). And this makes easy the backport to earlier releases: up to and including 6.12, btrfs also used PG_private_2, but without a ref_count increment.

Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation").

Link https://lkml.kernel.org/r/[email protected] open here

Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")

Signed-off-by Hugh Dickins <[email protected]>

Reported-by Will Deacon <[email protected]>

Closes https://lore.kernel.org/linux-mm/[email protected]/ open here

Acked-by Kiryl Shutsemau <[email protected]>

Acked-by David Hildenbrand <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Axel Rasmussen <[email protected]>

Cc Chris Li <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc John Hubbard <[email protected]>

Cc Keir Fraser <[email protected]>

Cc Konstantin Khlebnikov <[email protected]>

Cc Li Zhe <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Peter Xu <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Shivank Garg <[email protected]>

Cc Vlastimil Babka <[email protected]>

Cc Wei Xu <[email protected]>

Cc yangge <[email protected]>

Cc Yuanchu Xie <[email protected]>

Cc Yu Zhao <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

mm/gup.c +2 -1

index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(
 			continue;
 		}
 
-		if (!folio_test_lru(folio) && drain_allow) {
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
 			drain_allow = false;
 		}

Hugh Dickins

Reviewed-by John Hubbard Acked-by Kiryl Shutsemau Acked-by David Hildenbrand Acked-by David Hildenbrand Acked-by David Hildenbrand

2025-09-24 07:46 UTC #1 lore raw

Quoted 147 lines · On Sun, 21 Sep 2025, [email protected] wrote:

The patch below does not apply to the 6.6-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.6.y
git checkout FETCH_HEAD
git cherry-pick -x 98c6d259319ecf6e8d027abd3f14b81324b8c0ad
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2025092111-hedging-brunch-bdeb@gregkh' --subject-prefix 'PATCH 6.6.y' HEAD^..
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 98c6d259319ecf6e8d027abd3f14b81324b8c0ad Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Mon, 8 Sep 2025 15:15:03 -0700
Subject: [PATCH] mm/gup: check ref_count instead of lru before migration
Patch series "mm: better GUP pin lru_add_drain_all()", v2.
Series of lru_add_drain_all()-related patches, arising from recent mm/gup migration report from Will Deacon.
This patch (of 5):
Will Deacon reports:-
When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration.
It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.
In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table.
Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated.
This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.
Hugh Dickins adds:-
!folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.
5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.
And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.
Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.
Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation: but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all(). And this makes easy the backport to earlier releases: up to and including 6.12, btrfs also used PG_private_2, but without a ref_count increment.
Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation").
Link https://lkml.kernel.org/r/[email protected] open here
Link https://lkml.kernel.org/r/[email protected] open here
Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")
Signed-off-by Hugh Dickins <[email protected]>
Reported-by Will Deacon <[email protected]>
Closes https://lore.kernel.org/linux-mm/[email protected]/ open here
Acked-by Kiryl Shutsemau <[email protected]>
Acked-by David Hildenbrand <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Axel Rasmussen <[email protected]>
Cc Chris Li <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc John Hubbard <[email protected]>
Cc Keir Fraser <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Li Zhe <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Peter Xu <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Shivank Garg <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Wei Xu <[email protected]>
Cc yangge <[email protected]>
Cc Yuanchu Xie <[email protected]>
Cc Yu Zhao <[email protected]>
Cc <[email protected]>
Signed-off-by Andrew Morton <[email protected]>
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 9 lines

mm/gup.c +2 -1

@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(
 			continue;
 		}
 
-		if (!folio_test_lru(folio) && drain_allow) {
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
 			drain_allow = false;
 		}

Five gup/lrudrain backport patches against 6.6.108-rc1 attached.

From a6ec8a1f2b5b039aee80571767e927ff993ec70f Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Wed, 11 Jun 2025 15:13:14 +0200
Subject: [PATCH 1/5] mm/gup: revert "mm: gup: fix infinite loop within
__get_longterm_locked"

[ Upstream commit 517f496e1e61bd169d585dab4dd77e7147506322 ]

After commit 1aaf8c122918 ("mm: gup: fix infinite loop within __get_longterm_locked") we are able to longterm pin folios that are not supposed to get longterm pinned, simply because they temporarily have the LRU flag cleared (esp. temporarily isolated).

For example, two __get_longterm_locked() callers can race, or __get_longterm_locked() can race with anything else that temporarily isolates folios.

The introducing commit mentions the use case of a driver that uses vm_ops->fault to insert pages allocated through cma_alloc() into the page tables, assuming they can later get longterm pinned. These pages/ folios would never have the LRU flag set and consequently cannot get isolated. There is no known in-tree user making use of that so far, fortunately.

To handle that in the future -- and avoid retrying forever to isolate/migrate them -- we will need a different mechanism for the CMA area *owner* to indicate that it actually already allocated the page and is fine with longterm pinning it. The LRU flag is not suitable for that.

Probably we can lookup the relevant CMA area and query the bitmap; we only have have to care about some races, probably. If already allocated, we could just allow longterm pinning)

Anyhow, let's fix the "must not be longterm pinned" problem first by reverting the original commit.

Link https://lkml.kernel.org/r/[email protected] open here

Fixes 1aaf8c122918 ("mm: gup: fix infinite loop within __get_longterm_locked")

Signed-off-by David Hildenbrand <[email protected]>

Closes https://lore.kernel.org/all/20250522092755.GA3277597@tiffany/ open here

Reported-by Hyesoo Yu <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Peter Xu <[email protected]>

Cc Zhaoyang Huang <[email protected]>

Cc Aijun Sun <[email protected]>

Cc Alistair Popple <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

[ Revert v6.6.79 commit 933b08c0edfa ]

Signed-off-by Hugh Dickins <[email protected]>

---
 mm/gup.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

mm/gup.c +10 -4

index 29c719b3ab31..497d7ce43d39 100644
@@ -1946,14 +1946,14 @@ struct page *get_dump_page(unsigned long addr)
 /*
  * Returns the number of collected pages. Return value is always >= 0.
  */
-static void collect_longterm_unpinnable_pages(
+static unsigned long collect_longterm_unpinnable_pages(
 					struct list_head *movable_page_list,
 					unsigned long nr_pages,
 					struct page **pages)
 {
+	unsigned long i, collected = 0;
 	struct folio *prev_folio = NULL;
 	bool drain_allow = true;
-	unsigned long i;
 
 	for (i = 0; i < nr_pages; i++) {
 		struct folio *folio = page_folio(pages[i]);

@@ -1965,6 +1965,8 @@ static void collect_longterm_unpinnable_pages(
 		if (folio_is_longterm_pinnable(folio))
 			continue;
 
+		collected++;
+
 		if (folio_is_device_coherent(folio))
 			continue;

@@ -1986,6 +1988,8 @@ static void collect_longterm_unpinnable_pages(
 				    NR_ISOLATED_ANON + folio_is_file_lru(folio),
 				    folio_nr_pages(folio));
 	}
+
+	return collected;
 }
 
 /*

@@ -2078,10 +2082,12 @@ static int migrate_longterm_unpinnable_pages(
 static long check_and_migrate_movable_pages(unsigned long nr_pages,
 					    struct page **pages)
 {
+	unsigned long collected;
 	LIST_HEAD(movable_page_list);
 
-	collect_longterm_unpinnable_pages(&movable_page_list, nr_pages, pages);
-	if (list_empty(&movable_page_list))
+	collected = collect_longterm_unpinnable_pages(&movable_page_list,
+						nr_pages, pages);
+	if (!collected)
 		return 0;
 
 	return migrate_longterm_unpinnable_pages(&movable_page_list, nr_pages,

Signature · 1 line

2.51.0.534.gc79095c0ca-goog

From 26dc7668635eeabab84a36c97f00e399387ff0df Mon Sep 17 00:00:00 2001
From: Shivank Garg <[email protected]>
Date: Wed, 30 Apr 2025 10:01:51 +0000
Subject: [PATCH 2/5] mm: add folio_expected_ref_count() for reference count
calculation

[ Upstream commit 86ebd50224c0734d965843260d0dc057a9431c61 ]

Patch series " JFS: Implement migrate_folio for jfs_metapage_aops" v5.

This patchset addresses a warning that occurs during memory compaction due to JFS's missing migrate_folio operation. The warning was introduced by commit 7ee3647243e5 ("migrate: Remove call to ->writepage") which added explicit warnings when filesystem don't implement migrate_folio.

The syzbot reported following [1]:
jfs_metapage_aops does not implement migrate_folio

  WARNING: CPU: 1 PID: 5861 at mm/migrate.c:955 fallback_migrate_folio mm/migrate.c:953 [inline]
  WARNING: CPU: 1 PID: 5861 at mm/migrate.c:955 move_to_new_folio+0x70e/0x840 mm/migrate.c:1007
  Modules linked in:
  CPU: 1 UID: 0 PID: 5861 Comm: syz-executor280 Not tainted 6.15.0-rc1-next-20250411-syzkaller #0 PREEMPT(full)
  Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
  RIP: 0010:fallback_migrate_folio mm/migrate.c:953 [inline]
  RIP: 0010:move_to_new_folio+0x70e/0x840 mm/migrate.c:1007

To fix this issue, this series implement metapage_migrate_folio() for JFS which handles both single and multiple metapages per page configurations.

While most filesystems leverage existing migration implementations like filemap_migrate_folio(), buffer_migrate_folio_norefs() or buffer_migrate_folio() (which internally used folio_expected_refs()), JFS's metapage architecture requires special handling of its private data during migration. To support this, this series introduce the folio_expected_ref_count(), which calculates external references to a folio from page/swap cache, private data, and page table mappings.

This standardized implementation replaces the previous ad-hoc folio_expected_refs() function and enables JFS to accurately determine whether a folio has unexpected references before attempting migration.

Implement folio_expected_ref_count() to calculate expected folio reference counts from:
- Page/swap cache (1 per page)
- Private data (1)
- Page table mappings (1 per map)

While originally needed for page migration operations, this improved implementation standardizes reference counting by consolidating all refcount contributors into a single, reusable function that can benefit any subsystem needing to detect unexpected references to folios.

The folio_expected_ref_count() returns the sum of these external references without including any reference the caller itself might hold. Callers comparing against the actual folio_ref_count() must account for their own references separately.

Link https://syzkaller.appspot.com/bug?extid=8bb6fd945af4e0ad9299 [1]

Link https://lkml.kernel.org/r/[email protected] open here

Signed-off-by David Hildenbrand <[email protected]>

Signed-off-by Shivank Garg <[email protected]>

Co-developed-by David Hildenbrand <[email protected]>

Cc Alistair Popple <[email protected]>

Cc Dave Kleikamp <[email protected]>

Cc Donet Tom <[email protected]>

Cc Jane Chu <[email protected]>

Cc Kefeng Wang <[email protected]>

Cc Zi Yan <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Stable-dep-of: 98c6d259319e ("mm/gup: check ref_count instead of lru before migration") [ Take the new function in mm.h, removing "const" from its parameter to stop
build warnings; but avoid all the conflicts of using it in mm/migrate.c. ]

Signed-off-by Hugh Dickins <[email protected]>

---
 include/linux/mm.h | 55 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

include/linux/mm.h +55 -0

61 lines not on this page · raw message

Signature · 1 line

2.51.0.534.gc79095c0ca-goog

From 64467b0b9ff7ba2cddc4855aa8726afdafbc1444 Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Mon, 8 Sep 2025 15:15:03 -0700
Subject: [PATCH 3/5] mm/gup: check ref_count instead of lru before migration

[ Upstream commit 98c6d259319ecf6e8d027abd3f14b81324b8c0ad ]

Patch series "mm: better GUP pin lru_add_drain_all()", v2.

Series of lru_add_drain_all()-related patches, arising from recent mm/gup migration report from Will Deacon.

This patch (of 5):

Will Deacon reports:-

Hugh Dickins adds:-

Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation").

Link https://lkml.kernel.org/r/[email protected] open here

Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")

Signed-off-by Hugh Dickins <[email protected]>

Reported-by Will Deacon <[email protected]>

Closes https://lore.kernel.org/linux-mm/[email protected]/ open here

Acked-by Kiryl Shutsemau <[email protected]>

Acked-by David Hildenbrand <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Axel Rasmussen <[email protected]>

Cc Chris Li <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc John Hubbard <[email protected]>

Cc Keir Fraser <[email protected]>

Cc Konstantin Khlebnikov <[email protected]>

Cc Li Zhe <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Peter Xu <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Shivank Garg <[email protected]>

Cc Vlastimil Babka <[email protected]>

Cc Wei Xu <[email protected]>

Cc yangge <[email protected]>

Cc Yuanchu Xie <[email protected]>

Cc Yu Zhao <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

[ Clean cherry-pick now into this tree ]

Signed-off-by Hugh Dickins <[email protected]>

---
 mm/gup.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/gup.c +2 -1

index 497d7ce43d39..00ac2df7164c 100644
@@ -1975,7 +1975,8 @@ static unsigned long collect_longterm_unpinnable_pages(
 			continue;
 		}
 
-		if (!folio_test_lru(folio) && drain_allow) {
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
 			drain_allow = false;
 		}

Signature · 1 line

2.51.0.534.gc79095c0ca-goog

From 20de1b47cf004937b16bd364936a7dc934c3136a Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Mon, 8 Sep 2025 15:16:53 -0700
Subject: [PATCH 4/5] mm/gup: local lru_add_drain() to avoid
lru_add_drain_all()

[ Upstream commit a09a8a1fbb374e0053b97306da9dbc05bd384685 ]

In many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all().

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean
backports: can take 6.16 commit a03db236aebf ("gup: optimize longterm pin_user_pages() for large folio") first.

Link https://lkml.kernel.org/r/[email protected] open here

Signed-off-by Hugh Dickins <[email protected]>

Acked-by David Hildenbrand <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Axel Rasmussen <[email protected]>

Cc Chris Li <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc John Hubbard <[email protected]>

Cc Keir Fraser <[email protected]>

Cc Konstantin Khlebnikov <[email protected]>

Cc Li Zhe <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Peter Xu <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Shivank Garg <[email protected]>

Cc Vlastimil Babka <[email protected]>

Cc Wei Xu <[email protected]>

Cc Will Deacon <[email protected]>

Cc yangge <[email protected]>

Cc Yuanchu Xie <[email protected]>

Cc Yu Zhao <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

[ Resolved minor conflicts ]

Signed-off-by Hugh Dickins <[email protected]>

---
 mm/gup.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

mm/gup.c +11 -4

index 00ac2df7164c..5be764395e04 100644
@@ -1953,7 +1953,7 @@ static unsigned long collect_longterm_unpinnable_pages(
 {
 	unsigned long i, collected = 0;
 	struct folio *prev_folio = NULL;
-	bool drain_allow = true;
+	int drained = 0;
 
 	for (i = 0; i < nr_pages; i++) {
 		struct folio *folio = page_folio(pages[i]);

@@ -1975,10 +1975,17 @@ static unsigned long collect_longterm_unpinnable_pages(
 			continue;
 		}
 
-		if (drain_allow && folio_ref_count(folio) !=
-				   folio_expected_ref_count(folio) + 1) {
+		if (drained == 0 &&
+				folio_ref_count(folio) !=
+				folio_expected_ref_count(folio) + 1) {
+			lru_add_drain();
+			drained = 1;
+		}
+		if (drained == 1 &&
+				folio_ref_count(folio) !=
+				folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
-			drain_allow = false;
+			drained = 2;
 		}
 
 		if (!folio_isolate_lru(folio))

Signature · 1 line

2.51.0.534.gc79095c0ca-goog

From e390f5e36ddeefc66d2a33985549c1f00f8c18d8 Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Mon, 8 Sep 2025 15:23:15 -0700
Subject: [PATCH 5/5] mm: folio_may_be_lru_cached() unless folio_test_large()

[ Upstream commit 2da6de30e60dd9bb14600eff1cc99df2fa2ddae3 ]

mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain[_all]() on a large folio.

But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).

So ask if folio_may_be_lru_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch: it's just the per-CPU LRU caches, drained much later, which need care.

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".

Link https://lkml.kernel.org/r/[email protected] open here

Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")

Signed-off-by Hugh Dickins <[email protected]>

Acked-by David Hildenbrand <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Axel Rasmussen <[email protected]>

Cc Chris Li <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc John Hubbard <[email protected]>

Cc Keir Fraser <[email protected]>

Cc Konstantin Khlebnikov <[email protected]>

Cc Li Zhe <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Peter Xu <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Shivank Garg <[email protected]>

Cc Vlastimil Babka <[email protected]>

Cc Wei Xu <[email protected]>

Cc Will Deacon <[email protected]>

Cc yangge <[email protected]>

Cc Yuanchu Xie <[email protected]>

Cc Yu Zhao <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

[ Resolved conflicts in mm/swap.c ]

Signed-off-by Hugh Dickins <[email protected]>

---
 include/linux/swap.h | 10 ++++++++++
 mm/gup.c             |  4 ++--
 mm/mlock.c           |  6 +++---
 mm/swap.c            |  4 ++--
 4 files changed, 17 insertions(+), 7 deletions(-)

include/linux/swap.h +10 -0

index cb25db2a93dd..d7a5b7817987 100644
@@ -375,6 +375,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
 
+static inline bool folio_may_be_lru_cached(struct folio *folio)
+{
+	/*
+	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
+	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
+	 * will be sensible, but nobody has implemented and tested that yet.
+	 */
+	return !folio_test_large(folio);
+}
+
 extern atomic_t lru_disable_count;
 
 static inline bool lru_cache_disabled(void)

mm/gup.c +2 -2

index 5be764395e04..53154b63295a 100644
@@ -1975,13 +1975,13 @@ static unsigned long collect_longterm_unpinnable_pages(
 			continue;
 		}
 
-		if (drained == 0 &&
+		if (drained == 0 && folio_may_be_lru_cached(folio) &&
 				folio_ref_count(folio) !=
 				folio_expected_ref_count(folio) + 1) {
 			lru_add_drain();
 			drained = 1;
 		}
-		if (drained == 1 &&
+		if (drained == 1 && folio_may_be_lru_cached(folio) &&
 				folio_ref_count(folio) !=
 				folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();

mm/mlock.c +3 -3

index 06bdfab83b58..6858095c20dd 100644
@@ -256,7 +256,7 @@ void mlock_folio(struct folio *folio)
 
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }

@@ -279,7 +279,7 @@ void mlock_new_folio(struct folio *folio)
 
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, mlock_new(folio)) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }

@@ -300,7 +300,7 @@ void munlock_folio(struct folio *folio)
 	 */
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, folio) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }

mm/swap.c +2 -2

index 42082eba42de..8fde1a27aa48 100644
@@ -220,8 +220,8 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 static void folio_batch_add_and_move(struct folio_batch *fbatch,
 		struct folio *folio, move_fn_t move_fn)
 {
-	if (folio_batch_add(fbatch, folio) && !folio_test_large(folio) &&
-	    !lru_cache_disabled())
+	if (folio_batch_add(fbatch, folio) &&
+	    folio_may_be_lru_cached(folio) && !lru_cache_disabled())
 		return;
 	folio_batch_move_lru(fbatch, move_fn);
 }

Signature · 1 line

2.51.0.534.gc79095c0ca-goog