Skip to contentSkip to conversation

mm: better GUP pin lru_add_drain_all()

PATCH Superseded Hugh Dickins 3 participants 30 messages Aug 31, 2025 – Sep 8, 2025

Subsystems (7)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover mm: better GUP pin lru_add_drain_all()

Series of lru_add_drain_all()-related patches, arising from recent mm/gup migration report from Will Deacon. Based on 6.17-rc3 but apply to latest mm.git. I suggest all but 7/7 be hotfixes going to 6.17 and stable, but you might not be persuaded by "counter the increase" ones.

1/7 mm: fix folio_expected_ref_count() when PG_private_2 2/7 mm/gup: check ref_count instead of lru before migration 3/7 mm/gup: local lru_add_drain() to avoid lru_add_drain_all() 4/7 mm: Revert "mm/gup: clear the LRU flag of a page before 5/7 mm: Revert "mm: vmscan.c: fix OOM on swap stress test" 6/7 mm: folio_may_be_cached() unless folio_test_large() 7/7 mm: lru_add_drain_all() do local lru_add_drain() first

 include/linux/mm.h   |  4 ++--
 include/linux/swap.h | 10 ++++++++++
 mm/gup.c             |  6 +++++-
 mm/mlock.c           |  6 +++---
 mm/swap.c            | 53 ++++++++++++++++++++++++++++------------------------
 mm/vmscan.c          |  2 +-
 6 files changed, 50 insertions(+), 31 deletions(-)

Thanks,
Hugh

#1 lore raw

PATCH 1/7 mm: fix folio_expected_ref_count() when PG_private_2

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

But this went wrong earlier: folio_expected_ref_count() was based on (and replaced) mm/migrate.c's folio_expected_refs(), which has been using folio_test_private() since 6.0 converted to folios(): before that, expected_page_refs() was correctly using page_has_private().

Just a few filesystems are still using PG_private_2 a.k.a. PG_fscache. Potentially, this fix re-enables page migration on their folios; but it would not be surprising to learn that in practice those folios are not migratable for other reasons.

Fixes 86ebd50224c0 ("mm: add folio_expected_ref_count() for reference count calculation")
Fixes 108ca8358139 ("mm/migrate: Convert expected_page_refs() to folio_expected_refs()")
---
 include/linux/mm.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -2
index 1ae97a0b8ec7..ee8e535eadac 100644
@@ -2180,8 +2180,8 @@ static inline int folio_expected_ref_count(const struct folio *folio)
 	} else {
 		/* One reference per page from the pagecache. */
 		ref_count += !!folio->mapping << order;
-		/* One reference from PG_private. */
-		ref_count += folio_test_private(folio);
+		/* One reference from PG_private or PG_private_2. */
+		ref_count += folio_has_private(folio);
 	}
 
 	/* One reference per page table mapping. */
Signature · 1 line
2.51.0
#2 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.

We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.

#3 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

Quoted 8 lines

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.

We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.

I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my lru_add_drainage to 6.12 I was careful to check whether folio_expected_ref_count() would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.

I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in mm/filemap.c:

/**
* folio_end_private_2 - Clear PG_private_2 and wake any waiters.
* @folio: The folio.
*
* Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
* it. The folio reference held for PG_private_2 being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
void folio_end_private_2(struct folio *folio)

{
	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
	folio_wake_bit(folio, PG_private_2);
	folio_put(folio);
}

EXPORT_SYMBOL(folio_end_private_2);

That seems to be clear that PG_private_2 is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.

I was also anxious to work out whether PG_private with PG_private_2 would mean +1 or +2: I don't think I found any decisive statement, but traditional use of page_has_private() implied +1; and I expect there's no filesystem which actually could have both on the same folio.

Thanks,
Hugh

#4 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 46 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

Quoted 8 lines

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.

We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.

I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my lru_add_drainage to 6.12 I was careful to check whether folio_expected_ref_count() would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.

I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in mm/filemap.c:

/**
* folio_end_private_2 - Clear PG_private_2 and wake any waiters.
* @folio: The folio.
*
* Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
* it. The folio reference held for PG_private_2 being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
void folio_end_private_2(struct folio *folio)

{
	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
	folio_wake_bit(folio, PG_private_2);
	folio_put(folio);
}

EXPORT_SYMBOL(folio_end_private_2);

Quoted 7 lines

That seems to be clear that PG_private_2 is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.

I was also anxious to work out whether PG_private with PG_private_2 would mean +1 or +2: I don't think I found any decisive statement, but traditional use of page_has_private() implied +1; and I expect there's no filesystem which actually could have both on the same folio.

I think it's "+1", like we used to have.

I was seriously confused when discovering (iow, concerned about false positives):

PG_fscache = PG_private_2,

But in the end PG_fscache is only used in comments and e.g., __fscache_clear_page_bits() calls folio_end_private_2(). So both are really just aliases.

[Either PG_fscache should be dropped and referred to as PG_private_2, or PG_private_2 should be dropped and PG_fscache used instead. It's even inconsistently used in that fscache. file.

Or both should be dropped, of course, once we can actually get rid of it ...]

So PG_private_2 should not be used for any other purpose.

folio_start_private_2() / folio_end_private_2() indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.

The usage of private_2 is declared deprecated all over the place. So the question is if we really still care.

The ceph usage is guarded by CONFIG_CEPH_FSCACHE, the NFS one by NFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.

Now, one problem would be if migration / splitting / ... code where we use folio_expected_ref_count() cannot deal with that additional reference properly, in which case this patch would indeed cause harm.

If all folio_expected_ref_count() callers can deal with updating that reference, all good.

nfs_migrate_folio(), for example, has folio_test_private_2() handling in there (just wait until it is gone). ceph handles it during ceph_writepages_start(), but uses ordinary filemap_migrate_folio.

Long story short: this patch is problematic if one folio_expected_ref_count() users is not aware of how to handle that additional reference.

Signature · 3 lines
Cheers

David / dhildenb
#5 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 93 lines · David Hildenbrand
Quoted 46 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

Quoted 8 lines

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.

We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.

I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my lru_add_drainage to 6.12 I was careful to check whether folio_expected_ref_count() would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.

I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in mm/filemap.c:

/**
* folio_end_private_2 - Clear PG_private_2 and wake any waiters.
* @folio: The folio.
*
* Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
* it. The folio reference held for PG_private_2 being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
void folio_end_private_2(struct folio *folio)

{
	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
	folio_wake_bit(folio, PG_private_2);
	folio_put(folio);
}

EXPORT_SYMBOL(folio_end_private_2);

Quoted 7 lines

That seems to be clear that PG_private_2 is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.

I was also anxious to work out whether PG_private with PG_private_2 would mean +1 or +2: I don't think I found any decisive statement, but traditional use of page_has_private() implied +1; and I expect there's no filesystem which actually could have both on the same folio.

I think it's "+1", like we used to have.

I was seriously confused when discovering (iow, concerned about false positives):

PG_fscache = PG_private_2,

But in the end PG_fscache is only used in comments and e.g., __fscache_clear_page_bits() calls folio_end_private_2(). So both are really just aliases.

[Either PG_fscache should be dropped and referred to as PG_private_2, or PG_private_2 should be dropped and PG_fscache used instead. It's even inconsistently used in that fscache. file.

Or both should be dropped, of course, once we can actually get rid of it ...]

So PG_private_2 should not be used for any other purpose.

folio_start_private_2() / folio_end_private_2() indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.

The usage of private_2 is declared deprecated all over the place. So the question is if we really still care.

The ceph usage is guarded by CONFIG_CEPH_FSCACHE, the NFS one by NFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.

Now, one problem would be if migration / splitting / ... code where we use folio_expected_ref_count() cannot deal with that additional reference properly, in which case this patch would indeed cause harm.

If all folio_expected_ref_count() callers can deal with updating that reference, all good.

Quoted 8 lines

nfs_migrate_folio(), for example, has folio_test_private_2() handling in there (just wait until it is gone). ceph handles it during ceph_writepages_start(), but uses ordinary filemap_migrate_folio.

Long story short: this patch is problematic if one folio_expected_ref_count() users is not aware of how to handle that additional reference.

Case in point, I just stumbled over

commit 682a71a1b6b363bff71440f4eca6498f827a839d

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Fri Sep 2 20:46:46 2022 +0100

migrate: convert __unmap_and_move() to use folios

and

commit 8faa8ef5dd11abe119ad0c8ccd39f2064ca7ed0e

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Mon Jun 6 09:34:36 2022 -0400

mm/migrate: Convert fallback_migrate_page() to fallback_migrate_folio()

Use a folio throughout. migrate_page() will be converted to migrate_folio() later.

where we converted from page_has_private() to folio_test_private(). Maybe that's all sane, but it raises the question if migration (and maybe splitting) as a whole is no incompatible with PG_private_2

Signature · 3 lines
Cheers

David / dhildenb
#6 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 11 lines · David Hildenbrand
Quoted 10 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.

I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found that PG_private_2 always goes with refcount increment.

(Always? Well, until 6.13, btrfs used PG_private_2 without any such
increment: that's gone, so now it's consistently with refcount increment.)

Confusing, given David Howells removed deprecated use of PG_private_2 then later reverted the removal: I've not looked up which releases those came and went, but reverted in stable trees too, so story all the same; but maybe some of Matthew's mods interleaved between removal and revert.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.

Not how btrfs used to work, but it is how ceph and nfs work.

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 3 lines

We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.

I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my lru_add_drainage to 6.12 I was careful to check whether folio_expected_ref_count() would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.

I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in mm/filemap.c:

/**
* folio_end_private_2 - Clear PG_private_2 and wake any waiters.
* @folio: The folio.
*
* Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
* it. The folio reference held for PG_private_2 being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
void folio_end_private_2(struct folio *folio)
{
VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio); clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); folio_wake_bit(folio, PG_private_2);
folio_put(folio);
}
EXPORT_SYMBOL(folio_end_private_2);

That seems to be clear that PG_private_2 is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

I was also anxious to work out whether PG_private with PG_private_2 would mean +1 or +2: I don't think I found any decisive statement, but traditional use of page_has_private() implied +1; and I expect there's no filesystem which actually could have both on the same folio.

I think it's "+1", like we used to have.

I've given up worrying about that. I'm inclined to think it's +2, since there's no test_private when incrementing and decrementing for private_2; but I don't need to care any more.

Quoted 10 lines
Quoted 10 lines

I was seriously confused when discovering (iow, concerned about false positives):

PG_fscache = PG_private_2,

But in the end PG_fscache is only used in comments and e.g., __fscache_clear_page_bits() calls folio_end_private_2(). So both are really just aliases.

Quoted 8 lines
Quoted 8 lines

[Either PG_fscache should be dropped and referred to as PG_private_2, or PG_private_2 should be dropped and PG_fscache used instead. It's even inconsistently used in that fscache. file.

Or both should be dropped, of course, once we can actually get rid of it ...]

So PG_private_2 should not be used for any other purpose.

Yes, ghastly the hiding of one behind the other; that, and the PageFlags versus folio_flags, made it all tiresome to track down.

I have considered adding PG_Spanish_Inquisition = PG_private_2 since folio_expect_ref_count() ignoring PG_private_2 implies that no-one expects the PG_private_2.

Quoted 8 lines
Quoted 8 lines

folio_start_private_2() / folio_end_private_2() indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.

The usage of private_2 is declared deprecated all over the place. So the question is if we really still care.

Quoted 7 lines
Quoted 7 lines

The ceph usage is guarded by CONFIG_CEPH_FSCACHE, the NFS one by NFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.

Now, one problem would be if migration / splitting / ... code where we use folio_expected_ref_count() cannot deal with that additional reference properly, in which case this patch would indeed cause harm.

Yes, that appears to be why Matthew said NAK and "dangerously wrong".

So far as I could tell, there is no problem with nfs, it has, and has all along had, the appropriate release_folio and migrate_folio methods.

ceph used to have what's needed, but 6.0's changes from page_has_private() to folio_test_private() (the change from "has" either bit to "test" just the one bit really should have been highlighted) broke the migration of ceph's PG_private_2 folios.

(I think it may have got re-enabled in intervening releases: David Howells reinstated folio_has_private() inside fallback_migrate_folio()'s filemap_release_folio(), which may have been enough to get ceph's PG_private_2s migratable again; but then 6.15's ceph .migrate_folio = filemap_migrate_folio will have broken it again.)

Folio migration does not and never has copied over PG_private_2 from src to dst; so my 1/7 patch would have permitted migration of a ceph PG_private_2 src folio to a dst folio left with refcount 1 more than it should be (plus whatever the consequences of migrating such a folio which should have waited for the flag to be cleared first).

Earlier, I did intend to add protection against PG_private_2 into folio_migrate_mapping() and/or whatever else needs it in mm/migrate.c, as part of the 1/7 patch; and later submit a ceph patch to give it back the release_folio wait on PG_private_2 it wants.

But (a) I ran out of steam, and (b) I couldn't test it or advise ceph folks how to test it, and (c) guessed that Matthew would hate me populating the codebase with further references to PG_private_2, and (d) realized that this PG_private_2 thing is a transient condition (more like writeback than private) which probably nobody cares too much about (its lack of migration has gone unnoticed).

I'm just going to drop this 1/7, and add a (briefer than this!) paragraph to 2/7 == 1/6's commit message in v2 later today.

Quoted 29 lines
Quoted 4 lines

If all folio_expected_ref_count() callers can deal with updating that reference, all good.

Quoted 8 lines

nfs_migrate_folio(), for example, has folio_test_private_2() handling in there (just wait until it is gone). ceph handles it during ceph_writepages_start(), but uses ordinary filemap_migrate_folio.

Long story short: this patch is problematic if one folio_expected_ref_count() users is not aware of how to handle that additional reference.

Case in point, I just stumbled over

commit 682a71a1b6b363bff71440f4eca6498f827a839d

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Fri Sep 2 20:46:46 2022 +0100

migrate: convert __unmap_and_move() to use folios

and

commit 8faa8ef5dd11abe119ad0c8ccd39f2064ca7ed0e

Author: Matthew Wilcox (Oracle) <[email protected]>
Date:   Mon Jun 6 09:34:36 2022 -0400

mm/migrate: Convert fallback_migrate_page() to fallback_migrate_folio()

Quoted 7 lines

Use a folio throughout. migrate_page() will be converted to migrate_folio() later.

where we converted from page_has_private() to folio_test_private(). Maybe that's all sane, but it raises the question if migration (and maybe splitting) as a whole is no incompatible with PG_private_2

The commit I blamed in my notes was 108ca835, I think that's the one that changes "has" to "test" in the "expected" calculaton; but yes, 8faa8ef5 is significant for skipping the call to folio_release.

Hugh

#7 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 10 lines · Hugh Dickins
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.

I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found that PG_private_2 always goes with refcount increment.

Let me explain that better. No filesystem followed the documented rule that the refcount must be incremented by one if either PG_private or PG_private_2 was set. And no surprise; that's a very complicated rule for filesystems to follow. Many of them weren't even following the rule to increment the refcount by one when PG_private was set.

So some were incrementing the refcount by one if PG_private were set, but not bumping the refcount by one if PG_private_2 were set (I think this is how btrfs worked, and you seem to believe the same thing). Others were bumping the refcount by two if both PG_private and PG_private_2 were set (I think this is how netfs works today).

Quoted 20 lines
Quoted 3 lines
Quoted 3 lines

Now, one problem would be if migration / splitting / ... code where we use folio_expected_ref_count() cannot deal with that additional reference properly, in which case this patch would indeed cause harm.

Yes, that appears to be why Matthew said NAK and "dangerously wrong".

So far as I could tell, there is no problem with nfs, it has, and has all along had, the appropriate release_folio and migrate_folio methods.

ceph used to have what's needed, but 6.0's changes from page_has_private() to folio_test_private() (the change from "has" either bit to "test" just the one bit really should have been highlighted) broke the migration of ceph's PG_private_2 folios.

(I think it may have got re-enabled in intervening releases: David Howells reinstated folio_has_private() inside fallback_migrate_folio()'s filemap_release_folio(), which may have been enough to get ceph's PG_private_2s migratable again; but then 6.15's ceph .migrate_folio = filemap_migrate_folio will have broken it again.)

Quoted 5 lines

Folio migration does not and never has copied over PG_private_2 from src to dst; so my 1/7 patch would have permitted migration of a ceph PG_private_2 src folio to a dst folio left with refcount 1 more than it should be (plus whatever the consequences of migrating such a folio which should have waited for the flag to be cleared first).

But that's another problem. The current meaning of PG_fscache (and also that has changed over the years!) is that the data in the folio is being written to the fscache. So we _shouldn't_ migrate the folio as some piece of storage hardware is busy reading from the old folio. And if somebody else starts writing to the old folio, we'll have a corrupted fscache.

So the current behaviour where we set private_2 and bump the refcount, but don't take the private_2 status into account is the safe one, because the elevated refcount means we'll skip the PG_fscache folio. Maybe it'd be better to wait for it to clear. But since Dave Howells is busy killing it off, I'm just inclined to wait for that to happen.

Quoted 2 lines

I'm just going to drop this 1/7, and add a (briefer than this!) paragraph to 2/7 == 1/6's commit message in v2 later today.

Thank you!

#8 lore raw

Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2

Quoted 18 lines · Matthew Wilcox
Quoted 10 lines · Hugh Dickins
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins

6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.

I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found that PG_private_2 always goes with refcount increment.

Quoted 5 lines

Let me explain that better. No filesystem followed the documented rule that the refcount must be incremented by one if either PG_private or PG_private_2 was set. And no surprise; that's a very complicated rule for filesystems to follow. Many of them weren't even following the rule to increment the refcount by one when PG_private was set.

Thanks, yes, I hadn't realized that you were referring to the +1 (versus +2) part of it: yes, a quite unnecessarily difficult rule to follow.

...

Quoted 5 lines

So the current behaviour where we set private_2 and bump the refcount, but don't take the private_2 status into account is the safe one, because the elevated refcount means we'll skip the PG_fscache folio. Maybe it'd be better to wait for it to clear. But since Dave Howells is busy killing it off, I'm just inclined to wait for that to happen.

Yes, that's where my internalized-Matthew brought me in the end; though killing off PG_private_2 seems to have been just around the corner for a very long time.

It's a pity that there isn't much incentive on ceph folks to get it fixed: the one who suffers is the compactor or pinner.

Hugh

#9 lore raw

PATCH 2/7 mm/gup: check ref_count instead of lru before migration

In reply to Hugh Dickins

Will Deacon reports:-

When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration.

It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.

In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table.

Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated.

This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.

Hugh Dickins adds:-

!folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.

5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.

And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.

Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.

Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation") and 6.17 commit ("mm: fix folio_expected_ref_count() when PG_private_2").

Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")
---
 mm/gup.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +2 -1
index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(
 			continue;
 		}
 
-		if (!folio_test_lru(folio) && drain_allow) {
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
 			drain_allow = false;
 		}
Signature · 1 line
2.51.0
Acked-by on v1 2/7
#10 lore raw

Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration

Quoted 77 lines · Hugh Dickins

Will Deacon reports:-

When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration.

It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.

In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table.

Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated.

This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.

Hugh Dickins adds:-

!folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.

5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.

And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.

Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.

Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation") and 6.17 commit ("mm: fix folio_expected_ref_count() when PG_private_2").

Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")
---
  mm/gup.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 9 lines
mm/gup.c +2 -1
@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(
  			continue;
  		}
 
-		if (!folio_test_lru(folio) && drain_allow) {
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
  			lru_add_drain_all();
  			drain_allow = false;
  		}

In general, to the fix idea

But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.

If we cannot include it in folio_expected_ref_count(), it's all going to be a mess until PG_private_2 is removed for good.

So that part still needs to be figured out.

Signature · 3 lines
Cheers

David / dhildenb
#11 lore raw

Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration

Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long

collect_longterm_unpinnable_folios(

    	continue;
    }

- if (!folio_test_lru(folio) && drain_allow) {

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 5 lines
mm/gup.c +2 -0
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
     lru_add_drain_all();
     drain_allow = false;
    }

In general, to the fix idea

Quoted 1 line

Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.

Quoted 8 lines

But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.

If we cannot include it in folio_expected_ref_count(), it's all going to be a mess until PG_private_2 is removed for good.

So that part still needs to be figured out.

Here's that added paragraph:

Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation:
but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all(). And this makes easy the backport to earlier releases: up to and including 6.12, btrfs also used PG_private_2, but without a ref_count increment.

Hugh

#12 lore raw

Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration

Quoted 35 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long

collect_longterm_unpinnable_folios(

     	continue;
     }

- if (!folio_test_lru(folio) && drain_allow) {

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 5 lines
mm/gup.c +2 -0
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
      lru_add_drain_all();
      drain_allow = false;
     }

In general, to the fix idea

Quoted 1 line

Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.

Quoted 8 lines

But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.

If we cannot include it in folio_expected_ref_count(), it's all going to be a mess until PG_private_2 is removed for good.

So that part still needs to be figured out.

Here's that added paragraph:

Quoted 9 lines

Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation:
but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all().

Would we also loop forever trying to migrate these folios if they reside on ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.

Signature · 3 lines
Cheers

David / dhildenb
#13 lore raw

Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration

Quoted 47 lines · David Hildenbrand
Quoted 36 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long

collect_longterm_unpinnable_folios(

     	continue;
     }

- if (!folio_test_lru(folio) && drain_allow) {

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 5 lines
mm/gup.c +2 -0
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
      lru_add_drain_all();
      drain_allow = false;
     }

In general, to the fix idea

Quoted 1 line

Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.

Quoted 9 lines

But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.

If we cannot include it in folio_expected_ref_count(), it's all going to be a
mess until PG_private_2 is removed for good.

So that part still needs to be figured out.

Here's that added paragraph:

Quoted 9 lines

Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation:
but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all().

Quoted 3 lines

Would we also loop forever trying to migrate these folios if they reside on ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.

Loop around forever? That would be unfortunate (but I presume killable). But when I looked, it appeared that any failure of migrate_pages() there gets reported as -ENOMEM, which would end up as an OOM? But you know mm/gup.c very much better than I do.

If it does loop around, it's not so bad in the PG_private_2 case, because that's (nowadays always) a transient flag, much more like PG_writeback than PG_private.

But whatever, yes, the move from testing lru to checking ref_count makes no difference to that: the failure occurs in migration either way.

Hugh

#14 lore raw

Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration

Quoted 48 lines · Hugh Dickins
Quoted 47 lines · David Hildenbrand
Quoted 45 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0 -0
index adffe663594d..82aec6443c0a 100644
@@ -2307,7 +2307,8 @@ static unsigned long

collect_longterm_unpinnable_folios(

      	continue;
      }
    -		if (!folio_test_lru(folio) && drain_allow) {

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314

Quoted 5 lines
mm/gup.c +2 -0
+		if (drain_allow && folio_ref_count(folio) !=
+				   folio_expected_ref_count(folio) + 1) {
       lru_add_drain_all();
       drain_allow = false;
      }

In general, to the fix idea

Quoted 1 line

Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.

Quoted 9 lines

But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.

If we cannot include it in folio_expected_ref_count(), it's all going to be a
mess until PG_private_2 is removed for good.

So that part still needs to be figured out.

Here's that added paragraph:

Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation:
but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all().

Quoted 8 lines
Quoted 3 lines

Would we also loop forever trying to migrate these folios if they reside on ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.

Loop around forever? That would be unfortunate (but I presume killable). But when I looked, it appeared that any failure of migrate_pages() there gets reported as -ENOMEM, which would end up as an OOM? But you know mm/gup.c very much better than I do.

Yes, like I expected, we just bail out. __gup_longterm_locked() will not retry in that case. It's interesting that any migration failure is treated as -ENOMEM, but well, that's certainly material for a completely different discussion.

Thanks Hugh!

Signature · 3 lines
Cheers

David / dhildenb
#15 lore raw

PATCH 3/7 mm/gup: local lru_add_drain() to avoid lru_add_drain_all()

In reply to Hugh Dickins

In many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all().

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean backports: can take 6.16 commit a03db236aebf ("gup:
optimize longterm pin_user_pages() for large folio") first.

---
 mm/gup.c | 2 ++
 1 file changed, 2 insertions(+)
mm/gup.c +2 -0
index 82aec6443c0a..9f7c87f504a9 100644
@@ -2291,6 +2291,8 @@ static unsigned long collect_longterm_unpinnable_folios(
 	struct folio *folio;
 	long i = 0;
 
+	lru_add_drain();
+
 	for (folio = pofs_get_folio(pofs, i); folio;
 	     folio = pofs_next_folio(folio, pofs, &i)) {
 
Signature · 1 line
2.51.0
#16 lore raw

Re: [PATCH 3/7] mm/gup: local lru_add_drain() to avoid lru_add_drain_all()

Quoted 21 lines · Hugh Dickins

In many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all().

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean backports: can take 6.16 commit a03db236aebf ("gup:
optimize longterm pin_user_pages() for large folio") first.

---
  mm/gup.c | 2 ++
  1 file changed, 2 insertions(+)
mm/gup.c +0 -0
index 82aec6443c0a..9f7c87f504a9 100644

mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2298

Quoted 8 lines
mm/gup.c +2 -0
@@ -2291,6 +2291,8 @@ static unsigned long collect_longterm_unpinnable_folios(
  	struct folio *folio;
  	long i = 0;
 
+	lru_add_drain();
+
  	for (folio = pofs_get_folio(pofs, i); folio;
  	     folio = pofs_next_folio(folio, pofs, &i)) {
 

Do we really want to drain all the time we enter collect_longterm_unpinnable_folios(), or only if we detect an actual problem? (unexpected reference?)

Signature · 3 lines
Cheers

David / dhildenb
#17 lore raw

Re: [PATCH 3/7] mm/gup: local lru_add_drain() to avoid lru_add_drain_all()

Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Hugh Dickins

In many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all().

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean backports: can take 6.16 commit a03db236aebf ("gup:
optimize longterm pin_user_pages() for large folio") first.

---
  mm/gup.c | 2 ++
  1 file changed, 2 insertions(+)
mm/gup.c +0 -0
index 82aec6443c0a..9f7c87f504a9 100644
@@ -2291,6 +2291,8 @@ static unsigned long

collect_longterm_unpinnable_folios(

   struct folio *folio;
   long i = 0;

+ lru_add_drain();

Quoted 8 lines
Quoted 4 lines

+

   for (folio = pofs_get_folio(pofs, i); folio;
        folio = pofs_next_folio(folio, pofs, &i)) {

Do we really want to drain all the time we enter collect_longterm_unpinnable_folios(), or only if we detect an actual problem? (unexpected reference?)

It looked nice and simple to me (hmm, where's the blank line before lru_add_drain() gone? weird, something wrong with my mail setup), I've never avoided an lru_add_drain() before; but you're right, we dom't need to do that every time, fixed in v2 - thanks.

Hugh

#18 lore raw

PATCH 4/7 mm: Revert "mm/gup: clear the LRU flag of a page before adding to LRU batch"

In reply to Hugh Dickins

This reverts commit 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9:
now that collect_longterm_unpinnable_folios() is checking ref_count instead of lru, and mlock/munlock do not participate in the revised LRU flag clearing, those changes are misleading, and enlarge the window during which mlock/munlock may miss an mlock_count update.

It is possible (I'd hesitate to claim probable) that the greater likelihood of missed mlock_count updates would explain the "Realtime threads delayed due to kcompactd0" observed on 6.12 in the Link below. If that is the case, this reversion will help; but a complete solution needs also a further patch, beyond the scope of this series.

Included some 80-column cleanup around folio_batch_add_and_move().

The role of folio_test_clear_lru() (before taking per-memcg lru_lock) is questionable since 6.13 removed mem_cgroup_move_account() etc; but perhaps there are still some races which need it - not examined here.

---
 mm/swap.c | 50 ++++++++++++++++++++++++++------------------------
 1 file changed, 26 insertions(+), 24 deletions(-)
mm/swap.c +26 -24
index 3632dd061beb..6ae2d5680574 100644
@@ -164,6 +164,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 	for (i = 0; i < folio_batch_count(fbatch); i++) {
 		struct folio *folio = fbatch->folios[i];
 
+		/* block memcg migration while the folio moves between lru */
+		if (move_fn != lru_add && !folio_test_clear_lru(folio))
+			continue;
+
 		folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
 		move_fn(lruvec, folio);
 
@@ -176,14 +180,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 }
 
 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
-		struct folio *folio, move_fn_t move_fn,
-		bool on_lru, bool disable_irq)
+		struct folio *folio, move_fn_t move_fn, bool disable_irq)
 {
 	unsigned long flags;
 
-	if (on_lru && !folio_test_clear_lru(folio))
-		return;
-
 	folio_get(folio);
 
 	if (disable_irq)
@@ -191,8 +191,8 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 	else
 		local_lock(&cpu_fbatches.lock);
 
-	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || folio_test_large(folio) ||
-	    lru_cache_disabled())
+	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
+			folio_test_large(folio) || lru_cache_disabled())
 		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
 
 	if (disable_irq)
@@ -201,13 +201,13 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 		local_unlock(&cpu_fbatches.lock);
 }
 
-#define folio_batch_add_and_move(folio, op, on_lru)						\
-	__folio_batch_add_and_move(								\
-		&cpu_fbatches.op,								\
-		folio,										\
-		op,										\
-		on_lru,										\
-		offsetof(struct cpu_fbatches, op) >= offsetof(struct cpu_fbatches, lock_irq)	\
+#define folio_batch_add_and_move(folio, op)		\
+	__folio_batch_add_and_move(			\
+		&cpu_fbatches.op,			\
+		folio,					\
+		op,					\
+		offsetof(struct cpu_fbatches, op) >=	\
+		offsetof(struct cpu_fbatches, lock_irq)	\
 	)
 
 static void lru_move_tail(struct lruvec *lruvec, struct folio *folio)
@@ -231,10 +231,10 @@ static void lru_move_tail(struct lruvec *lruvec, struct folio *folio)
 void folio_rotate_reclaimable(struct folio *folio)
 {
 	if (folio_test_locked(folio) || folio_test_dirty(folio) ||
-	    folio_test_unevictable(folio))
+	    folio_test_unevictable(folio) || !folio_test_lru(folio))
 		return;
 
-	folio_batch_add_and_move(folio, lru_move_tail, true);
+	folio_batch_add_and_move(folio, lru_move_tail);
 }
 
 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
@@ -328,10 +328,11 @@ static void folio_activate_drain(int cpu)
 
 void folio_activate(struct folio *folio)
 {
-	if (folio_test_active(folio) || folio_test_unevictable(folio))
+	if (folio_test_active(folio) || folio_test_unevictable(folio) ||
+	    !folio_test_lru(folio))
 		return;
 
-	folio_batch_add_and_move(folio, lru_activate, true);
+	folio_batch_add_and_move(folio, lru_activate);
 }
 
 #else
@@ -507,7 +508,7 @@ void folio_add_lru(struct folio *folio)
 	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
 		folio_set_active(folio);
 
-	folio_batch_add_and_move(folio, lru_add, false);
+	folio_batch_add_and_move(folio, lru_add);
 }
 EXPORT_SYMBOL(folio_add_lru);
 
@@ -685,13 +686,13 @@ void lru_add_drain_cpu(int cpu)
 void deactivate_file_folio(struct folio *folio)
 {
 	/* Deactivating an unevictable folio will not accelerate reclaim */
-	if (folio_test_unevictable(folio))
+	if (folio_test_unevictable(folio) || !folio_test_lru(folio))
 		return;
 
 	if (lru_gen_enabled() && lru_gen_clear_refs(folio))
 		return;
 
-	folio_batch_add_and_move(folio, lru_deactivate_file, true);
+	folio_batch_add_and_move(folio, lru_deactivate_file);
 }
 
 /*
@@ -704,13 +705,13 @@ void deactivate_file_folio(struct folio *folio)
  */
 void folio_deactivate(struct folio *folio)
 {
-	if (folio_test_unevictable(folio))
+	if (folio_test_unevictable(folio) || !folio_test_lru(folio))
 		return;
 
 	if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio))
 		return;
 
-	folio_batch_add_and_move(folio, lru_deactivate, true);
+	folio_batch_add_and_move(folio, lru_deactivate);
 }
 
 /**
@@ -723,10 +724,11 @@ void folio_deactivate(struct folio *folio)
 void folio_mark_lazyfree(struct folio *folio)
 {
 	if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) ||
+	    !folio_test_lru(folio) ||
 	    folio_test_swapcache(folio) || folio_test_unevictable(folio))
 		return;
 
-	folio_batch_add_and_move(folio, lru_lazyfree, true);
+	folio_batch_add_and_move(folio, lru_lazyfree);
 }
 
 void lru_add_drain(void)
Signature · 1 line
2.51.0
Acked-by on v1 4/7
#19 lore raw

Re: [PATCH 4/7] mm: Revert "mm/gup: clear the LRU flag of a page before adding to LRU batch"

Quoted 14 lines · Hugh Dickins

This reverts commit 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9:
now that collect_longterm_unpinnable_folios() is checking ref_count instead of lru, and mlock/munlock do not participate in the revised LRU flag clearing, those changes are misleading, and enlarge the window during which mlock/munlock may miss an mlock_count update.

It is possible (I'd hesitate to claim probable) that the greater likelihood of missed mlock_count updates would explain the "Realtime threads delayed due to kcompactd0" observed on 6.12 in the Link below. If that is the case, this reversion will help; but a complete solution needs also a further patch, beyond the scope of this series.

Included some 80-column cleanup around folio_batch_add_and_move().

Quoted 8 lines

The role of folio_test_clear_lru() (before taking per-memcg lru_lock) is questionable since 6.13 removed mem_cgroup_move_account() etc; but perhaps there are still some races which need it - not examined here.

---

Signature · 3 lines
Cheers

David / dhildenb
#20 lore raw

PATCH 5/7 mm: Revert "mm: vmscan.c: fix OOM on swap stress test"

In reply to Hugh Dickins

This reverts commit 0885ef4705607936fc36a38fd74356e1c465b023: that was a fix to the reverted 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9.

---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
index a48aec8bfd92..674999999cd0 100644
@@ -4507,7 +4507,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
 	}
 
 	/* ineligible */
-	if (!folio_test_lru(folio) || zone > sc->reclaim_idx) {
+	if (zone > sc->reclaim_idx) {
 		gen = folio_inc_gen(lruvec, folio, false);
 		list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
 		return true;
Signature · 1 line
2.51.0
Acked-by on v1 5/7
#21 lore raw

Re: [PATCH 5/7] mm: Revert "mm: vmscan.c: fix OOM on swap stress test"

Quoted 3 lines · Hugh Dickins

This reverts commit 0885ef4705607936fc36a38fd74356e1c465b023: that was a fix to the reverted 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9.

Likely checkpatch will not like the long commit ids.

Quoted 3 lines

---

Signature · 3 lines
Cheers

David / dhildenb
#22 lore raw

PATCH 6/7 mm: folio_may_be_cached() unless folio_test_large()

In reply to Hugh Dickins

mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio.

But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).

So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".

---
 include/linux/swap.h | 10 ++++++++++
 mm/gup.c             |  5 +++--
 mm/mlock.c           |  6 +++---
 mm/swap.c            |  2 +-
 4 files changed, 17 insertions(+), 6 deletions(-)
include/linux/swap.h +10 -0
index 2fe6ed2cc3fd..b49a61c32238 100644
@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
 
+static inline bool folio_may_be_cached(struct folio *folio)
+{
+	/*
+	 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
+	 * Holding small numbers of low-order mTHP folios in per-CPU LRU cache
+	 * will be sensible, but nobody has implemented and tested that yet.
+	 */
+	return !folio_test_large(folio);
+}
+
 extern atomic_t lru_disable_count;
 
 static inline bool lru_cache_disabled(void)
mm/gup.c +3 -2
index 9f7c87f504a9..e70544c0f958 100644
@@ -2309,8 +2309,9 @@ static unsigned long collect_longterm_unpinnable_folios(
 			continue;
 		}
 
-		if (drain_allow && folio_ref_count(folio) !=
-				   folio_expected_ref_count(folio) + 1) {
+		if (drain_allow && folio_may_be_cached(folio) &&
+				folio_ref_count(folio) !=
+				folio_expected_ref_count(folio) + 1) {
 			lru_add_drain_all();
 			drain_allow = false;
 		}
mm/mlock.c +3 -3
index a1d93ad33c6d..427339dea380 100644
@@ -255,7 +255,7 @@ void mlock_folio(struct folio *folio)
 
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
@@ -278,7 +278,7 @@ void mlock_new_folio(struct folio *folio)
 
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, mlock_new(folio)) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
@@ -299,7 +299,7 @@ void munlock_folio(struct folio *folio)
 	 */
 	folio_get(folio);
 	if (!folio_batch_add(fbatch, folio) ||
-	    folio_test_large(folio) || lru_cache_disabled())
+	    !folio_may_be_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
 	local_unlock(&mlock_fbatch.lock);
 }
mm/swap.c +1 -1
index 6ae2d5680574..17438fd1f51a 100644
@@ -192,7 +192,7 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
 		local_lock(&cpu_fbatches.lock);
 
 	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
-			folio_test_large(folio) || lru_cache_disabled())
+			!folio_may_be_cached(folio) || lru_cache_disabled())
 		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
 
 	if (disable_irq)
Signature · 1 line
2.51.0
#23 lore raw

Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()

Quoted 31 lines · Hugh Dickins

mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio.

But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).

So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".

---
  include/linux/swap.h | 10 ++++++++++
  mm/gup.c             |  5 +++--
  mm/mlock.c           |  6 +++---
  mm/swap.c            |  2 +-
  4 files changed, 17 insertions(+), 6 deletions(-)
include/linux/swap.h +0 -0
index 2fe6ed2cc3fd..b49a61c32238 100644

include/linux/swap.h void folio_add_lru_vma(struct folio *, struct vm_area_struct *); · line 387

Quoted 3 lines
include/linux/swap.h +0 -0
@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
  void mark_page_accessed(struct page *);
  void folio_mark_accessed(struct folio *);
 

Two smaller things:

(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.

(2) With things like fscache in mind, the function can be a bit misleading.

So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.

Thinking again, "maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.

Maybe it's more like "supports being cached".

folio_lru_caching_supported()

Something like that, maybe? (again, unclear about lru/mlock cache abstraction)

Signature · 3 lines
Cheers

David / dhildenb
#24 lore raw

Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()

Quoted 49 lines · David Hildenbrand
Quoted 31 lines · Hugh Dickins

mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio.

But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).

So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".

---
  include/linux/swap.h | 10 ++++++++++
  mm/gup.c             |  5 +++--
  mm/mlock.c           |  6 +++---
  mm/swap.c            |  2 +-
  4 files changed, 17 insertions(+), 6 deletions(-)
include/linux/swap.h +0 -0
index 2fe6ed2cc3fd..b49a61c32238 100644

include/linux/swap.h void folio_add_lru_vma(struct folio *, struct · line 388

Quoted 4 lines
include/linux/swap.h +0 -0
@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct
 vm_area_struct *);
  void mark_page_accessed(struct page *);
  void folio_mark_accessed(struct folio *);
 

Two smaller things:

(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.

(2) With things like fscache in mind, the function can be a bit misleading.

So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.

Quoted 6 lines

Thinking again, "maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.

Maybe it's more like "supports being cached".

folio_lru_caching_supported()

folio_may_be_cached() -> folio_may_be_lru_cached(), yes, that's very much better, thanks.

(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but lru_cache_disable() gave us that terminology, and we've gone with the flow ever since. lru_add_drain() would be better named lru_cache_drain() now, I've always got hung up on "adding a drain".)

"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.

kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.

Hugh

#25 lore raw

Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()

Quoted 57 lines · Hugh Dickins
Quoted 49 lines · David Hildenbrand
Quoted 31 lines · Hugh Dickins

mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio.

But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).

So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.

Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".

---
   include/linux/swap.h | 10 ++++++++++
   mm/gup.c             |  5 +++--
   mm/mlock.c           |  6 +++---
   mm/swap.c            |  2 +-
   4 files changed, 17 insertions(+), 6 deletions(-)
include/linux/swap.h +0 -0
index 2fe6ed2cc3fd..b49a61c32238 100644

include/linux/swap.h void folio_add_lru_vma(struct folio *, struct · line 388

Quoted 4 lines
include/linux/swap.h +0 -0
@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct
 vm_area_struct *);
   void mark_page_accessed(struct page *);
   void folio_mark_accessed(struct folio *);
 

Two smaller things:

(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.

(2) With things like fscache in mind, the function can be a bit misleading.

So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.

Quoted 6 lines

Thinking again, "maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.

Maybe it's more like "supports being cached".

folio_lru_caching_supported()

Quoted 7 lines

folio_may_be_cached() -> folio_may_be_lru_cached(), yes, that's very much better, thanks.

(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but lru_cache_disable() gave us that terminology, and we've gone with the flow ever since. lru_add_drain() would be better named lru_cache_drain() now, I've always got hung up on "adding a drain".)

Yeah, the terminology is not that intuitive :)

Not sure if using "batched" instead of "cached" might be clearer long-term?

Quoted 4 lines

"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.

I would wish we could find something that also non-native speakers can immediately understand ;)

"may_get_lru_cached" / "may_get_lru_batched"?

/me could not even phrase it in German properly

Quoted 4 lines

kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.

So a comment that this is an internal helper might be nice. Or we just move it straight to mm/internal.h ?

Signature · 3 lines
Cheers

David / dhildenb
#26 lore raw

Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()

Quoted 1 line · David Hildenbrand

On 08.09.25 13:19, Hugh Dickins wrote:

...

Quoted 23 lines
Quoted 5 lines

(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but lru_cache_disable() gave us that terminology, and we've gone with the flow ever since. lru_add_drain() would be better named lru_cache_drain() now, I've always got hung up on "adding a drain".)

Yeah, the terminology is not that intuitive :)

Not sure if using "batched" instead of "cached" might be clearer long-term?

Quoted 4 lines

"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.

I would wish we could find something that also non-native speakers can immediately understand ;)

"may_get_lru_cached" / "may_get_lru_batched"?

/me could not even phrase it in German properly

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.

So a comment that this is an internal helper might be nice. Or we just move it straight to mm/internal.h ?

mm/internal.h, where we hide things (GFP_RECLAIM_MASK etc!) that belong elsewhere? No thanks.

David, I think you're over-thinking this: I'm coming to regret not just going with your excellent folio_test_large() optimization, and let someone else mess around with the naming.

I'll post later with my current naming, but if it's the Suggested-by that's making you uncomfortable, we can delete it.

Hugh

#27 lore raw

Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()

Quoted 30 lines · Hugh Dickins
Quoted 1 line · David Hildenbrand

On 08.09.25 13:19, Hugh Dickins wrote:

...

Quoted 27 lines
Quoted 5 lines

(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but lru_cache_disable() gave us that terminology, and we've gone with the flow ever since. lru_add_drain() would be better named lru_cache_drain() now, I've always got hung up on "adding a drain".)

Yeah, the terminology is not that intuitive :)

Not sure if using "batched" instead of "cached" might be clearer long-term?

Quoted 4 lines

"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.

I would wish we could find something that also non-native speakers can immediately understand ;)

"may_get_lru_cached" / "may_get_lru_batched"?

/me could not even phrase it in German properly

Quoted 4 lines

kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.

Quoted 5 lines
Quoted 2 lines

So a comment that this is an internal helper might be nice. Or we just move it straight to mm/internal.h ?

mm/internal.h, where we hide things (GFP_RECLAIM_MASK etc!) that belong elsewhere? No thanks.

Yes, or other mm-internal helpers that actually belong there. Like folio_raw_mapping() or folio_nr_pages_mapped().

Quoted 4 lines

David, I think you're over-thinking this: I'm coming to regret not just going with your excellent folio_test_large() optimization, and let someone else mess around with the naming.

Please don't feel like I'm pushing to hard here. If you feel the current naming is fine and for some reason I don't completely understand it should not be in mm/internal.h, all good.

Signature · 3 lines
Cheers

David / dhildenb
#28 lore raw

PATCH 7/7 mm: lru_add_drain_all() do local lru_add_drain() first

In reply to Hugh Dickins

No numbers to back this up, but it seemed obvious to me, that if there are competing lru_add_drain_all()ers, the work will be minimized if each flushes its own local queues before locking and doing cross-CPU drains.

---
 mm/swap.c | 3 +++
 1 file changed, 3 insertions(+)
mm/swap.c +3 -0
index 17438fd1f51a..4951f4d35e40 100644
@@ -834,6 +834,9 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	 */
 	this_gen = smp_load_acquire(&lru_drain_gen);
 
+	/* It helps everyone if we do our own local drain immediately. */
+	lru_add_drain();
+
 	mutex_lock(&lock);
 
 	/*
Signature · 1 line
2.51.0
Acked-by on v1 7/7
#29 lore raw

Re: [PATCH 7/7] mm: lru_add_drain_all() do local lru_add_drain() first

Quoted 4 lines · Hugh Dickins

No numbers to back this up, but it seemed obvious to me, that if there are competing lru_add_drain_all()ers, the work will be minimized if each flushes its own local queues before locking and doing cross-CPU drains.

Sounds reasonable

Signature · 3 lines
Cheers

David / dhildenb