mm: better GUP pin lru_add_drain_all()
Subsystems (7)
cover mm: better GUP pin lru_add_drain_all()
Series of lru_add_drain_all()-related patches, arising from recent mm/gup migration report from Will Deacon. Based on 6.17-rc3 but apply to latest mm.git. I suggest all but 7/7 be hotfixes going to 6.17 and stable, but you might not be persuaded by "counter the increase" ones.
1/7 mm: fix folio_expected_ref_count() when PG_private_2 2/7 mm/gup: check ref_count instead of lru before migration 3/7 mm/gup: local lru_add_drain() to avoid lru_add_drain_all() 4/7 mm: Revert "mm/gup: clear the LRU flag of a page before 5/7 mm: Revert "mm: vmscan.c: fix OOM on swap stress test" 6/7 mm: folio_may_be_cached() unless folio_test_large() 7/7 mm: lru_add_drain_all() do local lru_add_drain() first
include/linux/mm.h | 4 ++-- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 6 +++++- mm/mlock.c | 6 +++--- mm/swap.c | 53 ++++++++++++++++++++++++++++------------------------ mm/vmscan.c | 2 +- 6 files changed, 50 insertions(+), 31 deletions(-)
Thanks,
Hugh
PATCH 1/7 mm: fix folio_expected_ref_count() when PG_private_2
6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private().
But this went wrong earlier: folio_expected_ref_count() was based on (and replaced) mm/migrate.c's folio_expected_refs(), which has been using folio_test_private() since 6.0 converted to folios(): before that, expected_page_refs() was correctly using page_has_private().
Just a few filesystems are still using PG_private_2 a.k.a. PG_fscache. Potentially, this fix re-enables page migration on their folios; but it would not be surprising to learn that in practice those folios are not migratable for other reasons.
--- include/linux/mm.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -2
@@ -2180,8 +2180,8 @@ static inline int folio_expected_ref_count(const struct folio *folio) }else {/* One reference per page from the pagecache. */ ref_count += !!folio->mapping << order; -/* One reference from PG_private. */ - ref_count +=folio_test_private (folio); +/* One reference from PG_private or PG_private_2. */ + ref_count +=folio_has_private (folio); }/* One reference per page table mapping. */
Signature · 1 line
2.51.0
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.
We're achingly close to getting rid of PG_private_2. I think it's just ceph and nfs that still use it.
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().
Quoted 8 lines
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.
We're achingly close to getting rid of
PG_private_2. I think it's just ceph and nfs that still use it.
I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my lru_add_drainage to 6.12 I was careful to check whether folio_expected_ref_count() would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.
I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in mm/filemap.c:
/**
* folio_end_private_2 - Clear PG_private_2 and wake any waiters.
* @folio: The folio.
*
* Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
* it. The folio reference held for PG_private_2 being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
void folio_end_private_2(struct folio *folio)
{
VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
folio_wake_bit(folio, PG_private_2);
folio_put(folio);
}EXPORT_SYMBOL(folio_end_private_2);
That seems to be clear that PG_private_2 is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.
I was also anxious to work out whether PG_private with PG_private_2 would mean +1 or +2: I don't think I found any decisive statement, but traditional use of page_has_private() implied +1; and I expect there's no filesystem which actually could have both on the same folio.
Thanks,
Hugh
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 46 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().Quoted 8 lines
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.
We're achingly close to getting rid of
PG_private_2. I think it's just ceph and nfs that still use it.I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my
lru_add_drainageto 6.12 I was careful to check whetherfolio_expected_ref_count()would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in
mm/filemap.c:/**
*folio_end_private_2- ClearPG_private_2and wake any waiters.
* @folio: The folio.
*
* Clear thePG_private_2bit on a folio and wake up any sleepers waiting for
* it. The folio reference held forPG_private_2being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
voidfolio_end_private_2(struct folio *folio){ VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio); clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); folio_wake_bit(folio, PG_private_2); folio_put(folio); }
EXPORT_SYMBOL(folio_end_private_2);
Quoted 7 lines
That seems to be clear that
PG_private_2is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.I was also anxious to work out whether
PG_privatewithPG_private_2would mean +1 or +2: I don't think I found any decisive statement, but traditional use ofpage_has_private()implied +1; and I expect there's no filesystem which actually could have both on the same folio.
I think it's "+1", like we used to have.
I was seriously confused when discovering (iow, concerned about false positives):
PG_fscache = PG_private_2,
But in the end PG_fscache is only used in comments and e.g., __fscache_clear_page_bits() calls folio_end_private_2(). So both are really just aliases.
[Either PG_fscache should be dropped and referred to as PG_private_2, or PG_private_2 should be dropped and PG_fscache used instead. It's even inconsistently used in that fscache. file.
Or both should be dropped, of course, once we can actually get rid of it ...]
So PG_private_2 should not be used for any other purpose.
folio_start_private_2() / folio_end_private_2() indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.
The usage of private_2 is declared deprecated all over the place. So the question is if we really still care.
The ceph usage is guarded by CONFIG_CEPH_FSCACHE, the NFS one by NFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.
Now, one problem would be if migration / splitting / ... code where we use folio_expected_ref_count() cannot deal with that additional reference properly, in which case this patch would indeed cause harm.
If all folio_expected_ref_count() callers can deal with updating that reference, all good.
nfs_migrate_folio(), for example, has folio_test_private_2() handling in there (just wait until it is gone). ceph handles it during ceph_writepages_start(), but uses ordinary filemap_migrate_folio.
Long story short: this patch is problematic if one folio_expected_ref_count() users is not aware of how to handle that additional reference.
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 93 lines · David Hildenbrand
Quoted 46 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().Quoted 8 lines
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked. I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.
We're achingly close to getting rid of
PG_private_2. I think it's just ceph and nfs that still use it.I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my
lru_add_drainageto 6.12 I was careful to check whetherfolio_expected_ref_count()would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in
mm/filemap.c:/**
*folio_end_private_2- ClearPG_private_2and wake any waiters.
* @folio: The folio.
*
* Clear thePG_private_2bit on a folio and wake up any sleepers waiting for
* it. The folio reference held forPG_private_2being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
voidfolio_end_private_2(struct folio *folio){ VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio); clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); folio_wake_bit(folio, PG_private_2); folio_put(folio); }
EXPORT_SYMBOL(folio_end_private_2);Quoted 7 lines
That seems to be clear that
PG_private_2is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.I was also anxious to work out whether
PG_privatewithPG_private_2would mean +1 or +2: I don't think I found any decisive statement, but traditional use ofpage_has_private()implied +1; and I expect there's no filesystem which actually could have both on the same folio.I think it's "+1", like we used to have.
I was seriously confused when discovering (iow, concerned about false positives):
PG_fscache=PG_private_2,But in the end
PG_fscacheis only used in comments and e.g.,__fscache_clear_page_bits()callsfolio_end_private_2(). So both are really just aliases.[Either
PG_fscacheshould be dropped and referred to asPG_private_2, orPG_private_2should be dropped andPG_fscacheused instead. It's even inconsistently used in that fscache. file.Or both should be dropped, of course, once we can actually get rid of it ...]
So
PG_private_2should not be used for any other purpose.
folio_start_private_2()/folio_end_private_2()indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.The usage of
private_2is declared deprecated all over the place. So the question is if we really still care.The ceph usage is guarded by
CONFIG_CEPH_FSCACHE, the NFS one byNFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.Now, one problem would be if migration / splitting / ... code where we use
folio_expected_ref_count()cannot deal with that additional reference properly, in which case this patch would indeed cause harm.If all
folio_expected_ref_count()callers can deal with updating that reference, all good.
Quoted 8 lines
nfs_migrate_folio(), for example, hasfolio_test_private_2()handling in there (just wait until it is gone). ceph handles it duringceph_writepages_start(), but uses ordinaryfilemap_migrate_folio.Long story short: this patch is problematic if one
folio_expected_ref_count()users is not aware of how to handle that additional reference.
Case in point, I just stumbled over
commit 682a71a1b6b363bff71440f4eca6498f827a839d
Author: Matthew Wilcox (Oracle) <[email protected]> Date: Fri Sep 2 20:46:46 2022 +0100
migrate: convert __unmap_and_move() to use folios
and
commit 8faa8ef5dd11abe119ad0c8ccd39f2064ca7ed0e
Author: Matthew Wilcox (Oracle) <[email protected]> Date: Mon Jun 6 09:34:36 2022 -0400
mm/migrate: Convert fallback_migrate_page() to fallback_migrate_folio()
Use a folio throughout. migrate_page() will be converted to migrate_folio() later.
where we converted from page_has_private() to folio_test_private(). Maybe that's all sane, but it raises the question if migration (and maybe splitting) as a whole is no incompatible with PG_private_2
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 11 lines · David Hildenbrand
Quoted 10 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.
I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found that PG_private_2 always goes with refcount increment.
(Always? Well, until 6.13, btrfs used PG_private_2 without any such
increment: that's gone, so now it's consistently with refcount increment.)
Confusing, given David Howells removed deprecated use of PG_private_2 then later reverted the removal: I've not looked up which releases those came and went, but reverted in stable trees too, so story all the same; but maybe some of Matthew's mods interleaved between removal and revert.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I'm not sure if there's still documentation lying around that gets this wrong or if you're remembering how things used to be documented, but it's never how any filesystem has ever worked.
Not how btrfs used to work, but it is how ceph and nfs work.
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 3 lines
We're achingly close to getting rid of
PG_private_2. I think it's just ceph and nfs that still use it.I knew you were trying to get rid of it (hurrah! thank you), so when I tried porting my
lru_add_drainageto 6.12 I was careful to check whetherfolio_expected_ref_count()would need to add it to the accounting there:
apparently yes; but then I was surprised to find that it's still present in 6.17-rc, I'd assumed it gone long ago.I didn't try to read the filesystems (which could easily have been inconsistent about it) to understand: what convinced me amidst all the confusion was this comment and code in
mm/filemap.c:/**
*folio_end_private_2- ClearPG_private_2and wake any waiters.
* @folio: The folio.
*
* Clear thePG_private_2bit on a folio and wake up any sleepers waiting for
* it. The folio reference held forPG_private_2being set is released. *
* This is, for example, used when a netfs folio is being written to a local
* disk cache, thereby allowing writes to the cache for the same folio to be
* serialised.
*/
voidfolio_end_private_2(struct folio *folio)
{
VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);clear_bit_unlock(PG_private_2, folio_flags(folio, 0));folio_wake_bit(folio, PG_private_2);
folio_put(folio);
}EXPORT_SYMBOL(folio_end_private_2);That seems to be clear that
PG_private_2is matched by a folio reference, but perhaps you can explain it away - worth changing the comment if so.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
I was also anxious to work out whether
PG_privatewithPG_private_2would mean +1 or +2: I don't think I found any decisive statement, but traditional use ofpage_has_private()implied +1; and I expect there's no filesystem which actually could have both on the same folio.I think it's "+1", like we used to have.
I've given up worrying about that. I'm inclined to think it's +2, since there's no test_private when incrementing and decrementing for private_2; but I don't need to care any more.
Quoted 10 lines
Quoted 10 lines
I was seriously confused when discovering (iow, concerned about false positives):
PG_fscache=PG_private_2,But in the end
PG_fscacheis only used in comments and e.g.,__fscache_clear_page_bits()callsfolio_end_private_2(). So both are really just aliases.
Quoted 8 lines
Quoted 8 lines
[Either
PG_fscacheshould be dropped and referred to asPG_private_2, orPG_private_2should be dropped andPG_fscacheused instead. It's even inconsistently used in that fscache. file.Or both should be dropped, of course, once we can actually get rid of it ...]
So
PG_private_2should not be used for any other purpose.
Yes, ghastly the hiding of one behind the other; that, and the PageFlags versus folio_flags, made it all tiresome to track down.
I have considered adding PG_Spanish_Inquisition = PG_private_2 since folio_expect_ref_count() ignoring PG_private_2 implies that no-one expects the PG_private_2.
Quoted 8 lines
Quoted 8 lines
folio_start_private_2()/folio_end_private_2()indeed pair the flag with a reference. There are no other callers that would set/clear the flag without involving a reference.The usage of
private_2is declared deprecated all over the place. So the question is if we really still care.
Quoted 7 lines
Quoted 7 lines
The ceph usage is guarded by
CONFIG_CEPH_FSCACHE, the NFS one byNFS_FSCACHE, nothing really seems to prevent it from getting configured in easily.Now, one problem would be if migration / splitting / ... code where we use
folio_expected_ref_count()cannot deal with that additional reference properly, in which case this patch would indeed cause harm.
Yes, that appears to be why Matthew said NAK and "dangerously wrong".
So far as I could tell, there is no problem with nfs, it has, and has all along had, the appropriate release_folio and migrate_folio methods.
ceph used to have what's needed, but 6.0's changes from page_has_private() to folio_test_private() (the change from "has" either bit to "test" just the one bit really should have been highlighted) broke the migration of ceph's PG_private_2 folios.
(I think it may have got re-enabled in intervening releases: David Howells reinstated folio_has_private() inside fallback_migrate_folio()'s filemap_release_folio(), which may have been enough to get ceph's PG_private_2s migratable again; but then 6.15's ceph .migrate_folio = filemap_migrate_folio will have broken it again.)
Folio migration does not and never has copied over PG_private_2 from src to dst; so my 1/7 patch would have permitted migration of a ceph PG_private_2 src folio to a dst folio left with refcount 1 more than it should be (plus whatever the consequences of migrating such a folio which should have waited for the flag to be cleared first).
Earlier, I did intend to add protection against PG_private_2 into folio_migrate_mapping() and/or whatever else needs it in mm/migrate.c, as part of the 1/7 patch; and later submit a ceph patch to give it back the release_folio wait on PG_private_2 it wants.
But (a) I ran out of steam, and (b) I couldn't test it or advise ceph folks how to test it, and (c) guessed that Matthew would hate me populating the codebase with further references to PG_private_2, and (d) realized that this PG_private_2 thing is a transient condition (more like writeback than private) which probably nobody cares too much about (its lack of migration has gone unnoticed).
I'm just going to drop this 1/7, and add a (briefer than this!) paragraph to 2/7 == 1/6's commit message in v2 later today.
Quoted 29 lines
Quoted 4 lines
If all
folio_expected_ref_count()callers can deal with updating that reference, all good.Quoted 8 lines
nfs_migrate_folio(), for example, hasfolio_test_private_2()handling in there (just wait until it is gone). ceph handles it duringceph_writepages_start(), but uses ordinaryfilemap_migrate_folio.Long story short: this patch is problematic if one
folio_expected_ref_count()users is not aware of how to handle that additional reference.Case in point, I just stumbled over
commit
682a71a1b6b363bff71440f4eca6498f827a839dAuthor: Matthew Wilcox (Oracle) <[email protected]> Date: Fri Sep 2 20:46:46 2022 +0100migrate: convert
__unmap_and_move()to use foliosand
commit
8faa8ef5dd11abe119ad0c8ccd39f2064ca7ed0eAuthor: Matthew Wilcox (Oracle) <[email protected]> Date: Mon Jun 6 09:34:36 2022 -0400mm/migrate: Convert
fallback_migrate_page()tofallback_migrate_folio()
Quoted 7 lines
Use a folio throughout.
migrate_page()will be converted tomigrate_folio()later.where we converted from
page_has_private()tofolio_test_private(). Maybe that's all sane, but it raises the question if migration (and maybe splitting) as a whole is no incompatible withPG_private_2
The commit I blamed in my notes was 108ca835, I think that's the one that changes "has" to "test" in the "expected" calculaton; but yes, 8faa8ef5 is significant for skipping the call to folio_release.
Hugh
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 10 lines · Hugh Dickins
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.
I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found thatPG_private_2always goes with refcount increment.
Let me explain that better. No filesystem followed the documented rule that the refcount must be incremented by one if either PG_private or PG_private_2 was set. And no surprise; that's a very complicated rule for filesystems to follow. Many of them weren't even following the rule to increment the refcount by one when PG_private was set.
So some were incrementing the refcount by one if PG_private were set, but not bumping the refcount by one if PG_private_2 were set (I think this is how btrfs worked, and you seem to believe the same thing). Others were bumping the refcount by two if both PG_private and PG_private_2 were set (I think this is how netfs works today).
Quoted 20 lines
Quoted 3 lines
Quoted 3 lines
Now, one problem would be if migration / splitting / ... code where we use
folio_expected_ref_count()cannot deal with that additional reference properly, in which case this patch would indeed cause harm.Yes, that appears to be why Matthew said NAK and "dangerously wrong".
So far as I could tell, there is no problem with nfs, it has, and has all along had, the appropriate
release_folioandmigrate_foliomethods.ceph used to have what's needed, but 6.0's changes from
page_has_private()tofolio_test_private()(the change from "has" either bit to "test" just the one bit really should have been highlighted) broke the migration of ceph'sPG_private_2folios.(I think it may have got re-enabled in intervening releases: David Howells reinstated
folio_has_private()inside fallback_migrate_folio()'sfilemap_release_folio(), which may have been enough to get ceph'sPG_private_2smigratable again; but then 6.15's ceph .migrate_folio =filemap_migrate_foliowill have broken it again.)
Quoted 5 lines
Folio migration does not and never has copied over
PG_private_2from src to dst; so my 1/7 patch would have permitted migration of a cephPG_private_2src folio to a dst folio left with refcount 1 more than it should be (plus whatever the consequences of migrating such a folio which should have waited for the flag to be cleared first).
But that's another problem. The current meaning of PG_fscache (and also that has changed over the years!) is that the data in the folio is being written to the fscache. So we _shouldn't_ migrate the folio as some piece of storage hardware is busy reading from the old folio. And if somebody else starts writing to the old folio, we'll have a corrupted fscache.
So the current behaviour where we set private_2 and bump the refcount, but don't take the private_2 status into account is the safe one, because the elevated refcount means we'll skip the PG_fscache folio. Maybe it'd be better to wait for it to clear. But since Dave Howells is busy killing it off, I'm just inclined to wait for that to happen.
Quoted 2 lines
I'm just going to drop this 1/7, and add a (briefer than this!) paragraph to 2/7 == 1/6's commit message in v2 later today.
Thank you!
Re: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2
Quoted 18 lines · Matthew Wilcox
Quoted 10 lines · Hugh Dickins
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Hugh Dickins
6.16's
folio_expected_ref_count()is forgetting thePG_private_2flag, which (likePG_private, but not in addition toPG_private) counts for 1 more reference: it needs to be usingfolio_has_private()in place offolio_test_private().Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
No, it doesn't. I know it used to, but no filesystem was actually doing that. So I changed mm to match how filesystems actually worked.
I think Matthew may be remembering how he wanted it to behave (? but he wanted it to go away completely) rather than how it ended up behaving:
we've both found thatPG_private_2always goes with refcount increment.
Quoted 5 lines
Let me explain that better. No filesystem followed the documented rule that the refcount must be incremented by one if either
PG_privateorPG_private_2was set. And no surprise; that's a very complicated rule for filesystems to follow. Many of them weren't even following the rule to increment the refcount by one whenPG_privatewas set.
Thanks, yes, I hadn't realized that you were referring to the +1 (versus +2) part of it: yes, a quite unnecessarily difficult rule to follow.
...
Quoted 5 lines
So the current behaviour where we set
private_2and bump the refcount, but don't take theprivate_2status into account is the safe one, because the elevated refcount means we'll skip thePG_fscachefolio. Maybe it'd be better to wait for it to clear. But since Dave Howells is busy killing it off, I'm just inclined to wait for that to happen.
Yes, that's where my internalized-Matthew brought me in the end; though killing off PG_private_2 seems to have been just around the corner for a very long time.
It's a pity that there isn't much incentive on ceph folks to get it fixed: the one who suffers is the compactor or pinner.
Hugh
PATCH 2/7 mm/gup: check ref_count instead of lru before migration
In reply to Hugh DickinsWill Deacon reports:-
When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration.
It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.
In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table.
Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated.
This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.
Hugh Dickins adds:-
!folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.
5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.
And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.
Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.
Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm:
add folio_expected_ref_count() for reference count calculation") and 6.17 commit ("mm: fix folio_expected_ref_count() when PG_private_2").
--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +2 -1
@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(continue ; } -if (!folio_test_lru (folio) && drain_allow) { +if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }
Signature · 1 line
2.51.0
Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration
Quoted 77 lines · Hugh Dickins
Will Deacon reports:-
When taking a longterm GUP pin via
pin_user_pages(),__gup_longterm_locked()tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off tomigrate_pages()for the actual migration.It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and
pin_user_pages()will return-ENOMEMto its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which usespin_user_pages()on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely.In the failure case, 'crosvm' has called
mlock(MLOCK_ONFAULT)on its mapping of guest memory prior to the pinning. Subsequently, whenpin_user_pages()walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it withmlock_folio()and maps it in the page-table.Since commit
2fbb0c10d1e8("mm/munlock:mlock_page()munlock_page()batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example,mlock_folio()takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing bymlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained withlru_add_drain_all()but as long as a folio remains unprocessed on the batch, its refcount will be elevated.This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation.
Hugh Dickins adds:-
!folio_test_lru() has never been a very reliable way to tell if an
lru_add_drain_all()is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache.5.18 commit
2fbb0c10d1e8may have made it more unreliable. Then 6.11 commit33dfe9204f29("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported.And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU.
Instead, switch to an expected
ref_countcheck, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better.Note for stable backports: requires 6.16 commit
86ebd50224c0("mm:
addfolio_expected_ref_count()for reference count calculation") and 6.17 commit ("mm: fixfolio_expected_ref_count()whenPG_private_2").Reported-by Will Deacon <[email protected]>Fixes 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region")Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/gup.c +0
-0
mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314
Quoted 9 lines
mm/gup.c +2
-1@@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_folios(continue ; } -if (!folio_test_lru (folio) && drain_allow) { +if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }
In general, to the fix idea
But as raised in reply to patch #1, we have to be a bit careful about including private_2 in folio_expected_ref_count() at this point.
If we cannot include it in folio_expected_ref_count(), it's all going to be a mess until PG_private_2 is removed for good.
So that part still needs to be figured out.
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0
-0@@ -2307,7 +2307,8 @@ static unsigned longcollect_longterm_unpinnable_folios(
continue; }- if (!folio_test_lru(folio) &&
drain_allow) {mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314
Quoted 5 lines
mm/gup.c +2
-0+if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }In general, to the fix idea
Quoted 1 line
Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.
Quoted 8 lines
But as raised in reply to patch #1, we have to be a bit careful about including
private_2infolio_expected_ref_count()at this point.If we cannot include it in
folio_expected_ref_count(), it's all going to be a mess untilPG_private_2is removed for good.So that part still needs to be figured out.
Here's that added paragraph:
Note on PG_private_2: ceph and nfs are still using the deprecated PG_private_2 flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folio ref_count, folio_expected_ref_count() intentionally does not recognize it, and ceph folio migration currently depends on that for PG_private_2 folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into the collect_longterm_unpinnable_folios() calculation:
but longterm pinning of transiently PG_private_2 ceph and nfs folios (an uncommon case) may invoke a redundant lru_add_drain_all(). And this makes easy the backport to earlier releases: up to and including 6.12, btrfs also used PG_private_2, but without a ref_count increment.
Hugh
Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration
Quoted 35 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0
-0@@ -2307,7 +2307,8 @@ static unsigned longcollect_longterm_unpinnable_folios(
continue; }- if (!folio_test_lru(folio) &&
drain_allow) {mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314
Quoted 5 lines
mm/gup.c +2
-0+if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }In general, to the fix idea
Quoted 1 line
Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.
Quoted 8 lines
But as raised in reply to patch #1, we have to be a bit careful about including
private_2infolio_expected_ref_count()at this point.If we cannot include it in
folio_expected_ref_count(), it's all going to be a mess untilPG_private_2is removed for good.So that part still needs to be figured out.
Here's that added paragraph:
Quoted 9 lines
Note on
PG_private_2: ceph and nfs are still using the deprecatedPG_private_2flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folioref_count,folio_expected_ref_count()intentionally does not recognize it, and ceph folio migration currently depends on that forPG_private_2folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into thecollect_longterm_unpinnable_folios()calculation:
but longterm pinning of transientlyPG_private_2ceph and nfs folios (an uncommon case) may invoke a redundantlru_add_drain_all().
Would we also loop forever trying to migrate these folios if they reside on ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration
Quoted 47 lines · David Hildenbrand
Quoted 36 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0
-0@@ -2307,7 +2307,8 @@ static unsigned longcollect_longterm_unpinnable_folios(
continue; }- if (!folio_test_lru(folio) &&
drain_allow) {mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314
Quoted 5 lines
mm/gup.c +2
-0+if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }In general, to the fix idea
Quoted 1 line
Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.
Quoted 9 lines
But as raised in reply to patch #1, we have to be a bit careful about including
private_2infolio_expected_ref_count()at this point.If we cannot include it in
folio_expected_ref_count(), it's all going to be a
mess untilPG_private_2is removed for good.So that part still needs to be figured out.
Here's that added paragraph:
Quoted 9 lines
Note on
PG_private_2: ceph and nfs are still using the deprecatedPG_private_2flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folioref_count,folio_expected_ref_count()intentionally does not recognize it, and ceph folio migration currently depends on that forPG_private_2folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into thecollect_longterm_unpinnable_folios()calculation:
but longterm pinning of transientlyPG_private_2ceph and nfs folios (an uncommon case) may invoke a redundantlru_add_drain_all().
Quoted 3 lines
Would we also loop forever trying to migrate these folios if they reside on
ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.
Loop around forever? That would be unfortunate (but I presume killable). But when I looked, it appeared that any failure of migrate_pages() there gets reported as -ENOMEM, which would end up as an OOM? But you know mm/gup.c very much better than I do.
If it does loop around, it's not so bad in the PG_private_2 case, because that's (nowadays always) a transient flag, much more like PG_writeback than PG_private.
But whatever, yes, the move from testing lru to checking ref_count makes no difference to that: the failure occurs in migration either way.
Hugh
Re: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration
Quoted 48 lines · Hugh Dickins
Quoted 47 lines · David Hildenbrand
Quoted 45 lines · Hugh Dickins
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Hugh Dickins
mm/gup.c +0
-0@@ -2307,7 +2307,8 @@ static unsigned longcollect_longterm_unpinnable_folios(
continue; } - if (!folio_test_lru(folio) && drain_allow) {mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2314
Quoted 5 lines
mm/gup.c +2
-0+if (drain_allow &&folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }In general, to the fix idea
Quoted 1 line
Thanks, but I'd better not assume that in v2, even though code the same. Will depend on how you feel about added paragraph in v2 commit message.
Quoted 9 lines
But as raised in reply to patch #1, we have to be a bit careful about including
private_2infolio_expected_ref_count()at this point.If we cannot include it in
folio_expected_ref_count(), it's all going to be a
mess untilPG_private_2is removed for good.So that part still needs to be figured out.
Here's that added paragraph:
Note on
PG_private_2: ceph and nfs are still using the deprecatedPG_private_2flag, with the aid of netfs and filemap support functions. Although it is consistently matched by an increment of folioref_count,folio_expected_ref_count()intentionally does not recognize it, and ceph folio migration currently depends on that forPG_private_2folios to be rejected. New references to the deprecated flag are discouraged, so do not add it into thecollect_longterm_unpinnable_folios()calculation:
but longterm pinning of transientlyPG_private_2ceph and nfs folios (an uncommon case) may invoke a redundantlru_add_drain_all().
Quoted 8 lines
Quoted 3 lines
Would we also loop forever trying to migrate these folios if they reside on
ZONE_MOVABLE? I would assume that is already the case, that migration will always fail due to the raised reference.Loop around forever? That would be unfortunate (but I presume killable). But when I looked, it appeared that any failure of
migrate_pages()there gets reported as-ENOMEM, which would end up as an OOM? But you knowmm/gup.cvery much better than I do.
Yes, like I expected, we just bail out. __gup_longterm_locked() will not retry in that case. It's interesting that any migration failure is treated as -ENOMEM, but well, that's certainly material for a completely different discussion.
Thanks Hugh!
Signature · 3 lines
Cheers David / dhildenb
PATCH 3/7 mm/gup: local lru_add_drain() to avoid lru_add_drain_all()
In reply to Hugh DickinsIn many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all().
Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean backports: can take 6.16 commit a03db236aebf ("gup:
optimize longterm pin_user_pages() for large folio") first.
--- mm/gup.c | 2 ++ 1 file changed, 2 insertions(+)
mm/gup.c +2 -0
@@ -2291,6 +2291,8 @@ static unsigned long collect_longterm_unpinnable_folios(struct folio *folio;long i =0 ; +lru_add_drain (); +for (folio =pofs_get_folio (pofs, i); folio; folio =pofs_next_folio (folio, pofs, &i)) {
Signature · 1 line
2.51.0
Re: [PATCH 3/7] mm/gup: local lru_add_drain() to avoid lru_add_drain_all()
Quoted 21 lines · Hugh Dickins
In many cases, if
collect_longterm_unpinnable_folios()does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary locallru_add_drain(), than the later cross-CPUlru_add_drain_all().Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check
ref_countinstead of lru before migration". Note for clean backports: can take 6.16 commita03db236aebf("gup:
optimize longtermpin_user_pages()for large folio") first.Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- mm/gup.c | 2 ++ 1 file changed, 2 insertions(+)mm/gup.c +0
-0
mm/gup.c static unsigned long collect_longterm_unpinnable_folios( · line 2298
Quoted 8 lines
mm/gup.c +2
-0@@ -2291,6 +2291,8 @@ static unsigned long collect_longterm_unpinnable_folios(struct folio *folio;long i =0 ; +lru_add_drain (); +for (folio =pofs_get_folio (pofs, i); folio; folio =pofs_next_folio (folio, pofs, &i)) {
Do we really want to drain all the time we enter collect_longterm_unpinnable_folios(), or only if we detect an actual problem? (unexpected reference?)
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 3/7] mm/gup: local lru_add_drain() to avoid lru_add_drain_all()
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Hugh Dickins
In many cases, if
collect_longterm_unpinnable_folios()does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary locallru_add_drain(), than the later cross-CPUlru_add_drain_all().Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check
ref_countinstead of lru before migration". Note for clean backports: can take 6.16 commita03db236aebf("gup:
optimize longtermpin_user_pages()for large folio") first.Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- mm/gup.c | 2 ++ 1 file changed, 2 insertions(+)mm/gup.c +0
-0@@ -2291,6 +2291,8 @@ static unsigned longcollect_longterm_unpinnable_folios(
struct folio *folio; long i = 0;+
lru_add_drain();
Quoted 8 lines
Quoted 4 lines
+
for (folio = pofs_get_folio(pofs, i); folio; folio = pofs_next_folio(folio, pofs, &i)) {Do we really want to drain all the time we enter
collect_longterm_unpinnable_folios(), or only if we detect an actual problem? (unexpected reference?)
It looked nice and simple to me (hmm, where's the blank line before lru_add_drain() gone? weird, something wrong with my mail setup), I've never avoided an lru_add_drain() before; but you're right, we dom't need to do that every time, fixed in v2 - thanks.
Hugh
PATCH 4/7 mm: Revert "mm/gup: clear the LRU flag of a page before adding to LRU batch"
In reply to Hugh DickinsThis reverts commit 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9:
now that collect_longterm_unpinnable_folios() is checking ref_count instead of lru, and mlock/munlock do not participate in the revised LRU flag clearing, those changes are misleading, and enlarge the window during which mlock/munlock may miss an mlock_count update.
It is possible (I'd hesitate to claim probable) that the greater likelihood of missed mlock_count updates would explain the "Realtime threads delayed due to kcompactd0" observed on 6.12 in the Link below. If that is the case, this reversion will help; but a complete solution needs also a further patch, beyond the scope of this series.
Included some 80-column cleanup around folio_batch_add_and_move().
The role of folio_test_clear_lru() (before taking per-memcg lru_lock) is questionable since 6.13 removed mem_cgroup_move_account() etc; but perhaps there are still some races which need it - not examined here.
--- mm/swap.c | 50 ++++++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 24 deletions(-)
mm/swap.c +26 -24
@@ -164,6 +164,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)for (i =0 ; i <folio_batch_count (fbatch); i++) {struct folio *folio = fbatch->folios[i]; +/* block memcg migration while the folio moves between lru */ +if (move_fn != lru_add && !folio_test_clear_lru (folio)) +continue ; +folio_lruvec_relock_irqsave (folio, &lruvec, &flags);move_fn (lruvec, folio);
@@ -176,14 +180,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) }static void __folio_batch_add_and_move (struct folio_batch __percpu *fbatch, -struct folio *folio,move_fn_t move_fn, -bool on_lru,bool disable_irq) +struct folio *folio,move_fn_t move_fn,bool disable_irq) {unsigned long flags; -if (on_lru && !folio_test_clear_lru (folio)) -return ; -folio_get (folio);if (disable_irq)
@@ -191,8 +191,8 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,else local_lock (&cpu_fbatches.lock); -if (!folio_batch_add (this_cpu_ptr (fbatch), folio) ||folio_test_large (folio) || -lru_cache_disabled ()) +if (!folio_batch_add (this_cpu_ptr (fbatch), folio) || +folio_test_large (folio) ||lru_cache_disabled ())folio_batch_move_lru (this_cpu_ptr (fbatch), move_fn);if (disable_irq)
@@ -201,13 +201,13 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,local_unlock (&cpu_fbatches.lock); } -#define folio_batch_add_and_move (folio, op, on_lru) \ - __folio_batch_add_and_move( \ - &cpu_fbatches.op, \ - folio, \ - op, \ - on_lru, \ - offsetof(struct cpu_fbatches, op) >= offsetof(struct cpu_fbatches, lock_irq) \ +#define folio_batch_add_and_move (folio, op) \ + __folio_batch_add_and_move( \ + &cpu_fbatches.op, \ + folio, \ + op, \ + offsetof(struct cpu_fbatches, op) >= \ + offsetof(struct cpu_fbatches, lock_irq) \ )static void lru_move_tail (struct lruvec *lruvec,struct folio *folio)
@@ -231,10 +231,10 @@ static void lru_move_tail(struct lruvec *lruvec, struct folio *folio)void folio_rotate_reclaimable (struct folio *folio) {if (folio_test_locked (folio) ||folio_test_dirty (folio) || -folio_test_unevictable (folio)) +folio_test_unevictable (folio) || !folio_test_lru (folio))return ; -folio_batch_add_and_move (folio, lru_move_tail, true); +folio_batch_add_and_move (folio, lru_move_tail); } void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
@@ -328,10 +328,11 @@ static void folio_activate_drain(int cpu)void folio_activate (struct folio *folio) { -if (folio_test_active (folio) ||folio_test_unevictable (folio)) +if (folio_test_active (folio) ||folio_test_unevictable (folio) || + !folio_test_lru (folio))return ; -folio_batch_add_and_move (folio, lru_activate, true); +folio_batch_add_and_move (folio, lru_activate); }#else
@@ -507,7 +508,7 @@ void folio_add_lru(struct folio *folio) lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))folio_set_active (folio); -folio_batch_add_and_move (folio, lru_add, false); +folio_batch_add_and_move (folio, lru_add); }EXPORT_SYMBOL (folio_add_lru);
@@ -685,13 +686,13 @@ void lru_add_drain_cpu(int cpu)void deactivate_file_folio (struct folio *folio) {/* Deactivating an unevictable folio will not accelerate reclaim */ -if (folio_test_unevictable (folio)) +if (folio_test_unevictable (folio) || !folio_test_lru (folio))return ;if (lru_gen_enabled () &&lru_gen_clear_refs (folio))return ; -folio_batch_add_and_move (folio, lru_deactivate_file, true); +folio_batch_add_and_move (folio, lru_deactivate_file); }/*
@@ -704,13 +705,13 @@ void deactivate_file_folio(struct folio *folio)*/ void folio_deactivate (struct folio *folio) { -if (folio_test_unevictable (folio)) +if (folio_test_unevictable (folio) || !folio_test_lru (folio))return ;if (lru_gen_enabled () ?lru_gen_clear_refs (folio) : !folio_test_active (folio))return ; -folio_batch_add_and_move (folio, lru_deactivate, true); +folio_batch_add_and_move (folio, lru_deactivate); }/**
@@ -723,10 +724,11 @@ void folio_deactivate(struct folio *folio)void folio_mark_lazyfree (struct folio *folio) {if (!folio_test_anon (folio) || !folio_test_swapbacked (folio) || + !folio_test_lru (folio) ||folio_test_swapcache (folio) ||folio_test_unevictable (folio))return ; -folio_batch_add_and_move (folio, lru_lazyfree, true); +folio_batch_add_and_move (folio, lru_lazyfree); }void lru_add_drain (void )
Signature · 1 line
2.51.0
Re: [PATCH 4/7] mm: Revert "mm/gup: clear the LRU flag of a page before adding to LRU batch"
Quoted 14 lines · Hugh Dickins
This reverts commit
33dfe9204f29b415bbc0abb1a50642d1ba94f5e9:
now thatcollect_longterm_unpinnable_folios()is checkingref_countinstead of lru, and mlock/munlock do not participate in the revised LRU flag clearing, those changes are misleading, and enlarge the window during which mlock/munlock may miss anmlock_countupdate.It is possible (I'd hesitate to claim probable) that the greater likelihood of missed
mlock_countupdates would explain the "Realtime threads delayed due to kcompactd0" observed on 6.12 in the Link below. If that is the case, this reversion will help; but a complete solution needs also a further patch, beyond the scope of this series.Included some 80-column cleanup around
folio_batch_add_and_move().
Quoted 8 lines
The role of
folio_test_clear_lru()(before taking per-memcglru_lock) is questionable since 6.13 removedmem_cgroup_move_account()etc; but perhaps there are still some races which need it - not examined here.Link https://lore.kernel.org/linux-mm/DU0PR01MB10385345F7153F334100981888259A@DU0PR01MB10385.eurprd01.prod.exchangelabs.com/ open hereSigned-off-by Hugh Dickins <[email protected]>Cc <[email protected]>---
Signature · 3 lines
Cheers David / dhildenb
PATCH 5/7 mm: Revert "mm: vmscan.c: fix OOM on swap stress test"
In reply to Hugh DickinsThis reverts commit 0885ef4705607936fc36a38fd74356e1c465b023: that was a fix to the reverted 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9.
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -4507,7 +4507,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c }/* ineligible */ -if (!folio_test_lru (folio) || zone > sc->reclaim_idx) { +if (zone > sc->reclaim_idx) { gen =folio_inc_gen (lruvec, folio, false);list_move_tail (&folio->lru, &lrugen->folios[gen][type][zone]);return true;
Signature · 1 line
2.51.0
Re: [PATCH 5/7] mm: Revert "mm: vmscan.c: fix OOM on swap stress test"
Quoted 3 lines · Hugh Dickins
This reverts commit
0885ef4705607936fc36a38fd74356e1c465b023: that was a fix to the reverted33dfe9204f29b415bbc0abb1a50642d1ba94f5e9.
Likely checkpatch will not like the long commit ids.
Quoted 3 lines
Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>---
Signature · 3 lines
Cheers David / dhildenb
PATCH 6/7 mm: folio_may_be_cached() unless folio_test_large()
In reply to Hugh Dickinsmm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio.
But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).
So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.
Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration".
--- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 5 +++-- mm/mlock.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-)
include/linux/swap.h +10 -0
@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);void mark_page_accessed (struct page *);void folio_mark_accessed (struct folio *); +static inline bool folio_may_be_cached (struct folio *folio) +{ +/* +* Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. +* Holding small numbers of low-order mTHP folios in per-CPU LRU cache +* will be sensible, but nobody has implemented and tested that yet. +*/ +return !folio_test_large (folio); +} +extern atomic_t lru_disable_count;static inline bool lru_cache_disabled (void )
mm/gup.c +3 -2
@@ -2309,8 +2309,9 @@ static unsigned long collect_longterm_unpinnable_folios(continue ; } -if (drain_allow &&folio_ref_count (folio) != -folio_expected_ref_count (folio) +1 ) { +if (drain_allow &&folio_may_be_cached (folio) && +folio_ref_count (folio) != +folio_expected_ref_count (folio) +1 ) {lru_add_drain_all (); drain_allow = false; }
mm/mlock.c +3 -3
@@ -255,7 +255,7 @@ void mlock_folio(struct folio *folio)folio_get (folio);if (!folio_batch_add (fbatch,mlock_lru (folio)) || -folio_test_large (folio) ||lru_cache_disabled ()) + !folio_may_be_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch);local_unlock (&mlock_fbatch.lock); }
@@ -278,7 +278,7 @@ void mlock_new_folio(struct folio *folio)folio_get (folio);if (!folio_batch_add (fbatch,mlock_new (folio)) || -folio_test_large (folio) ||lru_cache_disabled ()) + !folio_may_be_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch);local_unlock (&mlock_fbatch.lock); }
@@ -299,7 +299,7 @@ void munlock_folio(struct folio *folio)*/ folio_get (folio);if (!folio_batch_add (fbatch, folio) || -folio_test_large (folio) ||lru_cache_disabled ()) + !folio_may_be_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch);local_unlock (&mlock_fbatch.lock); }
mm/swap.c +1 -1
@@ -192,7 +192,7 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,local_lock (&cpu_fbatches.lock);if (!folio_batch_add (this_cpu_ptr (fbatch), folio) || -folio_test_large (folio) ||lru_cache_disabled ()) + !folio_may_be_cached (folio) ||lru_cache_disabled ())folio_batch_move_lru (this_cpu_ptr (fbatch), move_fn);if (disable_irq)
Signature · 1 line
2.51.0
Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()
Quoted 31 lines · Hugh Dickins
mm/swap.candmm/mlock.cagree to drain any per-CPU batch as soon as a large folio is added: socollect_longterm_unpinnable_folios()just wastes effort when callinglru_add_drain_all()on a large folio.But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).
So ask if
folio_may_be_cached()rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check
ref_countinstead of lru before migration".Suggested-by David Hildenbrand <[email protected]>Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 5 +++-- mm/mlock.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-)include/linux/swap.h +0
-0
include/linux/swap.h void folio_add_lru_vma(struct folio *, struct vm_area_struct *); · line 387
Quoted 3 lines
include/linux/swap.h +0
-0@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);void mark_page_accessed (struct page *);void folio_mark_accessed (struct folio *);
Two smaller things:
(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.
(2) With things like fscache in mind, the function can be a bit misleading.
So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.
Thinking again, "maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.
Maybe it's more like "supports being cached".
folio_lru_caching_supported()
Something like that, maybe? (again, unclear about lru/mlock cache abstraction)
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()
Quoted 49 lines · David Hildenbrand
Quoted 31 lines · Hugh Dickins
mm/swap.candmm/mlock.cagree to drain any per-CPU batch as soon as a large folio is added: socollect_longterm_unpinnable_folios()just wastes effort when callinglru_add_drain_all()on a large folio.But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).
So ask if
folio_may_be_cached()rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check
ref_countinstead of lru before migration".Suggested-by David Hildenbrand <[email protected]>Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 5 +++-- mm/mlock.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h void folio_add_lru_vma(struct folio *, struct · line 388
Quoted 4 lines
include/linux/swap.h +0
-0@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);void mark_page_accessed (struct page *);void folio_mark_accessed (struct folio *);Two smaller things:
(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.
(2) With things like fscache in mind, the function can be a bit misleading.
So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it
folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.
Quoted 6 lines
Thinking again, "
maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.Maybe it's more like "supports being cached".
folio_lru_caching_supported()
folio_may_be_cached() -> folio_may_be_lru_cached(), yes, that's very much better, thanks.
(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but lru_cache_disable() gave us that terminology, and we've gone with the flow ever since. lru_add_drain() would be better named lru_cache_drain() now, I've always got hung up on "adding a drain".)
"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.
kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.
Hugh
Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()
Quoted 57 lines · Hugh Dickins
Quoted 49 lines · David Hildenbrand
Quoted 31 lines · Hugh Dickins
mm/swap.candmm/mlock.cagree to drain any per-CPU batch as soon as a large folio is added: socollect_longterm_unpinnable_folios()just wastes effort when callinglru_add_drain_all()on a large folio.But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented).
So ask if
folio_may_be_cached()rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch:
it's just the per-CPU LRU caches, drained much later, which need care.Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check
ref_countinstead of lru before migration".Suggested-by David Hildenbrand <[email protected]>Signed-off-by Hugh Dickins <[email protected]>Cc <[email protected]>--- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 5 +++-- mm/mlock.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h void folio_add_lru_vma(struct folio *, struct · line 388
Quoted 4 lines
include/linux/swap.h +0
-0@@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_struct *);void mark_page_accessed (struct page *);void folio_mark_accessed (struct folio *);Two smaller things:
(1) We have other "folio_maybe_*" functions, so this one should likely better start with that as well.
(2) With things like fscache in mind, the function can be a bit misleading.
So I wonder if (a) we should just add kerneldoc to document it clearly (lru cache, mlock cache?) and (b) maybe call it
folio_may_be_lru_cached(). Not sure if we can find a better abstraction for these two caches.Quoted 6 lines
Thinking again, "
maybe_cached" might be a bit misleading because it implements a very very very bad heuristic for small folios.Maybe it's more like "supports being cached".
folio_lru_caching_supported()
Quoted 7 lines
folio_may_be_cached()->folio_may_be_lru_cached(), yes, that's very much better, thanks.(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but
lru_cache_disable()gave us that terminology, and we've gone with the flow ever since.lru_add_drain()would be better namedlru_cache_drain()now, I've always got hung up on "adding a drain".)
Yeah, the terminology is not that intuitive :)
Not sure if using "batched" instead of "cached" might be clearer long-term?
Quoted 4 lines
"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.
I would wish we could find something that also non-native speakers can immediately understand ;)
"may_get_lru_cached" / "may_get_lru_batched"?
/me could not even phrase it in German properly
Quoted 4 lines
kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.
So a comment that this is an internal helper might be nice. Or we just move it straight to mm/internal.h ?
Signature · 3 lines
Cheers David / dhildenb
Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()
Quoted 1 line · David Hildenbrand
On 08.09.25 13:19, Hugh Dickins wrote:
...
Quoted 23 lines
Quoted 5 lines
(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but
lru_cache_disable()gave us that terminology, and we've gone with the flow ever since.lru_add_drain()would be better namedlru_cache_drain()now, I've always got hung up on "adding a drain".)Yeah, the terminology is not that intuitive :)
Not sure if using "batched" instead of "cached" might be clearer long-term?
Quoted 4 lines
"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.
I would wish we could find something that also non-native speakers can immediately understand ;)
"
may_get_lru_cached" / "may_get_lru_batched"?/me could not even phrase it in German properly
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.
So a comment that this is an internal helper might be nice. Or we just move it straight to
mm/internal.h?
mm/internal.h, where we hide things (GFP_RECLAIM_MASK etc!) that belong elsewhere? No thanks.
David, I think you're over-thinking this: I'm coming to regret not just going with your excellent folio_test_large() optimization, and let someone else mess around with the naming.
I'll post later with my current naming, but if it's the Suggested-by that's making you uncomfortable, we can delete it.
Hugh
Re: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large()
Quoted 30 lines · Hugh Dickins
Quoted 1 line · David Hildenbrand
On 08.09.25 13:19, Hugh Dickins wrote:
...
Quoted 27 lines
Quoted 5 lines
(Settimg aside that I've never perceived those pagevecs/batches as a "cache"; but
lru_cache_disable()gave us that terminology, and we've gone with the flow ever since.lru_add_drain()would be better namedlru_cache_drain()now, I've always got hung up on "adding a drain".)Yeah, the terminology is not that intuitive :)
Not sure if using "batched" instead of "cached" might be clearer long-term?
Quoted 4 lines
"may be" rather than "maybe" was intentional: perhaps too subtle, but to a native speaker it neatly expresses both the "we can do this" and "might this have been done" cases.
I would wish we could find something that also non-native speakers can immediately understand ;)
"
may_get_lru_cached" / "may_get_lru_batched"?/me could not even phrase it in German properly
Quoted 4 lines
kernel-doc? I don't think so, this is very much an mm-internal matter, and I don't care for the way kernel-doc forces us towards boilerplate ("@folio: The folio.") rather than helpful comment.
Quoted 5 lines
Quoted 2 lines
So a comment that this is an internal helper might be nice. Or we just move it straight to
mm/internal.h?
mm/internal.h, where we hide things (GFP_RECLAIM_MASKetc!) that belong elsewhere? No thanks.
Yes, or other mm-internal helpers that actually belong there. Like folio_raw_mapping() or folio_nr_pages_mapped().
Quoted 4 lines
David, I think you're over-thinking this: I'm coming to regret not just going with your excellent
folio_test_large()optimization, and let someone else mess around with the naming.
Please don't feel like I'm pushing to hard here. If you feel the current naming is fine and for some reason I don't completely understand it should not be in mm/internal.h, all good.
Signature · 3 lines
Cheers David / dhildenb
PATCH 7/7 mm: lru_add_drain_all() do local lru_add_drain() first
In reply to Hugh DickinsNo numbers to back this up, but it seemed obvious to me, that if there are competing lru_add_drain_all()ers, the work will be minimized if each flushes its own local queues before locking and doing cross-CPU drains.
--- mm/swap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/swap.c +3 -0
@@ -834,6 +834,9 @@ static inline void __lru_add_drain_all(bool force_all_cpus)*/ this_gen =smp_load_acquire (&lru_drain_gen); +/* It helps everyone if we do our own local drain immediately. */ +lru_add_drain (); +mutex_lock (&lock);/*
Signature · 1 line
2.51.0
Re: [PATCH 7/7] mm: lru_add_drain_all() do local lru_add_drain() first
Quoted 4 lines · Hugh Dickins
No numbers to back this up, but it seemed obvious to me, that if there are competing lru_add_drain_all()ers, the work will be minimized if each flushes its own local queues before locking and doing cross-CPU drains.
Sounds reasonable
Signature · 3 lines
Cheers David / dhildenb