Skip to contentSkip to conversation

mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()

PATCH Superseded Hui Zhu 3 participants 12 messages Oct 26 – Nov 7

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v2 1/1 mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +2 -4
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
 	    folio_test_swapcache(folio) || folio_test_unevictable(folio))
 		return;
 
+	lru_gen_clear_refs(folio);
 	lruvec_del_folio(lruvec, folio);
 	folio_clear_active(folio);
-	if (lru_gen_enabled())
-		lru_gen_clear_refs(folio);
-	else
-		folio_clear_referenced(folio);
+	folio_clear_referenced(folio);
 	/*
 	 * Lazyfree folios are clean anonymous folios.  They have
 	 * the swapbacked flag cleared, to distinguish them from normal
Signature · 1 line
2.53.0
reviewer Reviewed-by on v2 1/1
#1 lore raw
Quoted 45 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 600

Quoted 4 lines
mm/folio.c +1 -0
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
 
+       lru_gen_clear_refs(folio);

The only thing I'm not sure about is whether we need if (lru_gen_enabled()) here to keep the semantics consistent with other places where we use it.

...
2 676 mm/folio.c <<deactivate_file_folio>>

             if (lru_gen_enabled() && lru_gen_clear_refs(folio))

3 695 mm/folio.c <<folio_deactivate>>
if (lru_gen_enabled() ? lru_gen_clear_refs(folio) :
!folio_test_active(folio))

So maybe we still need

+ if (lru_gen_enabled())

        lru_gen_clear_refs(folio);

with that, feel free to add:

author
#2 lore raw

2026/10/26 17:47, Barry Song:

Quoted 58 lines · Hui Zhu
Quoted 44 lines

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

  mm/folio.c | 6 ++----
  1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 600

Quoted 4 lines
mm/folio.c +1 -0
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
             folio_test_swapcache(folio) || folio_test_unevictable(folio))
                 return;
 
+       lru_gen_clear_refs(folio);

The only thing I'm not sure about is whether we need if (lru_gen_enabled()) here to keep the semantics consistent with other places where we use it.

...

    2    676  mm/folio.c <<deactivate_file_folio>>
              if (lru_gen_enabled() && lru_gen_clear_refs(folio))
    3    695  mm/folio.c <<folio_deactivate>>
              if (lru_gen_enabled() ? lru_gen_clear_refs(folio) :
Quoted 6 lines

!folio_test_active(folio))

So maybe we still need

+ if (lru_gen_enabled())

         lru_gen_clear_refs(folio);

I think we maybe don't need it here.  The other two sites consume the return value, and that is what the check disambiguates:

    if (gen < 0)
        return true;

An off-list folio has gen == -1, so without lru_gen_enabled() both deactivate_file_folio() and folio_deactivate() would take that "true" and return early, skipping the deactivation.  lru_lazyfree() ignores the return value, since it re-adds the folio a few lines below anyway.

The side effects gate themselves, and on something more precise than the global key: a folio on a classic list always has gen == -1, which fill_evictable() even asserts,

    VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);

and drain_evictable() clears LRU_GEN_MASK on the way out.  So an unguarded call either clears the refs of a folio that is on a generation list, or does nothing at all - no flag writes and no lruvec access.  The classic LRU cannot be affected, including with CONFIG_LRU_GEN=n, where the stub inlines away.

Guarding on the key would reopen a narrower version of this bug. lru_gen_change_state() flips it before migrating any folio:

    if (enabled) static_branch_enable_cpuslocked(&lru_gen_caps[LRU_GEN_CORE]); else static_branch_disable_cpuslocked(&lru_gen_caps[LRU_GEN_CORE]);

    memcg = mem_cgroup_iter(NULL, NULL, NULL);

so while MGLRU is being turned off the key is already false, but every lruvec that drain_evictable() has not reached yet still has its folios on generation lists.  A MADV_FREE batch drained on another CPU in that window would keep PG_workingset.

What about add a comment instead of the check:

    /* no-op unless the folio is on a generation list */ lru_gen_clear_refs(folio);

Best,
Hui

Quoted 4 lines

with that, feel free to add:

reviewer Reviewed-by
#3 lore raw

On Tue, Oct 27, 2026 at 3:25 PM Hui Zhu <[email protected]> wrote:
[...]

Quoted 12 lines
Quoted 12 lines
Quoted 1 line

+ lru_gen_clear_refs(folio);

The only thing I'm not sure about is whether we need if (lru_gen_enabled()) here to keep the semantics consistent with other places where we use it.

...

    2    676  mm/folio.c <<deactivate_file_folio>>
              if (lru_gen_enabled() && lru_gen_clear_refs(folio))
    3    695  mm/folio.c <<folio_deactivate>>
              if (lru_gen_enabled() ? lru_gen_clear_refs(folio) :

!folio_test_active(folio))

Quoted 8 lines
Quoted 4 lines

So maybe we still need

+ if (lru_gen_enabled())

         lru_gen_clear_refs(folio);

I think we maybe don't need it here. The other two sites consume the return value, and that is what the check disambiguates:

You’re right. Thanks for the thorough investigation.

#4 lore raw
In reply to Hui Zhu
Quoted 46 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 10 lines
mm/folio.c +2 -4
                return;
 
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

And folio_set_lru_refs(.., 0) should be a little bit faster and unify CLRU / MGLRU. And we can detect lazy freed folios in sort_folios to avoid PID form touching them?

Something like:

mm/folio.c +0 -0
index 35e242b48870..416cd4beb182 100644
@@ -599,10 +599,7 @@ static void lru_lazyfree(struct lruvec *lruvec,

struct folio *folio)

        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);

- if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
- else
- folio_clear_referenced(folio);
+ folio_set_lru_refs(folio, 0);
/*
* Lazyfree folios are clean anonymous folios. They have
* the swapbacked flag cleared, to distinguish them from normal

mm/vmscan.c +0 -0
index 6a1af9b92349..e64b7f544d15 100644
@@ -4835,6 +4835,10 @@ static bool sort_folio(struct lruvec *lruvec,

struct folio *folio, struct scan_c

        if (!folio_evictable(folio))
                return false;

+ /* No need to protect lazyfreed folios */
+ if (folio_test_lazyfree(folio) && !folio_test_dirty(folio))
+ return false;
+

        /* promoted */
        if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
                list_move(&folio->lru, &lrugen->folios[gen][type][zone]);

---

BTW I also tried to cover this in MGLRU-FG series (not including the sort_folio part):
https://lore.kernel.org/linux-mm/[email protected]/

I tested keeping min(refs, LRU_REFS_WORKINGSET) / just set to 0 as in v4, nothing different from the workload (testing using Android apps as workload), so I think it's fine either way but the above patch I posted might help to impove it by a little?

And if CLRU didn't actually expect PG_workingset to be kept but that's just a leftover or trivial, maybe we can always do both folio_clear_workingset and folio_set_lru_refs(0) here?

reviewer
#5 lore raw
Quoted 50 lines · Kairui Song
Quoted 48 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
 
Quoted 13 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

I assume we can support this, but doesn't the code below also look a bit weird?

include/linux/mm_inline.h +0 -0
index 829977d399e6..66ed805a48f5 100644
@@ -431,6 +431,15 @@ static inline void folio_migrate_lru_refs(struct

folio *new, const struct folio

        if (folio_test_referenced(old))
                folio_set_referenced(new);

}
+
+static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs) +{
+ if (!refs)
+ folio_clear_referenced(folio);
+ else if (refs == 1)
+ folio_set_referenced(folio);
+}
+
#endif /* CONFIG_LRU_GEN */

[1] https://lore.kernel.org/linux-mm/[email protected]/

Best,
Barry

#6 lore raw
Quoted 66 lines · Barry Song
Quoted 50 lines · Kairui Song
Quoted 48 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
 
Quoted 13 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

Quoted 6 lines

I recall that in v1 [1], Hui mentioned:

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

PG_workingseet gets cleaned too, but if we thing carefully, it might be actually the right move, even for CLRU. Stepping back: what PG_referenced/PG_workingset encode is "this data has been accessed repeatedly, so protect it". MADV_FREE is the caller saying those bytes are garbage, the allocation that produced those accesses is gone, and whoever gets the memory next, could be a different owner with a different access pattern.

#7 lore raw
Quoted 70 lines · Kairui Song
Quoted 68 lines · Barry Song
Quoted 50 lines · Kairui Song
Quoted 48 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing the refs or PG_workingset.

A MADV_FREE folio therefore keeps the state it had before madvise(2), even though madvise_free_pte_range() has already cleared the accessed and dirty bits of its PTEs. lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Call lru_gen_clear_refs() while the folio is still on the generation list, as deactivate_file_folio() and folio_deactivate() already do. Its return value only says whether the folio is in the oldest generation, which lru_lazyfree() does not need because it re-adds the folio below. Keep folio_clear_referenced() unconditional: the classic LRU needs it, and it also covers the folios that lru_gen_clear_refs() skips.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")

---
Changelog:
v2:
Call lru_gen_clear_refs() before lruvec_del_folio() instead of open-coding the flag clears; drop the lru_gen_enabled() branch and the comment (Barry Song).

 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
 
Quoted 13 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

Quoted 7 lines
Quoted 4 lines

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

And it will be better if we can keep this operation off-list just like right now :), some later works could benefit from it.

reviewer
#8 lore raw

On Sat, Oct 31, 2026 at 3:27 PM Kairui Song <[email protected]> wrote:
[...]

Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 12 lines
Quoted 12 lines
 mm/folio.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
 
Quoted 13 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Quoted 5 lines
Quoted 2 lines

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

And it will be better if we can keep this operation off-list just like right now :), some later works could benefit from it.

lru_gen_clear_refs() is currently multifunctional. It clears refs/workingset, but also returns true to skip folio_batch_add_and_move() in both deactivate_file_folio() and folio_deactivate().

This actually looks a bit weird. However, following the current logic, I've drafted a version that we can use as a starting point for discussion:

mm/folio.c +5 -3
index dc2c796a4ea7..bc17cd3b6b6c 100644
@@ -444,12 +444,12 @@ static bool lru_gen_clear_refs(struct folio *folio)
        int type = folio_is_file_lru(folio);
        unsigned long seq;
 
-       if (gen < 0)
-               return true;
-
        folio_set_lru_refs(folio, 0);
        folio_clear_workingset(folio);
 
+       if (gen < 0)
+               return true;
+
        rcu_read_lock();
        seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
        rcu_read_unlock();
@@ -465,6 +465,8 @@ static void lru_gen_inc_refs(struct folio *folio)
 
 static bool lru_gen_clear_refs(struct folio *folio)
 {
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
        return false;
 }
 

The another option is:

mm/folio.c +10 -4
index dc2c796a4ea7..8d1fd788188c 100644
@@ -438,6 +438,12 @@ static void lru_gen_inc_refs(struct folio *folio)
        } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
+static inline void __lru_gen_clear_refs(struct folio *folio)
+{
+       folio_set_lru_refs(folio, 0);
+       folio_clear_workingset(folio);
+}
+
 static bool lru_gen_clear_refs(struct folio *folio)
 {
        int gen = folio_lru_gen(folio);
@@ -447,8 +453,7 @@ static bool lru_gen_clear_refs(struct folio *folio)
        if (gen < 0)
                return true;
 
-       folio_set_lru_refs(folio, 0);
-       folio_clear_workingset(folio);
+       __lru_gen_clear_refs(folio);
 
        rcu_read_lock();
        seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -463,9 +468,10 @@ static void lru_gen_inc_refs(struct folio *folio)
 {
 }
 
-static bool lru_gen_clear_refs(struct folio *folio)
+static inline void __lru_gen_clear_refs(struct folio *folio)
 {
-       return false;
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
 }
 
 #endif /* CONFIG_LRU_GEN */
@@ -659,10 +665,7 @@ static void lru_lazyfree(struct lruvec *lruvec,

struct folio *folio)

        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);

- if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
- else
- folio_clear_referenced(folio);
+ __lru_gen_clear_refs(folio);
/*
* Lazyfree folios are clean anonymous folios. They have
* the swapbacked flag cleared, to distinguish them from normal

I would personally prefer the latter. What do you think, Hui and Kairui?

Best,
Barry

author
#9 lore raw

2026/11/6 05:13, Barry Song:

Quoted 127 lines · Kairui Song

[...]

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
  mm/folio.c | 6 ++----
  1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
             folio_test_swapcache(folio) || folio_test_unevictable(folio))
                 return;
 
Quoted 12 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

And it will be better if we can keep this operation off-list just like right now :), some later works could benefit from it.

lru_gen_clear_refs() is currently multifunctional. It clears refs/workingset, but also returns true to skip folio_batch_add_and_move() in both deactivate_file_folio() and folio_deactivate().

This actually looks a bit weird. However, following the current logic, I've drafted a version that we can use as a starting point for discussion:

mm/folio.c +5 -3
index dc2c796a4ea7..bc17cd3b6b6c 100644
@@ -444,12 +444,12 @@ static bool lru_gen_clear_refs(struct folio *folio)
         int type = folio_is_file_lru(folio);
         unsigned long seq;
 
-       if (gen < 0)
-               return true;
-
         folio_set_lru_refs(folio, 0);
         folio_clear_workingset(folio);
 
+       if (gen < 0)
+               return true;
+
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
         rcu_read_unlock();
@@ -465,6 +465,8 @@ static void lru_gen_inc_refs(struct folio *folio)
 
  static bool lru_gen_clear_refs(struct folio *folio)
  {
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
         return false;
  }
 

The another option is:

mm/folio.c +11 -8
index dc2c796a4ea7..8d1fd788188c 100644
@@ -438,6 +438,12 @@ static void lru_gen_inc_refs(struct folio *folio)
         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
  }
 
+static inline void __lru_gen_clear_refs(struct folio *folio)
+{
+       folio_set_lru_refs(folio, 0);
+       folio_clear_workingset(folio);
+}
+
  static bool lru_gen_clear_refs(struct folio *folio)
  {
         int gen = folio_lru_gen(folio);
@@ -447,8 +453,7 @@ static bool lru_gen_clear_refs(struct folio *folio)
         if (gen < 0)
                 return true;
 
-       folio_set_lru_refs(folio, 0);
-       folio_clear_workingset(folio);
+       __lru_gen_clear_refs(folio);
 
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -463,9 +468,10 @@ static void lru_gen_inc_refs(struct folio *folio)
  {
  }
 
-static bool lru_gen_clear_refs(struct folio *folio)
+static inline void __lru_gen_clear_refs(struct folio *folio)
  {
-       return false;
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
  }
 
  #endif /* CONFIG_LRU_GEN */
@@ -659,10 +665,7 @@ static void lru_lazyfree(struct lruvec *lruvec,
 struct folio *folio)
 
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       __lru_gen_clear_refs(folio);
         /*
          * Lazyfree folios are clean anonymous folios.  They have
Quoted 1 line

* the swapbacked flag cleared, to distinguish them from normal

Hi Barry, Kairui,

Quoted 2 lines

I would personally prefer the latter. What do you think, Hui and Kairui?

I prefer option B too: it keeps the flag clearing independent of the list state and leaves the two deactivate callers untouched.

But I'd suggest splitting this in two:

1. Take v2 as the fix now. If that is acceptable, I will respin it with Cc: stable added.

2. Follow up with option B as a pure cleanup, no Fixes tag. The lazyfree site then becomes the off-list __lru_gen_clear_refs() call that Kairui wants for later work.

The reason is backporting. Every stable from 6.14 (cc8ec7be78ff) to 7.3 carries the regression, but B is written against folio_set_lru_refs(), which only landed in this merge window (64169ceea5b1e). None of the affected trees has that helper, so backporting B would drag the refs-flags rework along with it. v2 applies to all of them with only the mm/swap.c -> mm/folio.c rename to adjust.

B also starts clearing PG_workingset on the classic LRU. I agree with Kairui that this is the right semantics for MADV_FREE, but it is a change beyond this regression and deserves its own patch - it fits naturally in the MGLRU-FG flag cleanup, as does the sort_folio() lazyfree check.

Best,
Hui

Quoted 2 lines

Best,
Barry

reviewer
#10 lore raw
Quoted 141 lines · Hui Zhu

2026/11/6 05:13, Barry Song:

Quoted 127 lines · Kairui Song

[...]

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
  mm/folio.c | 6 ++----
  1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
             folio_test_swapcache(folio) || folio_test_unevictable(folio))
                 return;
 
Quoted 12 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

And it will be better if we can keep this operation off-list just like right now :), some later works could benefit from it.

lru_gen_clear_refs() is currently multifunctional. It clears refs/workingset, but also returns true to skip folio_batch_add_and_move() in both deactivate_file_folio() and folio_deactivate().

This actually looks a bit weird. However, following the current logic, I've drafted a version that we can use as a starting point for discussion:

mm/folio.c +5 -3
index dc2c796a4ea7..bc17cd3b6b6c 100644
@@ -444,12 +444,12 @@ static bool lru_gen_clear_refs(struct folio *folio)
         int type = folio_is_file_lru(folio);
         unsigned long seq;
 
-       if (gen < 0)
-               return true;
-
         folio_set_lru_refs(folio, 0);
         folio_clear_workingset(folio);
 
+       if (gen < 0)
+               return true;
+
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
         rcu_read_unlock();
@@ -465,6 +465,8 @@ static void lru_gen_inc_refs(struct folio *folio)
 
  static bool lru_gen_clear_refs(struct folio *folio)
  {
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
         return false;
  }
 

The another option is:

mm/folio.c +11 -8
index dc2c796a4ea7..8d1fd788188c 100644
@@ -438,6 +438,12 @@ static void lru_gen_inc_refs(struct folio *folio)
         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
  }
 
+static inline void __lru_gen_clear_refs(struct folio *folio)
+{
+       folio_set_lru_refs(folio, 0);
+       folio_clear_workingset(folio);
+}
+
  static bool lru_gen_clear_refs(struct folio *folio)
  {
         int gen = folio_lru_gen(folio);
@@ -447,8 +453,7 @@ static bool lru_gen_clear_refs(struct folio *folio)
         if (gen < 0)
                 return true;
 
-       folio_set_lru_refs(folio, 0);
-       folio_clear_workingset(folio);
+       __lru_gen_clear_refs(folio);
 
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -463,9 +468,10 @@ static void lru_gen_inc_refs(struct folio *folio)
  {
  }
 
-static bool lru_gen_clear_refs(struct folio *folio)
+static inline void __lru_gen_clear_refs(struct folio *folio)
  {
-       return false;
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
  }
 
  #endif /* CONFIG_LRU_GEN */
@@ -659,10 +665,7 @@ static void lru_lazyfree(struct lruvec *lruvec,
 struct folio *folio)
 
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       __lru_gen_clear_refs(folio);
         /*
          * Lazyfree folios are clean anonymous folios.  They have
Quoted 1 line

* the swapbacked flag cleared, to distinguish them from normal

Hi Barry, Kairui,

Quoted 2 lines

I would personally prefer the latter. What do you think, Hui and Kairui?

I prefer option B too: it keeps the flag clearing independent of the list state and leaves the two deactivate callers untouched.

Quoted 8 lines

But I'd suggest splitting this in two:

1. Take v2 as the fix now. If that is acceptable, I will respin it with Cc: stable added.

2. Follow up with option B as a pure cleanup, no Fixes tag. The lazyfree site then becomes the off-list __lru_gen_clear_refs() call that Kairui wants for later work.

Makes sense to me. I also personally hope this can be merged into mainline soon so that we can backport it to the Android common kernel. Having lazyfree folios with hotness information sitting on the file LRU really makes little sense.

I recall I once did the below to ensure that lazyfree folios get higher reclaim priority:

https://lore.kernel.org/linux-mm/[email protected]/

And it significantly improves Android performance, according to Gaoxu from Honor:

https://lore.kernel.org/linux-mm/[email protected]/

I didn't pursue that approach further, but at least yours should address part of the problem.

Best,
Barry

#11 lore raw
In reply to Hui Zhu
Quoted 136 lines · Hui Zhu

2026/11/6 05:13, Barry Song:

Quoted 127 lines · Kairui Song

[...]

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
  mm/folio.c | 6 ++----
  1 file changed, 2 insertions(+), 4 deletions(-)
mm/folio.c +0 -0
index 35e242b48870..ea3adb6a89d7 100644
@@ -597,12 +597,10 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
             folio_test_swapcache(folio) || folio_test_unevictable(folio))
                 return;
 
Quoted 12 lines

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 603

Quoted 8 lines
mm/folio.c +2 -4
+       lru_gen_clear_refs(folio);
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       folio_clear_referenced(folio);

Why can't we just call folio_set_lru_refs(.., 0), and should we clear PG_workingset? Keeping it helps PSI on refualt / redirty tracking (not sure if meaningful but CLRU does this), clearing it makes eviction better, but the eviction issue is MGLRU-only, CLRU keeps it.

I recall that in v1 [1], Hui mentioned:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

"Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN."

Or prehaps we can have a folio_clear_lru_refs() do in a unified way for For both CLRU / MGLRU? (Just like folio_migrate_lru_refs).

And it will be better if we can keep this operation off-list just like right now :), some later works could benefit from it.

lru_gen_clear_refs() is currently multifunctional. It clears refs/workingset, but also returns true to skip folio_batch_add_and_move() in both deactivate_file_folio() and folio_deactivate().

This actually looks a bit weird. However, following the current logic, I've drafted a version that we can use as a starting point for discussion:

mm/folio.c +5 -3
index dc2c796a4ea7..bc17cd3b6b6c 100644
@@ -444,12 +444,12 @@ static bool lru_gen_clear_refs(struct folio *folio)
         int type = folio_is_file_lru(folio);
         unsigned long seq;
 
-       if (gen < 0)
-               return true;
-
         folio_set_lru_refs(folio, 0);
         folio_clear_workingset(folio);
 
+       if (gen < 0)
+               return true;
+
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
         rcu_read_unlock();
@@ -465,6 +465,8 @@ static void lru_gen_inc_refs(struct folio *folio)
 
  static bool lru_gen_clear_refs(struct folio *folio)
  {
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
         return false;
  }
 

The another option is:

mm/folio.c +11 -8
index dc2c796a4ea7..8d1fd788188c 100644
@@ -438,6 +438,12 @@ static void lru_gen_inc_refs(struct folio *folio)
         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
  }
 
+static inline void __lru_gen_clear_refs(struct folio *folio)
+{
+       folio_set_lru_refs(folio, 0);
+       folio_clear_workingset(folio);
+}
+
  static bool lru_gen_clear_refs(struct folio *folio)
  {
         int gen = folio_lru_gen(folio);
@@ -447,8 +453,7 @@ static bool lru_gen_clear_refs(struct folio *folio)
         if (gen < 0)
                 return true;
 
-       folio_set_lru_refs(folio, 0);
-       folio_clear_workingset(folio);
+       __lru_gen_clear_refs(folio);
 
         rcu_read_lock();
         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -463,9 +468,10 @@ static void lru_gen_inc_refs(struct folio *folio)
  {
  }
 
-static bool lru_gen_clear_refs(struct folio *folio)
+static inline void __lru_gen_clear_refs(struct folio *folio)
  {
-       return false;
+       folio_clear_referenced(folio);
+       folio_clear_workingset(folio);
  }
 
  #endif /* CONFIG_LRU_GEN */
@@ -659,10 +665,7 @@ static void lru_lazyfree(struct lruvec *lruvec,
 struct folio *folio)
 
         lruvec_del_folio(lruvec, folio);
         folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);
+       __lru_gen_clear_refs(folio);
         /*
          * Lazyfree folios are clean anonymous folios.  They have
Quoted 1 line

* the swapbacked flag cleared, to distinguish them from normal

Hi Barry, Kairui,

Quoted 2 lines

I would personally prefer the latter. What do you think, Hui and Kairui?

Quoted 7 lines

I prefer option B too: it keeps the flag clearing independent of the list state and leaves the two deactivate callers untouched.

But I'd suggest splitting this in two:

1. Take v2 as the fix now. If that is acceptable, I will respin it with Cc: stable added.

OK, that makes sense, agree that we can patch that regression first as a clear backport indeed. Thanks!