Skip to contentSkip to conversation

mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()

PATCH Superseded Hui Zhu 2 participants 2 messages Oct 23

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH 1/1 mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing anything.

A MADV_FREE folio therefore keeps the refs and PG_workingset it had before madvise(2). lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Clear both flags here instead. lru_gen_clear_refs() only reports whether the folio is already in the oldest generation, which lru_lazyfree() does not need, and moving the call up would still miss the folios that are on the classic LRU while lru_gen_enabled() is true. Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN; PG_referenced gates LRU_REFS_MASK, so clearing it zeroes the refs count as well. PG_workingset is cleared only while MGLRU is running, which leaves the classic LRU unchanged.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")
---
 mm/folio.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
mm/folio.c +10 -4
index 47a437e0f7fd..03be101334eb 100644
@@ -599,10 +599,16 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
 
 	lruvec_del_folio(lruvec, folio);
 	folio_clear_active(folio);
-	if (lru_gen_enabled())
-		lru_gen_clear_refs(folio);
-	else
-		folio_clear_referenced(folio);
+	folio_clear_referenced(folio);
+	if (lru_gen_enabled()) {
+		/*
+		 * lru_gen_clear_refs() skips an off-list folio, which
+		 * lruvec_del_folio() has just made this one.  Left set,
+		 * PG_workingset keeps it in the last tier that sort_folio()
+		 * protects from eviction.
+		 */
+		folio_clear_workingset(folio);
+	}
 	/*
 	 * Lazyfree folios are clean anonymous folios.  They have
 	 * the swapbacked flag cleared, to distinguish them from normal
Signature · 1 line
2.53.0
#1 lore raw
Quoted 46 lines · Hui Zhu

From: Hui Zhu <[email protected]>

lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing anything.

A MADV_FREE folio therefore keeps the refs and PG_workingset it had before madvise(2). lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.

Clear both flags here instead. lru_gen_clear_refs() only reports whether the folio is already in the oldest generation, which lru_lazyfree() does not need, and moving the call up would still miss the folios that are on the classic LRU while lru_gen_enabled() is true. Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN; PG_referenced gates LRU_REFS_MASK, so clearing it zeroes the refs count as well. PG_workingset is cleared only while MGLRU is running, which leaves the classic LRU unchanged.

Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")
---
 mm/folio.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
mm/folio.c +0 -4
index 47a437e0f7fd..03be101334eb 100644
@@ -599,10 +599,16 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
 
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);
-       if (lru_gen_enabled())
-               lru_gen_clear_refs(folio);
-       else
-               folio_clear_referenced(folio);

mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 611

Quoted 10 lines
mm/folio.c +10 -0
+       folio_clear_referenced(folio);
+       if (lru_gen_enabled()) {
+               /*
+                * lru_gen_clear_refs() skips an off-list folio, which
+                * lruvec_del_folio() has just made this one.  Left set,
+                * PG_workingset keeps it in the last tier that sort_folio()
+                * protects from eviction.
+                */
+               folio_clear_workingset(folio);
+       }

So we can't do the below? But why?

mm/folio.c +0 -0
index 45c814f3857b..6b779a1ad2f6 100644
@@ -611,12 +611,10 @@ static void lru_lazyfree(struct lruvec *lruvec,

struct folio *folio)

            folio_test_swapcache(folio) || folio_test_unevictable(folio))
                return;
+       lru_gen_clear_refs(folio);
        lruvec_del_folio(lruvec, folio);
        folio_clear_active(folio);

- if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
- else
- folio_clear_referenced(folio);
+ folio_clear_referenced(folio);

Best Regards
Barry