mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()
Subsystems (2)
PATCH 1/1 mm/mglru: fix a no-op lru_gen_clear_refs() call in lru_lazyfree()
From: Hui Zhu <[email protected]>
lru_lazyfree() calls lru_gen_clear_refs() after lruvec_del_folio(), but lru_gen_del_folio() has already cleared LRU_GEN_MASK, so folio_lru_gen() returns -1 and lru_gen_clear_refs() exits at its "gen < 0" check without clearing anything.
A MADV_FREE folio therefore keeps the refs and PG_workingset it had before madvise(2). lru_gen_folio_seq() places it one generation younger, and PG_workingset puts it in the last tier, so sort_folio() promotes and protects it instead of evicting it and charges it to lrugen->protected[], which read_ctrl_pos() uses to balance anon against file reclaim. __folio_inc_gen() clears the refs but not PG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first, ttu_anon_lazyfree_folio() keeps it and lru_gen_eviction() packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.
Clear both flags here instead. lru_gen_clear_refs() only reports whether the folio is already in the oldest generation, which lru_lazyfree() does not need, and moving the call up would still miss the folios that are on the classic LRU while lru_gen_enabled() is true. Use folio_clear_referenced() rather than folio_set_lru_refs(), which is not defined without CONFIG_LRU_GEN; PG_referenced gates LRU_REFS_MASK, so clearing it zeroes the refs count as well. PG_workingset is cleared only while MGLRU is running, which leaves the classic LRU unchanged.
--- mm/folio.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
mm/folio.c +10 -4
@@ -599,10 +599,16 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio); -if (lru_gen_enabled ()) -lru_gen_clear_refs (folio); -else -folio_clear_referenced (folio); +folio_clear_referenced (folio); +if (lru_gen_enabled ()) { +/* +* lru_gen_clear_refs() skips an off-list folio, which +* lruvec_del_folio() has just made this one. Left set, +* PG_workingset keeps it in the last tier that sort_folio() +* protects from eviction. +*/ +folio_clear_workingset (folio); + }/* * Lazyfree folios are clean anonymous folios. They have * the swapbacked flag cleared, to distinguish them from normal
Signature · 1 line
2.53.0
Quoted 46 lines · Hui Zhu
From: Hui Zhu <[email protected]>
lru_lazyfree()callslru_gen_clear_refs()afterlruvec_del_folio(), butlru_gen_del_folio()has already clearedLRU_GEN_MASK, sofolio_lru_gen()returns -1 andlru_gen_clear_refs()exits at its "gen < 0" check without clearing anything.A
MADV_FREEfolio therefore keeps the refs andPG_workingsetit had before madvise(2).lru_gen_folio_seq()places it one generation younger, andPG_workingsetputs it in the last tier, sosort_folio()promotes and protects it instead of evicting it and charges it to lrugen->protected[], whichread_ctrl_pos()uses to balance anon against file reclaim.__folio_inc_gen()clears the refs but notPG_workingset, so this repeats every time the folio reaches the oldest generation. If the folio is redirtied first,ttu_anon_lazyfree_folio()keeps it andlru_gen_eviction()packs the stale state into the shadow entry, so a later refault is promoted to the youngest generation.Clear both flags here instead.
lru_gen_clear_refs()only reports whether the folio is already in the oldest generation, whichlru_lazyfree()does not need, and moving the call up would still miss the folios that are on the classic LRU whilelru_gen_enabled()is true. Usefolio_clear_referenced()rather thanfolio_set_lru_refs(), which is not defined withoutCONFIG_LRU_GEN;PG_referencedgatesLRU_REFS_MASK, so clearing it zeroes the refs count as well.PG_workingsetis cleared only while MGLRU is running, which leaves the classic LRU unchanged.Fixes cc8ec7be78ff ("mm/mglru: optimize deactivation")Signed-off-by Hui Zhu <[email protected]>--- mm/folio.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)mm/folio.c +0
-4@@ -599,10 +599,16 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio); -if (lru_gen_enabled ()) -lru_gen_clear_refs (folio); -else -folio_clear_referenced (folio);
mm/folio.c static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio) · line 611
Quoted 10 lines
mm/folio.c +10
-0+folio_clear_referenced (folio); +if (lru_gen_enabled ()) { +/* +* lru_gen_clear_refs() skips an off-list folio, which +* lruvec_del_folio() has just made this one. Left set, +* PG_workingset keeps it in the last tier that sort_folio() +* protects from eviction. +*/ +folio_clear_workingset (folio); + }
So we can't do the below? But why?
mm/folio.c +0 -0
@@ -611,12 +611,10 @@ static void lru_lazyfree(struct lruvec *lruvec,
struct folio *folio)
folio_test_swapcache(folio) || folio_test_unevictable(folio))
return;+ lru_gen_clear_refs(folio);
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);- if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
- else
- folio_clear_referenced(folio);
+ folio_clear_referenced(folio);
Best Regards
Barry