mm/mglru: preserve inactive placement when enabling MGLRU
Subsystems (3)
PATCH 1/1 mm/mglru: preserve inactive placement when enabling MGLRU
From: Ridong Chen <[email protected]>
When the LRU is switched to MGLRU (echo y > /sys/kernel/mm/lru_gen/ enabled), fill_evictable() re-inserts every folio via lru_gen_add_folio(..., false). With reclaiming hardcoded to false, an inactive anonymous folio (no PG_active, not in the swapcache) takes the "gen = MIN_NR_GENS" branch in lru_gen_folio_seq() and is seeded at seq = max_seq - 1, which lru_gen_is_active() treats as active. Its inactive placement is lost and NR_INACTIVE_ANON is folded into NR_ACTIVE_ANON.
Pass reclaiming=!active so a folio from an inactive list is seeded into an older generation. Folios from the active list carry PG_active and hit the first branch either way, so they are unchanged.
reclaiming also selects the insertion end in lru_gen_add_folio():list_add_tail() for inactive folios, list_add() for active ones. Both the legacy LRU and a MGLRU generation keep the hottest folios at the head and the coldest at the tail, and reclaim takes from the tail. To preserve that order the folio must be taken from the end matching the insertion end, so take inactive folios from the head and active folios from the tail; otherwise hot/cold would be reversed within the generation.
Tested on x86_64, next-20260812, 2G VM + 1G swap, ~1.5G anon pushed onto the inactive list before enabling MGLRU:
Active(anon) Inactive(anon)
before switch (legacy) 2952 1548792 kB after `echo y`, unpatched 1552052 0 kB after `echo y`, patched 15144 1536636 kB
Inactive file folios stay inactive either way (NR_INACTIVE_FILE is preserved).
--- mm/vmscan.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
mm/vmscan.c +27 -2
@@ -5311,7 +5311,23 @@ static bool fill_evictable(struct lruvec *lruvec)while (!list_empty (head)) {bool success; -struct folio *folio =lru_to_folio (head); +struct folio *folio; + +/* +* Both the legacy LRU and a MGLRU generation keep the +* hottest folios at the head and the coldest at the +* tail, and reclaim takes from the tail. To preserve +* that order, the end we take from must match the end +* lru_gen_add_folio() inserts at: inactive folios use +* reclaiming=true (list_add_tail), so take from the +* head; active folios use reclaiming=false (list_add), +* so take from the tail. Taking from the wrong end +* would reverse hot/cold within the generation. +*/ +if (active) + folio =lru_to_folio (head); +else + folio = list_first_entry(head, struct folio, lru);VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio) != active, folio);
@@ -5319,7 +5335,16 @@ static bool fill_evictable(struct lruvec *lruvec)VM_WARN_ON_ONCE_FOLIO (folio_lru_gen (folio) !=-1 , folio);lruvec_del_folio (lruvec, folio); - success =lru_gen_add_folio (lruvec, folio, false); +/* +* With reclaiming=false, lru_gen_folio_seq() would seed +* an inactive folio near max_seq, which +* lru_gen_is_active() reports as active, so its inactive +* placement would be lost. Pass reclaiming=!active to +* seed it into the oldest generation instead. This +* reuses reclaiming beyond its folio_rotate_reclaimable() +* meaning; it also picks list_add_tail() above. +*/ + success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success);if (!--remaining)
Signature · 1 line
2.34.1
Quoted 56 lines · Ridong Chen
From: Ridong Chen <[email protected]>
When the LRU is switched to MGLRU (echo y >
/sys/kernel/mm/lru_gen/enabled),fill_evictable()re-inserts every folio vialru_gen_add_folio(..., false). With reclaiming hardcoded to false, an inactive anonymous folio (noPG_active, not in the swapcache) takes the "gen =MIN_NR_GENS" branch inlru_gen_folio_seq()and is seeded at seq =max_seq- 1, whichlru_gen_is_active()treats as active. Its inactive placement is lost andNR_INACTIVE_ANONis folded intoNR_ACTIVE_ANON.Pass reclaiming=!active so a folio from an inactive list is seeded into an older generation. Folios from the active list carry
PG_activeand hit the first branch either way, so they are unchanged.reclaiming also selects the insertion end in
lru_gen_add_folio():list_add_tail()for inactive folios,list_add()for active ones. Both the legacy LRU and a MGLRU generation keep the hottest folios at the head and the coldest at the tail, and reclaim takes from the tail. To preserve that order the folio must be taken from the end matching the insertion end, so take inactive folios from the head and active folios from the tail; otherwise hot/cold would be reversed within the generation.Tested on
x86_64, next-20260812, 2G VM + 1G swap, ~1.5G anon pushed onto the inactive list before enabling MGLRU:Active(anon) Inactive(anon)
before switch (legacy) 2952 1548792 kB after `echo y`, unpatched 1552052 0 kB after `echo y`, patched 15144 1536636 kBInactive file folios stay inactive either way (
NR_INACTIVE_FILEis preserved).Fixes 354ed5974429 ("mm: multi-gen LRU: kill switch")Suggested-by Barry Song <[email protected]>Assisted-by Claude:claude-opus-4-8Signed-off-by Ridong Chen <[email protected]>--- mm/vmscan.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)mm/vmscan.c +3
-1@@ -5311,7 +5311,23 @@ static bool fill_evictable(struct lruvec *lruvec)while (!list_empty (head)) {bool success; -struct folio *folio =lru_to_folio (head); +struct folio *folio; + +/*
mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5326
Quoted 10 lines
mm/vmscan.c +10
-0+* Both the legacy LRU and a MGLRU generation keep the +* hottest folios at the head and the coldest at the +* tail, and reclaim takes from the tail. To preserve +* that order, the end we take from must match the end +* lru_gen_add_folio() inserts at: inactive folios use +* reclaiming=true (list_add_tail), so take from the +* head; active folios use reclaiming=false (list_add), +* so take from the tail. Taking from the wrong end +* would reverse hot/cold within the generation. +*/
too many words, maybe just:
lru_gen_add_folio() uses list_add_tail() rather than list_add() when reclaiming is true. Match its ordering to avoid cold/hot Inversion.
Quoted 11 lines
mm/vmscan.c +4
-0+if (active) + folio =lru_to_folio (head); +else + folio = list_first_entry(head, struct folio, lru);VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio) != active, folio);@@ -5319,7 +5335,16 @@ static bool fill_evictable(struct lruvec *lruvec)VM_WARN_ON_ONCE_FOLIO (folio_lru_gen (folio) !=-1 , folio);lruvec_del_folio (lruvec, folio);
mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5346
Quoted 10 lines
mm/vmscan.c +9
-1- success =lru_gen_add_folio (lruvec, folio, false); +/* +* With reclaiming=false, lru_gen_folio_seq() would seed +* an inactive folio near max_seq, which +* lru_gen_is_active() reports as active, so its inactive +* placement would be lost. Pass reclaiming=!active to +* seed it into the oldest generation instead. This +* reuses reclaiming beyond its folio_rotate_reclaimable() +* meaning; it also picks list_add_tail() above. +*/
Maybe that's too verbose. How about:
"Borrow reclaiming=true to place inactive folios in the older gens"
mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5350
Quoted 4 lines
mm/vmscan.c +1
-0+ success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success);if (!--remaining)
This is admittedly a bit ugly, but it seems to be the simplest approach. Since switching MGLRU on and off is not a common scenario, we probably don't want to over-engineer it. So, with the above change:
Quoted 103 lines · Barry Song
Quoted 56 lines · Ridong Chen
From: Ridong Chen <[email protected]>
When the LRU is switched to MGLRU (echo y >
/sys/kernel/mm/lru_gen/enabled),fill_evictable()re-inserts every folio vialru_gen_add_folio(..., false). With reclaiming hardcoded to false, an inactive anonymous folio (noPG_active, not in the swapcache) takes the "gen =MIN_NR_GENS" branch inlru_gen_folio_seq()and is seeded at seq =max_seq- 1, whichlru_gen_is_active()treats as active. Its inactive placement is lost andNR_INACTIVE_ANONis folded intoNR_ACTIVE_ANON.Pass reclaiming=!active so a folio from an inactive list is seeded into an older generation. Folios from the active list carry
PG_activeand hit the first branch either way, so they are unchanged.reclaiming also selects the insertion end in
lru_gen_add_folio():list_add_tail()for inactive folios,list_add()for active ones. Both the legacy LRU and a MGLRU generation keep the hottest folios at the head and the coldest at the tail, and reclaim takes from the tail. To preserve that order the folio must be taken from the end matching the insertion end, so take inactive folios from the head and active folios from the tail; otherwise hot/cold would be reversed within the generation.Tested on
x86_64, next-20260812, 2G VM + 1G swap, ~1.5G anon pushed onto the inactive list before enabling MGLRU:Active(anon) Inactive(anon)
before switch (legacy) 2952 1548792 kB after `echo y`, unpatched 1552052 0 kB after `echo y`, patched 15144 1536636 kBInactive file folios stay inactive either way (
NR_INACTIVE_FILEis preserved).Fixes 354ed5974429 ("mm: multi-gen LRU: kill switch")Suggested-by Barry Song <[email protected]>Assisted-by Claude:claude-opus-4-8Signed-off-by Ridong Chen <[email protected]>--- mm/vmscan.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)mm/vmscan.c +3
-1@@ -5311,7 +5311,23 @@ static bool fill_evictable(struct lruvec *lruvec)while (!list_empty (head)) {bool success; -struct folio *folio =lru_to_folio (head); +struct folio *folio; + +/* mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5326
Quoted 10 lines
mm/vmscan.c +10
-0+* Both the legacy LRU and a MGLRU generation keep the +* hottest folios at the head and the coldest at the +* tail, and reclaim takes from the tail. To preserve +* that order, the end we take from must match the end +* lru_gen_add_folio() inserts at: inactive folios use +* reclaiming=true (list_add_tail), so take from the +* head; active folios use reclaiming=false (list_add), +* so take from the tail. Taking from the wrong end +* would reverse hot/cold within the generation. +*/ too many words, maybe just:
lru_gen_add_folio()useslist_add_tail()rather thanlist_add()when reclaiming is true. Match its ordering to avoid cold/hot Inversion.Quoted 11 lines
mm/vmscan.c +4
-0+if (active) + folio =lru_to_folio (head); +else + folio = list_first_entry(head, struct folio, lru);VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio) != active, folio);@@ -5319,7 +5335,16 @@ static bool fill_evictable(struct lruvec *lruvec)VM_WARN_ON_ONCE_FOLIO (folio_lru_gen (folio) !=-1 , folio);lruvec_del_folio (lruvec, folio);mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5346
Quoted 10 lines
mm/vmscan.c +9
-1- success =lru_gen_add_folio (lruvec, folio, false); +/* +* With reclaiming=false, lru_gen_folio_seq() would seed +* an inactive folio near max_seq, which +* lru_gen_is_active() reports as active, so its inactive +* placement would be lost. Pass reclaiming=!active to +* seed it into the oldest generation instead. This +* reuses reclaiming beyond its folio_rotate_reclaimable() +* meaning; it also picks list_add_tail() above. +*/ Maybe that's too verbose. How about:
"Borrow reclaiming=true to place inactive folios in the older gens"
Quoted 3 lines
mm/vmscan.c +1
-0+ success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success);
Quoted 7 lines
mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5350
Quoted 1 line
mm/vmscan.c +0
-0if (!--remaining)This is admittedly a bit ugly, but it seems to be the simplest approach. Since switching MGLRU on and off is not a common scenario, we probably don't want to over-engineer it. So, with the above change:
Acked-by Barry Song <[email protected]>
Thanks.
Will update.
Signature · 2 lines
Best regards Ridong