mm: mglru: fix swappiness behavior
Subsystems (4)
Barry Song (Xiaomi)
reviewerv7 cover mm: mglru: fix swappiness behavior
Classic LRU respects swappiness, while MGLRU only weakly honors it when selecting the scan type. The fallback logic can easily override the bias established by get_type_to_scan(). As a result, swappiness has a much smaller overall impact when changing from 1 to 200, as shown by the data presented later in this cover letter.
In this patchset, patches 1-8 from Kairui introduce a model that uses refaults to determine nr_to_scan[] for both file and anon in MGLRU. For each reclaim operation, we calculate how many file and anon folios to scan, similar to CLRU, rather than relying on get_type_to_scan() and fallback mechanisms to decide what to reclaim, which have proven rather unreliable in honoring swappiness. This model also allows scanning and reclaiming from the two youngest generations, including the youngest generation itself. This can help resolve OOM situations[1] and reduce the need for aging.
As Kairui mentioned, the OOM reproducer was demonstrated previously [2], and that series was merged in v7 [3]. The problem was largely alleviated as described, but not completely fixed. Under higher memory pressure, the system could still hit OOM. Before this series, Kairui could still reproduce OOM after hundreds of iterations. With the fixing patch in this series applied, the problem no longer occurs.
Patch 9 from Barry fixes a cold/hot inversion exacerbated by patches 1–8 (which also exists in mainline) and improves reclaim efficiency.
Patches 10-15 from Gaofei and Barry optimize aging based on folio distribution imbalance and improve reclaim behavior at extreme swappiness values such as 1 and 200.
Patch 16 from Bo Zhang addresses the GFP_NOIO issue in MGLRU.
After the patchset, the results track CLRU across the full swappiness range (see table below), with more file scanning at low swappiness and more anon scanning at high swappiness.
pgsteal_file
swappiness CLRU MGLRU MGLRU+patches 1 6,620,680 745,968 6,249,276 36 807,807 508,974 569,354 71 553,144 417,700 419,718 106 402,880 402,474 329,145 141 240,293 337,797 296,131 176 187,682 330,045 260,850 200 0 343,949 128
pgsteal_anon
swappiness CLRU MGLRU MGLRU+patches 1 4,436,199 2,671,934 4,960,289 36 2,858,117 3,032,747 3,066,513 71 3,042,392 2,827,735 3,258,388 106 3,292,622 3,184,314 3,217,699 141 3,612,869 2,999,000 3,417,135 176 3,827,154 3,134,127 3,720,778 200 13,825,283 3,269,753 14,551,425
pgscan_file
swappiness CLRU MGLRU MGLRU+patches 1 11,853,625 877,311 13,703,243 36 1,016,826 582,783 680,534 71 662,823 474,772 467,822 106 496,374 451,285 356,494 141 277,109 382,604 307,866 176 206,261 372,059 268,584 200 0 393,608 128
pgscan_anon
swappiness CLRU MGLRU MGLRU+patches 1 20,804,386 5,847,747 6,766,764 36 17,830,330 6,740,075 5,870,434 71 18,357,293 6,126,194 7,333,437 106 19,938,920 6,431,355 7,702,863 141 19,609,579 5,997,762 8,915,973 176 21,612,749 6,071,032 10,590,224 200 44,776,474 6,544,125 30,318,894
refault_file
swappiness CLRU MGLRU MGLRU+patches 1 6,523,070 667,302 6,171,580 36 722,110 431,585 492,267 71 477,062 340,241 342,626 106 327,610 324,747 262,786 141 161,827 265,959 224,831 176 115,551 258,515 184,601 200 0 268,049 88
refault_anon
swappiness CLRU MGLRU MGLRU+patches 1 867,728 552,829 1,382,052 36 419,635 657,745 675,629 71 536,299 611,345 706,445 106 580,742 693,689 718,644 141 706,151 639,850 785,236 176 706,499 685,086 906,427 200 4,693,850 733,273 5,489,076
With a normal swappiness setting (without extreme bias), I see almost no noticeable performance difference between the two versions.
For example, with swappiness=60, the average results over 5 rounds are:
metric W/o patch W/ patch real 1m49.273s 1m48.876s user 25m33.186s 25m31.449s sys 3m41.851s 3m39.935s
[1] https://github.com/ryncsn/emm-test-project/tree/master/file-anon-mix-pressure
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
v7:
* fix a build error of bisect and address some other sashiko minor comments;
* add no-populatable aging;
v6 link:
https://lore.kernel.org/linux-mm/[email protected]/
v6:
* rebase on top of mm-new(a lot of conflicts due to recent active mglru developments);
* only count eligible zones below sc->reclaim_idx for lruvec_evictable_size() and lru_gen_imbalanced();
* GFP_NOIO further fixing by Bo Zhang, exclude unreclaimable type in lruvec_evictable_size().
* Add Kunwu's reviewed-by, thanks!
* RFC v5 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v5:
* Remove the aging optimizations, which have been merged - thanks for Baolin, Baoqun, Lian Wang, Kairui, Kunwu, Ridong reviewing on this;
* Add Kairui's model for determining nr_to_scan[] for both types.
* Fix cold/hot inversion for rejected folios - Barry.
* Improve and clean up imbalanced aging, and optimize extreme swappiness values of 1 and 200 - Gaofei, Barry.
* Address the GFP_NOIO issue in MGLRU - Bo Zhang.
* RFC v4 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v4:
* Clean up isolate_folios() based on Baolin's feedback. Thanks!
* Further improve isolate_folios() and scan_folios(), including handling of exhausted reclaim types.
* Add fixes and performance improvements to inc_min_seq(), and reduce lock contention.
* Improve the handling of imbalanced aging and remaining generation-based aging.
* RFC v3 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v3:
- Added Ridong's patch of improving isolate_folios() readability;
- for lru_gen_imbalanced(), apply some suggestions from Zicheng and Kairui, considering lruvec size and use macro;
- drop "mm: mglru: only fall back when reclaim is running at high priority"
- mitigate aging for balanced swappiess value(near middle 100) as swappiness bias is less a concern for them; but slightly increased aging overhead might not be good.
* RFC v2 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!
Barry Song (Xiaomi) (5):
mm/mglru: Avoid placing rejected folios below populated gen mm/mglru: Run aging when one type has no populatable folios mm/mglru: Restrict eviction accounting to eligible zones mm/mglru: Reduce LRU lock contention by shrinking folio batches in
inc_min_seq
mm/mglru: Make swappiness=1 file reclaim more aggressive
Bo Zhang (1):
mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO
allocation
Kairui Song (8):
mm/mglru: report evictable sizes per type
mm/mglru: extract lruvec_populated_min_seq() helper mm/mglru: reclaim from the oldest populated gen mm/mglru: lift the forced protection of reclaimer mm/mglru: skip zones above reclaim_idx instead of rotating them mm/mglru: return for unavailable anon reclaim earlier mm/mglru: balance anon and file scan budget by refault rate mm/mglru: lift youngest gen protection for higher priority
lyugaofei (2):
mm/mglru: Run aging when pages are rare outside the youngest gen mm/mglru: Avoid prolonged anon reclaim with swappiness=200
include/linux/mm_inline.h | 22 +- mm/trace_vmscan.h | 4 +- mm/vmscan.c | 488 +++++++++++++++++++++++++------------- mm/vmscan.h | 5 +- 4 files changed, 343 insertions(+), 176 deletions(-)
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerPATCH v7 1/16 mm/mglru: report evictable sizes per type
From: Kairui Song <[email protected]>
Change lruvec_evictable_size() to return the number of evictable anonymous and file pages separately instead of a single total, so that callers can weigh the two types independently.
Update lruvec_is_sizable() and get_nr_to_scan() accordingly; the summed totals are unchanged, so there is no functional change.
--- mm/vmscan.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
mm/vmscan.c +13 -9
@@ -4192,28 +4192,31 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) +static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, +unsigned long nr[]) {int type; -unsigned long seq, total =0 ; +unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); + nr[LRU_GEN_ANON ] =0 ; + nr[LRU_GEN_FILE ] =0 ; +for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - total +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq); } - -return total; }static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) { -unsigned long total; +unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); - total =lruvec_evictable_size (lruvec, swappiness); +lruvec_evictable_size (lruvec, swappiness, sizes); + total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */ return total >> sc->priority;
@@ -5021,7 +5024,7 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc,struct mem_cgroup *memcg,int swappiness) { -unsigned long nr_to_scan, evictable; +unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ];struct pglist_data *pgdat =lruvec_pgdat (lruvec);/*
@@ -5037,7 +5040,8 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,return 0 ; } - evictable =lruvec_evictable_size (lruvec, swappiness); +lruvec_evictable_size (lruvec, swappiness, sizes); + evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ]; nr_to_scan =apply_proportional_protection (memcg, sc, evictable); nr_to_scan >>= sc->priority;
Signature · 1 line
2.34.1
Re: [PATCH v7 01/16] mm/mglru: report evictable sizes per type
Quoted 6 lines · Barry Song (Xiaomi)
From: Kairui Song <[email protected]>
Change
lruvec_evictable_size()to return the number of evictable anonymous and file pages separately instead of a single total, so that callers can weigh the two types independently.
Quoted 7 lines
Update
lruvec_is_sizable()andget_nr_to_scan()accordingly; the summed totals are unchanged, so there is no functional change.Signed-off-by Kairui Song <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>Reviewed-by Kunwu Chan <[email protected]>---
LGTM.
Barry Song (Xiaomi)
reviewerPATCH v7 2/16 mm/mglru: extract lruvec_populated_min_seq() helper
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
The search for the oldest populated generation of a type, previously embedded in try_to_inc_min_seq(), is extracted into the new lruvec_populated_min_seq() helper. It returns the oldest populated seq of a given type, bounded by the given max_seq so the search never goes beyond it.
No functional change.
--- mm/vmscan.c | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-)
mm/vmscan.c +29 -14
@@ -3991,30 +3991,45 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)return true; } +/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ +static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, +unsigned long max_seq) +{ +int gen, zone; +unsigned long min_seq; +struct lru_gen_folio *lrugen = &lruvec->lrugen; + +lockdep_assert_held (&lruvec->lru_lock); + + min_seq =READ_ONCE (lrugen->min_seq[type]); +while (min_seq <= max_seq) { + gen =lru_gen_from_seq (min_seq); +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +if (!list_empty (&lrugen->folios[gen][type][zone])) +return min_seq; + } + min_seq++; + } + +return max_seq; +} +static void try_to_inc_min_seq (struct lruvec *lruvec,int swappiness) { -int gen, type, zone; +int type;bool seq_inc_flag = false;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MIN_SEQ (lruvec); +unsigned long seq;VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); -/* find the oldest populated generation */ +/* Requires at least MIN_NR_GENS gens for basic functionality */ + seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { -while (min_seq[type] +MIN_NR_GENS <= lrugen->max_seq) { - gen =lru_gen_from_seq (min_seq[type]); - -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { -if (!list_empty (&lrugen->folios[gen][type][zone])) - goto next; - } - - min_seq[type]++; + min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); +if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; - } -next: - ; }/*
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 02/16] mm/mglru: extract lruvec_populated_min_seq() helper
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/mglru: shadowed local variable seq in try_to_inc_min_seq() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/mglru: shadowed local variable seq in try_to_inc_min_seq() --
commit 7a4927e9f0aa8623b1c83f69afd1fb935c342dd5
Author: Kairui Song <[email protected]>
mm/mglru: extract lruvec_populated_min_seq() helper
This commit extracts the search for the oldest populated generation of a type from try_to_inc_min_seq() into a new lruvec_populated_min_seq() helper.
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 21 lines
static void
try_to_inc_min_seq(struct lruvec *lruvec, int swappiness){mm/vmscan.c +4
-8-int gen, type, zone; +int type;bool seq_inc_flag = false;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MIN_SEQ (lruvec); +unsigned long seq;VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); -/* find the oldest populated generation */ +/* Requires at least MIN_NR_GENS gens for basic functionality */ + seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { -while (min_seq[type] +MIN_NR_GENS <= lrugen->max_seq) { - gen =lru_gen_from_seq (min_seq[type]); - -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { -if (!list_empty (&lrugen->folios[gen][type][zone])) - goto next;
mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 4033
Quoted 10 lines
mm/vmscan.c +2
-6- } - - min_seq[type]++; + min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); +if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; - } -next: - ; }
[Severity: Low]
This isn't a bug, but adding seq at function scope causes the existing inner declaration in try_to_inc_min_seq() to shadow it and recompute the same value:
mm/vmscan.c:try_to_inc_min_seq() {
...
if (swappiness && swappiness <= MAX_SWAPPINESS) {
unsigned long seq = lrugen->max_seq - MIN_NR_GENS;
...
}Dropping the inner declaration would avoid the -Wshadow warning under W=2 and reuse the value already computed.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Barry Song
reviewerRe: [PATCH v7 02/16] mm/mglru: extract lruvec_populated_min_seq() helper
Quoted 66 lines · On Thu, Nov 12, 2026 at 1:00 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/mglru: shadowed local variable seq intry_to_inc_min_seq()--commit
7a4927e9f0aa8623b1c83f69afd1fb935c342dd5
Author: Kairui Song <[email protected]>mm/mglru: extract
lruvec_populated_min_seq()helperThis commit extracts the search for the oldest populated generation of a type from
try_to_inc_min_seq()into a newlruvec_populated_min_seq()helper.Quoted 4 lines
mm/vmscan.c +0
-0[ ... ]
Quoted 21 lines
static void
try_to_inc_min_seq(struct lruvec *lruvec, int swappiness){mm/vmscan.c +4
-8-int gen, type, zone; +int type;bool seq_inc_flag = false;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MIN_SEQ (lruvec); +unsigned long seq;VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); -/* find the oldest populated generation */ +/* Requires at least MIN_NR_GENS gens for basic functionality */ + seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { -while (min_seq[type] +MIN_NR_GENS <= lrugen->max_seq) { - gen =lru_gen_from_seq (min_seq[type]); - -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { -if (!list_empty (&lrugen->folios[gen][type][zone])) - goto next;mm/vmscan.c
Quoted 10 lines
mm/vmscan.c +2
-6- } - - min_seq[type]++; + min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); +if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; - } -next: - ; }[Severity: Low]
This isn't a bug, but adding seq at function scope causes the existing inner declaration intry_to_inc_min_seq()to shadow it and recompute the same value:mm/vmscan.c:try_to_inc_min_seq() { ... if (swappiness && swappiness <= MAX_SWAPPINESS) { unsigned long seq = lrugen->max_seq - MIN_NR_GENS; ... }
Quoted 2 lines
Dropping the inner declaration would avoid the -Wshadow warning under W=2 and reuse the value already computed.
This makes sense to me, we can do:
mm/vmscan.c +0 -0
@@ -4033,8 +4033,6 @@ static void try_to_inc_min_seq(struct lruvec
*lruvec, int swappiness)
/* see the comment on lru_gen_folio */
if (swappiness && swappiness <= MAX_SWAPPINESS) {
- unsigned long seq = lrugen->max_seq - MIN_NR_GENS;
-
if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
min_seq[LRU_GEN_ANON] = seq;
else if (min_seq[LRU_GEN_FILE] > seq &&min_seq[LRU_GEN_ANON] < seq)
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Barry Song (Xiaomi)
reviewerPATCH v7 3/16 mm/mglru: reclaim from the oldest populated gen
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
Currently MGLRU tries to keep the gen numbers of the two types in sync, avoiding reducing the gen number of one type when the other type has more gens, as described in the comment in try_to_inc_min_seq().
This makes scan_folios() stuck reclaiming the type with fewer gens: it always scanned the oldest gen of the type, even when that gen was empty, so reclaim made no progress on that type unless forced aging created a new gen and eliminated the dead empty gen.
There are other approaches, like lifting the gen number synchronization [1]. However, that would break the gen layout used by lru_gen_folio_seq(): newly added folios would no longer have a well-defined hotness according to their gen number, which reduces accuracy. It is also not pressure-aware, since with gentle reclaim we still want to respect the gen layout (hotness).
Instead, reclaim from whichever populated oldest gen is left, but only when the default reclaim priority can't satisfy the reclaimer's requirement. This is typical for swappiness limitations: e.g., low swappiness asks the reclaimer to reclaim from file, but file gens are drained, so the reclaim cycle fails and priority is raised.
At the default priority the scan quota is 1/4096 of the evictable size, and the old behavior is kept there to respect the hotness distribution. Once priority is raised, the quota is at least 1/2048, so the hotness bias introduced by reclaiming from a younger gen is bounded by that quota, which seems fair. Global reclaim for MGLRU may start with DEF_PRIORITY / 2 (1/64); the ratio is still reasonable at that point, and when global pressure is more critical, we should reclaim harder.
Note that scan_folios() still refuses to reclaim a type left with only MIN_NR_GENS gens, so a completely drained type still requires aging. This change only avoids scanning empty gens when populated reclaimable gens remain.
--- mm/vmscan.c | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-)
mm/vmscan.c +19 -11
@@ -3311,11 +3311,9 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased) }/* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio) +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,int old_gen) { -int type =folio_is_file_lru (folio); -struct lru_gen_folio *lrugen = &lruvec->lrugen; -int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); +int new_gen;bool gen_increased; new_gen =__folio_inc_gen (folio, old_gen, &gen_increased);
@@ -4674,7 +4672,7 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent,******************************************************************************/ static bool sort_folio (struct lruvec *lruvec,struct folio *folio,struct scan_control *sc, -int tier_idx) +int min_gen,int tier_idx) {int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);
@@ -4692,14 +4690,14 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_creturn false;/* promoted */ -if (gen !=lru_gen_from_seq (lrugen->min_seq[type])) { +if (gen != min_gen) {list_move (&folio->lru, &lrugen->folios[gen][type][zone]);return true; }/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) { - gen =folio_inc_gen (lruvec, folio); + gen =folio_inc_gen (lruvec, folio, min_gen);list_move (&folio->lru, &lrugen->folios[gen][type][zone]);/* don't count the workingset being lazily promoted */
@@ -4714,7 +4712,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c/* ineligible */ if (zone > sc->reclaim_idx) { - gen =folio_inc_gen (lruvec, folio); + gen =folio_inc_gen (lruvec, folio, min_gen);list_move_tail (&folio->lru, &lrugen->folios[gen][type][zone]);return true; }
@@ -4757,17 +4755,27 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int scanned =0 ;int isolated =0 ;int skipped =0 ; +unsigned long min_seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); +/* +* Respect the gen hotness distribution at low priority to avoid +* aging and scan cost, but prefer to satisfy the reclaimers request +* when under pressure to respect swappiness more and reduce IO cost. +*/ +if (sc->priority ==DEF_PRIORITY ) + min_seq = lrugen->min_seq[type]; +else + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 ); +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS )return 0 ; - gen =lru_gen_from_seq (lrugen->min_seq[type]); - + gen =lru_gen_from_seq (min_seq);for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;
@@ -4785,7 +4793,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned += delta; -if (sort_folio (lruvec, folio, sc, tier)) +if (sort_folio (lruvec, folio, sc, gen, tier)) sorted += delta;else if (isolate_folio (lruvec, folio, sc)) {list_add (&folio->lru, list);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 03/16] mm/mglru: reclaim from the oldest populated gen
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/mglru: uninitialized sc.priority in lru_gen_seq_write() skews scan_folios() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/mglru: uninitialized sc.priority in lru_gen_seq_write() skews scan_folios()
--
commit c250a15f7ef946582fda6b278da38054c0ad3a66
Author: Kairui Song <[email protected]>
mm/mglru: reclaim from the oldest populated gen
This commit updates scan_folios() to scan from the oldest populated generation when sc->priority is raised from DEF_PRIORITY, passing that generation down to sort_folio() and folio_inc_gen().
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 10 lines
mm/vmscan.c +1
-0@@ -4758,17 +4756,27 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int scanned =0 ;int isolated =0 ;int skipped =0 ; +unsigned long min_seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list));
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4774
Quoted 10 lines
mm/vmscan.c +9
-0+/* +* Respect the gen hotness distribution at low priority to avoid +* aging and scan cost, but prefer to satisfy the reclaimers request +* when under pressure to respect swappiness more and reduce IO cost. +*/ +if (sc->priority ==DEF_PRIORITY ) + min_seq = lrugen->min_seq[type]; +else + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 );
[Severity: Low]
Does debugfs eviction via lru_gen_seq_write() take the raised-priority branch here because sc.priority is left zero-initialized?
lru_gen_seq_write() allocates struct scan_control sc on the stack without setting .priority = DEF_PRIORITY, so sc.priority is 0. When an eviction command runs while one type's oldest generation is empty and the other type keeps get_nr_gens() above MIN_NR_GENS, scan_folios() treats the call as raised priority and advances min_seq up to max_seq - MIN_NR_GENS + 1, evicting folios from generations younger than the requested min_gen_nr (including active generations that run_eviction() rejects).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Barry Song (Xiaomi)
reviewerPATCH v7 4/16 mm/mglru: lift the forced protection of reclaimer
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
When a type is drained down to MIN_NR_GENS gens, the reclaimer will fail to make any progress on that type without aging.
However aging comes with a cost, and it will take time to finish. Even if we do aging more proactively, when under great pressure, gens will still be drained very fast so the new gens produced by aging can't catch up with the consumption. With MIN_NR_GENS setting blocking the reclaim, we will run into OOM. This OOM can be reproduced by the previous test program [1] used for fixing another aging problem [2]. When running in great pressure, certain workloads will keep seeing premature OOM with MGLRU, and never see any OOM with classical LRU.
This might get worse if we implement async aging, or avoid hotness inversion in aging, as they will slow down the aging process even more.
To fix this, let's first drop the get_nr_gens() == MIN_NR_GENS early return and let the raised-priority scan bound cover max_seq itself.
Mapped folios will still survive as the rmap PTE check will keep them. Unmapped cache folios may thrashing more when under great pressure, which is really a problem that should be fixed with refault distance and frequecy aware promotoin instead [3].
--- mm/vmscan.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
mm/vmscan.c +25 -9
@@ -4672,7 +4672,7 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent,******************************************************************************/ static bool sort_folio (struct lruvec *lruvec,struct folio *folio,struct scan_control *sc, -int min_gen,int tier_idx) +int min_gen,int max_gen,int tier_idx) {int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);
@@ -4689,12 +4689,22 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_cif (!folio_evictable (folio))return false; -/* promoted */ +/* promoted, or moved due to reparenting. */ if (gen != min_gen) {list_move (&folio->lru, &lrugen->folios[gen][type][zone]);return true; } +/* +* We are reclaiming the last gen left. Folios in this gen can no +* longer be promoted or protected, so just isolate them. The one +* exception is lru_gen_reparent_memcg(), which is handled above. +*/ +if (min_gen == max_gen) { +VM_WARN_ON_ONCE (zone > sc->reclaim_idx); +return false; + } +/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) { gen =folio_inc_gen (lruvec, folio, min_gen);
@@ -4749,15 +4759,16 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct list_head *list, int *isolatedp) { int i; -int gen;enum node_stat_item item;int sorted =0 ;int scanned =0 ;int isolated =0 ;int skipped =0 ; +int gen, max_gen;unsigned long min_seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +DEFINE_MAX_SEQ (lruvec);VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list));
@@ -4770,18 +4781,24 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (sc->priority ==DEF_PRIORITY ) min_seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 ); - -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) -return 0 ; + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq); gen =lru_gen_from_seq (min_seq); + max_gen =lru_gen_from_seq (max_seq); +for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;int zone = (sc->reclaim_idx + i) %MAX_NR_ZONES ;struct list_head *head = &lrugen->folios[gen][type][zone]; +/* +* If there is only one gen left, rotating the unreclaimable +* zone become meaningless, just skip them. +*/ +if (gen == max_gen && zone > sc->reclaim_idx) +continue ; +while (!list_empty (head)) {struct folio *folio =lru_to_folio (head);int delta =folio_nr_pages (folio);
@@ -4792,8 +4809,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); scanned += delta; - -if (sort_folio (lruvec, folio, sc, gen, tier)) +if (sort_folio (lruvec, folio, sc, gen, max_gen, tier)) sorted += delta;else if (isolate_folio (lruvec, folio, sc)) {list_add (&folio->lru, list);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 04/16] mm/mglru: lift the forced protection of reclaimer
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/mglru: run_eviction() evicts up to max_seq due to sc.priority == 0 --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/mglru: run_eviction() evicts up to max_seq due to sc.priority == 0
--
commit cd86952996e1243807567af19d65b021c3573888
Author: Kairui Song <[email protected]>
mm/mglru: lift the forced protection of reclaimer
This commit drops the MIN_NR_GENS early return in scan_folios() and extends the raised-priority scan bound up to max_seq so reclaim can proceed when aging cannot keep up with pressure.
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4785
Quoted 9 lines
mm/vmscan.c +1
-4@@ -4771,18 +4782,24 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (sc->priority ==DEF_PRIORITY ) min_seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 ); - -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) -return 0 ; + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq);
[Severity: Low]
Can debugfs proactive eviction via lru_gen_seq_write() -> run_eviction() now scan and evict folios up to lrugen->max_seq instead of stopping at the requested target seq?
lru_gen_seq_write() leaves sc.priority zero-initialized (0 != DEF_PRIORITY), and run_eviction() loops while seq >= evictable_min_seq(min_seq, swappiness). When one evictable type has no folios in generations <= seq while the other still has min_seq <= seq, scan_folios() takes the sc->priority != DEF_PRIORITY branch and evicts folios from newer generations up to max_seq, where sort_folio() also bypasses tier and reference protection when min_gen == max_gen.
The commit message of later commit "mm/mglru: balance anon and file scan budget by refault rate" in this series notes setting .priority = DEF_PRIORITY for the debugfs scan_control, but that initialization is missing from the code.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Barry Song (Xiaomi)
reviewerPATCH v7 5/16 mm/mglru: skip zones above reclaim_idx instead of rotating them
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
scan_folios() used to visit every zone. Folios in zones above sc->reclaim_idx can never be isolated, so sort_folio() rotated them to a younger generation, the "ineligible" branch, or, when only one gen was left, skipped them. This wasted scan work on every pass, and the rotation kept the protected folios from aging.
Restrict both the scan and the gen selection to the eligible zones, like the legacy LRU, which only isolates folios in zones up to reclaim_idx:
- lruvec_populated_min_seq() takes a zone_limit, so the oldest populated gen is searched only among the zones the scan can reclaim from. A gen populated only in protected zones no longer stalls the scan.
- scan_folios() iterates the eligible zones in the descending order of the legacy zonelist walk, draining the zone at reclaim_idx first.
- The ineligible branch of sort_folio() is now dead and removed, with a VM_WARN_ON_ONCE_FOLIO() guarding the new invariant.
try_to_inc_min_seq() still considers all zones, so min_seq[type] is only bumped once the oldest gen is empty everywhere. At DEF_PRIORITY the search bound stays min_seq[type], so the scan still starts from the oldest gen regardless of whether it is populated.
--- mm/vmscan.c | 47 +++++++++++++++++++---------------------------- 1 file changed, 19 insertions(+), 28 deletions(-)
mm/vmscan.c +19 -28
@@ -3991,7 +3991,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, -unsigned long max_seq) +int zone_limit,unsigned long max_seq) {int gen, zone;unsigned long min_seq;
@@ -4002,7 +4002,7 @@ static unsigned long lruvec_populated_min_seq(struct lruvec *lruvec, int type, min_seq =READ_ONCE (lrugen->min_seq[type]);while (min_seq <= max_seq) { gen =lru_gen_from_seq (min_seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +for (zone =0 ; zone < zone_limit; zone++) {if (!list_empty (&lrugen->folios[gen][type][zone]))return min_seq; }
@@ -4025,7 +4025,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)/* Requires at least MIN_NR_GENS gens for basic functionality */ seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { - min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); + min_seq[type] =lruvec_populated_min_seq (lruvec, type,MAX_NR_ZONES , seq +1 );if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; }
@@ -4684,6 +4684,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_cstruct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE_FOLIO (gen >=MAX_NR_GENS , folio); +VM_WARN_ON_ONCE_FOLIO (zone > sc->reclaim_idx, folio);/* unevictable: let it through and the generic path will cull it */ if (!folio_evictable (folio))
@@ -4700,10 +4701,8 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c* longer be promoted or protected, so just isolate them. The one * exception is lru_gen_reparent_memcg(), which is handled above. */ -if (min_gen == max_gen) { -VM_WARN_ON_ONCE (zone > sc->reclaim_idx); +if (min_gen == max_gen)return false; - }/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) {
@@ -4720,13 +4719,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_creturn true; } -/* ineligible */ -if (zone > sc->reclaim_idx) { - gen =folio_inc_gen (lruvec, folio, min_gen); -list_move_tail (&folio->lru, &lrugen->folios[gen][type][zone]); -return true; - } -return false; }
@@ -4758,16 +4750,17 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int type, int tier, struct list_head *list, int *isolatedp) { - int i; enum node_stat_item item; +int zone_idx;int sorted =0 ;int scanned =0 ;int isolated =0 ;int skipped =0 ;int gen, max_gen; -unsigned long min_seq; +unsigned long seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct pglist_data *pgdat =lruvec_pgdat (lruvec);DEFINE_MAX_SEQ (lruvec);VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );
@@ -4779,24 +4772,22 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* when under pressure to respect swappiness more and reduce IO cost. */ if (sc->priority ==DEF_PRIORITY ) - min_seq = lrugen->min_seq[type]; + seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq); + seq = lrugen->max_seq; - gen =lru_gen_from_seq (min_seq); + seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq); -for (i =MAX_NR_ZONES ; i >0 ; i--) { +/* Scan eligible zones in descending order. */ +for (zone_idx = sc->reclaim_idx; zone_idx >=0 ; zone_idx--) {LIST_HEAD (moved);int skipped_zone =0 ; -int zone = (sc->reclaim_idx + i) %MAX_NR_ZONES ; -struct list_head *head = &lrugen->folios[gen][type][zone]; +struct zone *zone = &pgdat->node_zones[zone_idx]; +struct list_head *head = &lrugen->folios[gen][type][zone_idx]; -/* -* If there is only one gen left, rotating the unreclaimable -* zone become meaningless, just skip them. -*/ -if (gen == max_gen && zone > sc->reclaim_idx) +if (!managed_zone (zone))continue ;while (!list_empty (head)) {
@@ -4806,7 +4797,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_is_file_lru (folio) != type, folio); -VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); +VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone_idx, folio); scanned += delta;if (sort_folio (lruvec, folio, sc, gen, max_gen, tier))
@@ -4825,7 +4816,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (skipped_zone) {list_splice (&moved, head); -__count_zid_vm_events (PGSCAN_SKIP , zone, skipped_zone); +__count_zid_vm_events (PGSCAN_SKIP , zone_idx, skipped_zone); skipped += skipped_zone; }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 05/16] mm/mglru: skip zones above reclaim_idx instead of rotating them
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm/mglru: pinned min_seq in try_to_inc_min_seq() stalls aging --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: pinned min_seq in try_to_inc_min_seq() stalls aging --
commit a9558af5b6eefdf9bb9a64058218b290ef8ecc66
Author: Kairui Song <[email protected]>
mm/mglru: skip zones above reclaim_idx instead of rotating them
This commit restricts scan_folios() and lruvec_populated_min_seq() to zones up to sc->reclaim_idx instead of scanning and rotating folios in higher zones.
Quoted 31 lines
mm/vmscan.c +3
-3@@ -3992,7 +3992,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, -unsigned long max_seq) +int zone_limit,unsigned long max_seq) {int gen, zone;unsigned long min_seq;@@ -4003,7 +4003,7 @@ static unsigned long lruvec_populated_min_seq(struct lruvec *lruvec, int type, min_seq =READ_ONCE (lrugen->min_seq[type]);while (min_seq <= max_seq) { gen =lru_gen_from_seq (min_seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +for (zone =0 ; zone < zone_limit; zone++) {if (!list_empty (&lrugen->folios[gen][type][zone]))return min_seq; }@@ -4026,7 +4026,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)/* Requires at least MIN_NR_GENS gens for basic functionality */ seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { - min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); + min_seq[type] =lruvec_populated_min_seq (lruvec, type,MAX_NR_ZONES , seq +1 );if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; }
[ ... ]
Quoted 8 lines
mm/vmscan.c +1
-2@@ -4780,24 +4773,22 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* when under pressure to respect swappiness more and reduce IO cost. */ if (sc->priority ==DEF_PRIORITY ) - min_seq = lrugen->min_seq[type]; + seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq);
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4785
Quoted 10 lines
mm/vmscan.c +5
-2+ seq = lrugen->max_seq; - gen =lru_gen_from_seq (min_seq); + seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq); -for (i =MAX_NR_ZONES ; i >0 ; i--) { +/* Scan eligible zones in descending order. */ +for (zone_idx = sc->reclaim_idx; zone_idx >=0 ; zone_idx--) {
[Severity: High]
As reported by KunWu Chan on RFC v5 [1], does skipping zones above sc->reclaim_idx in scan_folios() while try_to_inc_min_seq() still checks all MAX_NR_ZONES stall aging during zone-constrained reclaim (for example, GFP_DMA32 reclaim when ZONE_NORMAL is populated)?
When an lruvec has MAX_NR_GENS (4) generations, lrugen->min_seq[type] starts at max_seq - 3. Because scan_folios() no longer rotates folios in zones above sc->reclaim_idx, any folios in those zones keep try_to_inc_min_seq() from advancing lrugen->min_seq[type] once eligible zones are drained.
In should_run_aging():
mm/vmscan.c:should_run_aging() {
...
if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
return true;
/* try to avoid aging, do gentle reclaim at the default priority */
if (sc->priority == DEF_PRIORITY)
return false;
/* better to run aging even though eviction is still possible */
return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq;
}With min_seq pinned at max_seq - 3, evictable_min_seq() + MIN_NR_GENS never reaches max_seq, so should_run_aging() returns false and scan_folios() drains eligible zones down to the youngest generation max_seq where sort_folio() disables tier protection and workingset promotion (min_gen == max_gen).
Even with the follow-up commits in this series (commit "mm/mglru: Run aging when pages are rare outside the youngest gen", commit "mm/mglru: Run aging when one type has no populatable folios", and commit "mm/mglru: Restrict eviction accounting to eligible zones"), at priorities above DEF_PRIORITY / 2 scan_folios() caps seq at max_seq - 1 and evict_folio_lists() returns 0 once older generations are empty in eligible zones, causing try_to_shrink_lruvec() to break out before calling should_run_aging() until priority drops to DEF_PRIORITY / 2 and scans max_seq without tier protection.
[1] https://lore.kernel.org/r/CAN_Opa-AScm9=EeXO_Uy8VjbQ=bqY9P++rFP8H1RhUB4dOt6xw@mail.gmail.com
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Barry Song
reviewerRe: [PATCH v7 05/16] mm/mglru: skip zones above reclaim_idx instead of rotating them
Quoted 98 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: pinnedmin_seqintry_to_inc_min_seq()stalls aging --commit
a9558af5b6eefdf9bb9a64058218b290ef8ecc66
Author: Kairui Song <[email protected]>mm/mglru: skip zones above
reclaim_idxinstead of rotating themThis commit restricts
scan_folios()andlruvec_populated_min_seq()to zones up tosc->reclaim_idxinstead of scanning and rotating folios in higher zones.Quoted 31 lines
mm/vmscan.c +3
-3@@ -3992,7 +3992,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, -unsigned long max_seq) +int zone_limit,unsigned long max_seq) {int gen, zone;unsigned long min_seq;@@ -4003,7 +4003,7 @@ static unsigned long lruvec_populated_min_seq(struct lruvec *lruvec, int type, min_seq =READ_ONCE (lrugen->min_seq[type]);while (min_seq <= max_seq) { gen =lru_gen_from_seq (min_seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +for (zone =0 ; zone < zone_limit; zone++) {if (!list_empty (&lrugen->folios[gen][type][zone]))return min_seq; }@@ -4026,7 +4026,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)/* Requires at least MIN_NR_GENS gens for basic functionality */ seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { - min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); + min_seq[type] =lruvec_populated_min_seq (lruvec, type,MAX_NR_ZONES , seq +1 );if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; }[ ... ]
Quoted 8 lines
mm/vmscan.c +1
-2@@ -4780,24 +4773,22 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* when under pressure to respect swappiness more and reduce IO cost. */ if (sc->priority ==DEF_PRIORITY ) - min_seq = lrugen->min_seq[type]; + seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq);mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4785
Quoted 10 lines
mm/vmscan.c +5
-2+ seq = lrugen->max_seq; - gen =lru_gen_from_seq (min_seq); + seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq); -for (i =MAX_NR_ZONES ; i >0 ; i--) { +/* Scan eligible zones in descending order. */ +for (zone_idx = sc->reclaim_idx; zone_idx >=0 ; zone_idx--) {[Severity: High]
As reported by KunWu Chan on RFC v5 [1], does skipping zones abovesc->reclaim_idxinscan_folios()whiletry_to_inc_min_seq()still checks allMAX_NR_ZONESstall aging during zone-constrained reclaim (for example,GFP_DMA32reclaim whenZONE_NORMALis populated)?When an lruvec has
MAX_NR_GENS(4) generations, lrugen->min_seq[type] starts atmax_seq- 3. Becausescan_folios()no longer rotates folios in zones abovesc->reclaim_idx, any folios in those zones keeptry_to_inc_min_seq()from advancing lrugen->min_seq[type] once eligible zones are drained.In
should_run_aging():mm/vmscan.c:should_run_aging() { ... if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; /* try to avoid aging, do gentle reclaim at the default priority */ if (sc->priority == DEF_PRIORITY) return false; /* better to run aging even though eviction is still possible */ return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq; }With
min_seqpinned atmax_seq- 3,evictable_min_seq()+MIN_NR_GENSnever reachesmax_seq, soshould_run_aging()returns false andscan_folios()drains eligible zones down to the youngest generationmax_seqwheresort_folio()disables tier protection and workingset promotion (min_gen==max_gen).
Quoted 7 lines
Even with the follow-up commits in this series (commit "mm/mglru: Run aging when pages are rare outside the youngest gen", commit "mm/mglru: Run aging when one type has no populatable folios", and commit "mm/mglru: Restrict eviction accounting to eligible zones"), at priorities above
DEF_PRIORITY/ 2scan_folios()caps seq atmax_seq- 1 andevict_folio_lists()returns 0 once older generations are empty in eligible zones, causingtry_to_shrink_lruvec()to break out before callingshould_run_aging()until priority drops to
I'm not sure why this is marked as high severity, as it seems more like a performance concern. Also, when we scan the youngest generation, tier protection and workingset promotion are no longer needed, as the folios are already in the youngest generation.
Actually, I created a silly test case that forcesGFP_DMA32 for anon folios:
mm/memory.c +0 -0
@@ -1251,7 +1251,7 @@ static inline struct folio
*folio_prealloc(struct mm_struct *src_mm,
if (need_zero)
new_folio = vma_alloc_zeroed_movable_folio(vma, addr);
else- new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, addr);
+ new_folio = vma_alloc_folio(GFP_USER | __GFP_DMA32, 0,
vma, addr);
if (!new_folio)
return NULL;
@@ -5053,7 +5053,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
if (!folio) {
/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IOdevices and zswap */
if (swap_entry_synchronous(si, entry))
- folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
+ folio = swapin_sync(entry,GFP_HIGHUSER_MOVABLE | __GFP_DMA32,
thp_swapin_suitable_orders(si, vmf) | BIT(0),
vmf, NULL, 0);
elseIn this case, the current patchset still performs much better on MGLRU than using CLRU.
MGLRU
*** Executing round 0 ***
set swappiness to 60
real 1m57.803s
user 25m34.833s
sys 6m8.024s
CLRU:
*** Executing round 0 ***
set swappiness to 60
Endless....(CLRU can't finish the kernel build at all)
If we do want to trigger aging earlier when evict_folio_lists() returns 0, rather than waiting until the priority drops to DEF_PRIORITY / 2, Bo Zhang shared an idea with me off-list that might address this issue.
mm/vmscan.c +0 -0
@@ -4033,8 +4033,6 @@ static void try_to_inc_min_seq(struct lruvec
*lruvec, int swappiness)
/* see the comment on lru_gen_folio */
if (swappiness && swappiness <= MAX_SWAPPINESS) {
- unsigned long seq = lrugen->max_seq - MIN_NR_GENS;
-
if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
min_seq[LRU_GEN_ANON] = seq;
else if (min_seq[LRU_GEN_FILE] > seq &&
min_seq[LRU_GEN_ANON] < seq)
@@ -5233,6 +5231,7 @@ static bool try_to_shrink_lruvec(struct lruvec
*lruvec, struct scan_control *sc)
unsigned long orig_file, orig_anon;
DEFINE_MAX_SEQ(lruvec);
bool no_populatable;+ long scanned;
if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) {
need_rotate = true;@@ -5241,8 +5240,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
orig_anon = nr[LRU_GEN_ANON];
orig_file = nr[LRU_GEN_FILE];- if (!evict_folio_lists(nr, lruvec, sc, swappiness,MIN_LRU_BATCH))
- break;
+ scanned = evict_folio_lists(nr, lruvec, sc,
swappiness, MIN_LRU_BATCH);
if (should_abort_scan(lruvec, sc))
break;
@@ -5263,6 +5261,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
should_age = true;
}+ if (!scanned)
+ break;
+
cond_resched();
}Quoted 6 lines
DEF_PRIORITY/ 2 and scansmax_seqwithout tier protection.[1] https://lore.kernel.org/r/CAN_Opa-AScm9=EeXO_Uy8VjbQ=bqY9P++rFP8H1RhUB4dOt6xw@mail.gmail.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Barry Song (Xiaomi)
reviewerPATCH v7 6/16 mm/mglru: return for unavailable anon reclaim earlier
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
No feature change, just shuffle some code, prepare for later commits.
--- mm/vmscan.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
mm/vmscan.c +13 -14
@@ -5055,20 +5055,6 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, struct mem_cgroup *memcg, int swappiness) {unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ]; -struct pglist_data *pgdat =lruvec_pgdat (lruvec); - -/* -* Proactive reclaim initiated by userspace for anonymous memory only. -* SWAPPINESS_ANON_ONLY is set only on the proactive reclaim path, so -* warn if it shows up elsewhere. When anon cannot be reclaimed (e.g. -* no swap), return 0 to skip the scan entirely, avoiding useless scan -* work when there is nothing eligible to reclaim. -*/ -if (swappiness ==SWAPPINESS_ANON_ONLY ) { -WARN_ON_ONCE (!sc->proactive); -if (!can_reclaim_anon_pages (memcg, pgdat->node_id, sc)) -return 0 ; - }lruvec_evictable_size (lruvec, swappiness, sizes); evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];
@@ -5121,6 +5107,19 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec); +/* +* Proactive reclaim initiated by userspace for anonymous memory only. +* SWAPPINESS_ANON_ONLY is set only on the proactive reclaim path, so +* warn if it shows up elsewhere. When anon cannot be reclaimed (e.g. +* no swap), return 0 to skip the scan entirely, avoiding useless scan +* work when there is nothing eligible to reclaim. +*/ +if (swappiness ==SWAPPINESS_ANON_ONLY ) { +WARN_ON_ONCE (!sc->proactive); +if (!can_reclaim_anon_pages (memcg,lruvec_pgdat (lruvec)->node_id, sc)) +return false; + } + nr_to_scan =get_nr_to_scan (lruvec, sc, memcg, swappiness);while (nr_to_scan >0 ) {int delta;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerPATCH v7 7/16 mm/mglru: balance anon and file scan budget by refault rate
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
Replace get_type_to_scan() and isolate_folios(), which selected one type per batch, with a proportional split of the scan budget between anon and file, and scan both types each round.
lru_gen_balance_scan() splits the total by
anon : file = swappiness * r_file * w_anon :
(MAX_SWAPPINESS - swappiness) * r_anon * w_file
where r_anon and r_file are the per-type refault rates scaled by 4096, clamped to [1, 4096]; a rate at the 4096 ceiling means the type refaults at least as often as it is evicted. The w factors bias the split toward the LRU sizes at mild swappiness and follow the IO cost ratio at the extremes. Both rate accumulators start at MIN_LRU_BATCH, so a type without eviction history reads as 100% until evictions accumulate, and the floor of 1 keeps a type that never refaults from zeroing out the other type's share. Swappiness 0 and SWAPPINESS_ANON_ONLY keep their single-type behavior; at swappiness == MAX_SWAPPINESS the file factor is zero, so anon gets the whole budget, as in the classical LRU; swappiness 1 no longer forces file-only scanning as the old code did.
The proportional protection and priority reduction are applied to the total once, before the split, and each budget is capped at its type's evictable size. Regular reclaim derives the total from the evictable sizes, so the budgets sum to it. The proactive path re-splits the remaining target each round, so eviction keeps working while either type makes progress, until the target is met or no progress is possible. The debugfs scan_control gets .priority = DEF_PRIORITY so eviction starts from the oldest gen and reports the default priority in the tracepoints.
--- mm/vmscan.c | 218 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 137 insertions(+), 81 deletions(-)
mm/vmscan.c +137 -81
@@ -4858,79 +4858,110 @@ static int get_tier_idx(struct lruvec *lruvec, int type)return tier -1 ; } -static int get_type_to_scan (struct lruvec *lruvec,int swappiness) +/* +* Balance the anon vs file scan budget using MGLRU's per-type refault +* rates, where swappiness is the relative IO cost of anon vs file (see +* Documentation/admin-guide/sysctl/vm.rst). +* +* On entry, nr[] holds the evictable size of each type, on return it +* holds each type's scan budget, bounded by its evictable size. +* +* LRU lock not needed: a stale snapshot merely skews the split for one pass, +* and MGLRU aging averages the fault rate so we can simply read it, no need +* to do any extra work here. If reclaim is blocked by unfair type hotness or +* stall aging, the priority will raise and a raised priority drives the +* aging. +* +* Calculation is basically based on following formula with a LRU size bias: +* +* anon : file = swappiness * r_file * w_anon : +* (MAX_SWAPPINESS - swappiness) * r_anon * w_file +* +* r_file / r_anon: refault rate for file / anon. +* w_file / w_anon: LRU size weight of file / anon. +* +* We don't calculate the scan cost, MGLRU uses lazy promotion, which will +* cancel the scan budget natually. +*/ +static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, +unsigned long total_scan,unsigned long nr[]) { -struct ctrl_pos sp, pv; +struct lru_gen_folio *lrugen = &lruvec->lrugen; +u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor; +u64 anon_scan, file_scan, total; +u64 anon_weight, file_weight; +int size_bias; -if (swappiness <=MIN_SWAPPINESS +1 ) -return LRU_GEN_FILE ; + total = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ]; +if (!total) +return ; -if (swappiness >=MAX_SWAPPINESS ) -return LRU_GEN_ANON ; -/* -* Compare the sum of all tiers of anon with that of file to determine -* which type to scan. -*/ -read_ctrl_pos (lruvec,LRU_GEN_ANON ,LRU_TIER_MIN ,LRU_TIER_MAX , - swappiness, &sp); -read_ctrl_pos (lruvec,LRU_GEN_FILE ,LRU_TIER_MIN ,LRU_TIER_MAX , -MAX_SWAPPINESS - swappiness, &pv); +if (swappiness <=MIN_SWAPPINESS ) { + nr[LRU_GEN_ANON ] =0 ; + nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]); +return ; + } -return positive_ctrl_err (&sp, &pv); -} +if (swappiness ==SWAPPINESS_ANON_ONLY ) { + nr[LRU_GEN_FILE ] =0 ; + nr[LRU_GEN_ANON ] =min_t (u64, total_scan, nr[LRU_GEN_ANON ]); +return ; + } -static inline bool is_single_type_reclaim (int swappiness) -{ -return swappiness ==MIN_SWAPPINESS || - swappiness ==SWAPPINESS_ANON_ONLY ; -} +for (int type =LRU_GEN_ANON ; type <=LRU_GEN_FILE ; type++) { +int hist =lru_hist_from_seq (READ_ONCE (lrugen->min_seq[type])); +/* start at MIN_LRU_BATCH to smooth the rate on gen start */ +u64 refaulted =MIN_LRU_BATCH , evicted =MIN_LRU_BATCH ; -static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int swappiness, -struct list_head *list,int *isolated, -int *isolate_type,int *isolate_scanned) -{ -bool type_fallback_allowed = !is_single_type_reclaim (swappiness); -int type =get_type_to_scan (lruvec, swappiness); -int total_scanned =0 , scanned, tier; -bool tried = false; +for (int tier =0 ; tier <MAX_NR_TIERS ; tier++) { + refaulted +=READ_ONCE (lrugen->avg_refaulted[type][tier]); + refaulted +=atomic_long_read (&lrugen->refaulted[hist][type][tier]); -retry: - tier =get_tier_idx (lruvec, type); - scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + evicted +=READ_ONCE (lrugen->avg_total[type][tier]); + evicted +=atomic_long_read (&lrugen->evicted[hist][type][tier]); + } - total_scanned += scanned; -if (*isolated) { - *isolate_type = type; - *isolate_scanned = scanned; -return total_scanned; +/* +* The rate scaled by 4096 to survive integer division, clamped +* to [1, 4096] (~0% - 100%), the floor of 1 keeps a type that +* never refaults from zeroing out the other type's share. +*/ + refault_rate[type] =mul_u64_u64_div_u64 (refaulted,4096 , evicted); + refault_rate[type] =clamp_t (u64, refault_rate[type],1 ,4096 ); }/* -* We are running out of the current reclaim type. Fall back to -* the other type if allowed. +* The more extreme the swappiness, the wider the IO cost gap between +* anon and file, so the split should follow the IO cost ratio rather +* than the sizes. With a mild swappiness, balance the scan by the +* actual LRU size more. */ -if (!scanned && !tried && type_fallback_allowed) { - type = !type; - tried = true; - type_fallback_allowed = false; - goto retry; - } + size_bias =max (1 ,min (swappiness,MAX_SWAPPINESS - swappiness)); + anon_weight = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ] / size_bias; + file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; + + anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; + +/* Cross-multiply the ratio above. */ + anon_scan =mul_u64_u64_div_u64 (total_scan, anon_factor, anon_factor + file_factor); + file_scan = (u64 )total_scan - anon_scan; +/* -* We scanned some folios but failed to isolate any due to promotions, -* protections, or races. Retry once to avoid a larger loop. +* Bound each type by its evictable size. Hand the exceeding part to +* the other type. Any excess left when total_scan exceeds both sizes +* is dropped, since it only stresses the scanner with no gain. */ -if (scanned && !tried) { - tried = true; - goto retry; - } - -return total_scanned; +if (anon_scan > nr[LRU_GEN_ANON ]) + file_scan += anon_scan - nr[LRU_GEN_ANON ]; +else if (file_scan > nr[LRU_GEN_FILE ]) + anon_scan += file_scan - nr[LRU_GEN_FILE ]; + nr[LRU_GEN_ANON ] =min_t (u64, anon_scan, nr[LRU_GEN_ANON ]); + nr[LRU_GEN_FILE ] =min_t (u64, file_scan, nr[LRU_GEN_FILE ]); }static int evict_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int swappiness) +struct scan_control *sc,int type,int swappiness) {LIST_HEAD (list);LIST_HEAD (clean);
@@ -4939,8 +4970,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,enum node_stat_item item;struct reclaim_stat stat;struct lru_gen_mm_walk *walk; -int scanned, reclaimed; -int isolated =0 , nr_isolated =0 , type, type_scanned; +int tier, scanned, reclaimed; +int isolated =0 , nr_isolated =0 ;unsigned long total_reclaimed =0 ;bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -4951,8 +4982,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,/* In case folio deletion left empty old gens, flush them */ try_to_inc_min_seq (lruvec, swappiness); - scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, - &list, &isolated, &type, &type_scanned); + tier =get_tier_idx (lruvec, type); + scanned =scan_folios (nr_to_scan, lruvec, sc, + type, tier, &list, &isolated); nr_isolated = isolated;if (nr_isolated) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
@@ -4974,7 +5006,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (isolated)handle_reclaim_writeback (isolated, pgdat, sc, &stat);trace_mm_vmscan_lru_shrink_inactive (pgdat->node_id, - type_scanned, reclaimed, &stat, sc, + scanned, reclaimed, &stat, sc, type ?LRU_INACTIVE_FILE :LRU_INACTIVE_ANON );list_for_each_entry_safe_reverse (folio, next, &list, lru) {
@@ -5051,18 +5083,18 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; } -static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc, -struct mem_cgroup *memcg,int swappiness) +static void lru_gen_prepare_scan (struct lruvec *lruvec,struct scan_control *sc, +struct mem_cgroup *memcg,int swappiness, +unsigned long nr_to_scan[]) { -unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ]; - -lruvec_evictable_size (lruvec, swappiness, sizes); - evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ]; +unsigned long total; - nr_to_scan =apply_proportional_protection (memcg, sc, evictable); - nr_to_scan >>= sc->priority; +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); + total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; -return nr_to_scan; + total =apply_proportional_protection (memcg, sc, total); + total >>= sc->priority; +lru_gen_balance_scan (lruvec, swappiness, total, nr_to_scan); }static bool should_abort_scan (struct lruvec *lruvec,struct scan_control *sc)
@@ -5095,6 +5127,30 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)return true; } +static long evict_folio_lists (unsigned long nr_to_scan[],struct lruvec *lruvec, +struct scan_control *sc,int swappiness,unsigned long max_batch) +{ +int type; +unsigned long batch, scanned, total_scanned =0 ; + +/* +* Scan both types with the given budgets; the caller checks if +* reclaim is satisfied after each pass. +*/ +for (type =LRU_GEN_FILE ; type >=LRU_GEN_ANON ; type--) { +if (!nr_to_scan[type]) +continue ; + + batch =min (nr_to_scan[type], max_batch); + scanned =evict_folios (batch, lruvec, sc, type, swappiness); + + nr_to_scan[type] -=min (nr_to_scan[type], scanned); + total_scanned += scanned; + } + +return total_scanned; +} +/* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg
@@ -5102,8 +5158,8 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)*/ static bool try_to_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) { +unsigned long nr[ANON_AND_FILE ];bool need_rotate = false, should_age = false; -long nr_batch, nr_to_scan;int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -5120,9 +5176,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)return false; } - nr_to_scan =get_nr_to_scan (lruvec, sc, memcg, swappiness); -while (nr_to_scan >0 ) { -int delta; +lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr); + +while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) {DEFINE_MAX_SEQ (lruvec);if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) {
@@ -5137,9 +5193,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) should_age = true; } - nr_batch =min (nr_to_scan,MIN_LRU_BATCH ); - delta =evict_folios (nr_batch, lruvec, sc, swappiness); -if (!delta) +if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ))break ;if (should_abort_scan (lruvec, sc))
@@ -5152,7 +5206,6 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; - nr_to_scan -= delta;cond_resched (); }
@@ -5764,8 +5817,8 @@ static int run_aging(struct lruvec *lruvec, unsigned long seq,static int run_eviction (struct lruvec *lruvec,unsigned long seq,struct scan_control *sc,int swappiness,unsigned long nr_to_reclaim) { -int nr_batch;DEFINE_MAX_SEQ (lruvec); +unsigned long nr_to_scan[ANON_AND_FILE ];if (seq +MIN_NR_GENS > max_seq)return -EINVAL ;
@@ -5781,8 +5834,11 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coif (sc->nr_reclaimed >= nr_to_reclaim)return 0 ; - nr_batch =min (nr_to_reclaim - sc->nr_reclaimed,MAX_LRU_BATCH ); -if (!evict_folios (nr_batch, lruvec, sc, swappiness)) +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lru_gen_balance_scan (lruvec, swappiness, + nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); + +if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))return 0 ;cond_resched ();
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 07/16] mm/mglru: balance anon and file scan budget by refault rate
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] mm/mglru: non-real-name handle in Co-developed-by and Signed-off-by trailers - [Medium] mm/mglru…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm/mglru: non-real-name handle in Co-developed-by and Signed-off-by trailers
- [Medium] mm/mglru: missing DEF_PRIORITY initialization in lru_gen_seq_write()
- [Medium] mm/mglru: lru_gen_balance_scan() budgets un-scannable young generations
--
commit 730defb97ea747f35c6996e786706ce8e9143280
Author: Kairui Song <[email protected]>
Quoted 34 lines
mm/mglru: balance anon and file scan budget by refault rate
Replace
get_type_to_scan()andisolate_folios(), which selected one type per batch, with a proportional split of the scan budget between anon and file, and scan both types each round.
lru_gen_balance_scan()splits the total byanon : file = swappiness *
r_file*w_anon:
(MAX_SWAPPINESS- swappiness) *r_anon*w_filewhere
r_anonandr_fileare the per-type refault rates scaled by 4096, clamped to [1, 4096]; a rate at the 4096 ceiling means the type refaults at least as often as it is evicted. The w factors bias the split toward the LRU sizes at mild swappiness and follow the IO cost ratio at the extremes. Both rate accumulators start atMIN_LRU_BATCH, so a type without eviction history reads as 100% until evictions accumulate, and the floor of 1 keeps a type that never refaults from zeroing out the other type's share. Swappiness 0 andSWAPPINESS_ANON_ONLYkeep their single-type behavior; at swappiness ==MAX_SWAPPINESSthe file factor is zero, so anon gets the whole budget, as in the classical LRU; swappiness 1 no longer forces file-only scanning as the old code did.The proportional protection and priority reduction are applied to the total once, before the split, and each budget is capped at its type's evictable size. Regular reclaim derives the total from the evictable sizes, so the budgets sum to it. The proactive path re-splits the remaining target each round, so eviction keeps working while either type makes progress, until the target is met or no progress is possible. The debugfs
scan_controlgets .priority =DEF_PRIORITYso eviction starts from the oldest gen and reports the default priority in the tracepoints.
Quoted 4 lines
Signed-off-by Kairui Song <[email protected]>Co-developed-by lyugaofei <[email protected]>Signed-off-by lyugaofei <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>
[Severity: High]
Under Documentation/process/submitting-patches.rst, Co-developed-by: and
Signed-off-by: tags must use a real full name rather than an account handle ("lyugaofei").
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 26 lines
mm/vmscan.c +9
-9@@ -5052,18 +5084,18 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; } -static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc, -struct mem_cgroup *memcg,int swappiness) +static void lru_gen_prepare_scan (struct lruvec *lruvec,struct scan_control *sc, +struct mem_cgroup *memcg,int swappiness, +unsigned long nr_to_scan[]) { -unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ]; - -lruvec_evictable_size (lruvec, swappiness, sizes); - evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ]; +unsigned long total; - nr_to_scan =apply_proportional_protection (memcg, sc, evictable); - nr_to_scan >>= sc->priority; +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); + total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; -return nr_to_scan; + total =apply_proportional_protection (memcg, sc, total); + total >>= sc->priority; +lru_gen_balance_scan (lruvec, swappiness, total, nr_to_scan); }
[ ... ]
Quoted 13 lines
mm/vmscan.c +13
-0+static long evict_folio_lists (unsigned long nr_to_scan[],struct lruvec *lruvec, +struct scan_control *sc,int swappiness,unsigned long max_batch) +{ +int type; +unsigned long batch, scanned, total_scanned =0 ; + +/* +* Scan both types with the given budgets; the caller checks if +* reclaim is satisfied after each pass. +*/ +for (type =LRU_GEN_FILE ; type >=LRU_GEN_ANON ; type--) { +if (!nr_to_scan[type]) +continue ;
mm/vmscan.c static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) · line 5152
Quoted 10 lines
mm/vmscan.c +10
-0+ + batch =min (nr_to_scan[type], max_batch); + scanned =evict_folios (batch, lruvec, sc, type, swappiness); + + nr_to_scan[type] -=min (nr_to_scan[type], scanned); + total_scanned += scanned; + } + +return total_scanned; +}
[Severity: Medium]
Can lru_gen_balance_scan() assign scan budget to a type whose scannable generations are empty, losing that budget instead of handing it to the other type?
Previously, isolate_folios() fell back to !type whenever scan_folios() returned 0. Now lru_gen_prepare_scan() caps each type's budget by lruvec_evictable_size(), which sums all generations from min_seq[type] through max_seq:
mm/vmscan.c:lruvec_evictable_size() {
...
for_each_evictable_type(type, swappiness) {
for (seq = min_seq[type]; seq <= max_seq; seq++)
nr[type] += lruvec_gen_size(lrugen, type, seq);
}
}At sc->priority == DEF_PRIORITY (or sc->priority > DEF_PRIORITY / 2 after commit "mm/mglru: lift youngest gen protection for higher priority"), scan_folios() only scans older generations and cannot scan max_seq - 1 or max_seq. When one type has drained its scannable generations while still holding folios in younger protected generations, lruvec_evictable_size() reports a non-zero nr[type], so lru_gen_balance_scan() does not transfer its share of total_scan to the other type.
When the other type receives a 0 share (such as at swappiness == MAX_SWAPPINESS when anon's scannable generations are empty, or when small total_scan rounds down to 0), evict_folio_lists() returns 0 and aborts try_to_shrink_lruvec() immediately even though the other type still has cold evictable folios. When both types receive non-zero budgets, evict_folios() returns 0 on the drained type and leaves its nr_to_scan entry unchanged, causing repeated no-op evict_folios() calls on every MIN_LRU_BATCH pass before discarding that unspent budget.
[ ... ]
Quoted 12 lines
mm/vmscan.c +1
-1@@ -5765,8 +5818,8 @@ static int run_aging(struct lruvec *lruvec, unsigned long seq,static int run_eviction (struct lruvec *lruvec,unsigned long seq,struct scan_control *sc,int swappiness,unsigned long nr_to_reclaim) { -int nr_batch;DEFINE_MAX_SEQ (lruvec); +unsigned long nr_to_scan[ANON_AND_FILE ];if (seq +MIN_NR_GENS > max_seq)return -EINVAL ;@@ -5782,8 +5835,11 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coif (sc->nr_reclaimed >= nr_to_reclaim)
mm/vmscan.c static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · line 5843
Quoted 10 lines
mm/vmscan.c +5
-2return 0 ; - nr_batch =min (nr_to_reclaim - sc->nr_reclaimed,MAX_LRU_BATCH ); -if (!evict_folios (nr_batch, lruvec, sc, swappiness)) +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lru_gen_balance_scan (lruvec, swappiness, + nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); + +if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))return 0 ;
[Severity: Medium]
Was .priority = DEF_PRIORITY accidentally dropped from the scan_control in lru_gen_seq_write()?
The commit message notes that the debugfs scan_control gets .priority = DEF_PRIORITY so eviction starts from the oldest generation and reports the default priority in tracepoints, but lru_gen_seq_write() leaves sc.priority zero-initialized:
mm/vmscan.c:lru_gen_seq_write() {
...
struct scan_control sc = {
.may_writepage = true,
.may_unmap = true,
.may_swap = true,
.reclaim_idx = MAX_NR_ZONES - 1,
.gfp_mask = GFP_KERNEL,
.proactive = true,
};
...
}
With sc.priority == 0, run_eviction() -> evict_folio_lists() -> evict_folios() -> scan_folios() takes the sc->priority != DEF_PRIORITY branch (seq = lrugen->max_seq) and can evict from the youngest generations whenever older generations of a type are empty, while reporting priority 0 in trace_mm_vmscan_lru_shrink_inactive().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Barry Song (Xiaomi)
reviewerPATCH v7 8/16 mm/mglru: lift youngest gen protection for higher priority
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
The forced protection of younger generations contradicts the goal of minimizing the I/O cost of reclaim. Under high memory pressure, MGLRU can repeatedly trigger premature OOMs, while the classic LRU does not.
The problem is that aging may fail to keep up with reclaim. When this happens, reclaimers may all abort prematurely because they see only empty generations or just two generations remaining.
Proactive aging does not solve this problem either. The number of threads performing proactive aging is limited, while the number of concurrent reclaimers can be much larger. This is a fundamental limitation of MGLRU. One possible solution is to use an aging lock. However, the aging lock introduces significant jitter under heavy memory pressure and has been reported to cause noticeable lags on Android.
Furthermore, the generation-based reclaim model can conflict with the goal of swappiness, which is to minimize I/O cost rather than blindly follow folio hotness.
The previous commit allowed reclaimers running at non-default priorities to reclaim up to lrugen->max_seq, while reclaimers at the default priority could reclaim only up to lrugen->min_seq.
This patch further refines generation protection based on reclaim priority. At the default priority, reclaimers can reclaim generations down to lrugen->min_seq - MIN_NR_GENS. As reclaim priority increases, reclaimers are progressively allowed to reclaim younger generations.
With this graduated protection, premature OOMs are no longer observed, while swappiness is also respected.
--- mm/vmscan.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
mm/vmscan.c +6 -4
@@ -4767,12 +4767,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (!list_empty (list));/* -* Respect the gen hotness distribution at low priority to avoid -* aging and scan cost, but prefer to satisfy the reclaimers request -* when under pressure to respect swappiness more and reduce IO cost. +* Respect hotness at default priority, and only reach into the +* newest generation when under severe pressure, so hot folios are +* not evicted as soon as priority is raised. */ if (sc->priority ==DEF_PRIORITY ) - seq = lrugen->min_seq[type]; + seq = lrugen->max_seq -MIN_NR_GENS ; +else if (sc->priority >DEF_PRIORITY /2 ) + seq = lrugen->max_seq -1 ;else seq = lrugen->max_seq;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerPATCH v7 9/16 mm/mglru: Avoid placing rejected folios below populated gen
In reply to Barry Song (Xiaomi)We used to scan only the oldest generation in scan_folios(). In evict_folios(), we checked whether lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]. If so, the folio was marked active and moved to younger generations.
This was itself buggy, as folios in generation min_seq[type] - 1 would not be marked active by folio_set_active(), potentially causing a cold/hot inversion.
Besides this issue, the old approach also differs from the current behavior, where we can scan fewer than two generations. Now that we can scan younger generations, including the youngest generation, this no longer prevents rejected folios from being placed in a generation below or at the populated generation. Such rejected folios would then be scanned again and again, resulting in extremely low pg_steal/pg_scan ratios. For example:
pgsteal_file: 2312605
pgscan_file: 36465693
reclamation efficiency: 6.34%
We pass scanned_seq to move_folios_to_lru() and make sure we either promote the folio or place it in the youngest generation when the populated generation is already the youngest one.
Also, folio_set_active() is no longer needed.
With this change, the results become:
pgsteal_file: 2683815
pgscan_file: 4224244
reclamation efficiency: 63.53%
The sys time for building the kernel also decreased from 4m30.532s to 3m50.188s while swappiness==1.
--- include/linux/mm_inline.h | 22 +++++++++++++++++++++- mm/vmscan.c | 24 ++++++++++++------------ 2 files changed, 33 insertions(+), 13 deletions(-)
include/linux/mm_inline.h +21 -1
@@ -332,7 +332,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); } -static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +static inline bool __lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq,bool reclaiming) {unsigned long seq;unsigned long flags;
@@ -347,6 +348,8 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return false; seq =lru_gen_folio_seq (lruvec, folio, reclaiming); +if (scanned_seq !=-1UL ) + seq =max (seq,min (scanned_seq +1 , lrugen->max_seq)); gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */
@@ -362,6 +365,17 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return true; } +static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +return __lru_gen_add_folio (lruvec, folio,-1UL , reclaiming); +} + +static inline bool lru_gen_add_rejected_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq) +{ +return __lru_gen_add_folio (lruvec, folio, scanned_seq, false); +} +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {unsigned long flags;
@@ -420,6 +434,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return false; } +static inline bool lru_gen_add_rejected_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq) +{ +return false; +} +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {return false;
mm/vmscan.c +12 -12
@@ -1864,7 +1864,8 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) +static unsigned int move_folios_to_lru (struct list_head *list,unsigned long seq, +bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;
@@ -1921,7 +1922,7 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate) !folio_test_referenced(folio) && folio_ref_count(folio) == folio_expected_ref_count(folio))lruvec_add_folio_tail (lruvec, folio); - else + else if (seq == -1UL || !lru_gen_add_rejected_folio(lruvec, folio, seq)) lruvec_add_folio(lruvec, folio); nr_pages = folio_nr_pages(folio);
@@ -2040,7 +2041,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed =shrink_folio_list (&folio_list, pgdat, sc, &stat, false,lruvec_memcg (lruvec)); -move_folios_to_lru (&folio_list, true); +move_folios_to_lru (&folio_list,-1UL , true);mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted);
@@ -2151,8 +2152,8 @@ static void shrink_active_list(unsigned long nr_to_scan,/* * Move folios back to the lru list. */ - nr_activate =move_folios_to_lru (&l_active, false); - nr_deactivate =move_folios_to_lru (&l_inactive, false); + nr_activate =move_folios_to_lru (&l_active,-1UL , false); + nr_deactivate =move_folios_to_lru (&l_inactive,-1UL , false);count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate);
@@ -4748,7 +4749,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +unsigned long *scanned_seq) {enum node_stat_item item;int zone_idx;
@@ -4779,6 +4781,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, seq = lrugen->max_seq; seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + *scanned_seq = seq; gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq);
@@ -4975,6 +4978,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int tier, scanned, reclaimed;int isolated =0 , nr_isolated =0 ;unsigned long total_reclaimed =0 ; +unsigned long scanned_seq;bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec);
@@ -4986,7 +4990,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, &list, &isolated); + type, tier, &list, &isolated, &scanned_seq); nr_isolated = isolated;if (nr_isolated) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
@@ -5012,8 +5016,6 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);list_for_each_entry_safe_reverse (folio, next, &list, lru) { -DEFINE_MIN_SEQ (lruvec); -/* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */ if (!folio_evictable (folio))continue ;
@@ -5032,11 +5034,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* so this effectively promotes them by at least one generation. */ folio_set_lru_refs (folio,0 ); -if (lru_gen_folio_seq (lruvec, folio, false) == min_seq[type]) -folio_set_active (folio); } -move_folios_to_lru (&list, false); +move_folios_to_lru (&list, scanned_seq, false); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) {
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerPATCH v7 10/16 mm/mglru: Run aging when pages are rare outside the youngest gen
In reply to Barry Song (Xiaomi)From: lyugaofei <[email protected]>
Run aging when the total number of pages outside the youngest generation across both types falls below 1/MAX_NR_GENS of the number of pages in the youngest generation. For extreme swappiness values such as 1 and 200, also consider the balance of the corresponding type.
This improves reclaim efficiency, especially at extreme swappiness values, while reducing system time.
To avoid the overhead of aging, we only allow imbalance-based aging when the reclaim priority is low enough that we might reclaim the youngest generation.
--- mm/vmscan.c | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-)
mm/vmscan.c +40 -1
@@ -5068,6 +5068,41 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,struct scan_control *sc, +int swappiness) +{ +unsigned long youngest[ANON_AND_FILE ] = {0 }, old[ANON_AND_FILE ] = {0 }; +struct lru_gen_folio *lrugen = &lruvec->lrugen; +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); +int type; + +if (sc->priority >DEF_PRIORITY /2 ) +return false; + +for_each_evictable_type (type, swappiness) { +unsigned long seq; + + youngest[type] =lruvec_gen_size (lrugen, type, max_seq); +for (seq = min_seq[type]; seq < max_seq; seq++) + old[type] +=lruvec_gen_size (lrugen, type, seq); + } + +/* the vast majority of pages sit in the youngest gen */ +if (youngest[LRU_GEN_ANON ] + youngest[LRU_GEN_FILE ] > + (old[LRU_GEN_ANON ] + old[LRU_GEN_FILE ]) *MAX_NR_GENS ) +return true; + +/* Swappiness extremes: the corresponding type matters */ +if (swappiness ==MIN_SWAPPINESS +1 ) +return youngest[LRU_GEN_FILE ] > old[LRU_GEN_FILE ] *MAX_NR_GENS ; + +if (swappiness ==MAX_SWAPPINESS ) +return youngest[LRU_GEN_ANON ] > old[LRU_GEN_ANON ] *MAX_NR_GENS ; + +return false; +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) {
@@ -5082,7 +5117,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* run aging when folios outside the youngest gen are rare */ +return lru_gen_imbalanced (lruvec, sc, swappiness); } static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc,
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 10/16] mm/mglru: Run aging when pages are rare outside the youngest gen
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm/mglru: single-word handle used in commit Author and Signed-off-by trailer --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: single-word handle used in commit Author and Signed-off-by trailer
--
commit e3df19968a61ac9c4f39eb92e538d862f157c5ad
Author: lyugaofei <[email protected]>
Quoted 11 lines
mm/mglru: Run aging when pages are rare outside the youngest gen
Run aging when the total number of pages outside the youngest generation across both types falls below 1/MAX_NR_GENS of the number of pages in the youngest generation. For extreme swappiness values such as 1 and 200, also consider the balance of the corresponding type.
This improves reclaim efficiency, especially at extreme swappiness values, while reducing system time.
Quoted 7 lines
To avoid the overhead of aging, we only allow imbalance-based aging when the reclaim priority is low enough that we might reclaim the youngest generation.
Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>
[Severity: High]
The Author header and the first Signed-off-by trailer use the single-word handle "lyugaofei" instead of a full real human name, which does not meet the Developer's Certificate of Origin requirements in Documentation/process/submitting-patches.rst.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Barry Song
reviewerRe: [PATCH v7 10/16] mm/mglru: Run aging when pages are rare outside the youngest gen
Quoted 29 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: single-word handle used in commit Author and Signed-off-by trailer
--commit
e3df19968a61ac9c4f39eb92e538d862f157c5ad
Author: lyugaofei <[email protected]>Quoted 11 lines
mm/mglru: Run aging when pages are rare outside the youngest gen
Run aging when the total number of pages outside the youngest generation across both types falls below 1/MAX_NR_GENS of the number of pages in the youngest generation. For extreme swappiness values such as 1 and 200, also consider the balance of the corresponding type.
This improves reclaim efficiency, especially at extreme swappiness values, while reducing system time.
Quoted 7 lines
To avoid the overhead of aging, we only allow imbalance-based aging when the reclaim priority is low enough that we might reclaim the youngest generation.
Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>
Quoted 5 lines
[Severity: High]
The Author header and the first Signed-off-by trailer use the single-word handle "lyugaofei" instead of a full real human name, which does not meet the Developer's Certificate of Origin requirements inDocumentation/process/submitting-patches.rst.
Okay, let me change it to
Gaofei Lyu <[email protected]>
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Barry Song (Xiaomi)
reviewerPATCH v7 11/16 mm/mglru: Run aging when one type has no populatable folios
In reply to Barry Song (Xiaomi)We may have scan budget remaining for anon or file but no folios in populatable generations to scan. In this case, scan_folios() may waste time scanning that type while the other type still has populatable folios.
Run aging in this case to make more folios populatable. This also helps respect the scan budget distribution determined by swappiness.
--- mm/vmscan.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
mm/vmscan.c +22 -8
@@ -5104,7 +5104,8 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, struct scan_control *sc, }static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq, -struct scan_control *sc,int swappiness) +struct scan_control *sc,int swappiness, +bool no_populatable) {DEFINE_MIN_SEQ (lruvec);
@@ -5116,6 +5117,13 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (sc->priority ==DEF_PRIORITY )return false; +/* +* Run aging if scan budget remains but no folios +* in populatable generations are available to scan. +*/ +if (no_populatable) +return true; +/* better to run aging even though eviction is still possible */ if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq)return true;
@@ -5220,20 +5228,17 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { +unsigned long orig_file, orig_anon;DEFINE_MAX_SEQ (lruvec); +bool no_populatable;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) { need_rotate = true;break ; } -if (should_run_aging (lruvec, max_seq, sc, swappiness)) { -trace_mm_mglru_aging (lruvec, sc, swappiness); -if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) - need_rotate = true; - should_age = true; - } - + orig_anon = nr[LRU_GEN_ANON ]; + orig_file = nr[LRU_GEN_FILE ];if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ))break ;
@@ -5247,6 +5252,15 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; + no_populatable = (nr[LRU_GEN_ANON ] && nr[LRU_GEN_ANON ] == orig_anon) || + (nr[LRU_GEN_FILE ] && nr[LRU_GEN_FILE ] == orig_file); +if (should_run_aging (lruvec, max_seq, sc, swappiness, no_populatable)) { +trace_mm_mglru_aging (lruvec, sc, swappiness); +if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) + need_rotate = true; + should_age = true; + } +cond_resched (); }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 11/16] mm/mglru: Run aging when one type has no populatable folios
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] mm/mglru: skipped aging in try_to_shrink_lruvec() when zero folios are scanned - [Medium] mm/mglru…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm/mglru: skipped aging in try_to_shrink_lruvec() when zero folios are scanned
- [Medium] mm/mglru: repeated aging in try_to_shrink_lruvec() when no_populatable persists
- [Low] mm/mglru: stale max_seq passed to should_run_aging() in try_to_shrink_lruvec()
--
commit b68a220713e0f1b121c4874a48ac05f55108264a
Author: Barry Song (Xiaomi) <[email protected]>
mm/mglru: Run aging when one type has no populatable folios
This commit moves the aging check in try_to_shrink_lruvec() after evict_folio_lists() and passes a no_populatable flag to should_run_aging() so aging runs when one LRU type makes no scan progress despite remaining scan budget.
Quoted 43 lines
mm/vmscan.c +11
-3@@ -5105,7 +5105,8 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, struct scan_control *sc, }static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq, -struct scan_control *sc,int swappiness) +struct scan_control *sc,int swappiness, +bool no_populatable) {DEFINE_MIN_SEQ (lruvec);@@ -5117,6 +5118,13 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (sc->priority ==DEF_PRIORITY )return false; +/* +* Run aging if scan budget remains but no folios +* in populatable generations are available to scan. +*/ +if (no_populatable) +return true; +/* better to run aging even though eviction is still possible */ if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq)return true;@@ -5221,20 +5229,17 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { +unsigned long orig_file, orig_anon;DEFINE_MAX_SEQ (lruvec); +bool no_populatable;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) { need_rotate = true;break ; } -if (should_run_aging (lruvec, max_seq, sc, swappiness)) { -trace_mm_mglru_aging (lruvec, sc, swappiness);
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5245
Quoted 10 lines
mm/vmscan.c +2
-5-if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) - need_rotate = true; - should_age = true; - } - + orig_anon = nr[LRU_GEN_ANON ]; + orig_file = nr[LRU_GEN_FILE ];if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ))break ;
[Severity: High]
Does moving should_run_aging() and try_to_inc_max_seq() after evict_folio_lists() skip aging whenever evict_folio_lists() scans zero folios?
mm/vmscan.c:scan_folios() {
...
if (sc->priority == DEF_PRIORITY)
seq = lrugen->max_seq - MIN_NR_GENS;
else if (sc->priority > DEF_PRIORITY / 2)
seq = lrugen->max_seq - 1;
else
seq = lrugen->max_seq;
seq = lruvec_populated_min_seq(lruvec, type, sc->reclaim_idx + 1, seq);
...
}At DEF_PRIORITY, when only MIN_NR_GENS generations remain (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq), scan_folios() caps seq to max_seq - MIN_NR_GENS, which is below min_seq and already empty. evict_folio_lists() then returns 0 and breaks out of the loop before reaching should_run_aging(), making the mandatory evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq aging check unreachable at DEF_PRIORITY.
Similarly, at DEF_PRIORITY / 2 < sc->priority < DEF_PRIORITY, scan_folios() caps seq to max_seq - 1. Whenever all evictable folios for the budgeted type(s) are in max_seq (for example when both types only have folios in max_seq, or when only one type has budget left), evict_folio_lists() returns 0 and breaks before aging can run, leaving need_rotate false as well so shrink_one() fails to rotate a sizable memcg.
Finally, when sc->priority <= DEF_PRIORITY / 2, running evict_folio_lists() before should_run_aging() scans directly from max_seq (where min_gen == max_gen in sort_folio() disables tier protection) before aging can split the youngest generation, and breaks without aging if the first batch satisfies should_abort_scan().
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5257
Quoted 6 lines
mm/vmscan.c +2
-0@@ -5248,6 +5253,15 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; + no_populatable = (nr[LRU_GEN_ANON ] && nr[LRU_GEN_ANON ] == orig_anon) || + (nr[LRU_GEN_FILE ] && nr[LRU_GEN_FILE ] == orig_file);
[Severity: Medium]
Can no_populatable stay true across every MIN_LRU_BATCH iteration and trigger repeated aging when one type still scans 0 folios after aging?
nr[] is computed once in lru_gen_prepare_scan() and is only decremented by scanned folios. If scan_folios() continues to return 0 for one type while the other type makes progress (for instance when sc->priority <= DEF_PRIORITY / 2 already scans up to max_seq, when max_seq has no eligible folios of that type, or when walk_mm() promotes all referenced folios into the new youngest generation), nr[type] remains equal to orig_anon or orig_file. In cgroup reclaim (!root_reclaim(sc)), the loop does not break on should_age, so should_run_aging() returns true and calls try_to_inc_max_seq() on every 64-page batch of the other type; in root_reclaim(sc), it runs unnecessary aging and aborts scanning the populated type after one extra batch.
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5262
Quoted 6 lines
mm/vmscan.c +6
-0+if (should_run_aging (lruvec, max_seq, sc, swappiness, no_populatable)) { +trace_mm_mglru_aging (lruvec, sc, swappiness); +if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) + need_rotate = true; + should_age = true; + }
[Severity: Low]
Can should_run_aging() see a stale max_seq alongside a fresh min_seq here?
DEFINE_MAX_SEQ(lruvec) snapshots max_seq at the start of the loop before evict_folio_lists(), whereas should_run_aging() reads min_seq after evict_folio_lists() finishes. If a concurrent reclaimer advances max_seq and min_seq during eviction, should_run_aging() compares the new min_seq against the old max_seq, which can falsely trigger the aging branch, fail try_to_inc_max_seq() on the stale max_seq, and set should_age = true so root_reclaim(sc) aborts on the next pass.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Barry Song
reviewerRe: [PATCH v7 11/16] mm/mglru: Run aging when one type has no populatable folios
Quoted 108 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm/mglru: skipped aging intry_to_shrink_lruvec()when zero folios are scanned
- [Medium] mm/mglru: repeated aging intry_to_shrink_lruvec()whenno_populatablepersists
- [Low] mm/mglru: stalemax_seqpassed toshould_run_aging()intry_to_shrink_lruvec()
--commit
b68a220713e0f1b121c4874a48ac05f55108264a
Author: Barry Song (Xiaomi) <[email protected]>mm/mglru: Run aging when one type has no populatable folios
This commit moves the aging check in
try_to_shrink_lruvec()afterevict_folio_lists()and passes ano_populatableflag toshould_run_aging()so aging runs when one LRU type makes no scan progress despite remaining scan budget.Quoted 43 lines
mm/vmscan.c +11
-3@@ -5105,7 +5105,8 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, struct scan_control *sc, }static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq, -struct scan_control *sc,int swappiness) +struct scan_control *sc,int swappiness, +bool no_populatable) {DEFINE_MIN_SEQ (lruvec);@@ -5117,6 +5118,13 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (sc->priority ==DEF_PRIORITY )return false; +/* +* Run aging if scan budget remains but no folios +* in populatable generations are available to scan. +*/ +if (no_populatable) +return true; +/* better to run aging even though eviction is still possible */ if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq)return true;@@ -5221,20 +5229,17 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { +unsigned long orig_file, orig_anon;DEFINE_MAX_SEQ (lruvec); +bool no_populatable;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) { need_rotate = true;break ; } -if (should_run_aging (lruvec, max_seq, sc, swappiness)) { -trace_mm_mglru_aging (lruvec, sc, swappiness);mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5245
Quoted 10 lines
mm/vmscan.c +2
-5-if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) - need_rotate = true; - should_age = true; - } - + orig_anon = nr[LRU_GEN_ANON ]; + orig_file = nr[LRU_GEN_FILE ];if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ))break ;[Severity: High]
Does movingshould_run_aging()andtry_to_inc_max_seq()afterevict_folio_lists()skip aging wheneverevict_folio_lists()scans zero folios?mm/vmscan.c:scan_folios() { ... if (sc->priority == DEF_PRIORITY) seq = lrugen->max_seq - MIN_NR_GENS; else if (sc->priority > DEF_PRIORITY / 2) seq = lrugen->max_seq - 1; else seq = lrugen->max_seq; seq = lruvec_populated_min_seq(lruvec, type, sc->reclaim_idx + 1, seq); ... }At
DEF_PRIORITY, when onlyMIN_NR_GENSgenerations remain (evictable_min_seq(min_seq, swappiness)+MIN_NR_GENS>max_seq),scan_folios()caps seq tomax_seq-MIN_NR_GENS, which is belowmin_seqand already empty.evict_folio_lists()then returns 0 and breaks out of the loop before reachingshould_run_aging(), making the mandatoryevictable_min_seq(min_seq, swappiness)+MIN_NR_GENS>max_seqaging check unreachable atDEF_PRIORITY.Similarly, at
DEF_PRIORITY/ 2 <sc->priority<DEF_PRIORITY,scan_folios()caps seq tomax_seq- 1. Whenever all evictable folios for the budgeted type(s) are inmax_seq(for example when both types only have folios inmax_seq, or when only one type has budget left),evict_folio_lists()returns 0 and breaks before aging can run, leavingneed_rotatefalse as well soshrink_one()fails to rotate a sizable memcg.
Quoted 5 lines
Finally, when
sc->priority<=DEF_PRIORITY/ 2, runningevict_folio_lists()beforeshould_run_aging()scans directly frommax_seq(wheremin_gen==max_geninsort_folio()disables tier protection) before aging can split the youngest generation, and breaks without aging if the first batch satisfiesshould_abort_scan().
As I mentioned in my earlier reply [1], Bo Zhang suggested the following change to me off-list:
@@ -5233,6 +5231,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
unsigned long orig_file, orig_anon;
DEFINE_MAX_SEQ(lruvec);
bool no_populatable;+ long scanned;
if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) {
need_rotate = true;@@ -5241,8 +5240,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
orig_anon = nr[LRU_GEN_ANON];
orig_file = nr[LRU_GEN_FILE];- if (!evict_folio_lists(nr, lruvec, sc, swappiness,MIN_LRU_BATCH))
- break;
+ scanned = evict_folio_lists(nr, lruvec, sc,
swappiness, MIN_LRU_BATCH);
if (should_abort_scan(lruvec, sc))
break;
@@ -5263,6 +5261,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
should_age = true;
}+ if (!scanned)
+ break;
+
cond_resched();
}This gives aging a chance to run even whenevict_folio_lists() returns 0.
[1] https://lore.kernel.org/linux-mm/CAGsJ_4xMUoTsPU-=_QVDqZ-p4Dt+vGJC5G8hed3PWNAk3iv8Bw@mail.gmail.com/
Quoted 5 lines
Quoted 5 lines
mm/vmscan.c +0
-0@@ -5248,6 +5253,15 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ;
Quoted 7 lines
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5256
Quoted 2 lines
mm/vmscan.c +2
-0+ no_populatable = (nr[LRU_GEN_ANON ] && nr[LRU_GEN_ANON ] == orig_anon) || + (nr[LRU_GEN_FILE ] && nr[LRU_GEN_FILE ] == orig_file);[Severity: Medium]
Canno_populatablestay true across everyMIN_LRU_BATCHiteration and trigger repeated aging when one type still scans 0 folios after aging?
This is very unlikely. As long as while (!list_empty(head)) holds, we will have a non-zero scanned count:
while (!list_empty(head)) {
struct folio *folio = lru_to_folio(head);
int delta = folio_nr_pages(folio);
scanned += delta;Aging should also help move some folios into
populatable generations.
Quoted 7 lines
nr[] is computed once in
lru_gen_prepare_scan()and is only decremented by scanned folios. Ifscan_folios()continues to return 0 for one type while the other type makes progress (for instance whensc->priority<=DEF_PRIORITY/ 2 already scans up tomax_seq, whenmax_seqhas no eligible folios of that type, or whenwalk_mm()promotes all referenced folios into the new youngest generation), nr[type] remains equal toorig_anonororig_file. In cgroup reclaim (!root_reclaim(sc)), the loop does not break
Folios shouldn't be promoted repeatedly, as earlier aging should have already cleared their PTE young bits.
Quoted 16 lines
on
should_age, soshould_run_aging()returns true and callstry_to_inc_max_seq()on every 64-page batch of the other type; inroot_reclaim(sc), it runs unnecessary aging and aborts scanning the populated type after one extra batch.mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5262
Quoted 6 lines
mm/vmscan.c +6
-0+if (should_run_aging (lruvec, max_seq, sc, swappiness, no_populatable)) { +trace_mm_mglru_aging (lruvec, sc, swappiness); +if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) + need_rotate = true; + should_age = true; + }[Severity: Low]
Canshould_run_aging()see a stalemax_seqalongside a freshmin_seqhere?
Quoted 7 lines
DEFINE_MAX_SEQ(lruvec)snapshotsmax_seqat the start of the loop beforeevict_folio_lists(), whereasshould_run_aging()readsmin_seqafterevict_folio_lists()finishes. If a concurrent reclaimer advancesmax_seqandmin_seqduring eviction,should_run_aging()compares the newmin_seqagainst the oldmax_seq, which can falsely trigger the aging branch, failtry_to_inc_max_seq()on the stalemax_seq, and setshould_age= true soroot_reclaim(sc)aborts on the next pass.
I guess this is a pre-existing race that could also occur before this patchset, but the patch somehow widens the race window. One possible approach is to read max_seq later, just before calling should_run_aging().
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Barry Song (Xiaomi)
reviewerPATCH v7 12/16 mm/mglru: Restrict eviction accounting to eligible zones
In reply to Barry Song (Xiaomi)For both lruvec_evictable_size() and lru_gen_imbalanced(), only count eligible zones below sc->reclaim_idx. This avoids inflating the number of scanned pages and allows aging to proceed more promptly when eligible zones have few folios.
--- mm/trace_vmscan.h | 4 ++-- mm/vmscan.c | 18 +++++++++--------- mm/vmscan.h | 5 +++-- 3 files changed, 14 insertions(+), 13 deletions(-)
mm/trace_vmscan.h +2 -2
@@ -488,9 +488,9 @@ TRACE_EVENT(mm_mglru_aging,unsigned long seq = max_seq -MAX_NR_GENS +1 + i; __entry->nr_anon[i] = -lruvec_gen_size (lrugen,LRU_GEN_ANON , seq); +lruvec_gen_size (lrugen,LRU_GEN_ANON , seq,MAX_NR_ZONES ); __entry->nr_file[i] = -lruvec_gen_size (lrugen,LRU_GEN_FILE , seq); +lruvec_gen_size (lrugen,LRU_GEN_FILE , seq,MAX_NR_ZONES ); } __entry->nid =lruvec_pgdat (lruvec)->node_id; __entry->priority = sc->priority;
mm/vmscan.c +9 -9
@@ -4206,8 +4206,8 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, -unsigned long nr[]) +static void lruvec_evictable_size (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long nr[]) {int type;unsigned long seq;
@@ -4220,7 +4220,7 @@ static void lruvec_evictable_size(struct lruvec *lruvec, int swappiness,for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - nr[type] +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); } }
@@ -4229,7 +4229,7 @@ static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); -lruvec_evictable_size (lruvec, swappiness, sizes); +lruvec_evictable_size (lruvec, sc, swappiness, sizes); total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */
@@ -5083,9 +5083,9 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, struct scan_control *sc,for_each_evictable_type (type, swappiness) {unsigned long seq; - youngest[type] =lruvec_gen_size (lrugen, type, max_seq); + youngest[type] =lruvec_gen_size (lrugen, type, max_seq, sc->reclaim_idx +1 );for (seq = min_seq[type]; seq < max_seq; seq++) - old[type] +=lruvec_gen_size (lrugen, type, seq); + old[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); }/* the vast majority of pages sit in the youngest gen */
@@ -5138,7 +5138,7 @@ static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc, {unsigned long total; -lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; total =apply_proportional_protection (memcg, sc, total);
@@ -5836,7 +5836,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)char mark = full && seq < min_seq[type] ?'x' :' ' ;seq_printf (m," %10lu%c" , -lruvec_gen_size (lrugen, type, seq), mark); +lruvec_gen_size (lrugen, type, seq,MAX_NR_ZONES ), mark); }seq_putc (m,'\n' );
@@ -5889,7 +5889,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coif (sc->nr_reclaimed >= nr_to_reclaim)return 0 ; -lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan);lru_gen_balance_scan (lruvec, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);
mm/vmscan.h +3 -2
@@ -119,12 +119,13 @@ struct scan_control {#ifdef CONFIG_LRU_GEN /* The number of pages of @type in the generation @seq, summed over all zones */ static inline unsigned long lruvec_gen_size (struct lru_gen_folio *lrugen, -int type,unsigned long seq) +int type,unsigned long seq, +int max_zone) {int gen =lru_gen_from_seq (seq);unsigned long size =0 ; -for (int zone =0 ; zone <MAX_NR_ZONES ; zone++) +for (int zone =0 ; zone < max_zone; zone++) size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L );return size;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 12/16] mm/mglru: Restrict eviction accounting to eligible zones
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm/mglru: premature OOM kill in lru_gen_age_node() on zone-constrained reclaim --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: premature OOM kill in lru_gen_age_node() on zone-constrained reclaim
--
commit 169c51ce627ac19052dda882e181615f7e7c984d
Author: Barry Song (Xiaomi) <[email protected]>
mm/mglru: Restrict eviction accounting to eligible zones
This commit restricts lruvec_evictable_size() and lru_gen_imbalanced() to count only zones at or below sc->reclaim_idx by adding a max_zone parameter to lruvec_gen_size().
Quoted 25 lines
mm/vmscan.c +3
-3@@ -4207,8 +4207,8 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, -unsigned long nr[]) +static void lruvec_evictable_size (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long nr[]) {int type;unsigned long seq;@@ -4221,7 +4221,7 @@ static void lruvec_evictable_size(struct lruvec *lruvec, int swappiness,for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - nr[type] +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); } }
mm/vmscan.c static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) · line 4236
Quoted 8 lines
mm/vmscan.c +1
-1@@ -4230,7 +4230,7 @@ static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); -lruvec_evictable_size (lruvec, swappiness, sizes); +lruvec_evictable_size (lruvec, sc, swappiness, sizes); total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */
[Severity: High]
Can restricting lruvec_is_sizable() to sc->reclaim_idx trigger a premature global OOM kill in lru_gen_age_node() during zone-constrained reclaim when lru_gen_min_ttl is enabled?
In balance_pgdat(), kswapd sets sc.reclaim_idx = highest_zoneidx and runs kswapd_age_node() -> lru_gen_age_node() before kswapd_shrink_node():
mm/vmscan.c:lru_gen_age_node() {
...
do {
struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
mem_cgroup_calculate_protection(NULL, memcg);
if (!reclaimable)
reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
...
if (!reclaimable && mutex_trylock(&oom_lock)) {
struct oom_control oc = {
.gfp_mask = sc->gfp_mask,
};
out_of_memory(&oc);
mutex_unlock(&oom_lock);
}
}Because user LRU pages are allocated from the highest zones and kept out of lower zones by lowmem_reserve, a wakeup with a constrained highest_zoneidx (such as ZONE_DMA or ZONE_DMA32) can find fewer than 1UL << sc->priority evictable pages in zones 0..sc->reclaim_idx in every memcg. lruvec_is_sizable() and lruvec_is_reclaimable() then return false for all memcgs, causing lru_gen_age_node() to invoke out_of_memory() with GFP_KERNEL even when higher zones have plenty of reclaimable memory older than min_ttl.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Barry Song
reviewerRe: [PATCH v7 12/16] mm/mglru: Restrict eviction accounting to eligible zones
Quoted 80 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/mglru: premature OOM kill inlru_gen_age_node()on zone-constrained reclaim
--commit
169c51ce627ac19052dda882e181615f7e7c984d
Author: Barry Song (Xiaomi) <[email protected]>mm/mglru: Restrict eviction accounting to eligible zones
This commit restricts
lruvec_evictable_size()andlru_gen_imbalanced()to count only zones at or belowsc->reclaim_idxby adding amax_zoneparameter tolruvec_gen_size().Quoted 25 lines
mm/vmscan.c +3
-3@@ -4207,8 +4207,8 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, -unsigned long nr[]) +static void lruvec_evictable_size (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long nr[]) {int type;unsigned long seq;@@ -4221,7 +4221,7 @@ static void lruvec_evictable_size(struct lruvec *lruvec, int swappiness,for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - nr[type] +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); } }mm/vmscan.c static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) · line 4236
Quoted 8 lines
mm/vmscan.c +1
-1@@ -4230,7 +4230,7 @@ static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); -lruvec_evictable_size (lruvec, swappiness, sizes); +lruvec_evictable_size (lruvec, sc, swappiness, sizes); total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */ [Severity: High]
Can restrictinglruvec_is_sizable()tosc->reclaim_idxtrigger a premature global OOM kill inlru_gen_age_node()during zone-constrained reclaim whenlru_gen_min_ttlis enabled?In
balance_pgdat(), kswapd sets sc.reclaim_idx =highest_zoneidxand runskswapd_age_node()->lru_gen_age_node()beforekswapd_shrink_node():mm/vmscan.c:lru_gen_age_node() { ... do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); mem_cgroup_calculate_protection(NULL, memcg); if (!reclaimable) reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl); } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); ... if (!reclaimable && mutex_trylock(&oom_lock)) { struct oom_control oc = { .gfp_mask = sc->gfp_mask, }; out_of_memory(&oc); mutex_unlock(&oom_lock); } }
Quoted 8 lines
Because user LRU pages are allocated from the highest zones and kept out of lower zones by
lowmem_reserve, a wakeup with a constrainedhighest_zoneidx(such asZONE_DMAorZONE_DMA32) can find fewer than 1UL <<sc->priorityevictable pages in zones 0..sc->reclaim_idx in every memcg.lruvec_is_sizable()andlruvec_is_reclaimable()then return false for all memcgs, causinglru_gen_age_node()to invokeout_of_memory()withGFP_KERNELeven when higher zones have plenty of reclaimable memory older thanmin_ttl.
I think this is a pre-existing issue. lruvec_is_reclaimable() uses the current sc->priority to estimate the reclaimable size, which can significantly underestimate it in min_ttl cases. Our patch somehow amplifies the problem because it uses the correct reclaim_idx to calculate the reclaimable size.
static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc,
unsigned long min_ttl)
{
int gen;
unsigned long birth;
int swappiness = get_swappiness(lruvec, sc);
struct mem_cgroup *memcg = lruvec_memcg(lruvec);
DEFINE_MIN_SEQ(lruvec);
if (mem_cgroup_below_min(NULL, memcg))
return false;
if (!lruvec_is_sizable(lruvec, sc))
return false;
gen = lru_gen_from_seq(evictable_min_seq(min_seq, swappiness));
birth = READ_ONCE(lruvec->lrugen.timestamps[gen]);
return time_is_before_jiffies(birth + min_ttl);
}I think we have two options: either drop this check and make sure lruvec_is_reclaimable() focuses solely on min_ttl:
if (!lruvec_is_sizable(lruvec, sc))
return false;
Or we can do the following, using the lowest possible kswapd sc->priority to estimate the reclaimable size:
@@ -4245,7 +4243,7 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc
if (mem_cgroup_below_min(NULL, memcg))
return false;
- if (!lruvec_is_sizable(lruvec, sc))
+ if (sc->priority <= 1 && !lruvec_is_sizable(lruvec, sc))
return false;
gen = lru_gen_from_seq(evictable_min_seq(min_seq, swappiness));Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Barry Song (Xiaomi)
reviewerPATCH v7 13/16 mm/mglru: Reduce LRU lock contention by shrinking folio batches in inc_min_seq
In reply to Barry Song (Xiaomi)Reduce the number of folios moved from the oldest generation to the second-oldest generation in each batch. This shortens the time the lruvec lock is held, allowing concurrent LRU operations to make progress and significantly reducing LRU lock contention.
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -3911,7 +3911,7 @@ static inline void flush_lru_batch(struct list_head *head, struct list_head **bastatic bool inc_min_seq (struct lruvec *lruvec,int type,int swappiness) {int zone; -int remaining =MAX_LRU_BATCH ; +int remaining =MIN_LRU_BATCH ;struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]);int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]);
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerPATCH v7 14/16 mm/mglru: Avoid prolonged anon reclaim with swappiness=200
In reply to Barry Song (Xiaomi)From: lyugaofei <[email protected]>
It has been observed that swappiness=200 can significantly slow down reclaim on some machines, with some workloads running 3x slower under MGLRU than under CLRU.
This is because swappiness=200 currently results in file_factor=0. Give file reclaim a small factor at lower priorities, for example, when sc->priority < DEF_PRIORITY / 2.
--- mm/vmscan.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
mm/vmscan.c +10 -5
@@ -4888,8 +4888,8 @@ static int get_tier_idx(struct lruvec *lruvec, int type)* We don't calculate the scan cost, MGLRU uses lazy promotion, which will * cancel the scan budget natually. */ -static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, -unsigned long total_scan,unsigned long nr[]) +static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long total_scan,unsigned long nr[]) {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor;
@@ -4946,7 +4946,12 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, int swappiness, file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; - file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; +if (swappiness ==MAX_SWAPPINESS ) + file_factor = sc->priority <DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_ANON ] * file_weight :0 ; +else + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * + refault_rate[LRU_GEN_ANON ] * file_weight;/* Cross-multiply the ratio above. */ anon_scan =mul_u64_u64_div_u64 (total_scan, anon_factor, anon_factor + file_factor);
@@ -5143,7 +5148,7 @@ static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc, total =apply_proportional_protection (memcg, sc, total); total >>= sc->priority; -lru_gen_balance_scan (lruvec, swappiness, total, nr_to_scan); +lru_gen_balance_scan (lruvec, sc, swappiness, total, nr_to_scan); }static bool should_abort_scan (struct lruvec *lruvec,struct scan_control *sc)
@@ -5890,7 +5895,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coreturn 0 ;lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); -lru_gen_balance_scan (lruvec, swappiness, +lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 14/16] mm/mglru: Avoid prolonged anon reclaim with swappiness=200
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] mm/mglru: commit Author and Signed-off-by use handle instead of real name - [Medium] mm/mglru: zero…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] mm/mglru: commit Author and Signed-off-by use handle instead of real name
- [Medium] mm/mglru: zero sc.priority in run_eviction() skews lru_gen_balance_scan()
--
commit 8a29a76d461ffd04769caa1f7ed98ca18efe1fb3
Author: lyugaofei <[email protected]>
Quoted 6 lines
mm/mglru: Avoid prolonged anon reclaim with swappiness=200
It has been observed that swappiness=200 can significantly slow down reclaim on some machines, with some workloads running 3x slower under MGLRU than under CLRU.
Quoted 7 lines
This is because swappiness=200 currently results in file_factor=0. Give file reclaim a small factor at lower priorities, for example, when
sc->priority<DEF_PRIORITY/ 2.Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>
[Severity: High]
Per Documentation/process/submitting-patches.rst, both the commit Author and the Signed-off-by trailer must use a known real full human name rather than an account handle ("lyugaofei").
Quoted 26 lines
mm/vmscan.c +8
-3@@ -4889,8 +4889,8 @@ static int get_tier_idx(struct lruvec *lruvec, int type)* We don't calculate the scan cost, MGLRU uses lazy promotion, which will * cancel the scan budget natually. */ -static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, -unsigned long total_scan,unsigned long nr[]) +static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long total_scan,unsigned long nr[]) {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor;@@ -4947,7 +4947,12 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, int swappiness, file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; - file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; +if (swappiness ==MAX_SWAPPINESS ) + file_factor = sc->priority <DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_ANON ] * file_weight :0 ; +else + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * + refault_rate[LRU_GEN_ANON ] * file_weight;
[ ... ]
mm/vmscan.c static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · line 5902
Quoted 9 lines
mm/vmscan.c +1
-1@@ -5891,7 +5896,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coreturn 0 ;lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); -lru_gen_balance_scan (lruvec, swappiness, +lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))
[Severity: Medium]
Does passing sc from run_eviction() into lru_gen_balance_scan() cause swappiness=200 via debugfs proactive reclaim to always calculate a non-zero file_factor?
lru_gen_seq_write() initializes sc on the stack without setting .priority to DEF_PRIORITY, leaving sc.priority zero-initialized when passed to run_cmd() -> run_eviction(). In lru_gen_balance_scan(), sc->priority < DEF_PRIORITY / 2 (0 < 6) is then unconditionally true, so proactive eviction with swappiness=200 always reclaims file pages instead of preferring anonymous memory.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Barry Song
reviewerRe: [PATCH v7 14/16] mm/mglru: Avoid prolonged anon reclaim with swappiness=200
Quoted 22 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] mm/mglru: commit Author and Signed-off-by use handle instead of real name
- [Medium] mm/mglru: zero sc.priority inrun_eviction()skewslru_gen_balance_scan()
--commit
8a29a76d461ffd04769caa1f7ed98ca18efe1fb3
Author: lyugaofei <[email protected]>Quoted 10 lines
mm/mglru: Avoid prolonged anon reclaim with swappiness=200
It has been observed that swappiness=200 can significantly slow down reclaim on some machines, with some workloads running 3x slower under MGLRU than under CLRU.
This is because swappiness=200 currently results in file_factor=0. Give file reclaim a small factor at lower priorities, for example, when
sc->priority<DEF_PRIORITY/ 2.
Quoted 8 lines
Quoted 3 lines
Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>[Severity: High]
PerDocumentation/process/submitting-patches.rst, both the commit Author and the Signed-off-by trailer must use a known real full human name rather than an account handle ("lyugaofei").
Let me change it to:
Gaofei Lyu <[email protected]>
Quoted 45 lines
Quoted 27 lines
mm/vmscan.c +8
-3@@ -4889,8 +4889,8 @@ static int get_tier_idx(struct lruvec *lruvec, int type)* We don't calculate the scan cost, MGLRU uses lazy promotion, which will * cancel the scan budget natually. */ -static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, -unsigned long total_scan,unsigned long nr[]) +static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long total_scan,unsigned long nr[]) {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor;@@ -4947,7 +4947,12 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, int swappiness, file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; - file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; +if (swappiness ==MAX_SWAPPINESS ) + file_factor = sc->priority <DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_ANON ] * file_weight :0 ; +else + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * + refault_rate[LRU_GEN_ANON ] * file_weight;[ ... ]
mm/vmscan.c static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · line 5902
Quoted 9 lines
mm/vmscan.c +1
-1@@ -5891,7 +5896,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coreturn 0 ;lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); -lru_gen_balance_scan (lruvec, swappiness, +lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))[Severity: Medium]
Does passing sc fromrun_eviction()intolru_gen_balance_scan()cause swappiness=200 via debugfs proactive reclaim to always calculate a non-zerofile_factor?
Quoted 6 lines
lru_gen_seq_write()initializes sc on the stack without setting .priority toDEF_PRIORITY, leaving sc.priority zero-initialized when passed torun_cmd()->run_eviction(). Inlru_gen_balance_scan(),sc->priority<DEF_PRIORITY/ 2 (0 < 6) is then unconditionally true, so proactive eviction with swappiness=200 always reclaims file pages instead of preferring anonymous memory.
Well, I'm getting tons of complaints about lru_gen_seq_write(), even though it's just a debug interface that probably very few people actually use.
If anyone is actually using it, please let me know. Otherwise, I think we should consider removing it in the long term. Both memcg and node already have proactive reclaim interfaces, so this one seems redundant and difficult to maintain.
All Sashiko comments seem to suggest:
@@ -5955,6 +5956,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src,
struct blk_plug plug;
int err = -EINVAL;
struct scan_control sc = {+ .priority = DEF_PRIORITY,
.may_writepage = true,
.may_unmap = true,
.may_swap = true,
But I think this approach could introduce its own problems. Suppose we have four generations for both file and anon, and users use the debugfs interface to reclaim the oldest generation.
1. Even at DEF_PRIORITY, we can currently scan up to lrugen->max_seq - MIN_NR_GENS, which means the second-oldest generation can also be reclaimed. This somehow violates the user's request to reclaim only the oldest generation.
2. With swappiness=1 or 200, if we run out of the preferred file or anon type in the oldest generation, we can fall back to the other type and reclaim all of its oldest generation, as we did before scan-budget mode was introduced. With the new approach, however, we stick to the preferred type at DEF_PRIORITY and don't fall back to the other type. As a result, evict_folio_lists() can return 0 earlier than it would with the fallback approach.
I guess the only way to address all these issues is to add a separate reclaim path for requests from lru_gen_seq_write(), with evict_folio_lists() handling these requests differently from regular reclaim to preserve the old behavior before this patchset introduced the budget-based model.
Something like the following, though it would be very ugly:
mm/vmscan.c +0 -0
@@ -4745,7 +4745,8 @@ static bool isolate_folio(struct lruvec *lruvec,
struct folio *folio, struct sca
static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
struct scan_control *sc, int type, int tier, struct list_head *list, int *isolatedp,
- unsigned long *scanned_seq)
+ unsigned long *scanned_seq,
+ bool cmd_proactive)
{
enum node_stat_item item;
int zone_idx;@@ -4768,7 +4769,9 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
* newest generation when under severe pressure, so hot folios are
* not evicted as soon as priority is raised.
*/
- if (sc->priority == DEF_PRIORITY)
+ if (unlikely(cmd_proactive))
+ seq = lrugen->min_seq[type];
+ else if (sc->priority == DEF_PRIORITY)
seq = lrugen->max_seq - MIN_NR_GENS;
else if (sc->priority > DEF_PRIORITY / 2)
seq = lrugen->max_seq - 1;@@ -4966,7 +4969,8 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc,
}
static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
- struct scan_control *sc, int type, int swappiness)
+ struct scan_control *sc, int type, int swappiness,
+ bool cmd_proactive)
{
LIST_HEAD(list);
LIST_HEAD(clean);@@ -4990,7 +4994,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
tier = get_tier_idx(lruvec, type);
scanned = scan_folios(nr_to_scan, lruvec, sc,
- type, tier, &list, &isolated, &scanned_seq);
+ type, tier, &list, &isolated,&scanned_seq, cmd_proactive);
nr_isolated = isolated;
if (nr_isolated)
__mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
@@ -5177,11 +5181,13 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)
}
static long evict_folio_lists(unsigned long nr_to_scan[], struct
lruvec *lruvec,
- struct scan_control *sc, int swappiness,
unsigned long max_batch)
+ struct scan_control *sc, int swappiness,
unsigned long max_batch,
+ bool cmd_proactive)
{
int type;
unsigned long batch, scanned, total_scanned = 0;+retry:
/*
* Scan both types with the given budgets; the caller checks if
* reclaim is satisfied after each pass.
@@ -5191,8 +5197,13 @@ static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec,
continue;
batch = min(nr_to_scan[type], max_batch);- scanned = evict_folios(batch, lruvec, sc, type, swappiness);
+ scanned = evict_folios(batch, lruvec, sc, type,
swappiness, cmd_proactive);
+ /* if it is a debugfs proactive reclaim, lend budget
to the other type */
+ if (!scanned && cmd_proactive && swappiness &&
swappiness <= MAX_SWAPPINESS) {
+ if (!scanned && cmd_proactive && swappiness &&
swappiness <= MAX_SWAPPINESS) {
+ nr_to_scan[!type] += nr_to_scan[type];
+ goto retry;
+ }
nr_to_scan[type] -= min(nr_to_scan[type], scanned);
total_scanned += scanned;
}@@ -5240,7 +5251,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
orig_anon = nr[LRU_GEN_ANON];
orig_file = nr[LRU_GEN_FILE];- scanned = evict_folio_lists(nr, lruvec, sc,
swappiness, MIN_LRU_BATCH);
+ scanned = evict_folio_lists(nr, lruvec, sc,
swappiness, MIN_LRU_BATCH, false);
if (should_abort_scan(lruvec, sc))
break;
@@ -5896,7 +5907,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co
lru_gen_balance_scan(lruvec, sc, swappiness,
nr_to_reclaim - sc->nr_reclaimed,nr_to_scan);
- if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness, MAX_LRU_BATCH))
+ if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness, MAX_LRU_BATCH, true))
return 0;
cond_resched();However, this would make the code much more complex and probably isn't worth the effort for a rarely used debugfs interface, especially given that we already have proactive reclaim interfaces for both memcg and nodes.
So perhaps we should simply set .priority = DEF_PRIORITY and live with the two less significant issues described above.
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Re: [PATCH v7 14/16] mm/mglru: Avoid prolonged anon reclaim with swappiness=200
Quoted 92 lines · Barry Song
Quoted 22 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] mm/mglru: commit Author and Signed-off-by use handle instead of real name
- [Medium] mm/mglru: zero sc.priority inrun_eviction()skewslru_gen_balance_scan()
--commit
8a29a76d461ffd04769caa1f7ed98ca18efe1fb3
Author: lyugaofei <[email protected]>Quoted 10 lines
mm/mglru: Avoid prolonged anon reclaim with swappiness=200
It has been observed that swappiness=200 can significantly slow down reclaim on some machines, with some workloads running 3x slower under MGLRU than under CLRU.
This is because swappiness=200 currently results in file_factor=0. Give file reclaim a small factor at lower priorities, for example, when
sc->priority<DEF_PRIORITY/ 2.Quoted 8 lines
Quoted 3 lines
Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>[Severity: High]
PerDocumentation/process/submitting-patches.rst, both the commit Author and the Signed-off-by trailer must use a known real full human name rather than an account handle ("lyugaofei").Let me change it to:
Gaofei Lyu <[email protected]>Quoted 45 lines
Quoted 27 lines
mm/vmscan.c +8
-3@@ -4889,8 +4889,8 @@ static int get_tier_idx(struct lruvec *lruvec, int type)* We don't calculate the scan cost, MGLRU uses lazy promotion, which will * cancel the scan budget natually. */ -static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, -unsigned long total_scan,unsigned long nr[]) +static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long total_scan,unsigned long nr[]) {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor;@@ -4947,7 +4947,12 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, int swappiness, file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; - file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; +if (swappiness ==MAX_SWAPPINESS ) + file_factor = sc->priority <DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_ANON ] * file_weight :0 ; +else + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * + refault_rate[LRU_GEN_ANON ] * file_weight;[ ... ]
mm/vmscan.c static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · line 5902
Quoted 9 lines
mm/vmscan.c +1
-1@@ -5891,7 +5896,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coreturn 0 ;lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); -lru_gen_balance_scan (lruvec, swappiness, +lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))[Severity: Medium]
Does passing sc fromrun_eviction()intolru_gen_balance_scan()cause swappiness=200 via debugfs proactive reclaim to always calculate a non-zerofile_factor?Quoted 6 lines
lru_gen_seq_write()initializes sc on the stack without setting .priority toDEF_PRIORITY, leaving sc.priority zero-initialized when passed torun_cmd()->run_eviction(). Inlru_gen_balance_scan(),sc->priority<DEF_PRIORITY/ 2 (0 < 6) is then unconditionally true, so proactive eviction with swappiness=200 always reclaims file pages instead of preferring anonymous memory.Well, I'm getting tons of complaints about
lru_gen_seq_write(), even though it's just a debug interface that probably very few people actually use.
Quoted 5 lines
If anyone is actually using it, please let me know. Otherwise, I think we should consider removing it in the long term. Both memcg and node already have proactive reclaim interfaces, so this one seems redundant and difficult to maintain.
For now I think we just be really lenient about that interface, e.g.g keep it at DEF_PRIORITY like in previous version
Quoted 157 lines
All Sashiko comments seem to suggest:
(quoted diff) +1
-0@@ -5955,6 +5956,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, struct blk_plug plug; int err = -EINVAL; struct scan_control sc = { + .priority = DEF_PRIORITY, .may_writepage = true, .may_unmap = true,.may_swap = true,
But I think this approach could introduce its own problems. Suppose we have four generations for both file and anon, and users use the debugfs interface to reclaim the oldest generation.
1. Even at
DEF_PRIORITY, we can currently scan up tolrugen->max_seq-MIN_NR_GENS, which means the second-oldest generation can also be reclaimed. This somehow violates the user's request to reclaim only the oldest generation.2. With swappiness=1 or 200, if we run out of the preferred file or anon type in the oldest generation, we can fall back to the other type and reclaim all of its oldest generation, as we did before scan-budget mode was introduced. With the new approach, however, we stick to the preferred type at
DEF_PRIORITYand don't fall back to the other type. As a result,evict_folio_lists()can return 0 earlier than it would with the fallback approach.I guess the only way to address all these issues is to add a separate reclaim path for requests from
lru_gen_seq_write(), withevict_folio_lists()handling these requests differently from regular reclaim to preserve the old behavior before this patchset introduced the budget-based model.Something like the following, though it would be very ugly:
mm/vmscan.c +0
-0@@ -4745,7 +4745,8 @@ static bool isolate_folio(struct lruvec *lruvec,struct folio *folio, struct sca
static int scan_folios(unsigned longnr_to_scan, struct lruvec *lruvec,
structscan_control*sc, int type, int tier, structlist_head*list, int *isolatedp,mm/vmscan.c +5
-2- unsigned long *scanned_seq) + unsigned long *scanned_seq, + bool cmd_proactive) {enum node_stat_item item;int zone_idx;@@ -4768,7 +4769,9 @@ static int scan_folios(unsigned long nr_to_scan,struct lruvec *lruvec, * newest generation when under severe pressure, so hot folios are * not evicted as soon as priority is raised. */ -if (sc->priority ==DEF_PRIORITY ) +if (unlikely (cmd_proactive)) + seq = lrugen->min_seq[type]; +else if (sc->priority ==DEF_PRIORITY ) seq = lrugen->max_seq -MIN_NR_GENS ;else if (sc->priority >DEF_PRIORITY /2 )seq = lrugen->max_seq - 1;mm/vmscan.c +2
-1@@ -4966,7 +4969,8 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, }static int evict_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int type,int swappiness) +struct scan_control *sc,int type,int swappiness, +bool cmd_proactive) {LIST_HEAD (list);LIST_HEAD(clean);mm/vmscan.c +1
-1@@ -4990,7 +4994,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, &list, &isolated, &scanned_seq); + type, tier, &list, &isolated, &scanned_seq, cmd_proactive); nr_isolated = isolated;if (
nr_isolated)
__mod_node_page_state(pgdat,NR_ISOLATED_ANON+ type,mm/vmscan.c +5
-2@@ -5177,11 +5181,13 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) } static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec, -struct scan_control *sc,int swappiness, unsigned long max_batch) + struct scan_control *sc, int swappiness, unsigned long max_batch, + bool cmd_proactive) { int type;unsigned long batch, scanned, total_scanned =0 ; +retry:/* * Scan both types with the given budgets; the caller checks if * reclaim is satisfied after each pass. @@ -5191,8 +5197,13 @@ static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec, continue; batch =min (nr_to_scan[type], max_batch); - scanned =evict_folios (batch, lruvec, sc, type, swappiness); + scanned =evict_folios (batch, lruvec, sc, type, swappiness, cmd_proactive); +/* if it is a debugfs proactive reclaim, lend budget to the other type */
+ if (!scanned &&cmd_proactive&& swappiness &&
swappiness <=MAX_SWAPPINESS) {
+ if (!scanned &&cmd_proactive&& swappiness &&
swappiness <=MAX_SWAPPINESS) {mm/vmscan.c +4
-1+ nr_to_scan[!type] += nr_to_scan[type]; + goto retry; + } nr_to_scan[type] -=min (nr_to_scan[type], scanned); total_scanned += scanned; }@@ -5240,7 +5251,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) orig_anon = nr[LRU_GEN_ANON ]; orig_file = nr[LRU_GEN_FILE ]; - scanned =evict_folio_lists (nr, lruvec, sc, swappiness, MIN_LRU_BATCH); + scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH , false);if (
should_abort_scan(lruvec, sc))
break;mm/vmscan.c +0
-0@@ -5896,7 +5907,7 @@ static int run_eviction(struct lruvec *lruvec,unsigned long seq, struct
scan_co
lru_gen_balance_scan(lruvec, sc, swappiness,
nr_to_reclaim-sc->nr_reclaimed,nr_to_scan);- if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness,MAX_LRU_BATCH))
+ if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness,MAX_LRU_BATCH, true))return 0; cond_resched();
Quoted 8 lines
However, this would make the code much more complex and probably isn't worth the effort for a rarely used debugfs interface, especially given that we already have proactive reclaim interfaces for both memcg and nodes.
So perhaps we should simply set .priority =
DEF_PRIORITYand live with the two less significant issues described above.
I agree with this, actually maybe later we can use something in scan_control like an extra field, CLRU has a lot of special tweaks in that, which also needs to be sorted.
Barry Song
reviewerRe: [PATCH v7 14/16] mm/mglru: Avoid prolonged anon reclaim with swappiness=200
On Thu, Nov 12, 2026 at 6:46 PM Kairui Song <[email protected]> wrote:
[...]
Quoted 197 lines
Quoted 26 lines
Quoted 15 lines
Review static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · line 5902
Quoted 9 lines
(quoted diff) +1
-1@@ -5891,7 +5896,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co return 0; lruvec_evictable_size(lruvec, sc, swappiness, nr_to_scan); - lru_gen_balance_scan(lruvec, swappiness, + lru_gen_balance_scan(lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); if (!evict_folio_lists(nr_to_scan, lruvec, sc, swappiness, MAX_LRU_BATCH))[Severity: Medium]
Does passing sc fromrun_eviction()intolru_gen_balance_scan()cause swappiness=200 via debugfs proactive reclaim to always calculate a non-zerofile_factor?Quoted 6 lines
lru_gen_seq_write()initializes sc on the stack without setting .priority toDEF_PRIORITY, leaving sc.priority zero-initialized when passed torun_cmd()->run_eviction(). Inlru_gen_balance_scan(),sc->priority<DEF_PRIORITY/ 2 (0 < 6) is then unconditionally true, so proactive eviction with swappiness=200 always reclaims file pages instead of preferring anonymous memory.Well, I'm getting tons of complaints about
lru_gen_seq_write(), even though it's just a debug interface that probably very few people actually use.Quoted 5 lines
If anyone is actually using it, please let me know. Otherwise, I think we should consider removing it in the long term. Both memcg and node already have proactive reclaim interfaces, so this one seems redundant and difficult to maintain.
For now I think we just be really lenient about that interface, e.g.g keep it at
DEF_PRIORITYlike in previous versionQuoted 162 lines
All Sashiko comments seem to suggest:
(quoted diff) +1
-0@@ -5955,6 +5956,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, struct blk_plug plug; int err = -EINVAL; struct scan_control sc = { + .priority = DEF_PRIORITY, .may_writepage = true, .may_unmap = true,.may_swap = true,
But I think this approach could introduce its own problems. Suppose we have four generations for both file and anon, and users use the debugfs interface to reclaim the oldest generation.
1. Even at
DEF_PRIORITY, we can currently scan up tolrugen->max_seq-MIN_NR_GENS, which means the second-oldest generation can also be reclaimed. This somehow violates the user's request to reclaim only the oldest generation.2. With swappiness=1 or 200, if we run out of the preferred file or anon type in the oldest generation, we can fall back to the other type and reclaim all of its oldest generation, as we did before scan-budget mode was introduced. With the new approach, however, we stick to the preferred type at
DEF_PRIORITYand don't fall back to the other type. As a result,evict_folio_lists()can return 0 earlier than it would with the fallback approach.I guess the only way to address all these issues is to add a separate reclaim path for requests from
lru_gen_seq_write(), withevict_folio_lists()handling these requests differently from regular reclaim to preserve the old behavior before this patchset introduced the budget-based model.Something like the following, though it would be very ugly:
mm/vmscan.c +0
-0@@ -4745,7 +4745,8 @@ static bool isolate_folio(struct lruvec *lruvec,struct folio *folio, struct sca
static int scan_folios(unsigned longnr_to_scan, struct lruvec *lruvec,
structscan_control*sc, int type, int tier, structlist_head*list, int *isolatedp,mm/vmscan.c +5
-2- unsigned long *scanned_seq) + unsigned long *scanned_seq, + bool cmd_proactive) {enum node_stat_item item;int zone_idx;@@ -4768,7 +4769,9 @@ static int scan_folios(unsigned long nr_to_scan,struct lruvec *lruvec, * newest generation when under severe pressure, so hot folios are * not evicted as soon as priority is raised. */ -if (sc->priority ==DEF_PRIORITY ) +if (unlikely (cmd_proactive)) + seq = lrugen->min_seq[type]; +else if (sc->priority ==DEF_PRIORITY ) seq = lrugen->max_seq -MIN_NR_GENS ;else if (sc->priority >DEF_PRIORITY /2 )seq = lrugen->max_seq - 1;mm/vmscan.c +2
-1@@ -4966,7 +4969,8 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, }static int evict_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int type,int swappiness) +struct scan_control *sc,int type,int swappiness, +bool cmd_proactive) {LIST_HEAD (list);LIST_HEAD(clean);mm/vmscan.c +1
-1@@ -4990,7 +4994,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, &list, &isolated, &scanned_seq); + type, tier, &list, &isolated, &scanned_seq, cmd_proactive); nr_isolated = isolated;if (
nr_isolated)
__mod_node_page_state(pgdat,NR_ISOLATED_ANON+ type,mm/vmscan.c +5
-2@@ -5177,11 +5181,13 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) } static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec, -struct scan_control *sc,int swappiness, unsigned long max_batch) + struct scan_control *sc, int swappiness, unsigned long max_batch, + bool cmd_proactive) { int type;unsigned long batch, scanned, total_scanned =0 ; +retry:/* * Scan both types with the given budgets; the caller checks if * reclaim is satisfied after each pass. @@ -5191,8 +5197,13 @@ static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec, continue; batch =min (nr_to_scan[type], max_batch); - scanned =evict_folios (batch, lruvec, sc, type, swappiness); + scanned =evict_folios (batch, lruvec, sc, type, swappiness, cmd_proactive); +/* if it is a debugfs proactive reclaim, lend budget to the other type */
+ if (!scanned &&cmd_proactive&& swappiness &&
swappiness <=MAX_SWAPPINESS) {
+ if (!scanned &&cmd_proactive&& swappiness &&
swappiness <=MAX_SWAPPINESS) {mm/vmscan.c +4
-1+ nr_to_scan[!type] += nr_to_scan[type]; + goto retry; + } nr_to_scan[type] -=min (nr_to_scan[type], scanned); total_scanned += scanned; }@@ -5240,7 +5251,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) orig_anon = nr[LRU_GEN_ANON ]; orig_file = nr[LRU_GEN_FILE ]; - scanned =evict_folio_lists (nr, lruvec, sc, swappiness, MIN_LRU_BATCH); + scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH , false);if (
should_abort_scan(lruvec, sc))
break;mm/vmscan.c +0
-0@@ -5896,7 +5907,7 @@ static int run_eviction(struct lruvec *lruvec,unsigned long seq, struct
scan_co
lru_gen_balance_scan(lruvec, sc, swappiness,
nr_to_reclaim-sc->nr_reclaimed,nr_to_scan);- if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness,MAX_LRU_BATCH))
+ if (!evict_folio_lists(nr_to_scan, lruvec, sc,
swappiness,MAX_LRU_BATCH, true))return 0; cond_resched();However, this would make the code much more complex and probably isn't worth the effort for a rarely used debugfs interface, especially given that we already have proactive reclaim interfaces for both memcg and nodes.
Quoted 7 lines
Quoted 3 lines
So perhaps we should simply set .priority =
DEF_PRIORITYand live with the two less significant issues described above.I agree with this, actually maybe later we can use something in
scan_controllike an extra field, CLRU has a lot of special tweaks in that, which also needs to be sorted.
Agreed. I'll add a few words to the changelog.
"Note that we are already lenient with the debugfs proactive reclaim interface. We may reclaim one generation ahead even at DEF_PRIORITY, and with extreme swappiness values such as 1 and 200, we may not fall back to the other type.
To address these cases and preserve the behavior of the previous model before scan-budget-based reclaim, we would need to add another flag to scan_control. However, this may not be worthwhile for a debugfs interface that is likely to be rarely used, especially given that we already have memory-level and node-level proactive reclaim interfaces."
Thanks,
Barry
Barry Song (Xiaomi)
reviewerPATCH v7 15/16 mm/mglru: Make swappiness=1 file reclaim more aggressive
In reply to Barry Song (Xiaomi)In MGLRU, swappiness=200 favors anon reclaim, while swappiness=1 favors file reclaim. Swappiness=0 allows only file reclaim, while swappiness=201 allows only anon reclaim. Thus, swappiness=1 and swappiness=200 are counterparts for file and anon reclaim, respectively.
Before this patchset, get_type_to_scan() always returned LRU_GEN_ANON for swappiness=200 and LRU_GEN_FILE for swappiness=1.
For swappiness=200, file_factor is set to 0 at higher reclaim priorities. Similarly, anon_factor is set to 0 for swappiness=1 at higher reclaim priorities, making the two settings behave similarly, although the thresholds are slightly different.
This allows file pages to be reclaimed more aggressively and benefits workloads that prioritize file reclaim by using swappiness=1.
--- mm/vmscan.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
mm/vmscan.c +5 -1
@@ -4945,7 +4945,11 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, anon_weight = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ] / size_bias; file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; - anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; +if (swappiness ==MIN_SWAPPINESS +1 ) + anon_factor = sc->priority <=DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_FILE ] * anon_weight :0 ; +else + anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight;if (swappiness ==MAX_SWAPPINESS ) file_factor = sc->priority <DEF_PRIORITY /2 ? refault_rate[LRU_GEN_ANON ] * file_weight :0 ;
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 15/16] mm/mglru: Make swappiness=1 file reclaim more aggressive
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/mglru: missing sc.priority init for lru_gen_balance_scan() via debugfs --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/mglru: missing sc.priority init for lru_gen_balance_scan() via debugfs
--
commit cf50aabb1ef5db31c58a09226fe4dbb55149d981
Author: Barry Song (Xiaomi) <[email protected]>
mm/mglru: Make swappiness=1 file reclaim more aggressive
This commit sets anon_factor to 0 in lru_gen_balance_scan() when swappiness is 1 and sc->priority is above DEF_PRIORITY / 2, making file reclaim more aggressive at normal reclaim priorities.
Quoted 5 lines
mm/vmscan.c +0
-0
mm/vmscan.c static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, · line 4953
Quoted 9 lines
mm/vmscan.c +5
-1@@ -4946,7 +4946,11 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, anon_weight = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ] / size_bias; file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; - anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; +if (swappiness ==MIN_SWAPPINESS +1 ) + anon_factor = sc->priority <=DEF_PRIORITY /2 ? + refault_rate[LRU_GEN_FILE ] * anon_weight :0 ; +else + anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight;
[Severity: Low]
Does the sc->priority <= DEF_PRIORITY / 2 check always evaluate to true when lru_gen_balance_scan() is called from the debugfs eviction path (lru_gen_seq_write() -> run_cmd() -> run_eviction())?
lru_gen_seq_write() allocates sc on the stack without initializing .priority, so sc.priority is 0 and anon_factor is never zeroed out for swappiness=1 during debugfs proactive eviction. Should lru_gen_seq_write() initialize .priority = DEF_PRIORITY in sc (as commit "mm/mglru: balance anon and file scan budget by refault rate" noted in its commit message)?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Barry Song (Xiaomi)
reviewerPATCH v7 16/16 mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO allocation
In reply to Barry Song (Xiaomi)From: Bo Zhang <[email protected]>
With GFP_NOIO, we cannot reclaim anon folios unless they are in the swap cache. CLRU skips anon reclaim when the swap cache is very low by commit 74d9aa0d8e98 ("mm: vmscan: avoid anon scanning for GFP_NOIO with low swapcache"). Apply the same logic to MGLRU.
The new scan-budget-based mechanism makes this straightforward to implement. Thanks also to Xiang Liu, who previously attempted to fix this before we had the scan-budget-based model [1].
--- mm/vmscan.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-)
mm/vmscan.c +10 -17
@@ -272,13 +272,6 @@ static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,if (!sc || (sc->gfp_mask & __GFP_IO))return false; -/* -* FIXME: MGLRU doesn't fully respect can_reclaim_anon_pages() for the -* scanning type, so only apply this to the traditional LRU for now. -*/ -if (lru_gen_enabled ()) -return false; -if (memcg) {struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);
@@ -4212,6 +4205,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *scint type;unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +int nid =lruvec_pgdat (lruvec)->node_id;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec);
@@ -4219,6 +4214,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc nr[LRU_GEN_FILE ] =0 ;for_each_evictable_type (type, swappiness) { +if (type ==LRU_GEN_ANON && !can_reclaim_anon_pages (memcg, nid, sc)) +continue ;for (seq = min_seq[type]; seq <= max_seq; seq++) nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); }
@@ -4893,23 +4890,19 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor; -u64 anon_scan, file_scan, total; +u64 anon_scan, file_scan;u64 anon_weight, file_weight;int size_bias; - total = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ]; -if (!total) -return ; - -if (swappiness <=MIN_SWAPPINESS ) { +if (swappiness <=MIN_SWAPPINESS ) nr[LRU_GEN_ANON ] =0 ; - nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]); -return ; - } -if (swappiness ==SWAPPINESS_ANON_ONLY ) { +if (swappiness ==SWAPPINESS_ANON_ONLY ) nr[LRU_GEN_FILE ] =0 ; + +if (!nr[LRU_GEN_ANON ] || !nr[LRU_GEN_FILE ]) { nr[LRU_GEN_ANON ] =min_t (u64, total_scan, nr[LRU_GEN_ANON ]); + nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]);return ; }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v7 16/16] mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO allocation
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm/mglru: stalled aging in try_to_shrink_lruvec() under GFP_NOIO --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm/mglru: stalled aging in try_to_shrink_lruvec() under GFP_NOIO
--
commit c2e07c75b5549aa295c630747e1cecd7625269f0
Author: Bo Zhang <[email protected]>
mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO allocation
This commit skips anonymous folio scanning in MGLRU when GFP_NOIO is set and swapcache is low by checking can_reclaim_anon_pages() in lruvec_evictable_size().
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 10 lines
mm/vmscan.c +2
-0@@ -4213,6 +4206,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *scint type;unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +int nid =lruvec_pgdat (lruvec)->node_id;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec);
mm/vmscan.c static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc · line 4222
Quoted 8 lines
mm/vmscan.c +2
-0@@ -4220,6 +4215,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc nr[LRU_GEN_FILE ] =0 ;for_each_evictable_type (type, swappiness) { +if (type ==LRU_GEN_ANON && !can_reclaim_anon_pages (memcg, nid, sc)) +continue ;for (seq = min_seq[type]; seq <= max_seq; seq++) nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); }
[Severity: Medium]
Can zeroing nr[LRU_GEN_ANON] here while swappiness remains non-zero stall MGLRU aging during GFP_NOIO reclaim?
When swap space is available and NUMA demotion is not, get_swappiness() stays in 1..MAX_SWAPPINESS even when can_reclaim_anon_pages() returns false. Because nr[LRU_GEN_ANON] is 0, evict_folio_lists() only scans file generations, so min_seq[LRU_GEN_ANON] stays at max_seq - 3 and try_to_inc_min_seq() clamps min_seq[LRU_GEN_FILE] to max_seq - MIN_NR_GENS:
mm/vmscan.c:try_to_inc_min_seq() {
...
if (swappiness && swappiness <= MAX_SWAPPINESS) {
unsigned long seq = lrugen->max_seq - MIN_NR_GENS;
if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
min_seq[LRU_GEN_ANON] = seq;
else if (min_seq[LRU_GEN_FILE] > seq && min_seq[LRU_GEN_ANON] < seq)
min_seq[LRU_GEN_FILE] = seq;
}
}Once populatable file generations are drained, evict_folio_lists() returns 0 and try_to_shrink_lruvec() breaks out of the loop before setting no_populatable, while should_run_aging() sees evictable_min_seq() at min_seq[LRU_GEN_ANON] (max_seq - 3) and lru_gen_imbalanced() still counts unreclaimable anon pages in old[LRU_GEN_ANON]. As a result, reclaim fails to age from DEF_PRIORITY down to DEF_PRIORITY / 2 + 1 and then directly evicts the youngest file generation (max_seq) without page-table aging or tier protection.
Should get_swappiness() return 0 when !can_reclaim_anon_pages() instead, so swappiness is 0 consistently across try_to_shrink_lruvec(), try_to_inc_min_seq(), should_run_aging(), and lru_gen_imbalanced()?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Barry Song
reviewerRe: [PATCH v7 16/16] mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO allocation
Quoted 62 lines · On Thu, Nov 12, 2026 at 1:01 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm/mglru: stalled aging intry_to_shrink_lruvec()underGFP_NOIO
--commit
c2e07c75b5549aa295c630747e1cecd7625269f0
Author: Bo Zhang <[email protected]>mm/mglru: Avoid scan anon while swapcache is low for
GFP_NOIOallocationThis commit skips anonymous folio scanning in MGLRU when
GFP_NOIOis set and swapcache is low by checkingcan_reclaim_anon_pages()inlruvec_evictable_size().Quoted 4 lines
mm/vmscan.c +0
-0[ ... ]
Quoted 10 lines
mm/vmscan.c +2
-0@@ -4213,6 +4206,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *scint type;unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +int nid =lruvec_pgdat (lruvec)->node_id;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec);mm/vmscan.c static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc · line 4222
Quoted 8 lines
mm/vmscan.c +2
-0@@ -4220,6 +4215,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc nr[LRU_GEN_FILE ] =0 ;for_each_evictable_type (type, swappiness) { +if (type ==LRU_GEN_ANON && !can_reclaim_anon_pages (memcg, nid, sc)) +continue ;for (seq = min_seq[type]; seq <= max_seq; seq++) nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); }[Severity: Medium]
Can zeroing nr[LRU_GEN_ANON] here while swappiness remains non-zero stall MGLRU aging duringGFP_NOIOreclaim?When swap space is available and NUMA demotion is not,
get_swappiness()stays in 1..MAX_SWAPPINESS even whencan_reclaim_anon_pages()returns false. Because nr[LRU_GEN_ANON] is 0,evict_folio_lists()only scans file generations, so min_seq[LRU_GEN_ANON] stays atmax_seq- 3 andtry_to_inc_min_seq()clamps min_seq[LRU_GEN_FILE] tomax_seq-MIN_NR_GENS:mm/vmscan.c:try_to_inc_min_seq() { ... if (swappiness && swappiness <= MAX_SWAPPINESS) { unsigned long seq = lrugen->max_seq - MIN_NR_GENS; if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq) min_seq[LRU_GEN_ANON] = seq; else if (min_seq[LRU_GEN_FILE] > seq && min_seq[LRU_GEN_ANON] < seq) min_seq[LRU_GEN_FILE] = seq; } }
Quoted 8 lines
Once populatable file generations are drained,
evict_folio_lists()returns 0 andtry_to_shrink_lruvec()breaks out of the loop before settingno_populatable, whileshould_run_aging()seesevictable_min_seq()at min_seq[LRU_GEN_ANON] (max_seq- 3) andlru_gen_imbalanced()still counts unreclaimable anon pages in old[LRU_GEN_ANON]. As a result, reclaim fails to age fromDEF_PRIORITYdown toDEF_PRIORITY/ 2 + 1 and then directly evicts the youngest file generation (max_seq) without page-table aging or tier protection.
This is a similar performance concern to the ones raised earlier [1][2]. As mentioned in my earlier reply, it could be addressed by the approach Bo Zhang suggested to me off-list:
@@ -5233,6 +5231,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
unsigned long orig_file, orig_anon;
DEFINE_MAX_SEQ(lruvec);
bool no_populatable;+ long scanned;
if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) {
need_rotate = true;@@ -5241,8 +5240,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
orig_anon = nr[LRU_GEN_ANON];
orig_file = nr[LRU_GEN_FILE];- if (!evict_folio_lists(nr, lruvec, sc, swappiness,MIN_LRU_BATCH))
- break;
+ scanned = evict_folio_lists(nr, lruvec, sc,
swappiness, MIN_LRU_BATCH);
if (should_abort_scan(lruvec, sc))
break;
@@ -5263,6 +5261,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
should_age = true;
}+ if (!scanned)
+ break;
+
cond_resched();
}[1] https://lore.kernel.org/linux-mm/CAGsJ_4xMUoTsPU-=_QVDqZ-p4Dt+vGJC5G8hed3PWNAk3iv8Bw@mail.gmail.com/
[2] https://lore.kernel.org/linux-mm/CAGsJ_4wFye_co4bnJwuaoyshZOE+gcWFthmWD3r94dQgVHp8GQ@mail.gmail.com/
Quoted 4 lines
Should
get_swappiness()return 0 when !can_reclaim_anon_pages() instead, so swappiness is 0 consistently acrosstry_to_shrink_lruvec(),try_to_inc_min_seq(),should_run_aging(), andlru_gen_imbalanced()?
No. Please don't suggest hacking get_swappiness() again. It would cause a cold/hot inversion in inc_min_seq().
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16