mm: mglru: fix swappiness behavior
Subsystems (4)
Barry Song (Xiaomi)
reviewerv9 cover mm: mglru: fix swappiness behavior
Classic LRU respects swappiness, while MGLRU only weakly honors it when selecting the scan type. The fallback logic can easily override the bias established by get_type_to_scan(). As a result, swappiness has a much smaller overall impact when changing from 1 to 200, as shown by the data presented later in this cover letter.
In this patchset, patches 1-8 from Kairui introduce a model that uses refaults to determine nr_to_scan[] for both file and anon in MGLRU. For each reclaim operation, we calculate how many file and anon folios to scan, similar to CLRU, rather than relying on get_type_to_scan() and fallback mechanisms to decide what to reclaim, which have proven rather unreliable in honoring swappiness. This model also allows scanning and reclaiming from the two youngest generations, including the youngest generation itself. This can help resolve OOM situations[1] and reduce the need for aging.
As Kairui mentioned, the OOM reproducer was demonstrated previously [2], and that series was merged in v7 [3]. The problem was largely alleviated as described, but not completely fixed. Under higher memory pressure, the system could still hit OOM. Before this series, Kairui could still reproduce OOM after hundreds of iterations. With the fixing patch in this series applied, the problem no longer occurs.
Patch 9 from Barry fixes a cold/hot inversion exacerbated by patches 1–8 (which also exists in mainline) and improves reclaim efficiency.
Patches 10-15 from Gaofei and Barry optimize aging based on folio distribution imbalance and improve reclaim behavior at extreme swappiness values such as 1 and 200.
Patch 16 from Bo Zhang addresses the GFP_NOIO issue in MGLRU.
After the patchset, the results track CLRU across the full swappiness range (see table below), with more file scanning at low swappiness and more anon scanning at high swappiness.
pgsteal_file
swappiness CLRU MGLRU MGLRU+patches 1 6,620,680 745,968 6,197,601 36 807,807 508,974 564,256 71 553,144 417,700 420,008 106 402,880 402,474 363,635 141 240,293 337,797 312,624 176 187,682 330,045 238,722 200 0 343,949 0
pgsteal_anon
swappiness CLRU MGLRU MGLRU+patches 1 4,436,199 2,671,934 5,011,932 36 2,858,117 3,032,747 3,039,483 71 3,042,392 2,827,735 3,132,431 106 3,292,622 3,184,314 3,238,658 141 3,612,869 2,999,000 3,306,901 176 3,827,154 3,134,127 3,714,411 200 13,825,283 3,269,753 14,481,445
pgscan_file
swappiness CLRU MGLRU MGLRU+patches 1 11,853,625 877,311 13,108,154 36 1,016,826 582,783 657,973 71 662,823 474,772 457,686 106 496,374 451,285 390,056 141 277,109 382,604 329,819 176 206,261 372,059 246,141 200 0 393,608 0
pgscan_anon
swappiness CLRU MGLRU MGLRU+patches 1 20,804,386 5,847,747 6,936,021 36 17,830,330 6,740,075 5,975,744 71 18,357,293 6,126,194 6,797,243 106 19,938,920 6,431,355 7,602,990 141 19,609,579 5,997,762 8,540,965 176 21,612,749 6,071,032 10,532,510 200 44,776,474 6,544,125 30,296,434
refault_file
swappiness CLRU MGLRU MGLRU+patches 1 6,523,070 667,302 6,118,731 36 722,110 431,585 486,604 71 477,062 340,241 342,746 106 327,610 324,747 286,465 141 161,827 265,959 233,860 176 115,551 258,515 171,174 200 0 268,049 0
refault_anon
swappiness CLRU MGLRU MGLRU+patches 1 867,728 552,829 1,368,563 36 419,635 657,745 636,567 71 536,299 611,345 689,528 106 580,742 693,689 707,227 141 706,151 639,850 753,779 176 706,499 685,086 934,553 200 4,693,850 733,273 5,491,234
With a normal swappiness setting (without extreme bias), I see almost no noticeable performance difference between the two versions.
For example, with swappiness=60, the average results over 5 rounds are:
metric W/o patch W/ patch real 1m49.273s 1m48.836s user 25m33.186s 25m33.368s sys 3m41.851s 3m38.388s
[1] https://github.com/ryncsn/emm-test-project/tree/master/file-anon-mix-pressure
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
v9:
* reduce one iteration for lruvec_populated_min_seq();
* let lruvec_is_reclaimable() count all zones, for min_ttl;
* refine lru_gen_imbalanced(), AGING_PRIORITY macro, roundup/down swappiness; per Kairui;
* let evict_folio_lists() return "drained" per Kairui, thanks!
* one bounded retry for scanned=0 and reduce concurrent aging
for no-populatable(drained) aging;
v8 link:
https://lore.kernel.org/linux-mm/[email protected]/
v8:
* Fixed Gaofei Lyu's name;
* for debugfs proactive reclaim, use DEF_PRIORITY;
* refine no-populatable aging (limit frequency, allow an aging after eviction returns 0 scanned)
* refine "restrict eviction accounting to eligible zones" for potential premature OOM;
* Collect Baolin's reviewed-by tag, thanks!
v7 link:
https://lore.kernel.org/linux-mm/[email protected]/
v7:
* fix a build error of bisect and address some other sashiko minor comments;
* add no-populatable aging;
v6 link:
https://lore.kernel.org/linux-mm/[email protected]/
v6:
* rebase on top of mm-new(a lot of conflicts due to recent active mglru developments);
* only count eligible zones below sc->reclaim_idx for lruvec_evictable_size() and lru_gen_imbalanced();
* GFP_NOIO further fixing by Bo Zhang, exclude unreclaimable type in lruvec_evictable_size().
* Add Kunwu's reviewed-by, thanks!
* RFC v5 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v5:
* Remove the aging optimizations, which have been merged - thanks for Baolin, Baoqun, Lian Wang, Kairui, Kunwu, Ridong reviewing on this;
* Add Kairui's model for determining nr_to_scan[] for both types.
* Fix cold/hot inversion for rejected folios - Barry.
* Improve and clean up imbalanced aging, and optimize extreme swappiness values of 1 and 200 - Gaofei, Barry.
* Address the GFP_NOIO issue in MGLRU - Bo Zhang.
* RFC v4 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v4:
* Clean up isolate_folios() based on Baolin's feedback. Thanks!
* Further improve isolate_folios() and scan_folios(), including handling of exhausted reclaim types.
* Add fixes and performance improvements to inc_min_seq(), and reduce lock contention.
* Improve the handling of imbalanced aging and remaining generation-based aging.
* RFC v3 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v3:
- Added Ridong's patch of improving isolate_folios() readability;
- for lru_gen_imbalanced(), apply some suggestions from Zicheng and Kairui, considering lruvec size and use macro;
- drop "mm: mglru: only fall back when reclaim is running at high priority"
- mitigate aging for balanced swappiess value(near middle 100) as swappiness bias is less a concern for them; but slightly increased aging overhead might not be good.
* RFC v2 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!
Barry Song (Xiaomi) (5):
mm/mglru: Avoid placing rejected folios below populated gen mm/mglru: Run aging when one type is drained
mm/mglru: Restrict eviction accounting to eligible zones mm/mglru: Reduce LRU lock contention by shrinking folio batches in
inc_min_seq
mm/mglru: Make swappiness=1 file reclaim more aggressive
Bo Zhang (1):
mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO
allocation
Gaofei Lyu (2):
mm/mglru: Run aging when pages are rare outside the youngest gen mm/mglru: Avoid prolonged anon reclaim with swappiness=200
Kairui Song (8):
mm/mglru: report evictable sizes per type
mm/mglru: extract lruvec_populated_min_seq() helper mm/mglru: reclaim from the oldest populated gen mm/mglru: lift the forced protection of reclaimer mm/mglru: skip zones above reclaim_idx instead of rotating them mm/mglru: return for unavailable anon reclaim earlier mm/mglru: balance anon and file scan budget by refault rate mm/mglru: lift youngest gen protection for higher priority
include/linux/mm_inline.h | 22 +- mm/trace_vmscan.h | 4 +- mm/vmscan.c | 514 +++++++++++++++++++++++++------------- mm/vmscan.h | 7 +- 4 files changed, 364 insertions(+), 183 deletions(-)
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 1/16 mm/mglru: report evictable sizes per type
From: Kairui Song <[email protected]>
Change lruvec_evictable_size() to return the number of evictable anonymous and file pages separately instead of a single total, so that callers can weigh the two types independently.
Update lruvec_is_sizable() and get_nr_to_scan() accordingly; the summed totals are unchanged, so there is no functional change.
--- mm/vmscan.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
mm/vmscan.c +13 -9
@@ -4193,28 +4193,31 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) +static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, +unsigned long nr[]) {int type; -unsigned long seq, total =0 ; +unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); + nr[LRU_GEN_ANON ] =0 ; + nr[LRU_GEN_FILE ] =0 ; +for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - total +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq); } - -return total; }static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) { -unsigned long total; +unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); - total =lruvec_evictable_size (lruvec, swappiness); +lruvec_evictable_size (lruvec, swappiness, sizes); + total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */ return total >> sc->priority;
@@ -5028,7 +5031,7 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc,struct mem_cgroup *memcg,int swappiness) { -unsigned long nr_to_scan, evictable; +unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ];struct pglist_data *pgdat =lruvec_pgdat (lruvec);/*
@@ -5044,7 +5047,8 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,return 0 ; } - evictable =lruvec_evictable_size (lruvec, swappiness); +lruvec_evictable_size (lruvec, swappiness, sizes); + evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ]; nr_to_scan =apply_proportional_protection (memcg, sc, evictable); nr_to_scan >>= sc->priority;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 2/16 mm/mglru: extract lruvec_populated_min_seq() helper
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
The search for the oldest populated generation of a type, previously embedded in try_to_inc_min_seq(), is extracted into the new lruvec_populated_min_seq() helper. It returns the oldest populated seq of a given type, bounded by the given max_seq so the search never goes beyond it.
No functional change.
--- mm/vmscan.c | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-)
mm/vmscan.c +29 -16
@@ -3992,30 +3992,45 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)return true; } +/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ +static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, +unsigned long max_seq) +{ +int gen, zone; +unsigned long min_seq; +struct lru_gen_folio *lrugen = &lruvec->lrugen; + +lockdep_assert_held (&lruvec->lru_lock); + + min_seq =READ_ONCE (lrugen->min_seq[type]); +while (min_seq < max_seq) { + gen =lru_gen_from_seq (min_seq); +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +if (!list_empty (&lrugen->folios[gen][type][zone])) +return min_seq; + } + min_seq++; + } + +return max_seq; +} +static void try_to_inc_min_seq (struct lruvec *lruvec,int swappiness) { -int gen, type, zone; +int type;bool seq_inc_flag = false;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MIN_SEQ (lruvec); +unsigned long seq;VM_WARN_ON_ONCE (!seq_is_valid (lruvec)); -/* find the oldest populated generation */ +/* Requires at least MIN_NR_GENS gens for basic functionality */ + seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { -while (min_seq[type] +MIN_NR_GENS <= lrugen->max_seq) { - gen =lru_gen_from_seq (min_seq[type]); - -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { -if (!list_empty (&lrugen->folios[gen][type][zone])) - goto next; - } - - min_seq[type]++; + min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); +if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; - } -next: - ; }/*
@@ -4027,8 +4042,6 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)/* see the comment on lru_gen_folio */ if (swappiness && swappiness <=MAX_SWAPPINESS ) { -unsigned long seq = lrugen->max_seq -MIN_NR_GENS ; -if (min_seq[LRU_GEN_ANON ] > seq && min_seq[LRU_GEN_FILE ] < seq) min_seq[LRU_GEN_ANON ] = seq;else if (min_seq[LRU_GEN_FILE ] > seq && min_seq[LRU_GEN_ANON ] < seq)
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 3/16 mm/mglru: reclaim from the oldest populated gen
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
Currently MGLRU tries to keep the gen numbers of the two types in sync, avoiding reducing the gen number of one type when the other type has more gens, as described in the comment in try_to_inc_min_seq().
This makes scan_folios() stuck reclaiming the type with fewer gens: it always scanned the oldest gen of the type, even when that gen was empty, so reclaim made no progress on that type unless forced aging created a new gen and eliminated the dead empty gen.
There are other approaches, like lifting the gen number synchronization [1]. However, that would break the gen layout used by lru_gen_folio_seq(): newly added folios would no longer have a well-defined hotness according to their gen number, which reduces accuracy. It is also not pressure-aware, since with gentle reclaim we still want to respect the gen layout (hotness).
Instead, reclaim from whichever populated oldest gen is left, but only when the default reclaim priority can't satisfy the reclaimer's requirement. This is typical for swappiness limitations: e.g., low swappiness asks the reclaimer to reclaim from file, but file gens are drained, so the reclaim cycle fails and priority is raised.
At the default priority the scan quota is 1/4096 of the evictable size, and the old behavior is kept there to respect the hotness distribution. Once priority is raised, the quota is at least 1/2048, so the hotness bias introduced by reclaiming from a younger gen is bounded by that quota, which seems fair. Global reclaim for MGLRU may start with DEF_PRIORITY / 2 (1/64); the ratio is still reasonable at that point, and when global pressure is more critical, we should reclaim harder.
Note that scan_folios() still refuses to reclaim a type left with only MIN_NR_GENS gens, so a completely drained type still requires aging. This change only avoids scanning empty gens when populated reclaimable gens remain.
--- mm/vmscan.c | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-)
mm/vmscan.c +19 -11
@@ -3312,11 +3312,9 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased) }/* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio) +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio,int old_gen) { -int type =folio_is_file_lru (folio); -struct lru_gen_folio *lrugen = &lruvec->lrugen; -int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); +int new_gen;bool gen_increased; new_gen =__folio_inc_gen (folio, old_gen, &gen_increased);
@@ -4679,7 +4677,7 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent,******************************************************************************/ static bool sort_folio (struct lruvec *lruvec,struct folio *folio,struct scan_control *sc, -int tier_idx) +int min_gen,int tier_idx) {int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);
@@ -4697,14 +4695,14 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_creturn false;/* promoted */ -if (gen !=lru_gen_from_seq (lrugen->min_seq[type])) { +if (gen != min_gen) {list_move (&folio->lru, &lrugen->folios[gen][type][zone]);return true; }/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) { - gen =folio_inc_gen (lruvec, folio); + gen =folio_inc_gen (lruvec, folio, min_gen);list_move (&folio->lru, &lrugen->folios[gen][type][zone]);/* don't count the workingset being lazily promoted */
@@ -4719,7 +4717,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c/* ineligible */ if (zone > sc->reclaim_idx) { - gen =folio_inc_gen (lruvec, folio); + gen =folio_inc_gen (lruvec, folio, min_gen);list_move_tail (&folio->lru, &lrugen->folios[gen][type][zone]);return true; }
@@ -4762,17 +4760,27 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int scanned =0 ;int isolated =0 ;int skipped =0 ; +unsigned long min_seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); +/* +* Respect the gen hotness distribution at low priority to avoid +* aging and scan cost, but prefer to satisfy the reclaimers request +* when under pressure to respect swappiness more and reduce IO cost. +*/ +if (sc->priority ==DEF_PRIORITY ) + min_seq = lrugen->min_seq[type]; +else + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 ); +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS )return 0 ; - gen =lru_gen_from_seq (lrugen->min_seq[type]); - + gen =lru_gen_from_seq (min_seq);for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;
@@ -4790,7 +4798,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned += delta; -if (sort_folio (lruvec, folio, sc, tier)) +if (sort_folio (lruvec, folio, sc, gen, tier)) sorted += delta;else if (isolate_folio (lruvec, folio, sc)) {list_add (&folio->lru, list);
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 4/16 mm/mglru: lift the forced protection of reclaimer
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
When a type is drained down to MIN_NR_GENS gens, the reclaimer will fail to make any progress on that type without aging.
However aging comes with a cost, and it will take time to finish. Even if we do aging more proactively, when under great pressure, gens will still be drained very fast so the new gens produced by aging can't catch up with the consumption. With MIN_NR_GENS setting blocking the reclaim, we will run into OOM. This OOM can be reproduced by the previous test program [1] used for fixing another aging problem [2]. When running in great pressure, certain workloads will keep seeing premature OOM with MGLRU, and never see any OOM with classical LRU.
This might get worse if we implement async aging, or avoid hotness inversion in aging, as they will slow down the aging process even more.
To fix this, let's first drop the get_nr_gens() == MIN_NR_GENS early return and let the raised-priority scan bound cover max_seq itself.
Mapped folios will still survive as the rmap PTE check will keep them. Unmapped cache folios may thrash more under heavy memory pressure. This should instead be addressed through refault-distance and frequency-aware promotion [3].
--- mm/vmscan.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
mm/vmscan.c +25 -9
@@ -4677,7 +4677,7 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent,******************************************************************************/ static bool sort_folio (struct lruvec *lruvec,struct folio *folio,struct scan_control *sc, -int min_gen,int tier_idx) +int min_gen,int max_gen,int tier_idx) {int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio);
@@ -4694,12 +4694,22 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_cif (!folio_evictable (folio))return false; -/* promoted */ +/* promoted, or moved due to reparenting. */ if (gen != min_gen) {list_move (&folio->lru, &lrugen->folios[gen][type][zone]);return true; } +/* +* We are reclaiming the last gen left. Folios in this gen can no +* longer be promoted or protected, so just isolate them. The one +* exception is lru_gen_reparent_memcg(), which is handled above. +*/ +if (min_gen == max_gen) { +VM_WARN_ON_ONCE (zone > sc->reclaim_idx); +return false; + } +/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) { gen =folio_inc_gen (lruvec, folio, min_gen);
@@ -4754,15 +4764,16 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct list_head *list, int *isolatedp) { int i; -int gen;enum node_stat_item item;int sorted =0 ;int scanned =0 ;int isolated =0 ;int skipped =0 ; +int gen, max_gen;unsigned long min_seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +DEFINE_MAX_SEQ (lruvec);VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list));
@@ -4775,18 +4786,24 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (sc->priority ==DEF_PRIORITY ) min_seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq -MIN_NR_GENS +1 ); - -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) -return 0 ; + min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq); gen =lru_gen_from_seq (min_seq); + max_gen =lru_gen_from_seq (max_seq); +for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;int zone = (sc->reclaim_idx + i) %MAX_NR_ZONES ;struct list_head *head = &lrugen->folios[gen][type][zone]; +/* +* If there is only one gen left, rotating the unreclaimable +* zone become meaningless, just skip them. +*/ +if (gen == max_gen && zone > sc->reclaim_idx) +continue ; +while (!list_empty (head)) {struct folio *folio =lru_to_folio (head);int delta =folio_nr_pages (folio);
@@ -4797,8 +4814,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); scanned += delta; - -if (sort_folio (lruvec, folio, sc, gen, tier)) +if (sort_folio (lruvec, folio, sc, gen, max_gen, tier)) sorted += delta;else if (isolate_folio (lruvec, folio, sc)) {list_add (&folio->lru, list);
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 5/16 mm/mglru: skip zones above reclaim_idx instead of rotating them
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
scan_folios() used to visit every zone. Folios in zones above sc->reclaim_idx can never be isolated, so sort_folio() rotated them to a younger generation, the "ineligible" branch, or, when only one gen was left, skipped them. This wasted scan work on every pass, and the rotation kept the protected folios from aging.
Restrict both the scan and the gen selection to the eligible zones, like the legacy LRU, which only isolates folios in zones up to reclaim_idx:
- lruvec_populated_min_seq() takes a zone_limit, so the oldest populated gen is searched only among the zones the scan can reclaim from. A gen populated only in protected zones no longer stalls the scan.
- scan_folios() iterates the eligible zones in the descending order of the legacy zonelist walk, draining the zone at reclaim_idx first.
- The ineligible branch of sort_folio() is now dead and removed, with a VM_WARN_ON_ONCE_FOLIO() guarding the new invariant.
try_to_inc_min_seq() still considers all zones, so min_seq[type] is only bumped once the oldest gen is empty everywhere. At DEF_PRIORITY the search bound stays min_seq[type], so the scan still starts from the oldest gen regardless of whether it is populated.
--- mm/vmscan.c | 47 +++++++++++++++++++---------------------------- 1 file changed, 19 insertions(+), 28 deletions(-)
mm/vmscan.c +19 -28
@@ -3992,7 +3992,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* Similar to lruvec_evictable_size, return the oldest populated seq of the type */ static unsigned long lruvec_populated_min_seq (struct lruvec *lruvec,int type, -unsigned long max_seq) +int zone_limit,unsigned long max_seq) {int gen, zone;unsigned long min_seq;
@@ -4003,7 +4003,7 @@ static unsigned long lruvec_populated_min_seq(struct lruvec *lruvec, int type, min_seq =READ_ONCE (lrugen->min_seq[type]);while (min_seq < max_seq) { gen =lru_gen_from_seq (min_seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +for (zone =0 ; zone < zone_limit; zone++) {if (!list_empty (&lrugen->folios[gen][type][zone]))return min_seq; }
@@ -4026,7 +4026,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)/* Requires at least MIN_NR_GENS gens for basic functionality */ seq = lrugen->max_seq -MIN_NR_GENS ;for_each_evictable_type (type, swappiness) { - min_seq[type] =lruvec_populated_min_seq (lruvec, type, seq +1 ); + min_seq[type] =lruvec_populated_min_seq (lruvec, type,MAX_NR_ZONES , seq +1 );if (min_seq[type] > lrugen->min_seq[type]) seq_inc_flag = true; }
@@ -4689,6 +4689,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_cstruct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE_FOLIO (gen >=MAX_NR_GENS , folio); +VM_WARN_ON_ONCE_FOLIO (zone > sc->reclaim_idx, folio);/* unevictable: let it through and the generic path will cull it */ if (!folio_evictable (folio))
@@ -4705,10 +4706,8 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c* longer be promoted or protected, so just isolate them. The one * exception is lru_gen_reparent_memcg(), which is handled above. */ -if (min_gen == max_gen) { -VM_WARN_ON_ONCE (zone > sc->reclaim_idx); +if (min_gen == max_gen)return false; - }/* protected */ if (tier > tier_idx || refs + workingset ==BIT (LRU_REFS_WIDTH ) +1 ) {
@@ -4725,13 +4724,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_creturn true; } -/* ineligible */ -if (zone > sc->reclaim_idx) { - gen =folio_inc_gen (lruvec, folio, min_gen); -list_move_tail (&folio->lru, &lrugen->folios[gen][type][zone]); -return true; - } -return false; }
@@ -4763,16 +4755,17 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int type, int tier, struct list_head *list, int *isolatedp) { - int i; enum node_stat_item item; +int zone_idx;int sorted =0 ;int scanned =0 ;int isolated =0 ;int skipped =0 ;int gen, max_gen; -unsigned long min_seq; +unsigned long seq;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct pglist_data *pgdat =lruvec_pgdat (lruvec);DEFINE_MAX_SEQ (lruvec);VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );
@@ -4784,24 +4777,22 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* when under pressure to respect swappiness more and reduce IO cost. */ if (sc->priority ==DEF_PRIORITY ) - min_seq = lrugen->min_seq[type]; + seq = lrugen->min_seq[type];else - min_seq =lruvec_populated_min_seq (lruvec, type, lrugen->max_seq); + seq = lrugen->max_seq; - gen =lru_gen_from_seq (min_seq); + seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq); -for (i =MAX_NR_ZONES ; i >0 ; i--) { +/* Scan eligible zones in descending order. */ +for (zone_idx = sc->reclaim_idx; zone_idx >=0 ; zone_idx--) {LIST_HEAD (moved);int skipped_zone =0 ; -int zone = (sc->reclaim_idx + i) %MAX_NR_ZONES ; -struct list_head *head = &lrugen->folios[gen][type][zone]; +struct zone *zone = &pgdat->node_zones[zone_idx]; +struct list_head *head = &lrugen->folios[gen][type][zone_idx]; -/* -* If there is only one gen left, rotating the unreclaimable -* zone become meaningless, just skip them. -*/ -if (gen == max_gen && zone > sc->reclaim_idx) +if (!managed_zone (zone))continue ;while (!list_empty (head)) {
@@ -4811,7 +4802,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_is_file_lru (folio) != type, folio); -VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); +VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone_idx, folio); scanned += delta;if (sort_folio (lruvec, folio, sc, gen, max_gen, tier))
@@ -4830,7 +4821,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (skipped_zone) {list_splice (&moved, head); -__count_zid_vm_events (PGSCAN_SKIP , zone, skipped_zone); +__count_zid_vm_events (PGSCAN_SKIP , zone_idx, skipped_zone); skipped += skipped_zone; }
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 6/16 mm/mglru: return for unavailable anon reclaim earlier
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
No feature change, just shuffle some code, prepare for later commits.
--- mm/vmscan.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
mm/vmscan.c +13 -14
@@ -5060,20 +5060,6 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, struct mem_cgroup *memcg, int swappiness) {unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ]; -struct pglist_data *pgdat =lruvec_pgdat (lruvec); - -/* -* Proactive reclaim initiated by userspace for anonymous memory only. -* SWAPPINESS_ANON_ONLY is set only on the proactive reclaim path, so -* warn if it shows up elsewhere. When anon cannot be reclaimed (e.g. -* no swap), return 0 to skip the scan entirely, avoiding useless scan -* work when there is nothing eligible to reclaim. -*/ -if (swappiness ==SWAPPINESS_ANON_ONLY ) { -WARN_ON_ONCE (!sc->proactive); -if (!can_reclaim_anon_pages (memcg, pgdat->node_id, sc)) -return 0 ; - }lruvec_evictable_size (lruvec, swappiness, sizes); evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];
@@ -5126,6 +5112,19 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec); +/* +* Proactive reclaim initiated by userspace for anonymous memory only. +* SWAPPINESS_ANON_ONLY is set only on the proactive reclaim path, so +* warn if it shows up elsewhere. When anon cannot be reclaimed (e.g. +* no swap), return 0 to skip the scan entirely, avoiding useless scan +* work when there is nothing eligible to reclaim. +*/ +if (swappiness ==SWAPPINESS_ANON_ONLY ) { +WARN_ON_ONCE (!sc->proactive); +if (!can_reclaim_anon_pages (memcg,lruvec_pgdat (lruvec)->node_id, sc)) +return false; + } + nr_to_scan =get_nr_to_scan (lruvec, sc, memcg, swappiness);while (nr_to_scan >0 ) {int delta;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 7/16 mm/mglru: balance anon and file scan budget by refault rate
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
Replace get_type_to_scan() and isolate_folios(), which selected one type per batch, with a proportional split of the scan budget between anon and file, and scan both types each round.
lru_gen_balance_scan() splits the total by
anon : file = swappiness * r_file * w_anon :
(MAX_SWAPPINESS - swappiness) * r_anon * w_file
where r_anon and r_file are the per-type refault rates scaled by 4096, clamped to [1, 4096]; a rate at the 4096 ceiling means the type refaults at least as often as it is evicted. The w factors bias the split toward the LRU sizes at mild swappiness and follow the IO cost ratio at the extremes. Both rate accumulators start at MIN_LRU_BATCH, so a type without eviction history reads as 100% until evictions accumulate, and the floor of 1 keeps a type that never refaults from zeroing out the other type's share. Swappiness 0 and SWAPPINESS_ANON_ONLY keep their single-type behavior; at swappiness == MAX_SWAPPINESS the file factor is zero, so anon gets the whole budget, as in the classical LRU; swappiness 1 no longer forces file-only scanning as the old code did.
The proportional protection and priority reduction are applied to the total once, before the split, and each budget is capped at its type's evictable size. Regular reclaim derives the total from the evictable sizes, so the budgets sum to it. The proactive path re-splits the remaining target each round, so eviction keeps working while either type makes progress, until the target is met or no progress is possible. The debugfs scan_control gets .priority = DEF_PRIORITY so eviction starts from the oldest gen and reports the default priority in the tracepoints.
Note that we are already lenient with the debugfs proactive reclaim interface. We may reclaim one generation ahead even at DEF_PRIORITY, and with extreme swappiness values such as 1 and 200, we may not fall back to the other type.
To address these cases and preserve the behavior of the previous model before scan-budget-based reclaim, we would need to add another flag to scan_control. However, this may not be worthwhile for a debugfs interface that is likely to be rarely used, especially given that we already have memory-level and node-level proactive reclaim interfaces.
--- mm/vmscan.c | 219 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 138 insertions(+), 81 deletions(-)
mm/vmscan.c +138 -81
@@ -4863,79 +4863,110 @@ static int get_tier_idx(struct lruvec *lruvec, int type)return tier -1 ; } -static int get_type_to_scan (struct lruvec *lruvec,int swappiness) +/* +* Balance the anon vs file scan budget using MGLRU's per-type refault +* rates, where swappiness is the relative IO cost of anon vs file (see +* Documentation/admin-guide/sysctl/vm.rst). +* +* On entry, nr[] holds the evictable size of each type, on return it +* holds each type's scan budget, bounded by its evictable size. +* +* LRU lock not needed: a stale snapshot merely skews the split for one pass, +* and MGLRU aging averages the fault rate so we can simply read it, no need +* to do any extra work here. If reclaim is blocked by unfair type hotness or +* stall aging, the priority will raise and a raised priority drives the +* aging. +* +* Calculation is basically based on following formula with a LRU size bias: +* +* anon : file = swappiness * r_file * w_anon : +* (MAX_SWAPPINESS - swappiness) * r_anon * w_file +* +* r_file / r_anon: refault rate for file / anon. +* w_file / w_anon: LRU size weight of file / anon. +* +* We don't calculate the scan cost. MGLRU uses lazy promotion, which will +* cancel the scan budget naturally. +*/ +static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, +unsigned long total_scan,unsigned long nr[]) { -struct ctrl_pos sp, pv; +struct lru_gen_folio *lrugen = &lruvec->lrugen; +u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor; +u64 anon_scan, file_scan, total; +u64 anon_weight, file_weight; +int size_bias; -if (swappiness <=MIN_SWAPPINESS +1 ) -return LRU_GEN_FILE ; + total = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ]; +if (!total) +return ; -if (swappiness >=MAX_SWAPPINESS ) -return LRU_GEN_ANON ; -/* -* Compare the sum of all tiers of anon with that of file to determine -* which type to scan. -*/ -read_ctrl_pos (lruvec,LRU_GEN_ANON ,LRU_TIER_MIN ,LRU_TIER_MAX , - swappiness, &sp); -read_ctrl_pos (lruvec,LRU_GEN_FILE ,LRU_TIER_MIN ,LRU_TIER_MAX , -MAX_SWAPPINESS - swappiness, &pv); +if (swappiness <=MIN_SWAPPINESS ) { + nr[LRU_GEN_ANON ] =0 ; + nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]); +return ; + } -return positive_ctrl_err (&sp, &pv); -} +if (swappiness ==SWAPPINESS_ANON_ONLY ) { + nr[LRU_GEN_FILE ] =0 ; + nr[LRU_GEN_ANON ] =min_t (u64, total_scan, nr[LRU_GEN_ANON ]); +return ; + } -static inline bool is_single_type_reclaim (int swappiness) -{ -return swappiness ==MIN_SWAPPINESS || - swappiness ==SWAPPINESS_ANON_ONLY ; -} +for (int type =LRU_GEN_ANON ; type <=LRU_GEN_FILE ; type++) { +int hist =lru_hist_from_seq (READ_ONCE (lrugen->min_seq[type])); +/* start at MIN_LRU_BATCH to smooth the rate on gen start */ +u64 refaulted =MIN_LRU_BATCH , evicted =MIN_LRU_BATCH ; -static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int swappiness, -struct list_head *list,int *isolated, -int *isolate_type,int *isolate_scanned) -{ -bool type_fallback_allowed = !is_single_type_reclaim (swappiness); -int type =get_type_to_scan (lruvec, swappiness); -int total_scanned =0 , scanned, tier; -bool tried = false; +for (int tier =0 ; tier <MAX_NR_TIERS ; tier++) { + refaulted +=READ_ONCE (lrugen->avg_refaulted[type][tier]); + refaulted +=atomic_long_read (&lrugen->refaulted[hist][type][tier]); -retry: - tier =get_tier_idx (lruvec, type); - scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + evicted +=READ_ONCE (lrugen->avg_total[type][tier]); + evicted +=atomic_long_read (&lrugen->evicted[hist][type][tier]); + } - total_scanned += scanned; -if (*isolated) { - *isolate_type = type; - *isolate_scanned = scanned; -return total_scanned; +/* +* The rate scaled by 4096 to survive integer division, clamped +* to [1, 4096] (~0% - 100%), the floor of 1 keeps a type that +* never refaults from zeroing out the other type's share. +*/ + refault_rate[type] =mul_u64_u64_div_u64 (refaulted,4096 , evicted); + refault_rate[type] =clamp_t (u64, refault_rate[type],1 ,4096 ); }/* -* We are running out of the current reclaim type. Fall back to -* the other type if allowed. +* The more extreme the swappiness, the wider the IO cost gap between +* anon and file, so the split should follow the IO cost ratio rather +* than the sizes. With a mild swappiness, balance the scan by the +* actual LRU size more. */ -if (!scanned && !tried && type_fallback_allowed) { - type = !type; - tried = true; - type_fallback_allowed = false; - goto retry; - } + size_bias =max (1 ,min (swappiness,MAX_SWAPPINESS - swappiness)); + anon_weight = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ] / size_bias; + file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; + + anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; + +/* Cross-multiply the ratio above. */ + anon_scan =mul_u64_u64_div_u64 (total_scan, anon_factor, anon_factor + file_factor); + file_scan = (u64 )total_scan - anon_scan; +/* -* We scanned some folios but failed to isolate any due to promotions, -* protections, or races. Retry once to avoid a larger loop. +* Bound each type by its evictable size. Hand the exceeding part to +* the other type. Any excess left when total_scan exceeds both sizes +* is dropped, since it only stresses the scanner with no gain. */ -if (scanned && !tried) { - tried = true; - goto retry; - } - -return total_scanned; +if (anon_scan > nr[LRU_GEN_ANON ]) + file_scan += anon_scan - nr[LRU_GEN_ANON ]; +else if (file_scan > nr[LRU_GEN_FILE ]) + anon_scan += file_scan - nr[LRU_GEN_FILE ]; + nr[LRU_GEN_ANON ] =min_t (u64, anon_scan, nr[LRU_GEN_ANON ]); + nr[LRU_GEN_FILE ] =min_t (u64, file_scan, nr[LRU_GEN_FILE ]); }static int evict_folios (unsigned long nr_to_scan,struct lruvec *lruvec, -struct scan_control *sc,int swappiness) +struct scan_control *sc,int type,int swappiness) {LIST_HEAD (list);LIST_HEAD (clean);
@@ -4944,8 +4975,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,enum node_stat_item item;struct reclaim_stat stat;struct lru_gen_mm_walk *walk; -int scanned, reclaimed; -int isolated =0 , nr_isolated =0 , type, type_scanned; +int tier, scanned, reclaimed; +int isolated =0 , nr_isolated =0 ;unsigned long total_reclaimed =0 ;bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -4956,8 +4987,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,/* In case folio deletion left empty old gens, flush them */ try_to_inc_min_seq (lruvec, swappiness); - scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, - &list, &isolated, &type, &type_scanned); + tier =get_tier_idx (lruvec, type); + scanned =scan_folios (nr_to_scan, lruvec, sc, + type, tier, &list, &isolated); nr_isolated = isolated;if (nr_isolated) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
@@ -4979,7 +5011,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (isolated)handle_reclaim_writeback (isolated, pgdat, sc, &stat);trace_mm_vmscan_lru_shrink_inactive (pgdat->node_id, - type_scanned, reclaimed, &stat, sc, + scanned, reclaimed, &stat, sc, type ?LRU_INACTIVE_FILE :LRU_INACTIVE_ANON );list_for_each_entry_safe_reverse (folio, next, &list, lru) {
@@ -5056,18 +5088,18 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; } -static long get_nr_to_scan (struct lruvec *lruvec,struct scan_control *sc, -struct mem_cgroup *memcg,int swappiness) +static void lru_gen_prepare_scan (struct lruvec *lruvec,struct scan_control *sc, +struct mem_cgroup *memcg,int swappiness, +unsigned long nr_to_scan[]) { -unsigned long nr_to_scan, evictable, sizes[ANON_AND_FILE ]; - -lruvec_evictable_size (lruvec, swappiness, sizes); - evictable = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ]; +unsigned long total; - nr_to_scan =apply_proportional_protection (memcg, sc, evictable); - nr_to_scan >>= sc->priority; +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); + total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; -return nr_to_scan; + total =apply_proportional_protection (memcg, sc, total); + total >>= sc->priority; +lru_gen_balance_scan (lruvec, swappiness, total, nr_to_scan); }static bool should_abort_scan (struct lruvec *lruvec,struct scan_control *sc)
@@ -5100,6 +5132,30 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)return true; } +static long evict_folio_lists (unsigned long nr_to_scan[],struct lruvec *lruvec, +struct scan_control *sc,int swappiness,unsigned long max_batch) +{ +int type; +unsigned long batch, scanned, total_scanned =0 ; + +/* +* Scan both types with the given budgets; the caller checks if +* reclaim is satisfied after each pass. +*/ +for (type =LRU_GEN_FILE ; type >=LRU_GEN_ANON ; type--) { +if (!nr_to_scan[type]) +continue ; + + batch =min (nr_to_scan[type], max_batch); + scanned =evict_folios (batch, lruvec, sc, type, swappiness); + + nr_to_scan[type] -=min (nr_to_scan[type], scanned); + total_scanned += scanned; + } + +return total_scanned; +} +/* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg
@@ -5107,8 +5163,8 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)*/ static bool try_to_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) { +unsigned long nr[ANON_AND_FILE ];bool need_rotate = false, should_age = false; -long nr_batch, nr_to_scan;int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -5125,9 +5181,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)return false; } - nr_to_scan =get_nr_to_scan (lruvec, sc, memcg, swappiness); -while (nr_to_scan >0 ) { -int delta; +lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr); + +while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) {DEFINE_MAX_SEQ (lruvec);if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) {
@@ -5142,9 +5198,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) should_age = true; } - nr_batch =min (nr_to_scan,MIN_LRU_BATCH ); - delta =evict_folios (nr_batch, lruvec, sc, swappiness); -if (!delta) +if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ))break ;if (should_abort_scan (lruvec, sc))
@@ -5157,7 +5211,6 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; - nr_to_scan -= delta;cond_resched (); }
@@ -5769,8 +5822,8 @@ static int run_aging(struct lruvec *lruvec, unsigned long seq,static int run_eviction (struct lruvec *lruvec,unsigned long seq,struct scan_control *sc,int swappiness,unsigned long nr_to_reclaim) { -int nr_batch;DEFINE_MAX_SEQ (lruvec); +unsigned long nr_to_scan[ANON_AND_FILE ];if (seq +MIN_NR_GENS > max_seq)return -EINVAL ;
@@ -5786,8 +5839,11 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coif (sc->nr_reclaimed >= nr_to_reclaim)return 0 ; - nr_batch =min (nr_to_reclaim - sc->nr_reclaimed,MAX_LRU_BATCH ); -if (!evict_folios (nr_batch, lruvec, sc, swappiness)) +lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lru_gen_balance_scan (lruvec, swappiness, + nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); + +if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ))return 0 ;cond_resched ();
@@ -5850,6 +5906,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, .may_writepage = true, .may_unmap = true, .may_swap = true, + .priority = DEF_PRIORITY, .reclaim_idx = MAX_NR_ZONES - 1, .gfp_mask = GFP_KERNEL, .proactive = true,
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 8/16 mm/mglru: lift youngest gen protection for higher priority
In reply to Barry Song (Xiaomi)From: Kairui Song <[email protected]>
The forced protection of younger generations contradicts the goal of minimizing the I/O cost of reclaim. Under high memory pressure, MGLRU can repeatedly trigger premature OOMs, while the classic LRU does not.
The problem is that aging may fail to keep up with reclaim. When this happens, reclaimers may all abort prematurely because they see only empty generations or just two generations remaining.
Proactive aging does not solve this problem either. The number of threads performing proactive aging is limited, while the number of concurrent reclaimers can be much larger. This is a fundamental limitation of MGLRU. One possible solution is to use an aging lock. However, the aging lock introduces significant jitter under heavy memory pressure and has been reported to cause noticeable lags on Android.
Furthermore, the generation-based reclaim model can conflict with the goal of swappiness, which is to minimize I/O cost rather than blindly follow folio hotness. The previous commit allowed reclaimers running at non-default priorities to reclaim up to lrugen->max_seq, while reclaimers at the default priority could reclaim only up to lrugen->min_seq.
This patch further refines generation protection based on reclaim priority. At the default priority, reclaimers can reclaim generations down to lrugen->max_seq - MIN_NR_GENS. As reclaim priority increases, reclaimers are progressively allowed to reclaim younger generations.
With this graduated protection, premature OOMs are no longer observed, while swappiness is also respected.
--- mm/vmscan.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
mm/vmscan.c +6 -4
@@ -4772,12 +4772,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (!list_empty (list));/* -* Respect the gen hotness distribution at low priority to avoid -* aging and scan cost, but prefer to satisfy the reclaimers request -* when under pressure to respect swappiness more and reduce IO cost. +* Respect hotness at default priority, and only reach into the +* newest generation when under severe pressure, so hot folios are +* not evicted as soon as priority is raised. */ if (sc->priority ==DEF_PRIORITY ) - seq = lrugen->min_seq[type]; + seq = lrugen->max_seq -MIN_NR_GENS ; +else if (sc->priority >DEF_PRIORITY /2 ) + seq = lrugen->max_seq -1 ;else seq = lrugen->max_seq;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 9/16 mm/mglru: Avoid placing rejected folios below populated gen
In reply to Barry Song (Xiaomi)We used to scan only the oldest generation in scan_folios(). In evict_folios(), we checked whether lru_gen_folio_seq(lruvec, folio, false) == min_seq[type]. If so, the folio was marked active and moved to younger generations.
This was itself buggy, as folios in generation min_seq[type] - 1 would not be marked active by folio_set_active(), potentially causing a cold/hot inversion.
Besides this issue, the old approach also differs from the current behavior, where we can scan fewer than two generations. Now that we can scan younger generations, including the youngest generation, this no longer prevents rejected folios from being placed in a generation below or at the populated generation. Such rejected folios would then be scanned again and again, resulting in extremely low pg_steal/pg_scan ratios. For example:
pgsteal_file: 2312605
pgscan_file: 36465693
reclamation efficiency: 6.34%
We pass scanned_seq to move_folios_to_lru() and make sure we either promote the folio or place it in the youngest generation when the populated generation is already the youngest one.
Also, folio_set_active() is no longer needed.
With this change, the results become:
pgsteal_file: 2683815
pgscan_file: 4224244
reclamation efficiency: 63.53%
The sys time for building the kernel also decreased from 4m30.532s to 3m50.188s while swappiness==1.
--- include/linux/mm_inline.h | 22 +++++++++++++++++++++- mm/vmscan.c | 24 ++++++++++++------------ 2 files changed, 33 insertions(+), 13 deletions(-)
include/linux/mm_inline.h +21 -1
@@ -332,7 +332,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); } -static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +static inline bool __lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq,bool reclaiming) {unsigned long seq;unsigned long flags;
@@ -347,6 +348,8 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return false; seq =lru_gen_folio_seq (lruvec, folio, reclaiming); +if (scanned_seq !=-1UL ) + seq =max (seq,min (scanned_seq +1 , lrugen->max_seq)); gen =lru_gen_from_seq (seq); flags = (gen +1UL ) <<LRU_GEN_PGOFF ;/* see the comment on MIN_NR_GENS about PG_active */
@@ -362,6 +365,17 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return true; } +static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +{ +return __lru_gen_add_folio (lruvec, folio,-1UL , reclaiming); +} + +static inline bool lru_gen_add_rejected_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq) +{ +return __lru_gen_add_folio (lruvec, folio, scanned_seq, false); +} +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {unsigned long flags;
@@ -420,6 +434,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,return false; } +static inline bool lru_gen_add_rejected_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq) +{ +return false; +} +static inline bool lru_gen_del_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) {return false;
mm/vmscan.c +12 -12
@@ -1865,7 +1865,8 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,* * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct list_head *list,bool do_rotate) +static unsigned int move_folios_to_lru (struct list_head *list,unsigned long seq, +bool do_rotate) {int nr_pages, nr_moved =0 ;struct lruvec *lruvec =NULL ;
@@ -1922,7 +1923,7 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate) !folio_test_referenced(folio) && folio_ref_count(folio) == folio_expected_ref_count(folio))lruvec_add_folio_tail (lruvec, folio); - else + else if (seq == -1UL || !lru_gen_add_rejected_folio(lruvec, folio, seq)) lruvec_add_folio(lruvec, folio); nr_pages = folio_nr_pages(folio);
@@ -2041,7 +2042,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed =shrink_folio_list (&folio_list, pgdat, sc, &stat, false,lruvec_memcg (lruvec)); -move_folios_to_lru (&folio_list, true); +move_folios_to_lru (&folio_list,-1UL , true);mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted);
@@ -2152,8 +2153,8 @@ static void shrink_active_list(unsigned long nr_to_scan,/* * Move folios back to the lru list. */ - nr_activate =move_folios_to_lru (&l_active, false); - nr_deactivate =move_folios_to_lru (&l_inactive, false); + nr_activate =move_folios_to_lru (&l_active,-1UL , false); + nr_deactivate =move_folios_to_lru (&l_inactive,-1UL , false);count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate);
@@ -4753,7 +4754,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +unsigned long *scanned_seq) {enum node_stat_item item;int zone_idx;
@@ -4784,6 +4786,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, seq = lrugen->max_seq; seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); + *scanned_seq = seq; gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq);
@@ -4980,6 +4983,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int tier, scanned, reclaimed;int isolated =0 , nr_isolated =0 ;unsigned long total_reclaimed =0 ; +unsigned long scanned_seq;bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec);
@@ -4991,7 +4995,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, &list, &isolated); + type, tier, &list, &isolated, &scanned_seq); nr_isolated = isolated;if (nr_isolated) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
@@ -5017,8 +5021,6 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);list_for_each_entry_safe_reverse (folio, next, &list, lru) { -DEFINE_MIN_SEQ (lruvec); -/* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */ if (!folio_evictable (folio))continue ;
@@ -5037,11 +5039,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* so this effectively promotes them by at least one generation. */ folio_set_lru_refs (folio,0 ); -if (lru_gen_folio_seq (lruvec, folio, false) == min_seq[type]) -folio_set_active (folio); } -move_folios_to_lru (&list, false); +move_folios_to_lru (&list, scanned_seq, false); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) {
Signature · 1 line
2.39.3 (Apple Git-146)
Re: [PATCH v9 09/16] mm/mglru: Avoid placing rejected folios below populated gen
Quoted 26 lines · Barry Song (Xiaomi)
We used to scan only the oldest generation in
scan_folios(). Inevict_folios(), we checked whetherlru_gen_folio_seq(lruvec, folio, false)== min_seq[type]. If so, the folio was marked active and moved to younger generations.This was itself buggy, as folios in generation min_seq[type] - 1 would not be marked active by
folio_set_active(), potentially causing a cold/hot inversion.Besides this issue, the old approach also differs from the current behavior, where we can scan fewer than two generations. Now that we can scan younger generations, including the youngest generation, this no longer prevents rejected folios from being placed in a generation below or at the populated generation. Such rejected folios would then be scanned again and again, resulting in extremely low
pg_steal/pg_scanratios. For example:
pgsteal_file: 2312605
pgscan_file: 36465693
reclamation efficiency: 6.34%We pass
scanned_seqtomove_folios_to_lru()and make sure we either promote the folio or place it in the youngest generation when the populated generation is already the youngest one.
Also,folio_set_active()is no longer needed.
Quoted 4 lines
With this change, the results become:
pgsteal_file: 2683815
pgscan_file: 4224244
reclamation efficiency: 63.53%
That's a really nice improvement! And I have seen the rejected folio got inserted to wrong gen many times, this will help to fix that too.
Quoted 16 lines
The sys time for building the kernel also decreased from 4m30.532s to 3m50.188s while swappiness==1.
Signed-off-by Barry Song (Xiaomi) <[email protected]>Reviewed-by Kunwu Chan <[email protected]>--- include/linux/mm_inline.h | 22 +++++++++++++++++++++- mm/vmscan.c | 24 ++++++++++++------------ 2 files changed, 33 insertions(+), 13 deletions(-)include/linux/mm_inline.h +0
-0
include/linux/mm_inline.h static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · line 336
Quoted 6 lines
include/linux/mm_inline.h +2
-1@@ -332,7 +332,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,return max (READ_ONCE (lrugen->max_seq) - gen +1 ,READ_ONCE (lrugen->min_seq[type])); } -static inline bool lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio,bool reclaiming) +static inline bool __lru_gen_add_folio (struct lruvec *lruvec,struct folio *folio, +unsigned long scanned_seq,bool reclaiming)
A little nitpick: I think the code will be cleaner if we use "long min_seq" and get rid of __lru_gen_add_folio, caller of lru_gen_add_folio just use -1 as min_seq if they don't have a specific gen limitation, but we can just leave this as it is now to avoid more chore update of this one.
...
Quoted 19 lines
include/linux/mm_inline.h +1
-3@@ -4991,7 +4995,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, &list, &isolated); + type, tier, &list, &isolated, &scanned_seq); nr_isolated = isolated;if (nr_isolated) __mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,@@ -5017,8 +5021,6 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);list_for_each_entry_safe_reverse (folio, next, &list, lru) { -DEFINE_MIN_SEQ (lruvec); -/* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */ if (!folio_evictable (folio))continue ;
include/linux/mm_inline.h static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5041
Quoted 5 lines
include/linux/mm_inline.h +0
-2@@ -5037,11 +5039,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* so this effectively promotes them by at least one generation. */ folio_set_lru_refs (folio,0 ); -if (lru_gen_folio_seq (lruvec, folio, false) == min_seq[type]) -folio_set_active (folio);
Nice, this part is causing trouble indeed.
Quoted 10 lines
include/linux/mm_inline.h +1
-2} -move_folios_to_lru (&list, false); +move_folios_to_lru (&list, scanned_seq, false); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) { --2.39.3 (Apple Git-146)
Thanks! Looks good to me.
Barry Song (Xiaomi)
reviewerPATCH v9 10/16 mm/mglru: Run aging when pages are rare outside the youngest gen
In reply to Barry Song (Xiaomi)From: Gaofei Lyu <[email protected]>
Run aging when the total number of pages outside the youngest generation across both types falls below 1/MAX_NR_GENS of the number of pages in the youngest generation. For extreme swappiness values such as 1 and 200, also consider the balance of the corresponding type.
This improves reclaim efficiency, especially at extreme swappiness values, while reducing system time.
To avoid the overhead of aging, we only allow imbalance-based aging when the reclaim priority is low enough that we might reclaim the youngest generation.
--- mm/vmscan.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-)
mm/vmscan.c +42 -2
@@ -2743,6 +2743,8 @@ static bool should_clear_pmd_young(void)#define get_memcg_gen (seq) ((seq) % MEMCG_NR_GENS)#define get_memcg_bin (bin) ((bin) % MEMCG_NR_BINS) +#define AGING_PRIORITY (DEF_PRIORITY / 2) +static int get_swappiness (struct lruvec *lruvec,struct scan_control *sc) {struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -4780,7 +4782,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,*/ if (sc->priority ==DEF_PRIORITY ) seq = lrugen->max_seq -MIN_NR_GENS ; -else if (sc->priority >DEF_PRIORITY /2 ) +else if (sc->priority >AGING_PRIORITY ) seq = lrugen->max_seq -1 ;else seq = lrugen->max_seq;
@@ -5073,6 +5075,40 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,struct scan_control *sc, +int swappiness) +{ +unsigned long youngest[ANON_AND_FILE ] = {0 }, old[ANON_AND_FILE ] = {0 }; +struct lru_gen_folio *lrugen = &lruvec->lrugen; +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); +int type; + +if (sc->priority >AGING_PRIORITY ) +return false; + +/* +* Round up or down corner swappiness, to balance the heavy biased +* type better +*/ +if (swappiness ==MIN_SWAPPINESS +1 ) + swappiness =MIN_SWAPPINESS ; +else if (swappiness ==MAX_SWAPPINESS ) + swappiness =MAX_SWAPPINESS +1 ; + +for_each_evictable_type (type, swappiness) { +unsigned long seq; + + youngest[type] =lruvec_gen_size (lrugen, type, max_seq); +for (seq = min_seq[type]; seq < max_seq; seq++) + old[type] +=lruvec_gen_size (lrugen, type, seq); + } + +/* the vast majority of pages sit in the youngest gen */ +return youngest[LRU_GEN_ANON ] + youngest[LRU_GEN_FILE ] > + (old[LRU_GEN_ANON ] + old[LRU_GEN_FILE ]) *MAX_NR_GENS ; +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) {
@@ -5087,7 +5123,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* run aging when folios outside the youngest gen are rare */ +return lru_gen_imbalanced (lruvec, sc, swappiness); } static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc,
Signature · 1 line
2.39.3 (Apple Git-146)
Re: [PATCH v9 10/16] mm/mglru: Run aging when pages are rare outside the youngest gen
Quoted 67 lines · Barry Song (Xiaomi)
From: Gaofei Lyu <[email protected]>
Run aging when the total number of pages outside the youngest generation across both types falls below 1/MAX_NR_GENS of the number of pages in the youngest generation. For extreme swappiness values such as 1 and 200, also consider the balance of the corresponding type.
This improves reclaim efficiency, especially at extreme swappiness values, while reducing system time.
To avoid the overhead of aging, we only allow imbalance-based aging when the reclaim priority is low enough that we might reclaim the youngest generation.
Signed-off-by Gaofei Lyu <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-)mm/vmscan.c +21
-1@@ -2743,6 +2743,8 @@ static bool should_clear_pmd_young(void)#define get_memcg_gen (seq) ((seq) % MEMCG_NR_GENS)#define get_memcg_bin (bin) ((bin) % MEMCG_NR_BINS) +#define AGING_PRIORITY (DEF_PRIORITY / 2) +static int get_swappiness (struct lruvec *lruvec,struct scan_control *sc) {struct mem_cgroup *memcg =lruvec_memcg (lruvec);@@ -4780,7 +4782,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,*/ if (sc->priority ==DEF_PRIORITY ) seq = lrugen->max_seq -MIN_NR_GENS ; -else if (sc->priority >DEF_PRIORITY /2 ) +else if (sc->priority >AGING_PRIORITY ) seq = lrugen->max_seq -1 ;else seq = lrugen->max_seq;@@ -5073,6 +5075,40 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,struct scan_control *sc, +int swappiness) +{ +unsigned long youngest[ANON_AND_FILE ] = {0 }, old[ANON_AND_FILE ] = {0 }; +struct lru_gen_folio *lrugen = &lruvec->lrugen; +DEFINE_MAX_SEQ (lruvec); +DEFINE_MIN_SEQ (lruvec); +int type; + +if (sc->priority >AGING_PRIORITY ) +return false; + +/* +* Round up or down corner swappiness, to balance the heavy biased +* type better +*/ +if (swappiness ==MIN_SWAPPINESS +1 ) + swappiness =MIN_SWAPPINESS ;
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5105
Quoted 10 lines
mm/vmscan.c +10
-0+else if (swappiness ==MAX_SWAPPINESS ) + swappiness =MAX_SWAPPINESS +1 ; + +for_each_evictable_type (type, swappiness) { +unsigned long seq; + + youngest[type] =lruvec_gen_size (lrugen, type, max_seq); +for (seq = min_seq[type]; seq < max_seq; seq++) + old[type] +=lruvec_gen_size (lrugen, type, seq); + }
A little nit: with the rounding above, I think we don't need youngest / old as an array, just an accumulative value seems enough now? But feel free to ignore this so we can avoid more update noise in this series.
Quoted 23 lines
mm/vmscan.c +11
-2+ +/* the vast majority of pages sit in the youngest gen */ +return youngest[LRU_GEN_ANON ] + youngest[LRU_GEN_FILE ] > + (old[LRU_GEN_ANON ] + old[LRU_GEN_FILE ]) *MAX_NR_GENS ; +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) {@@ -5087,7 +5123,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* run aging when folios outside the youngest gen are rare */ +return lru_gen_imbalanced (lruvec, sc, swappiness); } static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc, --
Quoted 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 11/16 mm/mglru: Run aging when one type is drained
In reply to Barry Song (Xiaomi)We may have remaining scan budget for anon or file but no folios in populatable generations to scan. In this case, scan_folios() may repeatedly scan that type without making progress, even when the other type still has populatable folios.
Run aging when a type is drained to make more folios available for reclaim. This also helps honor the scan budget distribution determined by swappiness.
Move aging after eviction to detect a lack of scan progress and avoid unnecessary aging when eviction can still make progress.
To avoid excessive aging, allow drained aging only once per try_to_shrink_lruvec() invocation. Also skip drained aging if another reclaimer has advanced max_seq during eviction.
At extreme swappiness values such as 1 and 200, reclaim initially focuses on the preferred type. Rely on imbalanced aging once sc->priority reaches DEF_PRIORITY / 2, rather than triggering drained aging for the unpreferred type. This avoids unnecessary aging and has shown better performance in testing.
If evict_folio_lists() returns zero, retry eviction once if max_seq has advanced since the aging decision.
--- mm/vmscan.c | 49 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 12 deletions(-)
mm/vmscan.c +37 -12
@@ -5175,7 +5175,8 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) }static long evict_folio_lists (unsigned long nr_to_scan[],struct lruvec *lruvec, -struct scan_control *sc,int swappiness,unsigned long max_batch) +struct scan_control *sc,int swappiness, +unsigned long max_batch,bool *drained) {int type;unsigned long batch, scanned, total_scanned =0 ;
@@ -5191,6 +5192,8 @@ static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec, batch =min (nr_to_scan[type], max_batch); scanned =evict_folios (batch, lruvec, sc, type, swappiness); +if (drained && !scanned) + *drained = true; nr_to_scan[type] -=min (nr_to_scan[type], scanned); total_scanned += scanned; }
@@ -5206,9 +5209,15 @@ static long evict_folio_lists(unsigned long nr_to_scan[], struct lruvec *lruvec,static bool try_to_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) {unsigned long nr[ANON_AND_FILE ]; -bool need_rotate = false, should_age = false; +bool need_rotate = false, should_age = false, retried = false;int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec); +/* +* Avoid aging the lagging, unpreferred type at extreme swappiness; +* rely on imbalanced aging once priority reaches AGING_PRIORITY +*/ +bool allow_drained_age = sc->priority <DEF_PRIORITY && + swappiness >MIN_SWAPPINESS +1 && swappiness <MAX_SWAPPINESS ;/* * Proactive reclaim initiated by userspace for anonymous memory only.
@@ -5226,22 +5235,17 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { +bool drained = false, do_drained_age;DEFINE_MAX_SEQ (lruvec); +unsigned long new_seq; +long scanned;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) { need_rotate = true;break ; } -if (should_run_aging (lruvec, max_seq, sc, swappiness)) { -trace_mm_mglru_aging (lruvec, sc, swappiness); -if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) - need_rotate = true; - should_age = true; - } - -if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH )) -break ; + scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH , &drained);if (should_abort_scan (lruvec, sc))break ;
@@ -5253,6 +5257,27 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; +/* +* Allow drained aging only once per invocation and skip it if +* max_seq advanced concurrently during eviction +*/ + new_seq =READ_ONCE (lruvec->lrugen.max_seq); + do_drained_age = drained && allow_drained_age && max_seq == new_seq; + allow_drained_age &= !do_drained_age; +if (do_drained_age ||should_run_aging (lruvec, new_seq, sc, swappiness)) { +trace_mm_mglru_aging (lruvec, sc, swappiness); +if (try_to_inc_max_seq (lruvec, new_seq, swappiness, false)) + need_rotate = true; + should_age = true; + } + +if (!scanned) { +/* Retry once if aging advanced max_seq */ +if (retried || new_seq ==READ_ONCE (lruvec->lrugen.max_seq)) +break ; + retried = true; + } +cond_resched (); }
@@ -5885,7 +5910,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_colru_gen_balance_scan (lruvec, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); -if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH )) +if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ,NULL ))return 0 ;cond_resched ();
Signature · 1 line
2.39.3 (Apple Git-146)
[email protected]
AI reviewRe: [PATCH v9 11/16] mm/mglru: Run aging when one type is drained
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] mm/mglru: scan_folios() scans seq below min_seq at DEF_PRIORITY - [Low] mm/mglru: missed eviction…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] mm/mglru: scan_folios() scans seq below min_seq at DEF_PRIORITY
- [Low] mm/mglru: missed eviction retry in try_to_shrink_lruvec() on concurrent aging
--
commit 230d3708cf744410ec1f04bfd86215ddf2b3c664
Author: Barry Song (Xiaomi) <[email protected]>
mm/mglru: Run aging when one type is drained
This commit moves MGLRU aging after eviction in try_to_shrink_lruvec() and triggers aging once per invocation when an evictable type is drained.
Quoted 4 lines
mm/vmscan.c +0
-0
[ ... ]
Quoted 14 lines
mm/vmscan.c +3
-0@@ -5226,22 +5235,17 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)lru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { +bool drained = false, do_drained_age;DEFINE_MAX_SEQ (lruvec); +unsigned long new_seq; +long scanned;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) { need_rotate = true;break ; }
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5248
Quoted 10 lines
mm/vmscan.c +1
-9-if (should_run_aging (lruvec, max_seq, sc, swappiness)) { -trace_mm_mglru_aging (lruvec, sc, swappiness); -if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) - need_rotate = true; - should_age = true; - } - -if (!evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH )) -break ; + scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH , &drained);
[Severity: Medium]
Can calling evict_folio_lists() before should_run_aging() cause scan_folios() to scan a generation below min_seq[type] at DEF_PRIORITY?
Previously, should_run_aging() ran first and incremented max_seq at DEF_PRIORITY whenever a type had only MIN_NR_GENS generations (min_seq[type] == max_seq - 1). With eviction moved ahead of aging, evict_folio_lists() -> evict_folios() -> scan_folios() now runs while min_seq[type] == max_seq - 1:
mm/vmscan.c:scan_folios() {
...
if (sc->priority == DEF_PRIORITY)
seq = lrugen->max_seq - MIN_NR_GENS;
else if (sc->priority > AGING_PRIORITY)
seq = lrugen->max_seq - 1;
else
seq = lrugen->max_seq;
seq = lruvec_populated_min_seq(lruvec, type, sc->reclaim_idx + 1, seq);
...
}Because seq is max_seq - 2 (< min_seq[type]), the while (min_seq < max_seq) loop in lruvec_populated_min_seq() does not execute and returns max_seq - 2. scan_folios() then scans the out-of-window slot lrugen->folios[lru_gen_from_seq(max_seq - 2)][type][zone_idx], prematurely isolating or promoting folios parked in that slot by memcg reparenting to await future aging, and re-adding rejected folios via move_folios_to_lru() at scanned_seq + 1 == min_seq[type] instead of promoting them above min_seq[type].
Quoted 17 lines
mm/vmscan.c +10
-0if (should_abort_scan (lruvec, sc))break ;@@ -5253,6 +5257,27 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (root_reclaim (sc) && should_age)break ; +/* +* Allow drained aging only once per invocation and skip it if +* max_seq advanced concurrently during eviction +*/ + new_seq =READ_ONCE (lruvec->lrugen.max_seq); + do_drained_age = drained && allow_drained_age && max_seq == new_seq; + allow_drained_age &= !do_drained_age; +if (do_drained_age ||should_run_aging (lruvec, new_seq, sc, swappiness)) { +trace_mm_mglru_aging (lruvec, sc, swappiness); +if (try_to_inc_max_seq (lruvec, new_seq, swappiness, false))
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) · line 5279
Quoted 10 lines
mm/vmscan.c +10
-0+ need_rotate = true; + should_age = true; + } + +if (!scanned) { +/* Retry once if aging advanced max_seq */ +if (retried || new_seq ==READ_ONCE (lruvec->lrugen.max_seq)) +break ; + retried = true; + }
[Severity: Low]
Should the retry check compare READ_ONCE(lruvec->lrugen.max_seq) against max_seq instead of new_seq?
CPU0 CPU1try_to_shrink_lruvec()
max_seq = lrugen.max_seq
evict_folio_lists() -> 0
try_to_inc_max_seq()
inc_max_seq()
lrugen.max_seq++ new_seq = lrugen.max_seq
max_seq == new_seq is false
/* aging skipped */
new_seq == lrugen.max_seq <-- breaks without retrying
When a concurrent reclaimer advances max_seq after scan_folios() finds no folios and before new_seq is read, do_drained_age is skipped because max_seq != new_seq and should_run_aging() is evaluated against the already incremented new_seq. Comparing lrugen.max_seq against new_seq then sees no change and breaks out of try_to_shrink_lruvec() without retrying eviction on the newly aged generation.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Barry Song (Xiaomi)
reviewerRe: [PATCH v9 11/16] mm/mglru: Run aging when one type is drained
Quoted 5 lines · On Sun, Nov 15, 2026 at 4:50 AM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] mm/mglru:scan_folios()scans seq belowmin_seqatDEF_PRIORITY
I guess reparenting could be a corner case where it's difficult to properly preserve the cold/hot ordering.
As mentioned in my earlier reply, I discussed this with Kairui, and we still prefer having lruvec_populated_min_seq() return max_seq instead of min_seq:
https://lore.kernel.org/linux-mm/CAGsJ_4xbDy8Qy=5yqmn+59jfxPm3WOgAH7zX9gwhJyO-6fiT6A@mail.gmail.com/
But for sure I can skip the scanning in scan_folios() if (seq < lrugen->min_seq[type]).
Quoted 3 lines
- [Low] mm/mglru: missed eviction retry in
try_to_shrink_lruvec()on concurrent aging
--
[...]
Quoted 6 lines
When a concurrent reclaimer advances
max_seqafterscan_folios()finds no folios and beforenew_seqis read,do_drained_ageis skipped becausemax_seq!=new_seqandshould_run_aging()is evaluated against the already incrementednew_seq. Comparing lrugen.max_seq againstnew_seqthen sees no change and breaks out oftry_to_shrink_lruvec()without retrying eviction on the newly aged generation.
I actually considered this before sending the patch. I think either approach is fine, and comparing against max_seq instead of new_seq would also be reasonable.
Comparing against new_seq still gives us a chance to retry, since we call try_to_inc_max_seq() here, which may advance lruvec->lrugen.max_seq beyond new_seq. By the time we reach this check, new_seq != READ_ONCE(lruvec->lrugen.max_seq) may already be true.
Overall, I think the following fix should address both comments:
From: "Barry Song (Xiaomi)" <[email protected]>
Date: Sun, 15 Nov 2026 10:24:32 +1300
Subject: [PATCH] Fix-mm-mglru-Run-aging-when-one-type-is-drained.patch
--- mm/vmscan.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/vmscan.c +4 -1
@@ -4788,6 +4788,9 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, seq = lrugen->max_seq; seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); +if (seq < lrugen->min_seq[type]) +return 0 ; + *scanned_seq = seq; gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq);
@@ -5278,7 +5281,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)if (!scanned) {/* Retry once if aging advanced max_seq */ -if (retried || new_seq ==READ_ONCE (lruvec->lrugen.max_seq)) +if (retried || max_seq ==READ_ONCE (lruvec->lrugen.max_seq))break ; retried = true; }
Signature · 1 line
2.34.1
Quoted 3 lines
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Re: [PATCH v9 11/16] mm/mglru: Run aging when one type is drained
Quoted 10 lines · Barry Song (Xiaomi)
Quoted 5 lines · On Sun, Nov 15, 2026 at 4:50 AM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] mm/mglru:scan_folios()scans seq belowmin_seqatDEF_PRIORITYI guess reparenting could be a corner case where it's difficult to properly preserve the cold/hot ordering.
I think we don't need to care about hotness for reparenting, MGLRU's reparent just forcefully squashes the old gen into the parent's gen, and most likely in the wrong window order, the hotness info is already severely skewed; there's no way to cover that until we run the aging. In fact I think we should patch reparent instead, maybe later, squashing everything into the oldest generation seems a more reasonable solution for reparenting?
And if the child's old generation was squashed into an out-of-window generation, these out-of-window cold folios will get evicted much later than the hot folios. In that case we even somehow "fixed" that by this patch :P, as long there is no leak, I think it's hard to define what is the right behavior.
Barry Song (Xiaomi)
reviewerPATCH v9 12/16 mm/mglru: Restrict eviction accounting to eligible zones
In reply to Barry Song (Xiaomi)For both lruvec_evictable_size() and lru_gen_imbalanced(), only count eligible zones up to sc->reclaim_idx. This avoids overestimating the number of pages available for scanning and allows aging to proceed more promptly when eligible zones contain few folios.
lru_gen_age_node() uses lruvec_is_reclaimable() to determine whether any memcg has generations old enough to reclaim under min_ttl. If none are reclaimable, it may trigger OOM. However, lruvec_is_reclaimable() also uses lruvec_is_sizable() to estimate the LRU size.
Previously, lruvec_is_sizable() counted folios across all zones. With accounting now restricted to zones up to sc->reclaim_idx, the estimated size may be significantly smaller, potentially leading to premature OOM.
To avoid this and simplify the implementation, allow lru_gen_age_node() to count folios from all zones.
Note that min_ttl currently has several issues and does not work as expected.
First, reclaim does not enforce min_ttl on a per-memcg basis, so folios from memcgs whose generations are all younger than min_ttl can still be reclaimed.
Second, the LRU size check is coupled with the min_ttl check, which can lead to premature OOM. For example, consider a system with 100 memcgs, each containing fewer than 16 MB of evictable folios. At priority 12, shifting each memcg's LRU size by sc->priority can yield zero, causing all memcgs to be considered unreclaimable even if their generations are older than min_ttl.
These issues should be addressed in a separate patch series rather than as part of this eviction accounting change.
--- mm/trace_vmscan.h | 4 ++-- mm/vmscan.c | 27 +++++++++++++++------------ mm/vmscan.h | 7 ++++--- 3 files changed, 21 insertions(+), 17 deletions(-)
mm/trace_vmscan.h +2 -2
@@ -488,9 +488,9 @@ TRACE_EVENT(mm_mglru_aging,unsigned long seq = max_seq -MAX_NR_GENS +1 + i; __entry->nr_anon[i] = -lruvec_gen_size (lrugen,LRU_GEN_ANON , seq); +lruvec_gen_size (lrugen,LRU_GEN_ANON , seq,MAX_NR_ZONES ); __entry->nr_file[i] = -lruvec_gen_size (lrugen,LRU_GEN_FILE , seq); +lruvec_gen_size (lrugen,LRU_GEN_FILE , seq,MAX_NR_ZONES ); } __entry->nid =lruvec_pgdat (lruvec)->node_id; __entry->priority = sc->priority;
mm/vmscan.c +15 -12
@@ -4207,8 +4207,9 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static void lruvec_evictable_size (struct lruvec *lruvec,int swappiness, -unsigned long nr[]) +static void lruvec_evictable_size (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long nr[], +int max_zone) {int type;unsigned long seq;
@@ -4221,16 +4222,17 @@ static void lruvec_evictable_size(struct lruvec *lruvec, int swappiness,for_each_evictable_type (type, swappiness) {for (seq = min_seq[type]; seq <= max_seq; seq++) - nr[type] +=lruvec_gen_size (lrugen, type, seq); + nr[type] +=lruvec_gen_size (lrugen, type, seq, max_zone); } } -static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) +static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc, +int max_zone) {unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); -lruvec_evictable_size (lruvec, swappiness, sizes); +lruvec_evictable_size (lruvec, sc, swappiness, sizes, max_zone); total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */
@@ -4249,7 +4251,8 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *scif (mem_cgroup_below_min (NULL , memcg))return false; -if (!lruvec_is_sizable (lruvec, sc)) +/* counting all zones to avoid premature OOM */ +if (!lruvec_is_sizable (lruvec, sc,MAX_NR_ZONES ))return false; gen =lru_gen_from_seq (evictable_min_seq (min_seq, swappiness));
@@ -5099,9 +5102,9 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, struct scan_control *sc,for_each_evictable_type (type, swappiness) {unsigned long seq; - youngest[type] =lruvec_gen_size (lrugen, type, max_seq); + youngest[type] =lruvec_gen_size (lrugen, type, max_seq, sc->reclaim_idx +1 );for (seq = min_seq[type]; seq < max_seq; seq++) - old[type] +=lruvec_gen_size (lrugen, type, seq); + old[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); }/* the vast majority of pages sit in the youngest gen */
@@ -5136,7 +5139,7 @@ static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc, {unsigned long total; -lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan, sc->reclaim_idx +1 ); total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; total =apply_proportional_protection (memcg, sc, total);
@@ -5323,7 +5326,7 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc)if (need_rotate)return MEMCG_LRU_YOUNG ; -if (lruvec_is_sizable (lruvec, sc)) +if (lruvec_is_sizable (lruvec, sc, sc->reclaim_idx +1 ))return 0 ;/* one retry if too small */
@@ -5853,7 +5856,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)char mark = full && seq < min_seq[type] ?'x' :' ' ;seq_printf (m," %10lu%c" , -lruvec_gen_size (lrugen, type, seq), mark); +lruvec_gen_size (lrugen, type, seq,MAX_NR_ZONES ), mark); }seq_putc (m,'\n' );
@@ -5906,7 +5909,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coif (sc->nr_reclaimed >= nr_to_reclaim)return 0 ; -lruvec_evictable_size (lruvec, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan, sc->reclaim_idx +1 );lru_gen_balance_scan (lruvec, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);
mm/vmscan.h +4 -3
@@ -117,14 +117,15 @@ struct scan_control { };#ifdef CONFIG_LRU_GEN -/* The number of pages of @type in the generation @seq, summed over all zones */ +/* Number of pages of @type in generation @seq below @max_zone */ static inline unsigned long lruvec_gen_size (struct lru_gen_folio *lrugen, -int type,unsigned long seq) +int type,unsigned long seq, +int max_zone) {int gen =lru_gen_from_seq (seq);unsigned long size =0 ; -for (int zone =0 ; zone <MAX_NR_ZONES ; zone++) +for (int zone =0 ; zone < max_zone; zone++) size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L );return size;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 13/16 mm/mglru: Reduce LRU lock contention by shrinking folio batches in inc_min_seq
In reply to Barry Song (Xiaomi)Reduce the number of folios moved from the oldest generation to the second-oldest generation in each batch. This shortens the time the lruvec lock is held, allowing concurrent LRU operations to make progress and significantly reducing LRU lock contention.
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -3914,7 +3914,7 @@ static inline void flush_lru_batch(struct list_head *head, struct list_head **bastatic bool inc_min_seq (struct lruvec *lruvec,int type,int swappiness) {int zone; -int remaining =MAX_LRU_BATCH ; +int remaining =MIN_LRU_BATCH ;struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]);int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]);
Signature · 1 line
2.39.3 (Apple Git-146)
Re: [PATCH v9 13/16] mm/mglru: Reduce LRU lock contention by shrinking folio batches in inc_min_seq
Quoted 25 lines · Barry Song (Xiaomi)
Reduce the number of folios moved from the oldest generation to the second-oldest generation in each batch. This shortens the time the lruvec lock is held, allowing concurrent LRU operations to make progress and significantly reducing LRU lock contention.
Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmscan.c +1
-2@@ -3914,7 +3914,7 @@ static inline void flush_lru_batch(struct list_head *head, struct list_head **bastatic bool inc_min_seq (struct lruvec *lruvec,int type,int swappiness) {int zone; -int remaining =MAX_LRU_BATCH ; +int remaining =MIN_LRU_BATCH ;struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]);int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); --
Quoted 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 14/16 mm/mglru: Avoid prolonged anon reclaim with swappiness=200
In reply to Barry Song (Xiaomi)From: Gaofei Lyu <[email protected]>
It has been observed that swappiness=200 can significantly slow down reclaim on some machines, with some workloads running 3x slower under MGLRU than under CLRU.
This is because swappiness=200 currently results in file_factor=0. Give file reclaim a small factor at lower priorities, for example, when sc->priority < DEF_PRIORITY / 2.
--- mm/vmscan.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
mm/vmscan.c +10 -5
@@ -4898,8 +4898,8 @@ static int get_tier_idx(struct lruvec *lruvec, int type)* We don't calculate the scan cost. MGLRU uses lazy promotion, which will * cancel the scan budget naturally. */ -static void lru_gen_balance_scan (struct lruvec *lruvec,int swappiness, -unsigned long total_scan,unsigned long nr[]) +static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc, +int swappiness,unsigned long total_scan,unsigned long nr[]) {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor;
@@ -4956,7 +4956,12 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, int swappiness, file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; - file_factor = (u64)(MAX_SWAPPINESS - swappiness) * refault_rate[LRU_GEN_ANON ] * file_weight; +if (swappiness ==MAX_SWAPPINESS ) + file_factor = sc->priority <AGING_PRIORITY ? + refault_rate[LRU_GEN_ANON ] * file_weight :0 ; +else + file_factor = (u64)(MAX_SWAPPINESS - swappiness) * + refault_rate[LRU_GEN_ANON ] * file_weight;/* Cross-multiply the ratio above. */ anon_scan =mul_u64_u64_div_u64 (total_scan, anon_factor, anon_factor + file_factor);
@@ -5144,7 +5149,7 @@ static void lru_gen_prepare_scan(struct lruvec *lruvec, struct scan_control *sc, total =apply_proportional_protection (memcg, sc, total); total >>= sc->priority; -lru_gen_balance_scan (lruvec, swappiness, total, nr_to_scan); +lru_gen_balance_scan (lruvec, sc, swappiness, total, nr_to_scan); }static bool should_abort_scan (struct lruvec *lruvec,struct scan_control *sc)
@@ -5910,7 +5915,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_coreturn 0 ;lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan, sc->reclaim_idx +1 ); -lru_gen_balance_scan (lruvec, swappiness, +lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan);if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ,NULL ))
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 15/16 mm/mglru: Make swappiness=1 file reclaim more aggressive
In reply to Barry Song (Xiaomi)In MGLRU, swappiness=200 favors anon reclaim, while swappiness=1 favors file reclaim. Swappiness=0 allows only file reclaim, while swappiness=201 allows only anon reclaim. Thus, swappiness=1 and swappiness=200 are counterparts for file and anon reclaim, respectively.
Before this patchset, get_type_to_scan() always returned LRU_GEN_ANON for swappiness=200 and LRU_GEN_FILE for swappiness=1.
For swappiness=200, file_factor is set to 0 at higher reclaim priorities. Similarly, for swappiness=1, anon_factor is set to 0 at higher reclaim priorities. This makes the two settings behave similarly, although the thresholds differ slightly based on testing.
This allows file pages to be reclaimed more aggressively and benefits workloads that prioritize file reclaim by using swappiness=1.
--- mm/vmscan.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
mm/vmscan.c +5 -1
@@ -4955,7 +4955,11 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, anon_weight = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ] / size_bias; file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias; - anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight; +if (swappiness ==MIN_SWAPPINESS +1 ) + anon_factor = sc->priority <=AGING_PRIORITY ? + refault_rate[LRU_GEN_FILE ] * anon_weight :0 ; +else + anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight;if (swappiness ==MAX_SWAPPINESS ) file_factor = sc->priority <AGING_PRIORITY ? refault_rate[LRU_GEN_ANON ] * file_weight :0 ;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerPATCH v9 16/16 mm/mglru: Avoid scan anon while swapcache is low for GFP_NOIO allocation
In reply to Barry Song (Xiaomi)From: Bo Zhang <[email protected]>
With GFP_NOIO, we cannot reclaim anon folios unless they are in the swap cache. CLRU skips anon reclaim when the swap cache is very low by commit 74d9aa0d8e98 ("mm: vmscan: avoid anon scanning for GFP_NOIO with low swapcache"). Apply the same logic to MGLRU.
The new scan-budget-based mechanism makes this straightforward to implement. Thanks also to Xiang Liu, who previously attempted to fix this before we had the scan-budget-based model [1].
One thing worth noting is that, for GFP_NOIO allocations, the lruvec size accounted for by lruvec_evictable_size() may be underestimated during lru_gen_age_node(), even though we count all zones for the min_ttl check.
As explained in the commit message, the proper solution is to decouple the size check from min_ttl handling. We leave that improvement to a separate patch series.
--- mm/vmscan.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-)
mm/vmscan.c +10 -17
@@ -272,13 +272,6 @@ static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,if (!sc || (sc->gfp_mask & __GFP_IO))return false; -/* -* FIXME: MGLRU doesn't fully respect can_reclaim_anon_pages() for the -* scanning type, so only apply this to the traditional LRU for now. -*/ -if (lru_gen_enabled ()) -return false; -if (memcg) {struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat);
@@ -4214,6 +4207,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *scint type;unsigned long seq;struct lru_gen_folio *lrugen = &lruvec->lrugen; +struct mem_cgroup *memcg =lruvec_memcg (lruvec); +int nid =lruvec_pgdat (lruvec)->node_id;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec);
@@ -4221,6 +4216,8 @@ static void lruvec_evictable_size(struct lruvec *lruvec, struct scan_control *sc nr[LRU_GEN_FILE ] =0 ;for_each_evictable_type (type, swappiness) { +if (type ==LRU_GEN_ANON && !can_reclaim_anon_pages (memcg, nid, sc)) +continue ;for (seq = min_seq[type]; seq <= max_seq; seq++) nr[type] +=lruvec_gen_size (lrugen, type, seq, max_zone); }
@@ -4903,23 +4900,19 @@ static void lru_gen_balance_scan(struct lruvec *lruvec, struct scan_control *sc, {struct lru_gen_folio *lrugen = &lruvec->lrugen;u64 refault_rate[ANON_AND_FILE ], anon_factor, file_factor; -u64 anon_scan, file_scan, total; +u64 anon_scan, file_scan;u64 anon_weight, file_weight;int size_bias; - total = nr[LRU_GEN_ANON ] + nr[LRU_GEN_FILE ]; -if (!total) -return ; - -if (swappiness <=MIN_SWAPPINESS ) { +if (swappiness <=MIN_SWAPPINESS ) nr[LRU_GEN_ANON ] =0 ; - nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]); -return ; - } -if (swappiness ==SWAPPINESS_ANON_ONLY ) { +if (swappiness ==SWAPPINESS_ANON_ONLY ) nr[LRU_GEN_FILE ] =0 ; + +if (!nr[LRU_GEN_ANON ] || !nr[LRU_GEN_FILE ]) { nr[LRU_GEN_ANON ] =min_t (u64, total_scan, nr[LRU_GEN_ANON ]); + nr[LRU_GEN_FILE ] =min_t (u64, total_scan, nr[LRU_GEN_FILE ]);return ; }
Signature · 1 line
2.39.3 (Apple Git-146)
Andrew Morton
maintainerQuoted 7 lines · On Sun, 15 Nov 2026 04:07:47 +0800 "Barry Song (Xiaomi)" <[email protected]>…
Classic LRU respects swappiness, while MGLRU only weakly honors it when selecting the scan type. The fallback logic can easily override the bias established by
get_type_to_scan(). As a result, swappiness has a much smaller overall impact when changing from 1 to 200, as shown by the data presented later in this cover letter.
Quoted 5 lines
In this patchset, patches 1-8 from Kairui introduce a model that uses refaults to determine nr_to_scan[] for both file and anon in MGLRU.
...
Thanks, I updated mm.git to this version. I included that Sashiko fixup.
The patchset does seem to be changing pretty quickly so I'll continue to hold it in the mm-new branch until things have settled down.
Quoted 8 lines
v9:
* reduce one iteration forlruvec_populated_min_seq();
* letlruvec_is_reclaimable()count all zones, formin_ttl;
* refinelru_gen_imbalanced(),AGING_PRIORITYmacro, roundup/down swappiness; per Kairui;
* letevict_folio_lists()return "drained" per Kairui, thanks!
* one bounded retry for scanned=0 and reduce concurrent aging
for no-populatable(drained) aging;
Here's how v9 altered mm.git:
mm/vmscan.c | 131 ++++++++++++++++++++++++-------------------------- mm/vmscan.h | 2 2 files changed, 66 insertions(+), 67 deletions(-)
mm/vmscan.c renamed from mm/vmscan.c~b +65 -66
@@ -2736,6 +2736,8 @@ static bool should_clear_pmd_young(void)#define get_memcg_gen (seq) ((seq) % MEMCG_NR_GENS)#define get_memcg_bin (bin) ((bin) % MEMCG_NR_BINS) +#define AGING_PRIORITY (DEF_PRIORITY / 2) +static int get_swappiness (struct lruvec *lruvec,struct scan_control *sc) {struct mem_cgroup *memcg =lruvec_memcg (lruvec);
@@ -3995,7 +3997,7 @@ static unsigned long lruvec_populated_milockdep_assert_held (&lruvec->lru_lock); min_seq =READ_ONCE (lrugen->min_seq[type]); -while (min_seq <= max_seq) { +while (min_seq < max_seq) { gen =lru_gen_from_seq (min_seq);for (zone =0 ; zone < zone_limit; zone++) {if (!list_empty (&lrugen->folios[gen][type][zone]))
@@ -4199,7 +4201,8 @@ static void set_initial_priority(struct }static void lruvec_evictable_size (struct lruvec *lruvec,struct scan_control *sc, -int swappiness,unsigned long nr[]) +int swappiness,unsigned long nr[], +int max_zone) {int type;unsigned long seq;
@@ -4216,16 +4219,17 @@ static void lruvec_evictable_size(structif (type ==LRU_GEN_ANON && !can_reclaim_anon_pages (memcg, nid, sc))continue ;for (seq = min_seq[type]; seq <= max_seq; seq++) - nr[type] +=lruvec_gen_size (lrugen, type, seq, sc->reclaim_idx +1 ); + nr[type] +=lruvec_gen_size (lrugen, type, seq, max_zone); } } -static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) +static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc, +int max_zone) {unsigned long total, sizes[ANON_AND_FILE ];int swappiness =get_swappiness (lruvec, sc); -lruvec_evictable_size (lruvec, sc, swappiness, sizes); +lruvec_evictable_size (lruvec, sc, swappiness, sizes, max_zone); total = sizes[LRU_GEN_ANON ] + sizes[LRU_GEN_FILE ];/* whether the size is big enough to be helpful */
@@ -4244,13 +4248,8 @@ static bool lruvec_is_reclaimable(structif (mem_cgroup_below_min (NULL , memcg))return false; -/* -* As in __alloc_pages_may_oom(), avoid triggering OOM for low-zone -* allocations. Also, defer the LRU size check until reclaim reaches -* a very low priority to avoid premature OOM. -*/ -if (sc->priority <=1 && sc->reclaim_idx >=ZONE_NORMAL && - !lruvec_is_sizable (lruvec, sc)) +/* counting all zones to avoid premature OOM */ +if (!lruvec_is_sizable (lruvec, sc,MAX_NR_ZONES ))return false; gen =lru_gen_from_seq (evictable_min_seq (min_seq, swappiness));
@@ -4783,12 +4782,15 @@ static int scan_folios(unsigned long nr_*/ if (sc->priority ==DEF_PRIORITY ) seq = lrugen->max_seq -MIN_NR_GENS ; -else if (sc->priority >DEF_PRIORITY /2 ) +else if (sc->priority >AGING_PRIORITY ) seq = lrugen->max_seq -1 ;else seq = lrugen->max_seq; seq =lruvec_populated_min_seq (lruvec, type, sc->reclaim_idx +1 , seq); +if (seq < lrugen->min_seq[type]) +return 0 ; + *scanned_seq = seq; gen =lru_gen_from_seq (seq); max_gen =lru_gen_from_seq (max_seq);
@@ -4893,8 +4895,8 @@ static int get_tier_idx(struct lruvec *l* r_file / r_anon: refault rate for file / anon. * w_file / w_anon: LRU size weight of file / anon. * -* We don't calculate the scan cost, MGLRU uses lazy promotion, which will -* cancel the scan budget natually. +* We don't calculate the scan cost. MGLRU uses lazy promotion, which will +* cancel the scan budget naturally. */ static void lru_gen_balance_scan (struct lruvec *lruvec,struct scan_control *sc,int swappiness,unsigned long total_scan,unsigned long nr[])
@@ -4950,12 +4952,12 @@ static void lru_gen_balance_scan(struct file_weight = nr[LRU_GEN_FILE ] + nr[LRU_GEN_ANON ] / size_bias;if (swappiness ==MIN_SWAPPINESS +1 ) - anon_factor = sc->priority <=DEF_PRIORITY /2 ? + anon_factor = sc->priority <=AGING_PRIORITY ? refault_rate[LRU_GEN_FILE ] * anon_weight :0 ;else anon_factor = (u64 )swappiness * refault_rate[LRU_GEN_FILE ] * anon_weight;if (swappiness ==MAX_SWAPPINESS ) - file_factor = sc->priority <DEF_PRIORITY /2 ? + file_factor = sc->priority <AGING_PRIORITY ? refault_rate[LRU_GEN_ANON ] * file_weight :0 ;else file_factor = (u64)(MAX_SWAPPINESS - swappiness) *
@@ -5090,9 +5092,18 @@ static bool lru_gen_imbalanced(struct lrDEFINE_MIN_SEQ (lruvec);int type; -if (sc->priority >DEF_PRIORITY /2 ) +if (sc->priority >AGING_PRIORITY )return false; +/* +* Round up or down corner swappiness, to balance the heavy biased +* type better +*/ +if (swappiness ==MIN_SWAPPINESS +1 ) + swappiness =MIN_SWAPPINESS ; +else if (swappiness ==MAX_SWAPPINESS ) + swappiness =MAX_SWAPPINESS +1 ; +for_each_evictable_type (type, swappiness) {unsigned long seq;
@@ -5102,23 +5113,12 @@ static bool lru_gen_imbalanced(struct lr }/* the vast majority of pages sit in the youngest gen */ -if (youngest[LRU_GEN_ANON ] + youngest[LRU_GEN_FILE ] > - (old[LRU_GEN_ANON ] + old[LRU_GEN_FILE ]) *MAX_NR_GENS ) -return true; - -/* Swappiness extremes: the corresponding type matters */ -if (swappiness ==MIN_SWAPPINESS +1 ) -return youngest[LRU_GEN_FILE ] > old[LRU_GEN_FILE ] *MAX_NR_GENS ; - -if (swappiness ==MAX_SWAPPINESS ) -return youngest[LRU_GEN_ANON ] > old[LRU_GEN_ANON ] *MAX_NR_GENS ; - -return false; +return youngest[LRU_GEN_ANON ] + youngest[LRU_GEN_FILE ] > + (old[LRU_GEN_ANON ] + old[LRU_GEN_FILE ]) *MAX_NR_GENS ; }static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq, -struct scan_control *sc,int swappiness, -bool no_populatable) +struct scan_control *sc,int swappiness) {DEFINE_MIN_SEQ (lruvec);
@@ -5130,13 +5130,6 @@ static bool should_run_aging(struct lruvif (sc->priority ==DEF_PRIORITY )return false; -/* -* Run aging if scan budget remains but no folios -* in populatable generations are available to scan. -*/ -if (no_populatable) -return true; -/* better to run aging even though eviction is still possible */ if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq)return true;
@@ -5151,7 +5144,7 @@ static void lru_gen_prepare_scan(struct {unsigned long total; -lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan, sc->reclaim_idx +1 ); total = nr_to_scan[LRU_GEN_ANON ] + nr_to_scan[LRU_GEN_FILE ]; total =apply_proportional_protection (memcg, sc, total);
@@ -5190,7 +5183,8 @@ static bool should_abort_scan(struct lru }static long evict_folio_lists (unsigned long nr_to_scan[],struct lruvec *lruvec, -struct scan_control *sc,int swappiness,unsigned long max_batch) +struct scan_control *sc,int swappiness, +unsigned long max_batch,bool *drained) {int type;unsigned long batch, scanned, total_scanned =0 ;
@@ -5206,6 +5200,8 @@ static long evict_folio_lists(unsigned l batch =min (nr_to_scan[type], max_batch); scanned =evict_folios (batch, lruvec, sc, type, swappiness); +if (drained && !scanned) + *drained = true; nr_to_scan[type] -=min (nr_to_scan[type], scanned); total_scanned += scanned; }
@@ -5221,16 +5217,15 @@ static long evict_folio_lists(unsigned lstatic bool try_to_shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) {unsigned long nr[ANON_AND_FILE ]; -bool need_rotate = false, should_age = false; +bool need_rotate = false, should_age = false, retried = false;int swappiness =get_swappiness (lruvec, sc);struct mem_cgroup *memcg =lruvec_memcg (lruvec);/* -* Avoid aging the lagging type under extreme swappiness, -* as reclaim of the unpreferred type is disabled until -* DEF_PRIORITY / 2. +* Avoid aging the lagging, unpreferred type at extreme swappiness; +* rely on imbalanced aging once priority reaches AGING_PRIORITY */ -bool allow_no_populatable_age = (sc->priority <=DEF_PRIORITY /2 ) || - (swappiness >MIN_SWAPPINESS +1 && swappiness <MAX_SWAPPINESS ); +bool allow_drained_age = sc->priority <DEF_PRIORITY && + swappiness >MIN_SWAPPINESS +1 && swappiness <MAX_SWAPPINESS ;/* * Proactive reclaim initiated by userspace for anonymous memory only.
@@ -5248,9 +5243,9 @@ static bool try_to_shrink_lruvec(structlru_gen_prepare_scan (lruvec, sc, memcg, swappiness, nr);while (nr[LRU_GEN_ANON ] >0 || nr[LRU_GEN_FILE ] >0 ) { -unsigned long orig_file, orig_anon; -unsigned long max_seq; -bool no_populatable; +bool drained = false, do_drained_age; +DEFINE_MAX_SEQ (lruvec); +unsigned long new_seq;long scanned;if (mem_cgroup_below_min (sc->target_mem_cgroup, memcg)) {
@@ -5258,9 +5253,7 @@ static bool try_to_shrink_lruvec(structbreak ; } - orig_anon = nr[LRU_GEN_ANON ]; - orig_file = nr[LRU_GEN_FILE ]; - scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH ); + scanned =evict_folio_lists (nr, lruvec, sc, swappiness,MIN_LRU_BATCH , &drained);if (should_abort_scan (lruvec, sc))break ;
@@ -5272,20 +5265,26 @@ static bool try_to_shrink_lruvec(structif (root_reclaim (sc) && should_age)break ; -/* Limit no_populatable aging to once per try_to_shrink_lruvec() */ - no_populatable = allow_no_populatable_age && - ((nr[LRU_GEN_ANON ] && nr[LRU_GEN_ANON ] == orig_anon) || - (nr[LRU_GEN_FILE ] && nr[LRU_GEN_FILE ] == orig_file)); - allow_no_populatable_age &= !no_populatable; - max_seq =READ_ONCE ((lruvec)->lrugen.max_seq); -if (should_run_aging (lruvec, max_seq, sc, swappiness, no_populatable)) { +/* +* Allow drained aging only once per invocation and skip it if +* max_seq advanced concurrently during eviction +*/ + new_seq =READ_ONCE (lruvec->lrugen.max_seq); + do_drained_age = drained && allow_drained_age && max_seq == new_seq; + allow_drained_age &= !do_drained_age; +if (do_drained_age ||should_run_aging (lruvec, new_seq, sc, swappiness)) {trace_mm_mglru_aging (lruvec, sc, swappiness); -if (try_to_inc_max_seq (lruvec, max_seq, swappiness, false)) +if (try_to_inc_max_seq (lruvec, new_seq, swappiness, false)) need_rotate = true; should_age = true; } -if (!scanned) -break ; + +if (!scanned) { +/* Retry once if aging advanced max_seq */ +if (retried || max_seq ==READ_ONCE (lruvec->lrugen.max_seq)) +break ; + retried = true; + }cond_resched (); }
@@ -5332,7 +5331,7 @@ static int shrink_one(struct lruvec *lruif (need_rotate)return MEMCG_LRU_YOUNG ; -if (lruvec_is_sizable (lruvec, sc)) +if (lruvec_is_sizable (lruvec, sc, sc->reclaim_idx +1 ))return 0 ;/* one retry if too small */
@@ -5915,11 +5914,11 @@ static int run_eviction(struct lruvec *lif (sc->nr_reclaimed >= nr_to_reclaim)return 0 ; -lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan); +lruvec_evictable_size (lruvec, sc, swappiness, nr_to_scan, sc->reclaim_idx +1 );lru_gen_balance_scan (lruvec, sc, swappiness, nr_to_reclaim - sc->nr_reclaimed, nr_to_scan); -if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH )) +if (!evict_folio_lists (nr_to_scan, lruvec, sc, swappiness,MAX_LRU_BATCH ,NULL ))return 0 ;cond_resched ();
mm/vmscan.h renamed from mm/vmscan.h~b +1 -1
@@ -117,7 +117,7 @@ struct scan_control { };#ifdef CONFIG_LRU_GEN -/* The number of pages of @type in the generation @seq, summed over all zones */ +/* Number of pages of @type in generation @seq below @max_zone */ static inline unsigned long lruvec_gen_size (struct lru_gen_folio *lrugen,int type,unsigned long seq,int max_zone)
_
Barry Song
reviewerQuoted 2 lines · On Sun, Nov 15, 2026, 11:09 Andrew Morton <[email protected]> wrote:
On Sun, 15 Nov 2026 04:07:47 +0800 "Barry Song (Xiaomi)" <
Quoted 11 lines · [email protected]> wrote:
Quoted 11 lines
Classic LRU respects swappiness, while MGLRU only weakly honors it when selecting the scan type. The fallback logic can easily override the bias established by
get_type_to_scan(). As a result, swappiness has a much smaller overall impact when changing from 1 to 200, as shown by the data presented later in this cover letter.In this patchset, patches 1-8 from Kairui introduce a model that uses refaults to determine nr_to_scan[] for both file and anon in MGLRU.
Quoted 8 lines
Quoted 2 lines
...
Thanks, I updated mm.git to this version. I included that Sashiko fixup.
The patchset does seem to be changing pretty quickly so I'll continue to hold it in the mm-new branch until things have settled down
Thanks very much, Andrew!
I'll be traveling soon, so I've been working day and night to try to get this settled before I leave.
Sashiko flagged some nasty corner cases, and Kairui also provided some helpful suggestions. So I've made a few additional changes in v9 to address these.
I expect the series to be much more stable from v9 onward.
Quoted 5 lines
Quoted 5 lines
v9:
* reduce one iteration forlruvec_populated_min_seq();
* letlruvec_is_reclaimable()count all zones, formin_ttl;
* refinelru_gen_imbalanced(),AGING_PRIORITYmacro, roundup/down
swappiness;
Quoted 4 lines
Quoted 4 lines
per Kairui;
* letevict_folio_lists()return "drained" per Kairui, thanks!
* one bounded retry for scanned=0 and reduce concurrent aging
Thanks,
Barry