mm: mglru: fix swappiness behavior
Subsystems (3)
Barry Song (Xiaomi)
reviewerv4 cover mm: mglru: fix swappiness behavior
The active/inactive LRU respects swappiness well. Anonymous page scanning and reclamation increase roughly linearly with swappiness, while file page scanning and reclamation decrease accordingly.
For example, with extreme swappiness values such as 1, the active/inactive LRU performs much more file reclaim, while at 200 it performs much more anonymous reclaim with little or even no file reclaim.
In contrast, MGLRU shows little change across different swappiness values, especially at the extremes of 1 and 200.
The data below was collected while building the kernel in a 1 GB MEMCG with -j20 and zRAM configured as swap.
pgsteal_file
Swappiness LRU MGLRU
--------------------------------
1 10788341 760931 36 1101082 526479 71 683084 406540 106 465165 380845 141 292018 361058 176 207810 332329 200 0 349617
pgsteal_anon
Swappiness LRU MGLRU
--------------------------------
1 4305108 2594416 36 2790004 2984411 71 2692223 2965396 106 2844429 3039293 141 3101425 3087067 176 3394419 3110359 200 13210927 3080592
pgscan_file
Swappiness LRU MGLRU
--------------------------------
1 18718146 887478 36 1485019 610943 71 849670 460520 106 561680 433057 141 368688 404839 176 234273 375214 200 0 400190
pgscan_anon
Swappiness LRU MGLRU
--------------------------------
1 29683174 6008050 36 27142699 5960053 71 25804052 6278323 106 26582642 6416301 141 28199275 6349025 176 29360048 6277269 200 62517850 6295609
refault_file
Swappiness LRU MGLRU
--------------------------------
1 10693128 587183 36 1018405 343244 71 602508 247697 106 388580 224246 141 216110 200426 176 127306 175176 200 0 189009
refault_anon
Swappiness LRU MGLRU
--------------------------------
1 593074 540688 36 231609 654679 71 263264 642920 106 286477 640700 141 331410 634729 176 390750 656967 200 4120283 642881
This patchset respects the type selected by get_type_to_scan(), which uses swappiness as its gain. It does so by running aging when the preferred type has few or no reclaimable folios, allowing more folios of that type to become reclaimable.
As we begin to respect swappiness more strictly, inc_min_seq() may be called more frequently. While this function resolves cold/hot inversion issues, it may itself introduce additional cold/hot inversions, which are addressed by this patchset. In addition, we use batching where possible to reduce the overhead of inc_min_seq().
Patches 1–3 improve readability and enhance the exhausted checks in isolate_folios() and scan_folios().
Patch 4 increases the impact of swappiness on type selection.
Patches 5–10 fix potential cold/hot inversions and use batching where possible to improve the performance of inc_min_seq().
Patches 11–15 improve should_run_aging() with additional generation and imbalance checks, particularly when few folios remain in the reclaimable generations.
Patch 16 reduces the batch size when moving folios from the oldest generation to the second-oldest generation in inc_min_seq() to reduce lock contention.
With this patchset, swappiness starts to behave more like it does with the active/inactive LRU.
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10788341 760931 1974207 36 1101082 526479 620248 71 683084 406540 414207 106 465165 380845 365887 141 292018 361058 335876 176 207810 332329 306476 200 0 349617 10569
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 4305108 2594416 3204425 36 2790004 2984411 2895280 71 2692223 2965396 2766936 106 2844429 3039293 3105011 141 3101425 3087067 2989129 176 3394419 3110359 3183651 200 13210927 3080592 13354629
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 18718146 887478 2753870 36 1485019 610943 712969 71 849670 460520 471135 106 561680 433057 419504 141 368688 404839 374698 176 234273 375214 340775 200 0 400190 10880
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 29683174 6008050 5645754 36 27142699 5960053 6056810 71 25804052 6278323 5877296 106 26582642 6416301 6397449 141 28199275 6349025 5951263 176 29360048 6277269 6076458 200 62517850 6295609 21506939
refault_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10693128 587183 1836987 36 1018405 343244 443893 71 602508 247697 245171 106 388580 224246 202345 141 216110 200426 178638 176 127306 175176 141618 200 0 189009 5003
refault_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 593074 540688 800149 36 231609 654679 609397 71 263264 642920 582514 106 286477 640700 653094 141 331410 634729 636040 176 390750 656967 667461 200 4120283 642881 4126464
It could be brought even closer to the active/inactive LRU by ignoring the potential cost of inc_min_seq() and allowing more aggressive aging. For now, however, we aim to balance the benefits of respecting swappiness against the overall system time. Across almost all swappiness levels, the kernel build workload shows that MGLRU with the patch achieves lower system and wall-clock times than the active/inactive LRU.
sys time
Swappiness LRU MGLRU+Patch
---------------------------------------
1 7m42.831s 6m15.160s 36 4m51.991s 3m42.020s 71 4m29.953s 3m39.259s 106 4m34.790s 3m48.321s 141 4m43.026s 3m42.552s 176 4m54.383s 3m46.840s 200 8m11.063s 7m28.910s
wall time
## Swappiness LRU MGLRU+Patch
1 3m08.386s 2m01.631s 36 1m54.989s 1m49.887s 71 1m52.556s 1m48.879s 106 1m51.885s 1m49.942s 141 1m52.143s 1m48.916s 176 1m51.925s 1m48.703s 200 2m02.300s 2m00.487s
RFC v4:
* Clean up isolate_folios() based on Baolin's feedback. Thanks!
* Further improve isolate_folios() and scan_folios(), including handling of exhausted reclaim types.
* Add fixes and performance improvements to inc_min_seq(), and reduce lock contention.
* Improve the handling of imbalanced aging and remaining generation-based aging.
* RFC v3 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v3:
- Added Ridong's patch of improving isolate_folios() readability;
- for lru_gen_imbalanced(), apply some suggestions from Zicheng and Kairui, considering lruvec size and use macro;
- drop "mm: mglru: only fall back when reclaim is running at high priority"
- mitigate aging for balanced swappiess value(near middle 100) as swappiness bias is less a concern for them; but slightly increased aging overhead might not be good.
* RFC v2 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!
Barry Song (Xiaomi) (13):
mm/mglru: improve scan_folios() exhaustion detection mm/mglru: retry the same type once if isolation fails due to races mm/mglru: boost swappiness responsiveness in get_type_to_scan() mm/mglru: batch update lrugen->nr_pages in inc_min_seq() mm/mglru: batch update lrugen->protected in inc_min_seq() mm/mglru: enhance cold/hot inversion handling in inc_min_seq() mm/mglru: exclude folios promoted by aging from protected in
inc_min_seq()
mm/mglru: move folios from oldest gen to second-oldest gen from head
to tail
mm/mglru: batch move folios to the second-oldest gen's LRU mm/mglru: skip gentle reclaim at DEF_PRIORITY for extreme swappiness mm/mglru: run aging if the preferred type has no reclaimable gens mm/mglru: dynamically scale aging threshold in lru_gen_imbalanced() mm/mglru: reduce folios pulled from the oldest gen in inc_min_seq()
Bo Zhang (1):
mm/mglru: remove redundant gens <= MIN_NR_GENS check in
should_run_aging()
Ridong Chen (1):
mm/mglru: improve readability of isolate_folios()
lyugaofei (1):
mm/mglru: run aging when pages are severely imbalanced across gens
mm/vmscan.c | 289 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 233 insertions(+), 56 deletions(-)
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 1/16 mm/mglru: improve readability of isolate_folios()
From: Ridong Chen <[email protected]>
The for_each_evictable_type() loop in isolate_folios() is misleading: it does not actually iterate over each evictable type. Instead, get_type_to_scan() selects the type to scan, while the iterator i merely bounds the number of attempts.
--- mm/vmscan.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-)
mm/vmscan.c +26 -20
@@ -4839,35 +4839,41 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)return positive_ctrl_err (&sp, &pv); } +static inline bool is_single_type_reclaim (int swappiness) +{ +return swappiness ==MIN_SWAPPINESS || + swappiness ==SWAPPINESS_ANON_ONLY ; +} +static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness,struct list_head *list,int *isolated,int *isolate_type,int *isolate_scanned) { -int i; -int total_scanned =0 ; +bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness); +int total_scanned =0 , scanned, tier; -for_each_evictable_type (i, swappiness) { -int scanned; -int tier =get_tier_idx (lruvec, type); +retry: + tier =get_tier_idx (lruvec, type); + scanned =scan_folios (nr_to_scan, lruvec, sc, + type, tier, list, isolated); - scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + total_scanned += scanned; +if (*isolated) { + *isolate_type = type; + *isolate_scanned = scanned; +return total_scanned; + } - total_scanned += scanned; -if (*isolated) { - *isolate_type = type; - *isolate_scanned = scanned; -break ; - } -/* -* If scanned > 0 and isolated == 0, avoid falling back to the -* other type, as this type remains sufficient. Falling back -* too readily can disrupt the positive_ctrl_err() bias. -*/ -if (!scanned) - type = !type; +/* +* We are running out of the current reclaim type. Fall back to +* the other type if allowed. +*/ +if (!scanned && type_fallback_allowed) { + type = !type; + type_fallback_allowed = false; + goto retry; }return total_scanned;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 2/16 mm/mglru: improve scan_folios() exhaustion detection
In reply to Barry Song (Xiaomi)Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when isolation makes no progress") uses scanned == 0 to determine whether scan_folios() has exhausted a reclaim type. However, this is not always sufficient. It is possible for scanned > 0, while the oldest reclaimable generation is exhausted after the first scan_folios() call.
We detect early_stop in scan_folios(). If we stop early for any reason, it means the current reclaim type is not exhausted yet. If early_stop is never reached, it means we have exhausted the current oldest generation without hitting any scanning limit.
Another issue is that if the lruvec has 4 generations, we might have exhausted the oldest generation while the second oldest generation is still reclaimable. In that case, this type is not exhausted yet.
Add an exhausted output argument to scan_folios() so it can explicitly report whether the reclaimable lists for the current type have been exhausted.
--- mm/vmscan.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-)
mm/vmscan.c +22 -6
@@ -4727,7 +4727,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +bool *exhausted) {int i;int gen;
@@ -4738,12 +4739,15 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int skipped =0 ;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +bool early_stop = false;VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) { + *exhausted = true;return 0 ; + } gen =lru_gen_from_seq (lrugen->min_seq[type]);
@@ -4774,8 +4778,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, skipped_zone += delta; } -if (!--remaining ||max (isolated, skipped_zone) >=MIN_LRU_BATCH ) +if (!--remaining ||max (isolated, skipped_zone) >=MIN_LRU_BATCH ) { + early_stop = true;break ; + } }if (skipped_zone) {
@@ -4784,8 +4790,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, skipped += skipped_zone; } -if (!remaining || isolated >=MIN_LRU_BATCH ) +if (!remaining || isolated >=MIN_LRU_BATCH ) { + early_stop = true;break ; + } } item =PGSCAN_KSWAPD +reclaimer_offset (sc);
@@ -4796,6 +4804,13 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); +/* +* If we didn't stop early, all reclaimable folios in the current +* generation have been scanned. We are exhausted if this is the last +* reclaimable generation. +*/ + *exhausted = !early_stop && + lrugen->min_seq[type] +MIN_NR_GENS == lrugen->max_seq; *isolatedp = isolated;return scanned; }
@@ -4853,11 +4868,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier; +bool exhausted; retry: tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, &exhausted); total_scanned += scanned;if (*isolated) {
@@ -4870,7 +4886,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* We are running out of the current reclaim type. Fall back to * the other type if allowed. */ -if (!scanned && type_fallback_allowed) { +if (exhausted && type_fallback_allowed) { type = !type; type_fallback_allowed = false; goto retry;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 3/16 mm/mglru: retry the same type once if isolation fails due to races
In reply to Barry Song (Xiaomi)If we are not exhausted (i.e., there are still folios in the reclaimable generations) but fail to isolate any folios due to promotions, protection, or races, give this type one more chance to avoid going through the outer loop again.
--- mm/vmscan.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/vmscan.c +9 -1
@@ -4868,7 +4868,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier; -bool exhausted; +bool exhausted, tried = false; retry: tier =get_tier_idx (lruvec, type);
@@ -4891,6 +4891,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, type_fallback_allowed = false; goto retry; } +/* +* We are not exhausted, but failed to isolate any folios due to +* races. Give this type one more chance to avoid a larger loop. +*/ +if (!exhausted && !tried) { + tried = true; + goto retry; + }return total_scanned; }
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 4/16 mm/mglru: boost swappiness responsiveness in get_type_to_scan()
In reply to Barry Song (Xiaomi)The linear weight in get_type_to_scan() can easily be overwhelmed by historical refault costs when swappiness deviates from neutral (100). As a result, user-configured swappiness often feels unresponsive under memory pressure.
Apply a smooth quadratic boost based on the distance from the neutral balance point (MAX_SWAPPINESS / 2). This amplifies the weight of the preferred scanning type (anon vs file) while avoiding hardware division via bitwise shift.
--- mm/vmscan.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
mm/vmscan.c +26 -2
@@ -4838,18 +4838,42 @@ static int get_tier_idx(struct lruvec *lruvec, int type)static int get_type_to_scan (struct lruvec *lruvec,int swappiness) {struct ctrl_pos sp, pv = {}; +int anon_gain, file_gain;if (swappiness <=MIN_SWAPPINESS +1 )return LRU_GEN_FILE ;if (swappiness >=MAX_SWAPPINESS )return LRU_GEN_ANON ; + +/* +* Apply a quadratic boost based on the distance from the neutral +* balance point (swappiness = MAX_SWAPPINESS / 2). +* +* A linear weight is easily overwhelmed by historical refault cost +* when swappiness deviates from neutral. The quadratic scaling +* amplifies the weight of the preferred type smoothly. +*/ +if (swappiness <MAX_SWAPPINESS /2 ) { +int delta = (MAX_SWAPPINESS /2 ) - swappiness; +int boost = (delta * delta) >>4 ; + + anon_gain = swappiness; + file_gain = (MAX_SWAPPINESS - swappiness) + boost; + }else { +int delta = swappiness - (MAX_SWAPPINESS /2 ); +int boost = (delta * delta) >>4 ; + + anon_gain = swappiness + boost; + file_gain =MAX_SWAPPINESS - swappiness; + } +/* * Compare the sum of all tiers of anon with that of file to determine * which type to scan. */ -read_ctrl_pos (lruvec,LRU_GEN_ANON ,MAX_NR_TIERS , swappiness, &sp); -read_ctrl_pos (lruvec,LRU_GEN_FILE ,MAX_NR_TIERS ,MAX_SWAPPINESS - swappiness, &pv); +read_ctrl_pos (lruvec,LRU_GEN_ANON ,MAX_NR_TIERS , anon_gain, &sp); +read_ctrl_pos (lruvec,LRU_GEN_FILE ,MAX_NR_TIERS , file_gain, &pv);return positive_ctrl_err (&sp, &pv); }
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 5/16 mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
In reply to Barry Song (Xiaomi)Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.
Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().
--- mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-)
mm/vmscan.c +35 -11
@@ -3295,20 +3295,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma }/* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio) +static int __folio_inc_gen (struct folio *folio,int old_gen,bool *increased) { -int type =folio_is_file_lru (folio); -struct lru_gen_folio *lrugen = &lruvec->lrugen; -int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]);unsigned long new_flags, old_flags =READ_ONCE (folio->flags.f); +int new_gen;VM_WARN_ON_ONCE_FOLIO (!(old_flags &LRU_GEN_MASK ), folio); do { new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; /* folio_update_gen() has promoted this page? */ - if (new_gen >= 0 && new_gen != old_gen) + if (new_gen >= 0 && new_gen != old_gen) { + if (increased) + *increased = false; return new_gen; + } new_gen = (old_gen + 1) % MAX_NR_GENS;
@@ -3316,8 +3317,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) new_flags |= (new_gen +1UL ) <<LRU_GEN_PGOFF ; }while (!try_cmpxchg (&folio->flags.f, &old_flags, new_flags)); -lru_gen_update_size (lruvec, folio, old_gen, new_gen); +if (increased) + *increased = true; +return new_gen; +} +static int folio_inc_gen (struct lruvec *lruvec,struct folio *folio) +{ +int type =folio_is_file_lru (folio); +struct lru_gen_folio *lrugen = &lruvec->lrugen; +int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); +bool gen_increased; + + new_gen =__folio_inc_gen (folio, old_gen, &gen_increased); +if (gen_increased) +lru_gen_update_size (lruvec, folio, old_gen, new_gen);return new_gen; }
@@ -3903,6 +3917,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]);int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]); +int target_gen = (old_gen +1 ) %MAX_NR_GENS ;/* For file type, skip the check if swappiness is anon only */ if (type && (swappiness ==SWAPPINESS_ANON_ONLY ))
@@ -3915,32 +3930,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* prevent cold/hot inversion if the type is evictable */ for (zone =0 ; zone <MAX_NR_ZONES ; zone++) {struct list_head *head = &lrugen->folios[old_gen][type][zone]; +unsigned long delta =0 ;while (!list_empty (head)) {struct folio *folio =lru_to_folio (head); +long nr_pages =folio_nr_pages (folio);int refs =folio_lru_refs (folio);bool workingset =folio_test_workingset (folio); +bool gen_increased;VM_WARN_ON_ONCE_FOLIO (folio_test_unevictable (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_test_active (folio), folio);VM_WARN_ON_ONCE_FOLIO (folio_is_file_lru (folio) != type, folio);VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); - new_gen =folio_inc_gen (lruvec, folio); + new_gen =__folio_inc_gen (folio, old_gen, &gen_increased);list_move_tail (&folio->lru, &lrugen->folios[new_gen][type][zone]); - +if (gen_increased) + delta += nr_pages;/* don't count the workingset being lazily promoted */ if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) {int tier =lru_tier_from_refs (refs, workingset); -int delta =folio_nr_pages (folio);WRITE_ONCE (lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + lrugen->protected[hist][type][tier] + nr_pages); }if (!--remaining) -return false; +break ; } +WRITE_ONCE (lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); +WRITE_ONCE (lrugen->nr_pages[target_gen][type][zone], + lrugen->nr_pages[target_gen][type][zone] + delta); +if (!remaining) +return false; } done:reset_ctrl_pos (lruvec, type, true);
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 6/16 mm/mglru: batch update lrugen->protected in inc_min_seq()
In reply to Barry Song (Xiaomi)Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead.
--- mm/vmscan.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
mm/vmscan.c +5 -3
@@ -3930,7 +3930,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* prevent cold/hot inversion if the type is evictable */ for (zone =0 ; zone <MAX_NR_ZONES ; zone++) {struct list_head *head = &lrugen->folios[old_gen][type][zone]; -unsigned long delta =0 ; +unsigned long protected[MAX_NR_TIERS ] = {}, delta =0 ;while (!list_empty (head)) {struct folio *folio =lru_to_folio (head);
@@ -3952,8 +3952,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) {int tier =lru_tier_from_refs (refs, workingset); -WRITE_ONCE (lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + nr_pages); + protected[tier] += nr_pages; }if (!--remaining)
@@ -3963,6 +3962,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) lrugen->nr_pages[old_gen][type][zone] - delta);WRITE_ONCE (lrugen->nr_pages[target_gen][type][zone], lrugen->nr_pages[target_gen][type][zone] + delta); +for (int tier =0 ; tier <MAX_NR_TIERS ; tier++) +WRITE_ONCE (lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + protected[tier]);if (!remaining)return false; }
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 7/16 mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
In reply to Barry Song (Xiaomi)During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.
--- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
mm/vmscan.c +5 -2
@@ -3945,9 +3945,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); new_gen =__folio_inc_gen (folio, old_gen, &gen_increased); -list_move_tail (&folio->lru, &lrugen->folios[new_gen][type][zone]); -if (gen_increased) +if (gen_increased) { delta += nr_pages; +list_move_tail (&folio->lru, &lrugen->folios[new_gen][type][zone]); + }else { +list_move (&folio->lru, &lrugen->folios[new_gen][type][zone]); + }/* don't count the workingset being lazily promoted */ if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) {int tier =lru_tier_from_refs (refs, workingset);
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 8/16 mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()
In reply to Barry Song (Xiaomi)Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio().
--- mm/vmscan.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
mm/vmscan.c +6 -7
@@ -3948,16 +3948,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)if (gen_increased) { delta += nr_pages;list_move_tail (&folio->lru, &lrugen->folios[new_gen][type][zone]); +/* don't count the workingset being lazily promoted */ +if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) { +int tier =lru_tier_from_refs (refs, workingset); + + protected[tier] += nr_pages; + } }else {list_move (&folio->lru, &lrugen->folios[new_gen][type][zone]); } -/* don't count the workingset being lazily promoted */ -if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) { -int tier =lru_tier_from_refs (refs, workingset); - - protected[tier] += nr_pages; - } -if (!--remaining)break ; }
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 9/16 mm/mglru: move folios from oldest gen to second-oldest gen from head to tail
In reply to Barry Song (Xiaomi)For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.
--- mm/vmscan.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
mm/vmscan.c +17 -2
@@ -190,8 +190,20 @@ struct scan_control { prefetchw(&prev->_field); \ } \ } while (0) +#define prefetchw_next_lru_folio (_folio, _base, _field) \ + do { \ + if ((_folio)->lru.next != _base) { \ + struct folio *next; \ + \ + next = list_entry((_folio)->lru.next, \ + struct folio, lru); \ + prefetchw(&next->_field); \ + } \ + } while (0) +#else #define prefetchw_prev_lru_folio (_folio, _base, _field) do { } while (0) +#define prefetchw_next_lru_folio (_folio, _base, _field) do { } while (0)#endif /*
@@ -3931,9 +3943,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)for (zone =0 ; zone <MAX_NR_ZONES ; zone++) {struct list_head *head = &lrugen->folios[old_gen][type][zone];unsigned long protected[MAX_NR_TIERS ] = {}, delta =0 ; +struct list_head *pos = head->next; -while (!list_empty (head)) { -struct folio *folio =lru_to_folio (head); +while (pos != head) { + struct folio *folio = list_entry(pos, struct folio, lru);long nr_pages =folio_nr_pages (folio);int refs =folio_lru_refs (folio);bool workingset =folio_test_workingset (folio);
@@ -3944,6 +3957,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)VM_WARN_ON_ONCE_FOLIO (folio_is_file_lru (folio) != type, folio);VM_WARN_ON_ONCE_FOLIO (folio_zonenum (folio) != zone, folio); +prefetchw_next_lru_folio (folio, head, flags); + pos = pos->next; new_gen =__folio_inc_gen (folio, old_gen, &gen_increased);if (gen_increased) { delta += nr_pages;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 10/16 mm/mglru: batch move folios to the second-oldest gen's LRU
In reply to Barry Song (Xiaomi)Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type.
--- mm/vmscan.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
mm/vmscan.c +20 -1
@@ -3922,6 +3922,19 @@ static void clear_mm_walk(void)kfree (walk); } +static inline void flush_lru_batch (struct list_head *head,struct list_head **batch_end, +struct list_head *dst) +{ +LIST_HEAD (movable); + +if (!*batch_end) +return ; + +list_cut_position (&movable, head, *batch_end); +list_splice_tail_init (&movable, dst); + *batch_end =NULL ; +} +static bool inc_min_seq (struct lruvec *lruvec,int type,int swappiness) {int zone;
@@ -3941,9 +3954,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)/* prevent cold/hot inversion if the type is evictable */ for (zone =0 ; zone <MAX_NR_ZONES ; zone++) { +struct list_head *target_list = &lrugen->folios[target_gen][type][zone];struct list_head *head = &lrugen->folios[old_gen][type][zone];unsigned long protected[MAX_NR_TIERS ] = {}, delta =0 ;struct list_head *pos = head->next; +struct list_head *batch_end =NULL ;while (pos != head) { struct folio *folio = list_entry(pos, struct folio, lru);
@@ -3962,7 +3977,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) new_gen =__folio_inc_gen (folio, old_gen, &gen_increased);if (gen_increased) { delta += nr_pages; -list_move_tail (&folio->lru, &lrugen->folios[new_gen][type][zone]); + batch_end = &folio->lru; +/* don't count the workingset being lazily promoted */ if (refs + workingset !=BIT (LRU_REFS_WIDTH ) +1 ) {int tier =lru_tier_from_refs (refs, workingset);
@@ -3970,11 +3986,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) protected[tier] += nr_pages; } }else { +flush_lru_batch (head, &batch_end, target_list);list_move (&folio->lru, &lrugen->folios[new_gen][type][zone]); }if (!--remaining)break ; } +flush_lru_batch (head, &batch_end, target_list); +WRITE_ONCE (lrugen->nr_pages[old_gen][type][zone], lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 11/16 mm/mglru: skip gentle reclaim at DEF_PRIORITY for extreme swappiness
In reply to Barry Song (Xiaomi)For extreme swappiness (<= MIN_SWAPPINESS + 1 or >= MAX_SWAPPINESS), skip gentle reclaim.
Falling back too easily, even at DEF_PRIORITY, undermines the effect of extreme swappiness.
This matches the logic in get_type_to_scan(), where we have:
if (swappiness <= MIN_SWAPPINESS + 1)
return LRU_GEN_FILE;
if (swappiness >= MAX_SWAPPINESS)
return LRU_GEN_ANON;
--- mm/vmscan.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
mm/vmscan.c +12 -2
@@ -281,6 +281,13 @@ static inline bool is_exec_file_folio(const struct folio *folio,return vma_flags_test (vma_flags,VMA_EXEC_BIT ) &&folio_is_file_lru (folio); } +/* See get_type_to_scan(): these values always select FILE or ANON */ +static inline bool is_extreme_swappiness (int swappiness) +{ +return swappiness <=MIN_SWAPPINESS +1 || + swappiness >=MAX_SWAPPINESS ; +} +static void set_task_reclaim_state (struct task_struct *task,struct reclaim_state *rs) {
@@ -5094,8 +5101,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true; -/* try to avoid aging, do gentle reclaim at the default priority */ -if (sc->priority ==DEF_PRIORITY ) +/* +* Try to avoid aging by doing gentle reclaim at the default +* priority. Skip gentle reclaim for extreme swappiness. +*/ +if (sc->priority ==DEF_PRIORITY && !is_extreme_swappiness (swappiness))return false;/* better to run aging even though eviction is still possible */
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 12/16 mm/mglru: run aging if the preferred type has no reclaimable gens
In reply to Barry Song (Xiaomi)Respect the type selected by get_type_to_scan(). If there are no reclaimable gens left for that type, run aging.
--- mm/vmscan.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/vmscan.c +5 -0
@@ -5095,12 +5095,17 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) { +int type =get_type_to_scan (lruvec, swappiness);DEFINE_MIN_SEQ (lruvec);/* have to run aging, since eviction is not possible anymore */ if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true; +/* run aging if the preferred type is exhausted */ +if (min_seq[type] +MIN_NR_GENS > max_seq) +return true; +/* * Try to avoid aging by doing gentle reclaim at the default * priority. Skip gentle reclaim for extreme swappiness.
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 13/16 mm/mglru: remove redundant gens <= MIN_NR_GENS check in should_run_aging()
In reply to Barry Song (Xiaomi)From: Bo Zhang <[email protected]>
If the following condition is true,
if (min_seq[type] + MIN_NR_GENS > max_seq)
return true;
then this one must also be true,
if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
return true;
The converse is not true, so the evictable_min_seq() check is redundant and can be removed.
--- mm/vmscan.c | 4 ---- 1 file changed, 4 deletions(-)
mm/vmscan.c +0 -4
@@ -5098,10 +5098,6 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,int type =get_type_to_scan (lruvec, swappiness);DEFINE_MIN_SEQ (lruvec); -/* have to run aging, since eviction is not possible anymore */ -if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq) -return true; -/* run aging if the preferred type is exhausted */ if (min_seq[type] +MIN_NR_GENS > max_seq)return true;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 14/16 mm/mglru: run aging when pages are severely imbalanced across gens
In reply to Barry Song (Xiaomi)From: lyugaofei <[email protected]>
This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
/*
* It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
* of the total number of pages for each generation. A reasonable range
* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
* aging cares about the upper bound of hot pages, while the eviction
* cares about the lower bound of cold pages.
*/
if (young * MIN_NR_GENS > total)
return true;
if (old * (MIN_NR_GENS + 2) < total)
return true;
But with a stricter condition: the younger generations must contain at least MAX_NR_GENS times as many folios as the older generations.
We also consider the cost of inc_min_seq(). If the oldest generation of the other type has fallen significantly behind, pulling those folios from the oldest generation to the second oldest generation can be very expensive. In this case, skip imbalance aging unless extreme swappiness is in use.
--- mm/vmscan.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 7 deletions(-)
mm/vmscan.c +50 -7
@@ -4219,20 +4219,28 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } +static inline unsigned long lruvec_gen_size (struct lru_gen_folio *lrugen, +int type,unsigned long seq) +{ +int gen =lru_gen_from_seq (seq); +unsigned long size =0 ; + +for (int zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); +return size; +} +static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) { -int gen, type, zone; +int type;unsigned long seq, total =0 ;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec);for_each_evictable_type (type, swappiness) { -for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen =lru_gen_from_seq (seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) - total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); - } +for (seq = min_seq[type]; seq <= max_seq; seq++) + total +=lruvec_gen_size (lrugen, type, seq); }return total;
@@ -5092,6 +5100,37 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,unsigned long max_seq, +struct scan_control *sc,int type,int swappiness) +{ +struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long young =0 , old =0 , lag =0 ; +DEFINE_MIN_SEQ (lruvec); + +/* we still have enough generations to reclaim */ +if (min_seq[type] +MIN_NR_GENS < max_seq) +return false; + +/* +* Trigger aging if the preferred type is running low on reclaimable +* folios, provided the generation lag of the other type remains small +* enough that inc_min_seq() introduces negligible overhead +*/ +for (unsigned long seq = min_seq[type]; seq <= max_seq; seq++) { +unsigned long size =lruvec_gen_size (lrugen, type, seq); + +if (seq +MIN_NR_GENS > max_seq) + young += size; +else + old += size; + } +if (min_seq[!type] +MAX_NR_GENS == max_seq +1 ) + lag +=lruvec_gen_size (lrugen, !type, min_seq[!type]); + +return young > old *MAX_NR_GENS && (lag <MAX_LRU_BATCH || + (is_extreme_swappiness (swappiness) && sc->priority >2 )); +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) {
@@ -5110,7 +5149,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* Run aging if the preferred type is severely imbalanced across gens */ +return lru_gen_imbalanced (lruvec, max_seq, sc, type, swappiness); } static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 15/16 mm/mglru: dynamically scale aging threshold in lru_gen_imbalanced()
In reply to Barry Song (Xiaomi)Currently, lru_gen_imbalanced() uses a fixed threshold (MAX_NR_GENS) to determine whether the young-to-old folio ratio warrants aging the preferred LRU type.
This fixed threshold may not scale well with large memory systems. Borrow the adaptive ratio calculation from inactive_is_low() by scaling the threshold with the square root of the memory size in GB.
--- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/vmscan.c +8 -1
@@ -5105,6 +5105,7 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq, {struct lru_gen_folio *lrugen = &lruvec->lrugen;unsigned long young =0 , old =0 , lag =0 ; +unsigned long inactive_ratio, gb;DEFINE_MIN_SEQ (lruvec);/* we still have enough generations to reclaim */
@@ -5127,7 +5128,13 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq,if (min_seq[!type] +MAX_NR_GENS == max_seq +1 ) lag +=lruvec_gen_size (lrugen, !type, min_seq[!type]); -return young > old *MAX_NR_GENS && (lag <MAX_LRU_BATCH || +/* +* Borrow the adaptive ratio from inactive_is_low(), and scale +* it by sqrt(MAX_NR_GENS) to make aging less aggressive +*/ + gb = (young + old) >> (30 -PAGE_SHIFT ); + inactive_ratio = gb ?int_sqrt (10 * gb *MAX_NR_GENS ) :MAX_NR_GENS ; +return young > old * inactive_ratio && (lag <MAX_LRU_BATCH || (is_extreme_swappiness (swappiness) && sc->priority >2 )); }
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v4 16/16 mm/mglru: reduce folios pulled from the oldest gen in inc_min_seq()
In reply to Barry Song (Xiaomi)Pulling folios from the oldest generation to the second-oldest generation with MAX_LRU_BATCH can take a long time, as the spinlock is held throughout the operation and blocks other users.
Reduce the remaining batch size significantly. This reduces the lock hold time, allowing concurrent reclaim, lru_cache_drain, and other operations to make progress.
--- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmscan.c +1 -1
@@ -3945,7 +3945,7 @@ static inline void flush_lru_batch(struct list_head *head, struct list_head **bastatic bool inc_min_seq (struct lruvec *lruvec,int type,int swappiness) {int zone; -int remaining =MAX_LRU_BATCH ; +int remaining =MAX_LRU_BATCH / (is_extreme_swappiness (swappiness) ?2 :8 );struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]);int new_gen, old_gen =lru_gen_from_seq (lrugen->min_seq[type]);
Signature · 1 line
2.34.1