mm: mglru: fix swappiness behavior
Subsystems (4)
Barry Song (Xiaomi)
reviewerv3 cover mm: mglru: fix swappiness behavior
RFC v3:
- Added Ridong's patch of improving isolate_folios() readability;
- for lru_gen_imbalanced(), apply some suggestions from Zicheng and Kairui, considering lruvec size and use macro;
- drop "mm: mglru: only fall back when reclaim is running at high priority"
- mitigate aging for balanced swappiess value(near middle 100) as swappiness bias is less a concern for them; but slightly increased aging overhead might not be good.
* RFC v2 link:
https://lore.kernel.org/linux-mm/[email protected]/
RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!
The active/inactive LRU respects swappiness well. Anonymous page scanning and reclamation increase roughly linearly with swappiness, while file page scanning and reclamation decrease accordingly.
For example, when swappiness reaches 200, both pgsteal_file and pgscan_file drop to zero while building the kernel in a 1 GB memcg. In contrast, MGLRU shows almost no change across different swappiness values.
pgsteal_file
Swappiness LRU MGLRU
--------------------------------
1 10567455 763612 36 990706 480205 71 688170 415848 106 446294 386164 141 286307 359196 176 201733 351686 200 0 330093
pgsteal_anon
Swappiness LRU MGLRU
--------------------------------
1 4410548 2726362 36 2465268 2762859 71 2677908 2885124 106 2737227 2841796 141 2984276 3035015 176 3381338 2938302 200 13116359 3113499
pgscan_file
Swappiness LRU MGLRU
--------------------------------
1 17997223 923094 36 1325674 539571 71 852345 464222 106 538207 464477 141 357253 412277 176 217536 399446 200 0 375902
pgscan_anon
Swappiness LRU MGLRU
--------------------------------
1 31639423 5987136 36 23441521 5753224 71 26067110 6101780 106 25619448 5782919 141 26842088 6234264 176 29200021 5980292 200 62193924 6413125
This patchset respects the type selected by positive_ctrl_err(), which uses swappiness as its gain. It does so by running aging when the preferred type has few or no reclaimable folios, allowing more folios of that type to become reclaimable.
For balanced swappiness values near the middle (for example, 60~140), the swappiness bias is less of a concern. We still avoid aggressive aging and allow fallback to reduce any potential aging overhead.
In other words, this patch strengthens reclaim type preference for extreme swappiness values while preserving the existing MGLRU behavior for balanced swappiness values.
With this patchset, swappiness starts to behave similarly to the active/inactive LRU.
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10567455 763612 1938714 36 990706 480205 427934 71 688170 415848 373450 106 446294 386164 368160 141 286307 359196 334467 176 201733 351686 293716 200 0 330093 25
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 4410548 2726362 3284272 36 2465268 2762859 3315112 71 2677908 2885124 2942357 106 2737227 2841796 2941288 141 2984276 3035015 3345933 176 3381338 2938302 3409531 200 13116359 3113499 14414151
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 17997223 923094 2612649 36 1325674 539571 514506 71 852345 464222 423155 106 538207 464477 436573 141 357253 412277 389644 176 217536 399446 333973 200 0 375902 25
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 31639423 5987136 6191482 36 23441521 5753224 6097925 71 26067110 6101780 6157755 106 25619448 5782919 6203221 141 26842088 6234264 6599502 176 29200021 5980292 6783074 200 62193924 6413125 22814377
Barry Song (Xiaomi) (4):
mm: mglru: prevent min_seq[type] from pointing to an empty generation
mm: mglru: let scan_folios() scan both reclaimable generations
mm: mglru: improve scan_folios() exhaustion detection
mm: mglru: run aging if the preferred type has no folios in
reclaimable gens
Ridong Chen (1):
mm/mglru: improve readability of isolate_folios()
lyugaofei (1):
mm: mglru: run aging when pages are severely imbalanced across gens
include/linux/mmzone.h | 6 +- mm/vmscan.c | 145 +++++++++++++++++++++++++++++++---------- 2 files changed, 112 insertions(+), 39 deletions(-)
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v3 1/6 mm: mglru: prevent min_seq[type] from pointing to an empty generation
In try_to_inc_min_seq(), min_seq[LRU_GEN_ANON] and min_seq[LRU_GEN_FILE] can be adjusted to point to empty generations to keep their gap within one. As a result, scan_folios() may repeatedly scan empty generations because it assumes the min_seq generation still contains folios. Likewise, should_run_aging() has no way to tell that the min_seq generation has already been drained.
This is quite confusing. I observed scan_folios() returning 0 even though the following check in scan_folios() doesn't take effect:
if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
return 0;There is no need to adjust min_seq[], since the reclaim logic already triggers aging when the number of generations reaches MIN_NR_GENS, and reclaim never reduces it below MIN_NR_GENS.
--- include/linux/mmzone.h | 6 ++---- mm/vmscan.c | 10 ---------- 2 files changed, 2 insertions(+), 14 deletions(-)
include/linux/mmzone.h +2 -4
@@ -552,10 +552,8 @@ enum {* The youngest generation number is stored in max_seq for both anon and file * types as they are aged on an equal footing. The oldest generation numbers are * stored in min_seq[] separately for anon and file types so that they can be -* incremented independently. Ideally min_seq[] are kept in sync when both anon -* and file types are evictable. However, to adapt to situations like extreme -* swappiness, they are allowed to be out of sync by at most -* MAX_NR_GENS-MIN_NR_GENS-1. +* incremented independently. For both file and anonymous memory, the minimum +* generation must be at least MIN_NR_GENS. * * The number of pages in each generation is eventually consistent and therefore * can be transiently negative when reset_batch_size() is pending.
mm/vmscan.c +0 -10
@@ -3972,16 +3972,6 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)if (!seq_inc_flag)return ; -/* see the comment on lru_gen_folio */ -if (swappiness && swappiness <=MAX_SWAPPINESS ) { -unsigned long seq = lrugen->max_seq -MIN_NR_GENS ; - -if (min_seq[LRU_GEN_ANON ] > seq && min_seq[LRU_GEN_FILE ] < seq) - min_seq[LRU_GEN_ANON ] = seq; -else if (min_seq[LRU_GEN_FILE ] > seq && min_seq[LRU_GEN_ANON ] < seq) - min_seq[LRU_GEN_FILE ] = seq; - } -for_each_evictable_type (type, swappiness) {if (min_seq[type] <= lrugen->min_seq[type])continue ;
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v3 2/6 mm: mglru: let scan_folios() scan both reclaimable generations
In reply to Barry Song (Xiaomi)When we have four generations, and scan_folios() exhausts the oldest one while the second-oldest still contains reclaimable folios, the current implementation doesn't move on to scan the second-oldest generation. Instead, it returns, leaving that generation with no chance to be scanned at the current sc->priority.
This doesn't seem right. Rather than breaking out and starting a larger next iteration, let's let scan_folios() continue scanning the second-oldest generation directly.
--- mm/vmscan.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
mm/vmscan.c +11 -2
@@ -4718,6 +4718,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int skipped =0 ;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long min_seq = lrugen->min_seq[type];VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list));
@@ -4725,8 +4726,8 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (get_nr_gens (lruvec, type) ==MIN_NR_GENS )return 0 ; - gen =lru_gen_from_seq (lrugen->min_seq[type]); - +next_gen: + gen =lru_gen_from_seq (min_seq);for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;
@@ -4768,6 +4769,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; } +/* +* This generation is exhausted across all zones, but the scan +* target has not been reached yet. Continue with the next +* reclaimable generation. +*/ +if (i ==0 && ++min_seq +MIN_NR_GENS <= lrugen->max_seq) + goto next_gen; + item =PGSCAN_KSWAPD +reclaimer_offset (sc);mod_lruvec_state (lruvec, item, isolated);mod_lruvec_state (lruvec,PGREFILL , sorted);
Signature · 1 line
2.34.1
Barry Song
reviewerRe: [RFC PATCH v3 2/6] mm: mglru: let scan_folios() scan both reclaimable generations
Quoted 40 lines · Barry Song (Xiaomi)
When we have four generations, and
scan_folios()exhausts the oldest one while the second-oldest still contains reclaimable folios, the current implementation doesn't move on to scan the second-oldest generation. Instead, it returns, leaving that generation with no chance to be scanned at the currentsc->priority.This doesn't seem right. Rather than breaking out and starting a larger next iteration, let's let
scan_folios()continue scanning the second-oldest generation directly.Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)mm/vmscan.c +3
-2@@ -4718,6 +4718,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int skipped =0 ;unsigned long remaining = nr_to_scan;struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long min_seq = lrugen->min_seq[type];VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list));@@ -4725,8 +4726,8 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (get_nr_gens (lruvec, type) ==MIN_NR_GENS )return 0 ; - gen =lru_gen_from_seq (lrugen->min_seq[type]); - +next_gen: + gen =lru_gen_from_seq (min_seq);for (i =MAX_NR_ZONES ; i >0 ; i--) {LIST_HEAD (moved);int skipped_zone =0 ;
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4778
Quoted 10 lines
mm/vmscan.c +7
-0@@ -4768,6 +4769,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; } +/* +* This generation is exhausted across all zones, but the scan +* target has not been reached yet. Continue with the next +* reclaimable generation. +*/ +if (i ==0 && ++min_seq +MIN_NR_GENS <= lrugen->max_seq) + goto next_gen;
https://sashiko.dev/#/patchset/20260731083843.37811-1-baohua%40kernel.org
“Does this cause all folios from the next generation to be skipped? When we increment the local min_seq and jump to next_gen, the global lrugen->min_seq[type] remains unchanged. When sort_folio() is called for these newer-generation folios, won't it misidentify them as concurrently promoted?
mm/vmscan.c:sort_folio() {
...
/* promoted */
if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
return true;
}...
}
Because gen now differs from the global lrugen->min_seq[type], won't this unconditionally return true, moving the folio to the head of the list instead of isolating it, and wasting CPU cycles while falsely inflating the sorted metric?”
Good catch. One way to fix it is to update
lrugen->min_seq[type] as well in scan_folios():
/*
* This generation is exhausted across all zones, but the scan
* target has not been reached yet. Continue with the next
* reclaimable generation.
*/
if (!reached_scan_target && ++min_seq + MIN_NR_GENS <= lrugen->max_seq) {
reset_ctrl_pos(lruvec, type, true);
WRITE_ONCE(lrugen->min_seq[type], min_seq);
goto next_gen;
}But this seems overly complicated, so I'd like to drop this patch in RFC v4 and instead rely on the outer loop to retry the remaining reclaimable generations.
The current issue is that when a reclaim type has four generations, scan_folios() may finish scanning the oldest generation while the second-oldest generation still contains reclaimable folios. However, it won't continue scanning that generation in the same reclaim round.
Thanks
Barry
Barry Song (Xiaomi)
reviewerRFC PATCH v3 3/6 mm/mglru: improve readability of isolate_folios()
In reply to Barry Song (Xiaomi)From: Ridong Chen <[email protected]>
The for_each_evictable_type() loop in isolate_folios() is misleading: it does not actually iterate over each evictable type. Instead, get_type_to_scan() selects the type to scan, while the iterator i merely bounds the number of attempts.
--- mm/vmscan.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)
mm/vmscan.c +27 -20
@@ -4828,35 +4828,42 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)return positive_ctrl_err (&sp, &pv); } +static inline bool is_single_type_reclaim (int swappiness) +{ +return swappiness ==MIN_SWAPPINESS || + swappiness ==SWAPPINESS_ANON_ONLY ; +} +static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness,struct list_head *list,int *isolated,int *isolate_type,int *isolate_scanned) { -int i; -int total_scanned =0 ; +bool single_type =is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness); +int total_scanned =0 , scanned, tier; +bool tried = false; -for_each_evictable_type (i, swappiness) { -int scanned; -int tier =get_tier_idx (lruvec, type); +retry: + tier =get_tier_idx (lruvec, type); + scanned =scan_folios (nr_to_scan, lruvec, sc, + type, tier, list, isolated); - scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + total_scanned += scanned; +if (*isolated) { + *isolate_type = type; + *isolate_scanned = scanned; +return total_scanned; + } - total_scanned += scanned; -if (*isolated) { - *isolate_type = type; - *isolate_scanned = scanned; -break ; - } -/* -* If scanned > 0 and isolated == 0, avoid falling back to the -* other type, as this type remains sufficient. Falling back -* too readily can disrupt the positive_ctrl_err() bias. -*/ -if (!scanned) - type = !type; +/* +* We are running out of the current reclaim type. Fall back to +* the other type if allowed. +*/ +if (!tried && !scanned && !single_type) { + type = !type; + tried = true; + goto retry; }return total_scanned;
Signature · 1 line
2.34.1
Baolin Wang
reviewerRe: [RFC PATCH v3 3/6] mm/mglru: improve readability of isolate_folios()
Quoted 8 lines · Barry Song (Xiaomi)
From: Ridong Chen <[email protected]>
The
for_each_evictable_type()loop inisolate_folios()is misleading: it does not actually iterate over each evictable type. Instead,get_type_to_scan()selects the type to scan, while the iteratorimerely bounds the number of attempts.
Quoted 4 lines
Signed-off-by Ridong Chen <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>---
Good cleanup. Some comments below.
Quoted 18 lines
mm/vmscan.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)mm/vmscan.c +6
-0@@ -4828,35 +4828,42 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)return positive_ctrl_err (&sp, &pv); } +static inline bool is_single_type_reclaim (int swappiness) +{ +return swappiness ==MIN_SWAPPINESS || + swappiness ==SWAPPINESS_ANON_ONLY ; +} + static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
mm/vmscan.c static int get_type_to_scan(struct lruvec *lruvec, int swappiness) · line 4845
Quoted 10 lines
mm/vmscan.c +3
-2struct scan_control *sc, int swappiness, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { - int i; -int total_scanned =0 ; + bool single_type = is_single_type_reclaim(swappiness);int type =get_type_to_scan (lruvec, swappiness); +int total_scanned =0 , scanned, tier; +bool tried = false;
The tried' is a bit confusing. How about 'type_fallback_allowed'? Or other more readable variable name?
In addition, the 'single_type' variable can be removed, and just initialize 'type_fallback_allowed' directly:
bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
Quoted 41 lines
mm/vmscan.c +18
-18-for_each_evictable_type (i, swappiness) { -int scanned; -int tier =get_tier_idx (lruvec, type); +retry: + tier =get_tier_idx (lruvec, type); + scanned =scan_folios (nr_to_scan, lruvec, sc, + type, tier, list, isolated); - scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + total_scanned += scanned; +if (*isolated) { + *isolate_type = type; + *isolate_scanned = scanned; +return total_scanned; + } - total_scanned += scanned; -if (*isolated) { - *isolate_type = type; - *isolate_scanned = scanned; -break ; - } -/* -* If scanned > 0 and isolated == 0, avoid falling back to the -* other type, as this type remains sufficient. Falling back -* too readily can disrupt the positive_ctrl_err() bias. -*/ -if (!scanned) - type = !type; +/* +* We are running out of the current reclaim type. Fall back to +* the other type if allowed. +*/ +if (!tried && !scanned && !single_type) { + type = !type; + tried = true; + goto retry; }return total_scanned;
Barry Song
reviewerRe: [RFC PATCH v3 3/6] mm/mglru: improve readability of isolate_folios()
Quoted 51 lines · Baolin Wang
Quoted 8 lines · Barry Song (Xiaomi)
From: Ridong Chen <[email protected]>
The
for_each_evictable_type()loop inisolate_folios()is misleading: it does not actually iterate over each evictable type. Instead,get_type_to_scan()selects the type to scan, while the iteratorimerely bounds the number of attempts.Quoted 4 lines
Signed-off-by Ridong Chen <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>---
Good cleanup. Some comments below.
Quoted 18 lines
mm/vmscan.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)mm/vmscan.c +6
-0@@ -4828,35 +4828,42 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)return positive_ctrl_err (&sp, &pv); } +static inline bool is_single_type_reclaim (int swappiness) +{ +return swappiness ==MIN_SWAPPINESS || + swappiness ==SWAPPINESS_ANON_ONLY ; +} + static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,mm/vmscan.c static int get_type_to_scan(struct lruvec *lruvec, int swappiness) · line 4845
Quoted 10 lines
mm/vmscan.c +3
-2struct scan_control *sc, int swappiness, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { - int i; -int total_scanned =0 ; + bool single_type = is_single_type_reclaim(swappiness);int type =get_type_to_scan (lruvec, swappiness); +int total_scanned =0 , scanned, tier; +bool tried = false;The tried' is a bit confusing. How about '
type_fallback_allowed'? Or other more readable variable name?
Quoted 4 lines
In addition, the '
single_type' variable can be removed, and just initialize 'type_fallback_allowed' directly:bool
type_fallback_allowed= !is_single_type_reclaim(swappiness);
Yep. It seems we can simply use type_fallback_allowed and drop tried entirely:
mm/vmscan.c +0 -0
@@ -4840,10 +4840,9 @@ static int isolate_folios(unsigned long
nr_to_scan, struct lruvec *lruvec,
struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned)
{
- bool single_type = is_single_type_reclaim(swappiness);
+ bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
int type = get_type_to_scan(lruvec, swappiness);
int total_scanned = 0, scanned, tier;- bool tried = false;
retry:
tier = get_tier_idx(lruvec, type);
@@ -4861,9 +4860,9 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
* We are running out of the current reclaim type. Fall back to
* the other type if allowed.
*/
- if (!tried && !scanned && !single_type) {
+ if (type_fallback_allowed && !scanned) {
type = !type;
- tried = true;
+ type_fallback_allowed = false;
goto retry;
}Baolin Wang
reviewerRe: [RFC PATCH v3 3/6] mm/mglru: improve readability of isolate_folios()
Quoted 53 lines · Barry Song
Quoted 51 lines · Baolin Wang
Quoted 8 lines · Barry Song (Xiaomi)
From: Ridong Chen <[email protected]>
The
for_each_evictable_type()loop inisolate_folios()is misleading: it does not actually iterate over each evictable type. Instead,get_type_to_scan()selects the type to scan, while the iteratorimerely bounds the number of attempts.Quoted 4 lines
Signed-off-by Ridong Chen <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>---
Good cleanup. Some comments below.
Quoted 18 lines
mm/vmscan.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)mm/vmscan.c +6
-0@@ -4828,35 +4828,42 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)return positive_ctrl_err (&sp, &pv); } +static inline bool is_single_type_reclaim (int swappiness) +{ +return swappiness ==MIN_SWAPPINESS || + swappiness ==SWAPPINESS_ANON_ONLY ; +} + static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,mm/vmscan.c static int get_type_to_scan(struct lruvec *lruvec, int swappiness) · line 4845
Quoted 10 lines
mm/vmscan.c +3
-2struct scan_control *sc, int swappiness, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { - int i; -int total_scanned =0 ; + bool single_type = is_single_type_reclaim(swappiness);int type =get_type_to_scan (lruvec, swappiness); +int total_scanned =0 , scanned, tier; +bool tried = false;The tried' is a bit confusing. How about '
type_fallback_allowed'? Or other more readable variable name?
Quoted 8 lines
Quoted 4 lines
In addition, the '
single_type' variable can be removed, and just initialize 'type_fallback_allowed' directly:bool
type_fallback_allowed= !is_single_type_reclaim(swappiness);Yep. It seems we can simply use
type_fallback_allowedand drop tried entirely:
Thanks. That's what I meant:)
Quoted 29 lines
mm/vmscan.c +1
-2@@ -4840,10 +4840,9 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { -bool single_type =is_single_type_reclaim (swappiness); +bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier; -bool tried = false; retry:tier = get_tier_idx(lruvec, type);mm/vmscan.c +2
-2@@ -4861,9 +4860,9 @@ static int isolate_folios(unsigned longnr_to_scan, struct lruvec *lruvec, * We are running out of the current reclaim type. Fall back to * the other type if allowed. */ -if (!tried && !scanned && !single_type) { +if (type_fallback_allowed && !scanned) { type = !type; - tried = true; + type_fallback_allowed = false; goto retry; }
Barry Song (Xiaomi)
reviewerRFC PATCH v3 4/6 mm: mglru: improve scan_folios() exhaustion detection
In reply to Barry Song (Xiaomi)Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when isolation makes no progress") uses scanned == 0 to determine whether scan_folios() has exhausted a reclaim type. However, this is not always sufficient. It is possible for scanned > 0, while the oldest reclaimable generation is exhausted after the first scan_folios() call.
Add an exhausted output argument to scan_folios() so it can explicitly report whether the reclaimable lists for the current type have been exhausted.
--- mm/vmscan.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
mm/vmscan.c +13 -5
@@ -4707,7 +4707,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +bool *exhausted) {int i;int gen;
@@ -4723,8 +4724,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) { + *exhausted = true;return 0 ; + } next_gen: gen =lru_gen_from_seq (min_seq);
@@ -4785,6 +4788,11 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); +/* +* This scan exhausted the reclaimable lists before reaching the +* scan target or accumulating enough isolated folios. +*/ + *exhausted = remaining >0 && isolated <MIN_LRU_BATCH ; *isolatedp = isolated;return scanned; }
@@ -4842,12 +4850,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool single_type =is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier; -bool tried = false; +bool tried = false, exhausted; retry: tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, &exhausted); total_scanned += scanned;if (*isolated) {
@@ -4860,7 +4868,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* We are running out of the current reclaim type. Fall back to * the other type if allowed. */ -if (!tried && !scanned && !single_type) { +if (!tried && exhausted && !single_type) { type = !type; tried = true; goto retry;
Signature · 1 line
2.34.1
Barry Song
reviewerRe: [RFC PATCH v3 4/6] mm: mglru: improve scan_folios() exhaustion detection
Quoted 44 lines · Barry Song (Xiaomi)
Commit
16b475d2ac3c("mm/mglru: avoid reclaim type fall back when isolation makes no progress") uses scanned == 0 to determine whetherscan_folios()has exhausted a reclaim type. However, this is not always sufficient. It is possible for scanned > 0, while the oldest reclaimable generation is exhausted after the firstscan_folios()call.Add an exhausted output argument to
scan_folios()so it can explicitly report whether the reclaimable lists for the current type have been exhausted.Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)mm/vmscan.c +5
-2@@ -4707,7 +4707,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +bool *exhausted) {int i;int gen;@@ -4723,8 +4724,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) { + *exhausted = true;return 0 ; + } next_gen: gen =lru_gen_from_seq (min_seq);
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4795
Quoted 8 lines
mm/vmscan.c +5
-0@@ -4785,6 +4788,11 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); +/* +* This scan exhausted the reclaimable lists before reaching the +* scan target or accumulating enough isolated folios. +*/ + *exhausted = remaining >0 && isolated <MIN_LRU_BATCH ;
https://sashiko.dev/#/patchset/20260731083843.37811-1-baohua%40kernel.org
"Does this logic properly account for early loop termination when large folios are skipped?
Looking at the scanning loop earlier in scan_folios():
} else {
list_move(&folio->lru, &moved);
skipped_zone += delta;
}
if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH)
break;If a large folio is skipped, skipped_zone can instantly reach or exceed MIN_LRU_BATCH. This causes the loop to break early while remaining is still greater than zero.
Because isolated would also be less than MIN_LRU_BATCH at this point, could this incorrectly evaluate the exhausted flag to true? This appears like it would signal to isolate_folios() that the current reclaim type is completely empty, triggering a premature fallback to the other reclaim type and breaking the cost-based reclaim balancing logic."
This is a good point. We could break out of a zone once skipped_zone >= MIN_LRU_BATCH, provided the zone still contains folios.
It seems I can fix this by checking whether we exited early after reaching the scan target due to any reason:
mm/vmscan.c +0 -0
@@ -4720,6 +4720,7 @@ static int scan_folios(unsigned long nr_to_scan,
struct lruvec *lruvec,
unsigned long remaining = nr_to_scan;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
unsigned long min_seq = lrugen->min_seq[type];+ bool reached_scan_target = false;
VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
VM_WARN_ON_ONCE(!list_empty(list));
@@ -4758,8 +4759,10 @@ static int scan_folios(unsigned long
nr_to_scan, struct lruvec *lruvec,
skipped_zone += delta;
}- if (!--remaining || max(isolated,skipped_zone) >= MIN_LRU_BATCH)
+ if (!--remaining || max(isolated,skipped_zone) >= MIN_LRU_BATCH) {
+ reached_scan_target = true;
break;
+ }
}
if (skipped_zone) {
@@ -4768,8 +4771,10 @@ static int scan_folios(unsigned long
nr_to_scan, struct lruvec *lruvec,
skipped += skipped_zone;
}- if (!remaining || isolated >= MIN_LRU_BATCH)
+ if (!remaining || isolated >= MIN_LRU_BATCH) {
+ reached_scan_target = true;
break;
+ }
}
/*@@ -4788,11 +4793,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
scanned, skipped, isolated,
type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
- /*
- * This scan exhausted the reclaimable lists before reaching the
- * scan target or accumulating enough isolated folios.
- */
- *exhausted = remaining > 0 && isolated < MIN_LRU_BATCH;
+ *exhausted = !reached_scan_target;
*isolatedp = isolated;
return scanned;}
Barry Song (Xiaomi)
reviewerRFC PATCH v3 5/6 mm: mglru: run aging when pages are severely imbalanced across gens
In reply to Barry Song (Xiaomi)From: lyugaofei <[email protected]>
This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
/*
* It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
* of the total number of pages for each generation. A reasonable range
* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
* aging cares about the upper bound of hot pages, while the eviction
* cares about the lower bound of cold pages.
*/
if (young * MIN_NR_GENS > total)
return true;
if (old * (MIN_NR_GENS + 2) < total)
return true;
But with a stricter condition: the younger generations must contain at least old_ratio times as many folios as the older generations. The old_ratio is derived from the lruvec size, so larger lruvecs use a higher old_ratio. This allows aging to keep folios of the preferred type distributed across the reclaimable generations.
Also, imbalanced aging is only applied to non-balanced swappiness values (for example, < 60 or > 140). For balanced swappiness values near the middle, the swappiness bias is less of a concern, while the slightly increased aging overhead might be unacceptable.
--- mm/vmscan.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)
mm/vmscan.c +58 -1
@@ -4973,9 +4973,62 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static inline bool swappiness_is_balanced (int swappiness) +{ +int range =MAX_SWAPPINESS -MIN_SWAPPINESS ; +int middle =MIN_SWAPPINESS + range /2 ; + +return swappiness >= middle - range /5 && + swappiness <= middle + range /5 ; +} + +static bool lru_gen_imbalanced (struct lruvec *lruvec,int type, +unsigned long max_seq,unsigned long min_seq, +struct scan_control *sc,int swappiness) +{ +struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long young =0 , old =0 , seq; +unsigned long old_ratio, gb; + +/* Skip swappiness bias for single-type reclaim or balanced swappiness */ +if (is_single_type_reclaim (swappiness) ||swappiness_is_balanced (swappiness)) +return false; + +/* More than two generations remain to reclaim */ +if (min_seq +MIN_NR_GENS < max_seq) +return false; + +/* +* Run aging if the only remaining reclaimable generation +* has few folios. +*/ +for (seq = min_seq; seq <= max_seq; seq++) { +int gen =lru_gen_from_seq (seq); +unsigned long size =0 ; +int zone; + +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); + +if (seq +MIN_NR_GENS > max_seq) + young += size; +else + old += size; + } +/* +* Copied from inactive_is_low(), but uses a higher old_ratio to +* make aging less aggressive. +*/ + gb = (young + old) >> (30 -PAGE_SHIFT ); + old_ratio = gb ?int_sqrt (10 * gb) :1 ; + old_ratio *=MAX_NR_GENS ; +return young > old * old_ratio; +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) { +int type =get_type_to_scan (lruvec, swappiness);DEFINE_MIN_SEQ (lruvec);/* have to run aging, since eviction is not possible anymore */
@@ -4987,7 +5040,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* Run aging if the preferred type is severely imbalanced across gens */ +return lru_gen_imbalanced (lruvec, type, max_seq, min_seq[type], sc, swappiness); } static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
Signature · 1 line
2.34.1
Barry Song (Xiaomi)
reviewerRFC PATCH v3 6/6 mm: mglru: run aging if the preferred type has no folios in reclaimable gens
In reply to Barry Song (Xiaomi)Respect the type selected by positive_ctrl_err(). If there are no reclaimable generations left for that type, run aging.
For balanced swappiness values near the middle, still allow gentle reclaim since the swappiness bias is less of a concern, while the slightly increased aging overhead might become a problem.
--- mm/vmscan.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/vmscan.c +4 -0
@@ -5035,6 +5035,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true; +/* run aging if the preferred type is exhausted for extreme swappiness */ +if (min_seq[type] +MIN_NR_GENS > max_seq && !swappiness_is_balanced (swappiness)) +return true; +/* try to avoid aging, do gentle reclaim at the default priority */ if (sc->priority ==DEF_PRIORITY )return false;
Signature · 1 line
2.34.1