mm: mglru: fix swappiness behavior
Subsystems (4)
Barry Song (Xiaomi)
reviewerv2 cover mm: mglru: fix swappiness behavior
RFC v2:
- Quickly address a few issues raised in the Sashiko comments so reviewers can ignore v1 and review a cleaner version instead.
https://sashiko.dev/#/patchset/[email protected]
Thanks, Sashiko!
The active/inactive LRU respects swappiness well. Anonymous page scanning and reclamation increase roughly linearly with swappiness, while file page scanning and reclamation decrease accordingly.
For example, when swappiness reaches 200, both pgsteal_file and pgscan_file drop to zero while building the kernel in a 1 GB memcg. In contrast, MGLRU shows almost no change across different swappiness values.
pgsteal_file
Swappiness LRU MGLRU
--------------------------------
1 10567455 763612 36 990706 480205 71 688170 415848 106 446294 386164 141 286307 359196 176 201733 351686 200 0 330093
pgsteal_anon
Swappiness LRU MGLRU
--------------------------------
1 4410548 2726362 36 2465268 2762859 71 2677908 2885124 106 2737227 2841796 141 2984276 3035015 176 3381338 2938302 200 13116359 3113499
pgscan_file
Swappiness LRU MGLRU
--------------------------------
1 17997223 923094 36 1325674 539571 71 852345 464222 106 538207 464477 141 357253 412277 176 217536 399446 200 0 375902
pgscan_anon
Swappiness LRU MGLRU
--------------------------------
1 31639423 5987136 36 23441521 5753224 71 26067110 6101780 106 25619448 5782919 141 26842088 6234264 176 29200021 5980292 200 62193924 6413125
This patchset respects the type selected by positive_ctrl_err(), which uses swappiness as its gain. It does so by:
1. Avoiding premature fallback to the other type. Only fall back when reclaim is running at high priority.
2. Running aging when the preferred type has few or no reclaimable folios, so more folios of that type become reclaimable.
With this patchset, swappiness starts to behave similarly to the active/inactive LRU.
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10567455 763612 2276977 36 990706 480205 485771 71 688170 415848 425893 106 446294 386164 391528 141 286307 359196 371844 176 201733 351686 322634 200 0 330093 1343
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 4410548 2726362 2212243 36 2465268 2762859 3036395 71 2677908 2885124 3111459 106 2737227 2841796 3021429 141 2984276 3035015 3341110 176 3381338 2938302 3232294 200 13116359 3113499 14144584
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 17997223 923094 3243420 36 1325674 539571 595233 71 852345 464222 523441 106 538207 464477 457391 141 357253 412277 434981 176 217536 399446 369302 200 0 375902 1344
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 31639423 5987136 4136386 36 23441521 5753224 5489636 71 26067110 6101780 5699571 106 25619448 5782919 5910594 141 26842088 6234264 6511215 176 29200021 5980292 6286023 200 62193924 6413125 22582174
Another possible approach is to decouple anonymous and file-backed aging by maintaining separate max_seq values for each type. This allows anonymous and file-backed memory to age and be reclaimed independently, enabling the swappiness-preferred type to be reclaimed more aggressively while allowing the other type to lag behind. This approach has already been adopted by projects such as CachyOS [1] and Chromium [2].
However, this approach requires substantial changes to MGLRU and fundamentally alters its design by breaking the shared aging timeline between anonymous and file-backed memory. This timeline is the foundation for mechanisms such as the PID controller and the min_ttl_ms thrashing protection.
That is why this patchset aims to fix the swappiness behavior without fundamentally changing MGLRU's design, with minimal changes.
[1] https://github.com/firelzrd/re-swappiness
[2] https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/929932351492d01f0aee37a0ac3be8c7bd88f80d
Barry Song (Xiaomi) (4):
mm: mglru: avoid scanning empty generations in scan_folios()
mm: mglru: only fall back when reclaim is running at high priority
mm: mglru: prevent min_seq[type] from pointing to an empty generation
mm: mglru: run aging if the preferred type has no folios in
reclaimable gens
lyugaofei (1):
mm: mglru: run aging when pages are severely imbalanced across gens
include/linux/mmzone.h | 6 ++-- mm/vmscan.c | 66 ++++++++++++++++++++++++++++++++---------- 2 files changed, 53 insertions(+), 19 deletions(-)
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerRFC PATCH v2 1/5 mm: mglru: avoid scanning empty generations in scan_folios()
Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when isolation makes no progress") only falls back to the other type when scanned == 0. However, I have frequently observed cases where scanned > 0, but the older reclaimable generation becomes empty after the first scan_folios(). As a result, the second scan_folios() for the same type performs a redundant scan over an empty generation.
We can avoid this by checking whether the reclaimable generation has become empty when scanned < nr_to_scan and we still have fewer than MIN_LRU_BATCH isolated folios after scan_folios().
--- mm/vmscan.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
mm/vmscan.c +7 -4
@@ -4852,11 +4852,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; }/* -* If scanned > 0 and isolated == 0, avoid falling back to the -* other type, as this type remains sufficient. Falling back -* too readily can disrupt the positive_ctrl_err() bias. +* If scanned >= nr_to_scan or isolated >= MIN_LRU_BATCH, +* avoid falling back to the other type. The preferred +* type is still reclaimable; otherwise, it would have +* already run out of reclaimable generations. Falling +* back too readily can disrupt the positive_ctrl_err() +* bias. */ -if (!scanned) +if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) type = !type; }
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song
reviewerRe: [RFC PATCH v2 1/5] mm: mglru: avoid scanning empty generations in scan_folios()
Quoted 28 lines · Barry Song (Xiaomi)
Commit
16b475d2ac3c("mm/mglru: avoid reclaim type fall back when isolation makes no progress") only falls back to the other type when scanned == 0. However, I have frequently observed cases where scanned > 0, but the older reclaimable generation becomes empty after the firstscan_folios(). As a result, the secondscan_folios()for the same type performs a redundant scan over an empty generation.We can avoid this by checking whether the reclaimable generation has become empty when scanned <
nr_to_scanand we still have fewer thanMIN_LRU_BATCHisolated folios afterscan_folios().Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)mm/vmscan.c +0
-2@@ -4852,11 +4852,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; }/* -* If scanned > 0 and isolated == 0, avoid falling back to the -* other type, as this type remains sufficient. Falling back
mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4862
Quoted 10 lines
mm/vmscan.c +7
-2-* too readily can disrupt the positive_ctrl_err() bias. +* If scanned >= nr_to_scan or isolated >= MIN_LRU_BATCH, +* avoid falling back to the other type. The preferred +* type is still reclaimable; otherwise, it would have +* already run out of reclaimable generations. Falling +* back too readily can disrupt the positive_ctrl_err() +* bias. */ -if (!scanned) +if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH )
https://sashiko.dev/#/patchset/20260726122123.7614-1-baohua%40kernel.org
"
mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4864
Quoted 4 lines
mm/vmscan.c +1
-1-if (!scanned) +if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) type = !type; }
Does the check for *isolated < MIN_LRU_BATCH have any effect here? Looking earlier in isolate_folios(), there is an if (*isolated) check that breaks out of the loop:
if (*isolated)
break;
Since execution only reaches this point if *isolated is exactly 0, isn't *isolated < MIN_LRU_BATCH unconditionally true?
Additionally, will this logic work correctly when large folios are involved? If scan_folios() encounters an un-isolatable large folio (for example, an order-9 THP with 512 pages), the scanned variable is incremented by the base page count (512).
Since nr_to_scan is capped at MIN_LRU_BATCH (64) by the caller evict_folios(), scanned < nr_to_scan (512 < 64) will evaluate to false even if the generation is completely empty.
Does this prevent the fallback and inadvertently defeat the optimization on systems utilizing large folios?
"
Right. What I actually need is a way to tell when scan_folios() returns after the current generation's LRU lists are empty, but I haven't found a good approach. Checking scanned < nr_to_scan works well for small folios, but not for large folios.
Maybe we could add an argument to scan_folios(), like this:
@@ -4707,7 +4711,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
struct scan_control *sc, int type, int tier,
- struct list_head *list, int *isolatedp)
+ struct list_head *list, int *isolatedp,
+ bool *exhausted)
{
int i;
int gen;@@ -4776,6 +4781,11 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
scanned, skipped, isolated,
type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
+ /*
+ * This scan exhausted the current generation before reaching the
+ * scan target or accumulating enough isolated folios.
+ */
+ *exhausted = remaining > 0 && isolated < MIN_LRU_BATCH;
*isolatedp = isolated;
return scanned;}
Then another question comes up: suppose we have four generations, and scan_folios() exhausts the oldest one while the second-oldest still contains reclaimable folios. The current implementation doesn't move on to scan the second-oldest generation. Instead, it returns, leaving that generation with no chance to be scanned at the current sc->priority. That doesn't seem right. Let me investigate this as well.
Thanks
Barry
Re: [RFC PATCH v2 1/5] mm: mglru: avoid scanning empty generations in scan_folios()
Hi Barry,
I tested a list-based exhaustion check on top of the v2 series.
Using remaining > 0 still misses the case where remaining reaches zero exactly when the last folio leaves the oldest generation. Since scan_folios() still holds the lruvec lock, checking the generation's zone lists directly avoids both this boundary and the scanned/nr_to_scan page-versus-folio mismatch.
The change below only replaces the exhaustion check. It keeps the priority-gated fallback unchanged, and reports the MIN_NR_GENS early return as exhausted to preserve the current v2 behavior.
mm/vmscan.c +21 -11
@@ -4635,7 +4635,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +bool *scan_exhausted) {int i;int gen;
@@ -4650,8 +4651,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) { + *scan_exhausted = true;return 0 ; + } gen =lru_gen_from_seq (lrugen->min_seq[type]);
@@ -4704,6 +4707,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); + *scan_exhausted = true; +for (i =0 ; i <MAX_NR_ZONES ; i++) { +if (!list_empty (&lrugen->folios[gen][type][i])) { + *scan_exhausted = false; +break ; + } + } + *isolatedp = isolated;return scanned; }
@@ -4757,11 +4768,13 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int type =get_type_to_scan (lruvec, swappiness);for_each_evictable_type (i, swappiness) { +bool scan_exhausted;int scanned;int tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, + &scan_exhausted); total_scanned += scanned;if (*isolated) {
@@ -4770,15 +4783,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; }/* -* If scanned >= nr_to_scan or isolated >= MIN_LRU_BATCH, -* avoid falling back to the other type. The preferred -* type is still reclaimable; otherwise, it would have -* already run out of reclaimable generations. Falling -* back too readily can disrupt the positive_ctrl_err() -* bias. Also, only fall back when reclaim is running at -* a high priority. +* The preferred type is still reclaimable unless this scan +* target is exhausted. Preserve the existing priority-gated +* fallback without inferring exhaustion from scanned, which +* counts base pages rather than iterations. */ -if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) { +if (scan_exhausted) {if (sc->priority >2 )break ; type = !type;
I built and booted this on an arm64 VM with MGLRU enabled. A fresh swappiness=200 pressure smoke with a 2 GiB file-cache setup and about 7 GiB of touched anonymous memory completed without a WARN/Oops/BUG/panic.
Would this be worth considering for the next revision? If I misunderstood the intended semantics, please ignore this.
Thanks,
Lian
Barry Song
reviewerRe: [RFC PATCH v2 1/5] mm: mglru: avoid scanning empty generations in scan_folios()
Quoted 42 lines · Lian Wang
Hi Barry,
I tested a list-based exhaustion check on top of the v2 series.
Using remaining > 0 still misses the case where remaining reaches zero exactly when the last folio leaves the oldest generation. Since
scan_folios()still holds the lruvec lock, checking the generation's zone lists directly avoids both this boundary and the scanned/nr_to_scan page-versus-folio mismatch.The change below only replaces the exhaustion check. It keeps the priority-gated fallback unchanged, and reports the
MIN_NR_GENSearly return as exhausted to preserve the current v2 behavior.mm/vmscan.c +5
-2@@ -4635,7 +4635,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list,int *isolatedp) +struct list_head *list,int *isolatedp, +bool *scan_exhausted) {int i;int gen;@@ -4650,8 +4651,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,VM_WARN_ON_ONCE (nr_to_scan >MAX_LRU_BATCH );VM_WARN_ON_ONCE (!list_empty (list)); -if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) +if (get_nr_gens (lruvec, type) ==MIN_NR_GENS ) { + *scan_exhausted = true;return 0 ; + } gen =lru_gen_from_seq (lrugen->min_seq[type]);
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4716
Quoted 10 lines
mm/vmscan.c +7
-0@@ -4704,6 +4707,14 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); + *scan_exhausted = true; +for (i =0 ; i <MAX_NR_ZONES ; i++) { +if (!list_empty (&lrugen->folios[gen][type][i])) { + *scan_exhausted = false; +break ; + } + }
Thanks very much. Your understanding is correct. We do have the case where the last folio leaves the list while all the other scan targets have already been satisfied. My understanding is that this should be a very low-probability case, so it may not be worth the added code complexity?
Quoted 41 lines
mm/vmscan.c +9
-9+ *isolatedp = isolated;return scanned; }@@ -4757,11 +4768,13 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int type =get_type_to_scan (lruvec, swappiness);for_each_evictable_type (i, swappiness) { +bool scan_exhausted;int scanned;int tier =get_tier_idx (lruvec, type); scanned =scan_folios (nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, + &scan_exhausted); total_scanned += scanned;if (*isolated) {@@ -4770,15 +4783,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,break ; }/* -* If scanned >= nr_to_scan or isolated >= MIN_LRU_BATCH, -* avoid falling back to the other type. The preferred -* type is still reclaimable; otherwise, it would have -* already run out of reclaimable generations. Falling -* back too readily can disrupt the positive_ctrl_err() -* bias. Also, only fall back when reclaim is running at -* a high priority. +* The preferred type is still reclaimable unless this scan +* target is exhausted. Preserve the existing priority-gated +* fallback without inferring exhaustion from scanned, which +* counts base pages rather than iterations. */ -if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) { +if (scan_exhausted) {if (sc->priority >2 )break ; type = !type;
Quoted 4 lines
I built and booted this on an arm64 VM with MGLRU enabled. A fresh swappiness=200 pressure smoke with a 2 GiB file-cache setup and about 7 GiB of touched anonymous memory completed without a WARN/Oops/BUG/panic.
Thanks! I'm going to send out RFC v3. I'd really appreciate any further testing you can do.
Quoted 3 lines
Would this be worth considering for the next revision? If I misunderstood the intended semantics, please ignore this.
Thanks! Your understanding of the corner case is 100% correct. It's just that, given how unlikely it is to occur, I don't think it would have any noticeable impact, so it may not be worth the added complexity.
Best Regards
Barry
Re: [RFC PATCH v2 1/5] mm: mglru: avoid scanning empty generations in scan_folios()
Hi Barry,
That makes sense. I agree that this rare corner case does not justify the added complexity for now, so please feel free to leave it out of v3.
I'll test v3, with a particular focus on the generation-imbalance aging behavior in 4/5, and report back with the results.
Thanks,
Lian
Barry Song (Xiaomi)
reviewerRFC PATCH v2 2/5 mm: mglru: only fall back when reclaim is running at high priority
In reply to Barry Song (Xiaomi)Respect the type selected by positive_ctrl_err(), where swappiness controls the relative weights of SP and PV. Falling back too readily undermines that bias. Only fall back after making a sufficient effort to reclaim from the preferred type.
--- mm/vmscan.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/vmscan.c +6 -2
@@ -4857,10 +4857,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* type is still reclaimable; otherwise, it would have * already run out of reclaimable generations. Falling * back too readily can disrupt the positive_ctrl_err() -* bias. +* bias. Also, only fall back when reclaim is running at +* a high priority. */ -if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) +if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) { +if (sc->priority >2 ) +break ; type = !type; + } }return total_scanned;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song
reviewerRe: [RFC PATCH v2 2/5] mm: mglru: only fall back when reclaim is running at high priority
Quoted 19 lines · Barry Song (Xiaomi)
Respect the type selected by
positive_ctrl_err(), where swappiness controls the relative weights of SP and PV. Falling back too readily undermines that bias. Only fall back after making a sufficient effort to reclaim from the preferred type.Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)mm/vmscan.c +0
-0@@ -4857,10 +4857,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,* type is still reclaimable; otherwise, it would have * already run out of reclaimable generations. Falling * back too readily can disrupt the positive_ctrl_err()
mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4867
Quoted 10 lines
mm/vmscan.c +6
-2-* bias. +* bias. Also, only fall back when reclaim is running at +* a high priority. */ -if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) +if (scanned < nr_to_scan && *isolated <MIN_LRU_BATCH ) { +if (sc->priority >2 ) +break ; type = !type; + }
https://sashiko.dev/#/patchset/20260726122123.7614-1-baohua%40kernel.org
"> diff --git a/mm/vmscan.c b/mm/vmscan.c
Quoted 3 lines
mm/vmscan.c +0
-0@@ -4867,10 +4867,14 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
[ ... ]
Quoted 4 lines
- if (scanned <
nr_to_scan&& *isolated <MIN_LRU_BATCH)
+ if (scanned <nr_to_scan&& *isolated <MIN_LRU_BATCH) {
+ if (sc->priority> 2)
+ break;
Does this logic artificially drop the scan priority and cause false memory pressure events when the preferred LRU type is empty?
By breaking the isolation loop instead of falling back to the populated LRU list, the direct reclaim path might falsely conclude that no pages can be isolated at the current priority. This forces the VM to artificially drop the scanning priority from the default priority down to 2.
This rapid priority drop could have two severe consequences. First, it aggressively invokes shrink_slab() at artificially high priorities, causing unnecessary eviction of active slab objects (like dentries and inodes) which results in extreme latency spikes.
Second, when sc->priority drops to 3, vmpressure_prio() triggers a critical memory pressure event. Userspace daemons (like lmkd or oomd) might respond by killing background apps, falsely assuming the system is completely out of memory, even though the non-preferred LRU list could be full of clean, evictable pages.
Quoted 5 lines
type = !type;
+ }
}return
total_scanned;"
I can change the priority to "DEF_PRIORITY - 2":
mm/vmscan.c +0 -0
@@ -4851,7 +4851,7 @@ static int isolate_folios(unsigned long
nr_to_scan, struct lruvec *lruvec,
* a high priority.
*/
if (scanned < nr_to_scan && *isolated < MIN_LRU_BATCH) {
- if (sc->priority > 2)
+ if (sc->priority > DEF_PRIORITY - 2)
break;
type = !type;
}And the result is still fine:
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10567455 763612 2191652 36 990706 480205 414483 71 688170 415848 386966 106 446294 386164 354976 141 286307 359196 348493 176 201733 351686 297113 200 0 330093 2544
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 4410548 2726362 3511241 36 2465268 2762859 3027445 71 2677908 2885124 3359352 106 2737227 2841796 3224762 141 2984276 3035015 3221214 176 3381338 2938302 3446206 200 13116359 3113499 14700981
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 17997223 923094 3178493 36 1325674 539571 517029 71 852345 464222 460869 106 538207 464477 411314 141 357253 412277 402429 176 217536 399446 333375 200 0 375902 3083
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 31639423 5987136 6184980 36 23441521 5753224 5727338 71 26067110 6101780 5932826 106 25619448 5782919 6105922 141 26842088 6234264 6147981 176 29200021 5980292 6707570 200 62193924 6413125 23288983
If I drop this patch entirely, swappiness still functions:
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 10567455 763612 2118900 36 990706 480205 426026 71 688170 415848 396647 106 446294 386164 376780 141 286307 359196 334243 176 201733 351686 287544 200 0 330093 309
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 4410548 2726362 3579570 36 2465268 2762859 3300899 71 2677908 2885124 3183903 106 2737227 2841796 3419872 141 2984276 3035015 3121920 176 3381338 2938302 3198154 200 13116359 3113499 14348490
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 17997223 923094 2898754 36 1325674 539571 509825 71 852345 464222 470746 106 538207 464477 434145 141 357253 412277 378242 176 217536 399446 322824 200 0 375902 694
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1 31639423 5987136 6594803 36 23441521 5753224 5979158 71 26067110 6101780 6200677 106 25619448 5782919 6480037 141 26842088 6234264 6106924 176 29200021 5980292 6553534 200 62193924 6413125 22763932
So I guess the other aging patches have a larger impact on improving swappiness behavior than this one.
Best Regards
Barry
Barry Song (Xiaomi)
reviewerRFC PATCH v2 3/5 mm: mglru: prevent min_seq[type] from pointing to an empty generation
In reply to Barry Song (Xiaomi)In try_to_inc_min_seq(), min_seq[LRU_GEN_ANON] and min_seq[LRU_GEN_FILE] can be fixed up to point to empty generations in order to keep their gap within one. As a result, scan_folios() may repeatedly scan empty generations. This is confusing, as I observed scan_folios() returning 0 even though the following check in scan_folios() doesn't take effect:
if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
return 0;There is no need to adjust min_seq[], since the reclaim logic already triggers aging when the number of generations reaches MIN_NR_GENS, and reclaim never reduces it below MIN_NR_GENS.
--- include/linux/mmzone.h | 6 ++---- mm/vmscan.c | 10 ---------- 2 files changed, 2 insertions(+), 14 deletions(-)
include/linux/mmzone.h +2 -4
@@ -552,10 +552,8 @@ enum {* The youngest generation number is stored in max_seq for both anon and file * types as they are aged on an equal footing. The oldest generation numbers are * stored in min_seq[] separately for anon and file types so that they can be -* incremented independently. Ideally min_seq[] are kept in sync when both anon -* and file types are evictable. However, to adapt to situations like extreme -* swappiness, they are allowed to be out of sync by at most -* MAX_NR_GENS-MIN_NR_GENS-1. +* incremented independently. For both file and anonymous memory, the minimum +* generation must be at least MIN_NR_GENS. * * The number of pages in each generation is eventually consistent and therefore * can be transiently negative when reset_batch_size() is pending.
mm/vmscan.c +0 -10
@@ -3972,16 +3972,6 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)if (!seq_inc_flag)return ; -/* see the comment on lru_gen_folio */ -if (swappiness && swappiness <=MAX_SWAPPINESS ) { -unsigned long seq = lrugen->max_seq -MIN_NR_GENS ; - -if (min_seq[LRU_GEN_ANON ] > seq && min_seq[LRU_GEN_FILE ] < seq) - min_seq[LRU_GEN_ANON ] = seq; -else if (min_seq[LRU_GEN_FILE ] > seq && min_seq[LRU_GEN_ANON ] < seq) - min_seq[LRU_GEN_FILE ] = seq; - } -for_each_evictable_type (type, swappiness) {if (min_seq[type] <= lrugen->min_seq[type])continue ;
Signature · 1 line
2.39.3 (Apple Git-146)
Barry Song (Xiaomi)
reviewerRFC PATCH v2 4/5 mm: mglru: run aging when pages are severely imbalanced across gens
In reply to Barry Song (Xiaomi)From: lyugaofei <[email protected]>
This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
/*
* It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
* of the total number of pages for each generation. A reasonable range
* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
* aging cares about the upper bound of hot pages, while the eviction
* cares about the lower bound of cold pages.
*/
if (young * MIN_NR_GENS > total)
return true;
if (old * (MIN_NR_GENS + 2) < total)
return true;
But with a stricter condition: the younger generations must contain at least four times as many folios as the older generations. This allows aging to keep folios of the preferred type spread across the reclaimable generations.
--- mm/vmscan.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
mm/vmscan.c +36 -1
@@ -4956,9 +4956,40 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,int type, +unsigned long max_seq,unsigned long min_seq, +int swappiness) +{ +struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long young =0 , old =0 , seq; + +/* +* reclaim is forced to a single type in those cases, so there is +* no need to consider the swappiness bias +*/ +if (swappiness ==MIN_SWAPPINESS || swappiness >MAX_SWAPPINESS ) +return false; + +for (seq = min_seq; seq <= max_seq; seq++) { +int gen =lru_gen_from_seq (seq); +unsigned long size =0 ; +int zone; + +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); + +if (seq +MIN_NR_GENS > max_seq) + young += size; +else + old += size; + } +return young > old *4 ; +} +static bool should_run_aging (struct lruvec *lruvec,unsigned long max_seq,struct scan_control *sc,int swappiness) { +int type =get_type_to_scan (lruvec, swappiness);DEFINE_MIN_SEQ (lruvec);/* have to run aging, since eviction is not possible anymore */
@@ -4970,7 +5001,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,return false;/* better to run aging even though eviction is still possible */ -return evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq; +if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS == max_seq) +return true; + +/* Run aging if the preferred type is severely imbalanced across gens */ +return lru_gen_imbalanced (lruvec, type, max_seq, min_seq[type], swappiness); } static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
Signature · 1 line
2.39.3 (Apple Git-146)
Re: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
Hi Barry, lyugaofei,
The cover letter results look good.
This change looks like it resolves the cross-type skew from 798c0330c2ca
[1] (if I understand correctly).
But:
removed (798c0330c2ca): old * 4 < total
4/5: old * 4 < young
Can a single type also skew, and does changing total to young have any downside?
Nit: the old form derived the constant from (MIN_NR_GENS + 2).
[1] https://lore.kernel.org/all/[email protected]/
Best,
Zicheng
Barry Song
reviewerRe: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
Quoted 7 lines · Zicheng Wang
Hi Barry, lyugaofei,
The cover letter results look good.
This change looks like it resolves the cross-type skew from
798c0330c2ca
[1] (if I understand correctly).
Hi Zicheng,
I guess it somehow moves MGLRU back toward its original design of maintaining some balance across generations.
Quoted 8 lines
But:
removed (
798c0330c2ca): old * 4 < total
4/5: old * 4 < youngCan a single type also skew, and does changing total to young have any downside?
The original code is a bit odd, as it compares a single young generation and a single old generation against the total, rather than comparing all young generations with all old generations.
if (seq == max_seq)
young += size;
else if (seq + MIN_NR_GENS == max_seq)
old += size;Also, for the swappiness bias, we care more about the preferred type than the other type. So comparing a single type, rather than the sum of both types, makes more sense.
Quoted 2 lines
Nit: the old form derived the constant from (
MIN_NR_GENS+ 2).
By comparing all young generations against all old generations, we no longer depend on the exact value of MIN_NR_GENS. The idea is to emulate inactive_is_low().
Quoted 3 lines
Thanks
Barry
Re: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
Hi Barry,
Quoted 7 lines
The original code is a bit odd, as it compares a single young generation and a single old generation against the total, rather than comparing all young generations with all old generations.
By comparing all young generations against all old generations, we no longer depend on the exact value of
MIN_NR_GENS. The idea is to emulateinactive_is_low().
Makes sense, the inactive_is_low() framing fits.
One gap: inactive_is_low()'s ratio is dynamic calculated. int_sqrt(10*1024) = 101 (ages only when inactive drops below ~1%), while patch 4/5's fixed 4 ages once old falls under 20% of young. It seems there will be considerable cost on big memcgs, or should the ratio scale with size?
Looking forward to the perf data.
Thanks,
Zicheng
Barry Song
reviewerRe: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
Quoted 13 lines · Zicheng Wang
Hi Barry,
Quoted 7 lines
The original code is a bit odd, as it compares a single young generation and a single old generation against the total, rather than comparing all young generations with all old generations.
By comparing all young generations against all old generations, we no longer depend on the exact value of
MIN_NR_GENS. The idea is to emulateinactive_is_low().Makes sense, the
inactive_is_low()framing fits.
Quoted 7 lines
One gap: inactive_is_low()'s ratio is dynamic calculated.
int_sqrt(10*1024)= 101 (ages only when inactive drops below ~1%), while patch 4/5's fixed 4 ages once old falls under 20% of young. It seems there will be considerable cost on big memcgs, or should the ratio scale with size?Looking forward to the perf data.
I don't have access to a machine with 1 TB of memory—or even more than 20 GB—to test this. But I think your comment makes sense. We shouldn't let a machine with a huge amount of memory in a single lruvec age too aggressively.
So maybe something like this:
mm/vmscan.c +0 -0
@@ -4968,6 +4968,7 @@ static bool lru_gen_imbalanced(struct lruvec
*lruvec, int type,
{
struct lru_gen_folio *lrugen = &lruvec->lrugen;
unsigned long young = 0, old = 0, seq;+ unsigned long old_ratio, gb;
/*
* reclaim is forced to a single type in those cases, so there is
@@ -4989,7 +4990,16 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, int type,
else
old += size;
}
- return young > old * 4;
+
+ /*
+ * Copied from the active/inactive LRU heuristic, but uses a higher
+ * old_ratio so fewer folios remain in the old generations, avoiding
+ * overly aggressive aging.
+ */
+ gb = (young + old) >> (30 - PAGE_SHIFT);
+ old_ratio = gb ? int_sqrt(10 * gb) : 1;
+ old <<= 2;
+ return young > old * old_ratio;}
Best Regards
Barry
Re: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
Quoted 20 lines · Barry Song
Quoted 19 lines · Zicheng Wang
Hi Barry,
Quoted 7 lines
The original code is a bit odd, as it compares a single young generation and a single old generation against the total, rather than comparing all young generations with all old generations.
By comparing all young generations against all old generations, we no longer depend on the exact value of
MIN_NR_GENS. The idea is to emulateinactive_is_low().Makes sense, the
inactive_is_low()framing fits.One gap: inactive_is_low()'s ratio is dynamic calculated.
int_sqrt(10*1024)= 101 (ages only when inactive drops below ~1%), while patch 4/5's fixed 4 ages once old falls under 20% of young. It seems there will be considerable cost on big memcgs, or should the ratio scale with size?
Quoted 3 lines
Quoted 1 line
Looking forward to the perf data.
I don't have access to a machine with 1 TB of memory—or even
Me neither.
Quoted 25 lines
more than 20 GB—to test this. But I think your comment makes sense. We shouldn't let a machine with a huge amount of memory in a single lruvec age too aggressively.
So maybe something like this:mm/vmscan.c +1
-0@@ -4968,6 +4968,7 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, int type, {struct lru_gen_folio *lrugen = &lruvec->lrugen;unsigned long young =0 , old =0 , seq; +unsigned long old_ratio, gb;/* * reclaim is forced to a single type in those cases, so there is
mm/vmscan.c +1
-1@@ -4989,7 +4990,16 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec, int type, else old += size; } -return young > old *4 ; +
mm/vmscan.c static bool lru_gen_imbalanced(struct lruvec · line 5004
Quoted 10 lines
mm/vmscan.c +9
-0+/* +* Copied from the active/inactive LRU heuristic, but uses a higher +* old_ratio so fewer folios remain in the old generations, avoiding +* overly aggressive aging. +*/ + gb = (young + old) >> (30 -PAGE_SHIFT ); + old_ratio = gb ?int_sqrt (10 * gb) :1 ; + old <<=2 ; +return young > old * old_ratio; }
Look good to me.
Also, this heuristic makes me believe more that MGLRU's `aging' could be exposed as a standalone feature, allowing users to tune it more flexibly.
Quoted 3 lines
Best Regards
Barry
Best,
Zicheng
Re: [RFC PATCH v2 4/5] mm: mglru: run aging when pages are severely imbalanced across gens
In reply to Barry Song (Xiaomi)Quoted 56 lines · Barry Song (Xiaomi)
From: lyugaofei <[email protected]>
This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit,
ac35a4902370("mm: multi-gen LRU:
minimal implementation"):
/*
* It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
* of the total number of pages for each generation. A reasonable range
* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
* aging cares about the upper bound of hot pages, while the eviction
* cares about the lower bound of cold pages.
*/
if (young *MIN_NR_GENS> total)
return true;
if (old * (MIN_NR_GENS+ 2) < total)
return true;But with a stricter condition: the younger generations must contain at least four times as many folios as the older generations. This allows aging to keep folios of the preferred type spread across the reclaimable generations.
Signed-off-by lyugaofei <[email protected]>Co-developed-by Barry Song (Xiaomi) <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)mm/vmscan.c +18
-0@@ -4956,9 +4956,40 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; } +static bool lru_gen_imbalanced (struct lruvec *lruvec,int type, +unsigned long max_seq,unsigned long min_seq, +int swappiness) +{ +struct lru_gen_folio *lrugen = &lruvec->lrugen; +unsigned long young =0 , old =0 , seq; + +/* +* reclaim is forced to a single type in those cases, so there is +* no need to consider the swappiness bias +*/ +if (swappiness ==MIN_SWAPPINESS || swappiness >MAX_SWAPPINESS ) +return false; + +for (seq = min_seq; seq <= max_seq; seq++) { +int gen =lru_gen_from_seq (seq); +unsigned long size =0 ; +int zone;
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4986
Quoted 10 lines
mm/vmscan.c +10
-0+ +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + size +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); + +if (seq +MIN_NR_GENS > max_seq) + young += size; +else + old += size; + } +return young > old *4 ;
Hi Barry,
MAX_NR_GEN could be tunable, would it be better to use a macro like MAX_NR_GEN or MIN_NR_GEN here?
And I know there are experiments to extend MGLRU to run aging periodically so each gen has a fixed age:
https://lwn.net/Articles/976985/
Not sure if this will conflict if this is a hardcoded aging behavior, and if we want things like that in the future, these are useful on servers, I think android might benefit foo.
Barry Song (Xiaomi)
reviewerRFC PATCH v2 5/5 mm: mglru: run aging if the preferred type has no folios in reclaimable gens
In reply to Barry Song (Xiaomi)Respect the type selected by positive_ctrl_err(). If there are no reclaimable gens left for that type, run aging.
--- mm/vmscan.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/vmscan.c +4 -0
@@ -4996,6 +4996,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true; +/* run aging if the preferred type is exhausted */ +if (min_seq[type] +MIN_NR_GENS > max_seq) +return true; +/* try to avoid aging, do gentle reclaim at the default priority */ if (sc->priority ==DEF_PRIORITY )return false;
Signature · 1 line
2.39.3 (Apple Git-146)
Baolin Wang
reviewerRe: [RFC PATCH v2 5/5] mm: mglru: run aging if the preferred type has no folios in reclaimable gens
Quoted 13 lines · Barry Song (Xiaomi)
Respect the type selected by
positive_ctrl_err(). If there are no reclaimable gens left for that type, run aging.Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 4 ++++ 1 file changed, 4 insertions(+)mm/vmscan.c +0
-0
mm/vmscan.c static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, · line 5005
Quoted 10 lines
mm/vmscan.c +4
-0@@ -4996,6 +4996,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true; +/* run aging if the preferred type is exhausted */ +if (min_seq[type] +MIN_NR_GENS > max_seq) +return true; +/* try to avoid aging, do gentle reclaim at the default priority */ if (sc->priority ==DEF_PRIORITY )return false;
Looks reasonable to me.
Additionally, for sc->priority > DEF_PRIORITY, is there the same check logic for the scan type?
if (min_seq[type] + MIN_NR_GENS == max_seq)
return true;
Barry Song
reviewerRe: [RFC PATCH v2 5/5] mm: mglru: run aging if the preferred type has no folios in reclaimable gens
Quoted 20 lines · Baolin Wang
Quoted 16 lines · Barry Song (Xiaomi)
Respect the type selected by
positive_ctrl_err(). If there are no reclaimable gens left for that type, run aging.Signed-off-by Barry Song (Xiaomi) <[email protected]>--- mm/vmscan.c | 4 ++++ 1 file changed, 4 insertions(+)mm/vmscan.c +0
-0@@ -4996,6 +4996,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,if (evictable_min_seq (min_seq, swappiness) +MIN_NR_GENS > max_seq)return true;
Quoted 9 lines
mm/vmscan.c static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, · line 5005
Quoted 7 lines
mm/vmscan.c +4
-0+/* run aging if the preferred type is exhausted */ +if (min_seq[type] +MIN_NR_GENS > max_seq) +return true; +/* try to avoid aging, do gentle reclaim at the default priority */ if (sc->priority ==DEF_PRIORITY )return false;Looks reasonable to me.
Yes, this is reasonable. However, I also noticed that it can slightly increase system time at moderate swappiness values during a kernel build.
At swappiness values near 100, overall performance is more important than strictly preserving the swappiness bias. To address this, I refined the behavior for balanced swappiness in v3:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 6 lines
Additionally, for
sc->priority>DEF_PRIORITY, is there the same check logic for the scan type?if (min_seq[type] +
MIN_NR_GENS==max_seq)return true;
We also have an imbalance check: if there are only three generations (min_seq[type] + MIN_NR_GENS == max_seq) and the folio distribution is heavily skewed across them, we trigger aging. This helps preserve the intended swappiness bias:
https://lore.kernel.org/linux-mm/[email protected]/
Thanks
Barry