mm: mglru: fix swappiness behavior

RFC PATCH Superseded Barry Song (Xiaomi) 1 participant 6 messages Jul 26

Subsystems (3)

Memory management MGLRU MM reclaim

lore

Share & tools

lore

Barry Song (Xiaomi)

author reviewer

2026-07-26 01:29 UTC #0 lore raw

cover mm: mglru: fix swappiness behavior

The active/inactive LRU respects swappiness well. Anonymous page scanning and reclamation increase roughly linearly with swappiness, while file page scanning and reclamation decrease accordingly.
For example, when swappiness reaches 200, both pgsteal_file and pgscan_file drop to zero while building the kernel in a 1 GB memcg. In contrast, MGLRU shows almost no change across different swappiness values.

pgsteal_file

Swappiness LRU MGLRU
--------------------------------

1             10567455      763612
36              990706      480205
71              688170      415848
106             446294      386164
141             286307      359196
176             201733      351686
200                  0      330093

pgsteal_anon

Swappiness LRU MGLRU
--------------------------------

1              4410548     2726362
36             2465268     2762859
71             2677908     2885124
106            2737227     2841796
141            2984276     3035015
176            3381338     2938302
200           13116359     3113499

pgscan_file

Swappiness LRU MGLRU
--------------------------------

1             17997223      923094
36             1325674      539571
71              852345      464222
106             538207      464477
141             357253      412277
176             217536      399446
200                  0      375902

pgscan_anon

Swappiness LRU MGLRU
--------------------------------

1             31639423     5987136
36            23441521     5753224
71            26067110     6101780
106           25619448     5782919
141           26842088     6234264
176           29200021     5980292
200           62193924     6413125

This patchset respects the type selected by positive_ctrl_err(), which uses swappiness as its gain. It does so by:

1. Avoiding premature fallback to the other type. Only fall back when reclaim is running at high priority.

2. Running aging when the preferred type has few or no reclaimable folios, so more folios of that type become reclaimable.

With this patchset, swappiness starts to behave similarly to the active/inactive LRU.

pgsteal_file

Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------

1             10567455      763612        3507619
36              990706      480205         528781
71              688170      415848         478478
106             446294      386164         411632
141             286307      359196         375137
176             201733      351686         298586
200                  0      330093              0

pgsteal_anon

Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------

1              4410548     2726362        2876888
36             2465268     2762859        3022062
71             2677908     2885124        3129555
106            2737227     2841796        2974192
141            2984276     3035015        3092280
176            3381338     2938302        3353395
200           13116359     3113499       14348518

pgscan_file

Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------

1             17997223      923094        5842416
36             1325674      539571         648111
71              852345      464222         569930
106             538207      464477         480767
141             357253      412277         431911
176             217536      399446         337768
200                  0      375902              0

pgscan_anon

Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------

1             31639423     5987136        4658593
36            23441521     5753224        5444726
71            26067110     6101780        5755804
106           25619448     5782919        5872415
141           26842088     6234264        6144096
176           29200021     5980292        6615500
200           62193924     6413125       22984986

Another possible approach is to decouple anonymous and file-backed aging by maintaining separate max_seq values for each type. This allows anonymous and file-backed memory to age and be reclaimed independently, enabling the swappiness-preferred type to be reclaimed more aggressively while allowing the other type to lag behind. This approach has already been adopted by projects such as CachyOS [1] and Chromium [2].

However, this approach requires substantial changes to MGLRU and fundamentally alters its design by breaking the shared aging timeline between anonymous and file-backed memory. This timeline is the foundation for mechanisms such as the PID controller and the min_ttl_ms thrashing protection.

That is why this patchset aims to fix the swappiness behavior without fundamentally changing MGLRU's design, with minimal changes.

[1] https://github.com/firelzrd/re-swappiness
[2] https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/929932351492d01f0aee37a0ac3be8c7bd88f80d

Barry Song (Xiaomi) (3):
mm: mglru: only fall back when reclaim is running at high priority
mm: mglru: do try_to_inc_min_seq if scanned==0
mm: mglru: run aging if the preferred type has no folios in
reclaimable gens

lyugaofei (1):
mm: mglru: run aging when pages are severely imbalanced across gens

 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author reviewer

2026-07-26 01:29 UTC #1 lore raw

RFC PATCH 1/4 mm: mglru: only fall back when reclaim is running at high priority

Respect the type selected by positive_ctrl_err(), where swappiness controls the relative weights of SP and PV. Falling back too readily violates that bias.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

mm/vmscan.c +5 -2

index 566c4e837c7d..f79402760776 100644
@@ -4855,9 +4855,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		 * If scanned > 0 and isolated == 0, avoid falling back to the
 		 * other type, as this type remains sufficient. Falling back
 		 * too readily can disrupt the positive_ctrl_err() bias.
+		 * Only fall back when reclaim is running at high priority.
 		 */
-		if (!scanned)
-			type = !type;
+		if (!scanned) {
+			if (!sc->priority)
+				type = !type;
+		}
 	}
 
 	return total_scanned;

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author reviewer

2026-07-26 01:29 UTC #2 lore raw

RFC PATCH 2/4 mm: mglru: do try_to_inc_min_seq if scanned==0

In reply to Barry Song (Xiaomi)

If scanned == 0, we are likely running out of folios in the reclaimable gens. Advance min_seq to keep it in sync with the current reclaimable state.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 1 +
 1 file changed, 1 insertion(+)

mm/vmscan.c +1 -0

index f79402760776..4f3a375de86e 100644
@@ -4858,6 +4858,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		 * Only fall back when reclaim is running at high priority.
 		 */
 		if (!scanned) {
+			try_to_inc_min_seq(lruvec, swappiness);
 			if (!sc->priority)
 				type = !type;
 		}

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author reviewer

2026-07-26 01:30 UTC #3 lore raw

RFC PATCH 3/4 mm: mglru: run aging when pages are severely imbalanced across gens

In reply to Barry Song (Xiaomi)

From: lyugaofei <[email protected]>

This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU:
minimal implementation"):
/*
* It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1)
* of the total number of pages for each generation. A reasonable range
* for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The
* aging cares about the upper bound of hot pages, while the eviction
* cares about the lower bound of cold pages.
*/
if (young * MIN_NR_GENS > total)
return true;
if (old * (MIN_NR_GENS + 2) < total)
return true;

But with a stricter condition: the youngest gen must have at least four times as many folios as the oldest. This helps keep folios of the preferred type distributed across the reclaimable gens.

Signed-off-by lyugaofei <[email protected]>

Co-developed-by Barry Song (Xiaomi) <[email protected]>

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

mm/vmscan.c +33 -1

index 4f3a375de86e..9aac4febc206 100644
@@ -4963,9 +4963,37 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	return scanned;
 }
 
+static bool lru_gen_imbalanced(struct lruvec *lruvec, int type,
+		unsigned long max_seq, unsigned long min_seq,
+		int swappiness)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	unsigned long young = 0, old = 0, seq;
+
+	if (swappiness == MIN_SWAPPINESS || swappiness > MAX_SWAPPINESS)
+		return false;
+
+	for (seq = min_seq; seq <= max_seq; seq++) {
+		int gen = lru_gen_from_seq(seq);
+		unsigned long size = 0;
+		int zone;
+
+		gen = lru_gen_from_seq(seq);
+		for (zone = 0; zone < MAX_NR_ZONES; zone++)
+			size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+
+		if (seq + MIN_NR_GENS > max_seq)
+			young += size;
+		else
+			old += size;
+	}
+	return young > old * 4;
+}
+
 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 			     struct scan_control *sc, int swappiness)
 {
+	int type = get_type_to_scan(lruvec, swappiness);
 	DEFINE_MIN_SEQ(lruvec);
 
 	/* have to run aging, since eviction is not possible anymore */

@@ -4977,7 +5005,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 		return false;
 
 	/* better to run aging even though eviction is still possible */
-	return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq;
+	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq)
+		return true;
+
+	/* Run aging if the preferred type is severely imbalanced across gens */
+	return lru_gen_imbalanced(lruvec, type, max_seq, min_seq[type], swappiness);
 }
 
 static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song (Xiaomi)

author reviewer

2026-07-26 01:30 UTC #4 lore raw

RFC PATCH 4/4 mm: mglru: run aging if the preferred type has no folios in reclaimable gens

In reply to Barry Song (Xiaomi)

Respect the type selected by positive_ctrl_err(). If there are no reclaimable gens left for that type, run aging.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 4 ++++
 1 file changed, 4 insertions(+)

mm/vmscan.c +4 -0

index 9aac4febc206..be34b8b946ad 100644
@@ -5000,6 +5000,10 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq,
 	if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq)
 		return true;
 
+	/* run aging if the preferred type is exhausted */
+	if (min_seq[type] + MIN_NR_GENS > max_seq)
+		return true;
+
 	/* try to avoid aging, do gentle reclaim at the default priority */
 	if (sc->priority == DEF_PRIORITY)
 		return false;

Signature · 1 line

2.39.3 (Apple Git-146)

Barry Song

author reviewer

2026-07-26 03:41 UTC #5 lore raw

In reply to Barry Song (Xiaomi)

Quoted 145 lines · Barry Song (Xiaomi)

The active/inactive LRU respects swappiness well. Anonymous page scanning and reclamation increase roughly linearly with swappiness, while file page scanning and reclamation decrease accordingly.
For example, when swappiness reaches 200, both pgsteal_file and pgscan_file drop to zero while building the kernel in a 1 GB memcg. In contrast, MGLRU shows almost no change across different swappiness values.
pgsteal_file
Swappiness LRU MGLRU
--------------------------------
1             10567455      763612
36              990706      480205
71              688170      415848
106             446294      386164
141             286307      359196
176             201733      351686
200                  0      330093
pgsteal_anon
Swappiness LRU MGLRU
--------------------------------
1              4410548     2726362
36             2465268     2762859
71             2677908     2885124
106            2737227     2841796
141            2984276     3035015
176            3381338     2938302
200           13116359     3113499
pgscan_file
Swappiness LRU MGLRU
--------------------------------
1             17997223      923094
36             1325674      539571
71              852345      464222
106             538207      464477
141             357253      412277
176             217536      399446
200                  0      375902
pgscan_anon
Swappiness LRU MGLRU
--------------------------------
1             31639423     5987136
36            23441521     5753224
71            26067110     6101780
106           25619448     5782919
141           26842088     6234264
176           29200021     5980292
200           62193924     6413125
This patchset respects the type selected by positive_ctrl_err(), which uses swappiness as its gain. It does so by:
1. Avoiding premature fallback to the other type. Only fall back when reclaim is running at high priority.
2. Running aging when the preferred type has few or no reclaimable folios, so more folios of that type become reclaimable.
With this patchset, swappiness starts to behave similarly to the active/inactive LRU.
pgsteal_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1             10567455      763612        3507619
36              990706      480205         528781
71              688170      415848         478478
106             446294      386164         411632
141             286307      359196         375137
176             201733      351686         298586
200                  0      330093              0
pgsteal_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1              4410548     2726362        2876888
36             2465268     2762859        3022062
71             2677908     2885124        3129555
106            2737227     2841796        2974192
141            2984276     3035015        3092280
176            3381338     2938302        3353395
200           13116359     3113499       14348518
pgscan_file
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1             17997223      923094        5842416
36             1325674      539571         648111
71              852345      464222         569930
106             538207      464477         480767
141             357253      412277         431911
176             217536      399446         337768
200                  0      375902              0
pgscan_anon
Swappiness LRU MGLRU MGLRU+Patch
-------------------------------------------------
1             31639423     5987136        4658593
36            23441521     5753224        5444726
71            26067110     6101780        5755804
106           25619448     5782919        5872415
141           26842088     6234264        6144096
176           29200021     5980292        6615500
200           62193924     6413125       22984986
Another possible approach is to decouple anonymous and file-backed aging by maintaining separate max_seq values for each type. This allows anonymous and file-backed memory to age and be reclaimed independently, enabling the swappiness-preferred type to be reclaimed more aggressively while allowing the other type to lag behind. This approach has already been adopted by projects such as CachyOS [1] and Chromium [2].
However, this approach requires substantial changes to MGLRU and fundamentally alters its design by breaking the shared aging timeline between anonymous and file-backed memory. This timeline is the foundation for mechanisms such as the PID controller and the min_ttl_ms thrashing protection.
That is why this patchset aims to fix the swappiness behavior without fundamentally changing MGLRU's design, with minimal changes.
[1] https://github.com/firelzrd/re-swappiness
[2] https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/929932351492d01f0aee37a0ac3be8c7bd88f80d

Quoted 8 lines

Barry Song (Xiaomi) (3):
mm: mglru: only fall back when reclaim is running at high priority
mm: mglru: do try_to_inc_min_seq if scanned==0
mm: mglru: run aging if the preferred type has no folios in
reclaimable gens
lyugaofei (1):
mm: mglru: run aging when pages are severely imbalanced across gens

I checked Sashiki's comment:
https://sashiko.dev/#/patchset/[email protected]

1. Sashiko says
"

Review static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4861

Quoted 5 lines

(quoted diff) +1 -0

@@ -4858,6 +4858,7 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 * Only fall back when reclaim is running at high priority.
 */
 if (!scanned) {
+ try_to_inc_min_seq(lruvec, swappiness);

Is this call to try_to_inc_min_seq() redundant?
The caller evict_folios() acquires the LRU lock and calls try_to_inc_min_seq() just before calling isolate_folios(). If scanned == 0 on the first iteration of the loop in isolate_folios(), the lists haven't changed since the previous try_to_inc_min_seq() call. This means get_nr_gens == MIN_NR_GENS, making this new call a strict no-op.
If scanned == 0 happens on the second iteration (because the first iteration processed folios without isolating any), try_to_inc_min_seq() might advance min_seq. However, the for_each_evictable_type() loop only iterates twice, so it terminates immediately without scanning the newly advanced generation.
After isolate_folios() returns, evict_folios() unconditionally calls try_to_inc_min_seq() again anyway. Does this logic successfully retry with new folios, or does it only perform unnecessary work?
"

I observed that scan_folios() always returns 0 because the list is empty, making the following early-return check appear to be ineffective:

        if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
                return 0;

So I thought min_seq should be advanced when scanned == 0, since there are no folios left in the reclaimable generation.

After taking another look, I realized that isn't the real issue. The problem is that try_to_inc_min_seq() doesn't always reflect the actual oldest reclaimable generation. It can leave min_seq pointing to an older generation even after that generation has no folios remaining:

static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)

{
        ...
        /* see the comment on lru_gen_folio */
        if (swappiness && swappiness <= MAX_SWAPPINESS) {
                unsigned long seq = lrugen->max_seq - MIN_NR_GENS;

                if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
                        min_seq[LRU_GEN_ANON] = seq;
                else if (min_seq[LRU_GEN_FILE] > seq &&
min_seq[LRU_GEN_ANON] < seq)
                        min_seq[LRU_GEN_FILE] = seq;
        }
}

I think the function itself should be fixed instead. Otherwise, I can leave it as is and simply drop the newly added call to try_to_inc_min_seq().

2. Sashiko says

Review static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4862

Quoted 1 line

(quoted diff) +1 -0
+ gen = lru_gen_from_seq(seq);

This isn't a bug, but is this second assignment of gen redundant since it was already initialized to the exact same value in its declaration?
"

This is right. I will fix it in RFC v2.

3. Sashiko says

mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4860

Quoted 13 lines

mm/vmscan.c +2 -2

index 566c4e837c7d5..f794027607767 100644
@@ -4855,9 +4855,12 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 * If scanned > 0 and isolated == 0, avoid falling back to the
 * other type, as this type remains sufficient. Falling back
 * too readily can disrupt the positive_ctrl_err() bias.
+ * Only fall back when reclaim is running at high priority.
 */
- if (!scanned)
- type = !type;
+ if (!scanned) {

Does this logic completely prevent kswapd from falling back to the other LRU type?
Looking at kswapd's reclaim loop in balance_pgdat(), it explicitly exits before sc.priority reaches 0:
mm/vmscan.c:balance_pgdat() {
...
if (raise_priority || !nr_reclaimed)
sc.priority--;
} while (sc.priority >= 1);
Because sc.priority never reaches 0 inside the kswapd reclaim path, kswapd will never satisfy this fallback condition. It will repeatedly scan the empty type, fail to make progress, drop its priority, and eventually give up and sleep.
For direct reclaim, this also means processes will stall fruitlessly through multiple priority drops (triggering VMSCAN_THROTTLE_NOPROGRESS delays) before finally reaching priority 0 and falling back, which could lead to severe stalling or premature OOM killer invocation.

mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 4864

Quoted 4 lines

mm/vmscan.c +3 -0

+ if (!sc->priority)
+ type = !type;
+ }
 }

This seems to be a valid concern. I could address it by:

mm/vmscan.c +0 -0

index be34b8b946ad..d69a0ed26f8c 100644
@@ -4859,7 +4859,7 @@ static int isolate_folios(unsigned long

nr_to_scan, struct lruvec *lruvec,

                 */
                if (!scanned) {
                        try_to_inc_min_seq(lruvec, swappiness);
-                       if (!sc->priority)
+                       if (sc->priority <= 2)
                                type = !type;
                }
        }

4. Sashiko says

"> But with a stricter condition: the youngest gen must have at

Quoted 3 lines

least four times as many folios as the oldest. This helps keep folios of the preferred type distributed across the reclaimable gens.

Is the commit message description accurate regarding "the youngest gen"? Looking at lru_gen_imbalanced(), the code checks:
if (seq + MIN_NR_GENS > max_seq)
Since MIN_NR_GENS is 2, wouldn't this condition group the two youngest generations into the young variable rather than just the single youngest generation?"

Yes, I'll update the description.

5. Sashiko says
"

mm/vmscan.c

Quoted 9 lines

mm/vmscan.c +9 -0

+static bool lru_gen_imbalanced(struct lruvec *lruvec, int type,
+ unsigned long max_seq, unsigned long min_seq,
+ int swappiness)
+{
+ struct lru_gen_folio *lrugen = &lruvec->lrugen;
+ unsigned long young = 0, old = 0, seq;
+
+ if (swappiness == MIN_SWAPPINESS || swappiness > MAX_SWAPPINESS)
+ return false;

Does this unexpectedly disable the imbalance heuristic for systems where swappiness is configured to 0?
When swappiness is 0 (MIN_SWAPPINESS), get_type_to_scan() correctly selects LRU_GEN_FILE, but this check causes an immediate return of false. Wouldn't file pages still need to be balanced across generations on these systems? "

Technically, we should maintain the balance for swappiness values of 0 and 201 as well. However, in those cases we always reclaim a single type, so the swappiness bias is irrelevant. We can therefore skip this.

Best Regards
Barry