mm/vmscan: throttle MGLRU eviction when isolated folios pile up
Subsystems (4)
PATCH v7 1/1 mm/vmscan: throttle MGLRU eviction when isolated folios pile up
From: Hui Zhu <[email protected]>
The legacy path throttles direct reclaim via too_many_isolated() when isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.
Add lru_gen_too_many_isolated() and throttle_evictable_types(), called from evict_folios() before the lruvec lock is taken since throttling sleeps.
The check is per lruvec rather than node-wide: reclaimers contend per lruvec, and a node-wide check would let isolation in one memcg throttle reclaim in another. The threshold is the evictable size of the type divided by MIN_NR_GENS, approximating one generation.
Since isolate_folios() picks the type from refault feedback and may fall back to the other one, throttle_evictable_types() collects the types that are not over-isolated and have evictable folios, and sleeps only when all of them are over-isolated. The mask is passed to isolate_folios() to restrict its choice and fallback, so an over-isolated type never blocks the other, and isolation never lands on a throttled or empty type.
An lruvec with no evictable folios at all is not over-isolation:
reclaim stops immediately instead of waiting. Without this, allowed == 0 would conflate "all types are empty" with "all types are over-isolated" and send the reclaimer to sleep, and since the kswapd exemption lives inside the per-type check, which empty types bypass, even kswapd would stall on an empty lruvec.
Like the legacy path, kswapd and !writeback_throttling_sane() reclaim are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and a dying task fakes progress to exit reclaim quickly.
This builds on the recently merged NR_ISOLATED accounting fix for the MGLRU reclaim path, which makes in-flight isolation visible in /proc/vmstat; the throttle itself uses a separate per-lruvec counter. The nr_isolated numbers in the test results below therefore come from that fix rather than from this patch.
Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (0x0007). "unpatched" and "patched" are the same tree built without/with this patch; every configuration is run with identical parameters on both kernels. Test scripts:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34
1) Synthetic (mglru_iso_repro.sh, mglru_iso_repro_v2.sh): threads write-touching a 48MB anonymous working set inside a 16MB memcg, with swap on a dm-delay device (50ms) to slow swap-out and lengthen the isolation window.
64 threads, 60s - neither kernel OOMs at this concurrency; the gain shows in reclaim quality:
unpatched patched
OOM kills 0 0
total touches 246,499,132,369 301,456,426,692 (+22%)
scan efficiency 0.0261 0.0194
mm_vmscan_throttled 0 35,645 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolated peak 0 230
The patched kernel completes ~22% more work in the same 60s: the throttle keeps concurrent reclaimers from trampling the same generation, so less CPU is burned in reclaim.
192 threads, 60s - over-isolation becomes fatal without the throttle:
unpatched patched
OOM kills 1 (task killed) 0
memcg oom events 51 0
total touches 0 (killed) 682,300,039,725
mm_vmscan_throttled 0 512,292
nr_isolated peak 0 335
The unpatched kernel cannot keep reclaim ahead of allocation and the task is OOM-killed; the patched kernel survives the full run and keeps reclaim making progress.
2) Real workload (mglru_build_repro.sh): a CI-style build pod - repeated from-scratch "make -j20 vmlinux" rounds inside a memory-limited cgroup, with the object tree on ext4 over dm-delay (10ms) to model slow storage. 20 concurrent compilers charge into one memcg and enter direct reclaim at the same time, while dirty object files stretch the isolate->shrink->putback window. Each kernel runs 1800s per pressure level and the metrics are cumulative counters over that window, which smooths run-to-run noise. The headline metric is the number of objects actually compiled (a full round is ~3,800 objects).
memory.max = 1024MB (extreme pressure):
unpatched patched
objects compiled 4,744 7,211 (+52%)
pgscan_direct 2,901,083,521 2,228,776,807 (-23%)
scan efficiency 0.195 0.242
workingset_refault 565,169,241 537,686,479
mm_vmscan_throttled 0 511,441 (all VMSCAN_THROTTLE_ISOLATED)
nr_isolated peak 0 1,152
memory.max = 1536MB (heavy pressure):
unpatched patched
objects compiled 18,969 25,124 (+32%)
pgscan_direct 1,223,905,508 816,812,074 (-33%)
scan efficiency 0.320 0.338
workingset_refault 390,048,986 273,442,445 (-30%)
memcg charge fails 8,344,775 5,868,167 (-30%)
round wall time 199-391s 182-275s
mm_vmscan_throttled 0 503,117
nr_isolated peak 0 1,053At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB of anon in total), so rounds end with a cc1 being memcg-OOM-killed on either kernel. Normalized per completed round the kill rate is the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the patched kernel simply reaches that point sooner because less time is burned thrashing, so it cycles through more rounds. This patch does not claim to avoid those OOMs - the point is that it performs 23-33% fewer futile LRU scans and gets 32-52% more build work done in the same wall time.
memory.max = 2048MB (light pressure - no-harm control):
unpatched patched
rounds completed 6 6
avg wall time/round 271s 270s
scan efficiency 0.984 0.978
mm_vmscan_throttled 0 0
nr_isolated peak 0 144When reclaim keeps up the throttle never fires and both kernels perform identically, so the check costs nothing on a system that is not over-isolating.
3) Service no-impact (mglru_redis_repro.sh and mglru_redis_shards_repro.sh): a redis-server serving a 1KB-value dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) - first as a single instance (100k keys, 200 clients), then as 8 shards with AOF persistence on a second dm-delay device sharing one cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O serialized on redis's single command thread, not isolation
contention: charge-induced reclaim is a trickle (~400 scans/s vs 10^5-10^6 scans/s in the build runs), isolation never piles up, and the throttle correctly stays dormant on both kernels:
unpatched patched
single instance 41.6 ops/s 41.2 ops/s
(avg 2999ms) (avg 2999ms)
8 shards aggregate 333-419 ops/s 310-358 ops/s
(mean 376, (mean 335,
4 runs) (*) 4 runs)
mm_vmscan_throttled 0 0 (all runs)
nr_isolated peak 0 0 (*) the aggregate is dominated by a preload-end residency lottery (which shards' datasets happen to stay hot) and swings by up to 26% between runs of the SAME kernel; with 4 runs per kernel the difference between the means is within that noise (Welch t=1.95, p~0.11). The throttle never fired in any patched run (the tracepoint covers preload as well), so both kernels execute the same reclaim path here, and the kernel-side counters agree across all 8 runs:
steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs document the no-impact side: the check is dormant unless isolation actually piles up.
---
Changelog:
v7:
Patch 1 was merged into mm-unstable, so this is resent as a single patch.
According to the comments of Sashiko, Return instead of waiting when the lruvec has no evictable folios at all: allowed == 0 previously conflated "all empty" with "all over-isolated" and stalled even kswapd for 20ms. Move the v6 cover-letter test data into the commit message. v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a memory-limited cgroup with its object tree on dm-delay-backed storage, measured at three pressure levels (1024/1536/2048MB), plus redis-based cache service runs (single instance and 8 shards sharing one cgroup) as no-impact data points. New and updated test scripts are in the gist. Rebased onto the latest mm-stable; patch 2 is adapted to the current isolate_folios() type-selection context (is_single_type_ reclaim()), with no semantic change. Re-run the tests and update the test logs.
v5:
According to the comments of Kairun and feedback from the test scripts, make the throttle check per lruvec (new nr_isolated counter) and skip empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop the throttle_is_throttled() helper extracted
from shrink_inactive_list() and leave the legacy path untouched. The new MGLRU-only throttle_evictable_types() only sleeps when all evictable types are over-isolated - v3 throttled as soon as any of them was, which unnecessarily blocked the reclaim of the other type and passes the mask of the remaining types to isolate_folios(), which restricts both its initial choice and its fallback, so that isolation never lands on a throttled type. The v3 gate did not constrain the type actually isolated, so the fallback could still pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundant nr_isolated check before restoring the NR_ISOLATED_* counters in evict_folios(). rename the extracted helper to throttle_is_throttled() to avoid confusion with the existing wake_throttle_isolated() naming space. Use for_each_evictable_type() in the MGLRU throttle to check each evictable type's isolation instead of only the type returned by get_type_to_scan(), since isolate_folios() may fall back to the other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable. Split into two patches; patch 2 is new and adds the too_many_isolated() throttling to the MGLRU eviction path, which v1 did not cover.
Add test infomations.
include/linux/mmzone.h | 2 + mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 172 insertions(+), 14 deletions(-)
include/linux/mmzone.h +2 -0
@@ -564,6 +564,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the number of folios isolated from this lruvec and being reclaimed */ +atomic_long_t nr_isolated[ANON_AND_FILE ];/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */
mm/vmscan.c +170 -14
244 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 101 lines · Hui Zhu
From: Hui Zhu <[email protected]>
The legacy path throttles direct reclaim via
too_many_isolated()when isolated folios pile up, but MGLRU'sevict_folios()isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.Add
lru_gen_too_many_isolated()andthrottle_evictable_types(), called fromevict_folios()before the lruvec lock is taken since throttling sleeps.The check is per lruvec rather than node-wide: reclaimers contend per lruvec, and a node-wide check would let isolation in one memcg throttle reclaim in another. The threshold is the evictable size of the type divided by
MIN_NR_GENS, approximating one generation.Since
isolate_folios()picks the type from refault feedback and may fall back to the other one,throttle_evictable_types()collects the types that are not over-isolated and have evictable folios, and sleeps only when all of them are over-isolated. The mask is passed toisolate_folios()to restrict its choice and fallback, so an over-isolated type never blocks the other, and isolation never lands on a throttled or empty type.An lruvec with no evictable folios at all is not over-isolation:
reclaim stops immediately instead of waiting. Without this, allowed == 0 would conflate "all types are empty" with "all types are over-isolated" and send the reclaimer to sleep, and since the kswapd exemption lives inside the per-type check, which empty types bypass, even kswapd would stall on an empty lruvec.Like the legacy path, kswapd and !writeback_throttling_sane() reclaim are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and a dying task fakes progress to exit reclaim quickly.
This builds on the recently merged
NR_ISOLATEDaccounting fix for the MGLRU reclaim path, which makes in-flight isolation visible in/proc/vmstat; the throttle itself uses a separate per-lruvec counter. Thenr_isolatednumbers in the test results below therefore come from that fix rather than from this patch.Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (
0x0007). "unpatched" and "patched" are the same tree built without/with this patch; every configuration is run with identical parameters on both kernels. Test scripts:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e341) Synthetic (
mglru_iso_repro.sh,mglru_iso_repro_v2.sh): threads write-touching a 48MB anonymous working set inside a 16MB memcg, with swap on a dm-delay device (50ms) to slow swap-out and lengthen the isolation window.64 threads, 60s - neither kernel OOMs at this concurrency; the gain shows in reclaim quality:
unpatched patched
OOM kills 0 0
total touches 246,499,132,369 301,456,426,692 (+22%)
scan efficiency 0.0261 0.0194
mm_vmscan_throttled0 35,645 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolatedpeak 0 230The patched kernel completes ~22% more work in the same 60s: the throttle keeps concurrent reclaimers from trampling the same generation, so less CPU is burned in reclaim.
192 threads, 60s - over-isolation becomes fatal without the throttle:
unpatched patched
OOM kills 1 (task killed) 0
memcg oom events 51 0
total touches 0 (killed) 682,300,039,725
mm_vmscan_throttled0 512,292
nr_isolatedpeak 0 335The unpatched kernel cannot keep reclaim ahead of allocation and the task is OOM-killed; the patched kernel survives the full run and keeps reclaim making progress.
2) Real workload (
mglru_build_repro.sh): a CI-style build pod - repeated from-scratch "make -j20 vmlinux" rounds inside a memory-limited cgroup, with the object tree on ext4 over dm-delay (10ms) to model slow storage. 20 concurrent compilers charge into one memcg and enter direct reclaim at the same time, while dirty object files stretch theisolate->shrink->putbackwindow. Each kernel runs 1800s per pressure level and the metrics are cumulative counters over that window, which smooths run-to-run noise. The headline metric is the number of objects actually compiled (a full round is ~3,800 objects).memory.max = 1024MB (extreme pressure):
unpatched patched objects compiled 4,744 7,211 (+52%) pgscan_direct 2,901,083,521 2,228,776,807 (-23%) scan efficiency 0.195 0.242 workingset_refault 565,169,241 537,686,479 mm_vmscan_throttled 0 511,441 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolatedpeak 0 1,152
Quoted 10 lines
memory.max = 1536MB (heavy pressure):
unpatched patched
objects compiled 18,969 25,124 (+32%)
pgscan_direct 1,223,905,508 816,812,074 (-33%)
scan efficiency 0.320 0.338
workingset_refault 390,048,986 273,442,445 (-30%)
memcg charge fails 8,344,775 5,868,167 (-30%)
round wall time 199-391s 182-275s
mm_vmscan_throttled 0 503,117
nr_isolated peak 0 1,053Hmm is this because of the type fallback? e.g. too many files isoalted -> scan anon instead / too many anon isolate -> scan file instead.
Quoted 34 lines
At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB of anon in total), so rounds end with a cc1 being memcg-OOM-killed on either kernel. Normalized per completed round the kill rate is the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the patched kernel simply reaches that point sooner because less time is burned thrashing, so it cycles through more rounds. This patch does not claim to avoid those OOMs - the point is that it performs 23-33% fewer futile LRU scans and gets 32-52% more build work done in the same wall time.
memory.max = 2048MB (light pressure - no-harm control):
unpatched patched rounds completed 6 6 avg wall time/round 271s 270s scan efficiency 0.984 0.978 mm_vmscan_throttled 0 0 nr_isolated peak 0 144When reclaim keeps up the throttle never fires and both kernels perform identically, so the check costs nothing on a system that is not over-isolating.
3) Service no-impact (
mglru_redis_repro.shandmglru_redis_shards_repro.sh): a redis-server serving a 1KB-value dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) - first as a single instance (100k keys, 200 clients), then as 8 shards with AOF persistence on a second dm-delay device sharing one cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O serialized on redis's single command thread, not isolation
contention: charge-induced reclaim is a trickle (~400 scans/s vs 10^5-10^6 scans/s in the build runs), isolation never piles up, and the throttle correctly stays dormant on both kernels:
Quoted 6 lines
unpatched patched
single instance 41.6 ops/s 41.2 ops/s
(avg 2999ms) (avg 2999ms)
8 shards aggregate 333-419 ops/s 310-358 ops/s
(mean 376, (mean 335,
4 runs) (*) 4 runs)Hmm, I know this could be noise but it looks like a regression?
Quoted 14 lines
mm_vmscan_throttled 0 0 (all runs) nr_isolated peak 0 0(*) the aggregate is dominated by a preload-end residency lottery (which shards' datasets happen to stay hot) and swings by up to 26% between runs of the SAME kernel; with 4 runs per kernel the difference between the means is within that noise (Welch t=1.95, p~0.11). The throttle never fired in any patched run (the tracepoint covers preload as well), so both kernels execute the same reclaim path here, and the kernel-side counters agree across all 8 runs:
steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs document the no-impact side: the check is dormant unless isolation actually piles up.
Quoted 1 line
Signed-off-by Hui Zhu <[email protected]>
Hi Hui
Looks like a good idea in general.
Quoted 105 lines
---
Changelog:
v7:
Patch 1 was merged into mm-unstable, so this is resent as a single patch.
According to the comments of Sashiko, Return instead of waiting when the lruvec has no evictable folios at all: allowed == 0 previously conflated "all empty" with "all over-isolated" and stalled even kswapd for 20ms. Move the v6 cover-letter test data into the commit message. v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a memory-limited cgroup with its object tree on dm-delay-backed storage, measured at three pressure levels (1024/1536/2048MB), plus redis-based cache service runs (single instance and 8 shards sharing one cgroup) as no-impact data points. New and updated test scripts are in the gist. Rebased onto the latest mm-stable; patch 2 is adapted to the currentisolate_folios()type-selection context (is_single_type_reclaim()), with no semantic change. Re-run the tests and update the test logs.
v5:
According to the comments of Kairun and feedback from the test scripts, make the throttle check per lruvec (newnr_isolatedcounter) and skip empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop thethrottle_is_throttled()helper extracted
fromshrink_inactive_list()and leave the legacy path untouched. The new MGLRU-onlythrottle_evictable_types()only sleeps when all evictable types are over-isolated - v3 throttled as soon as any of them was, which unnecessarily blocked the reclaim of the other type and passes the mask of the remaining types toisolate_folios(), which restricts both its initial choice and its fallback, so that isolation never lands on a throttled type. The v3 gate did not constrain the type actually isolated, so the fallback could still pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundantnr_isolatedcheck before restoring the NR_ISOLATED_* counters inevict_folios(). rename the extracted helper tothrottle_is_throttled()to avoid confusion with the existingwake_throttle_isolated()naming space. Usefor_each_evictable_type()in the MGLRU throttle to check each evictable type's isolation instead of only the type returned byget_type_to_scan(), sinceisolate_folios()may fall back to the other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable. Split into two patches; patch 2 is new and adds thetoo_many_isolated()throttling to the MGLRU eviction path, which v1 did not cover.
Add test infomations.include/linux/mmzone.h | 2 + mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 172 insertions(+), 14 deletions(-)include/linux/mmzone.h +2
-0@@ -564,6 +564,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the number of folios isolated from this lruvec and being reclaimed */ +atomic_long_t nr_isolated[ANON_AND_FILE ];/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */ mm/vmscan.c +7
-8@@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) +static unsigned long lruvec_type_evictable_size (struct lruvec *lruvec,int type) { -int gen, type, zone; +int gen, zone;unsigned long seq, total =0 ;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); -for_each_evictable_type (type, swappiness) { -for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen =lru_gen_from_seq (seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) - total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); - } +for (seq = min_seq[type]; seq <= max_seq; seq++) { + gen =lru_gen_from_seq (seq); +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); }return total; } +static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness)
mm/vmscan.c static void set_initial_priority(struct pglist_data *pgdat, struct scan_control · line 4362
Quoted 10 lines
mm/vmscan.c +10
-0+{ +unsigned long total =0 ; +int type; + +for_each_evictable_type (type, swappiness) + total +=lruvec_type_evictable_size (lruvec, type); + +return total; +} +
The helper split looks clean; it could be a stand alone commit I think.
Quoted 38 lines
mm/vmscan.c +31
-0static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) {unsigned long total;@@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness) swappiness ==SWAPPINESS_ANON_ONLY ; } +/* +* The MGLRU counterpart to too_many_isolated(). +* +* too_many_isolated() compares node-wide isolated counts against +* node-wide inactive sizes. It predates per-memcg LRU lists, which +* moved isolation to per-lruvec granularity. MGLRU reclaimers contend +* per lruvec, so this check is per lruvec as well, and isolation in +* one memcg does not throttle reclaim in another. +* +* The legacy threshold, the size of the inactive list, is not usable +* here either: in MGLRU the inactive counters are only a compatibility +* shim - they jump around when swap runs out or a memcg hits its swap +* limit, and proactive aging can inflate them. Compare against the +* total number of evictable pages of the type divided by MIN_NR_GENS, +* approximating the size of one generation - the same unit the +* original MGLRU aging heuristics used. +*/ +static bool lru_gen_too_many_isolated (struct lruvec *lruvec,int type, +struct scan_control *sc) +{ +unsigned long isolated, evictable; + +if (current_is_kswapd ()) +return false; + +if (!writeback_throttling_sane (sc)) +return false; + + isolated =atomic_long_read (&lruvec->lrugen.nr_isolated[type]); + evictable =lruvec_type_evictable_size (lruvec, type); +
mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5064
Quoted 10 lines
mm/vmscan.c +10
-0+/* +* GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so +* they won't be blocked by normal direct-reclaimers, forming a +* circular deadlock. +*/ +if (gfp_has_io_fs (sc->gfp_mask)) + evictable >>=3 ; + +return isolated > evictable /MIN_NR_GENS ; +}
The magic number of >>=3 looks a bit odd, others are fine to me.
Quoted 47 lines
mm/vmscan.c +47
-0+ +/* +* Unlike the legacy path, where the LRU list to isolate from is known +* before isolation, isolate_folios() picks the type from the refault +* feedback and may fall back to the other one. Therefore, instead of +* throttling on a single type, collect the evictable types that do not +* have too many isolated folios, and only sleep when all of them do. +* +* Returns the mask of the types isolate_folios() may isolate from, or +* 0 if reclaim should stop. Also sets @fatal to tell the caller that +* the task received a fatal signal while waiting. +*/ +static unsigned int throttle_evictable_types (struct lruvec *lruvec, +int swappiness, +struct scan_control *sc, +bool *fatal) +{ +unsigned int allowed; +bool stalled = false; +bool has_evictable; +struct pglist_data *pgdat =lruvec_pgdat (lruvec); +int i; + + *fatal = false; + +for (;;) { + allowed =0 ; + has_evictable = false; +for_each_evictable_type (i, swappiness) { +/* +* A type with no evictable folios has nothing to +* isolate. Don't allow it, otherwise reclaim can be +* redirected to it and spin making no progress, e.g. +* when the only type with folios is over-isolated. +*/ +if (!lruvec_type_evictable_size (lruvec, i)) +continue ; + + has_evictable = true; + +if (!lru_gen_too_many_isolated (lruvec, i, sc)) + allowed |=BIT (i); + } + +/* +* Nothing evictable at all is not over-isolation. Stop +* reclaim instead of waiting, so that nobody - including
mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5121
Quoted 10 lines
mm/vmscan.c +10
-0+* kswapd, which is exempt from the check below - stalls on +* an empty lruvec. +*/ +if (!has_evictable) +return 0 ; + +if (allowed) { +/* Wake up reclaimers waiting on the isolation to go down. */ +wake_throttle_isolated (pgdat); +return allowed;
This seems counterintuitive: we reach here because a reclaimer is starting, and we wake up even more reclaimer?
Quoted 11 lines
mm/vmscan.c +11
-0+ } + +/* +* All evictable types are over-isolated. Like the legacy +* path, wait once for concurrent reclaimers to put their +* isolated folios back; give up if that makes no progress. +*/ +if (stalled) +return 0 ; + + stalled = true;
mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5142
Quoted 10 lines
mm/vmscan.c +10
-0+reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ); + +/* We are about to die and free our memory. Return now. */ +if (fatal_signal_pending (current)) { + *fatal = true; +return 0 ; + } + } +} +
Hmm... this part looks a bit too complex, see below.
Quoted 43 lines
mm/vmscan.c +16
-4static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness, -struct list_head *list,int *isolated, -int *isolate_type,int *isolate_scanned) +unsigned int allowed,struct list_head *list, +int *isolated,int *isolate_type,int *isolate_scanned) {bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier;bool tried = false; +/* +* The preferred type may have been excluded by +* throttle_evictable_types(); start from the other one. +*/ +if (!(allowed &BIT (type))) + type = !type; + retry: tier =get_tier_idx (lruvec, type); scanned = scan_folios(nr_to_scan, lruvec, sc,@@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,/* * We are running out of the current reclaim type. Fall back to -* the other type if allowed. +* the other type if allowed. Only fall back to a type that is +* not throttled. */ -if (!scanned && !tried && type_fallback_allowed) { +if (!scanned && !tried && type_fallback_allowed && + (allowed &BIT (!type))) { type = !type; tried = true; type_fallback_allowed = false;@@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec); +unsigned int allowed; +bool fatal; +
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5224
Quoted 10 lines
mm/vmscan.c +10
-0+ allowed =throttle_evictable_types (lruvec, swappiness, sc, &fatal); +if (!allowed) { +/* +* We are about to die and free our memory. Like the legacy +* path, pretend some pages were reclaimed so reclaim +* unwinds quickly instead of looping back into the +* throttle. +*/ +if (fatal) + sc->nr_reclaimed +=SWAP_CLUSTER_MAX ;
And reporting a fake nr_reclaimed looks a bad thing to do in the first place.
It seems a lot of trouble is caused by the design of the reclaim loop, so will it work better if we calculate the number & type to reclaim first like CLRU?
There is a demonstration from Barry:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
And instead of throttle at the reclaim loop, balance the scan budge based on (size - isolation) number, and throttle a while LRU simply when the total isolated number is larger than total size / MIN_NR_GENS.
Just an idea for reference, seems could help get rid of a lot of complex logic.
2026/10/17 02:07, Kairui Song:
Quoted 484 lines · Hui Zhu
Quoted 100 lines
From: Hui Zhu <[email protected]>
The legacy path throttles direct reclaim via
too_many_isolated()when isolated folios pile up, but MGLRU'sevict_folios()isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.Add
lru_gen_too_many_isolated()andthrottle_evictable_types(), called fromevict_folios()before the lruvec lock is taken since throttling sleeps.The check is per lruvec rather than node-wide: reclaimers contend per lruvec, and a node-wide check would let isolation in one memcg throttle reclaim in another. The threshold is the evictable size of the type divided by
MIN_NR_GENS, approximating one generation.Since
isolate_folios()picks the type from refault feedback and may fall back to the other one,throttle_evictable_types()collects the types that are not over-isolated and have evictable folios, and sleeps only when all of them are over-isolated. The mask is passed toisolate_folios()to restrict its choice and fallback, so an over-isolated type never blocks the other, and isolation never lands on a throttled or empty type.An lruvec with no evictable folios at all is not over-isolation:
reclaim stops immediately instead of waiting. Without this, allowed == 0 would conflate "all types are empty" with "all types are over-isolated" and send the reclaimer to sleep, and since the kswapd exemption lives inside the per-type check, which empty types bypass, even kswapd would stall on an empty lruvec.Like the legacy path, kswapd and !writeback_throttling_sane() reclaim are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and a dying task fakes progress to exit reclaim quickly.
This builds on the recently merged
NR_ISOLATEDaccounting fix for the MGLRU reclaim path, which makes in-flight isolation visible in/proc/vmstat; the throttle itself uses a separate per-lruvec counter. Thenr_isolatednumbers in the test results below therefore come from that fix rather than from this patch.Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (
0x0007). "unpatched" and "patched" are the same tree built without/with this patch; every configuration is run with identical parameters on both kernels. Test scripts:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e341) Synthetic (
mglru_iso_repro.sh,mglru_iso_repro_v2.sh): threads write-touching a 48MB anonymous working set inside a 16MB memcg, with swap on a dm-delay device (50ms) to slow swap-out and lengthen the isolation window.64 threads, 60s - neither kernel OOMs at this concurrency; the gain shows in reclaim quality:
unpatched patched
OOM kills 0 0
total touches 246,499,132,369 301,456,426,692 (+22%)
scan efficiency 0.0261 0.0194
mm_vmscan_throttled0 35,645 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolatedpeak 0 230The patched kernel completes ~22% more work in the same 60s: the throttle keeps concurrent reclaimers from trampling the same generation, so less CPU is burned in reclaim.
192 threads, 60s - over-isolation becomes fatal without the throttle:
unpatched patched
OOM kills 1 (task killed) 0
memcg oom events 51 0
total touches 0 (killed) 682,300,039,725
mm_vmscan_throttled0 512,292
nr_isolatedpeak 0 335The unpatched kernel cannot keep reclaim ahead of allocation and the task is OOM-killed; the patched kernel survives the full run and keeps reclaim making progress.
2) Real workload (
mglru_build_repro.sh): a CI-style build pod - repeated from-scratch "make -j20 vmlinux" rounds inside a memory-limited cgroup, with the object tree on ext4 over dm-delay (10ms) to model slow storage. 20 concurrent compilers charge into one memcg and enter direct reclaim at the same time, while dirty object files stretch theisolate->shrink->putbackwindow. Each kernel runs 1800s per pressure level and the metrics are cumulative counters over that window, which smooths run-to-run noise. The headline metric is the number of objects actually compiled (a full round is ~3,800 objects).memory.max = 1024MB (extreme pressure):
unpatched patched
objects compiled 4,744 7,211 (+52%)
pgscan_direct2,901,083,521 2,228,776,807 (-23%)
scan efficiency 0.195 0.242
workingset_refault565,169,241 537,686,479
mm_vmscan_throttled0 511,441 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolatedpeak 0 1,152Quoted 10 lines
memory.max = 1536MB (heavy pressure):
unpatched patched
objects compiled 18,969 25,124 (+32%)
pgscan_direct1,223,905,508 816,812,074 (-33%)
scan efficiency 0.320 0.338
workingset_refault390,048,986 273,442,445 (-30%)
memcg charge fails 8,344,775 5,868,167 (-30%)
round wall time 199-391s 182-275s
mm_vmscan_throttled0 503,117
nr_isolatedpeak 0 1,053Hmm is this because of the type fallback? e.g. too many files isoalted -> scan anon instead / too many anon isolate -> scan file instead.
Quoted 33 lines
At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB of anon in total), so rounds end with a cc1 being memcg-OOM-killed on either kernel. Normalized per completed round the kill rate is the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the patched kernel simply reaches that point sooner because less time is burned thrashing, so it cycles through more rounds. This patch does not claim to avoid those OOMs - the point is that it performs 23-33% fewer futile LRU scans and gets 32-52% more build work done in the same wall time.
memory.max = 2048MB (light pressure - no-harm control):
unpatched patched
rounds completed 6 6
avg wall time/round 271s 270s
scan efficiency 0.984 0.978
mm_vmscan_throttled0 0
nr_isolatedpeak 0 144When reclaim keeps up the throttle never fires and both kernels perform identically, so the check costs nothing on a system that is not over-isolating.
3) Service no-impact (
mglru_redis_repro.shandmglru_redis_shards_repro.sh): a redis-server serving a 1KB-value dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) - first as a single instance (100k keys, 200 clients), then as 8 shards with AOF persistence on a second dm-delay device sharing one cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O serialized on redis's single command thread, not isolation
contention: charge-induced reclaim is a trickle (~400 scans/s vs 10^5-10^6 scans/s in the build runs), isolation never piles up, and the throttle correctly stays dormant on both kernels:Quoted 6 lines
unpatched patched
single instance 41.6 ops/s 41.2 ops/s
(avg 2999ms) (avg 2999ms)
8 shards aggregate 333-419 ops/s 310-358 ops/s
(mean 376, (mean 335,
4 runs) (*) 4 runs)Hmm, I know this could be noise but it looks like a regression?
Quoted 14 lines
mm_vmscan_throttled 0 0 (all runs) nr_isolated peak 0 0(*) the aggregate is dominated by a preload-end residency lottery (which shards' datasets happen to stay hot) and swings by up to 26% between runs of the SAME kernel; with 4 runs per kernel the difference between the means is within that noise (Welch t=1.95, p~0.11). The throttle never fired in any patched run (the tracepoint covers preload as well), so both kernels execute the same reclaim path here, and the kernel-side counters agree across all 8 runs:
steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs document the no-impact side: the check is dormant unless isolation actually piles up.Quoted 1 line
Signed-off-by Hui Zhu <[email protected]>Hi Hui
Looks like a good idea in general.
Quoted 105 lines
---
Changelog:
v7:
Patch 1 was merged into mm-unstable, so this is resent as a single patch.
According to the comments of Sashiko, Return instead of waiting when the lruvec has no evictable folios at all: allowed == 0 previously conflated "all empty" with "all over-isolated" and stalled even kswapd for 20ms. Move the v6 cover-letter test data into the commit message. v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a memory-limited cgroup with its object tree on dm-delay-backed storage, measured at three pressure levels (1024/1536/2048MB), plus redis-based cache service runs (single instance and 8 shards sharing one cgroup) as no-impact data points. New and updated test scripts are in the gist. Rebased onto the latest mm-stable; patch 2 is adapted to the currentisolate_folios()type-selection context (is_single_type_reclaim()), with no semantic change. Re-run the tests and update the test logs.
v5:
According to the comments of Kairun and feedback from the test scripts, make the throttle check per lruvec (newnr_isolatedcounter) and skip empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop thethrottle_is_throttled()helper extracted
fromshrink_inactive_list()and leave the legacy path untouched. The new MGLRU-onlythrottle_evictable_types()only sleeps when all evictable types are over-isolated - v3 throttled as soon as any of them was, which unnecessarily blocked the reclaim of the other type and passes the mask of the remaining types toisolate_folios(), which restricts both its initial choice and its fallback, so that isolation never lands on a throttled type. The v3 gate did not constrain the type actually isolated, so the fallback could still pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundantnr_isolatedcheck before restoring the NR_ISOLATED_* counters inevict_folios(). rename the extracted helper tothrottle_is_throttled()to avoid confusion with the existingwake_throttle_isolated()naming space. Usefor_each_evictable_type()in the MGLRU throttle to check each evictable type's isolation instead of only the type returned byget_type_to_scan(), sinceisolate_folios()may fall back to the other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable. Split into two patches; patch 2 is new and adds thetoo_many_isolated()throttling to the MGLRU eviction path, which v1 did not cover.
Add test infomations.include/linux/mmzone.h | 2 + mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 172 insertions(+), 14 deletions(-)include/linux/mmzone.h +2
-0@@ -564,6 +564,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the number of folios isolated from this lruvec and being reclaimed */ +atomic_long_t nr_isolated[ANON_AND_FILE ];/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */ mm/vmscan.c +7
-8@@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) +static unsigned long lruvec_type_evictable_size (struct lruvec *lruvec,int type) { -int gen, type, zone; +int gen, zone;unsigned long seq, total =0 ;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); -for_each_evictable_type (type, swappiness) { -for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen =lru_gen_from_seq (seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) - total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); - } +for (seq = min_seq[type]; seq <= max_seq; seq++) { + gen =lru_gen_from_seq (seq); +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); }return total; } +static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness)mm/vmscan.c static void set_initial_priority(struct pglist_data *pgdat, struct scan_control · line 4362
Quoted 10 lines
mm/vmscan.c +10
-0+{ +unsigned long total =0 ; +int type; + +for_each_evictable_type (type, swappiness) + total +=lruvec_type_evictable_size (lruvec, type); + +return total; +} +The helper split looks clean; it could be a stand alone commit I think.
Quoted 38 lines
mm/vmscan.c +31
-0static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) {unsigned long total;@@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness) swappiness ==SWAPPINESS_ANON_ONLY ; } +/* +* The MGLRU counterpart to too_many_isolated(). +* +* too_many_isolated() compares node-wide isolated counts against +* node-wide inactive sizes. It predates per-memcg LRU lists, which +* moved isolation to per-lruvec granularity. MGLRU reclaimers contend +* per lruvec, so this check is per lruvec as well, and isolation in +* one memcg does not throttle reclaim in another. +* +* The legacy threshold, the size of the inactive list, is not usable +* here either: in MGLRU the inactive counters are only a compatibility +* shim - they jump around when swap runs out or a memcg hits its swap +* limit, and proactive aging can inflate them. Compare against the +* total number of evictable pages of the type divided by MIN_NR_GENS, +* approximating the size of one generation - the same unit the +* original MGLRU aging heuristics used. +*/ +static bool lru_gen_too_many_isolated (struct lruvec *lruvec,int type, +struct scan_control *sc) +{ +unsigned long isolated, evictable; + +if (current_is_kswapd ()) +return false; + +if (!writeback_throttling_sane (sc)) +return false; + + isolated =atomic_long_read (&lruvec->lrugen.nr_isolated[type]); + evictable =lruvec_type_evictable_size (lruvec, type); +mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5064
Quoted 10 lines
mm/vmscan.c +10
-0+/* +* GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so +* they won't be blocked by normal direct-reclaimers, forming a +* circular deadlock. +*/ +if (gfp_has_io_fs (sc->gfp_mask)) + evictable >>=3 ; + +return isolated > evictable /MIN_NR_GENS ; +}The magic number of >>=3 looks a bit odd, others are fine to me.
Quoted 47 lines
mm/vmscan.c +47
-0+ +/* +* Unlike the legacy path, where the LRU list to isolate from is known +* before isolation, isolate_folios() picks the type from the refault +* feedback and may fall back to the other one. Therefore, instead of +* throttling on a single type, collect the evictable types that do not +* have too many isolated folios, and only sleep when all of them do. +* +* Returns the mask of the types isolate_folios() may isolate from, or +* 0 if reclaim should stop. Also sets @fatal to tell the caller that +* the task received a fatal signal while waiting. +*/ +static unsigned int throttle_evictable_types (struct lruvec *lruvec, +int swappiness, +struct scan_control *sc, +bool *fatal) +{ +unsigned int allowed; +bool stalled = false; +bool has_evictable; +struct pglist_data *pgdat =lruvec_pgdat (lruvec); +int i; + + *fatal = false; + +for (;;) { + allowed =0 ; + has_evictable = false; +for_each_evictable_type (i, swappiness) { +/* +* A type with no evictable folios has nothing to +* isolate. Don't allow it, otherwise reclaim can be +* redirected to it and spin making no progress, e.g. +* when the only type with folios is over-isolated. +*/ +if (!lruvec_type_evictable_size (lruvec, i)) +continue ; + + has_evictable = true; + +if (!lru_gen_too_many_isolated (lruvec, i, sc)) + allowed |=BIT (i); + } + +/* +* Nothing evictable at all is not over-isolation. Stop +* reclaim instead of waiting, so that nobody - including mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5121
Quoted 10 lines
mm/vmscan.c +10
-0+* kswapd, which is exempt from the check below - stalls on +* an empty lruvec. +*/ +if (!has_evictable) +return 0 ; + +if (allowed) { +/* Wake up reclaimers waiting on the isolation to go down. */ +wake_throttle_isolated (pgdat); +return allowed;This seems counterintuitive: we reach here because a reclaimer is starting, and we wake up even more reclaimer?
Quoted 11 lines
mm/vmscan.c +11
-0+ } + +/* +* All evictable types are over-isolated. Like the legacy +* path, wait once for concurrent reclaimers to put their +* isolated folios back; give up if that makes no progress. +*/ +if (stalled) +return 0 ; + + stalled = true;mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5142
Quoted 10 lines
mm/vmscan.c +10
-0+reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ); + +/* We are about to die and free our memory. Return now. */ +if (fatal_signal_pending (current)) { + *fatal = true; +return 0 ; + } + } +} +Hmm... this part looks a bit too complex, see below.
Quoted 43 lines
mm/vmscan.c +16
-4static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness, -struct list_head *list,int *isolated, -int *isolate_type,int *isolate_scanned) +unsigned int allowed,struct list_head *list, +int *isolated,int *isolate_type,int *isolate_scanned) {bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier;bool tried = false; +/* +* The preferred type may have been excluded by +* throttle_evictable_types(); start from the other one. +*/ +if (!(allowed &BIT (type))) + type = !type; + retry: tier =get_tier_idx (lruvec, type); scanned = scan_folios(nr_to_scan, lruvec, sc,@@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,/* * We are running out of the current reclaim type. Fall back to -* the other type if allowed. +* the other type if allowed. Only fall back to a type that is +* not throttled. */ -if (!scanned && !tried && type_fallback_allowed) { +if (!scanned && !tried && type_fallback_allowed && + (allowed &BIT (!type))) { type = !type; tried = true; type_fallback_allowed = false;@@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec); +unsigned int allowed; +bool fatal; +mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5224
Quoted 10 lines
mm/vmscan.c +10
-0+ allowed =throttle_evictable_types (lruvec, swappiness, sc, &fatal); +if (!allowed) { +/* +* We are about to die and free our memory. Like the legacy +* path, pretend some pages were reclaimed so reclaim +* unwinds quickly instead of looping back into the +* throttle. +*/ +if (fatal) + sc->nr_reclaimed +=SWAP_CLUSTER_MAX ;And reporting a fake
nr_reclaimedlooks a bad thing to do in the first place.It seems a lot of trouble is caused by the design of the reclaim loop, so will it work better if we calculate the number & type to reclaim first like CLRU?
There is a demonstration from Barry:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
Quoted 6 lines
And instead of throttle at the reclaim loop, balance the scan budge based on (size - isolation) number, and throttle a while LRU simply when the total isolated number is larger than total size /
MIN_NR_GENS.Just an idea for reference, seems could help get rid of a lot of complex logic.
Hi Kairui,
Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and I am quite interested in it.
I think that branch still needs an over-isolation guard. The scan budget answers "which type and how much", but it is computed once per try_to_shrink_lruvec() and does not see what other reclaimers are holding. Each reclaimer's in-flight isolation is bounded by the batch size rather than by the budget, so N reclaimers entering together all see the same state and all proceed.
On the other hand, that branch would also make the guard much simpler and clearer. Since evict_folios() already takes the type from its caller, the allowed mask, the type rewrite and the fallback restriction in isolate_folios() all go away, and the throttle reduces to a single check on the totals in the shrink loop, which is basically the second half of what you suggested.
So I would like to ask what the development plan for this branch is, and whether you think it is worth moving the MGLRU too_many_isolated() work over to it.
Best,
Hui
Barry Song
reviewerOn Tue, Oct 20, 2026 at 5:29 PM Hui Zhu <[email protected]> wrote:
[...]
Quoted 34 lines
Quoted 8 lines
It seems a lot of trouble is caused by the design of the reclaim loop, so will it work better if we calculate the number & type to reclaim first like CLRU?
There is a demonstration from Barry:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3Quoted 6 lines
And instead of throttle at the reclaim loop, balance the scan budge based on (size - isolation) number, and throttle a while LRU simply when the total isolated number is larger than total size /
MIN_NR_GENS.Just an idea for reference, seems could help get rid of a lot of complex logic.
Hi Kairui,
Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and I am quite interested in it.
I think that branch still needs an over-isolation guard. The scan budget answers "which type and how much", but it is computed once per
try_to_shrink_lruvec()and does not see what other reclaimers are holding. Each reclaimer's in-flight isolation is bounded by the batch size rather than by the budget, so N reclaimers entering together all see the same state and all proceed.On the other hand, that branch would also make the guard much simpler and clearer. Since
evict_folios()already takes the type from its caller, the allowed mask, the type rewrite and the fallback restriction inisolate_folios()all go away, and the throttle reduces to a single check on the totals in the shrink loop, which is basically the second half of what you suggested.
Quoted 3 lines
So I would like to ask what the development plan for this branch is, and whether you think it is worth moving the MGLRU
too_many_isolated()work over to it.
FYI, here is the RFC:
https://lore.kernel.org/linux-mm/[email protected]/
Best Regards
Barry
Quoted 2 lines · Hui Zhu
2026/10/17 02:07, Kairui Song:
Quoted 1 line · Hui Zhu
...
Quoted 344 lines
Quoted 332 lines
Hmm, I know this could be noise but it looks like a regression?
Quoted 14 lines
mm_vmscan_throttled 0 0 (all runs) nr_isolated peak 0 0(*) the aggregate is dominated by a preload-end residency lottery (which shards' datasets happen to stay hot) and swings by up to 26% between runs of the SAME kernel; with 4 runs per kernel the difference between the means is within that noise (Welch t=1.95, p~0.11). The throttle never fired in any patched run (the tracepoint covers preload as well), so both kernels execute the same reclaim path here, and the kernel-side counters agree across all 8 runs:
steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs document the no-impact side: the check is dormant unless isolation actually piles up.Quoted 1 line
Signed-off-by Hui Zhu <[email protected]>Hi Hui
Looks like a good idea in general.
Quoted 105 lines
---
Changelog:
v7:
Patch 1 was merged into mm-unstable, so this is resent as a single patch.
According to the comments of Sashiko, Return instead of waiting when the lruvec has no evictable folios at all: allowed == 0 previously conflated "all empty" with "all over-isolated" and stalled even kswapd for 20ms. Move the v6 cover-letter test data into the commit message. v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a memory-limited cgroup with its object tree on dm-delay-backed storage, measured at three pressure levels (1024/1536/2048MB), plus redis-based cache service runs (single instance and 8 shards sharing one cgroup) as no-impact data points. New and updated test scripts are in the gist. Rebased onto the latest mm-stable; patch 2 is adapted to the currentisolate_folios()type-selection context (is_single_type_reclaim()), with no semantic change. Re-run the tests and update the test logs.
v5:
According to the comments of Kairun and feedback from the test scripts, make the throttle check per lruvec (newnr_isolatedcounter) and skip empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop thethrottle_is_throttled()helper extracted
fromshrink_inactive_list()and leave the legacy path untouched. The new MGLRU-onlythrottle_evictable_types()only sleeps when all evictable types are over-isolated - v3 throttled as soon as any of them was, which unnecessarily blocked the reclaim of the other type and passes the mask of the remaining types toisolate_folios(), which restricts both its initial choice and its fallback, so that isolation never lands on a throttled type. The v3 gate did not constrain the type actually isolated, so the fallback could still pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundantnr_isolatedcheck before restoring the NR_ISOLATED_* counters inevict_folios(). rename the extracted helper tothrottle_is_throttled()to avoid confusion with the existingwake_throttle_isolated()naming space. Usefor_each_evictable_type()in the MGLRU throttle to check each evictable type's isolation instead of only the type returned byget_type_to_scan(), sinceisolate_folios()may fall back to the other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable. Split into two patches; patch 2 is new and adds thetoo_many_isolated()throttling to the MGLRU eviction path, which v1 did not cover.
Add test infomations.include/linux/mmzone.h | 2 + mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 172 insertions(+), 14 deletions(-)include/linux/mmzone.h +2
-0@@ -564,6 +564,8 @@ struct lru_gen_folio {/* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ];atomic_long_t refaulted[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS ]; +/* the number of folios isolated from this lruvec and being reclaimed */ +atomic_long_t nr_isolated[ANON_AND_FILE ];/* whether the multi-gen LRU is enabled */ bool enabled;/* the memcg generation this lru_gen_folio belongs to */ mm/vmscan.c +7
-8@@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority =clamp (priority,DEF_PRIORITY /2 ,DEF_PRIORITY ); } -static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness) +static unsigned long lruvec_type_evictable_size (struct lruvec *lruvec,int type) { -int gen, type, zone; +int gen, zone;unsigned long seq, total =0 ;struct lru_gen_folio *lrugen = &lruvec->lrugen;DEFINE_MAX_SEQ (lruvec);DEFINE_MIN_SEQ (lruvec); -for_each_evictable_type (type, swappiness) { -for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen =lru_gen_from_seq (seq); -for (zone =0 ; zone <MAX_NR_ZONES ; zone++) - total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); - } +for (seq = min_seq[type]; seq <= max_seq; seq++) { + gen =lru_gen_from_seq (seq); +for (zone =0 ; zone <MAX_NR_ZONES ; zone++) + total +=max (READ_ONCE (lrugen->nr_pages[gen][type][zone]),0L ); }return total; } +static unsigned long lruvec_evictable_size (struct lruvec *lruvec,int swappiness)mm/vmscan.c static void set_initial_priority(struct pglist_data *pgdat, struct scan_control · line 4362
Quoted 10 lines
mm/vmscan.c +10
-0+{ +unsigned long total =0 ; +int type; + +for_each_evictable_type (type, swappiness) + total +=lruvec_type_evictable_size (lruvec, type); + +return total; +} +The helper split looks clean; it could be a stand alone commit I think.
Quoted 38 lines
mm/vmscan.c +31
-0static bool lruvec_is_sizable (struct lruvec *lruvec,struct scan_control *sc) {unsigned long total;@@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness) swappiness ==SWAPPINESS_ANON_ONLY ; } +/* +* The MGLRU counterpart to too_many_isolated(). +* +* too_many_isolated() compares node-wide isolated counts against +* node-wide inactive sizes. It predates per-memcg LRU lists, which +* moved isolation to per-lruvec granularity. MGLRU reclaimers contend +* per lruvec, so this check is per lruvec as well, and isolation in +* one memcg does not throttle reclaim in another. +* +* The legacy threshold, the size of the inactive list, is not usable +* here either: in MGLRU the inactive counters are only a compatibility +* shim - they jump around when swap runs out or a memcg hits its swap +* limit, and proactive aging can inflate them. Compare against the +* total number of evictable pages of the type divided by MIN_NR_GENS, +* approximating the size of one generation - the same unit the +* original MGLRU aging heuristics used. +*/ +static bool lru_gen_too_many_isolated (struct lruvec *lruvec,int type, +struct scan_control *sc) +{ +unsigned long isolated, evictable; + +if (current_is_kswapd ()) +return false; + +if (!writeback_throttling_sane (sc)) +return false; + + isolated =atomic_long_read (&lruvec->lrugen.nr_isolated[type]); + evictable =lruvec_type_evictable_size (lruvec, type); +mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5064
Quoted 10 lines
mm/vmscan.c +10
-0+/* +* GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so +* they won't be blocked by normal direct-reclaimers, forming a +* circular deadlock. +*/ +if (gfp_has_io_fs (sc->gfp_mask)) + evictable >>=3 ; + +return isolated > evictable /MIN_NR_GENS ; +}The magic number of >>=3 looks a bit odd, others are fine to me.
Quoted 47 lines
mm/vmscan.c +47
-0+ +/* +* Unlike the legacy path, where the LRU list to isolate from is known +* before isolation, isolate_folios() picks the type from the refault +* feedback and may fall back to the other one. Therefore, instead of +* throttling on a single type, collect the evictable types that do not +* have too many isolated folios, and only sleep when all of them do. +* +* Returns the mask of the types isolate_folios() may isolate from, or +* 0 if reclaim should stop. Also sets @fatal to tell the caller that +* the task received a fatal signal while waiting. +*/ +static unsigned int throttle_evictable_types (struct lruvec *lruvec, +int swappiness, +struct scan_control *sc, +bool *fatal) +{ +unsigned int allowed; +bool stalled = false; +bool has_evictable; +struct pglist_data *pgdat =lruvec_pgdat (lruvec); +int i; + + *fatal = false; + +for (;;) { + allowed =0 ; + has_evictable = false; +for_each_evictable_type (i, swappiness) { +/* +* A type with no evictable folios has nothing to +* isolate. Don't allow it, otherwise reclaim can be +* redirected to it and spin making no progress, e.g. +* when the only type with folios is over-isolated. +*/ +if (!lruvec_type_evictable_size (lruvec, i)) +continue ; + + has_evictable = true; + +if (!lru_gen_too_many_isolated (lruvec, i, sc)) + allowed |=BIT (i); + } + +/* +* Nothing evictable at all is not over-isolation. Stop +* reclaim instead of waiting, so that nobody - including mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5121
Quoted 10 lines
mm/vmscan.c +10
-0+* kswapd, which is exempt from the check below - stalls on +* an empty lruvec. +*/ +if (!has_evictable) +return 0 ; + +if (allowed) { +/* Wake up reclaimers waiting on the isolation to go down. */ +wake_throttle_isolated (pgdat); +return allowed;This seems counterintuitive: we reach here because a reclaimer is starting, and we wake up even more reclaimer?
Quoted 11 lines
mm/vmscan.c +11
-0+ } + +/* +* All evictable types are over-isolated. Like the legacy +* path, wait once for concurrent reclaimers to put their +* isolated folios back; give up if that makes no progress. +*/ +if (stalled) +return 0 ; + + stalled = true;mm/vmscan.c static inline bool is_single_type_reclaim(int swappiness) · line 5142
Quoted 10 lines
mm/vmscan.c +10
-0+reclaim_throttle (pgdat,VMSCAN_THROTTLE_ISOLATED ); + +/* We are about to die and free our memory. Return now. */ +if (fatal_signal_pending (current)) { + *fatal = true; +return 0 ; + } + } +} +Hmm... this part looks a bit too complex, see below.
Quoted 43 lines
mm/vmscan.c +16
-4static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness, -struct list_head *list,int *isolated, -int *isolate_type,int *isolate_scanned) +unsigned int allowed,struct list_head *list, +int *isolated,int *isolate_type,int *isolate_scanned) {bool type_fallback_allowed = !is_single_type_reclaim (swappiness);int type =get_type_to_scan (lruvec, swappiness);int total_scanned =0 , scanned, tier;bool tried = false; +/* +* The preferred type may have been excluded by +* throttle_evictable_types(); start from the other one. +*/ +if (!(allowed &BIT (type))) + type = !type; + retry: tier =get_tier_idx (lruvec, type); scanned = scan_folios(nr_to_scan, lruvec, sc,@@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,/* * We are running out of the current reclaim type. Fall back to -* the other type if allowed. +* the other type if allowed. Only fall back to a type that is +* not throttled. */ -if (!scanned && !tried && type_fallback_allowed) { +if (!scanned && !tried && type_fallback_allowed && + (allowed &BIT (!type))) { type = !type; tried = true; type_fallback_allowed = false;@@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,bool skip_retry = false;struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec); +unsigned int allowed; +bool fatal; +mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · line 5224
Quoted 10 lines
mm/vmscan.c +10
-0+ allowed =throttle_evictable_types (lruvec, swappiness, sc, &fatal); +if (!allowed) { +/* +* We are about to die and free our memory. Like the legacy +* path, pretend some pages were reclaimed so reclaim +* unwinds quickly instead of looping back into the +* throttle. +*/ +if (fatal) + sc->nr_reclaimed +=SWAP_CLUSTER_MAX ;And reporting a fake
nr_reclaimedlooks a bad thing to do in the first place.It seems a lot of trouble is caused by the design of the reclaim loop, so will it work better if we calculate the number & type to reclaim first like CLRU?
There is a demonstration from Barry:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3Quoted 6 lines
And instead of throttle at the reclaim loop, balance the scan budge based on (size - isolation) number, and throttle a while LRU simply when the total isolated number is larger than total size /
MIN_NR_GENS.Just an idea for reference, seems could help get rid of a lot of complex logic.
Hi Kairui,
Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and I am quite interested in it.
Quoted 6 lines
I think that branch still needs an over-isolation guard. The scan budget answers "which type and how much", but it is computed once per
try_to_shrink_lruvec()and does not see what other reclaimers are holding. Each reclaimer's in-flight isolation is bounded by the batch size rather than by the budget, so N reclaimers entering together all see the same state and all proceed.
Yeah, exactly.
Quoted 10 lines
On the other hand, that branch would also make the guard much simpler and clearer. Since
evict_folios()already takes the type from its caller, the allowed mask, the type rewrite and the fallback restriction inisolate_folios()all go away, and the throttle reduces to a single check on the totals in the shrink loop, which is basically the second half of what you suggested.So I would like to ask what the development plan for this branch is, and whether you think it is worth moving the MGLRU
too_many_isolated()work over to it.
Thanks for looking into that!
I think the isolation protection is definitely needed; really appreciate your work!
However, many things are currently happening in parallel for MGLRU. Will the code be cleaner if worked on based on that branch? And MGLRU's budge will be more similiar with CLRU, so I hope maybe it can even be unified with CLRU for this isolation protection, just with a different gating (size / MIN_NR_GEN vs size of inactive), will be much lower LOC.