mm/vmscan: fix NR_ISOLATED accounting and throttle MGLRU eviction

From: Hui Zhu <[email protected]>

The legacy reclaim path has two mechanisms around isolated folios that MGLRU lacks:

1. NR_ISOLATED_ANON/FILE counters are updated when folios are isolated from the inactive lists. Compaction's too_many_isolated() relies on them to decide when to back off. The MGLRU reclaim path never updates them, so compaction cannot see MGLRU's in-flight isolation.

2. shrink_inactive_list() throttles direct reclaim via too_many_isolated() when isolated folios pile up. MGLRU's evict_folios() isolates folios without any such check, so many concurrent reclaimers can over-isolate the same (oldest) generation, leading to unnecessary swapping, thrashing and premature memcg OOM.

Patch 1 fixes the counter accounting in the MGLRU isolation path. Patch 2 adds a per-lruvec throttle, mirroring the legacy behavior but adapted to MGLRU's per-lruvec contention and its dynamic type selection/fallback.

Testing
=======
Two test scripts are provided to reproduce the problem and validate the fix. Both are available at:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34
Test environment: 10 CPU / 8GB QEMU guest, MGLRU enabled, a 16MB memory cgroup, anonymous working set, swap backed by dm-delay (50ms read/write delay) to slow swap-out and lengthen the isolation window.

mglru_iso_repro.sh (64 threads, 48MB working set, 60s):
Drives concurrent direct reclaim inside the memcg and measures scan efficiency, throttle events, in-flight isolation and throughput. Neither kernel OOMs at this concurrency; the value of the patch shows in reclaim quality:

                        unpatched       patched
  OOM kills             0               0
  mm_vmscan_throttled   0               35645 (all
                                        VMSCAN_THROTTLE_ISOLATED)
  nr_isolated peak      0 (invisible)   230
  total touches         246,499,132,369 301,456,426,692
  scan efficiency       0.0261          0.0194

The patched kernel completes ~22% more work in the same 60s: the throttle keeps concurrent reclaimers from trampling the same generation, so less CPU is burned in reclaim. Note nr_isolated is always 0 on the unpatched kernel - the over-isolation is invisible there, which is exactly what patch 1 fixes.

mglru_iso_repro_v2.sh (192 threads, 48MB working set, 60s):
Raises concurrency to the point where over-isolation becomes fatal:
unpatched patched

  OOM kills             1 (task killed) 0
  memcg oom events      51              0
  mm_vmscan_throttled   0               512292 (all

VMSCAN_THROTTLE_ISOLATED)
total touches 0 (killed) 682,300,003,972

With 192 threads the unpatched kernel cannot keep reclaim ahead of allocation and the task is OOM-killed; the patched kernel survives the full run and keeps reclaim making progress.

Changelog:
v5:
According to the comments of Kairun and feedback from the test scripts, make the throttle check per lruvec (new nr_isolated counter) and skip empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop the throttle_is_throttled() helper extracted
from shrink_inactive_list() and leave the legacy path untouched. The new MGLRU-only throttle_evictable_types() only sleeps when all evictable types are over-isolated - v3 throttled as soon as any of them was, which unnecessarily blocked the reclaim of the other type and passes the mask of the remaining types to isolate_folios(), which restricts both its initial choice and its fallback, so that isolation never lands on a throttled type. The v3 gate did not constrain the type actually isolated, so the fallback could still pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundant nr_isolated check before restoring the NR_ISOLATED_* counters in evict_folios(). rename the extracted helper to throttle_is_throttled() to avoid confusion with the existing wake_throttle_isolated() naming space. Use for_each_evictable_type() in the MGLRU throttle to check each evictable type's isolation instead of only the type returned by get_type_to_scan(), since isolate_folios() may fall back to the other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable. Split into two patches; patch 2 is new and adds the too_many_isolated() throttling to the MGLRU eviction path, which v1 did not cover.
Add test infomations.

Hui Zhu (2):
mm/vmscan: fix missing NR_ISOLATED counter update in MGLRU reclaim
path
mm/vmscan: throttle MGLRU eviction when isolated folios pile up

 include/linux/mmzone.h |   2 +
 mm/vmscan.c            | 173 +++++++++++++++++++++++++++++++++++++----
 2 files changed, 162 insertions(+), 13 deletions(-)

2.53.0

From: Hui Zhu <[email protected]>

MGLRU evict_folios() isolates folios from the LRU without updating the NR_ISOLATED_ANON/FILE counters, unlike the legacy shrink_inactive_list() path. This causes compaction's too_many_isolated() check to under-count isolated pages when MGLRU reclaim is active.

Add NR_ISOLATED counter updates in evict_folios(): increment after isolate_folios() and decrement after all retry passes complete, using the existing nr_isolated which holds the original isolated count.

---
 mm/vmscan.c | 5 +++++
 1 file changed, 5 insertions(+)

index c1404a59523d..98226bb021f3 100644
@@ -4892,6 +4892,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness,
 				 &list, &isolated, &type, &type_scanned);
 	nr_isolated = isolated;
+	if (nr_isolated)
+		__mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
+				      nr_isolated);
 
 	/* Scanning may have emptied the oldest gen, flush it */
 	if (scanned)

@@ -4954,6 +4957,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 		goto retry;
 	}
 
+	mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, -nr_isolated);
+
 	if (nr_isolated > total_reclaimed)
 		mod_lruvec_state(lruvec, PGROTATE_ANON + type,
 				 nr_isolated - total_reclaimed);

2.53.0

From: Hui Zhu <[email protected]>

The legacy path throttles direct reclaim via too_many_isolated() when isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.

Add lru_gen_too_many_isolated() and throttle_evictable_types(), called from evict_folios() before the lruvec lock is taken since throttling sleeps.

The check is per lruvec rather than node-wide: reclaimers contend per lruvec, and a node-wide check would let isolation in one memcg throttle reclaim in another. The threshold is the evictable size of the type divided by MIN_NR_GENS, approximating one generation.

Since isolate_folios() picks the type from refault feedback and may fall back to the other one, throttle_evictable_types() collects the types that are not over-isolated and have evictable folios, and sleeps only when all of them are over-isolated. The mask is passed to isolate_folios() to restrict its choice and fallback, so an over-isolated type never blocks the other, and isolation never lands on a throttled or empty type.

Like the legacy path, kswapd and !writeback_throttling_sane() reclaim are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and a dying task fakes progress to exit reclaim quickly.

Tested with threads write-touching a 48MB anonymous working set in a 16MB memcg backed by swap delayed 50ms via dm-delay. With 192 threads the task is OOM-killed without this patch and survives with it. With 64 threads both kernels survive the 60s run, and the patched kernel completes ~22% more touches (301B vs 246B).

---
 include/linux/mmzone.h |   2 +
 mm/vmscan.c            | 170 +++++++++++++++++++++++++++++++++++++----
 2 files changed, 158 insertions(+), 14 deletions(-)

index 94f9c3ff5416..819b6c059116 100644
@@ -580,6 +580,8 @@ struct lru_gen_folio {
 	/* can be modified without holding the LRU lock */
 	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
+	/* the number of folios isolated from this lruvec and being reclaimed */
+	atomic_long_t nr_isolated[ANON_AND_FILE];
 	/* whether the multi-gen LRU is enabled */
 	bool enabled;
 	/* the memcg generation this lru_gen_folio belongs to */

2.53.0