mm/vmscan: fix NR_ISOLATED accounting and throttling for MGLRU

From: Hui Zhu <[email protected]>

MGLRU evict_folios() isolates folios from the LRU without updating the NR_ISOLATED_ANON/FILE counters, unlike the legacy shrink_inactive_list() path. This causes compaction's too_many_isolated() check to under-count isolated pages when MGLRU reclaim is active.

Add NR_ISOLATED counter updates in evict_folios(): increment after isolate_folios() and decrement after all retry passes complete, using the existing nr_isolated which holds the original isolated count.

Signed-off-by Hui Zhu <[email protected]>

Reviewed-by Baolin Wang <[email protected]>

---

From: Hui Zhu <[email protected]>

The legacy path throttles direct reclaim in shrink_inactive_list() when too many isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.

With the NR_ISOLATED counters now updated in evict_folios(), extract the throttling loop from shrink_inactive_list() into throttle_is_throttled() and reuse it in evict_folios(). Since the type to isolate is unknown until isolation and isolate_folios() may fall back to the other type, check all evictable types with for_each_evictable_type() and throttle if any of them has too many isolated folios.

If a fatal signal is pending, fake reclaim progress the same way the legacy path does, so the dying task exits reclaim quickly instead of being held in the throttle.

Signed-off-by Hui Zhu <[email protected]>

---
 mm/vmscan.c | 71 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 59 insertions(+), 12 deletions(-)

mm/vmscan.c +4 -1

index 98226bb021f3..6fe8824430ac 100644
@@ -1819,7 +1819,7 @@ bool folio_isolate_lru(struct folio *folio)
  * the LRU list will go small and be scanned faster than necessary, leading to
  * unnecessary swapping, thrashing and OOM.
  */
-static bool too_many_isolated(struct pglist_data *pgdat, int file,
+static bool too_many_isolated(struct pglist_data *pgdat, bool file,
                struct scan_control *sc)
 {
        unsigned long inactive, isolated;

@@ -1856,6 +1856,37 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,
        return too_many;
 }
 
+/*
+ * Throttle reclaim if too many isolated folios are piling up. If this makes
+ * no progress, the caller is probably looping on unevictable folios, so give

Quoted 8 lines · Hui Zhu

From: Hui Zhu <[email protected]>

The legacy path throttles direct reclaim in shrink_inactive_list() when too many isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.

Quoted 8 lines

With the NR_ISOLATED counters now updated in evict_folios(), extract the throttling loop from shrink_inactive_list() into throttle_is_throttled() and reuse it in evict_folios(). Since the type to isolate is unknown until isolation and isolate_folios() may fall back to the other type, check all evictable types with for_each_evictable_type() and throttle if any of them has too many isolated folios.

I feel this is unlikely to work. MGLRU behaves quite differently from the active/inactive LRU, and its NR_INACTIVE_FILE accounting is also very different.

With the active/inactive LRU, shrink_inactive_list() ensures that we always have an inactive list with pages available for reclaim. With MGLRU, however, a generation can legitimately point to an empty list, so this assumption does not hold.

try_to_inc_min_seq:

         /* see the comment on lru_gen_folio */
         if (swappiness && swappiness <= MAX_SWAPPINESS) {
                 unsigned long seq = lrugen->max_seq - MIN_NR_GENS;

                 if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
                         min_seq[LRU_GEN_ANON] = seq;
                 else if (min_seq[LRU_GEN_FILE] > seq &&
min_seq[LRU_GEN_ANON] < seq)
                         min_seq[LRU_GEN_FILE] = seq;
         }

At that point, we have no inactive pages for the type, so the throttle will take effect when the following condition is true:

too_many = isolated > inactive;

With MGLRU, however, we can still fall back to the other type even when there are no inactive pages for the current type.

BTW, if we are hitting isolated > inactive with MGLRU, it probably means the generations are quite imbalanced—we are running out of reclaimable generations. In that case, we may actually want reclamation to proceed with aging instead.

Quoted 26 lines

If a fatal signal is pending, fake reclaim progress the same way the legacy path does, so the dying task exits reclaim quickly instead of being held in the throttle.

Signed-off-by Hui Zhu <[email protected]>

---
  mm/vmscan.c | 71 ++++++++++++++++++++++++++++++++++++++++++++---------
  1 file changed, 59 insertions(+), 12 deletions(-)

mm/vmscan.c +1 -1

index 98226bb021f3..6fe8824430ac 100644
@@ -1819,7 +1819,7 @@ bool folio_isolate_lru(struct folio *folio)
   * the LRU list will go small and be scanned faster than necessary, leading to
   * unnecessary swapping, thrashing and OOM.
   */
-static bool too_many_isolated(struct pglist_data *pgdat, int file,
+static bool too_many_isolated(struct pglist_data *pgdat, bool file,
                 struct scan_control *sc)
  {
         unsigned long inactive, isolated;

@@ -1856,6 +1856,37 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,
         return too_many;
  }
 

mm/vmscan.c static bool too_many_isolated(struct pglist_data *pgdat, int file, · line 1871

Quoted 13 lines

mm/vmscan.c +13 -0

+/*
+ * Throttle reclaim if too many isolated folios are piling up. If this makes
+ * no progress, the caller is probably looping on unevictable folios, so give
+ * up. Returns true to tell the caller to stop reclaiming, and sets @fatal
+ * if the task received a fatal signal while waiting, so that the caller can
+ * bail out faster.
+ */
+static bool throttle_is_throttled(struct pglist_data *pgdat, bool file,
+                                 struct scan_control *sc, bool *fatal)
+{
+       bool stalled = false;
+
+       *fatal = false;

TBH, I find the name quite weird :-)

Quoted 40 lines · Barry Song

Quoted 8 lines · Hui Zhu

From: Hui Zhu <[email protected]>

The legacy path throttles direct reclaim in shrink_inactive_list() when too many isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.

Quoted 8 lines

With the NR_ISOLATED counters now updated in evict_folios(), extract the throttling loop from shrink_inactive_list() into throttle_is_throttled() and reuse it in evict_folios(). Since the type to isolate is unknown until isolation and isolate_folios() may fall back to the other type, check all evictable types with for_each_evictable_type() and throttle if any of them has too many isolated folios.

I feel this is unlikely to work. MGLRU behaves quite differently from the active/inactive LRU, and its NR_INACTIVE_FILE accounting is also very different.

With the active/inactive LRU, shrink_inactive_list() ensures that we always have an inactive list with pages available for reclaim. With MGLRU, however, a generation can legitimately point to an empty list, so this assumption does not hold.

try_to_inc_min_seq:

         /* see the comment on lru_gen_folio */
         if (swappiness && swappiness <= MAX_SWAPPINESS) {
                 unsigned long seq = lrugen->max_seq - MIN_NR_GENS;

                 if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
                         min_seq[LRU_GEN_ANON] = seq;
                 else if (min_seq[LRU_GEN_FILE] > seq &&
min_seq[LRU_GEN_ANON] < seq)
                         min_seq[LRU_GEN_FILE] = seq;
         }

Quoted 7 lines

At that point, we have no inactive pages for the type, so the throttle will take effect when the following condition is true:

too_many = isolated > inactive;

With MGLRU, however, we can still fall back to the other type even when there are no inactive pages for the current type.

Yes, that's a valid concern. So I think we can check the isolation of both types for MGLRU to avoid this case:

The typical 'isolated > inactive' case is that we've tried our best with aging, but cold pages production can't keep up with isolation speed, especially under concurrent reclaim from multiple processes. In this case, I think throttling is reasonable.

Quoted 41 lines · Baolin Wang

Quoted 40 lines · Barry Song

Quoted 8 lines · Hui Zhu

From: Hui Zhu <[email protected]>

The legacy path throttles direct reclaim in shrink_inactive_list() when too many isolated folios pile up, but MGLRU's evict_folios() isolates folios without this check, which can lead to unnecessary swapping, thrashing and OOM.

Quoted 8 lines

With the NR_ISOLATED counters now updated in evict_folios(), extract the throttling loop from shrink_inactive_list() into throttle_is_throttled() and reuse it in evict_folios(). Since the type to isolate is unknown until isolation and isolate_folios() may fall back to the other type, check all evictable types with for_each_evictable_type() and throttle if any of them has too many isolated folios.

I feel this is unlikely to work. MGLRU behaves quite differently from the active/inactive LRU, and its NR_INACTIVE_FILE accounting is also very different.

With the active/inactive LRU, shrink_inactive_list() ensures that we always have an inactive list with pages available for reclaim. With MGLRU, however, a generation can legitimately point to an empty list, so this assumption does not hold.

try_to_inc_min_seq:

         /* see the comment on lru_gen_folio */
         if (swappiness && swappiness <= MAX_SWAPPINESS) {
                 unsigned long seq = lrugen->max_seq - MIN_NR_GENS;

                 if (min_seq[LRU_GEN_ANON] > seq && min_seq[LRU_GEN_FILE] < seq)
                         min_seq[LRU_GEN_ANON] = seq;
                 else if (min_seq[LRU_GEN_FILE] > seq &&
min_seq[LRU_GEN_ANON] < seq)
                         min_seq[LRU_GEN_FILE] = seq;
         }

Quoted 7 lines

Quoted 7 lines

At that point, we have no inactive pages for the type, so the throttle will take effect when the following condition is true:

too_many = isolated > inactive;

With MGLRU, however, we can still fall back to the other type even when there are no inactive pages for the current type.

Hi All, that's a very insightful concern.

But on a higher level, you can fall back to the other type regardless of the LRU? That's not something limited to MGLRU I think? I mean throttling on one type seems like a good idea if the other type is very reclaimable.

Quoted 4 lines

Yes, that's a valid concern. So I think we can check the isolation of both types for MGLRU to avoid this case:

...

Quoted 5 lines

The typical 'isolated > inactive' case is that we've tried our best with aging, but cold pages production can't keep up with isolation speed, especially under concurrent reclaim from multiple processes. In this case, I think throttling is reasonable.

Just an idea... How about we simplify this: just check the isolated vs total memory size? At least for MGLRU. MGLRU maintains NR_INACTIVE_* only as a compatibility shim for reporting meminfo, so it does not track the immediately reclaimable set the way the active/inactive LRU does, so the inactive value is a poor trigger for MGLRU: Aging moves the folios in batches so anonymous reading can be very jumpy. For file folios, PID can only promote unmapped page cache to the second oldest generation, meaning most file folios are considered inactive. This has been causing trouble for our dashboards for a long time and I think we can fix this by using the tier number intead of gen number as the inactive / active reading, that 's a different topic though.

And in fact I don't quite get why classical LRU only compares that isolated number to the inactive count only. The reclamer can move active folios to the inactive part anytime, so throttling when we are low on inactive doesn't always seem like a good idea?

I remember this was discussed sometime ago that in the long term we might better throttle it in a different way, it was once mentioned by Shakeel:
https://lore.kernel.org/linux-mm/[email protected]/

So perhaps we can throttle MGLRU more gently? e.g. just compare to the total number / MIN_NR_GENS, which means half of the memory is isolated for MGLRU? This doesn't look too pretty but I think it's better than using the inactive count for MGLRU. And if we introduce some gen-balance-based aging (whether that is a good idea is also another topic), the ideal goal of MGLRU will be evenly ditribute folios among gens so the current inactive reading will always target half of the total memory.

Another thing btw is that the function name throttle_is_throttled reads a bit strange to me :)