mm/vmscan: fix delayed flusher wakeup in MGLRU
Subsystems (3)
PATCH 1/1 mm/vmscan: fix delayed flusher wakeup in MGLRU
MGLRU currently decides whether to wake flusher threads in try_to_shrink_lruvec() using cumulative reclaim counters:
sc->nr.unqueued_dirty == sc->nr.file_taken
However, these counters are accumulated across multiple evict_folios() passes before the check is performed.
This can delay or suppress flusher wakeup when an earlier reclaim batch isolates only dirty file folios, but a later batch isolates clean file folios before try_to_shrink_lruvec() performs the final comparison.
For example:
batch 1: file_taken = 100, unqueued_dirty = 100
batch 2: file_taken += 60, unqueued_dirty += 0
Final check becomes 100 != 160 and flusher wakeup is skipped, even though reclaim was already blocked by dirty file folios in batch 1.
Classic reclaim avoids this by using per-batch values:
stat.nr_unqueued_dirty == nr_taken
and waking flushers immediately when the condition is met.
Make MGLRU use the same per-batch flusher wakeup behavior as classic reclaim by moving the flusher wakeup into evict_folios(), using batch-local isolation results from scan_folios() instead of the cumulative counters checked later in try_to_shrink_lruvec().
This avoids missed flusher wakeups and makes dirty folio reclaim behavior consistent with classic reclaim.
--- mm/vmscan.c | 46 ++++++++++++++++++++-------------------------- 1 file changed, 20 insertions(+), 26 deletions(-)
mm/vmscan.c +20 -26
@@ -4680,7 +4680,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list) +struct list_head *list, +unsigned long *file_taken) {int i;int gen;
@@ -4749,7 +4750,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);if (type ==LRU_GEN_FILE ) - sc->nr.file_taken += isolated; + *file_taken += isolated;/* * There might not be eligible folios due to reclaim_idx. Check the * remaining to prevent livelock if it's not making progress.
@@ -4798,7 +4799,8 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness, -int *type_scanned,struct list_head *list) +int *type_scanned,struct list_head *list, +unsigned long *file_taken) {int i;int type =get_type_to_scan (lruvec, swappiness);
@@ -4809,7 +4811,8 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, *type_scanned = type; - scanned =scan_folios (nr_to_scan, lruvec, sc, type, tier, list); + scanned =scan_folios (nr_to_scan, lruvec, sc, type, tier, + list, file_taken);if (scanned)return scanned;
@@ -4825,6 +4828,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int type;int scanned;int reclaimed; +unsigned long file_taken =0 ;LIST_HEAD (list);LIST_HEAD (clean);struct folio *folio;
@@ -4839,8 +4843,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,lruvec_lock_irq (lruvec); - scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, &type, &list); - + scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, + &type, &list, &file_taken); scanned +=try_to_inc_min_seq (lruvec, swappiness);if (evictable_min_seq (lrugen->min_seq, swappiness) +MIN_NR_GENS > lrugen->max_seq)
@@ -4852,6 +4856,14 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; retry: reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false, memcg); + +if (stat.nr_unqueued_dirty && stat.nr_unqueued_dirty == file_taken) { +wakeup_flusher_threads (WB_REASON_VMSCAN ); + +if (!writeback_throttling_sane (sc)) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); + } + sc->nr.file_taken += file_taken; sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; sc->nr_reclaimed += reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
@@ -5021,27 +5033,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) }/* -* If too many file cache in the coldest generation can't be evicted -* due to being dirty, wake up the flusher. +* Flusher wakeup and writeback throttling are handled in +* evict_folios() based on per-batch reclaim results. */ -if (sc->nr.unqueued_dirty && sc->nr.unqueued_dirty == sc->nr.file_taken) { -struct pglist_data *pgdat =lruvec_pgdat (lruvec); - -wakeup_flusher_threads (WB_REASON_VMSCAN ); - -/* -* For cgroupv1 dirty throttling is achieved by waking up -* the kernel flusher here and later waiting on folios -* which are in writeback to finish (see shrink_folio_list()). -* -* Flusher may not be able to issue writeback quickly -* enough for cgroupv1 writeback throttling to work -* on a large system. -*/ -if (!writeback_throttling_sane (sc)) -reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); - } -/* whether this lruvec should be rotated */ return nr_to_scan <0 ; }
Signature · 1 line
2.54.0
Andrew Morton
maintainerQuoted 9 lines · On Thu, 30 Apr 2026 00:24:41 +0530 Vineet Agarwal…
MGLRU currently decides whether to wake flusher threads in
try_to_shrink_lruvec()using cumulative reclaim counters:...
Make MGLRU use the same per-batch flusher wakeup behavior as classic reclaim by moving the flusher wakeup into
evict_folios(), using batch-local isolation results fromscan_folios()instead of the cumulative counters checked later intry_to_shrink_lruvec().
Thanks. AI review asked a couple of questions:
https://sashiko.dev/#/patchset/[email protected]
Quoted 149 lines · Vineet Agarwal
MGLRU currently decides whether to wake flusher threads in
try_to_shrink_lruvec()using cumulative reclaim counters:
sc->nr.unqueued_dirty==sc->nr.file_takenHowever, these counters are accumulated across multiple
evict_folios()passes before the check is performed.This can delay or suppress flusher wakeup when an earlier reclaim batch isolates only dirty file folios, but a later batch isolates clean file folios before
try_to_shrink_lruvec()performs the final comparison.For example:
batch 1:
file_taken= 100,unqueued_dirty= 100
batch 2:file_taken+= 60,unqueued_dirty+= 0Final check becomes 100 != 160 and flusher wakeup is skipped, even though reclaim was already blocked by dirty file folios in batch 1.
Classic reclaim avoids this by using per-batch values:
stat.nr_unqueued_dirty ==
nr_takenand waking flushers immediately when the condition is met.
Make MGLRU use the same per-batch flusher wakeup behavior as classic reclaim by moving the flusher wakeup into
evict_folios(), using batch-local isolation results fromscan_folios()instead of the cumulative counters checked later intry_to_shrink_lruvec().This avoids missed flusher wakeups and makes dirty folio reclaim behavior consistent with classic reclaim.
Fixes 1bc542c6a0d14 ("mm/vmscan: wake up flushers conditionally to avoid cgroup OOM")Signed-off-by Vineet Agarwal <[email protected]>--- mm/vmscan.c | 46 ++++++++++++++++++++-------------------------- 1 file changed, 20 insertions(+), 26 deletions(-)mm/vmscan.c +20
-27@@ -4680,7 +4680,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct scastatic int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list) +struct list_head *list, +unsigned long *file_taken) {int i;int gen;@@ -4749,7 +4750,7 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);if (type ==LRU_GEN_FILE ) - sc->nr.file_taken += isolated; + *file_taken += isolated;/* * There might not be eligible folios due to reclaim_idx. Check the * remaining to prevent livelock if it's not making progress. @@ -4798,7 +4799,8 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness)static int isolate_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int swappiness, -int *type_scanned,struct list_head *list) +int *type_scanned,struct list_head *list, +unsigned long *file_taken) {int i;int type =get_type_to_scan (lruvec, swappiness);@@ -4809,7 +4811,8 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, *type_scanned = type; - scanned =scan_folios (nr_to_scan, lruvec, sc, type, tier, list); + scanned =scan_folios (nr_to_scan, lruvec, sc, type, tier, + list, file_taken);if (scanned)return scanned;@@ -4825,6 +4828,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,int type;int scanned;int reclaimed; +unsigned long file_taken =0 ;LIST_HEAD (list);LIST_HEAD (clean);struct folio *folio;@@ -4839,8 +4843,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,lruvec_lock_irq (lruvec); - scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, &type, &list); - + scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, + &type, &list, &file_taken); scanned +=try_to_inc_min_seq (lruvec, swappiness);if (evictable_min_seq (lrugen->min_seq, swappiness) +MIN_NR_GENS > lrugen->max_seq)@@ -4852,6 +4856,14 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,return scanned; retry: reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false, memcg); + +if (stat.nr_unqueued_dirty && stat.nr_unqueued_dirty == file_taken) { +wakeup_flusher_threads (WB_REASON_VMSCAN ); + +if (!writeback_throttling_sane (sc)) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); + } + sc->nr.file_taken += file_taken; sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; sc->nr_reclaimed += reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,@@ -5021,27 +5033,9 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) }/* -* If too many file cache in the coldest generation can't be evicted -* due to being dirty, wake up the flusher. +* Flusher wakeup and writeback throttling are handled in +* evict_folios() based on per-batch reclaim results. */ -if (sc->nr.unqueued_dirty && sc->nr.unqueued_dirty == sc->nr.file_taken) { -struct pglist_data *pgdat =lruvec_pgdat (lruvec); - -wakeup_flusher_threads (WB_REASON_VMSCAN ); - -/* -* For cgroupv1 dirty throttling is achieved by waking up -* the kernel flusher here and later waiting on folios -* which are in writeback to finish (see shrink_folio_list()). -* -* Flusher may not be able to issue writeback quickly -* enough for cgroupv1 writeback throttling to work -* on a large system. -*/ -if (!writeback_throttling_sane (sc)) -reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); - } -/* whether this lruvec should be rotated */ return nr_to_scan <0 ; } --
Quoted 3 lines
2.54.0
Hi, Vineet,
Thanks for the patch, I got what you mean, but even with this patch, the wakeup and flush logic of MGLRU is still very different from classical LRU and doesn't perform well. No throttling, the batch is still over large, and it could over aggressively wake up the flusher as the folios are stuck at the tail.
There are a few more issues related, so there is already a ongoing work to fix them properly, and make them more unified:
https://lore.kernel.org/linux-mm/[email protected]/
Global throttling or isolation throttling are still missing after that, I think maybe we can focus on that part after that series. How do you think?
Hi Kairui,
Thanks for the detailed explanation and for pointing me to the series.
That makes sense — I see that the issue is part of a broader set of reclaim and writeback problems in MGLRU, and my patch only addressed a small part of it.
I’ll go through the series and take a closer look at the remaining gaps you mentioned, especially around throttling.
Thanks again for the guidance.
Vineet
Quoted 194 lines · Kairui Song
Quoted 32 lines · Vineet Agarwal
MGLRU currently decides whether to wake flusher threads in
try_to_shrink_lruvec()using cumulative reclaim counters:
sc->nr.unqueued_dirty==sc->nr.file_takenHowever, these counters are accumulated across multiple
evict_folios()passes before the check is performed.This can delay or suppress flusher wakeup when an earlier reclaim batch isolates only dirty file folios, but a later batch isolates clean file folios before
try_to_shrink_lruvec()performs the final comparison.For example:
batch 1:
file_taken= 100,unqueued_dirty= 100
batch 2:file_taken+= 60,unqueued_dirty+= 0Final check becomes 100 != 160 and flusher wakeup is skipped, even though reclaim was already blocked by dirty file folios in batch 1.
Classic reclaim avoids this by using per-batch values:
stat.nr_unqueued_dirty ==
nr_takenand waking flushers immediately when the condition is met.
Make MGLRU use the same per-batch flusher wakeup behavior as classic reclaim by moving the flusher wakeup into
evict_folios(), using batch-local isolation results fromscan_folios()instead of the cumulative counters checked later intry_to_shrink_lruvec().Quoted 4 lines
This avoids missed flusher wakeups and makes dirty folio reclaim behavior consistent with classic reclaim.
Fixes 1bc542c6a0d14 ("mm/vmscan: wake up flushers conditionally toavoid cgroup OOM")
mm/vmscan.c static bool isolate_folio(struct lruvec *lruvec,
Quoted 10 lines
Signed-off-by Vineet Agarwal <[email protected]>--- mm/vmscan.c | 46 ++++++++++++++++++++-------------------------- 1 file changed, 20 insertions(+), 26 deletions(-)mm/vmscan.c +0
-0@@ -4680,7 +4680,8 @@ static bool isolate_folio(struct lruvec *lruvec,struct folio *folio, struct sca
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan,
Quoted 10 lines
mm/vmscan.c +2
-1static int scan_folios (unsigned long nr_to_scan,struct lruvec *lruvec,struct scan_control *sc,int type,int tier, -struct list_head *list) +struct list_head *list, +unsigned long *file_taken) {int i;int gen;@@ -4749,7 +4750,7 @@ static int scan_folios(unsigned long nr_to_scan,struct lruvec *lruvec,
mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, · line 4751
Quoted 2 lines
mm/vmscan.c +0
-0scanned, skipped, isolated, type ? LRU_INACTIVE_FILE :
LRU_INACTIVE_ANON);mm/vmscan.c static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
Quoted 5 lines
mm/vmscan.c +1
-1if (type ==LRU_GEN_FILE ) - sc->nr.file_taken += isolated; + *file_taken += isolated;/* * There might not be eligible folios due to reclaim_idx. Check the
mm/vmscan.c static int get_type_to_scan(struct lruvec *lruvec,
Quoted 2 lines
mm/vmscan.c +0
-0* remaining to prevent livelock if it's not making progress. @@ -4798,7 +4799,8 @@ static int get_type_to_scan(struct lruvec *lruvec,int swappiness)
mm/vmscan.c static int get_type_to_scan(struct lruvec *lruvec, · line 4799
Quoted 2 lines
mm/vmscan.c +0
-0static int isolate_folios(unsigned long nr_to_scan, struct lruvec*lruvec,
mm/vmscan.c static int isolate_folios(unsigned long
Quoted 8 lines
mm/vmscan.c +2
-1struct scan_control *sc, int swappiness, - int *type_scanned, struct list_head *list) + int *type_scanned, struct list_head *list, + unsigned long *file_taken) { int i;int type =get_type_to_scan (lruvec, swappiness);@@ -4809,7 +4811,8 @@ static int isolate_folios(unsigned long
nr_to_scan, struct lruvec *lruvec,mm/vmscan.c static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
Quoted 4 lines
mm/vmscan.c +0
-1*type_scanned = type; - scanned = scan_folios(nr_to_scan, lruvec, sc, type,tier, list);
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan,
Quoted 6 lines
mm/vmscan.c +2
-0+ scanned =scan_folios (nr_to_scan, lruvec, sc, type, tier, + list, file_taken);if (scanned)return scanned;@@ -4825,6 +4828,7 @@ static int evict_folios(unsigned long nr_to_scan,struct lruvec *lruvec,
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan,
Quoted 8 lines
mm/vmscan.c +1
-0int type;int scanned;int reclaimed; +unsigned long file_taken =0 ;LIST_HEAD (list);LIST_HEAD (clean);struct folio *folio;@@ -4839,8 +4843,8 @@ static int evict_folios(unsigned long nr_to_scan,struct lruvec *lruvec,
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
Quoted 4 lines
mm/vmscan.c +0
-1lruvec_lock_irq (lruvec); - scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness,&type, &list);
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan,
Quoted 8 lines
mm/vmscan.c +2
-1- + scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, + &type, &list, &file_taken); scanned +=try_to_inc_min_seq (lruvec, swappiness);if (evictable_min_seq (lrugen->min_seq, swappiness) +MIN_NR_GENS lrugen->max_seq)@@ -4852,6 +4856,14 @@ static int evict_folios(unsigned long nr_to_scan,struct lruvec *lruvec,
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
Quoted 3 lines
mm/vmscan.c +0
-0return scanned; retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false,memcg);
mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, · line 4860
Quoted 2 lines
mm/vmscan.c +2
-0+ + if (stat.nr_unqueued_dirty && stat.nr_unqueued_dirty ==
file_taken) {mm/vmscan.c static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
Quoted 4 lines
mm/vmscan.c +4
-0+wakeup_flusher_threads (WB_REASON_VMSCAN ); + +if (!writeback_throttling_sane (sc)) + reclaim_throttle(pgdat,
VMSCAN_THROTTLE_WRITEBACK);mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec
Quoted 6 lines
mm/vmscan.c +2
-0+ } + sc->nr.file_taken += file_taken; sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; sc->nr_reclaimed += reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,@@ -5021,27 +5033,9 @@ static bool try_to_shrink_lruvec(struct lruvec*lruvec, struct
scan_control*sc)mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec · line 5024
Quoted 4 lines
mm/vmscan.c +0
-1}/* -* If too many file cache in the coldest generation can't be evicted
mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
Quoted 5 lines
mm/vmscan.c +2
-2-* due to being dirty, wake up the flusher. +* Flusher wakeup and writeback throttling are handled in +* evict_folios() based on per-batch reclaim results. */ - if (sc->nr.unqueued_dirty && sc->nr.unqueued_dirty ==
sc->nr.file_taken) {mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
Quoted 8 lines
mm/vmscan.c +0
-8-struct pglist_data *pgdat =lruvec_pgdat (lruvec); - -wakeup_flusher_threads (WB_REASON_VMSCAN ); - -/* -* For cgroupv1 dirty throttling is achieved by waking up -* the kernel flusher here and later waiting on folios -* which are in writeback to finish (see
shrink_folio_list()).mm/vmscan.c static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
Quoted 7 lines
mm/vmscan.c +0
-7-* -* Flusher may not be able to issue writeback quickly -* enough for cgroupv1 writeback throttling to work -* on a large system. -*/ -if (!writeback_throttling_sane (sc)) - reclaim_throttle(pgdat,
VMSCAN_THROTTLE_WRITEBACK);Quoted 9 lines
mm/vmscan.c +0
-3- } -/* whether this lruvec should be rotated */ return nr_to_scan <0 ; } --2.54.0
Hi, Vineet,
Thanks for the patch, I got what you mean, but even with this patch, the wakeup and flush logic of MGLRU is still very different from classical LRU and doesn't perform well. No throttling, the batch is still over large, and it could over aggressively wake up the flusher as the folios are stuck at the tail.
There are a few more issues related, so there is already a ongoing work to fix them properly, and make them more unified:
https://lore.kernel.org/linux-mm/[email protected]/
Global throttling or isolation throttling are still missing after that, I think maybe we can focus on that part after that series. How do you think?