mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

Extract the generation update into __folio_inc_gen(), which only updates the folio's generation and reports whether the generation was actually increased. Keep folio_inc_gen() as the wrapper that performs the LRU size accounting when needed.

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

mm/vmscan.c +11 -4

index 8f187d296b8e..07c22d51debd 100644
@@ -3918,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))

@@ -3927,35 +3928,47 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	if (!type && !swappiness)
 		goto done;
 
+	VM_WARN_ON_ONCE(get_nr_gens(lruvec, type) != MAX_NR_GENS);
+	VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) !=
+			lru_gen_is_active(lruvec, target_gen));
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);

mm/vmscan.c +2 -0

+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);

Signature · 1 line

2.34.1

Quoted 14 lines · On Fri, 28 Aug 2026 07:46:59 +0800 "Barry Song (Xiaomi)" <[email protected]>…

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 23 ++++++++++++++++++-----

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3969

Quoted 9 lines

                      if (!--remaining)

mm/vmscan.c +5 -1

-                             return false;
+                             break;
              }
+             WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+                        lrugen->nr_pages[old_gen][type][zone] - delta);
+             WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+                        lrugen->nr_pages[target_gen][type][zone] + delta);

Hi Barry,

One subtle point about the remaining handling:
when remaining reaches zero, we now break rather than return so that the accumulated delta is applied before returning.

As I understand it, this is required because __folio_inc_gen() has already changed the generation of the scanned folios, while lrugen->nr_pages[] is now updated only in batch.

So the invariant is that every successful generation increment must have its corresponding delta flushed before inc_min_seq() returns.

Is this the intended accounting invariant?

Quoted 17 lines · Kunwu Chan

Quoted 14 lines · On Fri, 28 Aug 2026 07:46:59 +0800 "Barry Song (Xiaomi)" <[email protected]>…

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 23 ++++++++++++++++++-----

[...]

Quoted 23 lines

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3969

Quoted 9 lines

                      if (!--remaining)

mm/vmscan.c +5 -1

-                             return false;
+                             break;
              }
+             WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+                        lrugen->nr_pages[old_gen][type][zone] - delta);
+             WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+                        lrugen->nr_pages[target_gen][type][zone] + delta);

Hi Barry,

One subtle point about the remaining handling:
when remaining reaches zero, we now break rather than return so that the accumulated delta is applied before returning.

As I understand it, this is required because __folio_inc_gen() has already changed the generation of the scanned folios, while lrugen->nr_pages[] is now updated only in batch.

So the invariant is that every successful generation increment must have its corresponding delta flushed before inc_min_seq() returns.

Quoted 1 line

Is this the intended accounting invariant?

Yes, Kunwu, you're right. We can't return when remaining == 0, otherwise we will miss the chance to flush the delta.

So changing return to break is intentional. Another option would be to duplicate the code that flushes the delta and keep the return, but that would be ugly since we would have to duplicate the code below:

             WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
                       lrugen->nr_pages[old_gen][type][zone] - delta);
             WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                        lrugen->nr_pages[target_gen][type][zone] + delta);

Thanks
Barry

Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.

It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().

Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Reviewed-by Kairui Song <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
  mm/vmscan.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)

mm/vmscan.c +0 -0

index 07c22d51debd..b10d1703d907 100644

mm/vmscan.c +2 -0

+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 
 
 

  			/* don't count the workingset being lazily promoted */
  			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
  				int tier = lru_tier_from_refs(refs, workingset);

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3954

Quoted 11 lines

mm/vmscan.c +2 -2

@@ -3949,9 +3949,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                      VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
                      new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-                     list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-                     if (gen_increased)
+                     if (gen_increased) {
                              delta += nr_pages;
+                             list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);

Nit.
This is a bit confusing to me.

When I read the code, I thought this implementation contradicts what the commit message says, which means I thought that we move the folios that have been promoted to the tail.

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3955

Quoted 1 line

mm/vmscan.c +1 -0

+                     } else {

Maybe adding a comment would make it clearer, like:

If gen_increased == false, it means the folio has been promoted, so move it to the head.

Quoted 3 lines

mm/vmscan.c +2 -0

+                             list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+                     }

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3960

Quoted 3 lines

mm/vmscan.c +0 -0

                      /* don't count the workingset being lazily promoted */
                      if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
                              int tier = lru_tier_from_refs(refs, workingset);

Overall, looks good to me.

Reviewed-by Ridong Chen <[email protected]>

Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.

Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
  mm/vmscan.c | 16 ++++++++--------
  1 file changed, 8 insertions(+), 8 deletions(-)

mm/vmscan.c +8 -1

index b10d1703d907..2b33d3f44682 100644
@@ -3952,17 +3952,17 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
  			if (gen_increased) {
  				delta += nr_pages;
  				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					WRITE_ONCE(lrugen->protected[hist][type][tier],
+						   lrugen->protected[hist][type][tier] + nr_pages);
+				}
  			} else {
  				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
  			}
-			/* don't count the workingset being lazily promoted */

Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio().

prefetchw_prev_lru_folio() is currently implemented as a macro with a potentially unused argument. This makes the helper harder to read and can also trigger checkpatch warnings about unused macro arguments.

Make it a static inline function and remove the unnecessary _field argument. The helper always prefetches the previous folio's flags, so there is no need to make the field configurable.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

mm/vmscan.c +15 -12

index 2b33d3f44682..81f95a968447 100644
@@ -182,17 +182,21 @@ struct scan_control {
 };
 
 #ifdef ARCH_HAS_PREFETCHW
-#define prefetchw_prev_lru_folio(_folio, _base, _field)			\
-	do {								\
-		if ((_folio)->lru.prev != _base) {			\
-			struct folio *prev;				\
-									\
-			prev = lru_to_folio(&(_folio->lru));		\
-			prefetchw(&prev->_field);			\
-		}							\
-	} while (0)
+static inline void prefetchw_prev_lru_folio(struct folio *folio,
+		struct list_head *base)
+{
+	if (folio->lru.prev != base) {
+		struct folio *prev;
+
+		prev = lru_to_folio(&folio->lru);
+		prefetchw(&prev->flags);
+	}
+}
 #else
-#define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+static inline void prefetchw_prev_lru_folio(struct folio *folio,
+		struct list_head *base)
+{
+}
 #endif
 
 /*

@@ -1695,7 +1699,7 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan,
 		struct folio *folio;
 
 		folio = lru_to_folio(src);
-		prefetchw_prev_lru_folio(folio, src, flags);
+		prefetchw_prev_lru_folio(folio, src);
 
 		nr_pages = folio_nr_pages(folio);
 		total_scan += nr_pages;
--

2.34.1

For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

mm/vmscan.c +19 -2

index 81f95a968447..17524e96fe64 100644
@@ -192,11 +192,27 @@ static inline void prefetchw_prev_lru_folio(struct folio *folio,
 		prefetchw(&prev->flags);
 	}
 }
+
+static inline void prefetchw_next_lru_folio(struct folio *folio,
+		struct list_head *base)
+{
+	if (folio->lru.next != base) {
+		struct folio *next;
+
+		next = list_entry(folio->lru.next, struct folio, lru);
+		prefetchw(&next->flags);
+	}
+}
 #else
 static inline void prefetchw_prev_lru_folio(struct folio *folio,
 		struct list_head *base)
 {
 }
+
+static inline void prefetchw_next_lru_folio(struct folio *folio,
+		struct list_head *base)
+{
+}
 #endif
 
 /*

@@ -3938,10 +3954,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		struct list_head *pos = head->next;
 		long delta = 0;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);

mm/vmscan.c +0 -1

 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
--

2.34.1

Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type.

Assisted-by gemini:gemini-3.6-flash

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

mm/vmscan.c +0 -0

index 17524e96fe64..2b6f3f05ce60 100644
@@ -3931,6 +3931,19 @@ static void clear_mm_walk(void)
                kfree(walk);
 }
 

                while (pos != head) {

mm/vmscan.c +1 -1

@@ -3974,7 +3989,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                        new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
                        if (gen_increased) {
                                delta += nr_pages;
-                               list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+                               batch_end = &folio->lru;
 
                                /* don't count the workingset being lazily promoted */
                                if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {

Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type.

Assisted-by gemini:gemini-3.6-flash

Signed-off-by Barry Song (Xiaomi) <[email protected]>

Reviewed-by Baoquan He <[email protected]>

Tested-by Xueyuan Chen <[email protected]>

---
 mm/vmscan.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

mm/vmscan.c +2 -0

index 17524e96fe64..2b6f3f05ce60 100644
@@ -3931,6 +3931,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)

mm/vmscan.c +4 -1

+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;

@@ -3953,8 +3966,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			lru_gen_is_active(lruvec, target_gen));
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 		long delta = 0;
 
 		while (pos != head) {

@@ -3974,7 +3989,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
 
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {

mm/vmscan.c +1 -1

+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
--

2.34.1