mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

PATCH Superseded Barry Song (Xiaomi) 5 participants 31 messages Aug 21 – Aug 27

Subsystems (3)

Memory management MGLRU MM reclaim

lore

Share & tools

lore

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #0 lore raw

cover mm/mglru: speed up inc_min_seq() and fix cold/hot inversions

This is an aging speedup series split out from the MGLRU swappiness series [1], with the inc_min_seq changes separated to make them easier to review.

Currently, inc_min_seq performance is crucial to both the swappiness fix and proactive aging. There are two problems with it:

1. It processes each folio one by one, while many operations can be batched or skipped. For example, a batch of folios can be moved together from the oldest generation to the second-oldest generation, and the associated counting can also be done in batches.

2. It may cause potential cold/hot inversion by placing promoted folios (which have been scanned and found to have young PTEs) behind non-promoted folios. A similar inversion can also occur among non-promoted folios, as tail folios from the oldest generation are placed before head folios when moving them to the second-oldest generation.

This series tries to batch operations as much as possible and fix the potential cold/hot inversion by keeping promoted folios ahead of non-promoted folios, while also preserving the order of non-promoted folios when moving them from the oldest generation to the second-oldest generation.

Minor issue: inc_min_seq() also counts protected folios improperly, as promoted folios should be skipped, as in sort_folio().

We need a stable workload with a stable number of folios to measure aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT to generate the microbenchmark below. It ages an LRU vec containing 512 MB of memory 100 times:

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#include <sys/mman.h>

#define SIZE (512UL * 1024 * 1024)
#define LRU_GEN "/sys/kernel/debug/lru_gen"
#define TARGET_CGROUP "/system.slice/agetest.scope"
#define START_GEN 3
#define END_GEN 103

static long long nsec_diff(const struct timespec *start,
const struct timespec *end)
{
return (end->tv_sec - start->tv_sec) * 1000000000LL +
(end->tv_nsec - start->tv_nsec);
}

static int find_memcg_id(void)
{
FILE *fp;
char line[4096];
int memcg_id;

fp = fopen(LRU_GEN, "r");
if (!fp) {
perror("fopen lru_gen");
return -1;
}

while (fgets(line, sizeof(line), fp)) {
char *p;

if (strncmp(line, "memcg ", 6))
continue;

p = line + 6;

if (sscanf(p, "%d", &memcg_id) != 1)
continue;

/*
* The memcg path follows the numeric ID. */
p = strchr(p, ' ');
if (!p)
continue;

if (strstr(p, TARGET_CGROUP)) {
fclose(fp);
return memcg_id;
}
}

fclose(fp);

fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP); return -1;
}

int main(void)
{
void *addr;
int memcg_id;
int fd;
long long total_ns = 0;

/*
* mmap 512 MB and touch every page.
*/
addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
return 1;
}

memset(addr, 0x55, SIZE);

printf("mmap: %p, size: %lu MB\n",
addr, SIZE / 1024 / 1024);

/*
* Find the memcg ID automatically.
*/
memcg_id = find_memcg_id();
if (memcg_id < 0)
return 1;

printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP); printf("aging generation %d -> %d\n",
START_GEN, END_GEN);

fd = open(LRU_GEN, O_WRONLY);
if (fd < 0) {
perror("open lru_gen");
return 1;
}

for (int gen = START_GEN; gen <= END_GEN; gen++) {
char buf[128];
int len;
struct timespec start, end;
long long ns;

len = snprintf(buf, sizeof(buf),
"+ %d 0 %d\n", memcg_id, gen);

clock_gettime(CLOCK_MONOTONIC, &start);

if (write(fd, buf, len) != len) {
perror("write lru_gen");
close(fd);
return 1;
}

clock_gettime(CLOCK_MONOTONIC, &end);

ns = nsec_diff(&start, &end);
total_ns += ns;

printf("gen %3d: %8.3f ms\n",
gen, ns / 1000000.0);
fflush(stdout);
}

close(fd);

printf("\nTotal: %.3f ms\n",
total_ns / 1000000.0);
printf("Average: %.3f ms\n",
total_ns / (double)(END_GEN - START_GEN + 1) / 1000000.0);

while (1)
sleep(1);

return 0;
}

Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest

I’m seeing inc_min_seq() become significantly faster:

W/o patch:

Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.433 ms
gen   4:    0.949 ms
gen   5:    2.535 ms
gen   6:    5.043 ms
gen   7:    5.041 ms
gen   8:    5.027 ms
...
gen 100:    5.035 ms
gen 101:    5.011 ms
gen 102:    5.029 ms
gen 103:    5.056 ms

Total:   503.946 ms
Average: 4.990 ms

W/ patch:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

The average aging time drops from 4.990 ms to 2.334 ms!

[1] https://lore.kernel.org/linux-mm/[email protected]/

Barry Song (Xiaomi) (6):
mm/mglru: batch update lrugen->nr_pages in inc_min_seq() mm/mglru: batch update lrugen->protected in inc_min_seq() mm/mglru: enhance cold/hot inversion handling in inc_min_seq() mm/mglru: exclude folios promoted by aging from protected in

    inc_min_seq()

mm/mglru: move folios from oldest gen to second-oldest gen from head
to tail
mm/mglru: batch move folios to the second-oldest gen's LRU

 mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 80 insertions(+), 18 deletions(-)

Signature · 1 line

2.34.1

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #1 lore raw

PATCH 1/6 mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

mm/vmscan.c +35 ~~-11~~

index c1404a59523d..0d74fc00abd3 100644
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
 {
-	int type = folio_is_file_lru(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	int new_gen;
 
 	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
 
 	do {
 		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
+		if (new_gen >= 0 && new_gen != old_gen) {
+			if (increased)
+				*increased = false;
 			return new_gen;
+		}
 
 		new_gen = (old_gen + 1) % MAX_NR_GENS;

@@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
 
-	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+	if (increased)
+		*increased = true;
+	return new_gen;
+}
 
+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	bool gen_increased;
+
+	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+	if (gen_increased)
+		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 	return new_gen;
 }

@@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))

@@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		unsigned long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);
 			}
 
 			if (!--remaining)
-				return false;
+				break;
 		}
+		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);

Signature · 1 line

2.34.1

Lian Wang (ProcessMission)

2026-08-22 01:42 UTC #2 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

From: Lian Wang <[email protected]>

Hi Barry,

A small nit:

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3934

Quoted 1 line

mm/vmscan.c +1 -0

+		unsigned long delta = 0;

lrugen->nr_pages is a signed long and is documented as being able to transiently go negative while reset_batch_size() is pending. Could delta remain a long as well, so this arithmetic stays signed?

In 2/6, where the declarations are combined, this could be:

	unsigned long protected[MAX_NR_TIERS] = {};
	long delta = 0;

Thanks,
Lian

Barry Song

author reviewer

2026-08-25 21:38 UTC #3 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

Quoted 5 lines · Lian Wang (ProcessMission)

From: Lian Wang <[email protected]>
Hi Barry,

Quoted 7 lines

A small nit:
Quoted 1 line
+ unsigned long delta = 0;
lrugen->nr_pages is a signed long and is documented as being able to transiently go negative while reset_batch_size() is pending. Could delta remain a long as well, so this arithmetic stays signed?

Thanks, Lian, for the review. Changing it from unsigned long to long makes sense to me.

Quoted 6 lines

In 2/6, where the declarations are combined, this could be:
        unsigned long protected[MAX_NR_TIERS] = {};
        long delta = 0;

Best Regards
Barry

Baoquan He

reviewer

2026-08-26 08:23 UTC #4 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

In reply to Barry Song (Xiaomi)

Hi Barry,

Quoted 5 lines · Barry Song (Xiaomi)

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

This patch refactor code to split out __folio_inc_gen() and introduce gen_increased, this is the base of later patches. You seem to only mention the minor optimization of lrugen->nr_pages. IMHO, this refactoring can be split out to an independent patch. The lrugen->nr_pages can be integrated with other patch, e.g patch 2 or patch 6.

Quoted 4 lines

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().

For the justification of skipping __lru_update_size(), there's a
precondition: get_nr_gens(lruvec, type) == MAX_NR_GENS; With this, the oldest gen and 2nd oldest gen are both inactive. Calling __lru_update_size() may waste tiny cpu, while skipping it may cause issue in future if the precondition is changed. Do you think adding a VM_WARN_ON_ONCE is necessary?

VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) ||

		lru_gen_is_active(lruvec, target_gen));

Thanks
Baoquan

Quoted 118 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

mm/vmscan.c +35 ~~-11~~

index c1404a59523d..0d74fc00abd3 100644
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
 {
-	int type = folio_is_file_lru(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
+	int new_gen;
 
 	VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio);
 
 	do {
 		new_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
 		/* folio_update_gen() has promoted this page? */
-		if (new_gen >= 0 && new_gen != old_gen)
+		if (new_gen >= 0 && new_gen != old_gen) {
+			if (increased)
+				*increased = false;
 			return new_gen;
+		}
 
 		new_gen = (old_gen + 1) % MAX_NR_GENS;

@@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
 		new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
 
-	lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+	if (increased)
+		*increased = true;
+	return new_gen;
+}
 
+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+{
+	int type = folio_is_file_lru(folio);
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	bool gen_increased;
+
+	new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+	if (gen_increased)
+		lru_gen_update_size(lruvec, folio, old_gen, new_gen);
 	return new_gen;
 }

@@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int hist = lru_hist_from_seq(lrugen->min_seq[type]);
 	int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
+	int target_gen = (old_gen + 1) % MAX_NR_GENS;
 
 	/* For file type, skip the check if swappiness is anon only */
 	if (type && (swappiness == SWAPPINESS_ANON_ONLY))

@@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
+		unsigned long delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);
+			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);
+			bool gen_increased;
 
 			VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
-			new_gen = folio_inc_gen(lruvec, folio);
+			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-
+			if (gen_increased)
+				delta += nr_pages;
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
-				int delta = folio_nr_pages(folio);
 
 				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + delta);
+					   lrugen->protected[hist][type][tier] + nr_pages);
 			}
 
 			if (!--remaining)
-				return false;
+				break;
 		}
+		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
+			   lrugen->nr_pages[old_gen][type][zone] - delta);
+		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
+			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		if (!remaining)
+			return false;
 	}
 done:
 	reset_ctrl_pos(lruvec, type, true);

Signature · 1 line

2.34.1

Kairui Song

2026-08-27 03:21 UTC #5 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

In reply to Barry Song (Xiaomi)

Quoted 7 lines · Barry Song (Xiaomi)

Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.

Quoted 8 lines

Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)

Hello Barry

Thanks for the patch!

mm/vmscan.c static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma · line 3299

Quoted 11 lines

mm/vmscan.c +1 -1

index c1404a59523d..0d74fc00abd3 100644
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)

I feel the naming is a bit confusing, the __ prefix doesn't tell how it differs from folio_inc_gen very well, maybe just name one folio_inc_gen (the old gen could be any gen), another one is folio_inc_min_gen (the old gen can only be min_seq), and with sanity check in folio_inc_min_gen that expects get_nr_gens == 4, and lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This could be a build-time sanity check instead of a runtime debug check.

I saw baoquan also mentioned the debug check on lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1), which I agree.

And BTW I'm suggesting changing the definition of active/inactive for MGLRU from gen-based to refs-based. The gen-based active/inactive reading has been giving us headaches for years and is a main blocker for server production, and I think no one cares about the active/inactive reading for MGLRU for desktop / mobile right now (especially anon active / inactive, which is basiclaly random number at this point :P, and files are almost always stuck at inactive).

However that contradicts this change by a lot, folio_inc_gen will always have to update the statistic, see (was planning to send RFC after more test on Android):

https://github.com/ryncsn/linux/commit/d13a3abdd22a97eca32a7207fce56cc6030a3bd7

This is another reason why I'm trying to avoid aging rather than optimizing it. I once tried some aging optimization two year ago, the result was good by then but I gave up due to the added complexity:
https://lore.kernel.org/linux-mm/[email protected]/

I realized the main blocker was cmpxchg and maybe async aging with a refined reclaim cycle was the better solution rather than making things complex.

But anyway, let's process with this; perhaps we'll need to find a different way to batch aging later to fit the implementation of the FG, OOM prevention or refs-based idea.

Barry Song

author reviewer

2026-08-27 11:21 UTC #6 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

Quoted 30 lines · Kairui Song

Quoted 7 lines · Barry Song (Xiaomi)
Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.
Quoted 8 lines
Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)
Hello Barry
Thanks for the patch!
Quoted 8 lines
mm/vmscan.c +0 -0
index c1404a59523d..0d74fc00abd3 100644
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 

Quoted 11 lines

mm/vmscan.c static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma · line 3299
Quoted 3 lines
mm/vmscan.c +1 -1
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
I feel the naming is a bit confusing, the __ prefix doesn't tell how it differs from folio_inc_gen very well, maybe just name one folio_inc_gen (the old gen could be any gen), another one is folio_inc_min_gen (the old gen can only be min_seq), and with sanity check in folio_inc_min_gen that expects get_nr_gens == 4, and lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This could be a build-time sanity check instead of a runtime debug check.

Thanks very much for your suggestion, Kairui.

I tried splitting this into two functions, folio_inc_min_gen() and folio_inc_gen(). I tried a couple of approaches, but I think I’ll give up on this direction.
folio_inc_min_gen() and folio_inc_gen() share some common code while also having some differences. It’s hard to avoid ugly code duplication because folio_inc_gen() cannot call folio_inc_min_gen(). However, folio_inc_gen() can call __folio_inc_gen().

The __ prefix currently indicates that we don’t update the LRU sizes in this case. The sizes can either be updated in a batch later or updated once by folio_inc_gen().
Maybe we could add a comment to make this easier to read, or use a better name for __folio_inc_gen()?

Quoted 5 lines

I saw baoquan also mentioned the debug check on lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1), which I agree.

Agreed. But I feel we only need to do this once in inc_min_seq(), rather than for each folio.

Best Regards
Barry

Kairui Song

2026-08-27 11:31 UTC #7 lore raw

Re: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq()

Quoted 54 lines · Barry Song

Quoted 30 lines · Kairui Song
Quoted 7 lines · Barry Song (Xiaomi)
Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero.
Quoted 8 lines
Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size().
Signed-off-by Barry Song (Xiaomi) <[email protected]>
---
 mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 11 deletions(-)
Hello Barry
Thanks for the patch!
Quoted 8 lines
mm/vmscan.c +0 -0
index c1404a59523d..0d74fc00abd3 100644
@@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma
 }
 
Quoted 11 lines
mm/vmscan.c static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma · line 3299
Quoted 3 lines
mm/vmscan.c +1 -1
 /* protect pages accessed multiple times through file descriptors */
-static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
+static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
I feel the naming is a bit confusing, the __ prefix doesn't tell how it differs from folio_inc_gen very well, maybe just name one folio_inc_gen (the old gen could be any gen), another one is folio_inc_min_gen (the old gen can only be min_seq), and with sanity check in folio_inc_min_gen that expects get_nr_gens == 4, and lru_gen_is_active(min_seq) == lru_gen_is_active(min_seq + 1)? This could be a build-time sanity check instead of a runtime debug check.
Thanks very much for your suggestion, Kairui.
I tried splitting this into two functions, folio_inc_min_gen() and folio_inc_gen(). I tried a couple of approaches, but I think I’ll give up on this direction.
folio_inc_min_gen() and folio_inc_gen() share some common code while also having some differences. It’s hard to avoid ugly code duplication because folio_inc_gen() cannot call folio_inc_min_gen(). However, folio_inc_gen() can call __folio_inc_gen().

Quoted 5 lines

The __ prefix currently indicates that we don’t update the LRU sizes in this case. The sizes can either be updated in a batch later or updated once by folio_inc_gen().
Maybe we could add a comment to make this easier to read, or use a better name for __folio_inc_gen()?

OK, thanks for the explanation. Let's just keep the name then.

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #8 lore raw

PATCH 2/6 mm/mglru: batch update lrugen->protected in inc_min_seq()

In reply to Barry Song (Xiaomi)

Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/vmscan.c +5 -3

index 0d74fc00abd3..99ee3c833d54 100644
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
-		unsigned long delta = 0;
+		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);

@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
 
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + nr_pages);
+				protected[tier] += nr_pages;
 			}
 
 			if (!--remaining)

@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
 			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + protected[tier]);
 		if (!remaining)
 			return false;
 	}

Signature · 1 line

2.34.1

Baoquan He

reviewer

2026-08-26 09:10 UTC #9 lore raw

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

Quoted 3 lines · Barry Song (Xiaomi)

Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead.

Wondering how much efficiency this can bring, is there a number for this standalone patch?

Quoted 42 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/vmscan.c +5 -3

index 0d74fc00abd3..99ee3c833d54 100644
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
-		unsigned long delta = 0;
+		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
 		while (!list_empty(head)) {
 			struct folio *folio = lru_to_folio(head);

@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);
 
-				WRITE_ONCE(lrugen->protected[hist][type][tier],
-					   lrugen->protected[hist][type][tier] + nr_pages);
+				protected[tier] += nr_pages;
 			}
 
 			if (!--remaining)

@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
 			   lrugen->nr_pages[target_gen][type][zone] + delta);
+		for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+			WRITE_ONCE(lrugen->protected[hist][type][tier],
+				   lrugen->protected[hist][type][tier] + protected[tier]);
 		if (!remaining)
 			return false;
 	}

Signature · 1 line

2.34.1

Barry Song

author reviewer

2026-08-27 05:09 UTC #10 lore raw

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

Quoted 8 lines · Baoquan He

Quoted 3 lines · Barry Song (Xiaomi)
Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead.
Wondering how much efficiency this can bring, is there a number for this standalone patch?

This is a good question. I reverted the protection batching to check its impact:

mm/vmscan.c +0 -0

index 996b48344ed0..c9a2fd9ad844 100644
@@ -3957,7 +3957,7 @@ static bool inc_min_seq(struct lruvec *lruvec,

int type, int swappiness)
for (zone = 0; zone < MAX_NR_ZONES; zone++) {
struct list_head *target_list =

&lrugen->folios[target_gen][type][zone];
                struct list_head *head = &lrugen->folios[old_gen][type][zone];
-               unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+               unsigned long delta = 0;
                struct list_head *pos = head->next;
                struct list_head *batch_end = NULL;

@@ -3984,7 +3984,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)

                                if (refs + workingset !=

BIT(LRU_REFS_WIDTH) + 1) {
int tier =
lru_tier_from_refs(refs, workingset);

- protected[tier] += nr_pages;
+
WRITE_ONCE(lrugen->protected[hist][type][tier],
+

lrugen->protected[hist][type][tier] + nr_pages);
                                }
                        } else {
                                flush_lru_batch(head, &batch_end, target_list);
@@ -3999,9 +4000,6 @@ static bool inc_min_seq(struct lruvec *lruvec,
int type, int swappiness)
                           lrugen->nr_pages[old_gen][type][zone] - delta);
                WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                           lrugen->nr_pages[target_gen][type][zone] + delta);
-               for (int tier = 0; tier < MAX_NR_TIERS; tier++)
-                       WRITE_ONCE(lrugen->protected[hist][type][tier],
-                                  lrugen->protected[hist][type][tier]
+ protected[tier]);
                if (!remaining)
                        return false;
        }

And I see:

With protection batching:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

Without protection batching:

Running scope as unit: agetest.scope
mmap: 0x775682e00000, size: 512 MB
memcg: 12823 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.542 ms
gen   4:    0.948 ms
gen   5:    2.338 ms
...
gen 101:    2.340 ms
gen 102:    2.342 ms
gen 103:    2.347 ms

Total:   240.505 ms
Average: 2.381 ms

It’s 2.334 vs. 2.381, which is really minor.

I’m curious to see what the data looks like on ARM. If there’s no significant difference, we may drop this patch in v2 to reduce the amount of code change.

Xueyuan, since you’ve been testing this patchset on ARM[1], would you mind checking this on ARM as well?

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 29 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/vmscan.c +2 -3

index 0d74fc00abd3..99ee3c833d54 100644
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
      /* prevent cold/hot inversion if the type is evictable */
      for (zone = 0; zone < MAX_NR_ZONES; zone++) {
              struct list_head *head = &lrugen->folios[old_gen][type][zone];
-             unsigned long delta = 0;
+             unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
              while (!list_empty(head)) {
                      struct folio *folio = lru_to_folio(head);

@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                      if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
                              int tier = lru_tier_from_refs(refs, workingset);
 
-                             WRITE_ONCE(lrugen->protected[hist][type][tier],
-                                        lrugen->protected[hist][type][tier] + nr_pages);
+                             protected[tier] += nr_pages;
                      }

Quoted 14 lines

Quoted 12 lines

mm/vmscan.c +3 -1

                      if (!--remaining)

@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                         lrugen->nr_pages[old_gen][type][zone] - delta);
              WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                         lrugen->nr_pages[target_gen][type][zone] + delta);
+             for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+                     WRITE_ONCE(lrugen->protected[hist][type][tier],
+                                lrugen->protected[hist][type][tier] + protected[tier]);
              if (!remaining)
                      return false;
      }
--

Quoted 2 lines

2.34.1

Thanks
Barry

Xueyuan Chen

2026-08-27 12:14 UTC #11 lore raw

Re: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq()

Quoted 99 lines · Barry Song

Quoted 8 lines · Baoquan He

Quoted 3 lines · Barry Song (Xiaomi)
Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead.
Wondering how much efficiency this can bring, is there a number for this standalone patch?

This is a good question. I reverted the protection batching to check its impact:

mm/vmscan.c +1 -1

index 996b48344ed0..c9a2fd9ad844 100644
@@ -3957,7 +3957,7 @@ static bool inc_min_seq(struct lruvec *lruvec,
 int type, int swappiness)
        for (zone = 0; zone < MAX_NR_ZONES; zone++) {
                struct list_head *target_list =
 &lrugen->folios[target_gen][type][zone];
                struct list_head *head = &lrugen->folios[old_gen][type][zone];
-               unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+               unsigned long delta = 0;
                struct list_head *pos = head->next;

                struct list_head *batch_end = NULL;

mm/vmscan.c +1 -1

@@ -3984,7 +3984,8 @@ static bool inc_min_seq(struct lruvec *lruvec,
 int type, int swappiness)
                                if (refs + workingset !=
 BIT(LRU_REFS_WIDTH) + 1) {
                                        int tier =
 lru_tier_from_refs(refs, workingset);
 
-                                       protected[tier] += nr_pages;
+

WRITE_ONCE(lrugen->protected[hist][type][tier],
+

lrugen->protected[hist][type][tier] + nr_pages);
                                }
                        } else {
                                flush_lru_batch(head, &batch_end, target_list);

mm/vmscan.c +1 -3

@@ -3999,9 +4000,6 @@ static bool inc_min_seq(struct lruvec *lruvec,
 int type, int swappiness)
                           lrugen->nr_pages[old_gen][type][zone] - delta);
                WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                           lrugen->nr_pages[target_gen][type][zone] + delta);
-               for (int tier = 0; tier < MAX_NR_TIERS; tier++)
-                       WRITE_ONCE(lrugen->protected[hist][type][tier],
-                                  lrugen->protected[hist][type][tier]
+ protected[tier]);
                if (!remaining)
                        return false;
        }

And I see:

With protection batching:

Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Total:   235.718 ms
Average: 2.334 ms

Without protection batching:

Running scope as unit: agetest.scope
mmap: 0x775682e00000, size: 512 MB
memcg: 12823 (/system.slice/agetest.scope)
aging generation 3 -> 103

gen   3:    7.542 ms
gen   4:    0.948 ms
gen   5:    2.338 ms
...
gen 101:    2.340 ms
gen 102:    2.342 ms
gen 103:    2.347 ms

Total:   240.505 ms
Average: 2.381 ms

Quoted 9 lines

It’s 2.334 vs. 2.381, which is really minor.
I’m curious to see what the data looks like on ARM. If there’s no significant difference, we may drop this patch in v2 to reduce the amount of code change.
Xueyuan, since you’ve been testing this patchset on ARM[1], would you mind checking this on ARM as well?

Hi Barry,

I reverted patch 2 and ran the test again on arm64, with THP=never like yours:

full series: 2.964 ms
w/o patch 2: 2.996 ms

So it's 2.964 vs. 2.996, about 1.1% here too. Consistent with your x86 data.

Thanks,
Xueyuan

Quoted 48 lines

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 29 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/vmscan.c +2 -3

index 0d74fc00abd3..99ee3c833d54 100644
@@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
      /* prevent cold/hot inversion if the type is evictable */
      for (zone = 0; zone < MAX_NR_ZONES; zone++) {
              struct list_head *head = &lrugen->folios[old_gen][type][zone];
-             unsigned long delta = 0;
+             unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 
              while (!list_empty(head)) {
                      struct folio *folio = lru_to_folio(head);

@@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                      if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
                              int tier = lru_tier_from_refs(refs, workingset);
 
-                             WRITE_ONCE(lrugen->protected[hist][type][tier],
-                                        lrugen->protected[hist][type][tier] + nr_pages);
+                             protected[tier] += nr_pages;
                      }

Quoted 14 lines

Quoted 12 lines

mm/vmscan.c +3 -1

                      if (!--remaining)

@@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                         lrugen->nr_pages[old_gen][type][zone] - delta);
              WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
                         lrugen->nr_pages[target_gen][type][zone] + delta);
+             for (int tier = 0; tier < MAX_NR_TIERS; tier++)
+                     WRITE_ONCE(lrugen->protected[hist][type][tier],
+                                lrugen->protected[hist][type][tier] + protected[tier]);
              if (!remaining)
                      return false;
      }
--

Quoted 2 lines

2.34.1

Thanks
Barry

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #12 lore raw

PATCH 3/6 mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

In reply to Barry Song (Xiaomi)

During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.

It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().

Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

mm/vmscan.c +5 -2

index 99ee3c833d54..3b618a51cde2 100644
@@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-			if (gen_increased)
+			if (gen_increased) {
 				delta += nr_pages;
+				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			} else {
+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);

Signature · 1 line

2.34.1

Baoquan He

reviewer

2026-08-26 08:57 UTC #13 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 10 lines · Barry Song (Xiaomi)

During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().

While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.

Quoted 31 lines

Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

mm/vmscan.c +5 -2

index 99ee3c833d54..3b618a51cde2 100644
@@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-			list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-			if (gen_increased)
+			if (gen_increased) {
 				delta += nr_pages;
+				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			} else {
+				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+			}
 			/* don't count the workingset being lazily promoted */
 			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 				int tier = lru_tier_from_refs(refs, workingset);

Signature · 1 line

2.34.1

Barry Song

author reviewer

2026-08-26 21:43 UTC #14 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 7 lines · Baoquan He

Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.

Quoted 9 lines

Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.

Hi Baoquan,

Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?

I can explain the design in more detail.

Yes. For both promoted and protected folios, sort_folio() moves them to the head of the corresponding generation.

Here, inc_min_seq() is a bit different. We are overlapping max_seq and min_seq, so the min_seq generation should be moved to the second-oldest generation. Therefore, I think non-promoted folios should be placed at the tail.
They are genuinely not promoted, so they shouldn't be at the head.

For example, suppose we have the following folios:

Second-oldest gen: f1, f2, f3, f4

Oldest gen: f5 (promoted), f6 (not promoted),
f7 (promoted), f8 (not promoted)

Without my patchset, the result is:

Second-oldest:

f1, f2, f3, f4, f8 (promoted), f7 (not promoted),
f6 (not promoted), f5 (promoted)

So you can see that both promoted and non-promoted folios are at the tail of the second-oldest generation?

With my patchset, the result is:

Second-oldest:

f5 (promoted), f8 (promoted), f1, f2, f3, f4,
f7 (not promoted), f6 (not promoted)

Best Regards
Barry

Baoquan He

reviewer

2026-08-27 00:46 UTC #15 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 14 lines · Barry Song

Quoted 13 lines · Baoquan He
Quoted 10 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().

Quoted 8 lines

Quoted 3 lines
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?

Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.

Quoted 38 lines

I can explain the design in more detail.
Yes. For both promoted and protected folios, sort_folio() moves them to the head of the corresponding generation.
Here, inc_min_seq() is a bit different. We are overlapping max_seq and min_seq, so the min_seq generation should be moved to the second-oldest generation. Therefore, I think non-promoted folios should be placed at the tail.
They are genuinely not promoted, so they shouldn't be at the head.
For example, suppose we have the following folios:
Second-oldest gen: f1, f2, f3, f4
Oldest gen: f5 (promoted), f6 (not promoted),
f7 (promoted), f8 (not promoted)
Without my patchset, the result is:
Second-oldest:
f1, f2, f3, f4, f8 (promoted), f7 (not promoted),
f6 (not promoted), f5 (promoted)
So you can see that both promoted and non-promoted folios are at the tail of the second-oldest generation?
With my patchset, the result is:
Second-oldest:
f5 (promoted), f8 (promoted), f1, f2, f3, f4,
f7 (not promoted), f6 (not promoted)
Best Regards
Barry

Barry Song

author reviewer

2026-08-27 01:24 UTC #16 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 22 lines · Baoquan He

Quoted 20 lines · Barry Song
Quoted 7 lines · Baoquan He
Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
Quoted 9 lines
Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,

Quoted 7 lines

Quoted 2 lines
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?
Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.

I think it might be fine for sort_folio() to move protected folios to the head, since those folios have either been accessed multiple times or have reached a tier higher than tier_idx. They are sort of hot in theory, right?

if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)

But for inc_min_seq(), it is just catching up to make sure the newest generation doesn't overlap with the oldest generation. Those non-promoted folios themselves aren't hot , so I feel these are actually different?

Best Regards
Barry

Baoquan He

reviewer

2026-08-27 02:14 UTC #17 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 35 lines · Barry Song

Quoted 22 lines · Baoquan He
Quoted 20 lines · Barry Song
Quoted 7 lines · Baoquan He
Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
Quoted 9 lines
Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,
Quoted 7 lines
Quoted 2 lines
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?
Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.
I think it might be fine for sort_folio() to move protected folios to the head, since those folios have either been accessed multiple times or have reached a tier higher than tier_idx. They are sort of hot in theory, right?

Quoted 5 lines

if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
But for inc_min_seq(), it is just catching up to make sure the newest generation doesn't overlap with the oldest generation. Those non-promoted folios themselves aren't hot , so I feel these are actually different?

I got your point, sort_folio() considers the hottness, inc_min_seq() doesn't. I agree with you now. Thanks for the explanation.

BUT no matter what it is, protected folios, lazily promoted folios, and no matter where it is, put in head of next gen or tail of next gen, their refs are cleared by folio_inc_gen(). Then in sort_folio(), they are all tier 0 of the oldest gen and must be reclaimed.

So here, I think differentiating them and moving them into head or tail doesn't make sense, the thing is whether if we need do something to retain refs of folios when gen_increased. At least, for lazily promoted folios, it should not be put in the tail of next gen and refs cleared. What do you think?

Thanks
Baoquan

Baoquan He

reviewer

2026-08-27 02:20 UTC #18 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 42 lines · Baoquan He

Quoted 35 lines · Barry Song
Quoted 22 lines · Baoquan He
Quoted 20 lines · Barry Song
Quoted 7 lines · Baoquan He
Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
Quoted 9 lines
Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,
Quoted 7 lines
Quoted 2 lines
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?
Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.
I think it might be fine for sort_folio() to move protected folios to the head, since those folios have either been accessed multiple times or have reached a tier higher than tier_idx. They are sort of hot in theory, right?
Quoted 5 lines
if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
But for inc_min_seq(), it is just catching up to make sure the newest generation doesn't overlap with the oldest generation. Those non-promoted folios themselves aren't hot , so I feel these are actually different?

Quoted 7 lines

I got your point, sort_folio() considers the hottness, inc_min_seq() doesn't. I agree with you now. Thanks for the explanation.
BUT no matter what it is, protected folios, lazily promoted folios, and no matter where it is, put in head of next gen or tail of next gen, their refs are cleared by folio_inc_gen(). Then in sort_folio(), they are all tier 0 of the oldest gen and must be reclaimed.

Or mm walking will take a long time, it doesn't matter much about the refs in next gen in inc_min_seq() because there are a lot of chances refs are updated when it comes to sort_folio()?

Quoted 9 lines

So here, I think differentiating them and moving them into head or tail doesn't make sense, the thing is whether if we need do something to retain refs of folios when gen_increased. At least, for lazily promoted folios, it should not be put in the tail of next gen and refs cleared. What do you think?
Thanks
Baoquan

Kairui Song

2026-08-27 04:31 UTC #19 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

In reply to Baoquan He

Quoted 43 lines · Baoquan He

Quoted 35 lines · Barry Song
Quoted 22 lines · Baoquan He
Quoted 20 lines · Barry Song
Quoted 7 lines · Baoquan He
Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
Quoted 9 lines
Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,
Quoted 7 lines
Quoted 2 lines
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?
Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.
I think it might be fine for sort_folio() to move protected folios to the head, since those folios have either been accessed multiple times or have reached a tier higher than tier_idx. They are sort of hot in theory, right?
Quoted 5 lines
if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
But for inc_min_seq(), it is just catching up to make sure the newest generation doesn't overlap with the oldest generation. Those non-promoted folios themselves aren't hot , so I feel these are actually different?

Quoted 7 lines

I got your point, sort_folio() considers the hottness, inc_min_seq() doesn't. I agree with you now. Thanks for the explanation.
BUT no matter what it is, protected folios, lazily promoted folios, and no matter where it is, put in head of next gen or tail of next gen, their refs are cleared by folio_inc_gen(). Then in sort_folio(), they are all tier 0 of the oldest gen and must be reclaimed.

Hi all,

I think this part is not true? a folio's PG_workingset will never be gone after set, that pinns the folio to tier 3 through folio_inc_gen.

In fact, I consider this a pitfall rather than a gain; many workloads have seen regression due to the over protection of such folios. Especially with that "refs + workingset == BIT(LRU_REFS_WIDTH) + 1"

Resetting the tier to a lower position is better if the folio is promoted by one gen due to protection, to avoid over protection IMO.

Changing that semantic will require many other adjustments; I suggest we just ignore that here.

Quoted 6 lines

So here, I think differentiating them and moving them into head or tail doesn't make sense, the thing is whether if we need do something to retain refs of folios when gen_increased. At least, for lazily promoted folios, it should not be put in the tail of next gen and refs cleared. What do you think?

Actually, retaining refs has a very limited effect on the current MGLRU implementation.

Basically I agree we should keep the refs if the folios are not lazy promoted or protected, e.g. in inc_min_seq. But somehow we need a way to "soft reset" the refs to a slight lower value so the higher tier in lower gen won't looks hotter than lower tiers in higher gen.

This is not very doable right now, the closest thing we can archive is just reset the refs field and not touch the PG_workingset bit, which already done right now. (BTW there is a LRU_REFS_WORKIGNSET reset in the FG series for this :-)

Baoquan He

reviewer

2026-08-27 06:14 UTC #20 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

Quoted 52 lines · Kairui Song

Quoted 43 lines · Baoquan He
Quoted 35 lines · Barry Song
Quoted 22 lines · Baoquan He
Quoted 20 lines · Barry Song
Quoted 7 lines · Baoquan He
Quoted 5 lines · Barry Song (Xiaomi)
During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation.
Quoted 9 lines
Quoted 5 lines
It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios().
While sort_folio() move protected folio to the head of next gen too. It only moves ineligible folios to the tail of next gen.
Hi Baoquan,
Quoted 7 lines
Quoted 2 lines
Thanks for the review! I’m not quite sure I understand what you mean :-) Could you please clarify what you’re suggesting?
Sorry for the confusion, Barry. I meant this is a good one, and sort_folio() has the similar issue in which the protected folios are moved to the head, wondering if that need be adjusted too. One consistent rule for both is better.
I think it might be fine for sort_folio() to move protected folios to the head, since those folios have either been accessed multiple times or have reached a tier higher than tier_idx. They are sort of hot in theory, right?
Quoted 5 lines
if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1)
But for inc_min_seq(), it is just catching up to make sure the newest generation doesn't overlap with the oldest generation. Those non-promoted folios themselves aren't hot , so I feel these are actually different?
Quoted 7 lines
I got your point, sort_folio() considers the hottness, inc_min_seq() doesn't. I agree with you now. Thanks for the explanation.
BUT no matter what it is, protected folios, lazily promoted folios, and no matter where it is, put in head of next gen or tail of next gen, their refs are cleared by folio_inc_gen(). Then in sort_folio(), they are all tier 0 of the oldest gen and must be reclaimed.

Quoted 4 lines

Hi all,
I think this part is not true? a folio's PG_workingset will never be gone after set, that pinns the folio to tier 3 through folio_inc_gen.

You are right, I missed the PG_workingset part.

But then folios of refs 0 will get the same treatment as folios of refs 4, even though later the tier_idx == 1 or 2 in sort_folio(). This sounds not reasonable.

Quoted 7 lines

In fact, I consider this a pitfall rather than a gain; many workloads have seen regression due to the over protection of such folios. Especially with that "refs + workingset == BIT(LRU_REFS_WIDTH) + 1"
Resetting the tier to a lower position is better if the folio is promoted by one gen due to protection, to avoid over protection IMO.

Yeah, I got your point and agree. I am thinking of this too, agree we should respect more on gen than tier for protected folio moving, while it's not easy to implement.

Quoted 3 lines

.
Changing that semantic will require many other adjustments; I suggest we just ignore that here.

Hmm, my thinking is if we should differentiate folios w/ refs with folios w/o refs or folios w/ a certain lower refs. Moving them all to the head of next gen, this at least giving them more time to live and update refs because eviction pick folios from the tail; or moving them to tail of next gen, retain limited refs.

Quoted 16 lines

Quoted 7 lines
So here, I think differentiating them and moving them into head or tail doesn't make sense, the thing is whether if we need do something to retain refs of folios when gen_increased. At least, for lazily promoted folios, it should not be put in the tail of next gen and refs cleared. What do you think?
Actually, retaining refs has a very limited effect on the current MGLRU implementation.
Basically I agree we should keep the refs if the folios are not lazy promoted or protected, e.g. in inc_min_seq. But somehow we need a way to "soft reset" the refs to a slight lower value so the higher tier in lower gen won't looks hotter than lower tiers in higher gen.

Quoted 4 lines

This is not very doable right now, the closest thing we can archive is just reset the refs field and not touch the PG_workingset bit, which already done right now. (BTW there is a LRU_REFS_WORKIGNSET reset in the FG series for this :-)

Yeah. I rechecked your patchset and saw the changing, as I said privately, the big patch better be split into smaller ones like Barry has done in this patchset according to logic unit, then reviewing and discussing will be much easier.

Kairui Song

Reviewed-by on v1 3/6

2026-08-27 04:38 UTC #21 lore raw

Re: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq()

In reply to Barry Song (Xiaomi)

Quoted 40 lines · Barry Song (Xiaomi)

Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

mm/vmscan.c +5 -3

index 99ee3c833d54..3b618a51cde2 100644
@@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
                        VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
                        new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
-                       list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
-                       if (gen_increased)
+                       if (gen_increased) {
                                delta += nr_pages;
+                               list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+                       } else {
+                               list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+                       }
                        /* don't count the workingset being lazily promoted */
                        if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
                                int tier = lru_tier_from_refs(refs, workingset);
--

Quoted 3 lines

2.34.1

It looks more consistent with the one in sort_folio, the idea LGTM:

Reviewed-by Kairui Song <[email protected]>

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #22 lore raw

PATCH 4/6 mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

In reply to Barry Song (Xiaomi)

Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio().

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

mm/vmscan.c +6 -7

index 3b618a51cde2..7bd01875fade 100644
@@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (gen_increased) {
 				delta += nr_pages;
 				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					protected[tier] += nr_pages;
+				}
 			} else {
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-
-				protected[tier] += nr_pages;
-			}
-
 			if (!--remaining)
 				break;
 		}

Signature · 1 line

2.34.1

Baoquan He

reviewer Reviewed-by on v1 4/6

2026-08-26 08:57 UTC #23 lore raw

Re: [PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq()

Quoted 2 lines · Barry Song (Xiaomi)

Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio().

Makes sense, the change makes it consistent with sort_filio.

Reviewed-by Baoquan He <[email protected]>

Quoted 36 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

mm/vmscan.c +6 -7

index 3b618a51cde2..7bd01875fade 100644
@@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			if (gen_increased) {
 				delta += nr_pages;
 				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				/* don't count the workingset being lazily promoted */
+				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
+					int tier = lru_tier_from_refs(refs, workingset);
+
+					protected[tier] += nr_pages;
+				}
 			} else {
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
-			/* don't count the workingset being lazily promoted */
-			if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
-				int tier = lru_tier_from_refs(refs, workingset);
-
-				protected[tier] += nr_pages;
-			}
-
 			if (!--remaining)
 				break;
 		}

Signature · 1 line

2.34.1

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #24 lore raw

PATCH 5/6 mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

In reply to Barry Song (Xiaomi)

For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

mm/vmscan.c +17 -2

index 7bd01875fade..2fd82b2ca4d1 100644
@@ -191,8 +191,20 @@ struct scan_control {
 			prefetchw(&prev->_field);			\
 		}							\
 	} while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)			\
+	do {								\
+		if ((_folio)->lru.next != _base) {			\
+			struct folio *next;				\
+									\
+			next = list_entry((_folio)->lru.next,		\
+					struct folio, lru);		\
+			prefetchw(&next->_field);			\
+		}							\
+	} while (0)
+
 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*

@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+		struct list_head *pos = head->next;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);

@@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
+			prefetchw_next_lru_folio(folio, head, flags);
+			pos = pos->next;
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;

Signature · 1 line

2.34.1

Kairui Song

2026-08-22 05:45 UTC #25 lore raw

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

Quoted 8 lines · Barry Song (Xiaomi)

For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.
Signed-off-by Barry Song (Xiaomi) <[email protected]>

Hi Barry

This looks a really good idea, thanks!

Quoted 13 lines

---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

mm/vmscan.c +1 -0

index 7bd01875fade..2fd82b2ca4d1 100644
@@ -191,8 +191,20 @@ struct scan_control {
                        prefetchw(&prev->_field);                       \
                }                                                       \
        } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)                        \

mm/vmscan.c struct scan_control { · line 204

Quoted 10 lines

mm/vmscan.c +10 -0

+       do {                                                            \
+               if ((_folio)->lru.next != _base) {                      \
+                       struct folio *next;                             \
+                                                                       \
+                       next = list_entry((_folio)->lru.next,           \
+                                       struct folio, lru);             \
+                       prefetchw(&next->_field);                       \
+               }                                                       \
+       } while (0)
+

I got following warning from checkpatch:

● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
function-like macro
● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
function-like macro
● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
function-like macro

Maybe you could try b4; it helps run these checks automatically. Feel free to ignore if you think these warning at pointless.

Quoted 7 lines

mm/vmscan.c +1 -0

 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3949

Quoted 8 lines

mm/vmscan.c +2 -2

@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
        for (zone = 0; zone < MAX_NR_ZONES; zone++) {
                struct list_head *head = &lrugen->folios[old_gen][type][zone];
                unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+               struct list_head *pos = head->next;
 
-               while (!list_empty(head)) {
-                       struct folio *folio = lru_to_folio(head);
+               while (pos != head) {

Do we need to change the while condition now? Since this commit still moves folios one by one, will it stop when the list is empty?

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3962

Quoted 2 lines

mm/vmscan.c +2 -0

+                       prefetchw_next_lru_folio(folio, head, flags);
+                       pos = pos->next;

I tried prefetching in MGLRU previously, and it didn't look very good but there is no regression either; perhaps it's very arch-dependent. Anyway, I think we can keep it here, maybe further optimize the prefetch later.

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3965

Quoted 3 lines

mm/vmscan.c +0 -0

                        new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
                        if (gen_increased) {
                                delta += nr_pages;

The rest looks good to me, thanks!

Barry Song

author reviewer

2026-08-25 21:32 UTC #26 lore raw

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

Quoted 48 lines · Kairui Song

Quoted 8 lines · Barry Song (Xiaomi)
For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.
Signed-off-by Barry Song (Xiaomi) <[email protected]>
Hi Barry
This looks a really good idea, thanks!
Quoted 13 lines
---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
mm/vmscan.c +1 -0
index 7bd01875fade..2fd82b2ca4d1 100644
@@ -191,8 +191,20 @@ struct scan_control {
                        prefetchw(&prev->_field);                       \
                }                                                       \
        } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)                        \
mm/vmscan.c struct scan_control { · line 204
Quoted 10 lines
mm/vmscan.c +10 -0
+       do {                                                            \
+               if ((_folio)->lru.next != _base) {                      \
+                       struct folio *next;                             \
+                                                                       \
+                       next = list_entry((_folio)->lru.next,           \
+                                       struct folio, lru);             \
+                       prefetchw(&next->_field);                       \
+               }                                                       \
+       } while (0)
+
I got following warning from checkpatch:
● checkpatch.pl: 92: WARNING: Argument '_folio' is not used in
function-like macro
● checkpatch.pl: 92: WARNING: Argument '_base' is not used in
function-like macro
● checkpatch.pl: 92: WARNING: Argument '_field' is not used in
function-like macro

Quoted 2 lines

Maybe you could try b4; it helps run these checks automatically. Feel free to ignore if you think these warning at pointless.

Yep. I vividly remember adding this rule to the coding style, and Xining added the corresponding checkpatch.pl change:

commit 6813216bbdba1 ("Documentation: coding-style: ask function-like macros to evaluate parameters")
commit b1be5844c1a01 ("scripts: checkpatch: check unused parameters for function-like macro")

But once we started writing the code, I was influenced by its context and ended up following that context instead.

Quoted 13 lines

mm/vmscan.c +2 -0

 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*

@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
        for (zone = 0; zone < MAX_NR_ZONES; zone++) {
                struct list_head *head = &lrugen->folios[old_gen][type][zone];
                unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+               struct list_head *pos = head->next;

Quoted 6 lines

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3949
Quoted 3 lines
mm/vmscan.c +1 -2
-               while (!list_empty(head)) {
-                       struct folio *folio = lru_to_folio(head);
+               while (pos != head) {
Do we need to change the while condition now? Since this commit still moves folios one by one, will it stop when the list is empty?

For this patch, we don't need to change the while condition. That change is more relevant to the next patch:
https://lore.kernel.org/linux-mm/[email protected]/

So I guess I could move this while (pos != head) change to the next patch if that makes the review easier.

Quoted 8 lines

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3962
Quoted 3 lines
mm/vmscan.c +2 -0
+                       prefetchw_next_lru_folio(folio, head, flags);
+                       pos = pos->next;
I tried prefetching in MGLRU previously, and it didn't look very good but there is no regression either; perhaps it's very arch-dependent. Anyway, I think we can keep it here, maybe further optimize the prefetch later.

Yep. I guess we may want to keep it to help architectures that are sensitive to prefetching.

Quoted 6 lines

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3965

Quoted 4 lines

mm/vmscan.c +0 -0

                        new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
                        if (gen_increased) {
                                delta += nr_pages;

The rest looks good to me, thanks!

Thanks for the review!

Best Regards
Barry

Baoquan He

reviewer Reviewed-by on v1 5/6

2026-08-26 09:06 UTC #27 lore raw

Re: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail

In reply to Barry Song (Xiaomi)

Quoted 5 lines · Barry Song (Xiaomi)

For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion.

This is great, do we need do the similar thing on scan_folios()? While it's not relevant to this patch.

Reviewed-by Baoquan He <[email protected]>

Quoted 56 lines

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

mm/vmscan.c +17 -2

index 7bd01875fade..2fd82b2ca4d1 100644
@@ -191,8 +191,20 @@ struct scan_control {
 			prefetchw(&prev->_field);			\
 		}							\
 	} while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field)			\
+	do {								\
+		if ((_folio)->lru.next != _base) {			\
+			struct folio *next;				\
+									\
+			next = list_entry((_folio)->lru.next,		\
+					struct folio, lru);		\
+			prefetchw(&next->_field);			\
+		}							\
+	} while (0)
+
 #else
 #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0)
+#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0)
 #endif
 
 /*

@@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
+		struct list_head *pos = head->next;
 
-		while (!list_empty(head)) {
-			struct folio *folio = lru_to_folio(head);
+		while (pos != head) {
+			struct folio *folio = list_entry(pos, struct folio, lru);
 			long nr_pages = folio_nr_pages(folio);
 			int refs = folio_lru_refs(folio);
 			bool workingset = folio_test_workingset(folio);

@@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
 			VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
 
+			prefetchw_next_lru_folio(folio, head, flags);
+			pos = pos->next;
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;

Signature · 1 line

2.34.1

Barry Song (Xiaomi)

author reviewer

2026-08-21 10:26 UTC #28 lore raw

PATCH 6/6 mm/mglru: batch move folios to the second-oldest gen's LRU

In reply to Barry Song (Xiaomi)

Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type.

Assisted-by gemini:gemini-3.6-flash

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

mm/vmscan.c +20 -1

index 2fd82b2ca4d1..996b48344ed0 100644
@@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)
+{
+	LIST_HEAD(movable);
+
+	if (!*batch_end)
+		return;
+
+	list_cut_position(&movable, head, *batch_end);
+	list_splice_tail_init(&movable, dst);
+	*batch_end = NULL;
+}
+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;

@@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 
 		while (pos != head) {
 			struct folio *folio = list_entry(pos, struct folio, lru);

@@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
+
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 					int tier = lru_tier_from_refs(refs, workingset);

@@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 					protected[tier] += nr_pages;
 				}
 			} else {
+				flush_lru_batch(head, &batch_end, target_list);
 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
 			if (!--remaining)
 				break;
 		}
+		flush_lru_batch(head, &batch_end, target_list);
+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],

Signature · 1 line

2.34.1

Baoquan He

reviewer Reviewed-by on v1 6/6

2026-08-26 09:34 UTC #29 lore raw

Re: [PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU

Quoted 59 lines · Barry Song (Xiaomi)

Assisted-by gemini:gemini-3.6-flash

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/vmscan.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

mm/vmscan.c +17 -1

index 2fd82b2ca4d1..996b48344ed0 100644
@@ -3923,6 +3923,19 @@ static void clear_mm_walk(void)
 		kfree(walk);
 }
 
+static inline void flush_lru_batch(struct list_head *head, struct list_head **batch_end,
+				   struct list_head *dst)
+{
+	LIST_HEAD(movable);
+
+	if (!*batch_end)
+		return;
+
+	list_cut_position(&movable, head, *batch_end);
+	list_splice_tail_init(&movable, dst);
+	*batch_end = NULL;
+}
+
 static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 {
 	int zone;

@@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 
 	/* prevent cold/hot inversion if the type is evictable */
 	for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+		struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
 		struct list_head *head = &lrugen->folios[old_gen][type][zone];
 		unsigned long protected[MAX_NR_TIERS] = {}, delta = 0;
 		struct list_head *pos = head->next;
+		struct list_head *batch_end = NULL;
 
 		while (pos != head) {
 			struct folio *folio = list_entry(pos, struct folio, lru);

@@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 			new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
 			if (gen_increased) {
 				delta += nr_pages;
-				list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
+				batch_end = &folio->lru;
+
 				/* don't count the workingset being lazily promoted */
 				if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
 					int tier = lru_tier_from_refs(refs, workingset);

mm/vmscan.c static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · line 3990

Quoted 4 lines

mm/vmscan.c +1 -0

@@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
 					protected[tier] += nr_pages;
 				}
 			} else {
+				flush_lru_batch(head, &batch_end, target_list);

How about this? then you only have "struct list_head *batch_end" in 2nd parameter. And explicit resetting batch_end makes code more readable?

				flush_lru_batch(head, batch_end, target_list);
				batch_end = NULL;

Anyway, personal preference, not strong opinion.

Other than the nit,

Reviewed-by Baoquan He <[email protected]>

Quoted 13 lines

mm/vmscan.c +2 -0

 				list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
 			}
 			if (!--remaining)
 				break;
 		}
+		flush_lru_batch(head, &batch_end, target_list);
+
 		WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
 			   lrugen->nr_pages[old_gen][type][zone] - delta);
 		WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],

Signature · 1 line

2.34.1

Xueyuan Chen

Tested-by on the v1 series

2026-08-27 03:54 UTC #30 lore raw

In reply to Barry Song (Xiaomi)

Quoted 226 lines · Barry Song (Xiaomi)

This is an aging speedup series split out from the MGLRU swappiness series [1], with the inc_min_seq changes separated to make them easier to review.
Currently, inc_min_seq performance is crucial to both the swappiness fix and proactive aging. There are two problems with it:
1. It processes each folio one by one, while many operations can be batched or skipped. For example, a batch of folios can be moved together from the oldest generation to the second-oldest generation, and the associated counting can also be done in batches.
2. It may cause potential cold/hot inversion by placing promoted folios (which have been scanned and found to have young PTEs) behind non-promoted folios. A similar inversion can also occur among non-promoted folios, as tail folios from the oldest generation are placed before head folios when moving them to the second-oldest generation.
This series tries to batch operations as much as possible and fix the potential cold/hot inversion by keeping promoted folios ahead of non-promoted folios, while also preserving the order of non-promoted folios when moving them from the oldest generation to the second-oldest generation.
Minor issue: inc_min_seq() also counts protected folios improperly, as promoted folios should be skipped, as in sort_folio().
We need a stable workload with a stable number of folios to measure aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT to generate the microbenchmark below. It ages an LRU vec containing 512 MB of memory 100 times:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#include <sys/mman.h>
#define SIZE (512UL * 1024 * 1024)
#define LRU_GEN "/sys/kernel/debug/lru_gen"
#define TARGET_CGROUP "/system.slice/agetest.scope"
#define START_GEN 3
#define END_GEN 103
static long long nsec_diff(const struct timespec *start,
const struct timespec *end)
{
	return (end->tv_sec - start->tv_sec) * 1000000000LL +
	       (end->tv_nsec - start->tv_nsec);
}
static int find_memcg_id(void)
{
	FILE *fp;
	char line[4096];
	int memcg_id;

	fp = fopen(LRU_GEN, "r");
	if (!fp) {
		perror("fopen lru_gen");
		return -1;
	}

	while (fgets(line, sizeof(line), fp)) {
		char *p;

		if (strncmp(line, "memcg ", 6))
			continue;

		p = line + 6;

		if (sscanf(p, "%d", &memcg_id) != 1)
			continue;

		/*
		 * The memcg path follows the numeric ID.
		 */
		p = strchr(p, ' ');
		if (!p)
			continue;

		if (strstr(p, TARGET_CGROUP)) {
			fclose(fp);
			return memcg_id;
		}
	}

	fclose(fp);

	fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
	return -1;
}
int main(void)
{
	void *addr;
	int memcg_id;
	int fd;
	long long total_ns = 0;

	/*
	 * mmap 512 MB and touch every page.
	 */
	addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	if (addr == MAP_FAILED) {
		perror("mmap");
		return 1;
	}

	memset(addr, 0x55, SIZE);

	printf("mmap: %p, size: %lu MB\n",
	       addr, SIZE / 1024 / 1024);

	/*
	 * Find the memcg ID automatically.
	 */
	memcg_id = find_memcg_id();
	if (memcg_id < 0)
		return 1;

	printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
	printf("aging generation %d -> %d\n",
	       START_GEN, END_GEN);

	fd = open(LRU_GEN, O_WRONLY);
	if (fd < 0) {
		perror("open lru_gen");
		return 1;
	}

	for (int gen = START_GEN; gen <= END_GEN; gen++) {
		char buf[128];
		int len;
		struct timespec start, end;
		long long ns;

		len = snprintf(buf, sizeof(buf),
			       "+ %d 0 %d\n", memcg_id, gen);

		clock_gettime(CLOCK_MONOTONIC, &start);

		if (write(fd, buf, len) != len) {
			perror("write lru_gen");
			close(fd);
			return 1;
		}

		clock_gettime(CLOCK_MONOTONIC, &end);

		ns = nsec_diff(&start, &end);
		total_ns += ns;

		printf("gen %3d: %8.3f ms\n",
		       gen, ns / 1000000.0);
		fflush(stdout);
	}

	close(fd);

	printf("\nTotal:   %.3f ms\n",
	       total_ns / 1000000.0);
	printf("Average: %.3f ms\n",
	       total_ns / (double)(END_GEN - START_GEN + 1) /
	       1000000.0);

	while (1)
		sleep(1);

	return 0;
}
Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest
I’m seeing inc_min_seq() become significantly faster:
W/o patch:
Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.433 ms
gen   4:    0.949 ms
gen   5:    2.535 ms
gen   6:    5.043 ms
gen   7:    5.041 ms
gen   8:    5.027 ms
...
gen 100:    5.035 ms
gen 101:    5.011 ms
gen 102:    5.029 ms
gen 103:    5.056 ms
Total:   503.946 ms
Average: 4.990 ms
W/ patch:
Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen   3:    7.538 ms
gen   4:    0.937 ms
gen   5:    2.348 ms
gen   6:    2.300 ms
gen   7:    2.302 ms
gen   8:    2.294 ms
gen   9:    2.296 ms
...
gen 100:    2.292 ms
gen 101:    2.307 ms
gen 102:    2.293 ms
gen 103:    2.293 ms

Quoted 4 lines

Total:   235.718 ms
Average: 2.334 ms
The average aging time drops from 4.990 ms to 2.334 ms!

Hi Barry,

I tested this series on my arm64 machine (24 cores, 4K base pages) and reproduced the improvement with the microbenchmark from the cover letter.

THP=never (PTE):
baseline: 7.644 ms
patched: 2.964 ms (-61.2%)

THP=always (PMD):
baseline: 0.0373 ms
patched: 0.0292 ms (-21.8%)

The PTE-level gain is larger than your x86 numbers (-61.2% vs -53.2%). I suspect the per-folio cost of inc_min_seq() is higher on my arm64 machine (or on arm64 in general).

The gain is also larger at the PTE level than at the PMD level (-61.2% vs -21.8%), matching the much higher folio count (131072 vs 256).

Tested-by Xueyuan Chen <[email protected]>

thanks,
Xueyuan

Quoted 19 lines

[1] https://lore.kernel.org/linux-mm/[email protected]/
Barry Song (Xiaomi) (6):
mm/mglru: batch update lrugen->nr_pages in inc_min_seq() mm/mglru: batch update lrugen->protected in inc_min_seq() mm/mglru: enhance cold/hot inversion handling in inc_min_seq() mm/mglru: exclude folios promoted by aging from protected in
inc_min_seq()
mm/mglru: move folios from oldest gen to second-oldest gen from head
to tail
mm/mglru: batch move folios to the second-oldest gen's LRU
mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 80 insertions(+), 18 deletions(-)
Signature · 1 line
2.34.1