mm/mglru: deactivate anon folios into the oldest generation
Subsystems (2)
PATCH 1/1 mm/mglru: deactivate anon folios into the oldest generation
From: Ridong Chen <[email protected]>
folio_deactivate() is used by MADV_COLD and DAMOS_LRU_DEPRIO to move a folio towards eviction. lru_deactivate() does that by deleting the folio from its LRU list, clearing PG_active and PG_referenced, and adding it back with lruvec_add_folio().
With MGLRU, lru_gen_folio_seq() places a swap-backed folio that is not in the swap cache, i.e. an anon or shmem folio, into max_seq - 1, regardless of its refs. That generation is active per lru_gen_is_active(), so such a folio never reaches an inactive generation, and a folio that has aged into min_seq + 1 is moved to a younger generation instead. PGDEACTIVATE is still counted. Other folios are not affected: with their refs cleared they are placed into the oldest generation.
On a 4-generation memcg, 64MB of aged anon memory in min_seq + 1 ends up in max_seq - 1 after MADV_COLD, while 64MB of file memory in the same generation ends up in min_seq.
As a result, MADV_COLD can make anon memory less likely to be reclaimed. It can be reproduced with a single process in a memcg with swap enabled:
1) map two 64MB anon regions A and B and fault them in, A first;
2) age the memcg twice through /sys/kernel/debug/lru_gen, so that both
regions are in the same generation, min_seq + 1;
3) apply MADV_COLD to B only, A is the control;
4) reclaim the size of one region from anon only:
echo "64M swappiness=max" > memory.reclaim; 5) count the swapped out pages of A and B from /proc/self/pagemap.
B is expected to be swapped out instead of A. Three runs on mm-new:
A swapped out B (MADV_COLD) swapped out before 16384 pages 15 ~ 128 pages after 0 ~ 157 pages 16384 pages
Use lruvec_add_folio_tail() for MGLRU, which treats the folio as reclaiming and adds it to the tail of the oldest generation, the same place lru_deactivate_file() puts a clean file folio. The other folios, which were added to the head of the oldest generation, are now added to its tail as well, so they are reclaimed first as MADV_COLD intends.
---
This patch was developed with the help of an AI coding assistant (Claude).
The reproducer, run as root in a memcg with swap enabled and debugfs mounted, e.g. "mkdir /sys/fs/cgroup/t; ./reclaim-order /sys/fs/cgroup/t":
// SPDX-License-Identifier: GPL-2.0 /* * Two 64MB anon regions, A (hot) and B (MADV_COLD). Reclaim 64MB via * memory.reclaim (swappiness=max -> anon only). Report how much of each * region was swapped out (mincore can't see swap; use /proc/self/pagemap). */ #define _GNU_SOURCE #include <sched.h> #include <stdio.h> #include <stdlib.h> #include <stdint.h> #include <string.h> #include <unistd.h> #include <fcntl.h> #include <sys/mman.h>
static unsigned long long memcg_id;
static const char *cg_name;
/* force one aging step on node 0 of our memcg via debugfs */
static void aging(void)
{
FILE *f = fopen("/sys/kernel/debug/lru_gen", "r");
char line[512], cmd[128];
unsigned long max = 0, s, x1, x2, x3;
int in = 0, fd, len;
while (fgets(line, sizeof(line), f)) {
unsigned long long id; char path[256];
if (sscanf(line, "memcg %llu %255s", &id, path) == 2) {
in = !strcmp(path, cg_name);
if (in) memcg_id = id;
continue;
}
if (in && sscanf(line, " %lu %lu %lu %lu", &s, &x1, &x2, &x3) == 4)
max = s;
}
fclose(f);
len = snprintf(cmd, sizeof(cmd), "+ %llu 0 %lu 200 1\n", memcg_id, max);
fd = open("/sys/kernel/debug/lru_gen", O_WRONLY);
if (write(fd, cmd, len) != len) perror("aging");
close(fd);
}
static long swapped(char *p, size_t sz)
{
int fd = open("/proc/self/pagemap", O_RDONLY);
long n = 0;
for (size_t off = 0; off < sz; off += 4096) {
uint64_t e;
pread(fd, &e, 8, ((uintptr_t)(p + off) / 4096) * 8);
if (e & (1ULL << 62))
n++;
}
close(fd);
return n;
}
int main(int argc, char **argv)
{
const char *cg = argv[1];
cg_name = strrchr(cg, '/');
size_t sz = 64UL << 20;
char buf[256], *a, *b, *d;
cpu_set_t set;
int fd;
CPU_ZERO(&set); CPU_SET(0, &set);
sched_setaffinity(0, sizeof(set), &set);
snprintf(buf, sizeof(buf), "%s/cgroup.procs", cg);
fd = open(buf, O_WRONLY); dprintf(fd, "%d\n", getpid()); close(fd);
a = mmap(NULL, sz, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
b = mmap(NULL, sz, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
d = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
madvise(a, sz, MADV_NOHUGEPAGE); madvise(b, sz, MADV_NOHUGEPAGE);
/* A is faulted in first, so without MADV_COLD it is the older one */
for (size_t i = 0; i < sz; i += 4096) a[i] = 1;
for (size_t i = 0; i < sz; i += 4096) b[i] = 1;
d[0] = 1;
madvise(d, 4096, MADV_COLD); /* drain lru_add */
/* two agings: clear young bits of both A and B, both end at min_seq+1 */
aging();
aging();
madvise(b, sz, MADV_COLD);
madvise(d, 4096, MADV_COLD);
snprintf(buf, sizeof(buf), "%s/memory.reclaim", cg);
fd = open(buf, O_WRONLY);
if (dprintf(fd, "64M swappiness=max\n") < 0)
perror("memory.reclaim");
close(fd);
printf("hot A swapped: %5ld pages cold B swapped: %5ld pages (of %zu)\n",
swapped(a, sz), swapped(b, sz), sz >> 12);
return 0;
}mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/folio.c +10 -1
@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio); +/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);__count_vm_events (PGDEACTIVATE , nr_pages);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_pages);
base-commit: 2dc0136b1fdac153434ab02778fcdb32db5d0dae
Signature · 1 line
2.34.1
Quoted 46 lines · Ridong Chen
From: Ridong Chen <[email protected]>
folio_deactivate()is used byMADV_COLDandDAMOS_LRU_DEPRIOto move a folio towards eviction.lru_deactivate()does that by deleting the folio from its LRU list, clearingPG_activeandPG_referenced, and adding it back withlruvec_add_folio().With MGLRU,
lru_gen_folio_seq()places a swap-backed folio that is not in the swap cache, i.e. an anon or shmem folio, intomax_seq- 1, regardless of its refs. That generation is active perlru_gen_is_active(), so such a folio never reaches an inactive generation, and a folio that has aged intomin_seq+ 1 is moved to a younger generation instead. PGDEACTIVATE is still counted. Other folios are not affected: with their refs cleared they are placed into the oldest generation.On a 4-generation memcg, 64MB of aged anon memory in
min_seq+ 1 ends up inmax_seq- 1 afterMADV_COLD, while 64MB of file memory in the same generation ends up inmin_seq.As a result,
MADV_COLDcan make anon memory less likely to be reclaimed. It can be reproduced with a single process in a memcg with swap enabled:1) map two 64MB anon regions A and B and fault them in, A first;
2) age the memcg twice through/sys/kernel/debug/lru_gen, so that bothregions are in the same generation, min_seq + 1; 3) apply MADV_COLD to B only, A is the control; 4) reclaim the size of one region from anon only: echo "64M swappiness=max" > memory.reclaim;5) count the swapped out pages of A and B from
/proc/self/pagemap.B is expected to be swapped out instead of A. Three runs on mm-new:
A swapped out B (MADV_COLD) swapped out before 16384 pages 15 ~ 128 pages after 0 ~ 157 pages 16384 pagesUse
lruvec_add_folio_tail()for MGLRU, which treats the folio as reclaiming and adds it to the tail of the oldest generation, the same placelru_deactivate_file()puts a clean file folio. The other folios, which were added to the head of the oldest generation, are now added to its tail as well, so they are reclaimed first asMADV_COLDintends.Fixes 081488051d28 ("mm/mglru: fix underprotected page cache")Signed-off-by Ridong Chen <[email protected]>---
...
Quoted 13 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);
mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);
Hi Ridong,
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]
Quoted 25 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
Quoted 3 lines
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Quoted 28 lines · Barry Song
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]Quoted 25 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
Quoted 7 lines
Quoted 3 lines
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Right, but lruvec_add_folio_tail also added the folio to the very tail of the inactive / oldest list, could it be a gain or problem if we do that too for CLRU?
Quoted 34 lines · Kairui Song
Quoted 32 lines · Barry Song
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]Quoted 28 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
Quoted 7 lines
Quoted 3 lines
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Right, but
lruvec_add_folio_tailalso added the folio to the very tail of the inactive / oldest list, could it be a gain or problem if we do that too for CLRU?
My gut feeling is that MADV_COLD says the memory is cold. it seems to make sense to do it all the same for both mglru and clru - to the tail of inactive(CLRU) or oldest gen(MGLRU).
not quite sure if userspace is expecting things like this. I feel it should expect this if it says it is cold?
Quoted 27 lines · Barry Song
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]Quoted 25 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
Quoted 7 lines
Quoted 3 lines
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Hi Kairui, Barry,
Thanks for taking a look.
Barry is right about the reason. With the classic LRU, lruvec_add_folio() adds the deactivated folio to the head of the inactive list. With MGLRU, lruvec_add_folio() can't put an anon folio into the oldest generation at all: lru_gen_folio_seq() places a swap-backed folio that is not in the swap cache into max_seq - 1, regardless of its refs. The only way to reach the oldest generation is the reclaiming path of lruvec_add_folio_tail(), and that path adds the folio to the tail.
And you are right that this makes it more aggressive than the classic LRU. With this patch, a deactivated folio is evicted before the folios already in the oldest generation, while the classic LRU puts it behind the folios already on the inactive list. It also changes file folios:
before this patch they are added to the head of the oldest generation, the same as the classic LRU, and now they go to the tail.
So the problem in the current code is that a deactivated anon folio is promoted into an active generation, and with this patch it goes one step further than the classic LRU in the other direction.
Signature · 2 lines
Best regards Ridong
Quoted 59 lines · Ridong Chen
Quoted 27 lines · Barry Song
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]Quoted 25 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
Quoted 7 lines
Quoted 3 lines
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Hi Kairui, Barry,
Thanks for taking a look.
Barry is right about the reason. With the classic LRU,
lruvec_add_folio()adds the deactivated folio to the head of the inactive list. With MGLRU,lruvec_add_folio()can't put an anon folio into the oldest generation at all:lru_gen_folio_seq()places a swap-backed folio that is not in the swap cache intomax_seq- 1, regardless of its refs. The only way to reach the oldest generation is the reclaiming path oflruvec_add_folio_tail(), and that path adds the folio to the tail.And you are right that this makes it more aggressive than the classic LRU. With this patch, a deactivated folio is evicted before the folios already in the oldest generation, while the classic LRU puts it behind the folios already on the inactive list. It also changes file folios:
before this patch they are added to the head of the oldest generation, the same as the classic LRU, and now they go to the tail.
Quoted 3 lines
So the problem in the current code is that a deactivated anon folio is promoted into an active generation, and with this patch it goes one step further than the classic LRU in the other direction.
I think MADV_COLD is fundamentally broken for anonymous folios under MGLRU!
An anonymous folio may already be in an older generation, but lru_deactivate() can actually promote it to the second-youngest generation, effectively making it hotter instead of colder!
Right. For anonymous folios, we have no way to insert them at the head of the oldest generation without introducing a new flag like PG_cold, which is unlikely to be accepted.
If we instead insert them at the tail of the oldest generation, I'm not sure whether that would be consistent with the historical semantics of MADV_COLD.
But I've been thinking about how to insert a folio at the head of the oldest generation. Why can't we simply move the folio directly from its current generation to the head of the oldest generation in lru_gen_deactivate_folio()?
For example, something along these lines:
lru_gen_deactivate_folio()
{
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int gen = lru_gen_from_seq(lrugen->min_seq[type]);
if (folio_lru_gen(folio) < 0)
return false;
folio_clear_active(folio);
folio_clear_referenced(folio);
list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
}Right now, lru_deactivate() seems unnecessarily complicated, incurring additional CPU overhead and LRU lock contention:
1. Remove the folio from the LRU list.
2. Add the folio to the LRU cache.
3. Drain the LRU cache.
4. Add the folio back to the LRU list.
Why do we need all these intermediate steps instead of moving the folio directly in a single operation, given that we're already manipulating the LRU list?
Am I missing something?
Thanks,
Barry
Quoted 87 lines · Barry Song
Quoted 59 lines · Ridong Chen
Quoted 27 lines · Barry Song
On Thu, Nov 12, 2026 at 4:38 PM Kairui Song <[email protected]> wrote:
[...]Quoted 25 lines
Quoted 12 lines
mm/folio.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)mm/folio.c +0
-1@@ -583,7 +583,16 @@ static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)lruvec_del_folio (lruvec, folio);folio_clear_active (folio);folio_clear_referenced (folio); -lruvec_add_folio (lruvec, folio);mm/folio.c static void lru_deactivate(struct lruvec *lruvec, struct folio *folio) · line 595
Quoted 10 lines
mm/folio.c +10
-0+/* +* For MGLRU, lruvec_add_folio() places anon folios that are not in +* the swap cache into the second youngest generation, which is an +* active one. Add the folio to the oldest generation instead, like +* lru_deactivate_file() does for clean folios. +*/ +if (lru_gen_enabled ()) +lruvec_add_folio_tail (lruvec, folio); +else +lruvec_add_folio (lruvec, folio);Hi Ridong,
Quoted 7 lines
Quoted 3 lines
One question here, why MGLRU/CLRU have to use two different ways to add the folio back? Will this cause over aggressive reclaim of the folio?
I guess that is because on mglru, that is lacing a way to add the folio to the head of the oldest generation for anon folios while CLRU can do that :-)
Hi Kairui, Barry,
Thanks for taking a look.
Barry is right about the reason. With the classic LRU,
lruvec_add_folio()adds the deactivated folio to the head of the inactive list. With MGLRU,lruvec_add_folio()can't put an anon folio into the oldest generation at all:lru_gen_folio_seq()places a swap-backed folio that is not in the swap cache intomax_seq- 1, regardless of its refs. The only way to reach the oldest generation is the reclaiming path oflruvec_add_folio_tail(), and that path adds the folio to the tail.And you are right that this makes it more aggressive than the classic LRU. With this patch, a deactivated folio is evicted before the folios already in the oldest generation, while the classic LRU puts it behind the folios already on the inactive list. It also changes file folios:
before this patch they are added to the head of the oldest generation, the same as the classic LRU, and now they go to the tail.Quoted 3 lines
So the problem in the current code is that a deactivated anon folio is promoted into an active generation, and with this patch it goes one step further than the classic LRU in the other direction.
I think
MADV_COLDis fundamentally broken for anonymous folios under MGLRU!An anonymous folio may already be in an older generation, but
lru_deactivate()can actually promote it to the second-youngest generation, effectively making it hotter instead of colder!Right. For anonymous folios, we have no way to insert them at the head of the oldest generation without introducing a new flag like
PG_cold, which is unlikely to be accepted.If we instead insert them at the tail of the oldest generation, I'm not sure whether that would be consistent with the historical semantics of
MADV_COLD.But I've been thinking about how to insert a folio at the head of the oldest generation. Why can't we simply move the folio directly from its current generation to the head of the oldest generation in
lru_gen_deactivate_folio()?For example, something along these lines:
lru_gen_deactivate_folio()
Quoted 14 lines
{
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int gen = lru_gen_from_seq(lrugen->min_seq[type]);
if (folio_lru_gen(folio) < 0)
return false;
folio_clear_active(folio);
folio_clear_referenced(folio);
list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
}I think that works, and it is better than what this patch does. It puts both anon and file folios at the head of the oldest generation, which matches the classic LRU, and it avoids the two problems I mentioned (too aggressive for anon, and moving file folios from the head to the tail).
A plain list_move() is not enough though. We also need to:
- update the gen bits in folio->flags with a cmpxchg loop, as folio_inc_gen() does, since folio_update_gen() can change them locklessly from the page table walk;
- fix up lrugen->nr_pages and the LRU sizes. lru_gen_update_size() warns on an active -> inactive move ("demotion requires isolation"), so this would be done as a deletion from old_gen followed by an addition to new_gen.
min_seq only advances under the LRU lock, so it is stable here. Something like:
static bool lru_gen_deactivate_folio(struct lruvec *lruvec,
struct folio *folio)
{
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int new_gen = lru_gen_from_seq(lrugen->min_seq[type]);
unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
int old_gen;
do {
old_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
if (old_gen < 0)
return false;
new_flags = old_flags & ~LRU_GEN_MASK;
new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
if (old_gen != new_gen) {
lru_gen_update_size(lruvec, folio, old_gen, -1);
lru_gen_update_size(lruvec, folio, -1, new_gen);
}
list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
return true;
}Quoted 8 lines
Right now,
lru_deactivate()seems unnecessarily complicated, incurring additional CPU overhead and LRU lock contention:1. Remove the folio from the LRU list.
2. Add the folio to the LRU cache.
3. Drain the LRU cache.
4. Add the folio back to the LRU list.
Quoted 6 lines
Why do we need all these intermediate steps instead of moving the folio directly in a single operation, given that we're already manipulating the LRU list?
Am I missing something?
I don't think the batch itself is the problem. The folio stays on the LRU while it sits in the per-cpu batch; only when the batch is drained do we take the LRU lock once for up to a batch of folios and do del + add under it. So the batch reduces lock contention rather than adding to it. What the in-place move saves is the del/add pair inside the lock.
So I plan to keep the batching and, in lru_deactivate(), call lru_gen_deactivate_folio() instead of lruvec_del_folio() + lruvec_add_folio() when MGLRU is enabled. I'll test it and send v2 if that sounds good to you.
Signature · 2 lines
Best regards Ridong
Quoted 87 lines · Ridong Chen
Quoted 17 lines · Barry Song
For example, something along these lines:
lru_gen_deactivate_folio(){ int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); int gen = lru_gen_from_seq(lrugen->min_seq[type]); if (folio_lru_gen(folio) < 0) return false; folio_clear_active(folio); folio_clear_referenced(folio); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); }I think that works, and it is better than what this patch does. It puts both anon and file folios at the head of the oldest generation, which matches the classic LRU, and it avoids the two problems I mentioned (too aggressive for anon, and moving file folios from the head to the tail).
A plain
list_move()is not enough though. We also need to:- update the gen bits in
folio->flagswith a cmpxchg loop, asfolio_inc_gen()does, sincefolio_update_gen()can change them locklessly from the page table walk;
- fix uplrugen->nr_pagesand the LRU sizes.lru_gen_update_size()warns on an active -> inactive move ("demotion requires isolation"), so this would be done as a deletion fromold_genfollowed by an addition tonew_gen.
min_seqonly advances under the LRU lock, so it is stable here. Something like:static bool lru_gen_deactivate_folio(struct lruvec *lruvec,
struct folio *folio){ int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen = lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f); int old_gen; do { old_gen = ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; if (old_gen < 0) return false; new_flags = old_flags & ~LRU_GEN_MASK; new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF; } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); if (old_gen != new_gen) { lru_gen_update_size(lruvec, folio, old_gen, -1); lru_gen_update_size(lruvec, folio, -1, new_gen); } list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); return true; }Quoted 8 lines
Right now,
lru_deactivate()seems unnecessarily complicated, incurring additional CPU overhead and LRU lock contention:1. Remove the folio from the LRU list.
2. Add the folio to the LRU cache.
3. Drain the LRU cache.
4. Add the folio back to the LRU list.Quoted 6 lines
Why do we need all these intermediate steps instead of moving the folio directly in a single operation, given that we're already manipulating the LRU list?
Am I missing something?
I don't think the batch itself is the problem. The folio stays on the LRU while it sits in the per-cpu batch; only when the batch is drained do we take the LRU lock once for up to a batch of folios and do del + add under it. So the batch reduces lock contention rather than adding to it. What the in-place move saves is the del/add pair inside the lock.
Quoted 4 lines
So I plan to keep the batching and, in
lru_deactivate(), calllru_gen_deactivate_folio()instead oflruvec_del_folio()+lruvec_add_folio()when MGLRU is enabled. I'll test it and send v2 if that sounds good to you.
Right, keeping the batch sounds like a good idea.
And I think we can make this cleaner if we build it on top of:
https://lore.kernel.org/linux-mm/CAMgjq7A1dWJ6DoF6+CxfjysxjzAChWrdk=z_JvtLMyFWdoYTyw@mail.gmail.com/
By having a "lru_gen_add_folio(lruvec, folio, min_seq, max_seq, tail)"
lru_gen_add_folio can cap the target window by the given [min_seq, max_seq], -1 means no limitation. Avoid a lot of redundant code, and we still have just one unifed place to add a folio into the gens (except lazy promotion), I think.