mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
RFC 1/1 mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages.
An MGLRU page cache page is eligible for promotion when:
1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.
When a page is accessed through a file descriptor, folio_inc_refs() is invoked. In this function, we check whether the page is referenced, is part of the working set, and Belongs to the active list. If all these conditions are met, the page is added to the promotion list. The per-process task task_numa_promotion_work() take the pages from the promotion list and promotes them to a higher memory tier.
Test process:
We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier
Test Results MGLRU
---------------------------------------------------------------
Pages on higher | Pages Lower tier | Pages on Lower Tier |
Tier | promotion off | Promotion On |
---------------------------------------------------------------
1.5s | 3.2s |During Promotion - 6.6s |
| |After Promotion - 1.6s |
| | |---------------------------------------------------------------
Test Results LRU
---------------------------------------------------------------
Pages on higher | Pages Lower tier | Pages on Lower Tier |
Tier | promotion off | Promotion On |
---------------------------------------------------------------
1.5s | 3.2s |During Promotion - 4.2s |
| | - 3.4s |
| | - 5.6s |
| |After Promotion - 1.6s |
| | |---------------------------------------------------------------
In LRU, pages are initially added to the inactive list. When a page is referenced, it is moved to the active list, and promotion to a higher memory tier occurs from the active list. This process often requires multiple reads to trigger promotion. In contrast, MGLRU adds new pages to the youngest generation immediately. As a result, we observe that MGLRU promotes pages on the first read itself, unlike LRU which takes multiple reads to trigger promotion.
This difference also impacts read latency:
For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.
For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.
MGLRU and LRU are showing similar performance benefit.
[1] https://lore.kernel.org/all/[email protected]/
--- mm/swap.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
mm/swap.c +21 -2
@@ -386,6 +386,9 @@ static void __lru_cache_activate_folio(struct folio *folio)static void lru_gen_inc_refs (struct folio *folio) { +struct mem_cgroup *memcg; +struct lruvec *lruvec; +int gen;unsigned long new_flags, old_flags =READ_ONCE (folio->flags);if (folio_test_unevictable (folio))
@@ -399,13 +402,29 @@ static void lru_gen_inc_refs(struct folio *folio)do {if ((old_flags &LRU_REFS_MASK ) ==LRU_REFS_MASK ) { -if (!folio_test_workingset (folio)) +if (!folio_test_workingset (folio)) {folio_set_workingset (folio); -return ; +return ; + } + goto promo_candid; } new_flags = old_flags +BIT (LRU_REFS_PGOFF ); }while (!try_cmpxchg (&folio->flags, &old_flags, new_flags)); + +promo_candid: +if (!folio_test_isolated (folio) && + (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) && + numa_pagecache_promotion_enabled) { + memcg =folio_memcg (folio); +if (memcg) { + lruvec =mem_cgroup_lruvec (memcg,folio_pgdat (folio)); + gen =folio_lru_gen (folio); + +if ((gen <MAX_NR_GENS ) &&lru_gen_is_active (lruvec, gen)) +promotion_candidate (folio); + } + } }static bool lru_gen_clear_refs (struct folio *folio)
Signature · 1 line
2.43.5
Quoted 9 lines · Donet Tom
An MGLRU page cache page is eligible for promotion when:
1. Memory Tiering and
pagecache_promotion_enabledare enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.
... why do you only promote folios if they belong to a memcg?
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 426
Quoted 12 lines
mm/swap.c +12
-0+promo_candid: +if (!folio_test_isolated (folio) && + (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) && + numa_pagecache_promotion_enabled) { + memcg =folio_memcg (folio); +if (memcg) { + lruvec =mem_cgroup_lruvec (memcg,folio_pgdat (folio)); + gen =folio_lru_gen (folio); + +if ((gen <MAX_NR_GENS ) &&lru_gen_is_active (lruvec, gen)) +promotion_candidate (folio); + }
Should there be an 'else' clause here? Or should it be:
lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio)); gen = folio_lru_gen(folio); if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) promotion_candidate(folio);
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Donet Tom
An MGLRU page cache page is eligible for promotion when:
1. Memory Tiering and
pagecache_promotion_enabledare enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.... why do you only promote folios if they belong to a memcg?
I missed adding non memcg folios for promotion . I will add them too.
Quoted 15 lines
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 426
Quoted 13 lines
mm/swap.c +12
-0+promo_candid: +if (!folio_test_isolated (folio) && + (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) && + numa_pagecache_promotion_enabled) { + memcg =folio_memcg (folio); +if (memcg) { + lruvec =mem_cgroup_lruvec (memcg,folio_pgdat (folio)); + gen =folio_lru_gen (folio); + +if ((gen <MAX_NR_GENS ) &&lru_gen_is_active (lruvec, gen)) +promotion_candidate (folio); + }Should there be an 'else' clause here? Or should it be:
Quoted 5 lines
lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio)); gen = folio_lru_gen(folio); if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) promotion_candidate(folio);
Yes, there should be an else block to handle the promotion of non memcg folios.
Does this approach look correct to you?
} while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); +
+promo_candid:
+ if (!folio_test_isolated(folio) &&
+ (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + numa_pagecache_promotion_enabled) {
+ pgdat = folio_pgdat(folio);
+ gen = folio_lru_gen(folio);
+
+ if (mem_cgroup_disabled())
+ lruvec = &pgdat->__lruvec;
+ else {
+ memcg = folio_memcg(folio);
+ lruvec = mem_cgroup_lruvec(memcg, pgdat); + }
+
+ if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) + promotion_candidate(folio);
+ }
}
Quoted 11 lines · Donet Tom
Quoted 7 lines
Should there be an 'else' clause here? Or should it be:
lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio)); gen = folio_lru_gen(folio); if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) promotion_candidate(folio);Yes, there should be an else block to handle the promotion of non memcg folios.
Does this approach look correct to you?
No. Look at the implementation of folio_memcg(), and mem_cgroup_lruvec() for that matter.
Quoted 23 lines
} while (!try_cmpxchg(&folio->flags,
&old_flags,new_flags));mm/swap.c +4
-0+ +promo_candid: + if (!folio_test_isolated(folio) && + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)&&
+numa_pagecache_promotion_enabled) { + pgdat =folio_pgdat(folio); + gen =folio_lru_gen(folio);(quoted diff) +11
-0+ + if (mem_cgroup_disabled()) + lruvec = &pgdat->__lruvec; + else { + memcg = folio_memcg(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + } + + if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) + promotion_candidate(folio); + }}
On Wed, Jan 15, 2025 at 06:06:25AM -0600, Donet Tom wrote:
... snip ...
Thank you for taking the time to do this, I don't have enough background with MGLRU to have done this quickly. I'll pull this onto my branch and carry it if you don't mind so we can keep things tracked.
I'll send you an updated RFC before i send out v4 and add appropriate tags.
Quoted 8 lines
This difference also impacts read latency:
For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.
For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.
Do you have a thought on a good test we can use to compare these strategies?
We decided against promotion on first-access because there are many easy-to-imagine scenarios where that will clearly harm performance.
We're planning to do some workload testing soon so we can get actual benefit numbers.
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 418
Quoted 4 lines
mm/swap.c +4
-0+promo_candid: +if (!folio_test_isolated (folio) && + (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) && + numa_pagecache_promotion_enabled) {
I am considering putting this in some inline wrapper with some likely() tags to clean this up a bit and optimize the fall-through cases since i've seen some measurable differences when left as-is.
Thoughts on this are welcome
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 420
Quoted 2 lines
mm/swap.c +2
-0+ memcg =folio_memcg (folio); +if (memcg) {
Also curious, why only promote when the folio is a member of a memcg?
~Gregory
Quoted 8 lines · Gregory Price
On Wed, Jan 15, 2025 at 06:06:25AM -0600, Donet Tom wrote:
... snip ...Thank you for taking the time to do this, I don't have enough background with MGLRU to have done this quickly. I'll pull this onto my branch and carry it if you don't mind so we can keep things tracked.
I'll send you an updated RFC before i send out v4 and add appropriate tags.
Sure. Thank you.
Quoted 11 lines
Quoted 9 lines
This difference also impacts read latency:
For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.
For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.
Do you have a thought on a good test we can use to compare these strategies?
I am also thinking on the same. I will come back with some tests which can compare these strategies.
Quoted 3 lines
We decided against promotion on first-access because there are many easy-to-imagine scenarios where that will clearly harm performance.
Yes, I agree. After the first access, it gets promoted, but if those pages are not accessed afterward, it will become an overhead.
I will check if we have any mechanism to restrict it.
Quoted 3 lines
We're planning to do some workload testing soon so we can get actual benefit numbers.
Could you please let me know what workload you are planning? I can try it on my system as well.
Quoted 8 lines
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 418
Quoted 5 lines
mm/swap.c +4
-0+promo_candid: +if (!folio_test_isolated (folio) && + (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) && + numa_pagecache_promotion_enabled) {I am considering putting this in some inline wrapper with some
likely()tags to clean this up a bit and optimize the fall-through cases since i've seen some measurable differences when left as-is.
Sounds good to me.
Quoted 6 lines
Thoughts on this are welcome
mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 420
Quoted 2 lines
mm/swap.c +2
-0+ memcg =folio_memcg (folio); +if (memcg) {Also curious, why only promote when the folio is a member of a memcg?
I missed non memcg pages. I will send a V2 with that.
~Donet
Quoted 2 lines
~Gregory
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Donet,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250115-200910 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250115120625.3785-1-donettom%40linux.ibm.com patch subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages config: arm64-randconfig-003-20250116 (https://download.01.org/0day-ci/archive/20250117/[email protected]/config) compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250117/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/swap.c:413:4: error: use of undeclared identifier 'sysctl_numa_balancing_mode'
(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
^
mm/swap.c:413:33: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
^
mm/swap.c:414:3: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
numa_pagecache_promotion_enabled) {
^
>> mm/swap.c:421:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
promotion_candidate(folio);
^4 errors generated.
vim +/promotion_candidate +421 mm/swap.c
382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 struct mem_cgroup *memcg;
386 struct lruvec *lruvec;
387 int gen;
388 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
389
390 if (folio_test_unevictable(folio))
391 return;
392
393 /* see the comment on LRU_REFS_FLAGS */
394 if (!folio_test_referenced(folio)) {
395 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
396 return;
397 }
398
399 do {
400 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
401 if (!folio_test_workingset(folio)) {
402 folio_set_workingset(folio);
403 return;
404 }
405 goto promo_candid;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410
411 promo_candid:
412 if (!folio_test_isolated(folio) &&
> 413 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
414 numa_pagecache_promotion_enabled) {
415 memcg = folio_memcg(folio);
416 if (memcg) {
417 lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
418 gen = folio_lru_gen(folio);
419
420 if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
> 421 promotion_candidate(folio); 422 }
423 }
424 }
425
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Donet,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250115-200910 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250115120625.3785-1-donettom%40linux.ibm.com patch subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages config: i386-buildonly-randconfig-003-20250116 (https://download.01.org/0day-ci/archive/20250117/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250117/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from mm/swap.c:26:
include/linux/mm_inline.h:47:41: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
47 | __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
| ~~~~~~~~~~~ ^ ~~~
include/linux/mm_inline.h:49:22: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
49 | NR_ZONE_LRU_BASE + lru, nr_pages);
| ~~~~~~~~~~~~~~~~ ^ ~~~
>> mm/swap.c:413:4: error: use of undeclared identifier 'sysctl_numa_balancing_mode'
413 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^
>> mm/swap.c:413:33: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
413 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^
>> mm/swap.c:414:3: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
414 | numa_pagecache_promotion_enabled) {
| ^
>> mm/swap.c:421:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
421 | promotion_candidate(folio);
| ^2 warnings and 4 errors generated.
vim +/sysctl_numa_balancing_mode +413 mm/swap.c
382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 struct mem_cgroup *memcg;
386 struct lruvec *lruvec;
387 int gen;
388 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
389
390 if (folio_test_unevictable(folio))
391 return;
392
393 /* see the comment on LRU_REFS_FLAGS */
394 if (!folio_test_referenced(folio)) {
395 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
396 return;
397 }
398
399 do {
400 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
401 if (!folio_test_workingset(folio)) {
402 folio_set_workingset(folio);
403 return;
404 }
405 goto promo_candid;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410
411 promo_candid:
412 if (!folio_test_isolated(folio) &&
> 413 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> 414 numa_pagecache_promotion_enabled) {
415 memcg = folio_memcg(folio);
416 if (memcg) {
417 lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
418 gen = folio_lru_gen(folio);
419
420 if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
> 421 promotion_candidate(folio); 422 }
423 }
424 }
425
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki