mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
RFC PATCH v2 1/1 mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages.
An MGLRU page cache page is eligible for promotion when:
1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. folio reference count is maximun (LRU_REFS_MASK).
When a page is accessed through a file descriptor, folio_inc_refs() is invoked. The first access will set the folio’s referenced flag, and subsequent accesses will increment the reference count in the folio flag (reference counter size in folio flags is 2 bits). Once the referenced flag is set, and the folio’s reference count reaches the maximum value (LRU_REFS_MASK), the working set flag will be set as well.
If a folio has both the referenced and working set flags set, and its reference count equals LRU_REFS_MASK, it becomes a good candidate for promotion. These pages will be added to the promotion list. The per-process task task_numa_promotion_work() takes the pages from the promotion list and promotes them to a higher memory tier.
In the MGLRU, for folios accessed through a file descriptor, if the folio’s referenced and working set flags are set, and the folio's reference count is equal to LRU_REFS_MASK, the folio is lazily promoted to the second oldest generation in the eviction path. When folio_inc_gen() does this, it clears the LRU_REFS_FLAGS so that lru_gen_inc_refs() can start over.
Test process:
We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier
Test Results MGLRU
----------------------------------------------------------------
Pages on higher | Pages Lower tier | Pages on Lower Tier |
Tier | promotion off | Promotion On |
----------------------------------------------------------------
0.48s | 1.6s |During Promotion - 3.3s |
| |After Promotion - 0.48s |
| | |----------------------------------------------------------------
Test Results LRU
----------------------------------------------------------------
Pages on higher | Pages Lower tier | Pages on Lower Tier |
Tier | promotion off | Promotion On |
----------------------------------------------------------------
0.48s | 1.6s |During Promotion - 3.3s |
| |After Promotion - 0.48s |
| | |----------------------------------------------------------------
MGLRU and LRU are showing similar performance benefit.
[1] https://lore.kernel.org/all/[email protected]/
---v1->v2
In V1, the folios that were part of the memcg and the active MGLRU list were being promoted. However, in MGLRU, file pages accessed through file descriptors are moved to the second oldest generation. This second oldest generation may not necessarily be part of the active list. Furthermore, this movement to the second oldest generation only happens in the eviction path, so if the system is not under memory pressure, this movement will not occur. As a result, hot pages can be present in any generation. If the reference count is at its maximum and the referenced and working set flags are set, the page becomes a candidate for promotion.
v1 - https://lore.kernel.org/all/[email protected]/
--- mm/swap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swap.c +6 -1
@@ -399,8 +399,13 @@ static void lru_gen_inc_refs(struct folio *folio) do { if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) + if (!folio_test_workingset(folio)) { folio_set_workingset(folio); + } else if (!folio_test_isolated(folio) && + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + numa_pagecache_promotion_enabled) { + promotion_candidate(folio); + } return; }
Signature · 1 line
2.43.5
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Donet,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com patch subject: [RFC PATCH v2] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages config: hexagon-randconfig-002-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config) compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project 7eccafc3c84606587a175c0a8c1ebea6e4fb21cd) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/swap.c:401:8: error: use of undeclared identifier 'sysctl_numa_balancing_mode'
401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^
>> mm/swap.c:401:37: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^
>> mm/swap.c:402:8: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
402 | numa_pagecache_promotion_enabled) {
| ^
>> mm/swap.c:403:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
403 | promotion_candidate(folio);
| ^4 errors generated.
vim +/sysctl_numa_balancing_mode +401 mm/swap.c
382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
386
387 if (folio_test_unevictable(folio))
388 return;
389
390 /* see the comment on LRU_REFS_FLAGS */
391 if (!folio_test_referenced(folio)) {
392 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
393 return;
394 }
395
396 do {
397 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
398 if (!folio_test_workingset(folio)) {
399 folio_set_workingset(folio);
400 } else if (!folio_test_isolated(folio) &&
> 401 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> 402 numa_pagecache_promotion_enabled) {
> 403 promotion_candidate(folio); 404 }
405 return;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410 }
411
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 65 lines · kernel test robot
Hi Donet,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com patch subject: [RFC PATCH v2] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages config: hexagon-randconfig-002-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config) compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project 7eccafc3c84606587a175c0a8c1ebea6e4fb21cd) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
mm/swap.c:401:8: error: use of undeclared identifier 'sysctl_numa_balancing_mode'401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && | ^Quoted 1 line
Quoted 1 line
mm/swap.c:401:37: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && | ^Quoted 1 line
Quoted 1 line
mm/swap.c:402:8: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'402 | numa_pagecache_promotion_enabled) { | ^Quoted 1 line
Quoted 1 line
mm/swap.c:403:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]403 | promotion_candidate(folio); | ^4 errors generated.
vim +/sysctl_numa_balancing_mode +401
mm/swap.c382 383 static void lru_gen_inc_refs(struct folio *folio) 384 { 385 unsigned long new_flags, old_flags = READ_ONCE(folio->flags); 386 387 if (folio_test_unevictable(folio)) 388 return; 389 390 /* see the comment on LRU_REFS_FLAGS */ 391 if (!folio_test_referenced(folio)) { 392 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced)); 393 return; 394 } 395 396 do { 397 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { 398 if (!folio_test_workingset(folio)) { 399 folio_set_workingset(folio); 400 } else if (!folio_test_isolated(folio) &&Quoted 3 lines
401 (
sysctl_numa_balancing_mode&NUMA_BALANCING_MEMORY_TIERING) &&
402numa_pagecache_promotion_enabled) {
403promotion_candidate(folio);
Quoted 8 lines
404 }
405 return;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410 }
411 Thanks for the report. I will fix this and send a V3.
- Donet
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Donet,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com patch subject: [RFC PATCH v2] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages config: i386-buildonly-randconfig-003-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/swap.c: In function 'lru_gen_inc_refs':
>> mm/swap.c:401:36: error: 'sysctl_numa_balancing_mode' undeclared (first use in this function)
401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
mm/swap.c:401:36: note: each undeclared identifier is reported only once for each function it appears in
>> mm/swap.c:401:65: error: 'NUMA_BALANCING_MEMORY_TIERING' undeclared (first use in this function)
401 | (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/swap.c:402:36: error: 'numa_pagecache_promotion_enabled' undeclared (first use in this function)
402 | numa_pagecache_promotion_enabled) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/swap.c:403:33: error: implicit declaration of function 'promotion_candidate' [-Werror=implicit-function-declaration]
403 | promotion_candidate(folio);
| ^~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/sysctl_numa_balancing_mode +401 mm/swap.c
382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
386
387 if (folio_test_unevictable(folio))
388 return;
389
390 /* see the comment on LRU_REFS_FLAGS */
391 if (!folio_test_referenced(folio)) {
392 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
393 return;
394 }
395
396 do {
397 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
398 if (!folio_test_workingset(folio)) {
399 folio_set_workingset(folio);
400 } else if (!folio_test_isolated(folio) &&
> 401 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> 402 numa_pagecache_promotion_enabled) {
> 403 promotion_candidate(folio); 404 }
405 return;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410 }
411
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki