Skip to contentSkip to conversation

mm/swap.c: Enable promotion of unmapped MGLRU page cache pages

RFC PATCH In review Donet Tom 2 participants 4 messages Apr 3, 2025 – Apr 4, 2025 Memory management

lore
Share & tools
lore

Donet Tom

author
#0 lore raw

RFC PATCH v2 1/1 mm/swap.c: Enable promotion of unmapped MGLRU page cache pages

This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages.

An MGLRU page cache page is eligible for promotion when:

1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. folio reference count is maximun (LRU_REFS_MASK).

When a page is accessed through a file descriptor, folio_inc_refs() is invoked. The first access will set the folio’s referenced flag, and subsequent accesses will increment the reference count in the folio flag (reference counter size in folio flags is 2 bits). Once the referenced flag is set, and the folio’s reference count reaches the maximum value (LRU_REFS_MASK), the working set flag will be set as well.

If a folio has both the referenced and working set flags set, and its reference count equals LRU_REFS_MASK, it becomes a good candidate for promotion. These pages will be added to the promotion list. The per-process task task_numa_promotion_work() takes the pages from the promotion list and promotes them to a higher memory tier.

In the MGLRU, for folios accessed through a file descriptor, if the folio’s referenced and working set flags are set, and the folio's reference count is equal to LRU_REFS_MASK, the folio is lazily promoted to the second oldest generation in the eviction path. When folio_inc_gen() does this, it clears the LRU_REFS_FLAGS so that lru_gen_inc_refs() can start over.

Test process:
We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier

Test Results MGLRU
----------------------------------------------------------------

Pages on higher   | Pages Lower tier |  Pages on Lower Tier    |
   Tier           |  promotion off   |   Promotion On          |
----------------------------------------------------------------
  0.48s           |    1.6s          |During Promotion - 3.3s  |
                  |                  |After Promotion  - 0.48s |
                  |                  |                         |

----------------------------------------------------------------

Test Results LRU
----------------------------------------------------------------

Pages on higher   | Pages Lower tier |  Pages on Lower Tier    |
   Tier           |  promotion off   |   Promotion On          |
----------------------------------------------------------------
   0.48s          |    1.6s          |During Promotion - 3.3s  |
                  |                  |After Promotion  - 0.48s |
                  |                  |                         |

----------------------------------------------------------------

MGLRU and LRU are showing similar performance benefit.

[1] https://lore.kernel.org/all/[email protected]/

---
v1->v2

In V1, the folios that were part of the memcg and the active MGLRU list were being promoted. However, in MGLRU, file pages accessed through file descriptors are moved to the second oldest generation. This second oldest generation may not necessarily be part of the active list. Furthermore, this movement to the second oldest generation only happens in the eviction path, so if the system is not under memory pressure, this movement will not occur. As a result, hot pages can be present in any generation. If the reference count is at its maximum and the referenced and working set flags are set, the page becomes a candidate for promotion.

v1 - https://lore.kernel.org/all/[email protected]/

---
 mm/swap.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/swap.c +6 -1
index b2341bc18452..f3c19d563556 100644
@@ -399,8 +399,13 @@ static void lru_gen_inc_refs(struct folio *folio)
 
 	do {
 		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
-			if (!folio_test_workingset(folio))
+			if (!folio_test_workingset(folio)) {
 				folio_set_workingset(folio);
+			} else if (!folio_test_isolated(folio) &&
+				  (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+				   numa_pagecache_promotion_enabled) {
+				promotion_candidate(folio);
+			}
 			return;
 		}
 
Signature · 1 line
2.43.5
#1 lore raw
kernel test robot noticed the following build errors:

Hi Donet,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com
patch subject: [RFC PATCH v2]  mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
config: hexagon-randconfig-002-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project 7eccafc3c84606587a175c0a8c1ebea6e4fb21cd)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> mm/swap.c:401:8: error: use of undeclared identifier 'sysctl_numa_balancing_mode'
     401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                                    ^
>> mm/swap.c:401:37: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
     401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                                                                 ^
>> mm/swap.c:402:8: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
     402 |                                    numa_pagecache_promotion_enabled) {
         |                                    ^
>> mm/swap.c:403:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     403 |                                 promotion_candidate(folio);
         |                                 ^

4 errors generated.

vim +/sysctl_numa_balancing_mode +401 mm/swap.c

382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
386
387 if (folio_test_unevictable(folio))
388 return;
389
390 /* see the comment on LRU_REFS_FLAGS */
391 if (!folio_test_referenced(folio)) {
392 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
393 return;
394 }
395
396 do {
397 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
398 if (!folio_test_workingset(folio)) {
399 folio_set_workingset(folio);
400 } else if (!folio_test_isolated(folio) &&

 > 401					  (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 > 402					   numa_pagecache_promotion_enabled) {
 > 403					promotion_candidate(folio);

404 }
405 return;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410 }
411

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#2 lore raw
Quoted 65 lines · kernel test robot

Hi Donet,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com
patch subject: [RFC PATCH v2]  mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
config: hexagon-randconfig-002-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project 7eccafc3c84606587a175c0a8c1ebea6e4fb21cd)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

mm/swap.c:401:8: error: use of undeclared identifier 'sysctl_numa_balancing_mode'

      401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
          |                                    ^
Quoted 1 line
Quoted 1 line

mm/swap.c:401:37: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'

      401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
          |                                                                 ^
Quoted 1 line
Quoted 1 line

mm/swap.c:402:8: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'

      402 |                                    numa_pagecache_promotion_enabled) {
          |                                    ^
Quoted 1 line
Quoted 1 line

mm/swap.c:403:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

      403 |                                 promotion_candidate(folio);
          |                                 ^

4 errors generated.

vim +/sysctl_numa_balancing_mode +401 mm/swap.c

    382	
    383	static void lru_gen_inc_refs(struct folio *folio)
    384	{
    385		unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
    386	
    387		if (folio_test_unevictable(folio))
    388			return;
    389	
    390		/* see the comment on LRU_REFS_FLAGS */
    391		if (!folio_test_referenced(folio)) {
    392			set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
    393			return;
    394		}
    395	
    396		do {
    397			if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
    398				if (!folio_test_workingset(folio)) {
    399					folio_set_workingset(folio);
    400				} else if (!folio_test_isolated(folio) &&
Quoted 3 lines

401 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
402 numa_pagecache_promotion_enabled) {
403 promotion_candidate(folio);

Quoted 8 lines
    404				}
    405				return;
    406			}
    407	
    408			new_flags = old_flags + BIT(LRU_REFS_PGOFF);
    409		} while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
    410	}
    411	

Thanks for the report. I will fix this and send a V3.

- Donet

#3 lore raw
In reply to Donet Tom
kernel test robot noticed the following build errors:

Hi Donet,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14 next-20250404] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250403-221222
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250403141032.22743-1-donettom%40linux.ibm.com
patch subject: [RFC PATCH v2]  mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
config: i386-buildonly-randconfig-003-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/swap.c: In function 'lru_gen_inc_refs':

>> mm/swap.c:401:36: error: 'sysctl_numa_balancing_mode' undeclared (first use in this function)
     401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~
   mm/swap.c:401:36: note: each undeclared identifier is reported only once for each function it appears in
>> mm/swap.c:401:65: error: 'NUMA_BALANCING_MEMORY_TIERING' undeclared (first use in this function)
     401 |                                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                                                                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/swap.c:402:36: error: 'numa_pagecache_promotion_enabled' undeclared (first use in this function)
     402 |                                    numa_pagecache_promotion_enabled) {
         |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/swap.c:403:33: error: implicit declaration of function 'promotion_candidate' [-Werror=implicit-function-declaration]
     403 |                                 promotion_candidate(folio);
         |                                 ^~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/sysctl_numa_balancing_mode +401 mm/swap.c

382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
386
387 if (folio_test_unevictable(folio))
388 return;
389
390 /* see the comment on LRU_REFS_FLAGS */
391 if (!folio_test_referenced(folio)) {
392 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
393 return;
394 }
395
396 do {
397 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
398 if (!folio_test_workingset(folio)) {
399 folio_set_workingset(folio);
400 } else if (!folio_test_isolated(folio) &&

 > 401					  (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 > 402					   numa_pagecache_promotion_enabled) {
 > 403					promotion_candidate(folio);

404 }
405 return;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410 }
411

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki