Skip to contentSkip to conversation

mm/swap.c: Enable promotion of unmapped MGLRU page cache pages

RFC Superseded Donet Tom 4 participants 8 messages Jan 15, 2025 – Jan 17, 2025 Memory management

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Donet Tom

author
#0 lore raw

RFC 1/1 mm/swap.c: Enable promotion of unmapped MGLRU page cache pages

This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages.

An MGLRU page cache page is eligible for promotion when:

1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.

When a page is accessed through a file descriptor, folio_inc_refs() is invoked. In this function, we check whether the page is referenced, is part of the working set, and Belongs to the active list. If all these conditions are met, the page is added to the promotion list. The per-process task task_numa_promotion_work() take the pages from the promotion list and promotes them to a higher memory tier.

Test process:
We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier

Test Results MGLRU
---------------------------------------------------------------

Pages on higher   | Pages Lower tier |  Pages on Lower Tier   |
   Tier           |  promotion off   |   Promotion On         |
---------------------------------------------------------------
 1.5s             |    3.2s          |During Promotion - 6.6s |
                  |                  |After Promotion  - 1.6s |
                  |                  |                        |

---------------------------------------------------------------

Test Results LRU
---------------------------------------------------------------

Pages on higher   | Pages Lower tier |  Pages on Lower Tier   |
   Tier           |  promotion off   |   Promotion On         |
---------------------------------------------------------------
 1.5s             |    3.2s          |During Promotion - 4.2s |
                  |                  |                 - 3.4s |
                  |                  |                 - 5.6s |
                  |                  |After Promotion  - 1.6s |
                  |                  |                        |

---------------------------------------------------------------

In LRU, pages are initially added to the inactive list. When a page is referenced, it is moved to the active list, and promotion to a higher memory tier occurs from the active list. This process often requires multiple reads to trigger promotion. In contrast, MGLRU adds new pages to the youngest generation immediately. As a result, we observe that MGLRU promotes pages on the first read itself, unlike LRU which takes multiple reads to trigger promotion.

This difference also impacts read latency:

For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.

For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.

MGLRU and LRU are showing similar performance benefit.

[1] https://lore.kernel.org/all/[email protected]/

---
 mm/swap.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
mm/swap.c +21 -2
index b2341bc18452..121de1d7e938 100644
@@ -386,6 +386,9 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 static void lru_gen_inc_refs(struct folio *folio)
 {
+	struct mem_cgroup *memcg;
+	struct lruvec *lruvec;
+	int gen;
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
 
 	if (folio_test_unevictable(folio))
@@ -399,13 +402,29 @@ static void lru_gen_inc_refs(struct folio *folio)
 
 	do {
 		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
-			if (!folio_test_workingset(folio))
+			if (!folio_test_workingset(folio)) {
 				folio_set_workingset(folio);
-			return;
+				return;
+			}
+			goto promo_candid;
 		}
 
 		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
 	} while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
+
+promo_candid:
+	if (!folio_test_isolated(folio) &&
+		(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		numa_pagecache_promotion_enabled) {
+		memcg = folio_memcg(folio);
+		if (memcg) {
+			lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
+			gen = folio_lru_gen(folio);
+
+			if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
+				promotion_candidate(folio);
+		}
+	}
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
Signature · 1 line
2.43.5
#1 lore raw
Quoted 9 lines · Donet Tom

An MGLRU page cache page is eligible for promotion when:

1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.

... why do you only promote folios if they belong to a memcg?

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 426

Quoted 12 lines
mm/swap.c +12 -0
+promo_candid:
+	if (!folio_test_isolated(folio) &&
+		(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		numa_pagecache_promotion_enabled) {
+		memcg = folio_memcg(folio);
+		if (memcg) {
+			lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
+			gen = folio_lru_gen(folio);
+
+			if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
+				promotion_candidate(folio);
+		}

Should there be an 'else' clause here? Or should it be:

		lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
		gen = folio_lru_gen(folio);
		if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
			promotion_candidate(folio);
author
#2 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 9 lines · Donet Tom

An MGLRU page cache page is eligible for promotion when:

1. Memory Tiering and pagecache_promotion_enabled are enabled
2. It resides in a lower memory tier.
3. It is referenced.
4. It is part of the working set.
5. It belongs to the active list.
For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list.

... why do you only promote folios if they belong to a memcg?

I missed adding non memcg folios for promotion . I will add them too.

Quoted 15 lines

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 426

Quoted 13 lines
mm/swap.c +12 -0
+promo_candid:
+	if (!folio_test_isolated(folio) &&
+		(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		numa_pagecache_promotion_enabled) {
+		memcg = folio_memcg(folio);
+		if (memcg) {
+			lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
+			gen = folio_lru_gen(folio);
+
+			if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
+				promotion_candidate(folio);
+		}

Should there be an 'else' clause here? Or should it be:

Quoted 5 lines
		lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
		gen = folio_lru_gen(folio);
		if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
			promotion_candidate(folio);

Yes, there should be an else block to handle the promotion of non memcg folios.

Does this approach look correct to you?

        } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); +
+promo_candid:
+       if (!folio_test_isolated(folio) &&
+               (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && +               numa_pagecache_promotion_enabled) {
+               pgdat = folio_pgdat(folio);
+               gen = folio_lru_gen(folio);
+
+               if (mem_cgroup_disabled())
+                       lruvec = &pgdat->__lruvec;
+               else {
+                       memcg = folio_memcg(folio);
+                       lruvec = mem_cgroup_lruvec(memcg, pgdat); +               }
+
+               if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) +                       promotion_candidate(folio);
+       }
 }

#3 lore raw
Quoted 11 lines · Donet Tom
Quoted 7 lines

Should there be an 'else' clause here? Or should it be:

		lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
		gen = folio_lru_gen(folio);
		if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
			promotion_candidate(folio);

Yes, there should be an else block to handle the promotion of non memcg folios.

Does this approach look correct to you?

No. Look at the implementation of folio_memcg(), and mem_cgroup_lruvec() for that matter.

Quoted 23 lines

        } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));

mm/swap.c +4 -0
+
+promo_candid:
+       if (!folio_test_isolated(folio) &&
+               (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)

&&
+               numa_pagecache_promotion_enabled) { +               pgdat = folio_pgdat(folio); +               gen = folio_lru_gen(folio);

(quoted diff) +11 -0
+
+               if (mem_cgroup_disabled())
+                       lruvec = &pgdat->__lruvec;
+               else {
+                       memcg = folio_memcg(folio);
+                       lruvec = mem_cgroup_lruvec(memcg, pgdat);
+               }
+
+               if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
+                       promotion_candidate(folio);
+       }

 }

#4 lore raw
In reply to Donet Tom

On Wed, Jan 15, 2025 at 06:06:25AM -0600, Donet Tom wrote:
... snip ...

Thank you for taking the time to do this, I don't have enough background with MGLRU to have done this quickly. I'll pull this onto my branch and carry it if you don't mind so we can keep things tracked.

I'll send you an updated RFC before i send out v4 and add appropriate tags.

Quoted 8 lines

This difference also impacts read latency:

For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.

For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.

Do you have a thought on a good test we can use to compare these strategies?

We decided against promotion on first-access because there are many easy-to-imagine scenarios where that will clearly harm performance.

We're planning to do some workload testing soon so we can get actual benefit numbers.

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 418

Quoted 4 lines
mm/swap.c +4 -0
+promo_candid:
+	if (!folio_test_isolated(folio) &&
+		(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		numa_pagecache_promotion_enabled) {

I am considering putting this in some inline wrapper with some likely() tags to clean this up a bit and optimize the fall-through cases since i've seen some measurable differences when left as-is.

Thoughts on this are welcome

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 420

Quoted 2 lines
mm/swap.c +2 -0
+		memcg = folio_memcg(folio);
+		if (memcg) {

Also curious, why only promote when the folio is a member of a memcg?

~Gregory

author
#5 lore raw
Quoted 8 lines · Gregory Price

On Wed, Jan 15, 2025 at 06:06:25AM -0600, Donet Tom wrote:
... snip ...

Thank you for taking the time to do this, I don't have enough background with MGLRU to have done this quickly. I'll pull this onto my branch and carry it if you don't mind so we can keep things tracked.

I'll send you an updated RFC before i send out v4 and add appropriate tags.

Sure. Thank you.

Quoted 11 lines
Quoted 9 lines

This difference also impacts read latency:

For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion.

For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately.

Do you have a thought on a good test we can use to compare these strategies?

I am also thinking on the same. I will come back with some tests which can compare these strategies.

Quoted 3 lines

We decided against promotion on first-access because there are many easy-to-imagine scenarios where that will clearly harm performance.

Yes, I agree. After the first access, it gets promoted, but if those pages are not accessed afterward, it will become an overhead.
I will check if we have any mechanism to restrict it.

Quoted 3 lines

We're planning to do some workload testing soon so we can get actual benefit numbers.

Could you please let me know what workload you are planning? I can try it on my system as well.

Quoted 8 lines

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 418

Quoted 5 lines
mm/swap.c +4 -0
+promo_candid:
+	if (!folio_test_isolated(folio) &&
+		(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		numa_pagecache_promotion_enabled) {

I am considering putting this in some inline wrapper with some likely() tags to clean this up a bit and optimize the fall-through cases since i've seen some measurable differences when left as-is.

Sounds good to me.

Quoted 6 lines

Thoughts on this are welcome

mm/swap.c static void lru_gen_inc_refs(struct folio *folio) · line 420

Quoted 2 lines
mm/swap.c +2 -0
+		memcg = folio_memcg(folio);
+		if (memcg) {

Also curious, why only promote when the folio is a member of a memcg?

I missed non memcg pages. I will send a V2 with that.

~Donet

Quoted 2 lines

~Gregory

#6 lore raw
In reply to Donet Tom
kernel test robot noticed the following build errors:

Hi Donet,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250115-200910
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250115120625.3785-1-donettom%40linux.ibm.com
patch subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
config: arm64-randconfig-003-20250116 (https://download.01.org/0day-ci/archive/20250117/[email protected]/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250117/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/swap.c:413:4: error: use of undeclared identifier 'sysctl_numa_balancing_mode'

                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
                    ^
   mm/swap.c:413:33: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
                   (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
                                                 ^
   mm/swap.c:414:3: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
                   numa_pagecache_promotion_enabled) {
                   ^
>> mm/swap.c:421:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
                                   promotion_candidate(folio);
                                   ^

4 errors generated.

vim +/promotion_candidate +421 mm/swap.c

382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 struct mem_cgroup *memcg;
386 struct lruvec *lruvec;
387 int gen;
388 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
389
390 if (folio_test_unevictable(folio))
391 return;
392
393 /* see the comment on LRU_REFS_FLAGS */
394 if (!folio_test_referenced(folio)) {
395 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
396 return;
397 }
398
399 do {
400 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
401 if (!folio_test_workingset(folio)) {
402 folio_set_workingset(folio);
403 return;
404 }
405 goto promo_candid;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410
411 promo_candid:
412 if (!folio_test_isolated(folio) &&

 > 413			(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
   414			numa_pagecache_promotion_enabled) {
   415			memcg = folio_memcg(folio);
   416			if (memcg) {
   417				lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
   418				gen = folio_lru_gen(folio);
   419	
   420				if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
 > 421					promotion_candidate(folio);

422 }
423 }
424 }
425

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#7 lore raw
In reply to Donet Tom
kernel test robot noticed the following build errors:

Hi Donet,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Donet-Tom/mm-swap-c-Enable-promotion-of-unmapped-MGLRU-page-cache-pages/20250115-200910
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250115120625.3785-1-donettom%40linux.ibm.com
patch subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages
config: i386-buildonly-randconfig-003-20250116 (https://download.01.org/0day-ci/archive/20250117/[email protected]/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250117/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from mm/swap.c:26:
include/linux/mm_inline.h:47:41: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]

      47 |         __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
         |                                    ~~~~~~~~~~~ ^ ~~~
   include/linux/mm_inline.h:49:22: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
      49 |                                 NR_ZONE_LRU_BASE + lru, nr_pages);
         |                                 ~~~~~~~~~~~~~~~~ ^ ~~~
>> mm/swap.c:413:4: error: use of undeclared identifier 'sysctl_numa_balancing_mode'
     413 |                 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                  ^
>> mm/swap.c:413:33: error: use of undeclared identifier 'NUMA_BALANCING_MEMORY_TIERING'
     413 |                 (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
         |                                               ^
>> mm/swap.c:414:3: error: use of undeclared identifier 'numa_pagecache_promotion_enabled'
     414 |                 numa_pagecache_promotion_enabled) {
         |                 ^
>> mm/swap.c:421:5: error: call to undeclared function 'promotion_candidate'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     421 |                                 promotion_candidate(folio);
         |                                 ^

2 warnings and 4 errors generated.

vim +/sysctl_numa_balancing_mode +413 mm/swap.c

382
383 static void lru_gen_inc_refs(struct folio *folio)
384 {
385 struct mem_cgroup *memcg;
386 struct lruvec *lruvec;
387 int gen;
388 unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
389
390 if (folio_test_unevictable(folio))
391 return;
392
393 /* see the comment on LRU_REFS_FLAGS */
394 if (!folio_test_referenced(folio)) {
395 set_mask_bits(&folio->flags, LRU_REFS_MASK, BIT(PG_referenced));
396 return;
397 }
398
399 do {
400 if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
401 if (!folio_test_workingset(folio)) {
402 folio_set_workingset(folio);
403 return;
404 }
405 goto promo_candid;
406 }
407
408 new_flags = old_flags + BIT(LRU_REFS_PGOFF);
409 } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags));
410
411 promo_candid:
412 if (!folio_test_isolated(folio) &&

 > 413			(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 > 414			numa_pagecache_promotion_enabled) {
   415			memcg = folio_memcg(folio);
   416			if (memcg) {
   417				lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio));
   418				gen = folio_lru_gen(folio);
   419	
   420				if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen))
 > 421					promotion_candidate(folio);

422 }
423 }
424 }
425

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki