Skip to contentSkip to conversation

mm: Multi-Gen LRU: fix use mm/page_idle/bitmap

RFC In review Henry Huang 5 participants 19 messages Dec 6, 2023 – Jan 12, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

When Multi-Gen LRU is enabled, we can't use /sys/kernel/mm/page_idle/bitmap to track memory which is actually been used.

Multi-Gen LRU page-table walker clears pte young flag, but it doesn't clear page idle flag. When we use /sys/kernel/mm/page_idle/bitmap to check whether one page is accessed, it would tell us this page is idle, but actually this page has been accessed.

For those unmapped filecache pages, page idle flag would not been cleared in folio_mark_accessed if Multi-Gen LRU is enabled. So we couln't use /sys/kernel/mm/page_idle/bitmap to check whether a filecache page is read or written.

What's more, /sys/kernel/mm/page_idle/bitmap also clears pte young flag. If one page is accessed, it would set page young flag. Multi-Gen LRU page-table walker should check both page&pte young flags.

Changes in V2:

We move folio_clear_idle into get_pfn_folio to improve efficiency of read /sys/kernel/mm/page_idle/bitmap.

Sometimes memcg of process may be changed, but memcg of page would not be changed. Page idle flag would not be cleared when multi-gen LRU scan this process's page-table.
We think it's better to clear page idle flag as soon as possible when walking page-table. For those non-idle pages, it's not necessary to walk rmap and clear pte young(read /sys/kernel/mm/page_idle/bitmap) if multi-gen LRU page-table walker already scanned once before.

Henry Huang (1):
mm: Multi-Gen LRU: fix use mm/page_idle/bitmap

 mm/swap.c   |  3 +++
 mm/vmscan.c | 40 ++++++++++++++++++++++++++++------------
 2 files changed, 31 insertions(+), 12 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

RFC v2 1/1 mm: Multi-Gen LRU: fix use mm/page_idle/bitmap

Multi-Gen LRU page-table walker clears pte young flag, but it doesn't clear page idle flag. When we use /sys/kernel/mm/page_idle/bitmap to check whether one page is accessed, it would tell us this page is idle, but actually this page has been accessed.

For those unmapped filecache pages, page idle flag would not been cleared in folio_mark_accessed if Multi-Gen LRU is enabled. So we couln't use /sys/kernel/mm/page_idle/bitmap to check whether a filecache page is read or written.

What's more, /sys/kernel/mm/page_idle/bitmap also clears pte young flag. If one page is accessed, it would set page young flag. Multi-Gen LRU page-table walker should check both page&pte young flags.

how-to-reproduce-problem

idle_page_track
a tools to track process accessed memory during a specific time
usage
idle_page_track $pid $time
how-it-works
1. scan process vma from /proc/$pid/maps
2. vfn --> pfn from /proc/$pid/pagemap
3. write /sys/kernel/mm/page_idle/bitmap to
mark phy page idle flag and clear pte young flag
4. sleep $time
5. read /sys/kernel/mm/page_idle/bitmap to test_and_clear pte young flag and
return whether phy page is accessed

test ---- test program

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, const char *argv[])

 {
     char *buf = NULL;
     char pipe_info[4096];
     int n;
     int fd = -1;

     buf = malloc(1024*1024*1024UL);
     memset(buf, 0, 1024*1024*1024UL);
     fd = open("access.pipe", O_RDONLY);
     if (fd < 0)
         goto out;
     while (1) {
         n = read(fd, pipe_info, sizeof(pipe_info));
         if (!n) {
             sleep(1);
             continue;
         } else if (n < 0) {
             break;
         }
         memset(buf, 0, 1024*1024*1024UL);
         puts("finish access");
      }
 out:
     if (fd >=0)
         close(fd);
     if (buf)
         free(buf);

     return 0;

}

prepare:
mkfifo access.pipe
./test
ps -ef | grep test
root 4106 3148 8 06:47 pts/0 00:00:01 ./test

We use /sys/kernel/debug/lru_gen to simulate mglru page-table scan.

case 1: mglru walker break page_idle ./idle_page_track 4106 60 &
sleep 5; echo 1 > access.pipe
sleep 5; echo '+ 8 0 6 1 1' > /sys/kernel/debug/lru_gen

the output of idle_page_track is:
Est(s) Ref(MB)
64.822 1.00
only found 1MB were accessed during 64.822s, but actually 1024MB were accessed.

case 2: page_idle break mglru walker
echo 1 > access.pipe
./idle_page_track 4106 10
echo '+ 8 0 7 1 1' > /sys/kernel/debug/lru_gen
lru gen status:
memcg 8 /user.slice
node 0

          5     772458       1065        9735
          6     737435     262244          72
          7     538053       1184         632
          8      59404       6422           0

almost pages should be in max_seq-1 queue, but actually not.

---
 mm/swap.c   |  3 +++
 mm/vmscan.c | 40 ++++++++++++++++++++++++++++------------
 2 files changed, 31 insertions(+), 12 deletions(-)
mm/swap.c +3 -0
index cd8f0150ba3a..4bd14aabdc10 100644
@@ -416,6 +416,9 @@ static void folio_inc_refs(struct folio *folio)
 {
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags);
 
+	if (folio_test_idle(folio))
+		folio_clear_idle(folio);
+
 	if (folio_test_unevictable(folio))
 		return;
 
mm/vmscan.c +28 -12
index 96abaa5a973e..c1dd9e7bc315 100644
@@ -3297,7 +3297,7 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned
 #endif
 
 static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
-				   struct pglist_data *pgdat, bool can_swap)
+				   struct pglist_data *pgdat, bool can_swap, bool clear_idle)
 {
 	struct folio *folio;
 
@@ -3306,6 +3306,10 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
 		return NULL;
 
 	folio = pfn_folio(pfn);
+
+	if (clear_idle && folio_test_idle(folio))
+		folio_clear_idle(folio);
+
 	if (folio_nid(folio) != pgdat->node_id)
 		return NULL;
 
@@ -3355,6 +3359,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		unsigned long pfn;
 		struct folio *folio;
 		pte_t ptent = ptep_get(pte + i);
+		bool is_pte_young;
 
 		total++;
 		walk->mm_stats[MM_LEAF_TOTAL]++;
@@ -3363,16 +3368,20 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		if (pfn == -1)
 			continue;
 
-		if (!pte_young(ptent)) {
-			walk->mm_stats[MM_LEAF_OLD]++;
+		is_pte_young = !!pte_young(ptent);
+		folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pte_young);
+		if (!folio) {
+			if (!is_pte_young)
+				walk->mm_stats[MM_LEAF_OLD]++;
 			continue;
 		}
 
-		folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap);
-		if (!folio)
+		if (!folio_test_clear_young(folio) && !is_pte_young) {
+			walk->mm_stats[MM_LEAF_OLD]++;
 			continue;
+		}
 
-		if (!ptep_test_and_clear_young(args->vma, addr, pte + i))
+		if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i))
 			VM_WARN_ON_ONCE(true);
 
 		young++;
@@ -3435,6 +3444,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 	do {
 		unsigned long pfn;
 		struct folio *folio;
+		bool is_pmd_young;
 
 		/* don't round down the first address */
 		addr = i ? (*first & PMD_MASK) + i * PMD_SIZE : *first;
@@ -3449,11 +3459,15 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 			goto next;
 		}
 
-		folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap);
+		is_pmd_young = !!pmd_young(pmd[i]);
+		folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pmd_young);
 		if (!folio)
 			goto next;
 
-		if (!pmdp_test_and_clear_young(vma, addr, pmd + i))
+		if (is_pmd_young && !pmdp_test_and_clear_young(vma, addr, pmd + i))
+			VM_WARN_ON_ONCE(true);
+
+		if (!folio_test_clear_young(folio) && !is_pmd_young)
 			goto next;
 
 		walk->mm_stats[MM_LEAF_YOUNG]++;
@@ -4025,19 +4039,21 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) {
 		unsigned long pfn;
 		pte_t ptent = ptep_get(pte + i);
+		bool is_pte_young;
 
 		pfn = get_pte_pfn(ptent, pvmw->vma, addr);
 		if (pfn == -1)
 			continue;
 
-		if (!pte_young(ptent))
+		is_pte_young = !!pte_young(ptent);
+		folio = get_pfn_folio(pfn, memcg, pgdat, can_swap, is_pte_young);
+		if (!folio)
 			continue;
 
-		folio = get_pfn_folio(pfn, memcg, pgdat, can_swap);
-		if (!folio)
+		if (!folio_test_clear_young(folio) && !is_pte_young)
 			continue;
 
-		if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i))
+		if (is_pte_young && !ptep_test_and_clear_young(pvmw->vma, addr, pte + i))
 			VM_WARN_ON_ONCE(true);
 
 		young++;
Signature · 1 line
2.43.0
#2 lore raw
Quoted 99 lines · Henry Huang

Multi-Gen LRU page-table walker clears pte young flag, but it doesn't clear page idle flag. When we use /sys/kernel/mm/page_idle/bitmap to check whether one page is accessed, it would tell us this page is idle, but actually this page has been accessed.

For those unmapped filecache pages, page idle flag would not been cleared in folio_mark_accessed if Multi-Gen LRU is enabled. So we couln't use /sys/kernel/mm/page_idle/bitmap to check whether a filecache page is read or written.

What's more, /sys/kernel/mm/page_idle/bitmap also clears pte young flag. If one page is accessed, it would set page young flag. Multi-Gen LRU page-table walker should check both page&pte young flags.

how-to-reproduce-problem

idle_page_track
a tools to track process accessed memory during a specific time
usage
idle_page_track $pid $time
how-it-works
1. scan process vma from /proc/$pid/maps
2. vfn --> pfn from /proc/$pid/pagemap
3. write /sys/kernel/mm/page_idle/bitmap to
mark phy page idle flag and clear pte young flag
4. sleep $time
5. read /sys/kernel/mm/page_idle/bitmap to test_and_clear pte young flag and
return whether phy page is accessed

test ---- test program

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, const char *argv[])

 {
     char *buf = NULL;
     char pipe_info[4096];
     int n;
     int fd = -1;

     buf = malloc(1024*1024*1024UL);
     memset(buf, 0, 1024*1024*1024UL);
     fd = open("access.pipe", O_RDONLY);
     if (fd < 0)
         goto out;
     while (1) {
         n = read(fd, pipe_info, sizeof(pipe_info));
         if (!n) {
             sleep(1);
             continue;
         } else if (n < 0) {
             break;
         }
         memset(buf, 0, 1024*1024*1024UL);
         puts("finish access");
      }
 out:
     if (fd >=0)
         close(fd);
     if (buf)
         free(buf);

     return 0;

}

prepare:
mkfifo access.pipe
./test
ps -ef | grep test
root 4106 3148 8 06:47 pts/0 00:00:01 ./test

We use /sys/kernel/debug/lru_gen to simulate mglru page-table scan.

case 1: mglru walker break page_idle ./idle_page_track 4106 60 &
sleep 5; echo 1 > access.pipe
sleep 5; echo '+ 8 0 6 1 1' > /sys/kernel/debug/lru_gen

the output of idle_page_track is:
Est(s) Ref(MB)
64.822 1.00
only found 1MB were accessed during 64.822s, but actually 1024MB were accessed.

case 2: page_idle break mglru walker
echo 1 > access.pipe
./idle_page_track 4106 10
echo '+ 8 0 7 1 1' > /sys/kernel/debug/lru_gen
lru gen status:
memcg 8 /user.slice
node 0

Quoted 7 lines
          5     772458       1065        9735
          6     737435     262244          72
          7     538053       1184         632
          8      59404       6422           0

almost pages should be in max_seq-1 queue, but actually not.

It's never intended for MGLRU to support page_idle/bitmap or PG_idle/young because:
1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.
2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

[1] https://lore.kernel.org/[email protected]/

#3 lore raw

Thanks for replying this RFC.

Quoted 3 lines

1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.

In our environment, we use /sys/kernel/mm/page_idle/bitmap to check pages whether were accessed during a peroid of time. We manage all pages idle time in userspace. Then use a prediction algorithm to select pages to reclaim. These pages would more likely be idled for a long time.

We only need kernel to tell use whether a page is accessed, a boolean value in kernel is enough for our case.

Quoted 3 lines

2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

Yes, at first time, we try using multi-gen LRU proactvie scan and exporting generation&refs number to do the same thing.

But there are serveral problems:

1. multi-gen LRU only care about self-memcg pages. In our environment, it's likely to see that different memcg's process share pages. multi-gen LRU only update gen of pages in current memcg. It's hard to judge a page whether is accessed depends on gen update.

We still have no ideas how to solve this problem.

2. We set swappiness 0, and use proactive scan to select cold pages & proactive reclaim to swap anon pages. But we can't control passive scan(can_swap = false), which would make anon pages cold/hot inversion in inc_min_seq.

Is it a good idea to include a interface to config passive scan option?

Signature · 1 line
2.43.0
#15 lore raw
In reply to Henry Huang
kernel test robot noticed the following build errors:

Hi Henry,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Henry-Huang/mm-Multi-Gen-LRU-fix-use-mm-page_idle-bitmap/20231206-205133
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/951fb7edab535cf522def4f5f2613947ed7b7d28.1701853894.git.henry.hj%40antgroup.com
patch subject: [RFC v2] mm: Multi-Gen LRU: fix use mm/page_idle/bitmap
config: i386-buildonly-randconfig-002-20231209 (https://download.01.org/0day-ci/archive/20231209/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231209/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/vmscan.c: In function 'get_pfn_folio':

>> mm/vmscan.c:3310:27: error: implicit declaration of function 'folio_test_idle'; did you mean 'folio_test_type'? [-Werror=implicit-function-declaration]
    3310 |         if (clear_idle && folio_test_idle(folio))
         |                           ^~~~~~~~~~~~~~~
         |                           folio_test_type
>> mm/vmscan.c:3311:17: error: implicit declaration of function 'folio_clear_idle'; did you mean 'folio_clear_head'? [-Werror=implicit-function-declaration]
    3311 |                 folio_clear_idle(folio);
         |                 ^~~~~~~~~~~~~~~~
         |                 folio_clear_head
   mm/vmscan.c: In function 'walk_pte_range':
>> mm/vmscan.c:3379:22: error: implicit declaration of function 'folio_test_clear_young'; did you mean 'folio_test_clear_lru'? [-Werror=implicit-function-declaration]
    3379 |                 if (!folio_test_clear_young(folio) && !is_pte_young) {
         |                      ^~~~~~~~~~~~~~~~~~~~~~
         |                      folio_test_clear_lru

cc1: some warnings being treated as errors

vim +3310 mm/vmscan.c

3298
3299 static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
3300 struct pglist_data *pgdat, bool can_swap, bool clear_idle)
3301 {
3302 struct folio *folio;
3303
3304 /* try to avoid unnecessary memory loads */
3305 if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat))
3306 return NULL;
3307
3308 folio = pfn_folio(pfn);
3309

> 3310		if (clear_idle && folio_test_idle(folio))
> 3311			folio_clear_idle(folio);
  3312	
  3313		if (folio_nid(folio) != pgdat->node_id)
  3314			return NULL;
  3315	
  3316		if (folio_memcg_rcu(folio) != memcg)
  3317			return NULL;
  3318	
  3319		/* file VMAs can contain anon pages from COW */
  3320		if (!folio_is_file_lru(folio) && !can_swap)
  3321			return NULL;
  3322	
  3323		return folio;
  3324	}
  3325	
  3326	static bool suitable_to_scan(int total, int young)
  3327	{
  3328		int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8);
  3329	
  3330		/* suitable if the average number of young PTEs per cacheline is >=1 */
  3331		return young * n >= total;
  3332	}
  3333	
  3334	static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
  3335				   struct mm_walk *args)
  3336	{
  3337		int i;
  3338		pte_t *pte;
  3339		spinlock_t *ptl;
  3340		unsigned long addr;
  3341		int total = 0;
  3342		int young = 0;
  3343		struct lru_gen_mm_walk *walk = args->private;
  3344		struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
  3345		struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
  3346		int old_gen, new_gen = lru_gen_from_seq(walk->max_seq);
  3347	
  3348		pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl);
  3349		if (!pte)
  3350			return false;
  3351		if (!spin_trylock(ptl)) {
  3352			pte_unmap(pte);
  3353			return false;
  3354		}
  3355	
  3356		arch_enter_lazy_mmu_mode();
  3357	restart:
  3358		for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
  3359			unsigned long pfn;
  3360			struct folio *folio;
  3361			pte_t ptent = ptep_get(pte + i);
  3362			bool is_pte_young;
  3363	
  3364			total++;
  3365			walk->mm_stats[MM_LEAF_TOTAL]++;
  3366	
  3367			pfn = get_pte_pfn(ptent, args->vma, addr);
  3368			if (pfn == -1)
  3369				continue;
  3370	
  3371			is_pte_young = !!pte_young(ptent);
  3372			folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pte_young);
  3373			if (!folio) {
  3374				if (!is_pte_young)
  3375					walk->mm_stats[MM_LEAF_OLD]++;
  3376				continue;
  3377			}
  3378	
> 3379			if (!folio_test_clear_young(folio) && !is_pte_young) {

3380 walk->mm_stats[MM_LEAF_OLD]++;
3381 continue;
3382 }
3383
3384 if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i))
3385 VM_WARN_ON_ONCE(true);
3386
3387 young++;
3388 walk->mm_stats[MM_LEAF_YOUNG]++;
3389
3390 if (pte_dirty(ptent) && !folio_test_dirty(folio) &&
3391 !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
3392 !folio_test_swapcache(folio)))
3393 folio_mark_dirty(folio);
3394
3395 old_gen = folio_update_gen(folio, new_gen);
3396 if (old_gen >= 0 && old_gen != new_gen)
3397 update_batch_size(walk, folio, old_gen, new_gen);
3398 }
3399
3400 if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
3401 goto restart;
3402
3403 arch_leave_lazy_mmu_mode();
3404 pte_unmap_unlock(pte, ptl);
3405
3406 return suitable_to_scan(total, young);
3407 }
3408

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#16 lore raw
In reply to Henry Huang
kernel test robot noticed the following build errors:

Hi Henry,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Henry-Huang/mm-Multi-Gen-LRU-fix-use-mm-page_idle-bitmap/20231206-205133
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/951fb7edab535cf522def4f5f2613947ed7b7d28.1701853894.git.henry.hj%40antgroup.com
patch subject: [RFC v2] mm: Multi-Gen LRU: fix use mm/page_idle/bitmap
config: i386-randconfig-013-20231209 (https://download.01.org/0day-ci/archive/20231209/[email protected]/config)
compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231209/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> mm/vmscan.c:3310:20: error: call to undeclared function 'folio_test_idle'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
           if (clear_idle && folio_test_idle(folio))
                             ^
>> mm/vmscan.c:3311:3: error: call to undeclared function 'folio_clear_idle'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
                   folio_clear_idle(folio);
                   ^
>> mm/vmscan.c:3379:8: error: call to undeclared function 'folio_test_clear_young'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
                   if (!folio_test_clear_young(folio) && !is_pte_young) {
                        ^
   mm/vmscan.c:3379:8: note: did you mean 'folio_test_clear_lru'?
   include/linux/page-flags.h:476:2: note: 'folio_test_clear_lru' declared here
           TESTCLEARFLAG(LRU, lru, PF_HEAD)
           ^
   include/linux/page-flags.h:416:6: note: expanded from macro 'TESTCLEARFLAG'
   bool folio_test_clear_##lname(struct folio *folio)                      \
        ^
   <scratch space>:139:1: note: expanded from here
   folio_test_clear_lru
   ^
   mm/vmscan.c:4053:8: error: call to undeclared function 'folio_test_clear_young'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
                   if (!folio_test_clear_young(folio) && !is_pte_young)
                        ^

4 errors generated.

Kconfig warnings: (for reference only)
WARNING: unmet direct dependencies detected for DRM_I915_DEBUG_GEM
Depends on [n]: HAS_IOMEM [=y] && DRM_I915 [=m] && EXPERT [=y] && DRM_I915_WERROR [=n] Selected by [m]:
- DRM_I915_DEBUG [=y] && HAS_IOMEM [=y] && DRM_I915 [=m] && EXPERT [=y] && !COMPILE_TEST [=n]

vim +/folio_test_idle +3310 mm/vmscan.c

3298
3299 static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
3300 struct pglist_data *pgdat, bool can_swap, bool clear_idle)
3301 {
3302 struct folio *folio;
3303
3304 /* try to avoid unnecessary memory loads */
3305 if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat))
3306 return NULL;
3307
3308 folio = pfn_folio(pfn);
3309

> 3310		if (clear_idle && folio_test_idle(folio))
> 3311			folio_clear_idle(folio);
  3312	
  3313		if (folio_nid(folio) != pgdat->node_id)
  3314			return NULL;
  3315	
  3316		if (folio_memcg_rcu(folio) != memcg)
  3317			return NULL;
  3318	
  3319		/* file VMAs can contain anon pages from COW */
  3320		if (!folio_is_file_lru(folio) && !can_swap)
  3321			return NULL;
  3322	
  3323		return folio;
  3324	}
  3325	
  3326	static bool suitable_to_scan(int total, int young)
  3327	{
  3328		int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8);
  3329	
  3330		/* suitable if the average number of young PTEs per cacheline is >=1 */
  3331		return young * n >= total;
  3332	}
  3333	
  3334	static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
  3335				   struct mm_walk *args)
  3336	{
  3337		int i;
  3338		pte_t *pte;
  3339		spinlock_t *ptl;
  3340		unsigned long addr;
  3341		int total = 0;
  3342		int young = 0;
  3343		struct lru_gen_mm_walk *walk = args->private;
  3344		struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec);
  3345		struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
  3346		int old_gen, new_gen = lru_gen_from_seq(walk->max_seq);
  3347	
  3348		pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl);
  3349		if (!pte)
  3350			return false;
  3351		if (!spin_trylock(ptl)) {
  3352			pte_unmap(pte);
  3353			return false;
  3354		}
  3355	
  3356		arch_enter_lazy_mmu_mode();
  3357	restart:
  3358		for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
  3359			unsigned long pfn;
  3360			struct folio *folio;
  3361			pte_t ptent = ptep_get(pte + i);
  3362			bool is_pte_young;
  3363	
  3364			total++;
  3365			walk->mm_stats[MM_LEAF_TOTAL]++;
  3366	
  3367			pfn = get_pte_pfn(ptent, args->vma, addr);
  3368			if (pfn == -1)
  3369				continue;
  3370	
  3371			is_pte_young = !!pte_young(ptent);
  3372			folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pte_young);
  3373			if (!folio) {
  3374				if (!is_pte_young)
  3375					walk->mm_stats[MM_LEAF_OLD]++;
  3376				continue;
  3377			}
  3378	
> 3379			if (!folio_test_clear_young(folio) && !is_pte_young) {

3380 walk->mm_stats[MM_LEAF_OLD]++;
3381 continue;
3382 }
3383
3384 if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i))
3385 VM_WARN_ON_ONCE(true);
3386
3387 young++;
3388 walk->mm_stats[MM_LEAF_YOUNG]++;
3389
3390 if (pte_dirty(ptent) && !folio_test_dirty(folio) &&
3391 !(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
3392 !folio_test_swapcache(folio)))
3393 folio_mark_dirty(folio);
3394
3395 old_gen = folio_update_gen(folio, new_gen);
3396 if (old_gen >= 0 && old_gen != new_gen)
3397 update_batch_size(walk, folio, old_gen, new_gen);
3398 }
3399
3400 if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
3401 goto restart;
3402
3403 arch_leave_lazy_mmu_mode();
3404 pte_unmap_unlock(pte, ptl);
3405
3406 return suitable_to_scan(total, young);
3407 }
3408

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#4 lore raw
In reply to Henry Huang
Quoted 9 lines · Henry Huang

Thanks for replying this RFC.

Quoted 3 lines

1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.

In our environment, we use /sys/kernel/mm/page_idle/bitmap to check pages whether were accessed during a peroid of time.

Is it a production environment? If so, what's your
1. scan interval
2. memory size

I'm trying to understand why scalability isn't a problem for you. On an average server, there are hundreds of millions of PFNs, so it'd be very expensive to use that ABI even for a time interval of minutes.

Quoted 3 lines

We manage all pages
idle time in userspace. Then use a prediction algorithm to select pages to reclaim. These pages would more likely be idled for a long time.

"There is a system in place now that is based on a user-space process that reads a bitmap stored in sysfs, but it has a high CPU and memory overhead, so a new approach is being tried."
https://lwn.net/Articles/787611/

Could you elaborate how you solved this problem?

Quoted 2 lines

We only need kernel to tell use whether a page is accessed, a boolean value in kernel is enough for our case.

How do you define "accessed"? I.e., through page tables or file descriptors or both?

Quoted 11 lines
Quoted 3 lines

2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

Yes, at first time, we try using multi-gen LRU proactvie scan and exporting generation&refs number to do the same thing.

But there are serveral problems:

1. multi-gen LRU only care about self-memcg pages. In our environment, it's likely to see that different memcg's process share pages.

This is related to my question above: are those pages mapped into different memcgs or not?

Quoted 2 lines

multi-gen LRU only update gen of pages in current memcg. It's hard to judge a page whether is accessed depends on gen update.

This depends. I'd be glad to elaborate after you clarify the above.

Quoted 6 lines

We still have no ideas how to solve this problem.

2. We set swappiness 0, and use proactive scan to select cold pages & proactive reclaim to swap anon pages. But we can't control passive scan(can_swap = false), which would make anon pages cold/hot inversion in inc_min_seq.

There is an option to prevent the inversion, IIUC, the force_scan option is what you are looking for.

#17 lore raw
In reply to Henry Huang
Quoted 99 lines · Henry Huang

Multi-Gen LRU page-table walker clears pte young flag, but it doesn't clear page idle flag. When we use /sys/kernel/mm/page_idle/bitmap to check whether one page is accessed, it would tell us this page is idle, but actually this page has been accessed.

For those unmapped filecache pages, page idle flag would not been cleared in folio_mark_accessed if Multi-Gen LRU is enabled. So we couln't use /sys/kernel/mm/page_idle/bitmap to check whether a filecache page is read or written.

What's more, /sys/kernel/mm/page_idle/bitmap also clears pte young flag. If one page is accessed, it would set page young flag. Multi-Gen LRU page-table walker should check both page&pte young flags.

how-to-reproduce-problem

idle_page_track
a tools to track process accessed memory during a specific time
usage
idle_page_track $pid $time
how-it-works
1. scan process vma from /proc/$pid/maps
2. vfn --> pfn from /proc/$pid/pagemap
3. write /sys/kernel/mm/page_idle/bitmap to
mark phy page idle flag and clear pte young flag
4. sleep $time
5. read /sys/kernel/mm/page_idle/bitmap to test_and_clear pte young flag and
return whether phy page is accessed

test ---- test program

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, const char *argv[])

 {
     char *buf = NULL;
     char pipe_info[4096];
     int n;
     int fd = -1;

     buf = malloc(1024*1024*1024UL);
     memset(buf, 0, 1024*1024*1024UL);
     fd = open("access.pipe", O_RDONLY);
     if (fd < 0)
         goto out;
     while (1) {
         n = read(fd, pipe_info, sizeof(pipe_info));
         if (!n) {
             sleep(1);
             continue;
         } else if (n < 0) {
             break;
         }
         memset(buf, 0, 1024*1024*1024UL);
         puts("finish access");
      }
 out:
     if (fd >=0)
         close(fd);
     if (buf)
         free(buf);

     return 0;

}

prepare:
mkfifo access.pipe
./test
ps -ef | grep test
root 4106 3148 8 06:47 pts/0 00:00:01 ./test

We use /sys/kernel/debug/lru_gen to simulate mglru page-table scan.

case 1: mglru walker break page_idle ./idle_page_track 4106 60 &
sleep 5; echo 1 > access.pipe
sleep 5; echo '+ 8 0 6 1 1' > /sys/kernel/debug/lru_gen

the output of idle_page_track is:
Est(s) Ref(MB)
64.822 1.00
only found 1MB were accessed during 64.822s, but actually 1024MB were accessed.

case 2: page_idle break mglru walker
echo 1 > access.pipe
./idle_page_track 4106 10
echo '+ 8 0 7 1 1' > /sys/kernel/debug/lru_gen
lru gen status:
memcg 8 /user.slice
node 0

Quoted 7 lines
          5     772458       1065        9735
          6     737435     262244          72
          7     538053       1184         632
          8      59404       6422           0

almost pages should be in max_seq-1 queue, but actually not.

Regarding the change itself, it'd cause a slight regression to other use cases (details below).

Quoted 9 lines
mm/vmscan.c +1 -0
@@ -3355,6 +3359,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                unsigned long pfn;
                struct folio *folio;
                pte_t ptent = ptep_get(pte + i);
+               bool is_pte_young;
 
                total++;
                walk->mm_stats[MM_LEAF_TOTAL]++;

mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3367

Quoted 5 lines
mm/vmscan.c +0 -2
@@ -3363,16 +3368,20 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                if (pfn == -1)
                        continue;
 
-               if (!pte_young(ptent)) {
-                       walk->mm_stats[MM_LEAF_OLD]++;

Most overhead from page table scanning normally comes from get_pfn_folio() because it almost always causes a cache miss. This is like a pointer dereference, whereas scanning PTEs is like streaming an array (bad vs good cache performance).

pte_young() is here to avoid an unnecessary cache miss from get_pfn_folio(). Also see the first comment in get_pfn_folio(). It should be easy to verify the regression -- FlameGraph from the memcached benchmark in the original commit message should do it.

Would a tracepoint here work for you?

Quoted 20 lines
mm/vmscan.c +9 -3
+               is_pte_young = !!pte_young(ptent);
+               folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pte_young);
+               if (!folio) {
+                       if (!is_pte_young)
+                               walk->mm_stats[MM_LEAF_OLD]++;
                        continue;
                }
 
-               folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap);
-               if (!folio)
+               if (!folio_test_clear_young(folio) && !is_pte_young) {
+                       walk->mm_stats[MM_LEAF_OLD]++;
                        continue;
+               }
 
-               if (!ptep_test_and_clear_young(args->vma, addr, pte + i))
+               if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i))
                        VM_WARN_ON_ONCE(true);
 
                young++;
#5 lore raw
In reply to Yu Zhao
Quoted 10 lines · Yu Zhao
Quoted 9 lines

Thanks for replying this RFC.

Quoted 3 lines

1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.

In our environment, we use /sys/kernel/mm/page_idle/bitmap to check pages whether were accessed during a peroid of time.

Quoted 7 lines

Is it a production environment? If so, what's your
1. scan interval
2. memory size

I'm trying to understand why scalability isn't a problem for you. On an average server, there are hundreds of millions of PFNs, so it'd be very expensive to use that ABI even for a time interval of minutes.

Thanks for replying.

Our scan interval is 10 minutes and total memory size is 512GB. We perferred to reclaim pages which idle age > 1 hour at least.

Quoted 10 lines
Quoted 3 lines

We manage all pages
idle time in userspace. Then use a prediction algorithm to select pages to reclaim. These pages would more likely be idled for a long time.

"There is a system in place now that is based on a user-space process that reads a bitmap stored in sysfs, but it has a high CPU and memory overhead, so a new approach is being tried."
https://lwn.net/Articles/787611/

Could you elaborate how you solved this problem?

In out environment, we found that we take average 0.4 core and 300MB memory to do scan, basic analyse and reclaim idle pages.

For reducing cpu & memroy usage, we do:
1. We implement a ratelimiter to control rate of scan and reclaim.
2. All pages info & idle age were stored in local DB file. Our prediction algorithm don't need all pages info in memory at the same time.

In out environment, about 1/3 memory was attemped to allocate as THP, which may save some cpu usage of scan.

Quoted 5 lines
Quoted 2 lines

We only need kernel to tell use whether a page is accessed, a boolean value in kernel is enough for our case.

How do you define "accessed"? I.e., through page tables or file descriptors or both?

both

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

Yes, at first time, we try using multi-gen LRU proactvie scan and exporting generation&refs number to do the same thing.

Quoted 7 lines
Quoted 4 lines

But there are serveral problems:

1. multi-gen LRU only care about self-memcg pages. In our environment, it's likely to see that different memcg's process share pages.

This is related to my question above: are those pages mapped into different memcgs or not?

There is a case:
There are two cgroup A, B (B is child cgroup of A) Process in A create a file and use mmap to read/write this file. Process in B mmap this file and usually read this file.

Quoted 9 lines
Quoted 6 lines

We still have no ideas how to solve this problem.

2. We set swappiness 0, and use proactive scan to select cold pages & proactive reclaim to swap anon pages. But we can't control passive scan(can_swap = false), which would make anon pages cold/hot inversion in inc_min_seq.

There is an option to prevent the inversion, IIUC, the force_scan option is what you are looking for.

It seems that doesn't work now.

static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) {
......

    for (type = ANON_AND_FILE - 1; type >= 0; type--) {
        if (get_nr_gens(lruvec, type) != MAX_NR_GENS)
            continue;

        VM_WARN_ON_ONCE(!force_scan && (type == LRU_GEN_FILE || can_swap));

        if (inc_min_seq(lruvec, type, can_swap))
            continue;

        spin_unlock_irq(&lruvec->lru_lock);
        cond_resched();
        goto restart;
    }

.....
}

force_scan is not a parameter of inc_min_seq.
In our environment, swappiness is 0, so can_swap would be false.

static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap)

{
    int zone;
    int remaining = MAX_LRU_BATCH;
    struct lru_gen_folio *lrugen = &lruvec->lrugen;
    int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);

    if (type == LRU_GEN_ANON && !can_swap)
        goto done;

......
}

If can_swap is false, would pass anon lru list.

What's more, in passive scan, force_scan is also false.

static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool can_swap) {
......

    /* skip this lruvec as it's low on cold folios */
    return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0;
}

Is it a good idea to include a global parameter no_inversion, and modify inc_min_seq like this:

static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap)

{
    int zone;
    int remaining = MAX_LRU_BATCH;
    struct lru_gen_folio *lrugen = &lruvec->lrugen;
    int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);

- if (type == LRU_GEN_ANON && !can_swap)
+ if (type == LRU_GEN_ANON && !can_swap && !no_inversion)

        goto done;

......
}

Signature · 1 line
2.43.0
#18 lore raw
In reply to Yu Zhao
Quoted 23 lines · Yu Zhao

Regarding the change itself, it'd cause a slight regression to other use cases (details below).

Quoted 9 lines
mm/vmscan.c +1 -0
@@ -3355,6 +3359,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                unsigned long pfn;
                struct folio *folio;
                pte_t ptent = ptep_get(pte + i);
+               bool is_pte_young;
 
                total++;
                walk->mm_stats[MM_LEAF_TOTAL]++;

mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3367

Quoted 5 lines
mm/vmscan.c +0 -2
@@ -3363,16 +3368,20 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                if (pfn == -1)
                        continue;
 
-               if (!pte_young(ptent)) {
-                       walk->mm_stats[MM_LEAF_OLD]++;

Most overhead from page table scanning normally comes from get_pfn_folio() because it almost always causes a cache miss. This is like a pointer dereference, whereas scanning PTEs is like streaming an array (bad vs good cache performance).

Quoted 29 lines

pte_young() is here to avoid an unnecessary cache miss from get_pfn_folio(). Also see the first comment in get_pfn_folio(). It should be easy to verify the regression -- FlameGraph from the memcached benchmark in the original commit message should do it.

Would a tracepoint here work for you?

Quoted 20 lines
mm/vmscan.c +9 -3
+               is_pte_young = !!pte_young(ptent);
+               folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, is_pte_young);
+               if (!folio) {
+                       if (!is_pte_young)
+                               walk->mm_stats[MM_LEAF_OLD]++;
                        continue;
                }
 
-               folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap);
-               if (!folio)
+               if (!folio_test_clear_young(folio) && !is_pte_young) {
+                       walk->mm_stats[MM_LEAF_OLD]++;
                        continue;
+               }
 
-               if (!ptep_test_and_clear_young(args->vma, addr, pte + i))
+               if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i))
                        VM_WARN_ON_ONCE(true);
 
                young++;

Thanks for replying.

For avoiding below:
1. confict between page_idle/bitmap and mglru scan
2. performance downgrade in mglru page-table scan if call get_pfn_folio for each pte.

We have a new idea:
1. Include a new api under /sys/kernel/mm/page_idle, support mark idle flag only, without rmap walking or clearing pte young.
2. Use mglru proactive scan to clear page idle flag.

workflows:
t1 t2
mark pages idle mglru scan and check pages idle

It's easy for us to know that whether a page is accessed during t1~t2.

Some code changes like these:

We clear idle flags in get_pfn_folio, and in walk_pte_range we still follow original design.

static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
struct pglist_data *pgdat, bool can_swap, bool clear_idle)

{
	struct folio *folio;

	/* try to avoid unnecessary memory loads */
	if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat))
		return NULL;

	folio = pfn_folio(pfn);

+
+ if (clear_idle && folio_test_idle(folio))
+ folio_clear_idle(folio);
+

	if (folio_nid(folio) != pgdat->node_id)
		return NULL;

	if (folio_memcg_rcu(folio) != memcg)
		return NULL;

	/* file VMAs can contain anon pages from COW */
	if (!folio_is_file_lru(folio) && !can_swap)
		return NULL;

	return folio;
}

static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
struct mm_walk *args)
{
...

	for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
		unsigned long pfn;
		struct folio *folio;
		pte_t ptent = ptep_get(pte + i);

		total++;
		walk->mm_stats[MM_LEAF_TOTAL]++;

		pfn = get_pte_pfn(ptent, args->vma, addr);
		if (pfn == -1)
			continue;

		if (!pte_young(ptent)) {
			walk->mm_stats[MM_LEAF_OLD]++;
			continue;
		}

+ folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap, true);

-		folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap);
		if (!folio)
			continue;

...
}

Is it a good idea or not ?

Signature · 1 line
2.43.0
#6 lore raw
In reply to Henry Huang
Quoted 16 lines · Henry Huang
Quoted 13 lines · Yu Zhao
Quoted 9 lines

Thanks for replying this RFC.

Quoted 3 lines

1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.

In our environment, we use /sys/kernel/mm/page_idle/bitmap to check pages whether were accessed during a peroid of time.

Is it a production environment? If so, what's your
1. scan interval
2. memory size

Quoted 8 lines
Quoted 3 lines

I'm trying to understand why scalability isn't a problem for you. On an average server, there are hundreds of millions of PFNs, so it'd be very expensive to use that ABI even for a time interval of minutes.

Thanks for replying.

Our scan interval is 10 minutes and total memory size is 512GB. We perferred to reclaim pages which idle age > 1 hour at least.

Yes, that makes sense. We have similar use cases, details below.

Quoted 42 lines
Quoted 10 lines
Quoted 3 lines

We manage all pages
idle time in userspace. Then use a prediction algorithm to select pages to reclaim. These pages would more likely be idled for a long time.

"There is a system in place now that is based on a user-space process that reads a bitmap stored in sysfs, but it has a high CPU and memory overhead, so a new approach is being tried."
https://lwn.net/Articles/787611/

Could you elaborate how you solved this problem?

In out environment, we found that we take average 0.4 core and 300MB memory to do scan, basic analyse and reclaim idle pages.

For reducing cpu & memroy usage, we do:
1. We implement a ratelimiter to control rate of scan and reclaim.
2. All pages info & idle age were stored in local DB file. Our prediction algorithm don't need all pages info in memory at the same time.

In out environment, about 1/3 memory was attemped to allocate as THP, which may save some cpu usage of scan.

Quoted 5 lines
Quoted 2 lines

We only need kernel to tell use whether a page is accessed, a boolean value in kernel is enough for our case.

How do you define "accessed"? I.e., through page tables or file descriptors or both?

both

Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

Yes, at first time, we try using multi-gen LRU proactvie scan and exporting generation&refs number to do the same thing.

But there are serveral problems:

1. multi-gen LRU only care about self-memcg pages. In our environment, it's likely to see that different memcg's process share pages.

Quoted 7 lines
Quoted 2 lines

This is related to my question above: are those pages mapped into different memcgs or not?

There is a case:
There are two cgroup A, B (B is child cgroup of A) Process in A create a file and use mmap to read/write this file. Process in B mmap this file and usually read this file.

Yes, actually we have a private patch to solve a similar problem. Basically it finds VMAs from other processes in different memcgs that share a mapping and jumps to those VMAs to scan them. We can upstream it for you if you find it useful too.

Quoted 43 lines
Quoted 9 lines
Quoted 6 lines

We still have no ideas how to solve this problem.

2. We set swappiness 0, and use proactive scan to select cold pages & proactive reclaim to swap anon pages. But we can't control passive scan(can_swap = false), which would make anon pages cold/hot inversion in inc_min_seq.

There is an option to prevent the inversion, IIUC, the force_scan option is what you are looking for.

It seems that doesn't work now.

static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) {
......

    for (type = ANON_AND_FILE - 1; type >= 0; type--) {
        if (get_nr_gens(lruvec, type) != MAX_NR_GENS)
            continue;

        VM_WARN_ON_ONCE(!force_scan && (type == LRU_GEN_FILE || can_swap));

        if (inc_min_seq(lruvec, type, can_swap))
            continue;

        spin_unlock_irq(&lruvec->lru_lock);
        cond_resched();
        goto restart;
    }

.....
}

force_scan is not a parameter of inc_min_seq.
In our environment, swappiness is 0, so can_swap would be false.

static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap)

{
    int zone;
    int remaining = MAX_LRU_BATCH;
    struct lru_gen_folio *lrugen = &lruvec->lrugen;
    int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);

    if (type == LRU_GEN_ANON && !can_swap)
        goto done;
Quoted 6 lines

......
}

If can_swap is false, would pass anon lru list.

What's more, in passive scan, force_scan is also false.

Ok, I see what you mean. (I thought "passive" means proactive scans triggered by the debugfs interface, but it actually means "reactive" scans triggered by memory pressure.) We actually have a private patch too to solve this. But there is a corner case here: that private change, which is essentially the same as what you suggested, can stall direct reclaim when there is tons of cold anon memory. E.g., if there is 300GB anon memory in the oldest generation which can't be swapped, calling inc_min_seq() with can_swap being true would stall the direct reclaim. Does it make sense?

Let me check the state of those private patches and get back to you in a couple of days. Thanks!

#7 lore raw
Quoted 2 lines · Yu Zhao

Let me check the state of those private patches and get back to you in a couple of days. Thanks!

Thanks very much!
Looking forward your patches.

Signature · 1 line
2.43.0
reviewer
#8 lore raw
In reply to Henry Huang

Hi Henry, I have a question on memcg charging for the shared pages. Replied inline.

Quoted 67 lines · Henry Huang
Quoted 10 lines · Yu Zhao
Quoted 9 lines

Thanks for replying this RFC.

Quoted 3 lines

1. page_idle/bitmap isn't a capable interface at all -- yes, Google proposed the idea [1], but we don't really use it anymore because of its poor scalability.

In our environment, we use /sys/kernel/mm/page_idle/bitmap to check pages whether were accessed during a peroid of time.

Quoted 7 lines

Is it a production environment? If so, what's your
1. scan interval
2. memory size

I'm trying to understand why scalability isn't a problem for you. On an average server, there are hundreds of millions of PFNs, so it'd be very expensive to use that ABI even for a time interval of minutes.

Thanks for replying.

Our scan interval is 10 minutes and total memory size is 512GB. We perferred to reclaim pages which idle age > 1 hour at least.

Quoted 10 lines
Quoted 3 lines

We manage all pages
idle time in userspace. Then use a prediction algorithm to select pages to reclaim. These pages would more likely be idled for a long time.

"There is a system in place now that is based on a user-space process that reads a bitmap stored in sysfs, but it has a high CPU and memory overhead, so a new approach is being tried."
https://lwn.net/Articles/787611/

Could you elaborate how you solved this problem?

In out environment, we found that we take average 0.4 core and 300MB memory to do scan, basic analyse and reclaim idle pages.

For reducing cpu & memroy usage, we do:
1. We implement a ratelimiter to control rate of scan and reclaim.
2. All pages info & idle age were stored in local DB file. Our prediction algorithm don't need all pages info in memory at the same time.

In out environment, about 1/3 memory was attemped to allocate as THP, which may save some cpu usage of scan.

Quoted 5 lines
Quoted 2 lines

We only need kernel to tell use whether a page is accessed, a boolean value in kernel is enough for our case.

How do you define "accessed"? I.e., through page tables or file descriptors or both?

both

Quoted 12 lines
Quoted 11 lines
Quoted 3 lines

2. PG_idle/young, being a boolean value, has poor granularity. If anyone must use page_idle/bitmap for some specific reason, I'd recommend exporting generation numbers instead.

Yes, at first time, we try using multi-gen LRU proactvie scan and exporting generation&refs number to do the same thing.

But there are serveral problems:

1. multi-gen LRU only care about self-memcg pages. In our environment, it's likely to see that different memcg's process share pages.

Quoted 7 lines
Quoted 2 lines

This is related to my question above: are those pages mapped into different memcgs or not?

There is a case:
There are two cgroup A, B (B is child cgroup of A) Process in A create a file and use mmap to read/write this file. Process in B mmap this file and usually read this file.\

How does the shared memory get charged to the cgroups? Does it all go to cgroup A or B exclusively, or do some pages get charged to each one?

Quoted 78 lines
Quoted 10 lines
Quoted 7 lines

We still have no ideas how to solve this problem.

2. We set swappiness 0, and use proactive scan to select cold pages & proactive reclaim to swap anon pages. But we can't control passive scan(can_swap = false), which would make anon pages cold/hot inversion in inc_min_seq.

There is an option to prevent the inversion, IIUC, the force_scan option is what you are looking for.

It seems that doesn't work now.

static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) {
......

    for (type = ANON_AND_FILE - 1; type >= 0; type--) {
        if (get_nr_gens(lruvec, type) != MAX_NR_GENS)
            continue;

        VM_WARN_ON_ONCE(!force_scan && (type == LRU_GEN_FILE || can_swap));

        if (inc_min_seq(lruvec, type, can_swap))
            continue;

        spin_unlock_irq(&lruvec->lru_lock);
        cond_resched();
        goto restart;
    }

.....
}

force_scan is not a parameter of inc_min_seq.
In our environment, swappiness is 0, so can_swap would be false.

static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap)

{
    int zone;
    int remaining = MAX_LRU_BATCH;
    struct lru_gen_folio *lrugen = &lruvec->lrugen;
    int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);

    if (type == LRU_GEN_ANON && !can_swap)
        goto done;

......
}

If can_swap is false, would pass anon lru list.

What's more, in passive scan, force_scan is also false.

static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool can_swap) {
......

    /* skip this lruvec as it's low on cold folios */
    return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0;
}

Is it a good idea to include a global parameter no_inversion, and modify inc_min_seq like this:

static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap)

{
    int zone;
    int remaining = MAX_LRU_BATCH;
    struct lru_gen_folio *lrugen = &lruvec->lrugen;
    int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
(quoted diff) +1 -1
-   if (type == LRU_GEN_ANON && !can_swap)
+   if (type == LRU_GEN_ANON && !can_swap && !no_inversion)
        goto done;
......

}

--
2.43.0

#9 lore raw

Thanks for replyting.

Quoted 3 lines · Yuanchu Xie

How does the shared memory get charged to the cgroups? Does it all go to cgroup A or B exclusively, or do some pages get charged to each one?

Some pages get charged to cgroup A, and the other get charged to cgroup B.

Signature · 1 line
2.43.0
#10 lore raw
Quoted 9 lines · Henry Huang

Thanks for replyting.

Quoted 3 lines · Yuanchu Xie

How does the shared memory get charged to the cgroups? Does it all go to cgroup A or B exclusively, or do some pages get charged to each one?

Some pages get charged to cgroup A, and the other get charged to cgroup B.

Just a side note:
We can potentially "fix" this, but it doesn't mean this is a good practice. In fact, I think this is an anti-pattern to memcgs:
resources should be preferably isolated between memcgs, or if a resource has to be shared between memcgs, it should be charged in a predetermined way, not randomly to one of the memcgs sharing it.

#11 lore raw
Quoted 7 lines · Yu Zhao
Quoted 7 lines

Thanks for replyting.

Quoted 3 lines · Yuanchu Xie

How does the shared memory get charged to the cgroups? Does it all go to cgroup A or B exclusively, or do some pages get charged to each one?

Quoted 9 lines
Quoted 1 line

Some pages get charged to cgroup A, and the other get charged to cgroup B.

Just a side note:
We can potentially "fix" this, but it doesn't mean this is a good practice. In fact, I think this is an anti-pattern to memcgs:
resources should be preferably isolated between memcgs, or if a resource has to be shared between memcgs, it should be charged in a predetermined way, not randomly to one of the memcgs sharing it.

Very interesting thread, a few questions for Henry to understand the situation better:

- is the lack of predeterministic charging a problem for you? Are you initially faulting it in a manner that charges it to the "right" memcg and the refault of it after periodic reclaim can causing the charge to appear "randomly," i.e. to whichever process happened to access it next?

- are pages ever shared between different memcg hierarchies? You mentioned sharing between processes in A and A/B, but I'm wondering if there is sharing between two different memcg hierarchies where root is the only common ancestor?

- do you anticipate a shorter scan period at some point? Proactively reclaiming all memory colder than one hour is a long time :) Are you concerned at all about the cost of doing your current idle bit harvesting approach becoming too expensive if you significantly reduce the scan period?

- is proactive reclaim being driven by writing to memory.reclaim, by enforcing a smaller memory.high, or something else?

Looking forward to learning more about your particular issue.

#12 lore raw

Thanks for replying.

Quoted 5 lines · David Rientjes

- is the lack of predeterministic charging a problem for you? Are you initially faulting it in a manner that charges it to the "right" memcg and the refault of it after periodic reclaim can causing the charge to appear "randomly," i.e. to whichever process happened to access it next?

Actually at begin, all pages got charged to cgroup A, but with memory pressure or after proactive reclaim. Some pages would be dropped or swapped. Task in cgroup B visit this shared memory before task in cgroup A, would make these pages charged to cgroup B.

This is common in our enviorment.

Quoted 4 lines

- are pages ever shared between different memcg hierarchies? You mentioned sharing between processes in A and A/B, but I'm wondering if there is sharing between two different memcg hierarchies where root is the only common ancestor?

Yes, there is a another really common case:
If docker graph driver is overlayfs, different docker containers use the same image, or share same low layers, would share file cache of public bin or lib(i.e libc.so).

Quoted 5 lines

- do you anticipate a shorter scan period at some point? Proactively reclaiming all memory colder than one hour is a long time :) Are you concerned at all about the cost of doing your current idle bit harvesting approach becoming too expensive if you significantly reduce the scan period?

We don't want the owner of the application to feel a significant performance downgrade when using swap. There is a high risk to reclaim pages which idle age are less than 1 hour. We have internal test and data analysis to support it.

We disabled global swappiness and memcg swapinness. Only proactive reclaim can swap anon pages.

What's more, we see that mglru has a more efficient way to scan pte access bit. We perferred to use mglru scan help us scan and select idle pages.

Quoted 2 lines

- is proactive reclaim being driven by writing to memory.reclaim, by enforcing a smaller memory.high, or something else?

Because all pages info and idle age are stored in userspace, kernel can't get these information directly. We have a private patch include a new reclaim interface to support reclaim pages with specific pfns.

Signature · 1 line
2.43.0
reviewer
#13 lore raw
Quoted 9 lines · Henry Huang
Quoted 4 lines

- are pages ever shared between different memcg hierarchies? You mentioned sharing between processes in A and A/B, but I'm wondering if there is sharing between two different memcg hierarchies where root is the only common ancestor?

Yes, there is a another really common case:
If docker graph driver is overlayfs, different docker containers use the same image, or share same low layers, would share file cache of public bin or lib(i.e libc.so).

Does this present a problem with setting memcg limits or OOMs? It seems like deterministically charging shared pages would be highly desirable. Mina Almasry previously proposed a memcg= mount option to implement deterministic charging[1], but it wasn't a generic sharing mechanism. Nonetheless, the problem remains, and it would be interesting to learn if this presents any issues for you.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 12 lines
Quoted 6 lines

- do you anticipate a shorter scan period at some point? Proactively reclaiming all memory colder than one hour is a long time :) Are you concerned at all about the cost of doing your current idle bit harvesting approach becoming too expensive if you significantly reduce the scan period?

We don't want the owner of the application to feel a significant performance downgrade when using swap. There is a high risk to reclaim pages which idle age are less than 1 hour. We have internal test and data analysis to support it.

Quoted 5 lines

We disabled global swappiness and memcg swapinness. Only proactive reclaim can swap anon pages.

What's more, we see that mglru has a more efficient way to scan pte access bit. We perferred to use mglru scan help us scan and select idle pages.

I'm working on a kernel driver/per-memcg interface to perform aging with MGLRU, including configuration for the MGLRU page scanning optimizations. I suspect scanning the PTE accessed bits for pages charged to a foreign memcg ad-hoc has some performance implications, and the more general solution is to charge in a predetermined way, which makes the scanning on behalf of the foreign memcg a bit cleaner. This is possible nonetheless, but a bit hacky. Let me know you have any ideas.

Quoted 7 lines
Quoted 3 lines

- is proactive reclaim being driven by writing to memory.reclaim, by enforcing a smaller memory.high, or something else?

Because all pages info and idle age are stored in userspace, kernel can't get these information directly. We have a private patch include a new reclaim interface to support reclaim pages with specific pfns.

Thanks for sharing! It's been enlightening to learn about different prod environments.

#14 lore raw

Thanks for replying.

Quoted 8 lines · Yuanchu Xie

Does this present a problem with setting memcg limits or OOMs? It seems like deterministically charging shared pages would be highly desirable. Mina Almasry previously proposed a memcg= mount option to implement deterministic charging[1], but it wasn't a generic sharing mechanism. Nonetheless, the problem remains, and it would be interesting to learn if this presents any issues for you.

[1] https://lore.kernel.org/linux-mm/[email protected]/

In this case, total size of shared memory usually is small(< 100MB). What's more, almost shared pages were file cache. So it doesn't present any problems.

Quoted 8 lines

I'm working on a kernel driver/per-memcg interface to perform aging with MGLRU, including configuration for the MGLRU page scanning optimizations. I suspect scanning the PTE accessed bits for pages charged to a foreign memcg ad-hoc has some performance implications, and the more general solution is to charge in a predetermined way, which makes the scanning on behalf of the foreign memcg a bit cleaner. This is possible nonetheless, but a bit hacky. Let me know you have any ideas.

Wow! per-memcg interface is also what we need.
It's hardly to control user's behaviors in our envrionment. We can't promise that all process who share memory would be in same memcg.
Maybe kernel should provide new interface to make shared page charge more predictable. I think it would take some overhead to do this.

The key problem of us is that we can't check whether page is accesed(no gen or ref changed) in this case.
page belongs to A, but maybe process in B read/write this page more frequently. we may treat this page as cold page but accutly hot page.
Maybe just call folio_mark_accessed instead of folio_update_gen(should hold memcg lru lock) for those remote memcg pages?

Signature · 1 line
2.43.0