mm/mglru: Fix young counter undercount for large folios

From: Hui Zhu <[email protected]>

In lru_gen_look_around(), the young counter tracks the number of young PTEs. The original folio's contribution is represented by the initial value of young: test_and_clear_young_ptes_notify() is called on it at function entry, and the function returns early if it is not young. In the subsequent loop, the original folio is skipped (its accessed bits were already cleared), so it is not double-counted.

However, young is initialized to 1 regardless of the folio size. When the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added to the bloom filter and reducing aging accuracy for mTHP workloads.

Initialize young to nr so the original folio is accounted the same way as other young folios in the loop (young += nr).

Signed-off-by Hui Zhu <[email protected]>

---

  mm/vmscan.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

mm/vmscan.c +1 -1

index bc324e37c5f1..264017850a55 100644
@@ -4192,7 +4192,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
  	unsigned long end;
  	struct lru_gen_mm_walk *walk;
  	struct folio *last = NULL;
-	int young = 1;
+	int young = nr;
  	pte_t *pte = pvmw->pte;
  	unsigned long addr = pvmw->address;
  	struct vm_area_struct *vma = pvmw->vma;

Quoted 15 lines · Hui Zhu

From: Hui Zhu <[email protected]>

In lru_gen_look_around(), the young counter tracks the number of young PTEs. The original folio's contribution is represented by the initial value of young: test_and_clear_young_ptes_notify() is called on it at function entry, and the function returns early if it is not young. In the subsequent loop, the original folio is skipped (its accessed bits were already cleared), so it is not double-counted.

However, young is initialized to 1 regardless of the folio size. When the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added to the bloom filter and reducing aging accuracy for mTHP workloads.

Quoted 5 lines

Initialize young to nr so the original folio is accounted the same way as other young folios in the loop (young += nr).

Signed-off-by Hui Zhu <[email protected]>

---

Good catch. Please also add the Fixes tag:

Fixes 56e5b60b2114 ("mm: support batched checking of the young flag for

MGLRU")

With that,

Reviewed-by Baolin Wang <[email protected]>

Quoted 4 lines

  mm/vmscan.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

mm/vmscan.c bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) · line 4198

Quoted 13 lines

mm/vmscan.c +1 -1

index bc324e37c5f1..264017850a55 100644
@@ -4192,7 +4192,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
      unsigned long end;
      struct lru_gen_mm_walk *walk;
      struct folio *last = NULL;
-     int young = 1;
+     int young = nr;
      pte_t *pte = pvmw->pte;
      unsigned long addr = pvmw->address;
      struct vm_area_struct *vma = pvmw->vma;

Quoted 25 lines · Baolin Wang

Quoted 15 lines · Hui Zhu

From: Hui Zhu <[email protected]>

In lru_gen_look_around(), the young counter tracks the number of young PTEs. The original folio's contribution is represented by the initial value of young: test_and_clear_young_ptes_notify() is called on it at function entry, and the function returns early if it is not young. In the subsequent loop, the original folio is skipped (its accessed bits were already cleared), so it is not double-counted.

However, young is initialized to 1 regardless of the folio size. When the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added to the bloom filter and reducing aging accuracy for mTHP workloads.

Quoted 5 lines

Initialize young to nr so the original folio is accounted the same way as other young folios in the loop (young += nr).

Signed-off-by Hui Zhu <[email protected]>

---

Quoted 7 lines

Good catch. Please also add the Fixes tag:

Fixes 56e5b60b2114 ("mm: support batched checking of the young flag for

MGLRU")

With that,

Reviewed-by Baolin Wang <[email protected]>

Hi Baolin, Hui,

I am not convinced this is the correct patch. test_and_clear_young_ptes_notify() only indicates that there is at least one young PTE among the nr PTEs; it does not mean that all of the PTEs are young.

Am I missing something?

bool contpte_test_and_clear_young_ptes(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep, unsigned int nr)

{
         ...
         unsigned long end = addr + nr * PAGE_SIZE;
         bool young = false;

         ptep = contpte_align_addr_ptep(&addr, &end, ptep, nr);
         for (; addr != end; ptep++, addr += PAGE_SIZE)
                 young |= __ptep_test_and_clear_young(vma, addr, ptep);

         return young;
}

EXPORT_SYMBOL_GPL(contpte_test_and_clear_young_ptes);

Quoted 4 lines

Quoted 4 lines

   mm/vmscan.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)

Quoted 14 lines

mm/vmscan.c bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) · line 4198

Quoted 13 lines

mm/vmscan.c +1 -1

index bc324e37c5f1..264017850a55 100644
@@ -4192,7 +4192,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
       unsigned long end;
       struct lru_gen_mm_walk *walk;
       struct folio *last = NULL;
-     int young = 1;
+     int young = nr;
       pte_t *pte = pvmw->pte;
       unsigned long addr = pvmw->address;
       struct vm_area_struct *vma = pvmw->vma;

Thanks
Barry

Quoted 35 lines · Barry Song

Quoted 25 lines · Baolin Wang

Quoted 15 lines · Hui Zhu

From: Hui Zhu <[email protected]>

In lru_gen_look_around(), the young counter tracks the number of young PTEs. The original folio's contribution is represented by the initial value of young: test_and_clear_young_ptes_notify() is called on it at function entry, and the function returns early if it is not young. In the subsequent loop, the original folio is skipped (its accessed bits were already cleared), so it is not double-counted.

However, young is initialized to 1 regardless of the folio size. When the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added to the bloom filter and reducing aging accuracy for mTHP workloads.

Quoted 5 lines

Initialize young to nr so the original folio is accounted the same way as other young folios in the loop (young += nr).

Signed-off-by Hui Zhu <[email protected]>

---

Quoted 7 lines

Good catch. Please also add the Fixes tag:

Fixes 56e5b60b2114 ("mm: support batched checking of the young flag for

MGLRU")

With that,

Reviewed-by Baolin Wang <[email protected]>

Quoted 7 lines

Hi Baolin, Hui,

I am not convinced this is the correct patch. test_and_clear_young_ptes_notify() only indicates that there is at least one young PTE among the nr PTEs; it does not mean that all of the PTEs are young.

Am I missing something?

You are right. But I explained why this is done in my original commit 56e5b60b2114:

"
Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:

1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.

2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn't seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.
"

Based on this, I think changing it to 'nr' is reasonable. For an accessed large folio, it's better to have the bloom filter rescan the PMD and keep it in memory instead of reclaiming it incorrectly.

Quoted 71 lines · Baolin Wang

Quoted 35 lines · Barry Song

Quoted 25 lines · Baolin Wang

Quoted 15 lines · Hui Zhu

From: Hui Zhu <[email protected]>

In lru_gen_look_around(), the young counter tracks the number of young PTEs. The original folio's contribution is represented by the initial value of young: test_and_clear_young_ptes_notify() is called on it at function entry, and the function returns early if it is not young. In the subsequent loop, the original folio is skipped (its accessed bits were already cleared), so it is not double-counted.

However, young is initialized to 1 regardless of the folio size. When the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added to the bloom filter and reducing aging accuracy for mTHP workloads.

Quoted 5 lines

Initialize young to nr so the original folio is accounted the same way as other young folios in the loop (young += nr).

Signed-off-by Hui Zhu <[email protected]>

---

Quoted 7 lines

Good catch. Please also add the Fixes tag:

Fixes 56e5b60b2114 ("mm: support batched checking of the young flag for

MGLRU")

With that,

Reviewed-by Baolin Wang <[email protected]>

Quoted 7 lines

Hi Baolin, Hui,

I am not convinced this is the correct patch. test_and_clear_young_ptes_notify() only indicates that there is at least one young PTE among the nr PTEs; it does not mean that all of the PTEs are young.

Am I missing something?

You are right. But I explained why this is done in my original commit 56e5b60b2114:

"
Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:

1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.

2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn't seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.
"

Quoted 4 lines

Based on this, I think changing it to 'nr' is reasonable. For an accessed large folio, it's better to have the bloom filter rescan the PMD and keep it in memory instead of reclaiming it incorrectly.

I am not sure if this is the best policy, but we don't seem to have a practical way to get the exact number of accessed PTEs, so this may be acceptable.

clarify that this is intentional?

" However, young is initialized to 1 regardless of the folio size. When
the original folio is a large folio with nr PTEs, its young count is underestimated by nr - 1. This inconsistency can cause suitable_to_scan() to return false, preventing the PMD from being added"

Its young count is not underestimated; we are intentionally overestimating it. Also, nr does not necessarily equal folio_nr_pages(), does it?