support batched checking of the young flag for MGLRU

PATCH Applied Baolin Wang 3 participants 15 messages Mar 6 – Mar 10

Subsystems (6)

Memory management MM core arm64 MGLRU +2 more subsystems

lore

Share & tools

lore

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #0 lore raw

v3 cover support batched checking of the young flag for MGLRU

This is a follow-up to the previous work [1], to support batched checking of the young flag for MGLRU.

Similarly, batched checking of young flag for large folios can improve performance during large-folio reclamation when MGLRU is enabled. I observed noticeable performance improvements (see patch 5) on an Arm64 machine that supports contiguous PTEs. All mm-selftests are passed.

Patch 1 - 3: cleanup patches.
Patch 4: add a new generic batched PTE helper: test_and_clear_young_ptes(). Patch 5: support batched young flag checking for MGLRU.
Patch 6: implement the Arm64 arch-specific test_and_clear_young_ptes().

[1] https://lore.kernel.org/all/[email protected]/

Changes from v2:
v2: https://lore.kernel.org/all/[email protected]/
- Update the commit message of patch 5 (per David).
- Fix some coding style issues (per David).
- Remove 'cur_pte' variable in lru_gen_look_around() (per David).
- Move 'ptes += nr;' to the suitable place in folio_referenced_one() (per David).
- Add acked tag from David. Thanks.

Changes from v1:
v1: https://lore.kernel.org/all/[email protected]/
- Update some commit message (per David).
- Fix some coding style issues (per David).
- Use VM_WARN_ON_ONCE_FOLIO() instead of VM_WARN_ON_FOLIO() (per Rik).
- Define a generic ptep_test_and_clear_young_notify() (per David).
- Remove redundant 'nr > 1' check in folio_referenced_one() (per David).
- Rename some variables' to make code more clear (per David).
- Add a new patch to rename some functions (per David).
- Update the young counters with the batched count for MGLRU.
- Collect reviewed tags from Rik, Barry, Alistair and David. Thanks.

Baolin Wang (6):
mm: use inline helper functions instead of ugly macros
mm: rename ptep/pmdp_clear_young_notify() to
ptep/pmdp_test_and_clear_young_notify()
mm: rmap: add a ZONE_DEVICE folio warning in folio_referenced()
mm: add a batched helper to clear the young flag for large folios
mm: support batched checking of the young flag for MGLRU
arm64: mm: implement the architecture-specific
test_and_clear_young_ptes()

 arch/arm64/include/asm/pgtable.h | 18 +++++++----
 include/linux/mmu_notifier.h     | 54 --------------------------------
 include/linux/mmzone.h           |  5 +--
 include/linux/pgtable.h          | 37 ++++++++++++++++++++++
 mm/internal.h                    | 52 ++++++++++++++++++++++++++++++
 mm/rmap.c                        | 29 ++++++++---------
 mm/vmscan.c                      | 45 +++++++++++++++++++-------
 7 files changed, 152 insertions(+), 88 deletions(-)

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #1 lore raw

PATCH v3 1/6 mm: use inline helper functions instead of ugly macros

People have already complained that these *_clear_young_notify() related macros are very ugly, so let's use inline helpers to make them more readable.

In addition, we cannot implement these inline helper functions in the mmu_notifier.h file, because some arch-specific files will include the mmu_notifier.h, which introduces header compilation dependencies and causes build errors (e.g., arch/arm64/include/asm/tlbflush.h). Moreover, since these functions are only used in the mm, implementing these inline helpers in the mm/internal.h header seems reasonable.

Reviewed-by Rik van Riel <[email protected]>

Reviewed-by Barry Song <[email protected]>

Acked-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---
 include/linux/mmu_notifier.h | 54 ------------------------------------
 mm/internal.h                | 52 ++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 54 deletions(-)

include/linux/mmu_notifier.h +0 ~~-54~~

index 8450e18a87c2..3705d350c863 100644
@@ -516,55 +516,6 @@ static inline void mmu_notifier_range_init_owner(
 	range->owner = owner;
 }
 
-#define clear_flush_young_ptes_notify(__vma, __address, __ptep, __nr)	\
-({									\
-	int __young;							\
-	struct vm_area_struct *___vma = __vma;				\
-	unsigned long ___address = __address;				\
-	unsigned int ___nr = __nr;					\
-	__young = clear_flush_young_ptes(___vma, ___address, __ptep, ___nr);	\
-	__young |= mmu_notifier_clear_flush_young(___vma->vm_mm,	\
-						  ___address,		\
-						  ___address +		\
-						  ___nr * PAGE_SIZE);	\
-	__young;							\
-})
-
-#define pmdp_clear_flush_young_notify(__vma, __address, __pmdp)		\
-({									\
-	int __young;							\
-	struct vm_area_struct *___vma = __vma;				\
-	unsigned long ___address = __address;				\
-	__young = pmdp_clear_flush_young(___vma, ___address, __pmdp);	\
-	__young |= mmu_notifier_clear_flush_young(___vma->vm_mm,	\
-						  ___address,		\
-						  ___address +		\
-							PMD_SIZE);	\
-	__young;							\
-})
-
-#define ptep_clear_young_notify(__vma, __address, __ptep)		\
-({									\
-	int __young;							\
-	struct vm_area_struct *___vma = __vma;				\
-	unsigned long ___address = __address;				\
-	__young = ptep_test_and_clear_young(___vma, ___address, __ptep);\
-	__young |= mmu_notifier_clear_young(___vma->vm_mm, ___address,	\
-					    ___address + PAGE_SIZE);	\
-	__young;							\
-})
-
-#define pmdp_clear_young_notify(__vma, __address, __pmdp)		\
-({									\
-	int __young;							\
-	struct vm_area_struct *___vma = __vma;				\
-	unsigned long ___address = __address;				\
-	__young = pmdp_test_and_clear_young(___vma, ___address, __pmdp);\
-	__young |= mmu_notifier_clear_young(___vma->vm_mm, ___address,	\
-					    ___address + PMD_SIZE);	\
-	__young;							\
-})
-
 #else /* CONFIG_MMU_NOTIFIER */
 
 struct mmu_notifier_range {

@@ -652,11 +603,6 @@ static inline void mmu_notifier_subscriptions_destroy(struct mm_struct *mm)
 
 #define mmu_notifier_range_update_to_read_only(r) false
 
-#define clear_flush_young_ptes_notify clear_flush_young_ptes
-#define pmdp_clear_flush_young_notify pmdp_clear_flush_young
-#define ptep_clear_young_notify ptep_test_and_clear_young
-#define pmdp_clear_young_notify pmdp_test_and_clear_young
-
 static inline void mmu_notifier_synchronize(void)
 {
 }

mm/internal.h +52 -0

62 lines not on this page · raw message

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #5 lore raw

PATCH v3 5/6 mm: support batched checking of the young flag for MGLRU

In reply to Baolin Wang

Use the batched helper test_and_clear_young_ptes_notify() to check and clear the young flag to improve the performance during large folio reclamation when MGLRU is enabled.

Meanwhile, we can also support batched checking the young and dirty flag when MGLRU walks the mm's pagetable to update the folios' generation counter. Since MGLRU also checks the PTE dirty bit, use folio_pte_batch_flags() with FPB_MERGE_YOUNG_DIRTY set to detect batches of PTEs for a large folio.

Then we can remove the ptep_test_and_clear_young_notify() since it has no users now.

Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:

1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.

2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn’t seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.

Reviewed-by Rik van Riel <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---
 include/linux/mmzone.h |  5 +++--
 mm/internal.h          |  6 ------
 mm/rmap.c              | 28 +++++++++++++--------------
 mm/vmscan.c            | 43 +++++++++++++++++++++++++++++++-----------
 4 files changed, 49 insertions(+), 33 deletions(-)

include/linux/mmzone.h +3 -2

index b53b95abe287..7bd0134c241c 100644
@@ -684,7 +684,7 @@ struct lru_gen_memcg {
 
 void lru_gen_init_pgdat(struct pglist_data *pgdat);
 void lru_gen_init_lruvec(struct lruvec *lruvec);
-bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
+bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr);
 
 void lru_gen_init_memcg(struct mem_cgroup *memcg);
 void lru_gen_exit_memcg(struct mem_cgroup *memcg);

@@ -706,7 +706,8 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec)
 {
 }
 
-static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
+static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw,
+		unsigned int nr)
 {
 	return false;
 }

mm/internal.h +0 -6

index 8cdd5d8e43fb..95b583e7e4f7 100644
@@ -1848,10 +1848,4 @@ static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,
 
 #endif /* CONFIG_MMU_NOTIFIER */
 
-static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
-		unsigned long addr, pte_t *ptep)
-{
-	return test_and_clear_young_ptes_notify(vma, addr, ptep, 1);
-}
-
 #endif	/* __MM_INTERNAL_H */

mm/rmap.c +14 ~~-14~~

index 2d94b3ba52da..6398d7eef393 100644
@@ -958,25 +958,20 @@ static bool folio_referenced_one(struct folio *folio,
 			return false;
 		}
 
+		if (pvmw.pte && folio_test_large(folio)) {
+			const unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
+			const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
+			pte_t pteval = ptep_get(pvmw.pte);
+
+			nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr);
+		}
+
 		if (lru_gen_enabled() && pvmw.pte) {
-			if (lru_gen_look_around(&pvmw))
+			if (lru_gen_look_around(&pvmw, nr))
 				referenced++;
 		} else if (pvmw.pte) {
-			if (folio_test_large(folio)) {
-				unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
-				unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
-				pte_t pteval = ptep_get(pvmw.pte);
-
-				nr = folio_pte_batch(folio, pvmw.pte,
-						     pteval, max_nr);
-			}
-
-			ptes += nr;
 			if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr))
 				referenced++;
-			/* Skip the batched PTEs */
-			pvmw.pte += nr - 1;
-			pvmw.address += (nr - 1) * PAGE_SIZE;
 		} else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
 			if (pmdp_clear_flush_young_notify(vma, address,
 						pvmw.pmd))

@@ -986,6 +981,7 @@ static bool folio_referenced_one(struct folio *folio,
 			WARN_ON_ONCE(1);
 		}
 
+		ptes += nr;
 		pra->mapcount -= nr;
 		/*
 		 * If we are sure that we batched the entire folio,

@@ -995,6 +991,10 @@ static bool folio_referenced_one(struct folio *folio,
 			page_vma_mapped_walk_done(&pvmw);
 			break;
 		}
+
+		/* Skip the batched PTEs */
+		pvmw.pte += nr - 1;
+		pvmw.address += (nr - 1) * PAGE_SIZE;
 	}
 
 	if (referenced)

mm/vmscan.c +32 ~~-11~~

102 lines not on this page · raw message

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #2 lore raw

PATCH v3 2/6 mm: rename ptep/pmdp_clear_young_notify() to ptep/pmdp_test_and_clear_young_notify()

In reply to Baolin Wang

Rename ptep/pmdp_clear_young_notify() to ptep/pmdp_test_and_clear_young_notify() to make the function names consistent.

Acked-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---
 mm/internal.h | 8 ++++----
 mm/vmscan.c   | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

mm/internal.h +4 -4

index 05eb0303f277..f45f97df0d28 100644
@@ -1819,7 +1819,7 @@ static inline int pmdp_clear_flush_young_notify(struct vm_area_struct *vma,
 	return young;
 }
 
-static inline int ptep_clear_young_notify(struct vm_area_struct *vma,
+static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
 		unsigned long addr, pte_t *ptep)
 {
 	int young;

@@ -1829,7 +1829,7 @@ static inline int ptep_clear_young_notify(struct vm_area_struct *vma,
 	return young;
 }
 
-static inline int pmdp_clear_young_notify(struct vm_area_struct *vma,
+static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,
 		unsigned long addr, pmd_t *pmdp)
 {
 	int young;

@@ -1843,8 +1843,8 @@ static inline int pmdp_clear_young_notify(struct vm_area_struct *vma,
 
 #define clear_flush_young_ptes_notify	clear_flush_young_ptes
 #define pmdp_clear_flush_young_notify	pmdp_clear_flush_young
-#define ptep_clear_young_notify	ptep_test_and_clear_young
-#define pmdp_clear_young_notify	pmdp_test_and_clear_young
+#define ptep_test_and_clear_young_notify	ptep_test_and_clear_young
+#define pmdp_test_and_clear_young_notify	pmdp_test_and_clear_young
 
 #endif /* CONFIG_MMU_NOTIFIER */

mm/vmscan.c +4 -4

index 52adb37d1b01..e3425b4db755 100644
@@ -3504,7 +3504,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 		if (!folio)
 			continue;
 
-		if (!ptep_clear_young_notify(args->vma, addr, pte + i))
+		if (!ptep_test_and_clear_young_notify(args->vma, addr, pte + i))
 			continue;
 
 		if (last != folio) {

@@ -3595,7 +3595,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area
 		if (!folio)
 			goto next;
 
-		if (!pmdp_clear_young_notify(vma, addr, pmd + i))
+		if (!pmdp_test_and_clear_young_notify(vma, addr, pmd + i))
 			goto next;
 
 		if (last != folio) {

@@ -4185,7 +4185,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
 
-	if (!ptep_clear_young_notify(vma, addr, pte))
+	if (!ptep_test_and_clear_young_notify(vma, addr, pte))
 		return false;
 
 	if (spin_is_contended(pvmw->ptl))

@@ -4237,7 +4237,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 		if (!folio)
 			continue;
 
-		if (!ptep_clear_young_notify(vma, addr, pte + i))
+		if (!ptep_test_and_clear_young_notify(vma, addr, pte + i))
 			continue;
 
 		if (last != folio) {

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #3 lore raw

PATCH v3 3/6 mm: rmap: add a ZONE_DEVICE folio warning in folio_referenced()

In reply to Baolin Wang

The folio_referenced() is used to test whether a folio was referenced during reclaim. Moreover, ZONE_DEVICE folios are controlled by their device driver, have a lifetime tied to that driver, and are never placed on the LRU list. That means we should never try to reclaim ZONE_DEVICE folios, so add a warning to catch this unexpected behavior in folio_referenced() to avoid confusion, as discussed in the previous thread[1].

[1] https://lore.kernel.org/all/[email protected]/

Reviewed-by Alistair Popple <[email protected]>

Acked-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---
 mm/rmap.c | 1 +
 1 file changed, 1 insertion(+)

mm/rmap.c +1 -0

index 603186ff4ba5..2d94b3ba52da 100644
@@ -1065,6 +1065,7 @@ int folio_referenced(struct folio *folio, int is_locked,
 		.invalid_vma = invalid_folio_referenced_vma,
 	};
 
+	VM_WARN_ON_ONCE_FOLIO(folio_is_zone_device(folio), folio);
 	*vm_flags = 0;
 	if (!pra.mapcount)
 		return 0;

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #4 lore raw

PATCH v3 4/6 mm: add a batched helper to clear the young flag for large folios

In reply to Baolin Wang

Currently, MGLRU will call ptep_test_and_clear_young_notify() to check and clear the young flag for each PTE sequentially, which is inefficient for large folios reclamation.

Moreover, on Arm64 architecture, which supports contiguous PTEs, the Arm64- specific ptep_test_and_clear_young() already implements an optimization to clear the young flags for PTEs within a contiguous range. However, this is not sufficient. Similar to the Arm64 specific clear_flush_young_ptes(), we can extend this to perform batched operations for the entire large folio (which might exceed the contiguous range: CONT_PTE_SIZE).

Thus, we can introduce a new batched helper: test_and_clear_young_ptes() and its wrapper test_and_clear_young_ptes_notify() which are consistent with the existing functions, to perform batched checking of the young flags for large folios, which can help improve performance during large folio reclamation when MGLRU is enabled. And it will be overridden by the architecture that implements a more efficient batch operation in the following patches.

Signed-off-by Baolin Wang <[email protected]>

---
 include/linux/pgtable.h | 37 +++++++++++++++++++++++++++++++++++++
 mm/internal.h           | 16 +++++++++++-----
 2 files changed, 48 insertions(+), 5 deletions(-)

include/linux/pgtable.h +37 -0

index d2767a4c027b..17d961c612fc 100644
@@ -1103,6 +1103,43 @@ static inline int clear_flush_young_ptes(struct vm_area_struct *vma,
 }
 #endif
 
+#ifndef test_and_clear_young_ptes
+/**
+ * test_and_clear_young_ptes - Mark PTEs that map consecutive pages of the same
+ *			       folio as old
+ * @vma: The virtual memory area the pages are mapped into.
+ * @addr: Address the first page is mapped at.
+ * @ptep: Page table pointer for the first entry.
+ * @nr: Number of entries to clear access bit.
+ *
+ * May be overridden by the architecture; otherwise, implemented as a simple
+ * loop over ptep_test_and_clear_young().
+ *
+ * Note that PTE bits in the PTE range besides the PFN can differ. For example,
+ * some PTEs might be write-protected.
+ *
+ * Context: The caller holds the page table lock.  The PTEs map consecutive
+ * pages that belong to the same folio.  The PTEs are all in the same PMD.
+ *
+ * Returns: whether any PTE was young.
+ */
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep, unsigned int nr)
+{
+	int young = 0;
+
+	for (;;) {
+		young |= ptep_test_and_clear_young(vma, addr, ptep);
+		if (--nr == 0)
+			break;
+		ptep++;
+		addr += PAGE_SIZE;
+	}
+
+	return young;
+}
+#endif
+
 /*
  * On some architectures hardware does not set page access bit when accessing
  * memory page, it is responsibility of software setting this bit. It brings

mm/internal.h +11 -5

index f45f97df0d28..8cdd5d8e43fb 100644
@@ -1819,13 +1819,13 @@ static inline int pmdp_clear_flush_young_notify(struct vm_area_struct *vma,
 	return young;
 }
 
-static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
-		unsigned long addr, pte_t *ptep)
+static inline int test_and_clear_young_ptes_notify(struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep, unsigned int nr)
 {
 	int young;
 
-	young = ptep_test_and_clear_young(vma, addr, ptep);
-	young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE);
+	young = test_and_clear_young_ptes(vma, addr, ptep, nr);
+	young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + nr * PAGE_SIZE);
 	return young;
 }

@@ -1843,9 +1843,15 @@ static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,
 
 #define clear_flush_young_ptes_notify	clear_flush_young_ptes
 #define pmdp_clear_flush_young_notify	pmdp_clear_flush_young
-#define ptep_test_and_clear_young_notify	ptep_test_and_clear_young
+#define test_and_clear_young_ptes_notify	test_and_clear_young_ptes
 #define pmdp_test_and_clear_young_notify	pmdp_test_and_clear_young
 
 #endif /* CONFIG_MMU_NOTIFIER */
 
+static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep)
+{
+	return test_and_clear_young_ptes_notify(vma, addr, ptep, 1);
+}
+
 #endif	/* __MM_INTERNAL_H */

Signature · 1 line

2.47.3

Baolin Wang

author reviewer

2026-03-06 06:44 UTC #7 lore raw

PATCH v3 6/6 arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to Baolin Wang

Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable batched checking of young flags, improving performance during large folio reclamation when MGLRU is enabled.

While we're at it, simplify ptep_test_and_clear_young() by calling test_and_clear_young_ptes(). Since callers guarantee that PTEs are present before calling these functions, we can use pte_cont() to check the CONT_PTE flag instead of pte_valid_cont().

Performance testing:
Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface. I can observe 60%+ performance improvement on my Arm64 32-core server (and about 15% improvement on my X86 machine).

W/o patchset:
real 0m0.470s
user 0m0.000s
sys 0m0.470s

W/ patchset:
real 0m0.180s
user 0m0.001s
sys 0m0.179s

Reviewed-by Rik van Riel <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---
 arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

arch/arm64/include/asm/pgtable.h +12 -6

index aa4b13da6371..ab451d20e4c5 100644
@@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 	return __ptep_get_and_clear(mm, addr, ptep);
 }
 
+#define test_and_clear_young_ptes test_and_clear_young_ptes
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep,
+					    unsigned int nr)
+{
+	if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
+		return __ptep_test_and_clear_young(vma, addr, ptep);
+
+	return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
+}
+
 #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
 static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,
 				unsigned long addr, pte_t *ptep)
 {
-	pte_t orig_pte = __ptep_get(ptep);
-
-	if (likely(!pte_valid_cont(orig_pte)))
-		return __ptep_test_and_clear_young(vma, addr, ptep);
-
-	return contpte_test_and_clear_young_ptes(vma, addr, ptep, 1);
+	return test_and_clear_young_ptes(vma, addr, ptep, 1);
 }
 
 #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH

Signature · 1 line

2.47.3

David Hildenbrand (Arm)

maintainer Acked-by on v3 5/6

2026-03-06 14:44 UTC #6 lore raw

Re: [PATCH v3 5/6] mm: support batched checking of the young flag for MGLRU

In reply to Baolin Wang

Quoted 29 lines · Baolin Wang

Use the batched helper test_and_clear_young_ptes_notify() to check and clear the young flag to improve the performance during large folio reclamation when MGLRU is enabled.
Meanwhile, we can also support batched checking the young and dirty flag when MGLRU walks the mm's pagetable to update the folios' generation counter. Since MGLRU also checks the PTE dirty bit, use folio_pte_batch_flags() with FPB_MERGE_YOUNG_DIRTY set to detect batches of PTEs for a large folio.
Then we can remove the ptep_test_and_clear_young_notify() since it has no users now.
Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:
1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.
2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn’t seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.

Quoted 3 lines

Reviewed-by Rik van Riel <[email protected]>
Signed-off-by Baolin Wang <[email protected]>
---

Thanks!

Acked-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David

David Hildenbrand (Arm)

maintainer

2026-03-06 14:47 UTC #8 lore raw

Re: [PATCH v3 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to Baolin Wang

Quoted 39 lines · Baolin Wang

Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable batched checking of young flags, improving performance during large folio reclamation when MGLRU is enabled.
While we're at it, simplify ptep_test_and_clear_young() by calling test_and_clear_young_ptes(). Since callers guarantee that PTEs are present before calling these functions, we can use pte_cont() to check the CONT_PTE flag instead of pte_valid_cont().
Performance testing:
Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface. I can observe 60%+ performance improvement on my Arm64 32-core server (and about 15% improvement on my X86 machine).
W/o patchset:
real 0m0.470s
user 0m0.000s
sys 0m0.470s
W/ patchset:
real 0m0.180s
user 0m0.001s
sys 0m0.179s
Reviewed-by Rik van Riel <[email protected]>
Signed-off-by Baolin Wang <[email protected]>
---
 arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
arch/arm64/include/asm/pgtable.h +0 -0
index aa4b13da6371..ab451d20e4c5 100644
@@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 	return __ptep_get_and_clear(mm, addr, ptep);
 }
 

arch/arm64/include/asm/pgtable.h static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · line 1824

Quoted 10 lines

arch/arm64/include/asm/pgtable.h +10 -0

+#define test_and_clear_young_ptes test_and_clear_young_ptes
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep,
+					    unsigned int nr)
+{
+	if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
+		return __ptep_test_and_clear_young(vma, addr, ptep);
+
+	return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
+}

Thinking out loud, what would happen if

(a) The range spans multiple possible cont ranges (like, 64 ptes).

(b) The first pte is !pte_cont(), but some others in there are?

Signature · 3 lines

Cheers,

David

Andrew Morton

maintainer

2026-03-06 23:20 UTC #13 lore raw

In reply to Baolin Wang

Quoted 7 lines · On Fri, 6 Mar 2026 14:43:36 +0800 Baolin Wang <[email protected]>…

This is a follow-up to the previous work [1], to support batched checking of the young flag for MGLRU.
Similarly, batched checking of young flag for large folios can improve performance during large-folio reclamation when MGLRU is enabled. I observed noticeable performance improvements (see patch 5) on an Arm64 machine that supports contiguous PTEs. All mm-selftests are passed.

Thanks, I updated mm-new with this.

Quoted 8 lines

Changes from v2:
v2: https://lore.kernel.org/all/[email protected]/
- Update the commit message of patch 5 (per David).
- Fix some coding style issues (per David).
- Remove 'cur_pte' variable in lru_gen_look_around() (per David).
- Move 'ptes += nr;' to the suitable place in folio_referenced_one() (per David).
- Add acked tag from David. Thanks.

Here's how v3 altered mm.git:

 include/linux/pgtable.h |    3 +--
 mm/rmap.c               |    2 +-
 mm/vmscan.c             |   11 +++++------
 3 files changed, 7 insertions(+), 9 deletions(-)

include/linux/pgtable.h renamed from include/linux/pgtable.h~b +1 -2

@@ -1124,8 +1124,7 @@ static inline int clear_flush_young_ptes
  * Returns: whether any PTE was young.
  */
 static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
-					    unsigned long addr, pte_t *ptep,
-					    unsigned int nr)
+		unsigned long addr, pte_t *ptep, unsigned int nr)
 {
 	int young = 0;

mm/rmap.c renamed from mm/rmap.c~b +1 -1

@@ -964,7 +964,6 @@ static bool folio_referenced_one(struct
 			pte_t pteval = ptep_get(pvmw.pte);
 
 			nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr);
-			ptes += nr;
 		}
 
 		if (lru_gen_enabled() && pvmw.pte) {

@@ -982,6 +981,7 @@ static bool folio_referenced_one(struct
 			WARN_ON_ONCE(1);
 		}
 
+		ptes += nr;
 		pra->mapcount -= nr;
 		/*
 		 * If we are sure that we batched the entire folio,

mm/vmscan.c renamed from mm/vmscan.c~b +5 -6

@@ -4201,7 +4201,6 @@ bool lru_gen_look_around(struct page_vma
 	struct lruvec *lruvec;
 	struct lru_gen_mm_state *mm_state;
 	unsigned long max_seq;
-	pte_t *cur_pte;
 	int gen;
 
 	lockdep_assert_held(pvmw->ptl);

@@ -4247,10 +4246,10 @@ bool lru_gen_look_around(struct page_vma
 
 	pte -= (addr - start) / PAGE_SIZE;
 
-	for (i = 0, addr = start, cur_pte = pte; addr != end;
-	     i += nr, cur_pte += nr, addr += nr * PAGE_SIZE) {
+	for (i = 0, addr = start; addr != end;
+	     i += nr, pte += nr, addr += nr * PAGE_SIZE) {
 		unsigned long pfn;
-		pte_t ptent = ptep_get(cur_pte);
+		pte_t ptent = ptep_get(pte);
 
 		nr = 1;
 		pfn = get_pte_pfn(ptent, vma, addr, pgdat);

@@ -4264,11 +4263,11 @@ bool lru_gen_look_around(struct page_vma
 		if (folio_test_large(folio)) {
 			const unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
 
-			nr = folio_pte_batch_flags(folio, NULL, cur_pte, &ptent,
+			nr = folio_pte_batch_flags(folio, NULL, pte, &ptent,
 						   max_nr, FPB_MERGE_YOUNG_DIRTY);
 		}
 
-		if (!test_and_clear_young_ptes_notify(vma, addr, cur_pte, nr))
+		if (!test_and_clear_young_ptes_notify(vma, addr, pte, nr))
 			continue;
 
 		if (last != folio) {

Baolin Wang

author reviewer

2026-03-07 01:28 UTC #9 lore raw

Re: [PATCH v3 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to David Hildenbrand (Arm)

Quoted 40 lines · David Hildenbrand (Arm)

Quoted 39 lines · Baolin Wang
Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable batched checking of young flags, improving performance during large folio reclamation when MGLRU is enabled.
While we're at it, simplify ptep_test_and_clear_young() by calling test_and_clear_young_ptes(). Since callers guarantee that PTEs are present before calling these functions, we can use pte_cont() to check the CONT_PTE flag instead of pte_valid_cont().
Performance testing:
Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface. I can observe 60%+ performance improvement on my Arm64 32-core server (and about 15% improvement on my X86 machine).
W/o patchset:
real 0m0.470s
user 0m0.000s
sys 0m0.470s
W/ patchset:
real 0m0.180s
user 0m0.001s
sys 0m0.179s
Reviewed-by Rik van Riel <[email protected]>
Signed-off-by Baolin Wang <[email protected]>
---
  arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
  1 file changed, 12 insertions(+), 6 deletions(-)
arch/arm64/include/asm/pgtable.h +0 -0
index aa4b13da6371..ab451d20e4c5 100644
@@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
  	return __ptep_get_and_clear(mm, addr, ptep);
  }
 

Quoted 12 lines

arch/arm64/include/asm/pgtable.h static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · line 1824

Quoted 10 lines

arch/arm64/include/asm/pgtable.h +10 -0

+#define test_and_clear_young_ptes test_and_clear_young_ptes
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep,
+					    unsigned int nr)
+{
+	if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
+		return __ptep_test_and_clear_young(vma, addr, ptep);
+
+	return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
+}

Thinking out loud, what would happen if

Good questions, I think the contpte_test_and_clear_young_ptes() takes that into account.

Quoted 1 line

(a) The range spans multiple possible cont ranges (like, 64 ptes).

The contpte_test_and_clear_young_ptes() will call contpte_align_addr_ptep() to align the range to cont‑block boundary, that means the range can span multiple cont blocks.

int contpte_test_and_clear_young_ptes(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep, unsigned int nr)

{
	unsigned long end = addr + nr * PAGE_SIZE;
	int young = 0;

	ptep = contpte_align_addr_ptep(&addr, &end, ptep, nr);
	for (; addr != end; ptep++, addr += PAGE_SIZE)
		young |= __ptep_test_and_clear_young(vma, addr, ptep);

	return young;
}

Quoted 2 lines

(b) The first pte is !pte_cont(), but some others in there are?

IMO they can’t be handled in a single batch. Since the folio_pte_batch() will group consecutive !cont PTEs into one batch and consecutive cont PTEs into another (assume all PTEs belong to a single large folio), because their PTE entries have different CONT bits.

Even if the callers do so, contpte_align_addr_ptep() will check the pte_cont() of the start and end address to align the range appropriately.

Baolin Wang

author reviewer

2026-03-07 01:29 UTC #14 lore raw

In reply to Andrew Morton

Quoted 21 lines · Andrew Morton

Quoted 7 lines · On Fri, 6 Mar 2026 14:43:36 +0800 Baolin Wang <[email protected]>…
This is a follow-up to the previous work [1], to support batched checking of the young flag for MGLRU.
Similarly, batched checking of young flag for large folios can improve performance during large-folio reclamation when MGLRU is enabled. I observed noticeable performance improvements (see patch 5) on an Arm64 machine that supports contiguous PTEs. All mm-selftests are passed.
Thanks, I updated mm-new with this.
Quoted 8 lines
Changes from v2:
v2: https://lore.kernel.org/all/[email protected]/
- Update the commit message of patch 5 (per David).
- Fix some coding style issues (per David).
- Remove 'cur_pte' variable in lru_gen_look_around() (per David).
- Move 'ptes += nr;' to the suitable place in folio_referenced_one() (per David).
- Add acked tag from David. Thanks.

Quoted 1 line

Here's how v3 altered mm.git:

Thanks Andrew. LGTM.

David Hildenbrand (Arm)

maintainer

2026-03-09 14:39 UTC #10 lore raw

Re: [PATCH v3 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to Baolin Wang

Quoted 7 lines

Quoted 2 lines
(b) The first pte is !pte_cont(), but some others in there are?
IMO they can’t be handled in a single batch. Since the folio_pte_batch() will group consecutive !cont PTEs into one batch and consecutive cont PTEs into another (assume all PTEs belong to a single large folio), because their PTE entries have different CONT bits.

Interesting, thanks. I thought that folio_pte_batch() would be able to batch that.

But yes, pte_batch_hint() relies on the CONT bit still being set after a ptep_get(). I wonder whether we should document somewhere that the arm implementation depends on that.

This might be something to look into in the future. (make folio_pte_batch() ignore cont information when comparing and make the arm implementation be able to deal with that).

Assume we unmapped the last page of a large folio. Ideally, we'd be able to process the remaining THP pieces (all ptes) in a single operation. I guess right now it would be two.

Signature · 3 lines

Cheers,

David

David Hildenbrand (Arm)

maintainer Reviewed-by on v3 6/6

2026-03-09 14:40 UTC #12 lore raw

Re: [PATCH v3 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to Baolin Wang

Quoted 20 lines · Baolin Wang

Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable batched checking of young flags, improving performance during large folio reclamation when MGLRU is enabled.
While we're at it, simplify ptep_test_and_clear_young() by calling test_and_clear_young_ptes(). Since callers guarantee that PTEs are present before calling these functions, we can use pte_cont() to check the CONT_PTE flag instead of pte_valid_cont().
Performance testing:
Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface. I can observe 60%+ performance improvement on my Arm64 32-core server (and about 15% improvement on my X86 machine).
W/o patchset:
real 0m0.470s
user 0m0.000s
sys 0m0.470s

Quoted 8 lines

W/ patchset:
real 0m0.180s
user 0m0.001s
sys 0m0.179s
Reviewed-by Rik van Riel <[email protected]>
Signed-off-by Baolin Wang <[email protected]>
---

Reviewed-by David Hildenbrand (Arm) <[email protected]>

Signature · 3 lines

Cheers,

David

Baolin Wang

author reviewer

2026-03-10 02:51 UTC #11 lore raw

Re: [PATCH v3 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes()

In reply to David Hildenbrand (Arm)

Quoted 14 lines · David Hildenbrand (Arm)

Quoted 7 lines
Quoted 2 lines
(b) The first pte is !pte_cont(), but some others in there are?
IMO they can’t be handled in a single batch. Since the folio_pte_batch() will group consecutive !cont PTEs into one batch and consecutive cont PTEs into another (assume all PTEs belong to a single large folio), because their PTE entries have different CONT bits.
Interesting, thanks. I thought that folio_pte_batch() would be able to batch that.
But yes, pte_batch_hint() relies on the CONT bit still being set after a ptep_get(). I wonder whether we should document somewhere that the arm implementation depends on that.

Quoted 7 lines

This might be something to look into in the future. (make folio_pte_batch() ignore cont information when comparing and make the arm implementation be able to deal with that).
Assume we unmapped the last page of a large folio. Ideally, we'd be able to process the remaining THP pieces (all ptes) in a single operation. I guess right now it would be two.

Right. Let me investigate this further and see if I can figure out how to optimize this case.

Thanks for reviewing.