support batched checking of the young flag for MGLRU

Rename ptep/pmdp_clear_young_notify() to ptep/pmdp_test_and_clear_young_notify() to make the function names consistent.

Suggested-by David Hildenbrand (Arm) <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

Currently, MGLRU will call ptep_test_and_clear_young_notify() to check and clear the young flag for each PTE sequentially, which is inefficient for large folios reclamation.

Moreover, on Arm64 architecture, which supports contiguous PTEs, the Arm64- specific ptep_test_and_clear_young() already implements an optimization to clear the young flags for PTEs within a contiguous range. However, this is not sufficient. Similar to the Arm64 specific clear_flush_young_ptes(), we can extend this to perform batched operations for the entire large folio (which might exceed the contiguous range: CONT_PTE_SIZE).

Thus, we can introduce a new batched helper: test_and_clear_young_ptes() and its wrapper test_and_clear_young_ptes_notify() which are consistent with the existing functions, to perform batched checking of the young flags for large folios, which can help improve performance during large folio reclamation when MGLRU is enabled. And it will be overridden by the architecture that implements a more efficient batch operation in the following patches.

Signed-off-by Baolin Wang <[email protected]>

---
 include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++
 mm/internal.h           | 16 +++++++++++-----
 2 files changed, 49 insertions(+), 5 deletions(-)

include/linux/pgtable.h +13 -0

index 776993d4567b..29bd9fd04e1e 100644
@@ -1103,6 +1103,44 @@ static inline int clear_flush_young_ptes(struct vm_area_struct *vma,
 }
 #endif
 
+#ifndef test_and_clear_young_ptes
+/**
+ * test_and_clear_young_ptes - Mark PTEs that map consecutive pages of the same
+ *			       folio as old
+ * @vma: The virtual memory area the pages are mapped into.
+ * @addr: Address the first page is mapped at.
+ * @ptep: Page table pointer for the first entry.
+ * @nr: Number of entries to clear access bit.
+ *
+ * May be overridden by the architecture; otherwise, implemented as a simple
+ * loop over ptep_test_and_clear_young().
+ *
+ * Note that PTE bits in the PTE range besides the PFN can differ. For example,

include/linux/pgtable.h +14 -0

+{
+	int young = 0;
+
+	for (;;) {
+		young |= ptep_test_and_clear_young(vma, addr, ptep);
+		if (--nr == 0)
+			break;
+		ptep++;
+		addr += PAGE_SIZE;
+	}
+
+	return young;
+}
+#endif

Use the batched helper test_and_clear_young_ptes_notify() to check and clear the young flag to improve the performance during large folio reclamation when MGLRU is enabled.

Meanwhile, we can also support batched checking the young and dirty flag when MGLRU walks the mm's pagetable to update the folios' generation counter. Since MGLRU also checks the PTE dirty bit, use folio_pte_batch_flags() with FPB_MERGE_YOUNG_DIRTY set to detect batches of PTEs for a large folio.

Then we can remove the ptep_test_and_clear_young_notify() since it has no users now.

Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:

1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.

2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn’t seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.

Reviewed-by Rik van Riel <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---

index a5f0a264ad56..a1b3967afe41 100644

mm/internal.h +0 -6

@@ -1843,10 +1843,4 @@ static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,
 
 #endif /* CONFIG_MMU_NOTIFIER */
 
-static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
-		unsigned long addr, pte_t *ptep)
-{
-	return test_and_clear_young_ptes_notify(vma, addr, ptep, 1);
-}
-
 #endif	/* __MM_INTERNAL_H */

mm/rmap.c +0 -0

index 11cc6171344f..beb423f3e8ec 100644

mm/rmap.c +7 -14

+		}
+
 		if (lru_gen_enabled() && pvmw.pte) {
-			if (lru_gen_look_around(&pvmw))
+			if (lru_gen_look_around(&pvmw, nr))
 				referenced++;
 		} else if (pvmw.pte) {
-			if (folio_test_large(folio)) {
-				unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
-				unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
-				pte_t pteval = ptep_get(pvmw.pte);
-
-				nr = folio_pte_batch(folio, pvmw.pte,
-						     pteval, max_nr);
-			}
-
-			ptes += nr;
 			if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr))
 				referenced++;
-			/* Skip the batched PTEs */
-			pvmw.pte += nr - 1;
-			pvmw.address += (nr - 1) * PAGE_SIZE;
 		} else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
 			if (pmdp_clear_flush_young_notify(vma, address,
 						pvmw.pmd))

@@ -995,6 +991,10 @@ static bool folio_referenced_one(struct folio *folio,
 			page_vma_mapped_walk_done(&pvmw);
 			break;
 		}
+
+		/* Skip the batched PTEs */
+		pvmw.pte += nr - 1;
+		pvmw.address += (nr - 1) * PAGE_SIZE;
 	}
 
 	if (referenced)

mm/vmscan.c +1 -0

index 0a5622420987..7457b3c06fa3 100644
@@ -3474,6 +3474,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
 	DEFINE_MAX_SEQ(walk->lruvec);
 	int gen = lru_gen_from_seq(max_seq);
+	unsigned int nr;
 	pmd_t pmdval;
 
 	pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl);

@@ -3492,11 +3493,13 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 
 	lazy_mmu_mode_enable();

mm/vmscan.c +3 -2

 	walk_update_folio(walk, last, gen, dirty);

@@ -4166,7 +4178,7 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
  * the PTE table to the Bloom filter. This forms a feedback loop between the
  * eviction and the aging.
  */
-bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
+bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
 {
 	int i;
 	bool dirty;

@@ -4184,12 +4196,13 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	struct lruvec *lruvec;
 	struct lru_gen_mm_state *mm_state;
 	unsigned long max_seq;
+	pte_t *cur_pte;
 	int gen;
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
 
-	if (!ptep_test_and_clear_young_notify(vma, addr, pte))
+	if (!test_and_clear_young_ptes_notify(vma, addr, pte, nr))
 		return false;
 
 	if (spin_is_contended(pvmw->ptl))

@@ -4229,10 +4242,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 

Quoted 49 lines · Baolin Wang

Currently, MGLRU will call ptep_test_and_clear_young_notify() to check and clear the young flag for each PTE sequentially, which is inefficient for large folios reclamation.

Moreover, on Arm64 architecture, which supports contiguous PTEs, the Arm64- specific ptep_test_and_clear_young() already implements an optimization to clear the young flags for PTEs within a contiguous range. However, this is not sufficient. Similar to the Arm64 specific clear_flush_young_ptes(), we can extend this to perform batched operations for the entire large folio (which might exceed the contiguous range: CONT_PTE_SIZE).

Thus, we can introduce a new batched helper: test_and_clear_young_ptes() and its wrapper test_and_clear_young_ptes_notify() which are consistent with the existing functions, to perform batched checking of the young flags for large folios, which can help improve performance during large folio reclamation when MGLRU is enabled. And it will be overridden by the architecture that implements a more efficient batch operation in the following patches.

Signed-off-by Baolin Wang <[email protected]>

---
  include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++
  mm/internal.h           | 16 +++++++++++-----
  2 files changed, 49 insertions(+), 5 deletions(-)

include/linux/pgtable.h +17 -0

index 776993d4567b..29bd9fd04e1e 100644
@@ -1103,6 +1103,44 @@ static inline int clear_flush_young_ptes(struct vm_area_struct *vma,
  }
  #endif
 
+#ifndef test_and_clear_young_ptes
+/**
+ * test_and_clear_young_ptes - Mark PTEs that map consecutive pages of the same
+ *			       folio as old
+ * @vma: The virtual memory area the pages are mapped into.
+ * @addr: Address the first page is mapped at.
+ * @ptep: Page table pointer for the first entry.
+ * @nr: Number of entries to clear access bit.
+ *
+ * May be overridden by the architecture; otherwise, implemented as a simple
+ * loop over ptep_test_and_clear_young().
+ *
+ * Note that PTE bits in the PTE range besides the PFN can differ. For example,
+ * some PTEs might be write-protected.
+ *
+ * Context: The caller holds the page table lock.  The PTEs map consecutive
+ * pages that belong to the same folio.  The PTEs are all in the same PMD.

include/linux/pgtable.h static inline int clear_flush_young_ptes(struct vm_area_struct *vma, · line 1128

Quoted 6 lines

include/linux/pgtable.h +6 -0

+ *
+ * Returns: whether any PTE was young.
+ */
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep,
+					    unsigned int nr)

Two tabs ...

What happened to using a boolen as return type and for "int young"?

Quoted 23 lines · Baolin Wang

Use the batched helper test_and_clear_young_ptes_notify() to check and clear the young flag to improve the performance during large folio reclamation when MGLRU is enabled.

Meanwhile, we can also support batched checking the young and dirty flag when MGLRU walks the mm's pagetable to update the folios' generation counter. Since MGLRU also checks the PTE dirty bit, use folio_pte_batch_flags() with FPB_MERGE_YOUNG_DIRTY set to detect batches of PTEs for a large folio.

Then we can remove the ptep_test_and_clear_young_notify() since it has no users now.

Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' counter with the batched count in the lru_gen_look_around() and walk_pte_range(). However, the batched operations may inflate these two counters, because in a large folio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have been accessed within a large folio is not very meaningful, since the mm core actually tracks access/dirty on a per-folio basis, not per page). The impact analysis is as follows:

1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is mainly for debugging.

Quoted 5 lines

2. The 'young' counter is used to decide whether to place the current PMD entry into the bloom filters by suitable_to_scan() (so that next time we can check whether it has been accessed again), which may set the hash bit in the bloom filters for a PMD entry that hasn’t seen much access. However, bloom filters inherently allow some error, so this effect appears negligible.

Doesn't checkpatch complain about long lines in the patch description?

Quoted 3 lines

Reviewed-by Rik van Riel <[email protected]>

Signed-off-by Baolin Wang <[email protected]>

---

...

Quoted 19 lines

index a5f0a264ad56..a1b3967afe41 100644

mm/internal.h +0 -6

@@ -1843,10 +1843,4 @@ static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,
 
  #endif /* CONFIG_MMU_NOTIFIER */
 
-static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *vma,
-		unsigned long addr, pte_t *ptep)
-{
-	return test_and_clear_young_ptes_notify(vma, addr, ptep, 1);
-}
-
  #endif	/* __MM_INTERNAL_H */

mm/rmap.c +0 -0

index 11cc6171344f..beb423f3e8ec 100644

mm/rmap.c static bool folio_referenced_one(struct folio *folio, · line 967

Quoted 10 lines

mm/rmap.c +7 -0

@@ -958,25 +958,21 @@ static bool folio_referenced_one(struct folio *folio,
  			return false;
  		}
 
+		if (pvmw.pte && folio_test_large(folio)) {
+			const unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
+			const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
+			pte_t pteval = ptep_get(pvmw.pte);
+
+			nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr);
+			ptes += nr;

Could we move that "ptes += nr;" just before the "pra->mapcount -= nr;"?

Would make the whole thing look less weird (only incrementing "ptes" with large folios).

Quoted 51 lines

mm/rmap.c +7 -14

+		}
+
  		if (lru_gen_enabled() && pvmw.pte) {
-			if (lru_gen_look_around(&pvmw))
+			if (lru_gen_look_around(&pvmw, nr))
  				referenced++;
  		} else if (pvmw.pte) {
-			if (folio_test_large(folio)) {
-				unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
-				unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
-				pte_t pteval = ptep_get(pvmw.pte);
-
-				nr = folio_pte_batch(folio, pvmw.pte,
-						     pteval, max_nr);
-			}
-
-			ptes += nr;
  			if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr))
  				referenced++;
-			/* Skip the batched PTEs */
-			pvmw.pte += nr - 1;
-			pvmw.address += (nr - 1) * PAGE_SIZE;
  		} else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
  			if (pmdp_clear_flush_young_notify(vma, address,
  						pvmw.pmd))

@@ -995,6 +991,10 @@ static bool folio_referenced_one(struct folio *folio,
  			page_vma_mapped_walk_done(&pvmw);
  			break;
  		}
+
+		/* Skip the batched PTEs */
+		pvmw.pte += nr - 1;
+		pvmw.address += (nr - 1) * PAGE_SIZE;
  	}
 
  	if (referenced)

mm/vmscan.c +1 -0

index 0a5622420987..7457b3c06fa3 100644
@@ -3474,6 +3474,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
  	struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
  	DEFINE_MAX_SEQ(walk->lruvec);
  	int gen = lru_gen_from_seq(max_seq);
+	unsigned int nr;
  	pmd_t pmdval;
 
  	pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl);

@@ -3492,11 +3493,13 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
 
  	lazy_mmu_mode_enable();

mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3502

Quoted 10 lines

mm/vmscan.c +4 -2

  restart:
-	for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
+	for (i = pte_index(start), addr = start; addr != end; i += nr, addr += nr * PAGE_SIZE) {
  		unsigned long pfn;
  		struct folio *folio;
-		pte_t ptent = ptep_get(pte + i);
+		pte_t *cur_pte = pte + i;
+		pte_t ptent = ptep_get(cur_pte);
 
+		nr = 1;

Looking at this again, we should get rid of "i" completely and instead
* rename pte to start_pte
* Add a new pte, which we increment in the loop

So we end up with something like

mm/vmscan.c +9 -8

index 54cf4924d223..150cbb2253b9 100644
@@ -3486,9 +3486,8 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio,
  static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                            struct mm_walk *args)
  {
-       int i;
         bool dirty;
-       pte_t *pte;
+       pte_t *start_pte, pte;
         spinlock_t *ptl;
         unsigned long addr;
         int total = 0;

@@ -3499,29 +3498,31 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
         struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec);
         DEFINE_MAX_SEQ(walk->lruvec);
         int gen = lru_gen_from_seq(max_seq);
+       unsigned int nr;
         pmd_t pmdval;
 
-       pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl);
-       if (!pte)
+       start_pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, &ptl);
+       if (!start_pte)
                 return false;
 
         if (!spin_trylock(ptl)) {
-               pte_unmap(pte);
+               pte_unmap(start_pte);
                 return true;
         }
 
         if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) {
-               pte_unmap_unlock(pte, ptl);
+               pte_unmap_unlock(start_pte, ptl);
                 return false;
         }
 
         lazy_mmu_mode_enable();
  restart:
-       for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
+       for (addr = start, pte = start_pte; addr != end; addr += nr * PAGE_SIZE, pte += nr) {
                 unsigned long pfn;
                 struct folio *folio;
-               pte_t ptent = ptep_get(pte + i);
+               pte_t ptent = ptep_get(pte);
 
+               nr = 1;
                 total++;
                 walk->mm_stats[MM_LEAF_TOTAL]++;
 

mm/vmscan.c +0 -0

 >   	walk_update_folio(walk, last, gen, dirty);
 > @@ -4166,7 +4178,7 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)

Quoted 34 lines

mm/vmscan.c +6 -4

   * the PTE table to the Bloom filter. This forms a feedback loop between the
   * eviction and the aging.
   */
-bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
+bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)
  {
  	int i;
  	bool dirty;

@@ -4184,12 +4196,13 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
  	struct lruvec *lruvec;
  	struct lru_gen_mm_state *mm_state;
  	unsigned long max_seq;
+	pte_t *cur_pte;
  	int gen;
 
  	lockdep_assert_held(pvmw->ptl);
  	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
 
-	if (!ptep_test_and_clear_young_notify(vma, addr, pte))
+	if (!test_and_clear_young_ptes_notify(vma, addr, pte, nr))
  		return false;
 
  	if (spin_is_contended(pvmw->ptl))

@@ -4229,10 +4242,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 
  	pte -= (addr - start) / PAGE_SIZE;
 
-	for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) {
+	for (i = 0, addr = start, cur_pte = pte; addr != end;
+	     i += nr, cur_pte += nr, addr += nr * PAGE_SIZE) {
  		unsigned long pfn;
-		pte_t ptent = ptep_get(pte + i);
+		pte_t ptent = ptep_get(cur_pte);
 

mm/vmscan.c static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · line 3525

Quoted 1 line

mm/vmscan.c +1 -0

+		nr = 1;

Can't you just use pte and increment that, right?

"pte" is not used afterwards.

Quoted 66 lines · David Hildenbrand (Arm)

Quoted 65 lines · Baolin Wang

Currently, MGLRU will call ptep_test_and_clear_young_notify() to check and
clear the young flag for each PTE sequentially, which is inefficient for large folios reclamation.

Moreover, on Arm64 architecture, which supports contiguous PTEs, the Arm64-
specific ptep_test_and_clear_young() already implements an optimization to
clear the young flags for PTEs within a contiguous range. However, this is not
sufficient. Similar to the Arm64 specific clear_flush_young_ptes(), we can
extend this to perform batched operations for the entire large folio (which
might exceed the contiguous range: CONT_PTE_SIZE).

Thus, we can introduce a new batched helper:
test_and_clear_young_ptes() and
its wrapper test_and_clear_young_ptes_notify() which are consistent with the
existing functions, to perform batched checking of the young flags for large
folios, which can help improve performance during large folio reclamation when
MGLRU is enabled. And it will be overridden by the architecture that implements
a more efficient batch operation in the following patches.

Signed-off-by Baolin Wang <[email protected]>

---
  include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ mm/internal.h           | 16 +++++++++++-----

  2 files changed, 49 insertions(+), 5 deletions(-)

include/linux/pgtable.h +0 -0

index 776993d4567b..29bd9fd04e1e 100644
@@ -1103,6 +1103,44 @@ static inline int
 clear_flush_young_ptes(struct vm_area_struct *vma,
   }
   #endif

  +#ifndef test_and_clear_young_ptes
+/**
+ * test_and_clear_young_ptes - Mark PTEs that map consecutive pages of the same
+ *                   folio as old
+ * @vma: The virtual memory area the pages are mapped into.
+ * @addr: Address the first page is mapped at.
+ * @ptep: Page table pointer for the first entry.
+ * @nr: Number of entries to clear access bit.
+ *
+ * May be overridden by the architecture; otherwise, implemented as a simple
+ * loop over ptep_test_and_clear_young().
+ *
+ * Note that PTE bits in the PTE range besides the PFN can differ. For example,
+ * some PTEs might be write-protected.
+ *
+ * Context: The caller holds the page table lock.  The PTEs map consecutive
+ * pages that belong to the same folio.  The PTEs are all in the same PMD.

Quoted 8 lines

Quoted 6 lines

+ *
+ * Returns: whether any PTE was young.
+ */
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma, +                        unsigned long addr, pte_t *ptep,
+                        unsigned int nr)

Two tabs ...

Ah, yes, not sure why I missed this one :(

Quoted 1 line

What happened to using a boolen as return type and for "int young"?

As I replied to you previously [1], I’d like to do this in a follow-up patchset that converts all functions that check the young flag. Does that sound OK to you?

[1] https://lore.kernel.org/all/32c538ce-6af8-48a8-86fc-
[email protected]/

Quoted 11 lines

                 unsigned long pfn;
                 struct folio *folio;
-               pte_t ptent = ptep_get(pte + i);
+               pte_t ptent = ptep_get(pte);
  +               nr = 1;
                 total++; walk->mm_stats[MM_LEAF_TOTAL]++;
  @@ -3533,7 +3534,16 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,
                 if (!folio)
                         continue;

Thanks for the code. I considered simplifying the logic this way, but this logic would be incorrect. Because the 'start' can be updated by the get_next_vma() below, when we goto restart, we will get a new 'i' value by 'pte_index(start)', so I kept the 'i' variable.