Optimize mremap() for large folios
Subsystems (3)
v2 cover Optimize mremap() for large folios
Currently move_ptes() iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch using folio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number of ptep_get() calls (since on a contig block, ptep_get() on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs through get_and_clear_full_ptes, replacing ptep_get_and_clear.
Mapping 512K of memory, memsetting it, remapping it to src + 512K, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).
Test program for reference:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>
#define SIZE (1UL << 20) // 512 KB
int main(void) {
void *new_addr, *addr;
for (int i = 0; i < 10000; ++i) {
addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
return 1;
}
memset(addr, 0xAA, SIZE);
new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE);
if (new_addr != (addr + SIZE)) {
perror("mremap");
return 1;
}
munmap(new_addr, SIZE);
}}
v1->v2:
- Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducing mremap_folio_pte_batch(), fix loop iteration (Lorenzo)
- Merge patch 2 and 3 (Anshuman, Lorenzo)
- Fix maybe_contiguous_pte_pfns (Willy)
Dev Jain (2):
mm: Call pointers to ptes as ptep
mm: Optimize mremap() by PTE batching
include/linux/pgtable.h | 29 ++++++++++++++++++++++ mm/mremap.c | 54 +++++++++++++++++++++++++++++------------ 2 files changed, 68 insertions(+), 15 deletions(-)
Signature · 1 line
2.30.2
PATCH v2 1/2 mm: Call pointers to ptes as ptep
Avoid confusion between pte_t* and pte_t data types by suffixing pointer type variables with p. No functional change.
--- mm/mremap.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-)
mm/mremap.c +15 -14
@@ -176,7 +176,8 @@ static int move_ptes(struct pagetable_move_control *pmc,struct vm_area_struct *vma = pmc->old;bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm; -pte_t *old_pte, *new_pte, pte; +pte_t *old_ptep, *new_ptep; +pte_t pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;
@@ -211,8 +212,8 @@ static int move_ptes(struct pagetable_move_control *pmc,* We don't have to worry about the ordering of src and dst * pte locks because exclusive mmap_lock prevents deadlock. */ - old_pte =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); -if (!old_pte) { + old_ptep =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); +if (!old_ptep) { err = -EAGAIN ; goto out; }
@@ -223,10 +224,10 @@ static int move_ptes(struct pagetable_move_control *pmc,* mmap_lock, so this new_pte page is stable, so there is no need to get * pmdval and do pmd_same() check. */ - new_pte =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, + new_ptep =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, &new_ptl); -if (!new_pte) { -pte_unmap_unlock (old_pte, old_ptl); +if (!new_ptep) { +pte_unmap_unlock (old_ptep, old_ptl); err = -EAGAIN ; goto out; }
@@ -235,12 +236,12 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_pte++, old_addr +=PAGE_SIZE , - new_pte++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_pte))) +for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , + new_ptep++, new_addr +=PAGE_SIZE ) { +if (pte_none (ptep_get (old_ptep)))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_pte); + pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the
@@ -258,7 +259,7 @@ static int move_ptes(struct pagetable_move_control *pmc, pte =move_soft_dirty_pte (pte);if (need_clear_uffd_wp &&pte_marker_uffd_wp (pte)) -pte_clear (mm, new_addr, new_pte); +pte_clear (mm, new_addr, new_ptep);else {if (need_clear_uffd_wp) {if (pte_present (pte))
@@ -266,7 +267,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_pte, pte); +set_pte_at (mm, new_addr, new_ptep, pte); } }
@@ -275,8 +276,8 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_range (vma, old_end - len, old_end);if (new_ptl != old_ptl)spin_unlock (new_ptl); -pte_unmap (new_pte -1 ); -pte_unmap_unlock (old_pte -1 , old_ptl); +pte_unmap (new_ptep -1 ); +pte_unmap_unlock (old_ptep -1 , old_ptl); out:if (pmc->need_rmap_locks)drop_rmap_locks (vma);
Signature · 1 line
2.30.2
Re: [PATCH v2 1/2] mm: Call pointers to ptes as ptep
Quoted 5 lines · Dev Jain
Avoid confusion between pte_t* and
pte_tdata types by suffixing pointer type variables with p. No functional change.Signed-off-by Dev Jain <[email protected]>
Quoted 92 lines
--- mm/mremap.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-)mm/mremap.c +15
-15@@ -176,7 +176,8 @@ static int move_ptes(struct pagetable_move_control *pmc,struct vm_area_struct *vma = pmc->old;bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm; -pte_t *old_pte, *new_pte, pte; +pte_t *old_ptep, *new_ptep; +pte_t pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -211,8 +212,8 @@ static int move_ptes(struct pagetable_move_control *pmc,* We don't have to worry about the ordering of src and dst * pte locks because exclusive mmap_lock prevents deadlock. */ - old_pte =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); -if (!old_pte) { + old_ptep =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); +if (!old_ptep) { err = -EAGAIN ; goto out; }@@ -223,10 +224,10 @@ static int move_ptes(struct pagetable_move_control *pmc,* mmap_lock, so this new_pte page is stable, so there is no need to get * pmdval and do pmd_same() check. */ - new_pte =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, + new_ptep =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, &new_ptl); -if (!new_pte) { -pte_unmap_unlock (old_pte, old_ptl); +if (!new_ptep) { +pte_unmap_unlock (old_ptep, old_ptl); err = -EAGAIN ; goto out; }@@ -235,12 +236,12 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_pte++, old_addr +=PAGE_SIZE , - new_pte++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_pte))) +for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , + new_ptep++, new_addr +=PAGE_SIZE ) { +if (pte_none (ptep_get (old_ptep)))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_pte); + pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the @@ -258,7 +259,7 @@ static int move_ptes(struct pagetable_move_control *pmc, pte =move_soft_dirty_pte (pte);if (need_clear_uffd_wp &&pte_marker_uffd_wp (pte)) -pte_clear (mm, new_addr, new_pte); +pte_clear (mm, new_addr, new_ptep);else {if (need_clear_uffd_wp) {if (pte_present (pte))@@ -266,7 +267,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_pte, pte); +set_pte_at (mm, new_addr, new_ptep, pte); } }@@ -275,8 +276,8 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_range (vma, old_end - len, old_end);if (new_ptl != old_ptl)spin_unlock (new_ptl); -pte_unmap (new_pte -1 ); -pte_unmap_unlock (old_pte -1 , old_ptl); +pte_unmap (new_ptep -1 ); +pte_unmap_unlock (old_ptep -1 , old_ptl); out:if (pmc->need_rmap_locks)drop_rmap_locks (vma); --2.30.2
Re: [PATCH v2 1/2] mm: Call pointers to ptes as ptep
In reply to Dev JainQuoted 83 lines · Dev Jain
Avoid confusion between pte_t* and
pte_tdata types by suffixing pointer type variables with p. No functional change.Signed-off-by Dev Jain <[email protected]>--- mm/mremap.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-)mm/mremap.c +13
-12@@ -176,7 +176,8 @@ static int move_ptes(struct pagetable_move_control *pmc,struct vm_area_struct *vma = pmc->old;bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm; -pte_t *old_pte, *new_pte, pte; +pte_t *old_ptep, *new_ptep; +pte_t pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -211,8 +212,8 @@ static int move_ptes(struct pagetable_move_control *pmc,* We don't have to worry about the ordering of src and dst * pte locks because exclusive mmap_lock prevents deadlock. */ - old_pte =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); -if (!old_pte) { + old_ptep =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); +if (!old_ptep) { err = -EAGAIN ; goto out; }@@ -223,10 +224,10 @@ static int move_ptes(struct pagetable_move_control *pmc,* mmap_lock, so this new_pte page is stable, so there is no need to get * pmdval and do pmd_same() check. */ - new_pte =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, + new_ptep =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, &new_ptl); -if (!new_pte) { -pte_unmap_unlock (old_pte, old_ptl); +if (!new_ptep) { +pte_unmap_unlock (old_ptep, old_ptl); err = -EAGAIN ; goto out; }@@ -235,12 +236,12 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_pte++, old_addr +=PAGE_SIZE , - new_pte++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_pte))) +for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , + new_ptep++, new_addr +=PAGE_SIZE ) { +if (pte_none (ptep_get (old_ptep)))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_pte); + pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the @@ -258,7 +259,7 @@ static int move_ptes(struct pagetable_move_control *pmc, pte =move_soft_dirty_pte (pte);if (need_clear_uffd_wp &&pte_marker_uffd_wp (pte)) -pte_clear (mm, new_addr, new_pte); +pte_clear (mm, new_addr, new_ptep);else {if (need_clear_uffd_wp) {if (pte_present (pte))@@ -266,7 +267,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_pte, pte); +set_pte_at (mm, new_addr, new_ptep, pte); } }
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 283
Quoted 10 lines
mm/mremap.c +2
-2@@ -275,8 +276,8 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_range (vma, old_end - len, old_end);if (new_ptl != old_ptl)spin_unlock (new_ptl); -pte_unmap (new_pte -1 ); -pte_unmap_unlock (old_pte -1 , old_ptl); +pte_unmap (new_ptep -1 ); +pte_unmap_unlock (old_ptep -1 , old_ptl); out:if (pmc->need_rmap_locks)drop_rmap_locks (vma);
Re: [PATCH v2 1/2] mm: Call pointers to ptes as ptep
In reply to Dev JainQuoted 4 lines · Dev Jain
Avoid confusion between pte_t* and
pte_tdata types by suffixing pointer type variables with p. No functional change.Signed-off-by Dev Jain <[email protected]>
LGTM, thanks!
Quoted 92 lines
--- mm/mremap.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-)mm/mremap.c +15
-15@@ -176,7 +176,8 @@ static int move_ptes(struct pagetable_move_control *pmc,struct vm_area_struct *vma = pmc->old;bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm; -pte_t *old_pte, *new_pte, pte; +pte_t *old_ptep, *new_ptep; +pte_t pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -211,8 +212,8 @@ static int move_ptes(struct pagetable_move_control *pmc,* We don't have to worry about the ordering of src and dst * pte locks because exclusive mmap_lock prevents deadlock. */ - old_pte =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); -if (!old_pte) { + old_ptep =pte_offset_map_lock (mm, old_pmd, old_addr, &old_ptl); +if (!old_ptep) { err = -EAGAIN ; goto out; }@@ -223,10 +224,10 @@ static int move_ptes(struct pagetable_move_control *pmc,* mmap_lock, so this new_pte page is stable, so there is no need to get * pmdval and do pmd_same() check. */ - new_pte =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, + new_ptep =pte_offset_map_rw_nolock (mm, new_pmd, new_addr, &dummy_pmdval, &new_ptl); -if (!new_pte) { -pte_unmap_unlock (old_pte, old_ptl); +if (!new_ptep) { +pte_unmap_unlock (old_ptep, old_ptl); err = -EAGAIN ; goto out; }@@ -235,12 +236,12 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_pte++, old_addr +=PAGE_SIZE , - new_pte++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_pte))) +for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , + new_ptep++, new_addr +=PAGE_SIZE ) { +if (pte_none (ptep_get (old_ptep)))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_pte); + pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the @@ -258,7 +259,7 @@ static int move_ptes(struct pagetable_move_control *pmc, pte =move_soft_dirty_pte (pte);if (need_clear_uffd_wp &&pte_marker_uffd_wp (pte)) -pte_clear (mm, new_addr, new_pte); +pte_clear (mm, new_addr, new_ptep);else {if (need_clear_uffd_wp) {if (pte_present (pte))@@ -266,7 +267,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_pte, pte); +set_pte_at (mm, new_addr, new_ptep, pte); } }@@ -275,8 +276,8 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_range (vma, old_end - len, old_end);if (new_ptl != old_ptl)spin_unlock (new_ptl); -pte_unmap (new_pte -1 ); -pte_unmap_unlock (old_pte -1 , old_ptl); +pte_unmap (new_ptep -1 ); +pte_unmap_unlock (old_ptep -1 , old_ptl); out:if (pmc->need_rmap_locks)drop_rmap_locks (vma); --2.30.2
PATCH v2 2/2 mm: Optimize mremap() by PTE batching
In reply to Dev JainTo use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of vm_normal_folio(). We want to avoid the cost of vm_normal_folio() if the code path doesn't already require the folio. For arm64, pte_batch_hint() does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, use folio_pte_batch() to optimize move_ptes(). On arm64, if the ptes are painted with the contig bit, then ptep_get() will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number of ptep_get() calls. Next, ptep_get_and_clear() will eventually call contpte_try_unfold() on every contig block, thus flushing the TLB for the complete large folio range. Instead, use get_and_clear_full_ptes() so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.
--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)
include/linux/pgtable.h +29 -0
@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte; + +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 ); +} +#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,
mm/mremap.c +30 -7
@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + } +return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {
@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;
@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma
@@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) { + max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the
@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + } + pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 ); pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);
@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } }
Signature · 1 line
2.30.2
Andrew Morton
maintainerRe: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 28 lines · On Wed, 7 May 2025 11:32:56 +0530 Dev Jain <[email protected]> wrote:
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block....
include/linux/pgtable.h +4
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry.
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 385
Quoted 10 lines
include/linux/pgtable.h +10
-0+* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/
Gee, that isn't the prettiest interface we've ever invented.
Is there any prospect that this function will get another caller? If not then it's probably better to just open-code all this in the caller and fill it up with comments.
Anyway, review of this series is scanty. I'd normally enter wait-and-see mode, but this:
Quoted 2 lines
the average execution time reduces from 1.9 to
1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).
prompts me into push-it-along mode.
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 48 lines · Andrew Morton
Quoted 28 lines · On Wed, 7 May 2025 11:32:56 +0530 Dev Jain <[email protected]> wrote:
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block....
include/linux/pgtable.h +4
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 385
Quoted 10 lines
include/linux/pgtable.h +10
-0+* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ Gee, that isn't the prettiest interface we've ever invented.
Is there any prospect that this function will get another caller? If not then it's probably better to just open-code all this in the caller and fill it up with comments.
Quoted 9 lines
Anyway, review of this series is scanty. I'd normally enter wait-and-see mode, but this:
Quoted 2 lines
the average execution time reduces from 1.9 to
1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).prompts me into push-it-along mode.
There's been a ton of review on the v1 (see [0]), I will get to this one soon, but as you can see from the v1 review there's lots we need to check for correctness.
So it's coming :)
[0]: https://lore.kernel.org/linux-mm/[email protected]/
Let's not be too hasty to merge this though until we're sure it's safe (this is delicate code). But I'll be giving R-b tags on a finalised version anyway.
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Dev JainQuoted 33 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +3
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio.
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 384
Quoted 10 lines
include/linux/pgtable.h +10
-0+* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists.
ptep points to an entry in a PTE page. As long as it is not pointing to the last entry, ptep+1 should always exist. With PTRS_PER_PTE and sizeof(pte_t), you can check ptep address to figure out whether it is the last entry of a PTE page, right? Let me know if I misunderstand anything.
--
Best Regards,
Yan, Zi
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 43 lines · Zi Yan
Quoted 41 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +11
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible.
Quoted 8 lines
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 384
Quoted 2 lines
include/linux/pgtable.h +2
-0+* +* The caller must ensure that ptep + 1 exists. ptep points to an entry in a PTE page. As long as it is not pointing to the last entry, ptep+1 should always exist. With
PTRS_PER_PTEandsizeof(pte_t), you can check ptep address to figure out whether it is the last entry of a PTE page, right? Let me know if I misunderstand anything.
Sounds correct to me.
Quoted 5 lines
--
Best Regards,
Yan, Zi
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Zi YanQuoted 43 lines · Zi Yan
Quoted 41 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +11
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible.
Quoted 8 lines
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 384
Quoted 2 lines
include/linux/pgtable.h +2
-0+* +* The caller must ensure that ptep + 1 exists. ptep points to an entry in a PTE page. As long as it is not pointing to the last entry, ptep+1 should always exist. With
PTRS_PER_PTEandsizeof(pte_t), you can check ptep address to figure out whether it is the last entry of a PTE page, right? Let me know if I misunderstand anything.
Agreed, this not-the-last-pte-entry test is definitely required here just to prevent a potential unmapped access crash. But I do agree with Andrew that unless there are callers, this should be contained in the call site itself (mm/mremap.c) with a good explanation.
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Dev JainQuoted 67 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +29
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte; + +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 ); +} +#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +0
-0
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 179
Quoted 10 lines
mm/mremap.c +7
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ;
A small nit - s/nr/nr_pages ?
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 181
Quoted 2 lines
mm/mremap.c +2
-0+ +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) {
Like mentioned earlier in v1, could maybe_contiguous_pte_pfns() here add some additional cost for buffers that are actually not mapped to contig physical pages.
The test case you have mentioned in the cover demonstrating performance gains might have always been run just after boot, thus increasing the probability of contiguous physical mapping, which will not be the case on fragmented memory systems. In that case the proposed consecutive PFN comparison will always happen unconditionally without any benefit ?
Just curious.
From V1
--------------------------------------------------------------------maybe_contiguous_pte_pfns() cost will be applicable for memory areas greater than a single PAGE_SIZE (i.e max_nr != 1) ? This helper extracts an additional consecutive pte, ensures that it is valid mapped and extracts pfn before comparing for the span.
There is some cost associated with the above code sequence which looks justified for sequential access of memory buffers that has consecutive physical memory backing. But what happens when such buffers are less probable, will those buffers take a performance hit for all the comparisons that just turn out to be negative ?
--------------------------------------------------------------------
Quoted 21 lines
mm/mremap.c +9
-1+ folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + } +return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 206
Quoted 4 lines
mm/mremap.c +1
-0@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;
A small nit - s/max_nr/max_nr_pages ?
Quoted 46 lines
mm/mremap.c +5
-4/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma @@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) { + max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* mm/mremap.c +6
-2* If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the @@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + } + pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 ); pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } }
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 71 lines · Anshuman Khandual
Quoted 70 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +29
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte; + +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 ); +} +#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +0
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; }
Quoted 9 lines
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 179
Quoted 7 lines
mm/mremap.c +7
-0+/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ;A small nit - s/nr/nr_pages ?
Well, all other places nr is being used, so I would like to keep it simple and stick to convention :)
Quoted 8 lines
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 181
Quoted 3 lines
mm/mremap.c +2
-0+ +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) {Like mentioned earlier in v1, could
maybe_contiguous_pte_pfns()here add some additional cost for buffers that are actually not mapped to contig physical pages.
Quoted 5 lines
The test case you have mentioned in the cover demonstrating performance gains might have always been run just after boot, thus increasing the probability of contiguous physical mapping, which will not be the case on fragmented memory systems. In that case the proposed consecutive PFN comparison will always happen unconditionally without any benefit ?
I think you mean to say that the underlying folio may not be actually large but the buddy allocator distributed consecutive physical memory. Hmm...at this rate I am thinking that the overhead of vm_normal_folio() + folio_test_large() is acceptable and is less churn :) Would like to hear your thoughts.
Quoted 92 lines
Just curious.
From V1
--------------------------------------------------------------------
maybe_contiguous_pte_pfns()cost will be applicable for memory areas greater than a singlePAGE_SIZE(i.emax_nr!= 1) ? This helper extracts an additional consecutive pte, ensures that it is valid mapped and extracts pfn before comparing for the span.There is some cost associated with the above code sequence which looks justified for sequential access of memory buffers that has consecutive physical memory backing. But what happens when such buffers are less probable, will those buffers take a performance hit for all the comparisons that just turn out to be negative ?
--------------------------------------------------------------------Quoted 21 lines
mm/mremap.c +9
-1+ folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + } +return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 206
Quoted 4 lines
mm/mremap.c +1
-0@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;A small nit - s/max_nr/max_nr_pages ?
Quoted 46 lines
mm/mremap.c +5
-4/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma @@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) { + max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure
* to flush TLB before we drop the PTL for themm/mremap.c +6
-2@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + } + pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 ); pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } }
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Anshuman KhandualQuoted 1 line · Anshuman Khandual
On 5/7/25 11:32, Dev Jain wrote:
[snip]
Quoted 9 lines
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 179
Quoted 7 lines
mm/mremap.c +7
-0+/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ;A small nit - s/nr/nr_pages ?
Agree in principal, but I think nr_ptes is clearer. You might not even be dealing with a page, if you hit a pte_none() pte for instance.
So max_nr_ptes, nr_ptes.
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Dev JainBefore getting into the review, just to say thanks for refactoring as per my (and of course other's) comments, much appreciated and big improvement! :)
We're getting there...
Quoted 47 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +17
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte;
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 398
Quoted 10 lines
include/linux/pgtable.h +10
-0+ +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 );
Let's not do unlikely()'s unless we have data for them... it shouldn't mean 'what the programmer believes' :)
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 399
Quoted 1 line
include/linux/pgtable.h +1
-0+}
Yeah I'm with Andrew and Anshuman, I mean this is kind of a nasty interface (I mean _perhaps_ unavoidably) and we've done the relevant check in mremap_folio_pte_batch(), so let's just move it there with comments, as this
Quoted 16 lines
include/linux/pgtable.h +1
-0+#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +4
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{
mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 186
Quoted 10 lines
mm/mremap.c +10
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + }
This needs some refactoring, avoid nesting at all costs :)
We'll want to move the maybe_contiguous_pte_pfns() function over here, so that'll change things, but in general let's use a guard clause.
So an if block like:
if (foo) {
... bunch of logic ...
}Is better replaced with a guard clause so you have:
if (!foo)
return ...;
... bunch of logic ...
Here we could really expand things out to make things SUPER clear like:
static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr,
pte_t *ptep, pte_t pte, int max_nr)
{
const fpb_t flags;
struct folio *folio;
if (max_nr == 1)
return 1;
if (!maybe_contiguous_pte_pfns(ptep, pte)) // obviously replace with open code...
return 1;
folio = vm_normal_folio(vma, addr, pte);
if (!folio)
return 1;
if (!folio_test_large(folio))
return 1;
flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY;
return folio_pte_batch(folio, addr, ptep, pte, max_nr,
flags, NULL, NULL, NULL);
}I mean you could argue assign nr would be neater here, but you get the point!
David mentioned a point about this code over in v1 discussion (see [0]). Trying to bring converastion here to avoid it being split across old/new series. There he said:
David H:
Quoted 3 lines
(2) Do we really need "must be part of the same folio", or could be just batch over present ptes that map consecutive PFNs? In that case, a helper that avoids
folio_pte_batch()completely might be better.
Hm, if we didn't do the batch test, can we batch a split large folio here ok? I'm guessing we can in which case this check is actually limiting...
Are we _explicitly_ only considering the cont pte case and ignoring the split THP case?
[0]: https://lore.kernel.org/all/[email protected]/
And in what circumstances will the hint be set, with a present subsequent PTE but !folio_test_large()?
I guess the hint might not be taken? But then isn't the valid check just folio_test_large() and we don't need this batched check at all?
Is it only to avoid the split THP case?
We definitely need some clarity here, and a comment in the code explaining what's going on as this is subtle stuff.
Quoted 24 lines
mm/mremap.c +5
-1+return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 258
Quoted 8 lines
mm/mremap.c +2
-3@@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) {
Really nitty thing here but the indentation is all messed up here, I mean nothing is going to be nice but maybe indent by two tabs below 'for'.
I'm not a fan of this declaration of nr, typically in a for loop a declaration here would be the counter, so this is just confusing.
In the old implementation, declaring nr in the for loop would make sense, but in the newly refactored one you should just declare it at the top.
Also as per Anshuman review, I think nr_ptes, max_nr_ptes would be better.
I don't think 'nr' needs to be initialised either, since the conditional is 'old_addr < old_end' and you _should_ only perform the
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 261
Quoted 3 lines
mm/mremap.c +3
-0+ max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))
This seems broken.
You're missing a nr assignment here, so you'll happen to offset by the number of pages of the last folio you encountered?
Should be:
if (pte_none(old_pte)) {
nr_ptes = 1;
continue;
}Or, alternatively, you can reset nr_ptes to 1 at the start of each loop.
Quoted 8 lines
mm/mremap.c +0
-1continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure mm/mremap.c +0
-0* to flush TLB before we drop the PTL for the
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 279
Quoted 9 lines
mm/mremap.c +4
-1@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + }
Thanks this is much clearer compared to v1
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 280
Quoted 1 line
mm/mremap.c +1
-0+ pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 );
Nit but...
Can we have a comment indicating what the last parameter refers to? I think David maybe doens't like this so obviously if he prefers not that fine, but I'm thinking something like:
pte = get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, /*full=*/false);
I think we are good to just use 'false' here right? As it's only an int for historical purposes...
Quoted 15 lines
mm/mremap.c +1
-2pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } } --2.30.2
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 165 lines · Lorenzo Stoakes
Before getting into the review, just to say thanks for refactoring as per my (and of course other's) comments, much appreciated and big improvement! :)
We're getting there...
Quoted 47 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +17
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte;include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 398
Quoted 10 lines
include/linux/pgtable.h +10
-0+ +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 );Let's not do unlikely()'s unless we have data for them... it shouldn't mean 'what the programmer believes' :)
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 399
Quoted 1 line
include/linux/pgtable.h +1
-0+}Yeah I'm with Andrew and Anshuman, I mean this is kind of a nasty interface (I mean _perhaps_ unavoidably) and we've done the relevant check in
mremap_folio_pte_batch(), so let's just move it there with comments, as thisQuoted 16 lines
include/linux/pgtable.h +1
-0+#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +4
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 186
Quoted 10 lines
mm/mremap.c +10
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + }This needs some refactoring, avoid nesting at all costs :)
We'll want to move the
maybe_contiguous_pte_pfns()function over here, so that'll change things, but in general let's use a guard clause.So an if block like:
if (foo) { ... bunch of logic ... }Is better replaced with a guard clause so you have:
if (!foo)
return ...;... bunch of logic ...
Here we could really expand things out to make things SUPER clear like:
static int mremap_folio_pte_batch(struct
vm_area_struct*vma, unsigned long addr,
pte_t*ptep,pte_tpte, intmax_nr){ const fpb_t flags; struct folio *folio; if (max_nr == 1) return 1; if (!maybe_contiguous_pte_pfns(ptep, pte)) // obviously replace with open code... return 1; folio = vm_normal_folio(vma, addr, pte); if (!folio) return 1; if (!folio_test_large(folio)) return 1; flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; return folio_pte_batch(folio, addr, ptep, pte, max_nr, flags, NULL, NULL, NULL); }I mean you could argue assign nr would be neater here, but you get the point!
David mentioned a point about this code over in v1 discussion (see [0]). Trying to bring converastion here to avoid it being split across old/new series. There he said:
David H:
Quoted 3 lines
(2) Do we really need "must be part of the same folio", or could be just batch over present ptes that map consecutive PFNs? In that case, a helper that avoids
folio_pte_batch()completely might be better.Hm, if we didn't do the batch test, can we batch a split large folio here ok? I'm guessing we can in which case this check is actually limiting...
Are we _explicitly_ only considering the cont pte case and ignoring the split THP case?
[0]: https://lore.kernel.org/all/[email protected]/
And in what circumstances will the hint be set, with a present subsequent PTE but !folio_test_large()?
Quoted 7 lines
I guess the hint might not be taken? But then isn't the valid check just
folio_test_large()and we don't need this batched check at all?Is it only to avoid the split THP case?
We definitely need some clarity here, and a comment in the code explaining what's going on as this is subtle stuff.
The whole maybe_contiguous_pte_pfns() is really far from nice. Let's add something like that *only if absolutely required* for performance reasons and on top of this patch.
But let's clarify if we have to limit ourselves to a single folio at all.
Staring at it, I think the tricky bit is:
pte = get_and_clear_full_ptes();
That means that
a) if any PTE is dirty, the resulting PTE will be dirty.
b) if any PTE is young, the resulting PTE will be young.
So we could be marking PTEs as dirty/young from unrelated folios.
That's problematic.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
In reply to Lorenzo StoakesQuoted 165 lines · Lorenzo Stoakes
Before getting into the review, just to say thanks for refactoring as per my (and of course other's) comments, much appreciated and big improvement! :)
We're getting there...
Quoted 47 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +17
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte;include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 398
Quoted 10 lines
include/linux/pgtable.h +10
-0+ +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 );Let's not do unlikely()'s unless we have data for them... it shouldn't mean 'what the programmer believes' :)
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 399
Quoted 1 line
include/linux/pgtable.h +1
-0+}Yeah I'm with Andrew and Anshuman, I mean this is kind of a nasty interface (I mean _perhaps_ unavoidably) and we've done the relevant check in
mremap_folio_pte_batch(), so let's just move it there with comments, as thisQuoted 16 lines
include/linux/pgtable.h +1
-0+#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +4
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 186
Quoted 10 lines
mm/mremap.c +10
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + }This needs some refactoring, avoid nesting at all costs :)
We'll want to move the
maybe_contiguous_pte_pfns()function over here, so that'll change things, but in general let's use a guard clause.So an if block like:
if (foo) { ... bunch of logic ... }Is better replaced with a guard clause so you have:
if (!foo)
return ...;... bunch of logic ...
Here we could really expand things out to make things SUPER clear like:
static int mremap_folio_pte_batch(struct
vm_area_struct*vma, unsigned long addr,
pte_t*ptep,pte_tpte, intmax_nr){ const fpb_t flags; struct folio *folio; if (max_nr == 1) return 1; if (!maybe_contiguous_pte_pfns(ptep, pte)) // obviously replace with open code... return 1; folio = vm_normal_folio(vma, addr, pte); if (!folio) return 1; if (!folio_test_large(folio)) return 1; flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; return folio_pte_batch(folio, addr, ptep, pte, max_nr, flags, NULL, NULL, NULL); }I mean you could argue assign nr would be neater here, but you get the point!
David mentioned a point about this code over in v1 discussion (see [0]). Trying to bring converastion here to avoid it being split across old/new series. There he said:
David H:
Quoted 3 lines
(2) Do we really need "must be part of the same folio", or could be just batch over present ptes that map consecutive PFNs? In that case, a helper that avoids
folio_pte_batch()completely might be better.Hm, if we didn't do the batch test, can we batch a split large folio here ok? I'm guessing we can in which case this check is actually limiting...
Are we _explicitly_ only considering the cont pte case and ignoring the split THP case?
[0]: https://lore.kernel.org/all/[email protected]/
And in what circumstances will the hint be set, with a present subsequent PTE but !folio_test_large()?
Quoted 7 lines
I guess the hint might not be taken? But then isn't the valid check just
folio_test_large()and we don't need this batched check at all?Is it only to avoid the split THP case?
We definitely need some clarity here, and a comment in the code explaining what's going on as this is subtle stuff.
I am focussed only on batching large folios. Split THPs won't be batched; you can use pte_batch() (from David's refactoring) and
figure the split THP batch out, but then get_and_clear_full_ptes()
will be gathering a/d bits and smearing them across the batch, which will be incorrect. Even if we introduce a new version of get_and_clear_full_ptes() which does not gather a/d bits, then if the pte_batch actually belongs to a folio, then we will *not* be smearing a/d bits, which is again wrong. So in any case we must know what the underlying folio looks like :) So my agenda for v3 is,
- Incorporate your refactoring comments
- Remove maybe_contiguous_pte_pfns and just use vm_normal_folio + folio_test_large
- Fix indentation
Sounds good?
Quoted 115 lines
Quoted 25 lines
mm/mremap.c +5
-1+return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 258
Quoted 8 lines
mm/mremap.c +2
-3@@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) {Really nitty thing here but the indentation is all messed up here, I mean nothing is going to be nice but maybe indent by two tabs below 'for'.
I'm not a fan of this declaration of nr, typically in a for loop a declaration here would be the counter, so this is just confusing.
In the old implementation, declaring nr in the for loop would make sense, but in the newly refactored one you should just declare it at the top.
Also as per Anshuman review, I think
nr_ptes,max_nr_pteswould be better.I don't think 'nr' needs to be initialised either, since the conditional is '
old_addr<old_end' and you _should_ only perform themm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 261
Quoted 3 lines
mm/mremap.c +3
-0+ max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))This seems broken.
You're missing a nr assignment here, so you'll happen to offset by the number of pages of the last folio you encountered?
Should be:
if (pte_none(old_pte)) { nr_ptes = 1; continue; }Or, alternatively, you can reset
nr_ptesto 1 at the start of each loop.Quoted 8 lines
mm/mremap.c +0
-1continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 279
Quoted 9 lines
mm/mremap.c +4
-1@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + }Thanks this is much clearer compared to v1
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 280
Quoted 1 line
mm/mremap.c +1
-0+ pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 );Nit but...
Can we have a comment indicating what the last parameter refers to? I think David maybe doens't like this so obviously if he prefers not that fine, but I'm thinking something like:
pte =
get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, /*full=*/false);I think we are good to just use 'false' here right? As it's only an int for historical purposes...
Quoted 15 lines
mm/mremap.c +1
-2pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } } --2.30.2
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 176 lines · Dev Jain
Quoted 165 lines · Lorenzo Stoakes
Before getting into the review, just to say thanks for refactoring as per my (and of course other's) comments, much appreciated and big improvement! :)
We're getting there...
Quoted 47 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +17
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte;include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 398
Quoted 10 lines
include/linux/pgtable.h +10
-0+ +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 );Let's not do unlikely()'s unless we have data for them... it shouldn't mean 'what the programmer believes' :)
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 399
Quoted 1 line
include/linux/pgtable.h +1
-0+}Yeah I'm with Andrew and Anshuman, I mean this is kind of a nasty interface (I mean _perhaps_ unavoidably) and we've done the relevant check in
mremap_folio_pte_batch(), so let's just move it there with comments, as thisQuoted 16 lines
include/linux/pgtable.h +1
-0+#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +4
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 186
Quoted 10 lines
mm/mremap.c +10
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + }This needs some refactoring, avoid nesting at all costs :)
We'll want to move the
maybe_contiguous_pte_pfns()function over here, so that'll change things, but in general let's use a guard clause.So an if block like:
if (foo) { ... bunch of logic ... }Is better replaced with a guard clause so you have:
if (!foo)
return ...;... bunch of logic ...
Here we could really expand things out to make things SUPER clear like:
static int mremap_folio_pte_batch(struct
vm_area_struct*vma, unsigned long addr,
pte_t*ptep,pte_tpte, intmax_nr){ const fpb_t flags; struct folio *folio; if (max_nr == 1) return 1; if (!maybe_contiguous_pte_pfns(ptep, pte)) // obviously replace with open code... return 1; folio = vm_normal_folio(vma, addr, pte); if (!folio) return 1; if (!folio_test_large(folio)) return 1; flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; return folio_pte_batch(folio, addr, ptep, pte, max_nr, flags, NULL, NULL, NULL); }I mean you could argue assign nr would be neater here, but you get the point!
David mentioned a point about this code over in v1 discussion (see [0]). Trying to bring converastion here to avoid it being split across old/new series. There he said:
David H:
Quoted 3 lines
(2) Do we really need "must be part of the same folio", or could be just batch over present ptes that map consecutive PFNs? In that case, a helper that avoids
folio_pte_batch()completely might be better.Hm, if we didn't do the batch test, can we batch a split large folio here ok? I'm guessing we can in which case this check is actually limiting...
Are we _explicitly_ only considering the cont pte case and ignoring the split THP case?
[0]: https://lore.kernel.org/all/[email protected]/
And in what circumstances will the hint be set, with a present subsequent PTE but !folio_test_large()?
Quoted 7 lines
I guess the hint might not be taken? But then isn't the valid check just
folio_test_large()and we don't need this batched check at all?Is it only to avoid the split THP case?
We definitely need some clarity here, and a comment in the code explaining what's going on as this is subtle stuff.
Quoted 9 lines
I am focussed only on batching large folios. Split THPs won't be batched; you can use
pte_batch()(from David's refactoring) and
figure the split THP batch out, but thenget_and_clear_full_ptes()will be gathering a/d bits and smearing them across the batch, which will be incorrect. Even if we introduce a new version ofget_and_clear_full_ptes()which does not gather a/d bits, then if thepte_batchactually belongs to a folio, then we will *not* be smearing a/d bits, which is again wrong. So in any case we must know what the underlying folio looks like :) So my agenda for v3 is,
Right, ack, there being a large folio per se doesn't mean A/D collection is appropriate in THP case.
I guess then this is really _only_ about the mTHP case, where essentially the other PTEs are to be disregarded going forwards?
Quoted 7 lines
- Incorporate your refactoring comments
- Removemaybe_contiguous_pte_pfnsand just usevm_normal_folio+folio_test_large
- Fix indentationSounds good?
Sure, but can we hold off until the mprotect() stuff is done first please? I mean obviously you're free to do things as you like, but this will help workload-wise on my side :>)
Thanks!
Quoted 117 lines
Quoted 116 lines
Quoted 26 lines
mm/mremap.c +5
-1+return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 258
Quoted 8 lines
mm/mremap.c +2
-3@@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) {Really nitty thing here but the indentation is all messed up here, I mean nothing is going to be nice but maybe indent by two tabs below 'for'.
I'm not a fan of this declaration of nr, typically in a for loop a declaration here would be the counter, so this is just confusing.
In the old implementation, declaring nr in the for loop would make sense, but in the newly refactored one you should just declare it at the top.
Also as per Anshuman review, I think
nr_ptes,max_nr_pteswould be better.I don't think 'nr' needs to be initialised either, since the conditional is '
old_addr<old_end' and you _should_ only perform themm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 261
Quoted 3 lines
mm/mremap.c +3
-0+ max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))This seems broken.
You're missing a nr assignment here, so you'll happen to offset by the number of pages of the last folio you encountered?
Should be:
if (pte_none(old_pte)) { nr_ptes = 1; continue; }Or, alternatively, you can reset
nr_ptesto 1 at the start of each loop.Quoted 8 lines
mm/mremap.c +0
-1continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 279
Quoted 9 lines
mm/mremap.c +4
-1@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + }Thanks this is much clearer compared to v1
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 280
Quoted 1 line
mm/mremap.c +1
-0+ pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 );Nit but...
Can we have a comment indicating what the last parameter refers to? I think David maybe doens't like this so obviously if he prefers not that fine, but I'm thinking something like:
pte =
get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, /*full=*/false);I think we are good to just use 'false' here right? As it's only an int for historical purposes...
Quoted 15 lines
mm/mremap.c +1
-2pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } } --2.30.2
Re: [PATCH v2 2/2] mm: Optimize mremap() by PTE batching
Quoted 187 lines · Lorenzo Stoakes
Quoted 176 lines · Dev Jain
Quoted 165 lines · Lorenzo Stoakes
Before getting into the review, just to say thanks for refactoring as per my (and of course other's) comments, much appreciated and big improvement! :)
We're getting there...
Quoted 47 lines · Dev Jain
To use PTE batching, we want to determine whether the folio mapped by the PTE is large, thus requiring the use of
vm_normal_folio(). We want to avoid the cost ofvm_normal_folio()if the code path doesn't already require the folio. For arm64,pte_batch_hint()does the job. To generalize this hint, add a helper which will determine whether two consecutive PTEs point to consecutive PFNs, in which case there is a high probability that the underlying folio is large.
Next, usefolio_pte_batch()to optimizemove_ptes(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_get_and_clear()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/mremap.c | 37 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 7 deletions(-)include/linux/pgtable.h +17
-0@@ -369,6 +369,35 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) }#endif +/** +* maybe_contiguous_pte_pfns - Hint whether the page mapped by the pte belongs +* to a large folio. +* @ptep: Pointer to the page table entry. +* @pte: The page table entry. +* +* This helper is invoked when the caller wants to batch over a set of ptes +* mapping a large folio, but the concerned code path does not already have +* the folio. We want to avoid the cost of vm_normal_folio() only to find that +* the underlying folio was small; i.e keep the small folio case as fast as +* possible. +* +* The caller must ensure that ptep + 1 exists. +*/ +static inline bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte;include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 398
Quoted 10 lines
include/linux/pgtable.h +10
-0+ +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); +if (!pte_present (next_pte)) +return false; + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==1 );Let's not do unlikely()'s unless we have data for them... it shouldn't mean 'what the programmer believes' :)
include/linux/pgtable.h static inline pgd_t pgdp_get(pgd_t *pgdp) · line 399
Quoted 1 line
include/linux/pgtable.h +1
-0+}Yeah I'm with Andrew and Anshuman, I mean this is kind of a nasty interface (I mean _perhaps_ unavoidably) and we've done the relevant check in
mremap_folio_pte_batch(), so let's just move it there with comments, as thisQuoted 16 lines
include/linux/pgtable.h +1
-0+#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address,mm/mremap.c +4
-0@@ -170,6 +170,23 @@ static pte_t move_soft_dirty_pte(pte_t pte)return pte; } +/* mremap a batch of PTEs mapping the same large folio */ +static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t pte,int max_nr) +{mm/mremap.c static pte_t move_soft_dirty_pte(pte_t pte) · line 186
Quoted 10 lines
mm/mremap.c +10
-0+const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ; +struct folio *folio; +int nr =1 ; + +if ((max_nr !=1 ) &&maybe_contiguous_pte_pfns (ptep, pte)) { + folio =vm_normal_folio (vma, addr, pte); +if (folio &&folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, + flags,NULL ,NULL ,NULL ); + }This needs some refactoring, avoid nesting at all costs :)
We'll want to move the
maybe_contiguous_pte_pfns()function over here, so that'll change things, but in general let's use a guard clause.So an if block like:
if (foo) { ... bunch of logic ... }Is better replaced with a guard clause so you have:
if (!foo)
return ...;... bunch of logic ...
Here we could really expand things out to make things SUPER clear like:
static int mremap_folio_pte_batch(struct
vm_area_struct*vma, unsigned long addr,
pte_t*ptep,pte_tpte, intmax_nr){ const fpb_t flags; struct folio *folio; if (max_nr == 1) return 1; if (!maybe_contiguous_pte_pfns(ptep, pte)) // obviously replace with open code... return 1; folio = vm_normal_folio(vma, addr, pte); if (!folio) return 1; if (!folio_test_large(folio)) return 1; flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; return folio_pte_batch(folio, addr, ptep, pte, max_nr, flags, NULL, NULL, NULL); }I mean you could argue assign nr would be neater here, but you get the point!
David mentioned a point about this code over in v1 discussion (see [0]). Trying to bring converastion here to avoid it being split across old/new series. There he said:
David H:
Quoted 3 lines
(2) Do we really need "must be part of the same folio", or could be just batch over present ptes that map consecutive PFNs? In that case, a helper that avoids
folio_pte_batch()completely might be better.Hm, if we didn't do the batch test, can we batch a split large folio here ok? I'm guessing we can in which case this check is actually limiting...
Are we _explicitly_ only considering the cont pte case and ignoring the split THP case?
[0]: https://lore.kernel.org/all/[email protected]/
And in what circumstances will the hint be set, with a present subsequent PTE but !folio_test_large()?
Quoted 7 lines
I guess the hint might not be taken? But then isn't the valid check just
folio_test_large()and we don't need this batched check at all?Is it only to avoid the split THP case?
We definitely need some clarity here, and a comment in the code explaining what's going on as this is subtle stuff.
Quoted 9 lines
I am focussed only on batching large folios. Split THPs won't be batched; you can use
pte_batch()(from David's refactoring) and
figure the split THP batch out, but thenget_and_clear_full_ptes()will be gathering a/d bits and smearing them across the batch, which will be incorrect. Even if we introduce a new version ofget_and_clear_full_ptes()which does not gather a/d bits, then if thepte_batchactually belongs to a folio, then we will *not* be smearing a/d bits, which is again wrong. So in any case we must know what the underlying folio looks like :) So my agenda for v3 is,
Quoted 5 lines
Right, ack, there being a large folio per se doesn't mean A/D collection is appropriate in THP case.
I guess then this is really _only_ about the mTHP case, where essentially the other PTEs are to be disregarded going forwards?
Yes.
Quoted 7 lines
Quoted 7 lines
- Incorporate your refactoring comments
- Removemaybe_contiguous_pte_pfnsand just usevm_normal_folio+folio_test_large
- Fix indentation
Quoted 7 lines
Quoted 1 line
Sounds good?
Sure, but can we hold off until the
mprotect()stuff is done first please? I mean obviously you're free to do things as you like, but this will help workload-wise on my side :>)Thanks!
No problem, thanks for reviewing!
Quoted 118 lines
Quoted 118 lines
Quoted 117 lines
Quoted 27 lines
mm/mremap.c +5
-1+return nr; +} +static int move_ptes (struct pagetable_move_control *pmc,unsigned long extent,pmd_t *old_pmd,pmd_t *new_pmd) {@@ -177,7 +194,7 @@ static int move_ptes(struct pagetable_move_control *pmc,bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm;pte_t *old_ptep, *new_ptep; -pte_t pte; +pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;bool force_flush = false;@@ -186,6 +203,7 @@ static int move_ptes(struct pagetable_move_control *pmc,unsigned long old_end = old_addr + extent;unsigned long len = old_end - old_addr;int err =0 ; +int max_nr;/* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 258
Quoted 8 lines
mm/mremap.c +2
-3@@ -236,12 +254,13 @@ static int move_ptes(struct pagetable_move_control *pmc,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); -for (; old_addr < old_end; old_ptep++, old_addr +=PAGE_SIZE , - new_ptep++, new_addr +=PAGE_SIZE ) { -if (pte_none (ptep_get (old_ptep))) +for (int nr =1 ; old_addr < old_end; old_ptep += nr, old_addr += nr *PAGE_SIZE , + new_ptep += nr, new_addr += nr *PAGE_SIZE ) {Really nitty thing here but the indentation is all messed up here, I mean nothing is going to be nice but maybe indent by two tabs below 'for'.
I'm not a fan of this declaration of nr, typically in a for loop a declaration here would be the counter, so this is just confusing.
In the old implementation, declaring nr in the for loop would make sense, but in the newly refactored one you should just declare it at the top.
Also as per Anshuman review, I think
nr_ptes,max_nr_pteswould be better.I don't think 'nr' needs to be initialised either, since the conditional is '
old_addr<old_end' and you _should_ only perform themm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 261
Quoted 3 lines
mm/mremap.c +3
-0+ max_nr = (old_end - old_addr) >>PAGE_SHIFT ; + old_pte =ptep_get (old_ptep); +if (pte_none (old_pte))This seems broken.
You're missing a nr assignment here, so you'll happen to offset by the number of pages of the last folio you encountered?
Should be:
if (pte_none(old_pte)) { nr_ptes = 1; continue; }Or, alternatively, you can reset
nr_ptesto 1 at the start of each loop.Quoted 8 lines
mm/mremap.c +0
-1continue ; - pte =ptep_get_and_clear (mm, old_addr, old_ptep);/* * If we are remapping a valid PTE, make sure * to flush TLB before we drop the PTL for the
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 279
Quoted 9 lines
mm/mremap.c +4
-1@@ -253,8 +272,12 @@ static int move_ptes(struct pagetable_move_control *pmc,* the TLB entry for the old mapping has been * flushed. */ -if (pte_present (pte)) +if (pte_present (old_pte)) { + nr =mremap_folio_pte_batch (vma, old_addr, old_ptep, + old_pte, max_nr); force_flush = true; + }Thanks this is much clearer compared to v1
mm/mremap.c static int move_ptes(struct pagetable_move_control *pmc, · line 280
Quoted 1 line
mm/mremap.c +1
-0+ pte =get_and_clear_full_ptes (mm, old_addr, old_ptep, nr,0 );Nit but...
Can we have a comment indicating what the last parameter refers to? I think David maybe doens't like this so obviously if he prefers not that fine, but I'm thinking something like:
pte =
get_and_clear_full_ptes(mm, old_addr, old_ptep, nr, /*full=*/false);I think we are good to just use 'false' here right? As it's only an int for historical purposes...
Quoted 15 lines
mm/mremap.c +1
-2pte =move_pte (pte, old_addr, new_addr); pte =move_soft_dirty_pte (pte);@@ -267,7 +290,7 @@ static int move_ptes(struct pagetable_move_control *pmc,else if (is_swap_pte (pte)) pte =pte_swp_clear_uffd_wp (pte); } -set_pte_at (mm, new_addr, new_ptep, pte); +set_ptes (mm, new_addr, new_ptep, pte, nr); } } --2.30.2
Dev - a general comment here - but let's slow things down a little please :)
The mprotect() version of this is still outstanding fixes and likely will need quite a bit of checking before we can ensure it's stabilised.
And now we have this mremap() series as well which also has had quite a few quite significant issues that have needed addressing.
So can we try to focus on one at a time, and really try to nail down the series before moving on to the next?
We also have outstanding review on the v1, which has now been split, which does happen sometimes but perhaps suggests that it'd work better if you waited a couple days or such to ensure things are settled before sending a new version when there's quite a bit of feedback?
This isn't a criticism really, sorry I don't mean to sound negative or such
- but this is more a process thing so we reviewers can keep up with things, keep things rolling, and ensure you get your changes merged asap :)
Thanks, Lorenzo
Quoted 64 lines · Dev Jain
Currently
move_ptes()iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch usingfolio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number ofptep_get()calls (since on a contig block,ptep_get()on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs throughget_and_clear_full_ptes, replacingptep_get_and_clear.Mapping 512K of memory, memsetting it, remapping it to src + 512K, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).
Test program for reference:
#define
_GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>#define SIZE (1UL << 20) // 512 KB
int main(void) { void *new_addr, *addr; for (int i = 0; i < 10000; ++i) { addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 0xAA, SIZE); new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE); if (new_addr != (addr + SIZE)) { perror("mremap"); return 1; } munmap(new_addr, SIZE); }}
v1->v2:
- Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducingmremap_folio_pte_batch(), fix loop iteration (Lorenzo)
- Merge patch 2 and 3 (Anshuman, Lorenzo)
- Fixmaybe_contiguous_pte_pfns(Willy)Dev Jain (2):
mm: Call pointers to ptes as ptep
mm: Optimizemremap()by PTE batchinginclude/linux/pgtable.h | 29 ++++++++++++++++++++++ mm/mremap.c | 54 +++++++++++++++++++++++++++++------------ 2 files changed, 68 insertions(+), 15 deletions(-)--
2.30.2
Quoted 9 lines · Lorenzo Stoakes
Dev - a general comment here - but let's slow things down a little please :)
The
mprotect()version of this is still outstanding fixes and likely will need quite a bit of checking before we can ensure it's stabilised.And now we have this
mremap()series as well which also has had quite a few quite significant issues that have needed addressing.
Quoted 7 lines
So can we try to focus on one at a time, and really try to nail down the series before moving on to the next?
We also have outstanding review on the v1, which has now been split, which does happen sometimes but perhaps suggests that it'd work better if you waited a couple days or such to ensure things are settled before sending a new version when there's quite a bit of feedback?
Sure, I should have waited my bad, I usually do, this time I was in a haste with both series for no reason :( thanks for your detailed replies btw!
Quoted 72 lines
This isn't a criticism really, sorry I don't mean to sound negative or such
- but this is more a process thing so we reviewers can keep up with things, keep things rolling, and ensure you get your changes merged asap :)Thanks, Lorenzo
Quoted 64 lines · Dev Jain
Currently
move_ptes()iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch usingfolio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number ofptep_get()calls (since on a contig block,ptep_get()on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs throughget_and_clear_full_ptes, replacingptep_get_and_clear.Mapping 512K of memory, memsetting it, remapping it to src + 512K, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).
Test program for reference:
#define
_GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>#define SIZE (1UL << 20) // 512 KB
int main(void) { void *new_addr, *addr; for (int i = 0; i < 10000; ++i) { addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 0xAA, SIZE); new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE); if (new_addr != (addr + SIZE)) { perror("mremap"); return 1; } munmap(new_addr, SIZE); }}
v1->v2:
- Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducingmremap_folio_pte_batch(), fix loop iteration (Lorenzo)
- Merge patch 2 and 3 (Anshuman, Lorenzo)
- Fixmaybe_contiguous_pte_pfns(Willy)Dev Jain (2):
mm: Call pointers to ptes as ptep
mm: Optimizemremap()by PTE batchinginclude/linux/pgtable.h | 29 ++++++++++++++++++++++ mm/mremap.c | 54 +++++++++++++++++++++++++++++------------ 2 files changed, 68 insertions(+), 15 deletions(-)--
2.30.2
Quoted 15 lines · Dev Jain
Quoted 12 lines · Lorenzo Stoakes
Dev - a general comment here - but let's slow things down a little please :)
The
mprotect()version of this is still outstanding fixes and likely will need quite a bit of checking before we can ensure it's stabilised.And now we have this
mremap()series as well which also has had quite a few quite significant issues that have needed addressing.So can we try to focus on one at a time, and really try to nail down the series before moving on to the next?
Quoted 7 lines
Quoted 4 lines
We also have outstanding review on the v1, which has now been split, which does happen sometimes but perhaps suggests that it'd work better if you waited a couple days or such to ensure things are settled before sending a new version when there's quite a bit of feedback?
Sure, I should have waited my bad, I usually do, this time I was in a haste with both series for no reason :( thanks for your detailed replies btw!
No problem, I've done this in the past myself haha (sometimes even faster than you did :P), just a case of tweaking things to fit the mm process :)
Cheers, Lorenzo
Quoted 74 lines
Quoted 73 lines
This isn't a criticism really, sorry I don't mean to sound negative or such
- but this is more a process thing so we reviewers can keep up with things, keep things rolling, and ensure you get your changes merged asap :)Thanks, Lorenzo
Quoted 64 lines · Dev Jain
Currently
move_ptes()iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch usingfolio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number ofptep_get()calls (since on a contig block,ptep_get()on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs throughget_and_clear_full_ptes, replacingptep_get_and_clear.Mapping 512K of memory, memsetting it, remapping it to src + 512K, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64).
Test program for reference:
#define
_GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>#define SIZE (1UL << 20) // 512 KB
int main(void) { void *new_addr, *addr; for (int i = 0; i < 10000; ++i) { addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 0xAA, SIZE); new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE); if (new_addr != (addr + SIZE)) { perror("mremap"); return 1; } munmap(new_addr, SIZE); }}
v1->v2:
- Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducingmremap_folio_pte_batch(), fix loop iteration (Lorenzo)
- Merge patch 2 and 3 (Anshuman, Lorenzo)
- Fixmaybe_contiguous_pte_pfns(Willy)Dev Jain (2):
mm: Call pointers to ptes as ptep
mm: Optimizemremap()by PTE batchinginclude/linux/pgtable.h | 29 ++++++++++++++++++++++ mm/mremap.c | 54 +++++++++++++++++++++++++++++------------ 2 files changed, 68 insertions(+), 15 deletions(-)--
2.30.2
Quoted 21 lines · Lorenzo Stoakes
Quoted 20 lines · Dev Jain
Quoted 16 lines · Lorenzo Stoakes
Dev - a general comment here - but let's slow things down a little please :)
The
mprotect()version of this is still outstanding fixes and likely will need quite a bit of checking before we can ensure it's stabilised.And now we have this
mremap()series as well which also has had quite a few quite significant issues that have needed addressing.So can we try to focus on one at a time, and really try to nail down the series before moving on to the next?
We also have outstanding review on the v1, which has now been split, which does happen sometimes but perhaps suggests that it'd work better if you waited a couple days or such to ensure things are settled before sending a new version when there's quite a bit of feedback?
Quoted 5 lines
Quoted 2 lines
Sure, I should have waited my bad, I usually do, this time I was in a haste with both series for no reason :( thanks for your detailed replies btw!
No problem, I've done this in the past myself haha (sometimes even faster than you did :P), just a case of tweaking things to fit the mm process :)
We've all been there, desperately wanting to flush the todo list ... :)
Signature · 3 lines
Cheers, David / dhildenb