Optimize mprotect for large folios
Subsystems (6)
Dev Jain
reviewerv2 cover Optimize mprotect for large folios
This patchset optimizes the mprotect() system call for large folios by PTE-batching.
We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pages
Average execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 seconds
After the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 seconds
Observing T1/T2 and T3 before the patchset, we also remove the regression introduced by ptep_get() on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf: util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)
Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimize mprotect() by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version of ptep_modify_prot_start
arm64: Add batched version of ptep_modify_prot_commit
mm: Batch around can_change_pte_writable()
mm: Optimize mprotect() through PTE-batching
arch/arm64/include/asm/pgtable.h | 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)
Signature · 1 line
2.30.2
Dev Jain
reviewerPATCH v2 1/7 mm: Refactor code in mprotect
Reduce indentation in change_pte_range() by refactoring some of the code into a new function. No functional change.
--- mm/mprotect.c | 116 +++++++++++++++++++++++++++++--------------------- 1 file changed, 68 insertions(+), 48 deletions(-)
mm/mprotect.c +68 -48
@@ -83,6 +83,71 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } + + +static bool prot_numa_skip (struct vm_area_struct *vma,struct folio *folio, +int target_node) +{ +bool toptier; +int nid; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) || +folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) && +folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && + toptier) +return true; +return false; +} + +static bool prot_numa_avoid_fault (struct vm_area_struct *vma, +unsigned long addr,pte_t oldpte,int target_node) +{ +struct folio *folio; +int ret; + +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio ||folio_is_zone_device (folio) || +folio_test_ksm (folio)) +return true; + ret =prot_numa_skip (vma, folio, target_node); +if (ret) +return ret; +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio, +jiffies_to_msecs (jiffies)); +return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)
@@ -116,56 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,* Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ -if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) +if (prot_numa && +prot_numa_avoid_fault (vma, addr, + oldpte, target_node))continue ; -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) -continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); - } - oldpte =ptep_modify_prot_start (vma, addr, pte); ptent =pte_modify (oldpte, newprot);
Signature · 1 line
2.30.2
Re: [PATCH v2 1/7] mm: Refactor code in mprotect
Quoted 63 lines · Dev Jain
Reduce indentation in
change_pte_range()by refactoring some of the code into a new function. No functional change.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 116 +++++++++++++++++++++++++++++--------------------- 1 file changed, 68 insertions(+), 48 deletions(-)mm/mprotect.c +47
-0@@ -83,6 +83,71 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } + + +static bool prot_numa_skip (struct vm_area_struct *vma,struct folio *folio, +int target_node) +{ +bool toptier; +int nid; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) || +folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) && +folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && + toptier) +return true; +return false; +} + +static bool prot_numa_avoid_fault (struct vm_area_struct *vma, +unsigned long addr,pte_t oldpte,int target_node) +{ +struct folio *folio; +int ret;
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 142
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio ||folio_is_zone_device (folio) || +folio_test_ksm (folio)) +return true; + ret =prot_numa_skip (vma, folio, target_node);
What purpose does it solve to create additional helper prot_numa_skip(). IOW - why cannot all of these be inside prot_numa_avoid_fault() itself.
Quoted 71 lines
mm/mprotect.c +11
-48+if (ret) +return ret; +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio, +jiffies_to_msecs (jiffies)); +return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -116,56 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,* Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ -if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) +if (prot_numa && +prot_numa_avoid_fault (vma, addr, + oldpte, target_node))continue ; -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) -continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); - } - oldpte =ptep_modify_prot_start (vma, addr, pte); ptent =pte_modify (oldpte, newprot);
Dev Jain
reviewerRe: [PATCH v2 1/7] mm: Refactor code in mprotect
Quoted 71 lines · Anshuman Khandual
Quoted 68 lines · Dev Jain
Reduce indentation in
change_pte_range()by refactoring some of the code into a new function. No functional change.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 116 +++++++++++++++++++++++++++++--------------------- 1 file changed, 68 insertions(+), 48 deletions(-)mm/mprotect.c +52
-0@@ -83,6 +83,71 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } + + +static bool prot_numa_skip (struct vm_area_struct *vma,struct folio *folio, +int target_node) +{ +bool toptier; +int nid; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) || +folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) && +folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && + toptier) +return true; +return false; +} + +static bool prot_numa_avoid_fault (struct vm_area_struct *vma, +unsigned long addr,pte_t oldpte,int target_node) +{ +struct folio *folio; +int ret; + +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; +
Quoted 8 lines
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 142
Quoted 5 lines
mm/mprotect.c +5
-0+ folio =vm_normal_folio (vma, addr, oldpte); +if (!folio ||folio_is_zone_device (folio) || +folio_test_ksm (folio)) +return true; + ret =prot_numa_skip (vma, folio, target_node);What purpose does it solve to create additional helper
prot_numa_skip(). IOW - why cannot all of these be insideprot_numa_avoid_fault()itself.
That is in preparation for patch 2, so that I can skip a PTE batch.
Quoted 72 lines
Quoted 72 lines
mm/mprotect.c +11
-48+if (ret) +return ret; +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio, +jiffies_to_msecs (jiffies)); +return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -116,56 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,* Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ -if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) +if (prot_numa && +prot_numa_avoid_fault (vma, addr, + oldpte, target_node))continue ; -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) -continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); - } - oldpte =ptep_modify_prot_start (vma, addr, pte); ptent =pte_modify (oldpte, newprot);
Re: [PATCH v2 1/7] mm: Refactor code in mprotect
In reply to Dev JainFor changes like this, difftastic comes in very handy :)
Quoted 4 lines · Dev Jain
Reduce indentation in
change_pte_range()by refactoring some of the code into a new function. No functional change.Signed-off-by Dev Jain <[email protected]>
Overall a big fan of the intent of this patch! This is a nice cleanup, just need to nail down details.
Quoted 9 lines
--- mm/mprotect.c | 116 +++++++++++++++++++++++++++++--------------------- 1 file changed, 68 insertions(+), 48 deletions(-)mm/mprotect.c +0
-0
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 87
Quoted 5 lines
mm/mprotect.c +2
-0@@ -83,6 +83,71 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } + +
Nit: stray extra newline.
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 89
Quoted 2 lines
mm/mprotect.c +2
-0+static bool prot_numa_skip (struct vm_area_struct *vma,struct folio *folio, +int target_node)
This is a bit weird, it's like you have two functions to determine whether to skip a PTE entry, but named differently?
I think you say in response to a comment elsewhere that you intend to further split things up in subsequent patches, but this kinda bugs me as subjective as it is :)
I'd say rename prot_numa_avoid_fault() -> can_skip_prot_numa_pte()
And this to can_skip_prot_numa_folio()?
Then again, the below function does some folio stuff too, so I'm not sure exactly what the separation is? Can you explain?
Also it'd be good to add some brief comment, something like 'the prot_numa change-prot (cp) flag indicates that this protection change is due to NUMA hinting, we determine if we actually have work to do or can skip this folio entirely'.
Or equivalent in the below function.
Quoted 45 lines
mm/mprotect.c +45
-0+{ +bool toptier; +int nid; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) || +folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) && +folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && + toptier) +return true; +return false; +} + +static bool prot_numa_avoid_fault (struct vm_area_struct *vma, +unsigned long addr,pte_t oldpte,int target_node) +{ +struct folio *folio; +int ret; + +/* Avoid TLB flush if possible */
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 144
Quoted 10 lines
mm/mprotect.c +10
-0+if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio ||folio_is_zone_device (folio) || +folio_test_ksm (folio)) +return true; + ret =prot_numa_skip (vma, folio, target_node); +if (ret) +return ret;
This is a bit silly as it returns a boolean value, surely;
if (prot_numa_skip(vma, folio, target_node)) return true;
Is better?
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 147
Quoted 3 lines
mm/mprotect.c +3
-0+if (folio_use_access_time (folio)) +folio_xchg_access_time (folio, +jiffies_to_msecs (jiffies));
Why is this here and not in prot_numa_skip() or whatever we rename it to?
Quoted 69 lines
mm/mprotect.c +6
-49+return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -116,56 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,* Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ -if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) +if (prot_numa && +prot_numa_avoid_fault (vma, addr, + oldpte, target_node))continue ; -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) -continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); - } - oldpte =ptep_modify_prot_start (vma, addr, pte); ptent =pte_modify (oldpte, newprot); --2.30.2
Dev Jain
reviewerPATCH v2 2/7 mm: Optimize mprotect() by batch-skipping PTEs
In reply to Dev JainIn case of prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.
--- mm/mprotect.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
mm/mprotect.c +20 -7
@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,bool toptier;int nid; +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; +/* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) ||
@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, }static bool prot_numa_avoid_fault (struct vm_area_struct *vma, -unsigned long addr,pte_t oldpte,int target_node) +unsigned long addr,pte_t *pte,pte_t oldpte,int target_node, +int max_nr,int *nr) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;struct folio *folio;int ret;
@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,return true; folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) +if (!folio)return true; + ret =prot_numa_skip (vma, folio, target_node); -if (ret) +if (ret) { +if (folio_test_large (folio) && max_nr !=1 ) + *nr =folio_pte_batch (folio, addr, pte, oldpte, + max_nr, flags,NULL ,NULL ,NULL );return ret; + }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));
@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);
@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr = (end - addr) >> PAGE_SHIFT; pte_t ptent; /*
@@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa && -prot_numa_avoid_fault (vma, addr, - oldpte, target_node)) +prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, + max_nr, &nr))continue ; oldpte =ptep_modify_prot_start (vma, addr, pte);
@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr, addr += nr *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl);
Signature · 1 line
2.30.2
Re: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
Quoted 14 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 96
Quoted 6 lines
mm/mprotect.c +3
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,bool toptier;int nid; +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; +
Moving these here from prot_numa_avoid_fault() could have been done earlier, while adding prot_numa_skip() itself in the previous patch (in case this helper is determined to be really required).
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 135
Quoted 12 lines
mm/mprotect.c +3
-1/* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) ||@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, }static bool prot_numa_avoid_fault (struct vm_area_struct *vma, -unsigned long addr,pte_t oldpte,int target_node) +unsigned long addr,pte_t *pte,pte_t oldpte,int target_node, +int max_nr,int *nr) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;
Flags are all correct.
Quoted 6 lines
mm/mprotect.c +0
-0struct folio *folio;int ret;@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,return true;
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 149
Quoted 10 lines
mm/mprotect.c +4
-3folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) +if (!folio)return true; + ret =prot_numa_skip (vma, folio, target_node); -if (ret) +if (ret) { +if (folio_test_large (folio) && max_nr !=1 )
Conditional checks are all correct.
Quoted 16 lines
mm/mprotect.c +4
-0+ *nr =folio_pte_batch (folio, addr, pte, oldpte, + max_nr, flags,NULL ,NULL ,NULL );return ret; + }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 186
Quoted 4 lines
mm/mprotect.c +1
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr = 1;
'nr' resets each iteration.
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 3 lines
mm/mprotect.c +1
-0oldpte =ptep_get (pte);if (pte_present (oldpte)) { +int max_nr = (end - addr) >>PAGE_SHIFT ;
Small nit - 'max_nr' declaration could be moved earlier along with 'nr'.
Quoted 16 lines
mm/mprotect.c +3
-2pte_t ptent;/* @@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa && -prot_numa_avoid_fault (vma, addr, - oldpte, target_node)) +prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, + max_nr, &nr))continue ; oldpte =ptep_modify_prot_start (vma, addr, pte);
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 319
Quoted 8 lines
mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr, addr += nr *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl);
Otherwise LGTM
Dev Jain
reviewerRe: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
Quoted 18 lines · Anshuman Khandual
Quoted 17 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,bool toptier;int nid;
Quoted 7 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 96
Quoted 3 lines
mm/mprotect.c +3
-0+if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; +Moving these here from
prot_numa_avoid_fault()could have been done earlier, while addingprot_numa_skip()itself in the previous patch (in case this helper is determined to be really required).
True. I'll do that.
Quoted 58 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 135
Quoted 13 lines
mm/mprotect.c +3
-1/* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) ||@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, }static bool prot_numa_avoid_fault (struct vm_area_struct *vma, -unsigned long addr,pte_t oldpte,int target_node) +unsigned long addr,pte_t *pte,pte_t oldpte,int target_node, +int max_nr,int *nr) { +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;Flags are all correct.
Quoted 6 lines
mm/mprotect.c +0
-0struct folio *folio;int ret;@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,return true;mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 149
Quoted 10 lines
mm/mprotect.c +4
-3folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) +if (!folio)return true; + ret =prot_numa_skip (vma, folio, target_node); -if (ret) +if (ret) { +if (folio_test_large (folio) && max_nr !=1 )Conditional checks are all correct.
Quoted 16 lines
mm/mprotect.c +4
-0+ *nr =folio_pte_batch (folio, addr, pte, oldpte, + max_nr, flags,NULL ,NULL ,NULL );return ret; + }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 186
Quoted 4 lines
mm/mprotect.c +1
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr = 1;'nr' resets each iteration.
Quoted 5 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 3 lines
mm/mprotect.c +1
-0oldpte =ptep_get (pte);if (pte_present (oldpte)) { +int max_nr = (end - addr) >>PAGE_SHIFT ;Small nit - '
max_nr' declaration could be moved earlier along with 'nr'.
Sure.
Quoted 27 lines
Quoted 17 lines
mm/mprotect.c +3
-2pte_t ptent;/* @@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa && -prot_numa_avoid_fault (vma, addr, - oldpte, target_node)) +prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, + max_nr, &nr))continue ; oldpte =ptep_modify_prot_start (vma, addr, pte);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 319
Quoted 8 lines
mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr, addr += nr *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl);Otherwise LGTM
Re: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
In reply to Dev JainVery very very nitty on subject (sorry I realise this is annoying :P) - generally don't need to capitalise 'Optimize' here :>)
Generally I like the idea here. But some issues on impl.
Quoted 14 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 96
Quoted 6 lines
mm/mprotect.c +3
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,bool toptier;int nid; +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; +
Hm why not just put this here from the start? I think you should put this back in the prior commit.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 133
Quoted 10 lines
mm/mprotect.c +2
-1/* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) ||@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, }static bool prot_numa_avoid_fault (struct vm_area_struct *vma, -unsigned long addr,pte_t oldpte,int target_node) +unsigned long addr,pte_t *pte,pte_t oldpte,int target_node, +int max_nr,int *nr)
Hate this ptr to nr.
Why not just return nr, if it's 0 then skip? Simple!
Quoted 6 lines
mm/mprotect.c +1
-0{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;struct folio *folio;int ret;
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 146
Quoted 8 lines
mm/mprotect.c +2
-2@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,return true; folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) +if (!folio)return true; +
Very nitty, but stray extra line unless intended...
Not sure why we can't just put this !folio check in prot_numa_skip()?
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 151
Quoted 6 lines
mm/mprotect.c +4
-1ret =prot_numa_skip (vma, folio, target_node); -if (ret) +if (ret) { +if (folio_test_large (folio) && max_nr !=1 ) + *nr =folio_pte_batch (folio, addr, pte, oldpte, + max_nr, flags,NULL ,NULL ,NULL );
So max_nr can <= 0 too? Shouldn't this be max_nr > 1?
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 152
Quoted 1 line
mm/mprotect.c +0
-0return ret;
Again x = fn_return_bool(); if (x) { return x; } is a bit silly, just do if (fn_return_bool()) { return true; }.
If we return the number of pages, then this can become really simple, like:
I feel like maybe we should abstract the folio large handling here, though it'd be a tiny function so hm.
Anyway assuming we leave it in place, and return number of pages processed, this can become:
if (prot_numa_skip(vma, folio, target_node)) {
if (folio_test_large(folio) && max_nr > 1)
return folio_pte_batch(folio, addr, pte, oldpte, max_nr, flags,
NULL, NULL, NULL);
return 1;
}Which is neater I think!
Quoted 13 lines
mm/mprotect.c +2
-0+ }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 7 lines
mm/mprotect.c +2
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr = (end - addr) >> PAGE_SHIFT;
Not a fan of open-coding this. Since we already provide addr, why not just provide end as well and have prot_numa_avoid_fault() calculate it?
Quoted 16 lines
mm/mprotect.c +3
-2pte_t ptent;/* @@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa && -prot_numa_avoid_fault (vma, addr, - oldpte, target_node)) +prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, + max_nr, &nr))continue ; oldpte =ptep_modify_prot_start (vma, addr, pte);
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 316
Quoted 5 lines
mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr, addr += nr *PAGE_SIZE , addr != end);
This is icky, having 'nr' here like this.
But alternatives might be _even more_ icky (that is advancing both on prot_numa_avoid_fault() so probably we need to keep it like this.
Maybe more a moan at the C programming language tbh haha!
Quoted 6 lines
mm/mprotect.c +0
-1arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); --2.30.2
Dev Jain
reviewerRe: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
Quoted 62 lines · Lorenzo Stoakes
Very very very nitty on subject (sorry I realise this is annoying :P) - generally don't need to capitalise 'Optimize' here :>)
Generally I like the idea here. But some issues on impl.
Quoted 14 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 96
Quoted 6 lines
mm/mprotect.c +3
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,bool toptier;int nid; +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; +Hm why not just put this here from the start? I think you should put this back in the prior commit.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, · line 133
Quoted 10 lines
mm/mprotect.c +2
-1/* Also skip shared copy-on-write pages */ if (is_cow_mapping(vma->vm_flags) && (folio_maybe_dma_pinned(folio) ||@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio, }static bool prot_numa_avoid_fault (struct vm_area_struct *vma, -unsigned long addr,pte_t oldpte,int target_node) +unsigned long addr,pte_t *pte,pte_t oldpte,int target_node, +int max_nr,int *nr)Hate this ptr to nr.
Why not just return nr, if it's 0 then skip? Simple!
Quoted 6 lines
mm/mprotect.c +1
-0{ +const fpb_t flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;struct folio *folio;int ret;mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 146
Quoted 8 lines
mm/mprotect.c +2
-2@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,return true; folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) +if (!folio)return true; +Very nitty, but stray extra line unless intended...
Quoted 1 line
Not sure why we can't just put this !folio check in
prot_numa_skip()?
Because we won't be able to batch if the folio is NULL.
I think I really messed up by having separate patch 1 and 2. The real intent of patch 1 was to do batching in patch 2 *and* not have insane indentation. Perhaps I should merge them, or completely separate them logically, I'll figure this out.
Quoted 93 lines
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 151
Quoted 7 lines
mm/mprotect.c +4
-1ret =prot_numa_skip (vma, folio, target_node); -if (ret) +if (ret) { +if (folio_test_large (folio) && max_nr !=1 ) + *nr =folio_pte_batch (folio, addr, pte, oldpte, + max_nr, flags,NULL ,NULL ,NULL );So
max_nrcan <= 0 too? Shouldn't this bemax_nr> 1?mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 152
Quoted 1 line
mm/mprotect.c +0
-0return ret;Again x =
fn_return_bool(); if (x) { return x; } is a bit silly, just do if (fn_return_bool()) { return true; }.If we return the number of pages, then this can become really simple, like:
I feel like maybe we should abstract the folio large handling here, though it'd be a tiny function so hm.
Anyway assuming we leave it in place, and return number of pages processed, this can become:
if (prot_numa_skip(vma, folio, target_node)) { if (folio_test_large(folio) && max_nr > 1) return folio_pte_batch(folio, addr, pte, oldpte, max_nr, flags, NULL, NULL, NULL); return 1; }Which is neater I think!
Quoted 13 lines
mm/mprotect.c +2
-0+ }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 7 lines
mm/mprotect.c +2
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr = (end - addr) >> PAGE_SHIFT;Not a fan of open-coding this. Since we already provide addr, why not just provide end as well and have
prot_numa_avoid_fault()calculate it?Quoted 16 lines
mm/mprotect.c +3
-2pte_t ptent;/* @@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa && -prot_numa_avoid_fault (vma, addr, - oldpte, target_node)) +prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, + max_nr, &nr))continue ; oldpte =ptep_modify_prot_start (vma, addr, pte);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 316
Quoted 5 lines
mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr, addr += nr *PAGE_SIZE , addr != end);This is icky, having 'nr' here like this.
But alternatives might be
_evenmore_icky (that is advancing both onprot_numa_avoid_fault()so probably we need to keep it like this.Maybe more a moan at the C programming language tbh haha!
Quoted 6 lines
mm/mprotect.c +0
-1arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); --2.30.2
Ryan Roberts
reviewerRe: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
Quoted 68 lines · Dev Jain
Quoted 65 lines · Lorenzo Stoakes
Very very very nitty on subject (sorry I realise this is annoying :P) - generally don't need to capitalise 'Optimize' here :>)
Generally I like the idea here. But some issues on impl.
Quoted 14 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>---
mm/mprotect.c| 27 ++++++++++++++++++++-------1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, · line 97
Quoted 7 lines
mm/mprotect.c +3
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma,struct folio *folio, bool toptier; int nid; + if (folio_is_zone_device(folio) || folio_test_ksm(folio)) + return true; +Hm why not just put this here from the start? I think you should put this back in the prior commit.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, · line 134
Quoted 11 lines
mm/mprotect.c +0
-0/* Also skip shared copy-on-write pages */if (
is_cow_mapping(vma->vm_flags)&&
(folio_maybe_dma_pinned(folio)||mm/mprotect.c +2
-1@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma,struct folio *folio, } static bool prot_numa_avoid_fault(struct vm_area_struct *vma, - unsigned long addr, pte_t oldpte, int target_node) + unsigned long addr, pte_t *pte, pte_t oldpte, int target_node, + int max_nr, int *nr)Hate this ptr to nr.
Why not just return nr, if it's 0 then skip? Simple!
Quoted 6 lines
mm/mprotect.c +1
-0{ + const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; struct folio *folio; int ret;mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct · line 147
Quoted 9 lines
mm/mprotect.c +2
-2@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma, return true; folio = vm_normal_folio(vma, addr, oldpte); - if (!folio || folio_is_zone_device(folio) || - folio_test_ksm(folio)) + if (!folio) return true; +Very nitty, but stray extra line unless intended...
Quoted 7 lines
Quoted 1 line
Not sure why we can't just put this !folio check in
prot_numa_skip()?Because we won't be able to batch if the folio is NULL.
I think I really messed up by having separate patch 1 and 2. The real intent of patch 1 was to do batching in patch 2 *and* not have insane indentation. Perhaps I should merge them, or completely separate them logically, I'll figure this out.
I'd be inclined to just merge into single patch...
Quoted 72 lines
Quoted 72 lines
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 151
Quoted 8 lines
mm/mprotect.c +4
-1ret = prot_numa_skip(vma, folio, target_node); - if (ret) + if (ret) { + if (folio_test_large(folio) && max_nr != 1) + *nr = folio_pte_batch(folio, addr, pte, oldpte, + max_nr, flags, NULL, NULL, NULL);So
max_nrcan <= 0 too? Shouldn't this bemax_nr> 1?mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 152
Quoted 1 line
mm/mprotect.c +0
-0return ret;Again x =
fn_return_bool(); if (x) { return x; } is a bit silly, just do if (fn_return_bool()) { return true; }.If we return the number of pages, then this can become really simple, like:
I feel like maybe we should abstract the folio large handling here, though it'd be a tiny function so hm.
Anyway assuming we leave it in place, and return number of pages processed, this can become:
if (
prot_numa_skip(vma, folio, target_node)) {
if (folio_test_large(folio)&&max_nr> 1)
returnfolio_pte_batch(folio, addr, pte, oldpte, max_nr, flags, NULL, NULL, NULL);
return 1;
}Which is neater I think!
Quoted 13 lines
mm/mprotect.c +1
-0+ } if (folio_use_access_time(folio)) folio_xchg_access_time(folio,
jiffies_to_msecs(jiffies));mm/mprotect.c +1
-0@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb, bool prot_numa = cp_flags & MM_CP_PROT_NUMA; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + int nr; tlb_change_page_size(tlb, PAGE_SIZE); pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 7 lines
mm/mprotect.c +2
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb, flush_tlb_batched_pending(vma->vm_mm); arch_enter_lazy_mmu_mode(); do { + nr = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr = (end - addr) >> PAGE_SHIFT;Not a fan of open-coding this. Since we already provide addr, why not just provide end as well and have
prot_numa_avoid_fault()calculate it?Quoted 14 lines
mm/mprotect.c +0
-0pte_t ptent; /*@@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (
prot_numa&&
-prot_numa_avoid_fault(vma, addr, - oldpte, target_node)) + prot_numa_avoid_fault(vma, addr, pte, + oldpte,target_node, +max_nr, &nr)) continue;
Quoted 9 lines
Quoted 9 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 316
Quoted 7 lines
oldpte =
ptep_modify_prot_start(vma, addr, pte);mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - } while (pte++, addr += PAGE_SIZE, addr != end); + } while (pte += nr, addr += nr * PAGE_SIZE, addr != end);This is icky, having 'nr' here like this.
For better or worse, this is the pattern we have already established in other loops that are batching-aware. See zap_pte_range(), copy_pte_range(), etc. So I'd prefer to follow that pattern here, as Dev has done.
Thanks.
Ryan
Quoted 14 lines
Quoted 13 lines
But alternatives might be
_evenmore_icky (that is advancing both onprot_numa_avoid_fault()so probably we need to keep it like this.Maybe more a moan at the C programming language tbh haha!
Quoted 6 lines
mm/mprotect.c +0
-0arch_leave_lazy_mmu_mode(); pte_unmap_unlock(pte - 1, ptl);Signature · 1 line
2.30.2
Re: [PATCH v2 2/7] mm: Optimize mprotect() by batch-skipping PTEs
Quoted 71 lines · Ryan Roberts
Quoted 70 lines · Dev Jain
Quoted 66 lines · Lorenzo Stoakes
Very very very nitty on subject (sorry I realise this is annoying :P) - generally don't need to capitalise 'Optimize' here :>)
Generally I like the idea here. But some issues on impl.
Quoted 14 lines · Dev Jain
In case of
prot_numa, there are various cases in which we can skip to the next iteration. Since the skip condition is based on the folio and not the PTEs, we can skip a PTE batch.Signed-off-by Dev Jain <[email protected]>---
mm/mprotect.c| 27 ++++++++++++++++++++-------1 file changed, 20 insertions(+), 7 deletions(-)mm/mprotect.c +0
-0mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, · line 97
Quoted 7 lines
mm/mprotect.c +3
-0@@ -91,6 +91,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma,struct folio *folio, bool toptier; int nid; + if (folio_is_zone_device(folio) || folio_test_ksm(folio)) + return true; +Hm why not just put this here from the start? I think you should put this back in the prior commit.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, · line 134
Quoted 11 lines
mm/mprotect.c +0
-0/* Also skip shared copy-on-write pages */if (
is_cow_mapping(vma->vm_flags)&&
(folio_maybe_dma_pinned(folio)||mm/mprotect.c +2
-1@@ -126,8 +129,10 @@ static bool prot_numa_skip(struct vm_area_struct *vma,struct folio *folio, } static bool prot_numa_avoid_fault(struct vm_area_struct *vma, - unsigned long addr, pte_t oldpte, int target_node) + unsigned long addr, pte_t *pte, pte_t oldpte, int target_node, + int max_nr, int *nr)Hate this ptr to nr.
Why not just return nr, if it's 0 then skip? Simple!
Quoted 6 lines
mm/mprotect.c +1
-0{ + const fpb_t flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; struct folio *folio; int ret;mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct · line 147
Quoted 9 lines
mm/mprotect.c +2
-2@@ -136,12 +141,16 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma, return true; folio = vm_normal_folio(vma, addr, oldpte); - if (!folio || folio_is_zone_device(folio) || - folio_test_ksm(folio)) + if (!folio) return true; +Very nitty, but stray extra line unless intended...
Not sure why we can't just put this !folio check in
prot_numa_skip()?
Quoted 7 lines
Quoted 5 lines
Because we won't be able to batch if the folio is NULL.
I think I really messed up by having separate patch 1 and 2. The real intent of patch 1 was to do batching in patch 2 *and* not have insane indentation. Perhaps I should merge them, or completely separate them logically, I'll figure this out.
I'd be inclined to just merge into single patch...
Agreed!
Quoted 83 lines
Quoted 73 lines
Quoted 73 lines
mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 151
Quoted 9 lines
mm/mprotect.c +4
-1ret = prot_numa_skip(vma, folio, target_node); - if (ret) + if (ret) { + if (folio_test_large(folio) && max_nr != 1) + *nr = folio_pte_batch(folio, addr, pte, oldpte, + max_nr, flags, NULL, NULL, NULL);So
max_nrcan <= 0 too? Shouldn't this bemax_nr> 1?mm/mprotect.c static bool prot_numa_avoid_fault(struct vm_area_struct *vma, · line 152
Quoted 1 line
mm/mprotect.c +0
-0return ret;Again x =
fn_return_bool(); if (x) { return x; } is a bit silly, just do if (fn_return_bool()) { return true; }.If we return the number of pages, then this can become really simple, like:
I feel like maybe we should abstract the folio large handling here, though it'd be a tiny function so hm.
Anyway assuming we leave it in place, and return number of pages processed, this can become:
if (
prot_numa_skip(vma, folio, target_node)) {
if (folio_test_large(folio)&&max_nr> 1)
returnfolio_pte_batch(folio, addr, pte, oldpte, max_nr, flags, NULL, NULL, NULL);
return 1;
}Which is neater I think!
Quoted 13 lines
mm/mprotect.c +1
-0+ } if (folio_use_access_time(folio)) folio_xchg_access_time(folio,
jiffies_to_msecs(jiffies));mm/mprotect.c +1
-0@@ -159,6 +168,7 @@ static long change_pte_range(struct mmu_gather *tlb, bool prot_numa = cp_flags & MM_CP_PROT_NUMA; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + int nr; tlb_change_page_size(tlb, PAGE_SIZE); pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 189
Quoted 7 lines
mm/mprotect.c +2
-0@@ -173,8 +183,10 @@ static long change_pte_range(struct mmu_gather *tlb, flush_tlb_batched_pending(vma->vm_mm); arch_enter_lazy_mmu_mode(); do { + nr = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr = (end - addr) >> PAGE_SHIFT;Not a fan of open-coding this. Since we already provide addr, why not just provide end as well and have
prot_numa_avoid_fault()calculate it?Quoted 14 lines
mm/mprotect.c +0
-0pte_t ptent; /*@@ -182,8 +194,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (
prot_numa&&
-prot_numa_avoid_fault(vma, addr, - oldpte, target_node)) + prot_numa_avoid_fault(vma, addr, pte, + oldpte,target_node, +max_nr, &nr)) continue;Quoted 9 lines
Quoted 9 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 316
Quoted 7 lines
oldpte =
ptep_modify_prot_start(vma, addr, pte);mm/mprotect.c +1
-1@@ -300,7 +313,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - } while (pte++, addr += PAGE_SIZE, addr != end); + } while (pte += nr, addr += nr * PAGE_SIZE, addr != end);This is icky, having 'nr' here like this.
Quoted 3 lines
For better or worse, this is the pattern we have already established in other loops that are batching-aware. See
zap_pte_range(),copy_pte_range(), etc. So I'd prefer to follow that pattern here, as Dev has done.
Yeah I'm fine with keeping this 'nr' stuff, I don't think there's a great alternative.
Quoted 3 lines
Thanks.
Ryan
Cheers, Lorenzo
Quoted 16 lines
Quoted 15 lines
Quoted 14 lines
But alternatives might be
_evenmore_icky (that is advancing both onprot_numa_avoid_fault()so probably we need to keep it like this.Maybe more a moan at the C programming language tbh haha!
Quoted 6 lines
mm/mprotect.c +0
-1arch_leave_lazy_mmu_mode(); pte_unmap_unlock(pte - 1, ptl); --2.30.2
Dev Jain
reviewerPATCH v2 3/7 mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainBatch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
include/linux/pgtable.h +38 -0
@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings
Signature · 1 line
2.30.2
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 43 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +27
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +for (;;) {
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 930
Quoted 10 lines
include/linux/pgtable.h +10
-0+ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif
Is there any particular reason why the first loop here is while { } based where as the second one is a for { } loop ?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 934
Quoted 4 lines
include/linux/pgtable.h +1
-0+/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings
These helpers should be added with at least a single caller using them.
Dev Jain
reviewerRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 51 lines · Anshuman Khandual
Quoted 48 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +32
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ;
Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 930
Quoted 5 lines
include/linux/pgtable.h +5
-0+ old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif Is there any particular reason why the first loop here is while { } based where as the second one is a for { } loop ?
modify_prot_start_ptes follows the pattern of get_and_clear_full_ptes. modify_prot_commit_ptes follows the pattern of wrprotect_ptes.
Quoted 7 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 934
Quoted 5 lines
include/linux/pgtable.h +1
-0+/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings These helpers should be added with at least a single caller using them.
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */
I feel like you really should add a little more here, perhaps point out that it's batched etc.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)
This name is a bit confusing, it's not any ptes, it's those pte entries belonging to a large folio capped to the PTE table right that you are batching right?
Perhaps modify_prot_start_large_folio() ? Or something with 'batched' in the name?
We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 3 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +
are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 2 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {
This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr = min(nr_pte_entries_in_pte, nr_pgs_in_folio)
Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called ptep_modify_prot_start() on all of them?
This is quite a bit to a. put in a header like this and b. not comment/explain.
So maybe something like:
pte = ptep_modify_prot_start(vma, addr, ptep);
/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) {
pte_t inner_pte;
ptep++;
addr += PAGE_SIZE;
inner_pte = ptep_modify_prot_start(vma, addr, ptep);
/* We must propagate A/D state from tail PTEs. */
if (pte_dirty(inner_pte))
pte = pte_mkdirty(pte);
if (pte_young(inner_pte))
pte = pte_mkyoung(pte);
}Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);
Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */
Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)
Nit, but you put 'p' suffix on ptep but not on 'old_pte'?
I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left in old_pte)
It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely _at the very least_ need to document it in a comment.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;
Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
for (pg = 1; pg < nr; pg++) {
ptep++;
addr += PAGE_SIZE;
old_pte = pte_next_pfn(old_pte);
pte = pte_next_pfn(pte);
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
}There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It brings
--
2.30.2
Dev Jain
reviewerRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 17 lines · Lorenzo Stoakes
Quoted 16 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif
Quoted 4 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 1 line
include/linux/pgtable.h +1
-0+/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Sure. I couldn't easily figure out a way to write the documentation nicely, I'll do it this time.
Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 4 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing, it's not any ptes, it's those pte entries belonging to a large folio capped to the PTE table right that you are batching right?
yes, but I am just following the convention. See wrprotect_ptes(), etc. I don't have a strong preference anyways.
Quoted 3 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?
How about modify_prot_start_batched_ptes()?
Quoted 11 lines
We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 3 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
Shall I write in the comments that the range is supposed to be within a PTE table?
Quoted 6 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
Again, following the pattern of get_and_clear_full_ptes :)
Quoted 34 lines
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?This is quite a bit to a. put in a header like this and b. not comment/explain.
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }
Quoted 1 line
Would work better?
No preference, I'll do this then.
Quoted 9 lines
Quoted 6 lines
include/linux/pgtable.h +3
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep);
Quoted 6 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 4 lines
include/linux/pgtable.h +4
-0+if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
Because the a/d bits are per-folio; and, this will help us batch around can_change_pte_writable (return pte_dirty(pte)) and, batch around pte_needs_flush() for parisc.
Quoted 16 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 7 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
Quoted 3 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?
Because ptep is a pointer, and old_pte isn't.
Quoted 13 lines
I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.
Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Following wrprotect_ptes().
I agree that this is confusing, which is why I thought why it was done in the first place :) but I just followed what already is. I'll change this to a simple for loop if that is your inclination.
Quoted 36 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-1+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings -- 2.30.2
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 20 lines · Dev Jain
Quoted 17 lines · Lorenzo Stoakes
Quoted 16 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif
Quoted 7 lines
Quoted 4 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 1 line
include/linux/pgtable.h +1
-0+/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Sure. I couldn't easily figure out a way to write the documentation nicely, I'll do it this time.
Thanks! Though see the discussion with Ryan also.
Quoted 10 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 5 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing, it's not any ptes, it's those pte entries belonging to a large folio capped to the PTE table right that you are batching right?
Quoted 8 lines
yes, but I am just following the convention. See
wrprotect_ptes(), etc. I don't have a strong preference anyways.Quoted 3 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?How about
modify_prot_start_batched_ptes()?
I like this :) Ryan - that work for you, or do you feel _batched_ should be dropped here?
Quoted 13 lines
Quoted 5 lines
We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Quoted 7 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 3 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
Quoted 2 lines
Shall I write in the comments that the range is supposed to be within a PTE table?
Yeah that'd be helpful I think thanks!
Quoted 9 lines
Quoted 7 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 4 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
Again, following the pattern of
get_and_clear_full_ptes:)
Yeah, see discussion with Ryan :>)
Quoted 34 lines
Quoted 34 lines
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?This is quite a bit to a. put in a header like this and b. not comment/explain.
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }
Quoted 3 lines
Quoted 1 line
Would work better?
No preference, I'll do this then.
Thanks!
Quoted 17 lines
Quoted 10 lines
Quoted 7 lines
include/linux/pgtable.h +3
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep);Quoted 6 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 4 lines
include/linux/pgtable.h +4
-0+if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
Quoted 3 lines
Because the a/d bits are per-folio; and, this will help us batch around
can_change_pte_writable(returnpte_dirty(pte)) and, batch aroundpte_needs_flush()for parisc.
Understood, thanks!
Quoted 17 lines
Quoted 17 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 8 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
Quoted 5 lines
Quoted 3 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?Because ptep is a pointer, and
old_pteisn't.
Oops :P :) sorry, this is me being a little 'slow' here... I missed that. Carry on then :P
Quoted 23 lines
Quoted 14 lines
I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Quoted 4 lines
Following
wrprotect_ptes().
I agree that this is confusing, which is why I thought why it was done in the first place :) but I just followed what already is.
I'll change this to a simple for loop if that is your inclination.
No, I guess let's keep it as-is, Ryan pointed out there are perf considerations here. This one is a lot less egregious.
Quoted 23 lines
Quoted 23 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
include/linux/pgtable.h +6
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +}Quoted 8 lines
Quoted 8 lines
include/linux/pgtable.h +2
-1+#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings -- 2.30.2
Thanks, Lorenzo
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 45 lines · Lorenzo Stoakes
Quoted 20 lines · Dev Jain
Quoted 17 lines · Lorenzo Stoakes
Quoted 16 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif Quoted 7 lines
Quoted 4 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 1 line
include/linux/pgtable.h +1
-0+/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Sure. I couldn't easily figure out a way to write the documentation nicely, I'll do it this time.
Thanks! Though see the discussion with Ryan also.
Quoted 14 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 5 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing, it's not any ptes, it's those pte entries belonging to a large folio capped to the PTE table right that you are batching right?
yes, but I am just following the convention. See
wrprotect_ptes(), etc. I don't have a strong preference anyways.Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?How about
modify_prot_start_batched_ptes()?I like this :) Ryan - that work for you, or do you feel _batched_ should be dropped here?
modify_prot_start_folio_ptes ?
But I would rather go with
modify_prot_folio_ptes_start
The "batched" is implicit, and "large folio" is not required if it's more than one pte ...
:)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 55 lines · David Hildenbrand
Quoted 45 lines · Lorenzo Stoakes
Quoted 20 lines · Dev Jain
Quoted 17 lines · Lorenzo Stoakes
Quoted 16 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif Quoted 7 lines
Quoted 4 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 1 line
include/linux/pgtable.h +1
-0+/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Sure. I couldn't easily figure out a way to write the documentation nicely, I'll do it this time.
Thanks! Though see the discussion with Ryan also.
Quoted 14 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 5 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing, it's not any ptes, it's those pte entries belonging to a large folio capped to the PTE table right that you are batching right?
yes, but I am just following the convention. See
wrprotect_ptes(), etc. I don't have a strong preference anyways.Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?How about
modify_prot_start_batched_ptes()?I like this :) Ryan - that work for you, or do you feel _batched_ should be dropped here?
Quoted 8 lines
modify_prot_start_folio_ptes?But I would rather go with
modify_prot_folio_ptes_startThe "batched" is implicit, and "large folio" is not required if it's more than one pte ...
Yeah that works for me! The mention of folio with plural does neatly imply the rest.
Naming is hard :P
Quoted 8 lines
:)
--
Cheers,David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Lorenzo StoakesQuoted 22 lines · Lorenzo Stoakes
Quoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Quoted 5 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing,
On naming, the existing (modern) convention for single-pte helpers is to start the function name with ptep_. When we started adding batched versions, we took the approach of adding _ptes as a suffix. For example:
set_pte_at()ptep_get_and_clear_full()ptep_set_wrprotect()
set_ptes()get_and_clear_full_ptes()wrprotect_ptes()
In this case, we already have ptep_modify_prot_start() and ptep_modify_prot_commit() for the existing single-pte helper versions. So according to the convention (or at least how I interpret the convention), the proposed names seem reasonable.
Quoted 3 lines
it's not any ptes, it's those pte entries
belonging to a large folio capped to the PTE table right that you are batching right?
Yes, but again by convention, that is captured in the kerneldoc comment for the functions. We are operating on a batch of *ptes* not on a folio or batch of folios. But it is a requirement of the function that the batch of ptes all lie within a single large folio (i.e. the pfns are sequential).
Quoted 5 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Agreed!
Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 4 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
For better or worse, that's the established convention. See part of comment for set_ptes() for example:
"""
* Context: The caller holds the page table lock. The pages all belong
* to the same folio. The PTEs are all in the same PMD.
"""
Quoted 11 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)
Quoted 8 lines
Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?This is quite a bit to a. put in a header like this and b. not comment/explain.
This style is copied from get_and_clear_full_ptes(), which has this comment, which explains all this complexity. My vote would be to have a simple comment for this function:
/**
* get_and_clear_full_ptes - Clear present PTEs that map consecutive pages of
* the same folio, collecting dirty/accessed bits.
* @mm: Address space the pages are mapped into.
* @addr: Address the first page is mapped at.
* @ptep: Page table pointer for the first entry.
* @nr: Number of entries to clear.
* @full: Whether we are clearing a full mm.
*
* May be overridden by the architecture; otherwise, implemented as a simple
* loop over ptep_get_and_clear_full(), merging dirty/accessed bits into the
* returned PTE.
*
* Note that PTE bits in the PTE range besides the PFN can differ. For example,
* some PTEs might be write-protected.
*
* Context: The caller holds the page table lock. The PTEs map consecutive
* pages that belong to the same folio. The PTEs are all in the same PMD. */
Quoted 69 lines
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.
Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
I agree it's not pretty to look at. But apparently it's the most efficient. This is Willy's commit that started it all: Commit bcc6cc832573 ("mm: add default definition of set_ptes()").
For the record, I think all your comments make good sense, Lorenzo. But there is an established style, and personally I think at this point is it more confusing to break from that style.
Thanks,
Ryan
Quoted 36 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It brings
--
2.30.2
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 37 lines · Ryan Roberts
Quoted 22 lines · Lorenzo Stoakes
Quoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Quoted 5 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing,
On naming, the existing (modern) convention for single-pte helpers is to start the function name with
ptep_. When we started adding batched versions, we took the approach of adding_ptesas a suffix. For example:
set_pte_at()ptep_get_and_clear_full()ptep_set_wrprotect()
Quoted 9 lines
set_ptes()get_and_clear_full_ptes()wrprotect_ptes()In this case, we already have
ptep_modify_prot_start()andptep_modify_prot_commit()for the existing single-pte helper versions. So according to the convention (or at least how I interpret the convention), the proposed names seem reasonable.
Right, I'm fine with following convention (we should), I just find 'ptes' really ambiguous. It's not just a -set of PTE entries- it's very explicitly for a large folio. I'd interpret some 'ptes' case to mean 'any number of pte entries', though I suppose it'd not in practice be any different if that were the intended use.
However, the proposed use case is large folio 'sub' PTEs and it'd be useful in callers to know this is explicitly what you're doing.
I feel like '_batched_ptes' makes it clear it's a _specific_ set of PTE entriess you're after (not just in effect multiple PTE entries).
However, I'm less insistent on this with a comment that explains what's going on.
I don't want to hold this up with trivialities around naming...
ASIDE: I continue to absolutely HATE the ambiguity between 'PxD/PTE' and 'PxD/PTE entries' and the fact we use both as a short-hand for each other. But that's not related to this series, just a pet peeve... :)
Quoted 8 lines
Quoted 3 lines
it's not any ptes, it's those pte entries
belonging to a large folio capped to the PTE table right that you are batching right?Yes, but again by convention, that is captured in the kerneldoc comment for the functions. We are operating on a batch of *ptes* not on a folio or batch of folios. But it is a requirement of the function that the batch of ptes all lie within a single large folio (i.e. the pfns are sequential).
Ack, yeah don't love this nr stuff but fine if it's convention...
Quoted 7 lines
Quoted 5 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Agreed!
...and luckily we are aligned on this :)
Quoted 53 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 5 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
For better or worse, that's the established convention. See part of comment for
set_ptes()for example:"""
* Context: The caller holds the page table lock. The pages all belong
* to the same folio. The PTEs are all in the same PMD.
"""Quoted 11 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Quoted 8 lines
Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?This is quite a bit to a. put in a header like this and b. not comment/explain.
This style is copied from
get_and_clear_full_ptes(), which has this comment, which explains all this complexity. My vote would be to have a simple comment for this function:/**
*get_and_clear_full_ptes- Clear present PTEs that map consecutive pages of
* the same folio, collecting dirty/accessed bits.
* @mm: Address space the pages are mapped into.
* @addr: Address the first page is mapped at.
* @ptep: Page table pointer for the first entry.
* @nr: Number of entries to clear.
* @full: Whether we are clearing a full mm.
*
* May be overridden by the architecture; otherwise, implemented as a simple
* loop overptep_get_and_clear_full(), merging dirty/accessed bits into the
Quoted 9 lines
* returned PTE.
*
* Note that PTE bits in the PTE range besides the PFN can differ. For example,
* some PTEs might be write-protected.
*
* Context: The caller holds the page table lock. The PTEs map consecutive
* pages that belong to the same folio. The PTEs are all in the same PMD. */
OK I think the key bit here is 'consecutive pages of the same folio'.
I'd like at least a paragraph about implementation, yes the original function doesn't have that (and should imo), something like:
We perform the operation on the first PTE, then if any others follow, we invoke the ptep_modify_prot_start() for each and aggregate A/D bits.
Something like this.
Point taken on consistency though!
Quoted 78 lines
Quoted 69 lines
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Quoted 7 lines
I agree it's not pretty to look at. But apparently it's the most efficient. This is Willy's commit that started it all: Commit
bcc6cc832573("mm: add default definition ofset_ptes()").For the record, I think all your comments make good sense, Lorenzo. But there is an established style, and personally I think at this point is it more confusing to break from that style.
This isn't _quite_ style, I'd say it's implementation, we're kind of crossing over into something a little more I'd say :) but I mean I get your point, sure.
I mean, fine, if (I presume you're referring _only_ to the for (;;) case above) you are absolutely certain it is more performant in practice I wouldn't want to stand in the way of that.
I would at least like a comment in the commit message about propagating an odd loop for performance though to explain the 'qualities'... :)
Quoted 42 lines
Thanks,
RyanQuoted 36 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It brings
--
2.30.2
Ryan Roberts
reviewerRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 54 lines · Lorenzo Stoakes
Quoted 37 lines · Ryan Roberts
Quoted 22 lines · Lorenzo Stoakes
Quoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Quoted 5 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing,
On naming, the existing (modern) convention for single-pte helpers is to start the function name with
ptep_. When we started adding batched versions, we took the approach of adding_ptesas a suffix. For example:
set_pte_at()ptep_get_and_clear_full()ptep_set_wrprotect()Quoted 9 lines
set_ptes()get_and_clear_full_ptes()wrprotect_ptes()In this case, we already have
ptep_modify_prot_start()andptep_modify_prot_commit()for the existing single-pte helper versions. So according to the convention (or at least how I interpret the convention), the proposed names seem reasonable.Right, I'm fine with following convention (we should), I just find 'ptes' really ambiguous. It's not just a -set of PTE entries- it's very explicitly for a large folio. I'd interpret some 'ptes' case to mean 'any number of pte entries', though I suppose it'd not in practice be any different if that were the intended use.
Quoted 5 lines
However, the proposed use case is large folio 'sub' PTEs and it'd be useful in callers to know this is explicitly what you're doing.
I feel like '
_batched_ptes' makes it clear it's a _specific_ set of PTE entriess you're after (not just in effect multiple PTE entries).
I don't mind _batched_ptes. _pte_batch would be shorter though - what do you think?
But if we go with one of these, then we should consistently apply it to all the existing helpers IMHO - perhaps with a preparatory patch at the start of the series.
Quoted 3 lines
However, I'm less insistent on this with a comment that explains what's going on.
That would still get my vote :)
Quoted 2 lines
I don't want to hold this up with trivialities around naming...
There are TWO hard things in computer science; cache invalidation, naming, and off-by-one errors :)
Quoted 4 lines
ASIDE: I continue to absolutely HATE the ambiguity between 'PxD/PTE' and 'PxD/PTE entries' and the fact we use both as a short-hand for each other. But that's not related to this series, just a pet peeve... :)
I assume you are referring to the ambiguity between the *table* and the *entry* (which just goes to show how ambiguous it is I guess)... I also hate this and still trip over it all the time...
Quoted 13 lines
Quoted 9 lines
Quoted 4 lines
it's not any ptes, it's those pte entries
belonging to a large folio capped to the PTE table right that you are batching right?Yes, but again by convention, that is captured in the kerneldoc comment for the functions. We are operating on a batch of *ptes* not on a folio or batch of folios. But it is a requirement of the function that the batch of ptes all lie within a single large folio (i.e. the pfns are sequential).
Ack, yeah don't love this nr stuff but fine if it's convention...
Quoted 1 line
Quoted 1 line
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in
Quoted 49 lines
Quoted 6 lines
Quoted 4 lines
the name?
We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Agreed!
...and luckily we are aligned on this :)
Quoted 40 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 5 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
For better or worse, that's the established convention. See part of comment for
set_ptes()for example:"""
* Context: The caller holds the page table lock. The pages all belong
* to the same folio. The PTEs are all in the same PMD.
"""Quoted 11 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Quoted 8 lines
Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?This is quite a bit to a. put in a header like this and b. not comment/explain.
This style is copied from
get_and_clear_full_ptes(), which has this comment, which explains all this complexity. My vote would be to have a simple comment
Oops; I meant "similar" when my fingers somehow typed "simple"... This is not simple :)
Quoted 122 lines
Quoted 13 lines
for this function:
/**
*get_and_clear_full_ptes- Clear present PTEs that map consecutive pages of
* the same folio, collecting dirty/accessed bits.
* @mm: Address space the pages are mapped into.
* @addr: Address the first page is mapped at.
* @ptep: Page table pointer for the first entry.
* @nr: Number of entries to clear.
* @full: Whether we are clearing a full mm.
*
* May be overridden by the architecture; otherwise, implemented as a simple
* loop overptep_get_and_clear_full(), merging dirty/accessed bits into theQuoted 9 lines
* returned PTE.
*
* Note that PTE bits in the PTE range besides the PFN can differ. For example,
* some PTEs might be write-protected.
*
* Context: The caller holds the page table lock. The PTEs map consecutive
* pages that belong to the same folio. The PTEs are all in the same PMD. */OK I think the key bit here is 'consecutive pages of the same folio'.
I'd like at least a paragraph about implementation, yes the original function doesn't have that (and should imo), something like:
We perform the operation on the first PTE, then if any others follow, we invoke the
ptep_modify_prot_start()for each and aggregate A/D bits.Something like this.
Point taken on consistency though!
Quoted 78 lines
Quoted 69 lines
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Quoted 7 lines
I agree it's not pretty to look at. But apparently it's the most efficient. This is Willy's commit that started it all: Commit
bcc6cc832573("mm: add default definition ofset_ptes()").For the record, I think all your comments make good sense, Lorenzo. But there is an established style, and personally I think at this point is it more confusing to break from that style.
Quoted 7 lines
This isn't _quite_ style, I'd say it's implementation, we're kind of crossing over into something a little more I'd say :) but I mean I get your point, sure.
I mean, fine, if (I presume you're referring _only_ to the for (;;) case above) you are absolutely certain it is more performant in practice I wouldn't want to stand in the way of that.
No I'm not certain at all... I'm just saying that's been the argument in the past. I vaguely recall I even tried changing the loop style in batched helpers I implemented in the past and David asked me to stick with the established style.
Quoted 3 lines
I would at least like a comment in the commit message about propagating an odd loop for performance though to explain the 'qualities'... :)
Just to make it clear, I'm just trying to provide some historical context, I'm not arguing that all those decisions were perfect. How about we take these concrete steps:
- Stick with the _ptes naming convention
- Add kerneldoc comments for the 2 new functions that are very clear about what the function does and the requirements on the batch of ptes (just like the other batched helpers)
- Rework the looping styles in the 2 new functions to be more "standard"; let's not micro-optimize unless we have real evidence that it is useful.
- Merge this patch with the one that uses these new functions
How does that sound as a way forwards?
Thanks,
Ryan
Quoted 43 lines
Quoted 43 lines
Thanks,
RyanQuoted 36 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It brings
--
2.30.2
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 55 lines · Ryan Roberts
Quoted 54 lines · Lorenzo Stoakes
Quoted 37 lines · Ryan Roberts
Quoted 22 lines · Lorenzo Stoakes
Quoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Quoted 5 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing,
On naming, the existing (modern) convention for single-pte helpers is to start the function name with
ptep_. When we started adding batched versions, we took the approach of adding_ptesas a suffix. For example:
set_pte_at()ptep_get_and_clear_full()ptep_set_wrprotect()Quoted 9 lines
set_ptes()get_and_clear_full_ptes()wrprotect_ptes()In this case, we already have
ptep_modify_prot_start()andptep_modify_prot_commit()for the existing single-pte helper versions. So according to the convention (or at least how I interpret the convention), the proposed names seem reasonable.Right, I'm fine with following convention (we should), I just find 'ptes' really ambiguous. It's not just a -set of PTE entries- it's very explicitly for a large folio. I'd interpret some 'ptes' case to mean 'any number of pte entries', though I suppose it'd not in practice be any different if that were the intended use.
Quoted 7 lines
Quoted 5 lines
However, the proposed use case is large folio 'sub' PTEs and it'd be useful in callers to know this is explicitly what you're doing.
I feel like '
_batched_ptes' makes it clear it's a _specific_ set of PTE entriess you're after (not just in effect multiple PTE entries).I don't mind
_batched_ptes._pte_batchwould be shorter though - what do you think?
Sounds good!
Quoted 9 lines
But if we go with one of these, then we should consistently apply it to all the existing helpers IMHO - perhaps with a preparatory patch at the start of the series.
Quoted 3 lines
However, I'm less insistent on this with a comment that explains what's going on.
That would still get my vote :)
Awesome :)
Quoted 6 lines
Quoted 3 lines
I don't want to hold this up with trivialities around naming...
There are TWO hard things in computer science; cache invalidation, naming, and off-by-one errors :)
Haha yes... I continue to be surprised at how bloody hard it is as my career goes on...
Quoted 9 lines
Quoted 5 lines
ASIDE: I continue to absolutely HATE the ambiguity between 'PxD/PTE' and 'PxD/PTE entries' and the fact we use both as a short-hand for each other. But that's not related to this series, just a pet peeve... :)
I assume you are referring to the ambiguity between the *table* and the *entry* (which just goes to show how ambiguous it is I guess)... I also hate this and still trip over it all the time...
Yes. As do I, as does everybody I think... Sadly I think unavoidable :(
Quoted 58 lines
Quoted 58 lines
Quoted 10 lines
Quoted 5 lines
it's not any ptes, it's those pte entries
belonging to a large folio capped to the PTE table right that you are batching right?Yes, but again by convention, that is captured in the kerneldoc comment for the functions. We are operating on a batch of *ptes* not on a folio or batch of folios. But it is a requirement of the function that the batch of ptes all lie within a single large folio (i.e. the pfns are sequential).
Ack, yeah don't love this nr stuff but fine if it's convention...
Quoted 7 lines
Quoted 5 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Agreed!
...and luckily we are aligned on this :)
Quoted 35 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 5 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
For better or worse, that's the established convention. See part of comment for
set_ptes()for example:"""
* Context: The caller holds the page table lock. The pages all belong
* to the same folio. The PTEs are all in the same PMD.
"""Quoted 17 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
This is quite a bit to a. put in a header like this and b. not comment/explain.
This style is copied from
get_and_clear_full_ptes(), which has this comment, which explains all this complexity. My vote would be to have a simple commentOops; I meant "similar" when my fingers somehow typed "simple"... This is not simple :)
Ha, yeah indeed :P that makes more sense!
Quoted 127 lines
Quoted 127 lines
Quoted 14 lines
for this function:
/**
*get_and_clear_full_ptes- Clear present PTEs that map consecutive pages of
* the same folio, collecting dirty/accessed bits.
* @mm: Address space the pages are mapped into.
* @addr: Address the first page is mapped at.
* @ptep: Page table pointer for the first entry.
* @nr: Number of entries to clear.
* @full: Whether we are clearing a full mm.
*
* May be overridden by the architecture; otherwise, implemented as a simple
* loop overptep_get_and_clear_full(), merging dirty/accessed bits into theQuoted 9 lines
* returned PTE.
*
* Note that PTE bits in the PTE range besides the PFN can differ. For example,
* some PTEs might be write-protected.
*
* Context: The caller holds the page table lock. The PTEs map consecutive
* pages that belong to the same folio. The PTEs are all in the same PMD. */OK I think the key bit here is 'consecutive pages of the same folio'.
I'd like at least a paragraph about implementation, yes the original function doesn't have that (and should imo), something like:
We perform the operation on the first PTE, then if any others follow, we invoke the
ptep_modify_prot_start()for each and aggregate A/D bits.Something like this.
Point taken on consistency though!
Quoted 78 lines
Quoted 69 lines
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Quoted 7 lines
I agree it's not pretty to look at. But apparently it's the most efficient. This is Willy's commit that started it all: Commit
bcc6cc832573("mm: add default definition ofset_ptes()").For the record, I think all your comments make good sense, Lorenzo. But there is an established style, and personally I think at this point is it more confusing to break from that style.
This isn't _quite_ style, I'd say it's implementation, we're kind of crossing over into something a little more I'd say :) but I mean I get your point, sure.
Quoted 7 lines
Quoted 3 lines
I mean, fine, if (I presume you're referring _only_ to the for (;;) case above) you are absolutely certain it is more performant in practice I wouldn't want to stand in the way of that.
No I'm not certain at all... I'm just saying that's been the argument in the past. I vaguely recall I even tried changing the loop style in batched helpers I implemented in the past and David asked me to stick with the established style.
I definitely defer to David's expertise, but I feel there's some room here for improving things.
Quoted 8 lines
Quoted 4 lines
I would at least like a comment in the commit message about propagating an odd loop for performance though to explain the 'qualities'... :)
Just to make it clear, I'm just trying to provide some historical context, I'm not arguing that all those decisions were perfect. How about we take these concrete steps:
Ack sure.
Quoted 10 lines
- Stick with the
_ptesnaming convention
- Add kerneldoc comments for the 2 new functions that are very clear about what the function does and the requirements on the batch of ptes (just like the other batched helpers)
- Rework the looping styles in the 2 new functions to be more "standard"; let's not micro-optimize unless we have real evidence that it is useful.
- Merge this patch with the one that uses these new functionsHow does that sound as a way forwards?
Sounds good to me!
Cheers, Lorenzo
Quoted 48 lines
Thanks,
RyanQuoted 43 lines
Quoted 43 lines
Thanks,
RyanQuoted 36 lines
I think it's ok to duplicate this single line for the sake of clarity, also.
Which gives us:
unsigned long pg;
ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);for (pg = 1; pg < nr; pg++) { ptep++; addr += PAGE_SIZE; old_pte = pte_next_pfn(old_pte); pte = pte_next_pfn(pte); ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); }There are alternative approaches, but I think doing an infinite loop that breaks and especially the confusing 'if (--foo) break;' stuff is much harder to parse than a super simple ranged loop.
Quoted 14 lines
include/linux/pgtable.h +8
-0+ ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It brings
--
2.30.2
Re: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 300 lines · Lorenzo Stoakes
Quoted 55 lines · Ryan Roberts
Quoted 54 lines · Lorenzo Stoakes
Quoted 37 lines · Ryan Roberts
Quoted 22 lines · Lorenzo Stoakes
Quoted 13 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +0
-0include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 894
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ I feel like you really should add a little more here, perhaps point out that it's batched etc.
Quoted 5 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 897
Quoted 3 lines
include/linux/pgtable.h +3
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr)This name is a bit confusing,
On naming, the existing (modern) convention for single-pte helpers is to start the function name with
ptep_. When we started adding batched versions, we took the approach of adding_ptesas a suffix. For example:
set_pte_at()ptep_get_and_clear_full()ptep_set_wrprotect()Quoted 9 lines
set_ptes()get_and_clear_full_ptes()wrprotect_ptes()In this case, we already have
ptep_modify_prot_start()andptep_modify_prot_commit()for the existing single-pte helper versions. So according to the convention (or at least how I interpret the convention), the proposed names seem reasonable.Right, I'm fine with following convention (we should), I just find 'ptes' really ambiguous. It's not just a -set of PTE entries- it's very explicitly for a large folio. I'd interpret some 'ptes' case to mean 'any number of pte entries', though I suppose it'd not in practice be any different if that were the intended use.
Quoted 7 lines
Quoted 5 lines
However, the proposed use case is large folio 'sub' PTEs and it'd be useful in callers to know this is explicitly what you're doing.
I feel like '
_batched_ptes' makes it clear it's a _specific_ set of PTE entriess you're after (not just in effect multiple PTE entries).I don't mind
_batched_ptes._pte_batchwould be shorter though - what do you think?Sounds good!
Quoted 9 lines
But if we go with one of these, then we should consistently apply it to all the existing helpers IMHO - perhaps with a preparatory patch at the start of the series.
Quoted 3 lines
However, I'm less insistent on this with a comment that explains what's going on.
That would still get my vote :)
Awesome :)
Quoted 6 lines
Quoted 3 lines
I don't want to hold this up with trivialities around naming...
There are TWO hard things in computer science; cache invalidation, naming, and off-by-one errors :)
Haha yes... I continue to be surprised at how bloody hard it is as my career goes on...
Quoted 9 lines
Quoted 5 lines
ASIDE: I continue to absolutely HATE the ambiguity between 'PxD/PTE' and 'PxD/PTE entries' and the fact we use both as a short-hand for each other. But that's not related to this series, just a pet peeve... :)
I assume you are referring to the ambiguity between the *table* and the *entry* (which just goes to show how ambiguous it is I guess)... I also hate this and still trip over it all the time...
Yes. As do I, as does everybody I think... Sadly I think unavoidable :(
Quoted 58 lines
Quoted 58 lines
Quoted 10 lines
Quoted 5 lines
it's not any ptes, it's those pte entries
belonging to a large folio capped to the PTE table right that you are batching right?Yes, but again by convention, that is captured in the kerneldoc comment for the functions. We are operating on a batch of *ptes* not on a folio or batch of folios. But it is a requirement of the function that the batch of ptes all lie within a single large folio (i.e. the pfns are sequential).
Ack, yeah don't love this nr stuff but fine if it's convention...
Quoted 7 lines
Quoted 5 lines
Perhaps
modify_prot_start_large_folio()? Or something with 'batched' in the name?We definitely need to mention in comment or name or _somewhere_ the intent and motivation for this.
Agreed!
...and luckily we are aligned on this :)
Quoted 35 lines
Quoted 9 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 900
Quoted 5 lines
include/linux/pgtable.h +3
-0+{ +pte_t pte, tmp_pte; +are we not validating what 'nr' is? Even with debug asserts? I'm not sure I love this interface, where you require the user to know the number of remaining PTE entries in a PTE table.
For better or worse, that's the established convention. See part of comment for
set_ptes()for example:"""
* Context: The caller holds the page table lock. The pages all belong
* to the same folio. The PTEs are all in the same PMD.
"""Quoted 17 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 902
Quoted 3 lines
include/linux/pgtable.h +2
-0+ pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) {This loop is a bit horrible. It seems needlessly confusing and you're in _dire_ need of comments to explain what's going on.
So my understanding is, you have the user figure out:
nr =
min(nr_pte_entries_in_pte, nr_pgs_in_folio)Then, you want to return the pte entry belonging to the start of the large folio batch, but you want to adjust that pte value to propagate dirty and young page table flags if any page table entries within the range contain those page table flags, having called
ptep_modify_prot_start()on all of them?Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
This is quite a bit to a. put in a header like this and b. not comment/explain.
This style is copied from
get_and_clear_full_ptes(), which has this comment, which explains all this complexity. My vote would be to have a simple commentOops; I meant "similar" when my fingers somehow typed "simple"... This is not simple :)
Ha, yeah indeed :P that makes more sense!
Quoted 131 lines
Quoted 130 lines
Quoted 14 lines
for this function:
/**
*get_and_clear_full_ptes- Clear present PTEs that map consecutive pages of
* the same folio, collecting dirty/accessed bits.
* @mm: Address space the pages are mapped into.
* @addr: Address the first page is mapped at.
* @ptep: Page table pointer for the first entry.
* @nr: Number of entries to clear.
* @full: Whether we are clearing a full mm.
*
* May be overridden by the architecture; otherwise, implemented as a simple
* loop overptep_get_and_clear_full(), merging dirty/accessed bits into theQuoted 9 lines
* returned PTE.
*
* Note that PTE bits in the PTE range besides the PFN can differ. For example,
* some PTEs might be write-protected.
*
* Context: The caller holds the page table lock. The PTEs map consecutive
* pages that belong to the same folio. The PTEs are all in the same PMD. */OK I think the key bit here is 'consecutive pages of the same folio'.
I'd like at least a paragraph about implementation, yes the original function doesn't have that (and should imo), something like:
We perform the operation on the first PTE, then if any others follow, we invoke the
ptep_modify_prot_start()for each and aggregate A/D bits.Something like this.
Point taken on consistency though!
Quoted 78 lines
Quoted 69 lines
So maybe something like:
pte =
ptep_modify_prot_start(vma, addr, ptep);/* Iterate through large folio tail PTEs. */
for (pg = 1; pg < nr; pg++) { pte_t inner_pte; ptep++; addr += PAGE_SIZE; inner_pte = ptep_modify_prot_start(vma, addr, ptep); /* We must propagate A/D state from tail PTEs. */ if (pte_dirty(inner_pte)) pte = pte_mkdirty(pte); if (pte_young(inner_pte)) pte = pte_mkyoung(pte); }Would work better?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 909
Quoted 10 lines
include/linux/pgtable.h +7
-0+ ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);Why are you propagating these?
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 915
Quoted 6 lines
include/linux/pgtable.h +6
-0+ } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ Same comments as above, needs more meat on the bones!
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 917
Quoted 2 lines
include/linux/pgtable.h +2
-0+#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,Again need to reference large folio, batched or something relevant here, 'ptes' is super vague.
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 918
Quoted 1 line
include/linux/pgtable.h +1
-0+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)Nit, but you put 'p' suffix on ptep but not on '
old_pte'?I'm even more concerned about the 'nr' API here now.
So this is now a user-calculated:
min3(large_folio_pages, number of pte entries left in ptep,
number of pte entries left inold_pte)It really feels like something that should be calculated here, or at least be broken out more clearly.
You definitely
_atthe veryleast_need to document it in a comment.Quoted 8 lines
include/linux/pgtable.h static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, · line 923
Quoted 5 lines
include/linux/pgtable.h +5
-0+{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ;Why are you doing an infinite loop here with a break like this? Again feels needlessly confusing.
Quoted 7 lines
I agree it's not pretty to look at. But apparently it's the most efficient. This is Willy's commit that started it all: Commit
bcc6cc832573("mm: add default definition ofset_ptes()").For the record, I think all your comments make good sense, Lorenzo. But there is an established style, and personally I think at this point is it more confusing to break from that style.
This isn't _quite_ style, I'd say it's implementation, we're kind of crossing over into something a little more I'd say :) but I mean I get your point, sure.
I mean, fine, if (I presume you're referring _only_ to the for (;;) case above) you are absolutely certain it is more performant in practice I wouldn't want to stand in the way of that.
Quoted 6 lines
Quoted 3 lines
No I'm not certain at all... I'm just saying that's been the argument in the past. I vaguely recall I even tried changing the loop style in batched helpers I implemented in the past and David asked me to stick with the established style.
I definitely defer to David's expertise, but I feel there's some room here for improving things.
Yeah, I recall Willy introducing that scheme, arguing that it is the most efficient once. Can't argue with that :)
Signature · 3 lines
Cheers, David / dhildenb
kernel test robot
build botRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev Jainkernel test robot noticed the following build errors:
Hi Dev,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on arm64/for-next/core linus/master v6.15-rc4 next-20250429] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Dev-Jain/mm-Refactor-code-in-mprotect/20250429-133151 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250429052336.18912-4-dev.jain%40arm.com patch subject: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit config: arm-randconfig-001-20250430 (https://download.01.org/0day-ci/archive/20250430/[email protected]/config) compiler: arm-linux-gnueabi-gcc (GCC) 10.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250430/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
In file included from include/linux/mm.h:31,
from arch/arm/kernel/asm-offsets.c:12:
include/linux/pgtable.h: In function 'modify_prot_start_ptes':
include/linux/pgtable.h:901:8: error: implicit declaration of function 'ptep_modify_prot_start' [-Werror=implicit-function-declaration]
901 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~
include/linux/pgtable.h: In function 'modify_prot_commit_ptes':
include/linux/pgtable.h:921:3: error: implicit declaration of function 'ptep_modify_prot_commit' [-Werror=implicit-function-declaration]
921 | ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
include/linux/pgtable.h: At top level:
>> include/linux/pgtable.h:1356:21: error: conflicting types for 'ptep_modify_prot_start'
1356 | static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,
| ^~~~~~~~~~~~~~~~~~~~~~
include/linux/pgtable.h:901:8: note: previous implicit declaration of 'ptep_modify_prot_start' was here
901 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~
>> include/linux/pgtable.h:1367:20: warning: conflicting types for 'ptep_modify_prot_commit'
1367 | static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,
| ^~~~~~~~~~~~~~~~~~~~~~~
include/linux/pgtable.h:1367:20: error: static declaration of 'ptep_modify_prot_commit' follows non-static declaration
include/linux/pgtable.h:921:3: note: previous implicit declaration of 'ptep_modify_prot_commit' was here
921 | ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~ cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:98: arch/arm/kernel/asm-offsets.s] Error 1 shuffle=2044487564 make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1280: prepare0] Error 2 shuffle=2044487564
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2 shuffle=2044487564
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2 shuffle=2044487564
make: Target 'prepare' not remade because of errors.
vim +/ptep_modify_prot_start +1356 include/linux/pgtable.h
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1340
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1341 #ifndef __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1342 /*
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1343 * Start a pte protection read-modify-write transaction, which
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1344 * protects against asynchronous hardware modifications to the pte.
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1345 * The intention is not to prevent the hardware from making pte
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1346 * updates, but to prevent any updates it may make from being lost.
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1347 *
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1348 * This does not protect against other software modifications of the
2eb70aab25dd9b include/linux/pgtable.h Bhaskar Chowdhury 2021-05-06 1349 * pte; the appropriate pte lock must be held over the transaction.
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1350 *
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1351 * Note that this interface is intended to be batchable, meaning that
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1352 * ptep_modify_prot_commit may not actually update the pte, but merely
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1353 * queue the update to be done at some later time. The update must be
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1354 * actually committed before the pte lock is released, however.
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1355 */
0cbe3e26abe0cf include/asm-generic/pgtable.h Aneesh Kumar K.V 2019-03-05 @1356 static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1357 unsigned long addr,
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1358 pte_t *ptep)
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1359 {
0cbe3e26abe0cf include/asm-generic/pgtable.h Aneesh Kumar K.V 2019-03-05 1360 return __ptep_modify_prot_start(vma, addr, ptep);
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1361 }
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1362
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1363 /*
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1364 * Commit an update to a pte, leaving any hardware-controlled bits in
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1365 * the PTE unmodified.
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1366 */
0cbe3e26abe0cf include/asm-generic/pgtable.h Aneesh Kumar K.V 2019-03-05 @1367 static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1368 unsigned long addr,
04a8645304500b include/asm-generic/pgtable.h Aneesh Kumar K.V 2019-03-05 1369 pte_t *ptep, pte_t old_pte, pte_t pte)
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1370 {
0cbe3e26abe0cf include/asm-generic/pgtable.h Aneesh Kumar K.V 2019-03-05 1371 __ptep_modify_prot_commit(vma, addr, ptep, pte);
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1372 }
1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1373 #endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */
fe1a6875fcaaac include/asm-generic/pgtable.h Sebastian Siewior 2008-07-15 1374 #endif /* CONFIG_MMU */1ea0704e0da65b include/asm-generic/pgtable.h Jeremy Fitzhardinge 2008-06-16 1375
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev Jainkernel test robot noticed the following build errors:
Hi Dev,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on arm64/for-next/core linus/master v6.15-rc4 next-20250429] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Dev-Jain/mm-Refactor-code-in-mprotect/20250429-133151 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250429052336.18912-4-dev.jain%40arm.com patch subject: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit config: arm-randconfig-003-20250430 (https://download.01.org/0day-ci/archive/20250430/[email protected]/config) compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250430/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:31:
>> include/linux/pgtable.h:901:8: error: call to undeclared function 'ptep_modify_prot_start'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
901 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^
>> include/linux/pgtable.h:921:3: error: call to undeclared function 'ptep_modify_prot_commit'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
921 | ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^
>> include/linux/pgtable.h:1356:21: error: static declaration of 'ptep_modify_prot_start' follows non-static declaration
1356 | static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,
| ^
include/linux/pgtable.h:901:8: note: previous implicit declaration is here
901 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^
include/linux/pgtable.h:1367:20: error: static declaration of 'ptep_modify_prot_commit' follows non-static declaration
1367 | static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,
| ^
include/linux/pgtable.h:921:3: note: previous implicit declaration is here
921 | ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:98:11: warning: array index 3 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
98 | return (set->sig[3] | set->sig[2] |
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:98:25: warning: array index 2 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
98 | return (set->sig[3] | set->sig[2] |
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:11: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:27: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:115:5: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
115 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:115:21: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
115 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:157:1: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
157 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:138:8: note: expanded from macro '_SIG_SET_BINOP'
138 | a3 = a->sig[3]; a2 = a->sig[2]; \
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:157:1: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
157 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:138:24: note: expanded from macro '_SIG_SET_BINOP'
138 | a3 = a->sig[3]; a2 = a->sig[2]; \
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^ In file included from arch/arm/kernel/asm-offsets.c:12:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
vim +/ptep_modify_prot_start +901 include/linux/pgtable.h
893
894 /* See the comment for ptep_modify_prot_start */
895 #ifndef modify_prot_start_ptes
896 static inline pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
897 unsigned long addr, pte_t *ptep, unsigned int nr)
898 {
899 pte_t pte, tmp_pte;
900
> 901 pte = ptep_modify_prot_start(vma, addr, ptep);
902 while (--nr) {
903 ptep++;
904 addr += PAGE_SIZE;
905 tmp_pte = ptep_modify_prot_start(vma, addr, ptep);
906 if (pte_dirty(tmp_pte))
907 pte = pte_mkdirty(pte);
908 if (pte_young(tmp_pte))
909 pte = pte_mkyoung(pte);
910 }
911 return pte;
912 }
913 #endif
914
915 /* See the comment for ptep_modify_prot_commit */
916 #ifndef modify_prot_commit_ptes
917 static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
918 pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)
919 {
920 for (;;) {
> 921 ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); 922 if (--nr == 0)
923 break;
924 ptep++;
925 addr += PAGE_SIZE;
926 old_pte = pte_next_pfn(old_pte);
927 pte = pte_next_pfn(pte);
928 }
929 }
930 #endif
931
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Ryan Roberts
reviewerRe: [PATCH v2 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 2 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect. Architecture can override these helpers.
I would suggest merging this with patch #7 since that's where they actually get used. Then you can add a single patch after that to specialize them for arm64, which will give a performance win.
Thanks,
Ryan
Quoted 55 lines
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)include/linux/pgtable.h +38
-0@@ -891,6 +891,44 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +for (;;) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings
Dev Jain
reviewerPATCH v2 4/7 arm64: Add batched version of ptep_modify_prot_start
In reply to Dev JainOverride the generic definition to use get_and_clear_full_ptes(), so that we do a TLBI possibly only on the "contpte-edges" of the large PTE block, instead of doing it for every contpte block, which happens for ptep_get_and_clear().
--- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/mm/mmu.c | 12 +++++++++--- include/linux/pgtable.h | 4 ++++ mm/pgtable-generic.c | 16 +++++++++++----- 4 files changed, 29 insertions(+), 8 deletions(-)
arch/arm64/include/asm/pgtable.h +5 -0
@@ -1553,6 +1553,11 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep, +unsigned int nr); +#ifdef CONFIG_ARM64_CONTPTE /*
arch/arm64/mm/mmu.c +9 -3
@@ -1523,7 +1523,8 @@ static int __init prevent_bootmem_remove_init(void)early_initcall (prevent_bootmem_remove_init);#endif -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +pte_t modify_prot_start_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,unsigned int nr) {if (alternative_has_cap_unlikely (ARM64_WORKAROUND_2645198 )) {/*
@@ -1532,9 +1533,14 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte* in cases where cpu is affected with errata #2645198. */ if (pte_user_exec (ptep_get (ptep))) -return ptep_clear_flush (vma, addr, ptep); +return clear_flush_ptes (vma, addr, ptep, nr); } -return ptep_get_and_clear (vma->vm_mm, addr, ptep); +return get_and_clear_full_ptes (vma->vm_mm, addr, ptep, nr,0 ); +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ +return modify_prot_start_ptes (vma, addr, ptep,1 ); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,
include/linux/pgtable.h +4 -0
@@ -828,6 +828,10 @@ extern pte_t ptep_clear_flush(struct vm_area_struct *vma, pte_t *ptep);#endif +extern pte_t clear_flush_ptes (struct vm_area_struct *vma, +unsigned long address, +pte_t *ptep,unsigned int nr); +#ifndef __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH extern pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,
mm/pgtable-generic.c +11 -5
@@ -90,17 +90,23 @@ int ptep_clear_flush_young(struct vm_area_struct *vma, }#endif -#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH -pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, -pte_t *ptep) +pte_t clear_flush_ptes (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {struct mm_struct *mm = (vma)->vm_mm;pte_t pte; - pte =ptep_get_and_clear (mm, address, ptep); + pte =get_and_clear_full_ptes (mm, address, ptep, nr,0 );if (pte_accessible (mm, pte)) -flush_tlb_page (vma, address); +flush_tlb_range (vma, address, address + nr *PAGE_SIZE );return pte; } + +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH +pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep) +{ +return clear_flush_ptes (vma, address, ptep,1 ); +}#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE
Signature · 1 line
2.30.2
Re: [PATCH v2 4/7] arm64: Add batched version of ptep_modify_prot_start
Quoted 3 lines · Dev Jain
Override the generic definition to use
get_and_clear_full_ptes(), so that we do a TLBI possibly only on the "contpte-edges" of the large PTE block, instead of doing it for every contpte block, which happens forptep_get_and_clear().
Could you please explain what does "contpte-edges" really signify in the context of large PTE blocks ? Also how TLBI operation only on these edges will never run into the risk of missing TLB invalidation of some other mapped areas ?
Quoted 104 lines
Signed-off-by Dev Jain <[email protected]>--- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/mm/mmu.c | 12 +++++++++--- include/linux/pgtable.h | 4 ++++ mm/pgtable-generic.c | 16 +++++++++++----- 4 files changed, 29 insertions(+), 8 deletions(-)arch/arm64/include/asm/pgtable.h +5
-0@@ -1553,6 +1553,11 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep, +unsigned int nr); +#ifdef CONFIG_ARM64_CONTPTE /* arch/arm64/mm/mmu.c +9
-3@@ -1523,7 +1523,8 @@ static int __init prevent_bootmem_remove_init(void)early_initcall (prevent_bootmem_remove_init);#endif -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +pte_t modify_prot_start_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,unsigned int nr) {if (alternative_has_cap_unlikely (ARM64_WORKAROUND_2645198 )) {/* @@ -1532,9 +1533,14 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte* in cases where cpu is affected with errata #2645198. */ if (pte_user_exec (ptep_get (ptep))) -return ptep_clear_flush (vma, addr, ptep); +return clear_flush_ptes (vma, addr, ptep, nr); } -return ptep_get_and_clear (vma->vm_mm, addr, ptep); +return get_and_clear_full_ptes (vma->vm_mm, addr, ptep, nr,0 ); +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ +return modify_prot_start_ptes (vma, addr, ptep,1 ); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,include/linux/pgtable.h +4
-0@@ -828,6 +828,10 @@ extern pte_t ptep_clear_flush(struct vm_area_struct *vma, pte_t *ptep);#endif +extern pte_t clear_flush_ptes (struct vm_area_struct *vma, +unsigned long address, +pte_t *ptep,unsigned int nr); +#ifndef __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH extern pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,mm/pgtable-generic.c +11
-5@@ -90,17 +90,23 @@ int ptep_clear_flush_young(struct vm_area_struct *vma, }#endif -#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH -pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, -pte_t *ptep) +pte_t clear_flush_ptes (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {struct mm_struct *mm = (vma)->vm_mm;pte_t pte; - pte =ptep_get_and_clear (mm, address, ptep); + pte =get_and_clear_full_ptes (mm, address, ptep, nr,0 );if (pte_accessible (mm, pte)) -flush_tlb_page (vma, address); +flush_tlb_range (vma, address, address + nr *PAGE_SIZE );return pte; } + +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH +pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep) +{ +return clear_flush_ptes (vma, address, ptep,1 ); +}#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE
Dev Jain
reviewerRe: [PATCH v2 4/7] arm64: Add batched version of ptep_modify_prot_start
Quoted 9 lines · Anshuman Khandual
Quoted 3 lines · Dev Jain
Override the generic definition to use
get_and_clear_full_ptes(), so that we do a TLBI possibly only on the "contpte-edges" of the large PTE block, instead of doing it for every contpte block, which happens forptep_get_and_clear().Could you please explain what does "contpte-edges" really signify in the context of large PTE blocks ? Also how TLBI operation only on these edges will never run into the risk of missing TLB invalidation of some other mapped areas ?
We are doing a TLBI over the whole range already, in the mprotect code:
see tlb_flush_pte_range. What the arm64 internal API does, irrespective of the caller, is to do a TLBI for every contpte block in case of unfolding. We don't need that for the intermediate blocks because the caller does that. We do need a TLBI for the start and end contpte block, because in case the range we are invalidating partially covers them, then the caller will not do a TLBI for the non-overlapped PTEs of the block.
I'll explain some more in the changelog next version.
Quoted 105 lines
Quoted 105 lines
Signed-off-by Dev Jain <[email protected]>--- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/mm/mmu.c | 12 +++++++++--- include/linux/pgtable.h | 4 ++++ mm/pgtable-generic.c | 16 +++++++++++----- 4 files changed, 29 insertions(+), 8 deletions(-)arch/arm64/include/asm/pgtable.h +5
-0@@ -1553,6 +1553,11 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep, +unsigned int nr); +#ifdef CONFIG_ARM64_CONTPTE /* arch/arm64/mm/mmu.c +9
-3@@ -1523,7 +1523,8 @@ static int __init prevent_bootmem_remove_init(void)early_initcall (prevent_bootmem_remove_init);#endif -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +pte_t modify_prot_start_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,unsigned int nr) {if (alternative_has_cap_unlikely (ARM64_WORKAROUND_2645198 )) {/* @@ -1532,9 +1533,14 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte* in cases where cpu is affected with errata #2645198. */ if (pte_user_exec (ptep_get (ptep))) -return ptep_clear_flush (vma, addr, ptep); +return clear_flush_ptes (vma, addr, ptep, nr); } -return ptep_get_and_clear (vma->vm_mm, addr, ptep); +return get_and_clear_full_ptes (vma->vm_mm, addr, ptep, nr,0 ); +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ +return modify_prot_start_ptes (vma, addr, ptep,1 ); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,include/linux/pgtable.h +4
-0@@ -828,6 +828,10 @@ extern pte_t ptep_clear_flush(struct vm_area_struct *vma, pte_t *ptep);#endif +extern pte_t clear_flush_ptes (struct vm_area_struct *vma, +unsigned long address, +pte_t *ptep,unsigned int nr); +#ifndef __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH extern pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,mm/pgtable-generic.c +11
-5@@ -90,17 +90,23 @@ int ptep_clear_flush_young(struct vm_area_struct *vma, }#endif -#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH -pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, -pte_t *ptep) +pte_t clear_flush_ptes (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep,unsigned int nr) {struct mm_struct *mm = (vma)->vm_mm;pte_t pte; - pte =ptep_get_and_clear (mm, address, ptep); + pte =get_and_clear_full_ptes (mm, address, ptep, nr,0 );if (pte_accessible (mm, pte)) -flush_tlb_page (vma, address); +flush_tlb_range (vma, address, address + nr *PAGE_SIZE );return pte; } + +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH +pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, +pte_t *ptep) +{ +return clear_flush_ptes (vma, address, ptep,1 ); +}#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE
Re: [PATCH v2 4/7] arm64: Add batched version of ptep_modify_prot_start
Quoted 8 lines · Dev Jain
Quoted 5 lines · Anshuman Khandual
Quoted 3 lines · Dev Jain
Override the generic definition to use
get_and_clear_full_ptes(), so that we do a TLBI possibly only on the "contpte-edges" of the large PTE block, instead of doing it for every contpte block, which happens forptep_get_and_clear().
Quoted 8 lines
Quoted 4 lines
Could you please explain what does "contpte-edges" really signify in the context of large PTE blocks ? Also how TLBI operation only on these edges will never run into the risk of missing TLB invalidation of some other mapped areas ?
We are doing a TLBI over the whole range already, in the mprotect code:
seetlb_flush_pte_range. What the arm64 internal API does, irrespective of the caller, is to do a TLBI for every contpte block in case of unfolding. We don't need that for the intermediate blocks because the caller does that. We do need a TLBI for the start and end contpte block, because in case the range we are invalidating partially covers them, then the caller will not do a TLBI for the non-overlapped PTEs of the block.
But is not splitting the TLBI flush responsibility between the callers (intermediate blocks) and the platform API (contpte-edges) - some what problematic from a semantics perspective, and will be more susceptible for missing TLB flushes etc ?
Quoted 101 lines
I'll explain some more in the changelog next version.
Quoted 98 lines
Quoted 98 lines
Signed-off-by Dev Jain <[email protected]>---
arch/arm64/include/asm/pgtable.h| 5 +++++ arch/arm64/mm/mmu.c | 12 +++++++++--- include/linux/pgtable.h | 4 ++++ mm/pgtable-generic.c | 16 +++++++++++-----4 files changed, 29 insertions(+), 8 deletions(-)arch/arm64/include/asm/pgtable.h +0
-0@@ -1553,6 +1553,11 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte);+#define
modify_prot_start_ptesmodify_prot_start_ptesarch/arm64/include/asm/pgtable.h +4
-0+extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr); +#ifdef
CONFIG_ARM64_CONTPTE
/*arch/arm64/mm/mmu.c +9
-2@@ -1523,7 +1523,8 @@ static int __init prevent_bootmem_remove_init(void) early_initcall(prevent_bootmem_remove_init); #endif -pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, + pte_t *ptep, unsigned int nr) { if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) { /*@@ -1532,9 +1533,14 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte* in cases where cpu is affected with errata #2645198. */ if (pte_user_exec(ptep_get(ptep))) - return ptep_clear_flush(vma, addr, ptep); + return clear_flush_ptes(vma, addr, ptep, nr); } - return ptep_get_and_clear(vma->vm_mm, addr, ptep); + return get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, 0); +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ + return modify_prot_start_ptes(vma, addr, ptep, 1); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,include/linux/pgtable.h +3
-0@@ -828,6 +828,10 @@ extern pte_t ptep_clear_flush(struct vm_area_struct *vma, pte_t *ptep); #endif +extern pte_t clear_flush_ptes(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep, unsigned int nr); + #ifndef __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH extern pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,mm/pgtable-generic.c +0
-0@@ -90,17 +90,23 @@ int ptep_clear_flush_young(struct vm_area_struct *vma, } #endif-#ifndef
__HAVE_ARCH_PTEP_CLEAR_FLUSHmm/pgtable-generic.c +2
-2-pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, - pte_t *ptep) +pte_t clear_flush_ptes (struct vm_area_struct *vma,unsigned long address, + pte_t *ptep, unsigned int nr){
structmm_struct*mm = (vma)->vm_mm;
pte_tpte;
- pte =ptep_get_and_clear(mm, address, ptep);
+ pte =get_and_clear_full_ptes(mm, address, ptep, nr, 0); if (pte_accessible(mm, pte))
-flush_tlb_page(vma, address);
+flush_tlb_range(vma, address, address + nr * PAGE_SIZE); return pte;
}mm/pgtable-generic.c +7
-0+ +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH +pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, + pte_t *ptep) +{ + return clear_flush_ptes(vma, address, ptep, 1); +}#endif
#ifdefCONFIG_TRANSPARENT_HUGEPAGE
Dev Jain
reviewerRe: [PATCH v2 4/7] arm64: Add batched version of ptep_modify_prot_start
Quoted 16 lines · Anshuman Khandual
Quoted 13 lines · Dev Jain
Quoted 9 lines · Anshuman Khandual
Quoted 3 lines · Dev Jain
Override the generic definition to use
get_and_clear_full_ptes(), so that we do a TLBI possibly only on the "contpte-edges" of the large PTE block, instead of doing it for every contpte block, which happens forptep_get_and_clear().Could you please explain what does "contpte-edges" really signify in the context of large PTE blocks ? Also how TLBI operation only on these edges will never run into the risk of missing TLB invalidation of some other mapped areas ?
Quoted 8 lines
Quoted 3 lines
We are doing a TLBI over the whole range already, in the mprotect code:
seetlb_flush_pte_range. What the arm64 internal API does, irrespective of the caller, is to do a TLBI for every contpte block in case of unfolding. We don't need that for the intermediate blocks because the caller does that. We do need a TLBI for the start and end contpte block, because in case the range we are invalidating partially covers them, then the caller will not do a TLBI for the non-overlapped PTEs of the block.But is not splitting the TLBI flush responsibility between the callers (intermediate blocks) and the platform API (contpte-edges) - some what problematic from a semantics perspective, and will be more susceptible for missing TLB flushes etc ?
I seem to agree that this is a semantic problem. Although, we won't ever miss a TLB flush - we will only have extras. It is the responsibility of the caller to do the TLBI, the platform API is only checking whether we need some more TLBIs.
Quoted 103 lines
Quoted 102 lines
I'll explain some more in the changelog next version.
Quoted 98 lines
Quoted 98 lines
Signed-off-by Dev Jain <[email protected]>--- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/mm/mmu.c | 12 +++++++++--- include/linux/pgtable.h | 4 ++++ mm/pgtable-generic.c | 16 +++++++++++----- 4 files changed, 29 insertions(+), 8 deletions(-)arch/arm64/include/asm/pgtable.h +4
-0@@ -1553,6 +1553,11 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr); + #ifdef CONFIG_ARM64_CONTPTE /*arch/arm64/mm/mmu.c +9
-2@@ -1523,7 +1523,8 @@ static int __init prevent_bootmem_remove_init(void) early_initcall(prevent_bootmem_remove_init); #endif -pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, + pte_t *ptep, unsigned int nr) { if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) { /*@@ -1532,9 +1533,14 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte* in cases where cpu is affected with errata #2645198. */ if (pte_user_exec(ptep_get(ptep))) - return ptep_clear_flush(vma, addr, ptep); + return clear_flush_ptes(vma, addr, ptep, nr); } - return ptep_get_and_clear(vma->vm_mm, addr, ptep); + return get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, 0); +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ + return modify_prot_start_ptes(vma, addr, ptep, 1); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,include/linux/pgtable.h +3
-0@@ -828,6 +828,10 @@ extern pte_t ptep_clear_flush(struct vm_area_struct *vma, pte_t *ptep); #endif +extern pte_t clear_flush_ptes(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep, unsigned int nr); + #ifndef __HAVE_ARCH_PMDP_HUGE_CLEAR_FLUSH extern pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,mm/pgtable-generic.c +11
-4@@ -90,17 +90,23 @@ int ptep_clear_flush_young(struct vm_area_struct *vma, } #endif -#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH -pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, - pte_t *ptep) +pte_t clear_flush_ptes (struct vm_area_struct *vma,unsigned long address, + pte_t *ptep, unsigned int nr) { struct mm_struct *mm = (vma)->vm_mm; pte_t pte; - pte = ptep_get_and_clear(mm, address, ptep); + pte = get_and_clear_full_ptes(mm, address, ptep, nr, 0); if (pte_accessible(mm, pte)) - flush_tlb_page(vma, address); + flush_tlb_range(vma, address, address + nr * PAGE_SIZE); return pte; } + +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH +pte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, + pte_t *ptep) +{ + return clear_flush_ptes(vma, address, ptep, 1); +} #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE
Dev Jain
reviewerPATCH v2 5/7 arm64: Add batched version of ptep_modify_prot_commit
In reply to Dev JainOverride the generic definition to simply use set_ptes() to map the new ptes into the pagetable.
--- arch/arm64/include/asm/pgtable.h | 5 +++++ arch/arm64/mm/mmu.c | 9 ++++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +5 -0
@@ -1558,6 +1558,11 @@ extern pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, unsigned int nr); +#define modify_prot_commit_ptes modify_prot_commit_ptes +extern void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte, +unsigned int nr); +#ifdef CONFIG_ARM64_CONTPTE /*
arch/arm64/mm/mmu.c +8 -1
@@ -1543,10 +1543,17 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, ptereturn modify_prot_start_ptes (vma, addr, ptep,1 ); } +void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte, +unsigned int nr) +{ +set_ptes (vma->vm_mm, addr, ptep, pte, nr); +} +void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t pte) { -set_pte_at (vma->vm_mm, addr, ptep, pte); +modify_prot_commit_ptes (vma, addr, ptep, old_pte, pte,1 ); }/*
Signature · 1 line
2.30.2
Dev Jain
reviewerPATCH v2 6/7 mm: Batch around can_change_pte_writable()
In reply to Dev JainIn preparation for patch 7, we need to properly batch around can_change_pte_writable(). We batch around pte_needs_soft_dirty_wp() by the corresponding fpb flag, we batch around the page-anon exclusive check using folio_maybe_mapped_shared(); modify_prot_start_ptes() collects the dirty and access bits across the batch, therefore batching across pte_dirty(): this is correct since the dirty bit on the PTE really is just an indication that the folio got written to, so even if the PTE is not actually dirty (but one of the PTEs in the batch is), the wp-fault optimization can be made.
--- include/linux/mm.h | 4 ++-- mm/gup.c | 2 +- mm/huge_memory.c | 4 ++-- mm/memory.c | 6 +++--- mm/mprotect.c | 11 ++++++----- 5 files changed, 14 insertions(+), 13 deletions(-)
include/linux/mm.h +2 -2
@@ -2487,8 +2487,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);#define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) -bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte); +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr);extern long change_protection (struct mmu_gather *tlb,struct vm_area_struct *vma,unsigned long start,unsigned long end,unsigned long cp_flags);
mm/gup.c +1 -1
@@ -614,7 +614,7 @@ static inline bool can_follow_write_common(struct page *page,return false;/* -* See can_change_pte_writable(): we broke COW and could map the page +* See can_change_ptes_writable(): we broke COW and could map the page * writable if we have an exclusive anonymous page ... */ return page &&PageAnon (page) &&PageAnonExclusive (page);
mm/huge_memory.c +2 -2
@@ -2032,12 +2032,12 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,return false;if (!(vma->vm_flags &VM_SHARED )) { -/* See can_change_pte_writable(). */ +/* See can_change_ptes_writable(). */ page =vm_normal_page_pmd (vma, addr, pmd);return page &&PageAnon (page) &&PageAnonExclusive (page); } -/* See can_change_pte_writable(). */ +/* See can_change_ptes_writable(). */ return pmd_dirty (pmd); }
mm/memory.c +3 -3
@@ -750,7 +750,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, pte =pte_mkuffd_wp (pte);if ((vma->vm_flags &VM_WRITE ) && -can_change_pte_writable (vma, address, pte)) { +can_change_ptes_writable (vma, address, pte,NULL ,1 )) {if (folio_test_dirty (folio)) pte =pte_mkdirty (pte); pte =pte_mkwrite (pte, vma);
@@ -5796,7 +5796,7 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru ptent =pte_modify (ptent, vma->vm_page_prot); writable =pte_write (ptent);if (!writable && pte_write_upgrade && -can_change_pte_writable (vma, addr, ptent)) +can_change_ptes_writable (vma, addr, ptent,NULL ,1 )) writable = true; }
@@ -5837,7 +5837,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)*/ writable =pte_write (pte);if (!writable && pte_write_upgrade && -can_change_pte_writable (vma, vmf->address, pte)) +can_change_ptes_writable (vma, vmf->address, pte,NULL ,1 )) writable = true; folio =vm_normal_folio (vma, vmf->address, pte);
mm/mprotect.c +6 -5
@@ -40,8 +40,8 @@#include "internal.h" -bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;
@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio); }VM_WARN_ON_ONCE (is_zero_pfn (pte_pfn (pte)) &&pte_dirty (pte));
@@ -222,7 +223,7 @@ static long change_pte_range(struct mmu_gather *tlb,*/ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pte_write (ptent) && -can_change_pte_writable (vma, addr, ptent)) +can_change_ptes_writable (vma, addr, ptent, folio,1 )) ptent =pte_mkwrite (ptent, vma);ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent);
Signature · 1 line
2.30.2
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
Quoted 9 lines · Dev Jain
In preparation for patch 7, we need to properly batch around
can_change_pte_writable(). We batch aroundpte_needs_soft_dirty_wp()by the corresponding fpb flag, we batch around the page-anon exclusive check usingfolio_maybe_mapped_shared();modify_prot_start_ptes()collects the dirty and access bits across the batch, therefore batching acrosspte_dirty(): this is correct since the dirty bit on the PTE really is just an indication that the folio got written to, so even if the PTE is not actually dirty (but one of the PTEs in the batch is), the wp-fault optimization can be made.
If you want to add a batched version of can_change_pte_writable(), do it right away instead of just adding parameters to functions.
Then, add a simple
#define can_change_pte_writable(...) can_change_ptes_writable(..., 1);
So you don't have to touch all callers and don't have to update the function name in comments.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
In reply to Dev JainQuoted 11 lines
#include "internal.h"mm/mprotect.c +2
-2-bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 72
Quoted 8 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio);
Oh no, now I spot it. That is horribly wrong.
Please understand first what you are doing.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
Quoted 22 lines · David Hildenbrand
Quoted 11 lines
#include "internal.h"mm/mprotect.c +2
-2-bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 72
Quoted 8 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the cow.c selftest would catch that:
"vmsplice() + unmap in child with mprotect() optimization"
After fork() we have a R/O PTE in the parent. Our child then uses vmsplice() and unmaps the R/O PTE, meaning it is only left mapped by the parent.
ret = mprotect(mem, size, PROT_READ);
ret |= mprotect(mem, size, PROT_READ|PROT_WRITE);
should turn the PTE writable, although it shouldn't.
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
Quoted 27 lines · David Hildenbrand
Quoted 20 lines · David Hildenbrand
Quoted 9 lines
#include "internal.h"mm/mprotect.c +2
-2-bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 72
Quoted 8 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the
cow.cselftest would catch that:"
vmsplice()+ unmap in child withmprotect()optimization"
Quoted 7 lines
After
fork()we have a R/O PTE in the parent. Our child then usesvmsplice()and unmaps the R/O PTE, meaning it is only left mapped by the parent.ret =
mprotect(mem, size, PROT_READ);
ret |=mprotect(mem, size, PROT_READ|PROT_WRITE);should turn the PTE writable, although it shouldn't.
This makes me concerned about the stability of this series as a whole...
Quoted 3 lines
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
There are 25 tests that fail for the cow self-test with this series applied:
# [RUN] vmsplice() + unmap in child with mprotect() optimization ... with base page
# [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (16 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (16 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (16 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (32 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (32 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (32 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (64 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (64 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (64 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (128 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (128 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (128 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (256 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (256 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (256 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (512 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (512 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (512 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (1024 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (1024 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (1024 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with PTE-mapped THP (2048 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with single PTE of THP (2048 kB) # [RUN] vmsplice() + unmap in child with mprotect() optimization ... with partially shared THP (2048 kB)
Dev, please take a little more time to test your series :) the current patch set doesn't compile and needs fixes applied to do so, and we're at v2, and you've clearly not run self-tests as these also fail.
Please ensure you do a smoke test and check compilation before sending out, as well as running self tests also.
Thanks, Lorenzo
Quoted 6 lines
--
Cheers,David / dhildenb
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
Quoted 62 lines · Lorenzo Stoakes
Quoted 27 lines · David Hildenbrand
Quoted 20 lines · David Hildenbrand
Quoted 9 lines
#include "internal.h"mm/mprotect.c +2
-2-bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 72
Quoted 8 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the
cow.cselftest would catch that:"
vmsplice()+ unmap in child withmprotect()optimization"Quoted 7 lines
After
fork()we have a R/O PTE in the parent. Our child then usesvmsplice()and unmaps the R/O PTE, meaning it is only left mapped by the parent.ret =
mprotect(mem, size, PROT_READ);
ret |=mprotect(mem, size, PROT_READ|PROT_WRITE);should turn the PTE writable, although it shouldn't.
This makes me concerned about the stability of this series as a whole...
Quoted 3 lines
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
There are 25 tests that fail for the cow self-test with this series applied:
# [RUN]
vmsplice()+ unmap in child withmprotect()optimization ... with base page
# [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (512 kB)
Quoted 8 lines
# [RUN]
vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (512 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (512 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (2048 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (2048 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (2048 kB)
As expected ... :)
Signature · 3 lines
Cheers, David / dhildenb
Dev Jain
reviewerRe: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
In reply to Lorenzo StoakesQuoted 72 lines · Lorenzo Stoakes
Quoted 27 lines · David Hildenbrand
Quoted 20 lines · David Hildenbrand
Quoted 9 lines
#include "internal.h"mm/mprotect.c +2
-2-bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +bool can_change_ptes_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte,struct folio *folio,unsigned int nr) {struct page *page;mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 72
Quoted 8 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,* write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); +if (!folio) + folio =vm_normal_folio (vma, addr, pte); +return folio_test_anon (folio) && !folio_maybe_mapped_shared (folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the
cow.cselftest would catch that:"
vmsplice()+ unmap in child withmprotect()optimization"Quoted 7 lines
After
fork()we have a R/O PTE in the parent. Our child then usesvmsplice()and unmaps the R/O PTE, meaning it is only left mapped by the parent.ret =
mprotect(mem, size, PROT_READ);
ret |=mprotect(mem, size, PROT_READ|PROT_WRITE);should turn the PTE writable, although it shouldn't.
This makes me concerned about the stability of this series as a whole...
Quoted 3 lines
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
There are 25 tests that fail for the cow self-test with this series applied:
# [RUN]
vmsplice()+ unmap in child withmprotect()optimization ... with base page
# [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (16 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (32 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (64 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (128 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (256 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (512 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (512 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (512 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (1024 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with PTE-mapped THP (2048 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with single PTE of THP (2048 kB) # [RUN]vmsplice()+ unmap in child withmprotect()optimization ... with partially shared THP (2048 kB)
Quoted 6 lines
Dev, please take a little more time to test your series :) the current patch set doesn't compile and needs fixes applied to do so, and we're at v2, and you've clearly not run self-tests as these also fail.
Please ensure you do a smoke test and check compilation before sending out, as well as running self tests also.
Apologies, I over-confidently skipped over selftests, and didn't build for x86 :( Shall take care.
Quoted 9 lines
Thanks, Lorenzo
Quoted 6 lines
--
Cheers,David / dhildenb
Dev Jain
reviewerRe: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
In reply to David HildenbrandQuoted 31 lines · David Hildenbrand
Quoted 24 lines · David Hildenbrand
Quoted 11 lines
#include "
internal.h"
-bool can_change_pte_writable(structvm_area_struct*vma, unsigned long addr,
-pte_tpte)
+bool can_change_ptes_writable(structvm_area_struct*vma, unsigned long addr,
+pte_tpte, struct folio *folio, unsigned int nr) {
struct page *page;mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct · line 74
Quoted 10 lines
mm/mprotect.c +3
-2@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct*vma, unsigned long addr, * write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page = vm_normal_page(vma, addr, pte); - return page && PageAnon(page) && PageAnonExclusive(page); + if (!folio) + folio = vm_normal_folio(vma, addr, pte); + return folio_test_anon(folio) && !folio_maybe_mapped_shared (folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the
cow.cselftest would catch that:"
vmsplice()+ unmap in child withmprotect()optimization"
Quoted 9 lines
After
fork()we have a R/O PTE in the parent. Our child then usesvmsplice()and unmaps the R/O PTE, meaning it is only left mapped by the parent.ret =
mprotect(mem, size, PROT_READ);
ret |=mprotect(mem, size, PROT_READ|PROT_WRITE);should turn the PTE writable, although it shouldn't.
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
Hi David, I am afraid I don't understand my mistake :( PageAnon(page) boils down to folio_test_anon(folio). Next we want to determine whether the page underlying a PTE is mapped exclusively or not. I approximate this by folio_maybe_mapped_shared -> if the folio => all pages are mapped exclusively, then I convert the entire batch to writable. If one of the pages is mapped shared, then I do not convert the batch to writable, thus missing out on the optimization. As far as I understand, the test failure points out exactly this right?
Do you suggest an alternate way? My initial approach was to add a new flag to folio_pte_batch: FPB_IGNORE_ANON_EXCLUSIVE, but from an API design PoV Ryan pointed out that that looked bad.
Re: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
Quoted 44 lines · Dev Jain
Quoted 41 lines · David Hildenbrand
Quoted 25 lines · David Hildenbrand
Quoted 14 lines
#include "
internal.h"
-bool can_change_pte_writable(structvm_area_struct*vma, unsigned long addr,
-pte_tpte)
+bool can_change_ptes_writable(structvm_area_struct*vma, unsigned long addr,
+pte_tpte, struct folio *folio, unsigned int nr)
{
struct page *page;(quoted diff) +0
-0@@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, * write-fault handler similarly would map them writablewithout
Quoted 8 lines
* any additional checks while holding the PT lock. */
- page =vm_normal_page(vma, addr, pte);
- return page && PageAnon(page) && PageAnonExclusive(page); + if (!folio)
+ folio =vm_normal_folio(vma, addr, pte);
+ returnfolio_test_anon(folio)&& !folio_maybe_mapped_shared(folio);Oh no, now I spot it. That is horribly wrong.
Quoted 1 line
Please understand first what you are doing.
Also, would expect that the
cow.cselftest would catch that:"
vmsplice()+ unmap in child withmprotect()optimization"After
fork()we have a R/O PTE in the parent. Our child then usesvmsplice()and unmaps the R/O PTE, meaning it is only left mapped by the parent.ret =
mprotect(mem, size, PROT_READ);
ret |=mprotect(mem, size, PROT_READ|PROT_WRITE);should turn the PTE writable, although it shouldn't.
Quoted 7 lines
Quoted 3 lines
If that test case does not detect the issue you're introducing, we should look into adding a test case that detects it.
Hi David, I am afraid I don't understand my mistake :( PageAnon(page) boils down to
folio_test_anon(folio). Next we want to determine whether the page underlying a PTE is mapped exclusively or not.
No. :)
There is your mistake.
We need to know if this folio page is *exclusive* not if the folio is *mapped exclusively*.
I know, it's confusing, but that's an important distinction.
You really have to test all PAE bits. I recently sent a mail on how we could remove PAE and encode it in the PTE value itself (W || (!W +dirty)), which would mean that we could batch more easily. For the time being, we have to stick with what we have.
Signature · 3 lines
Cheers, David / dhildenb
kernel test robot
build botRe: [PATCH v2 6/7] mm: Batch around can_change_pte_writable()
In reply to Dev Jainkernel test robot noticed the following build warnings:
Hi Dev,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on arm64/for-next/core linus/master v6.15-rc4 next-20250429] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Dev-Jain/mm-Refactor-code-in-mprotect/20250429-133151 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250429052336.18912-7-dev.jain%40arm.com patch subject: [PATCH v2 6/7] mm: Batch around can_change_pte_writable() config: arm64-randconfig-002-20250430 (https://download.01.org/0day-ci/archive/20250430/[email protected]/config) compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250430/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> mm/mprotect.c:46:15: warning: unused variable 'page' [-Wunused-variable]
46 | struct page *page;
| ^~~~ mm/mprotect.c:226:51: error: use of undeclared identifier 'folio'
226 | can_change_ptes_writable(vma, addr, ptent, folio, 1))
| ^
1 warning and 1 error generated.
vim +/page +46 mm/mprotect.c
36f881883c5794 Kirill A. Shutemov 2015-06-24 42
695112a1385b39 Dev Jain 2025-04-29 43 bool can_change_ptes_writable(struct vm_area_struct *vma, unsigned long addr,
695112a1385b39 Dev Jain 2025-04-29 44 pte_t pte, struct folio *folio, unsigned int nr)
64fe24a3e05e5f David Hildenbrand 2022-06-14 45 {
64fe24a3e05e5f David Hildenbrand 2022-06-14 @46 struct page *page;
64fe24a3e05e5f David Hildenbrand 2022-06-14 47
7ea7e333842ed5 David Hildenbrand 2022-11-08 48 if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE)))
7ea7e333842ed5 David Hildenbrand 2022-11-08 49 return false;
64fe24a3e05e5f David Hildenbrand 2022-06-14 50
7ea7e333842ed5 David Hildenbrand 2022-11-08 51 /* Don't touch entries that are not even readable. */
d84887739d5c98 Nadav Amit 2022-11-08 52 if (pte_protnone(pte))
64fe24a3e05e5f David Hildenbrand 2022-06-14 53 return false;
64fe24a3e05e5f David Hildenbrand 2022-06-14 54
64fe24a3e05e5f David Hildenbrand 2022-06-14 55 /* Do we need write faults for softdirty tracking? */
f38ee285191813 Barry Song 2024-06-08 56 if (pte_needs_soft_dirty_wp(vma, pte))
64fe24a3e05e5f David Hildenbrand 2022-06-14 57 return false;
64fe24a3e05e5f David Hildenbrand 2022-06-14 58
64fe24a3e05e5f David Hildenbrand 2022-06-14 59 /* Do we need write faults for uffd-wp tracking? */
64fe24a3e05e5f David Hildenbrand 2022-06-14 60 if (userfaultfd_pte_wp(vma, pte))
64fe24a3e05e5f David Hildenbrand 2022-06-14 61 return false;
64fe24a3e05e5f David Hildenbrand 2022-06-14 62
64fe24a3e05e5f David Hildenbrand 2022-06-14 63 if (!(vma->vm_flags & VM_SHARED)) {
64fe24a3e05e5f David Hildenbrand 2022-06-14 64 /*
7ea7e333842ed5 David Hildenbrand 2022-11-08 65 * Writable MAP_PRIVATE mapping: We can only special-case on
7ea7e333842ed5 David Hildenbrand 2022-11-08 66 * exclusive anonymous pages, because we know that our
7ea7e333842ed5 David Hildenbrand 2022-11-08 67 * write-fault handler similarly would map them writable without
7ea7e333842ed5 David Hildenbrand 2022-11-08 68 * any additional checks while holding the PT lock.
64fe24a3e05e5f David Hildenbrand 2022-06-14 69 */
695112a1385b39 Dev Jain 2025-04-29 70 if (!folio)
695112a1385b39 Dev Jain 2025-04-29 71 folio = vm_normal_folio(vma, addr, pte);
695112a1385b39 Dev Jain 2025-04-29 72 return folio_test_anon(folio) && !folio_maybe_mapped_shared(folio);
64fe24a3e05e5f David Hildenbrand 2022-06-14 73 }
64fe24a3e05e5f David Hildenbrand 2022-06-14 74
fce831c92092ad David Hildenbrand 2024-05-22 75 VM_WARN_ON_ONCE(is_zero_pfn(pte_pfn(pte)) && pte_dirty(pte));
fce831c92092ad David Hildenbrand 2024-05-22 76
7ea7e333842ed5 David Hildenbrand 2022-11-08 77 /*
7ea7e333842ed5 David Hildenbrand 2022-11-08 78 * Writable MAP_SHARED mapping: "clean" might indicate that the FS still
7ea7e333842ed5 David Hildenbrand 2022-11-08 79 * needs a real write-fault for writenotify
7ea7e333842ed5 David Hildenbrand 2022-11-08 80 * (see vma_wants_writenotify()). If "dirty", the assumption is that the
7ea7e333842ed5 David Hildenbrand 2022-11-08 81 * FS was already notified and we can simply mark the PTE writable
7ea7e333842ed5 David Hildenbrand 2022-11-08 82 * just like the write-fault handler would do.
7ea7e333842ed5 David Hildenbrand 2022-11-08 83 */
d84887739d5c98 Nadav Amit 2022-11-08 84 return pte_dirty(pte);
64fe24a3e05e5f David Hildenbrand 2022-06-14 85 }
64fe24a3e05e5f David Hildenbrand 2022-06-14 86 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Dev Jain
reviewerPATCH v2 7/7 mm: Optimize mprotect() through PTE-batching
In reply to Dev JainThe common pte_present case does not require the folio. Elide the overhead of vm_normal_folio() for the small folio case, by making an approximation:
for arm64, pte_batch_hint() is conclusive. For other arches, if the pfns pointed to by the current and the next PTE are contiguous, check whether a large folio is actually mapped, and only then make the batch optimization. Reuse the folio from prot_numa case if possible. Since modify_prot_start_ptes() gathers access/dirty bits, it lets us batch around pte_needs_flush() (for parisc, the definition includes the access bit).
--- mm/mprotect.c | 49 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 14 deletions(-)
mm/mprotect.c +35 -14
@@ -129,7 +129,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, struct folio *folio,return false; } -static bool prot_numa_avoid_fault (struct vm_area_struct *vma, +static struct folio *prot_numa_avoid_fault (struct vm_area_struct *vma,unsigned long addr,pte_t *pte,pte_t oldpte,int target_node,int max_nr,int *nr) {
@@ -139,25 +139,37 @@ static bool prot_numa_avoid_fault(struct vm_area_struct *vma,/* Avoid TLB flush if possible */ if (pte_protnone (oldpte)) -return true; +return NULL ; folio =vm_normal_folio (vma, addr, oldpte);if (!folio) -return true; +return NULL ; ret =prot_numa_skip (vma, folio, target_node);if (ret) {if (folio_test_large (folio) && max_nr !=1 ) *nr =folio_pte_batch (folio, addr, pte, oldpte, max_nr, flags,NULL ,NULL ,NULL ); -return ret; +return NULL ; }if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -return false; +return folio; } +static bool maybe_contiguous_pte_pfns (pte_t *ptep,pte_t pte) +{ +pte_t *next_ptep, next_pte; + +if (pte_batch_hint (ptep, pte) !=1 ) +return true; + + next_ptep = ptep +1 ; + next_pte =ptep_get (next_ptep); + +return unlikely (pte_pfn (next_pte) -pte_pfn (pte) ==PAGE_SIZE ); +}static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)
@@ -188,19 +200,28 @@ static long change_pte_range(struct mmu_gather *tlb, oldpte =ptep_get (pte);if (pte_present (oldpte)) {int max_nr = (end - addr) >>PAGE_SHIFT ; +const fpb_t flags =FPB_IGNORE_DIRTY ; +struct folio *folio =NULL ;pte_t ptent;/* * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ -if (prot_numa && -prot_numa_avoid_fault (vma, addr, pte, - oldpte, target_node, - max_nr, &nr)) +if (prot_numa) { + folio =prot_numa_avoid_fault (vma, addr, pte, + oldpte, target_node, max_nr, &nr); +if (!folio)continue ; + } - oldpte =ptep_modify_prot_start (vma, addr, pte); +if (!folio && (max_nr !=1 ) &&maybe_contiguous_pte_pfns (pte, oldpte)) { + folio =vm_normal_folio (vma, addr, oldpte); +if (folio_test_large (folio)) + nr =folio_pte_batch (folio, addr, pte, + oldpte, max_nr, flags,NULL ,NULL ,NULL ); + } + oldpte =modify_prot_start_ptes (vma, addr, pte, nr); ptent =pte_modify (oldpte, newprot);if (uffd_wp)
@@ -223,13 +244,13 @@ static long change_pte_range(struct mmu_gather *tlb,*/ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pte_write (ptent) && -can_change_ptes_writable (vma, addr, ptent, folio,1 )) +can_change_ptes_writable (vma, addr, ptent, folio, nr)) ptent =pte_mkwrite (ptent, vma); -ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); +modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr);if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; +tlb_flush_pte_range (tlb, addr, nr *PAGE_SIZE ); + pages += nr; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte;
Signature · 1 line
2.30.2
Lance Yang
reviewerHey Dev,
Hmm... I also hit the same compilation errors:
In file included from ./include/linux/kasan.h:37,
from ./include/linux/slab.h:260,
from ./include/linux/crypto.h:19,
from arch/x86/kernel/asm-offsets.c:9:
./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:
./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~
./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 | tmp_pte = ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~
./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:
./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 | ptep_modify_prot_commit(vma, addr, ptep,
old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(struct vm_area_struct *, long unsigned int, pte_t *)’
1360 | static inline pte_t ptep_modify_prot_start(structvm_area_struct *vma,
| ^~~~~~~~~~~~~~~~~~~~~~
./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte = ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~
./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(struct vm_area_struct *, long unsigned int, pte_t *, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(struct
vm_area_struct *vma,
| ^~~~~~~~~~~~~~~~~~~~~~~
./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration
./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(struct vm_area_struct *, long unsigned int, pte_t *, pte_t, pte_t)’
925 | ptep_modify_prot_commit(vma, addr, ptep,
old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~ CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98: arch/x86/kernel/asm-offsets.s] Error 1 make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2 make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2
Well, modify_prot_start_ptes() calls ptep_modify_prot_start(), but x86 does not define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independent ptep_modify_prot_start() must be defined before modify_prot_start_ptes().
With the changes below, things work correctly now ;)
include/linux/pgtable.h +0 -38
@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, }#endif -/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,unsigned int nr) -{ -pte_t pte, tmp_pte; - - pte =ptep_modify_prot_start (vma, addr, ptep); -while (--nr) { - ptep++; - addr +=PAGE_SIZE ; - tmp_pte =ptep_modify_prot_start (vma, addr, ptep); -if (pte_dirty (tmp_pte)) - pte =pte_mkdirty (pte); -if (pte_young (tmp_pte)) - pte =pte_mkyoung (pte); - } -return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) -{ -for (;;) { -ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); -if (--nr ==0 ) -break ; - ptep++; - addr +=PAGE_SIZE ; - old_pte =pte_next_pfn (old_pte); - pte =pte_next_pfn (pte); - } -} -#endif -/* * On some architectures hardware does not set page access bit when
accessing
* memory page, it is responsibility of software setting this bit. It
brings
@@ -1375,6 +1337,45 @@ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,
__ptep_modify_prot_commit(vma, addr, ptep, pte);
}
#endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */
+
+/* See the comment for ptep_modify_prot_start */
+#ifndef modify_prot_start_ptes
+static inline pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep, unsigned int nr)
+{
+ pte_t pte, tmp_pte;
+
+ pte = ptep_modify_prot_start(vma, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = ptep_modify_prot_start(vma, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = pte_mkdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
+ return pte;
+}
+#endif
+
+/* See the comment for ptep_modify_prot_commit */
+#ifndef modify_prot_commit_ptes
+static inline void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr)
+{
+ for (;;) {
+ ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ old_pte = pte_next_pfn(old_pte);
+ pte = pte_next_pfn(pte);
+ }
+}
+#endif
+
#endif /* CONFIG_MMU */
/*
--
Thanks,
Lance
Quoted 46 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batchingarch/arm64/include/asm/pgtable.h | 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)
Dev Jain
reviewerQuoted 56 lines · Lance Yang
Hey Dev,
Hmm... I also hit the same compilation errors:
In file included from
./include/linux/kasan.h:37,
from./include/linux/slab.h:260,
from./include/linux/crypto.h:19,
fromarch/x86/kernel/asm-offsets.c:9:./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte =ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 |tmp_pte=ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(structvm_area_struct*, long unsigned int,pte_t*)’
1360 | static inlinepte_tptep_modify_prot_start(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte =ptep_modify_prot_start(vma, addr, ptep);
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98:arch/x86/kernel/asm-offsets.s] Error 1
make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2
make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/
objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2
Quoted 6 lines
Well,
modify_prot_start_ptes()callsptep_modify_prot_start(), but x86 does not define__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independentptep_modify_prot_start()must be defined beforemodify_prot_start_ptes().With the changes below, things work correctly now ;)
Ah thanks! My bad :(
Quoted 155 lines
include/linux/pgtable.h +0
-38@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } #endif -/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - pte_t pte, tmp_pte; - - pte = ptep_modify_prot_start(vma, addr, ptep); - while (--nr) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_modify_prot_start(vma, addr, ptep); - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes (struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) -{ - for (;;) { - ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - old_pte = pte_next_pfn(old_pte); - pte = pte_next_pfn(pte); - } -} -#endif - /** On some architectures hardware does not set page access bit when
accessing
* memory page, it is responsibility of software setting this bit. It
bringsinclude/linux/pgtable.h +39
-0@@ -1375,6 +1337,45 @@ static inline void ptep_modify_prot_commit(structvm_area_struct *vma, __ptep_modify_prot_commit(vma, addr, ptep, pte); } #endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, unsigned int nr) +{ + pte_t pte, tmp_pte; + + pte = ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, + pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) +{ + for (;;) { + ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + old_pte = pte_next_pfn(old_pte); + pte = pte_next_pfn(pte); + } +} +#endif + #endif /* CONFIG_MMU *//*
Signature · 2 lines
Thanks, LanceQuoted 46 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batching
arch/arm64/include/asm/pgtable.h| 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++
mm/gup.c | 2 +-
mm/huge_memory.c | 4 +-
mm/memory.c | 6 +-
mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++-9 files changed, 198 insertions(+), 72 deletions(-)
FWIW can confirm the same thing. Lance's fixes sort most of it out, but I also get this error:
mm/mprotect.c: In function ‘can_change_ptes_writable’:mm/mprotect.c:46:22: error: unused variable ‘page’ [-Werror=unused-variable]
46 | struct page *page;
| ^~~~
So you also need to remove this unused variable at the stop of can_change_ptes_writable().
Cheers, Lorenzo
Quoted 241 lines · Dev Jain
Quoted 72 lines · Lance Yang
Hey Dev,
Hmm... I also hit the same compilation errors:
In file included from
./include/linux/kasan.h:37,
from./include/linux/slab.h:260,
from./include/linux/crypto.h:19,
fromarch/x86/kernel/asm-offsets.c:9:./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 |tmp_pte=ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(structvm_area_struct*, long unsigned int,pte_t*)’
1360 | static inlinepte_tptep_modify_prot_start(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98:arch/x86/kernel/asm-offsets.s] Error 1
make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2
make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/
objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2Quoted 6 lines
Well,
modify_prot_start_ptes()callsptep_modify_prot_start(), but x86 does not define__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independentptep_modify_prot_start()must be defined beforemodify_prot_start_ptes().With the changes below, things work correctly now ;)
Ah thanks! My bad :(
Quoted 156 lines
include/linux/pgtable.h +0
-38@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } #endif -/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - pte_t pte, tmp_pte; - - pte = ptep_modify_prot_start(vma, addr, ptep); - while (--nr) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_modify_prot_start(vma, addr, ptep); - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes (struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) -{ - for (;;) { - ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - old_pte = pte_next_pfn(old_pte); - pte = pte_next_pfn(pte); - } -} -#endif - /** On some architectures hardware does not set page access bit when
accessing
* memory page, it is responsibility of software setting this bit. It
bringsinclude/linux/pgtable.h +39
-0@@ -1375,6 +1337,45 @@ static inline void ptep_modify_prot_commit(structvm_area_struct *vma, __ptep_modify_prot_commit(vma, addr, ptep, pte); } #endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, unsigned int nr) +{ + pte_t pte, tmp_pte; + + pte = ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, + pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) +{ + for (;;) { + ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + old_pte = pte_next_pfn(old_pte); + pte = pte_next_pfn(pte); + } +} +#endif + #endif /* CONFIG_MMU *//*
--Thanks,
LanceQuoted 47 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batching
arch/arm64/include/asm/pgtable.h| 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++
mm/gup.c | 2 +-
mm/huge_memory.c | 4 +-
mm/memory.c | 6 +-
mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++-9 files changed, 198 insertions(+), 72 deletions(-)
Dev Jain
reviewerQuoted 10 lines · Lorenzo Stoakes
FWIW can confirm the same thing. Lance's fixes sort most of it out, but I also get this error:
mm/mprotect.c: In function ‘can_change_ptes_writable’:mm/mprotect.c:46:22: error: unused variable ‘page’ [-Werror=unused-variable]46 | struct page *page; | ^~~~So you also need to remove this unused variable at the stop of
can_change_ptes_writable().
Strange that my build didn't catch this.
Quoted 245 lines
Cheers, Lorenzo
Quoted 241 lines · Dev Jain
Quoted 72 lines · Lance Yang
Hey Dev,
Hmm... I also hit the same compilation errors:
In file included from
./include/linux/kasan.h:37,
from./include/linux/slab.h:260,
from./include/linux/crypto.h:19,
fromarch/x86/kernel/asm-offsets.c:9:./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 |tmp_pte=ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(structvm_area_struct*, long unsigned int,pte_t*)’
1360 | static inlinepte_tptep_modify_prot_start(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
925 | ptep_modify_prot_commit(vma, addr, ptep,old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98:arch/x86/kernel/asm-offsets.s] Error 1
make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2
make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/
objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/
objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2Quoted 6 lines
Well,
modify_prot_start_ptes()callsptep_modify_prot_start(), but x86 does not define__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independentptep_modify_prot_start()must be defined beforemodify_prot_start_ptes().With the changes below, things work correctly now ;)
Ah thanks! My bad :(
Quoted 156 lines
include/linux/pgtable.h +0
-38@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } #endif -/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - pte_t pte, tmp_pte; - - pte = ptep_modify_prot_start(vma, addr, ptep); - while (--nr) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_modify_prot_start(vma, addr, ptep); - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes (struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) -{ - for (;;) { - ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - old_pte = pte_next_pfn(old_pte); - pte = pte_next_pfn(pte); - } -} -#endif - /** On some architectures hardware does not set page access bit when
accessing
* memory page, it is responsibility of software setting this bit. It
bringsinclude/linux/pgtable.h +39
-0@@ -1375,6 +1337,45 @@ static inline void ptep_modify_prot_commit(structvm_area_struct *vma, __ptep_modify_prot_commit(vma, addr, ptep, pte); } #endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, unsigned int nr) +{ + pte_t pte, tmp_pte; + + pte = ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, + pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) +{ + for (;;) { + ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + old_pte = pte_next_pfn(old_pte); + pte = pte_next_pfn(pte); + } +} +#endif + #endif /* CONFIG_MMU *//*
--Thanks,
LanceQuoted 47 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the
header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batchingarch/arm64/include/asm/pgtable.h | 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)
Lance Yang
reviewerQuoted 5 lines · Dev Jain
Quoted 2 lines · Lorenzo Stoakes
FWIW can confirm the same thing. Lance's fixes sort most of it out, but I also get this error:
Good catch!
Quoted 10 lines
Quoted 8 lines
mm/mprotect.c: In function ‘can_change_ptes_writable’:mm/mprotect.c:46:22: error: unused variable ‘page’ [-Werror=unused- variable] 46 | struct page *page;
| ^~~~So you also need to remove this unused variable at the stop of
can_change_ptes_writable().Strange that my build didn't catch this.
Well, to catch unused variable warnings with GCC, enable stricter checks by passing -Wunused-variable via KCFLAGS, and use -Werror=unused-variable to force the build to fail if any variable is declared but unused:
make -j$(nproc) KCFLAGS="-Wunused-variable -Werror=unused-variable"
Thanks,
Lance
Quoted 249 lines
Quoted 248 lines
Cheers, Lorenzo
Quoted 243 lines · Dev Jain
Quoted 72 lines · Lance Yang
Hey Dev,
Hmm... I also hit the same compilation errors:
In file included from
./include/linux/kasan.h:37,
from./include/linux/slab.h:260,
from./include/linux/crypto.h:19,
fromarch/x86/kernel/asm-offsets.c:9:./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 |tmp_pte=ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 |ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(structvm_area_struct*, long unsigned int,pte_t*)’
1360 | static inlinepte_tptep_modify_prot_start(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
925 |ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98:arch/x86/kernel/asm-offsets.s] Error 1
make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2
make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2Quoted 6 lines
Well,
modify_prot_start_ptes()callsptep_modify_prot_start(), but x86 does not define__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independentptep_modify_prot_start()must be defined beforemodify_prot_start_ptes().With the changes below, things work correctly now ;)
Ah thanks! My bad :(
Quoted 158 lines
include/linux/pgtable.h +0
-38@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, } #endif -/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - pte_t pte, tmp_pte; - - pte = ptep_modify_prot_start(vma, addr, ptep); - while (--nr) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_modify_prot_start(vma, addr, ptep); - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes (struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, pte_t old_pte, pte_t pte, unsigned int nr) -{ - for (;;) { - ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - old_pte = pte_next_pfn(old_pte); - pte = pte_next_pfn(pte); - } -} -#endif - /** On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It bringsinclude/linux/pgtable.h +0
-0@@ -1375,6 +1337,45 @@ static inline voidptep_modify_prot_commit(struct
vm_area_struct*vma,__ptep_modify_prot_commit(vma, addr, ptep, pte); }
#endif /*__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION*/include/linux/pgtable.h +25
-0+ +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, unsigned int nr) +{ + pte_t pte, tmp_pte; + + pte = ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct *vma,unsigned long addr,
+pte_t*ptep,pte_told_pte,pte_tpte, unsigned int nr)include/linux/pgtable.h +13
-0+{ + for (;;) { + ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + old_pte = pte_next_pfn(old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +#endif /*
CONFIG_MMU*//*
Signature · 2 lines
Thanks, LanceQuoted 48 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batching
arch/arm64/include/asm/pgtable.h| 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)
Dev Jain
reviewerQuoted 20 lines · Lance Yang
Quoted 5 lines · Dev Jain
Quoted 2 lines · Lorenzo Stoakes
FWIW can confirm the same thing. Lance's fixes sort most of it out, but I also get this error:
Good catch!
Quoted 9 lines
Quoted 8 lines
mm/mprotect.c: In function ‘can_change_ptes_writable’:mm/mprotect.c:46:22: error: unused variable ‘page’ [-Werror=unused- variable] 46 | struct page *page;
| ^~~~So you also need to remove this unused variable at the stop of
can_change_ptes_writable().
Quoted 7 lines
Quoted 1 line
Strange that my build didn't catch this.
Well, to catch unused variable warnings with GCC, enable stricter checks by passing -Wunused-variable via KCFLAGS, and use -Werror=unused- variable to force the build to fail if any variable is declared but unused:
make -j$(nproc) KCFLAGS="-Wunused-variable -Werror=unused-variable"
I thought this was the default, but thanks Lance!
Quoted 260 lines
Thanks,
LanceQuoted 254 lines
Quoted 253 lines
Cheers, Lorenzo
Quoted 248 lines · Dev Jain
Quoted 72 lines · Lance Yang
Hey Dev,
Hmm... I also hit the same compilation errors:
In file included from
./include/linux/kasan.h:37,
from./include/linux/slab.h:260,
from./include/linux/crypto.h:19,
fromarch/x86/kernel/asm-offsets.c:9:./include/linux/pgtable.h: In function ‘modify_prot_start_ptes’:./include/linux/pgtable.h:905:15: error: implicit declaration of function ‘ptep_modify_prot_start’ [-Werror=implicit-function-declaration]
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’./include/linux/pgtable.h:909:27: error: incompatible types when assigning to type ‘pte_t’ from type ‘int’
909 |tmp_pte=ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: In function ‘modify_prot_commit_ptes’:./include/linux/pgtable.h:925:17: error: implicit declaration of function ‘ptep_modify_prot_commit’ [-Werror=implicit-function-declaration]
925 |ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h: At top level:./include/linux/pgtable.h:1360:21: error: conflicting types for ‘ptep_modify_prot_start’; have ‘pte_t(structvm_area_struct*, long unsigned int,pte_t*)’
1360 | static inlinepte_tptep_modify_prot_start(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:905:15: note: previous implicit declaration of ‘ptep_modify_prot_start’ with type ‘int()’
905 | pte =ptep_modify_prot_start(vma, addr, ptep); | ^~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: warning: conflicting types for ‘ptep_modify_prot_commit’; have ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
1371 | static inline void ptep_modify_prot_commit(structvm_area_struct*vma,
| ^~~~~~~~~~~~~~~~~~~~~~~./include/linux/pgtable.h:1371:20: error: static declaration of ‘ptep_modify_prot_commit’ follows non-static declaration./include/linux/pgtable.h:925:17: note: previous implicit declaration of ‘ptep_modify_prot_commit’ with type ‘void(structvm_area_struct*, long unsigned int,pte_t*, pte_t, pte_t)’
925 |ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte);
| ^~~~~~~~~~~~~~~~~~~~~~~
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ libstring.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ libctype.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ str_error_r.o
CC/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ librbtree.o
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:98:arch/x86/kernel/asm-offsets.s] Error 1
make[1]: *** [/home/runner/work/mm-test-robot/mm-test-robot/linux/Makefile:1280: prepare0] Error 2
make[1]: *** Waiting for unfinished jobs....
LD/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/ objtool-in.o
LINK/home/runner/work/mm-test-robot/mm-test-robot/linux/tools/objtool/objtool
make: *** [Makefile:248: __sub-make] Error 2Quoted 6 lines
Well,
modify_prot_start_ptes()callsptep_modify_prot_start(), but x86 does not define__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION. To avoid implicit declaration errors, the architecture-independentptep_modify_prot_start()must be defined beforemodify_prot_start_ptes().With the changes below, things work correctly now ;)
Ah thanks! My bad :(
Quoted 163 lines
include/linux/pgtable.h +0
-0@@ -895,44 +895,6 @@ static inline void wrprotect_ptes(struct
mm_struct
*mm, unsigned long addr,
}
#endifinclude/linux/pgtable.h +0
-3-/* See the comment for ptep_modify_prot_start */ -#ifndef modify_prot_start_ptes -static inline pte_t modify_prot_start_ptes(struct vm_area_struct*vma,
- unsigned long addr,pte_t*ptep, unsigned int nr)include/linux/pgtable.h +0
-20-{ - pte_t pte, tmp_pte; - - pte = ptep_modify_prot_start(vma, addr, ptep); - while (--nr) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_modify_prot_start(vma, addr, ptep); - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - return pte; -} -#endif - -/* See the comment for ptep_modify_prot_commit */ -#ifndef modify_prot_commit_ptes -static inline void modify_prot_commit_ptes(struct vm_area_struct*vma,
unsigned long addr,
-pte_t*ptep,pte_told_pte,pte_tpte, unsigned int nr)include/linux/pgtable.h +0
-13-{ - for (;;) { - ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - old_pte = pte_next_pfn(old_pte); - pte = pte_next_pfn(pte); - } -} -#endif -/*
* On some architectures hardware does not set page access bit when accessing
* memory page, it is responsibility of software setting this bit. It bringsinclude/linux/pgtable.h +0
-0@@ -1375,6 +1337,45 @@ static inline voidptep_modify_prot_commit(struct
vm_area_struct*vma,__ptep_modify_prot_commit(vma, addr, ptep, pte); }
#endif /*__HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION*/include/linux/pgtable.h +4
-0+ +/* See the comment for ptep_modify_prot_start */ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes(struct vm_area_struct*vma,
+ unsigned long addr,pte_t*ptep, unsigned int nr)include/linux/pgtable.h +20
-0+{ + pte_t pte, tmp_pte; + + pte = ptep_modify_prot_start(vma, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_modify_prot_start(vma, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +/* See the comment for ptep_modify_prot_commit */ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes(struct vm_area_struct*vma,
unsigned long addr,
+pte_t*ptep,pte_told_pte,pte_tpte, unsigned int nr)include/linux/pgtable.h +13
-0+{ + for (;;) { + ptep_modify_prot_commit(vma, addr, ptep, old_pte, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + old_pte = pte_next_pfn(old_pte); + pte = pte_next_pfn(pte); + } +} +#endif +#endif /*
CONFIG_MMU*//*
Signature · 2 lines
Thanks, LanceQuoted 48 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batching
arch/arm64/include/asm/pgtable.h| 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)
Also,
On future respins can we please drop Namit from vmware from cc list?
I am being spammed with these messages whenever I reply here:
'A custom mail flow rule created by an admin at onevmw.onmicrosoft.com has blocked your message. Recipient is not authorized to accept external mail'
Or ask him to fix this :)
Cheers, Lorenzo
Quoted 49 lines · Dev Jain
This patchset optimizes the
mprotect()system call for large folios by PTE-batching.We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 7.9 seconds T2: 7.9 seconds T3: 4.2 secondsAfter the patchset:
T1: 2.1 seconds T2: 2.2 seconds T3: 4.2 secondsObserving T1/T2 and T3 before the patchset, we also remove the regression introduced by
ptep_get()on a contpte block. And, for large folios we get an almost 74% performance improvement.
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf:util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)Dev Jain (7):
mm: Refactor code in mprotect
mm: Optimizemprotect()by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
arm64: Add batched version ofptep_modify_prot_start
arm64: Add batched version ofptep_modify_prot_commit
mm: Batch aroundcan_change_pte_writable()
mm: Optimizemprotect()through PTE-batchingarch/arm64/include/asm/pgtable.h | 10 ++ arch/arm64/mm/mmu.c | 21 +++- include/linux/mm.h | 4 +- include/linux/pgtable.h | 42 ++++++++ mm/gup.c | 2 +- mm/huge_memory.c | 4 +- mm/memory.c | 6 +- mm/mprotect.c | 165 ++++++++++++++++++++----------- mm/pgtable-generic.c | 16 ++- 9 files changed, 198 insertions(+), 72 deletions(-)--
2.30.2
Quoted 1 line · Lorenzo Stoakes
Yes, Nadav is no longer working for VMWare.
.mailmap should already include the correct mapping to the gmail address AFAIKS?
Signature · 3 lines
Cheers, David / dhildenb