Optimize mprotect() for large folios
Subsystems (4)
v5 cover Optimize mprotect() for large folios
Use folio_pte_batch() to optimize change_pte_range(). On arm64, if the ptes are painted with the contig bit, then ptep_get() will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number of ptep_get() calls. Next, ptep_modify_prot_start() will eventually call contpte_try_unfold() on every contig block, thus flushing the TLB for the complete large folio range. Instead, use get_and_clear_full_ptes() so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.
For split folios, there will be no pte batching; the batch size returned by folio_pte_batch() will be 1. For pagetable split folios, the ptes will still point to the same large folio; for arm64, this results in the optimization described above, and for other arches, a minor improvement is expected due to a reduction in the number of function calls.
mm-selftests pass on arm64. I have some failing tests on my x86 VM already; no new tests fail as a result of this patchset.
We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pages
Average execution time on arm64, Apple M3:
Before the patchset:
T1: 2.1 seconds T2: 2 seconds T3: 1 second
After the patchset:
T1: 0.65 seconds T2: 0.7 seconds T3: 1.1 seconds
Observing T1/T2 and T3 before the patchset, we also remove the regression introduced by ptep_get() on a contpte block. And, for large folios we get an almost 74% performance improvement, albeit the trade-off being a slight degradation in the small folio case.
For x86:
Before the patchset:
T1: 3.75 seconds T2: 3.7 seconds T3: 3.85 seconds
After the patchset:
T1: 3.7 seconds T2: 3.7 seconds T3: 3.9 seconds
So there is a minor improvement due to reduction in number of function calls, and a slight degradation in the small folio case due to the overhead of vm_normal_folio() + folio_test_large().
Here is the test program:
#define _GNU_SOURCE
#include <sys/mman.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#define SIZE (1024*1024*1024)
unsigned long pmdsize = (1UL << 21);
unsigned long pagesize = (1UL << 12);
static void pte_map_thps(char *mem, size_t size)
{
size_t offs;
int ret = 0;
/* PTE-map each THP by temporarily splitting the VMAs. */
for (offs = 0; offs < size; offs += pmdsize) {
ret |= madvise(mem + offs, pagesize, MADV_DONTFORK);
ret |= madvise(mem + offs, pagesize, MADV_DOFORK);
}
if (ret) {
fprintf(stderr, "ERROR: mprotect() failed\n");
exit(1);
}
}int main(int argc, char *argv[])
{
char *p;
int ret = 0;
p = mmap((1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (p != (1UL << 30)) {
perror("mmap");
return 1;
}
memset(p, 0, SIZE);
if (madvise(p, SIZE, MADV_NOHUGEPAGE))
perror("madvise");
explicit_bzero(p, SIZE);
pte_map_thps(p, SIZE);
for (int loops = 0; loops < 40; loops++) {
if (mprotect(p, SIZE, PROT_READ))
perror("mprotect"), exit(1);
if (mprotect(p, SIZE, PROT_READ|PROT_WRITE))
perror("mprotect"), exit(1);
explicit_bzero(p, SIZE);
}
}---v4->v5:
- Add patch 4
- Add patch 1 (Lorenzo)
- For patch 2, instead of using nr_ptes returned from prot_numa_skip() as a dummy for whether to skip or not, make that function return boolean, and then use folio_pte_batch() to determine how much to skip
- Split can_change_pte_writable() (Lorenzo)
- Implement patch 6 in a better way
v3->v4:
- Refactor skipping logic into a new function, edit patch 1 subject to highlight it is only for MM_CP_PROT_NUMA case (David H)
- Refactor the optimization logic, add more documentation to the generic batched functions, do not add clear_flush_ptes, squash patch 4 and 5 (Ryan)
v2->v3:
- Add comments for the new APIs (Ryan, Lorenzo)
- Instead of refactoring, use a "skip_batch" label
- Move arm64 patches at the end (Ryan)
- In can_change_pte_writable(), check AnonExclusive page-by-page (David H)
- Resolve implicit declaration; tested build on x86 (Lance Yang)
v1->v2:
- Rebase onto mm-unstable (6ebffe676fcf: util_macros.h: make the header more resilient)
- Abridge the anon-exclusive condition (Lance Yang)
Dev Jain (7):
mm: Refactor MM_CP_PROT_NUMA skipping case into new function
mm: Optimize mprotect() for MM_CP_PROT_NUMA by batch-skipping PTEs
mm: Add batched versions of ptep_modify_prot_start/commit
mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
mm: Split can_change_pte_writable() into private and shared parts
mm: Optimize mprotect() by PTE batching
arm64: Add batched versions of ptep_modify_prot_start/commit
arch/arm64/include/asm/pgtable.h | 10 ++ arch/arm64/mm/mmu.c | 28 ++- include/linux/pgtable.h | 84 ++++++++- mm/internal.h | 11 +- mm/mprotect.c | 295 ++++++++++++++++++++++++------- 5 files changed, 352 insertions(+), 76 deletions(-)
Signature · 1 line
2.30.2
PATCH v5 1/7 mm: Refactor MM_CP_PROT_NUMA skipping case into new function
Reduce indentation by refactoring the prot_numa case into a new function. No functional change intended.
--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)
mm/mprotect.c +55 -46
@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, +pte_t oldpte,pte_t *pte,int target_node) +{ +struct folio *folio; +bool toptier; +int nid; + +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio) +return true; + +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) +return true; + +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); +return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)
@@ -117,53 +170,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) -continue ; - -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) +if (prot_numa_skip (vma, addr, oldpte, pte, + target_node))continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); } oldpte =ptep_modify_prot_start (vma, addr, pte);
Signature · 1 line
2.30.2
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
Quoted 4 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>
LGTM, so:
Quoted 127 lines
--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)mm/mprotect.c +55
-47@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, +pte_t oldpte,pte_t *pte,int target_node) +{ +struct folio *folio; +bool toptier; +int nid; + +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio) +return true; + +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + + toptier =node_is_toptier (nid); + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) +return true; + +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); +return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -117,53 +170,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) -continue ; - -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) +if (prot_numa_skip (vma, addr, oldpte, pte, + target_node))continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); } oldpte =ptep_modify_prot_start (vma, addr, pte); --2.30.2
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
In reply to Dev JainQuoted 5 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>
Quoted 15 lines
--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)mm/mprotect.c +3
-0@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, +pte_t oldpte,pte_t *pte,int target_node) +{
[...]
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 135
Quoted 9 lines
mm/mprotect.c +9
-0+/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) +return true; + +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));
Nit: I wonder if this should be moved elsewhere, since this isn't actually about 'skipping', even though the function is named prot_numa_skip().
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 138
Quoted 3 lines
mm/mprotect.c +3
-0+return false; +} +
Thanks
Barry
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
Quoted 33 lines · Barry Song
Quoted 4 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>Reviewed-by Barry Song <[email protected]>Quoted 15 lines
--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)mm/mprotect.c +3
-0@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, +pte_t oldpte,pte_t *pte,int target_node) +{[...]
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 135
Quoted 9 lines
mm/mprotect.c +9
-0+/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) +return true; + +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));
Quoted 3 lines
Nit: I wonder if this should be moved elsewhere, since this isn't actually about 'skipping', even though the function is named
prot_numa_skip().
Agreed, thanks. We can use prot_numa_skip() only to return true or false, and if returned false, we can call folio_xchg_access_time. I will wait for 2-3 days for any more comments and respin.
Quoted 6 lines
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 138
Quoted 4 lines
mm/mprotect.c +3
-0+return false; +} +Thanks
Barry
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
Quoted 40 lines · Dev Jain
Quoted 38 lines · Barry Song
Quoted 4 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>Reviewed-by Barry Song <[email protected]>Quoted 18 lines
---
mm/mprotect.c| 101
+++++++++++++++++++++++++++-----------------------1 file changed, 55 insertions(+), 46 deletions(-)mm/mprotect.c +3
-0@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, return pte_dirty(pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, + pte_t oldpte, pte_t *pte, int target_node) +{[...]
Quoted 11 lines
+ /*
+ * Skip scanning top tier node if normal numa
+ * balancing is disabled
+ */
+ if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL) && toptier)
+ return true;
+
+ if (folio_use_access_time(folio))
+folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
Quoted 7 lines
Quoted 3 lines
Nit: I wonder if this should be moved elsewhere, since this isn't actually about 'skipping', even though the function is named
prot_numa_skip().Agreed, thanks. We can use
prot_numa_skip()only to return true or false, and if returned false, we can callfolio_xchg_access_time. I will wait for 2-3 days for any more comments and respin.
Since Andrew has already pulled this and we are quite late into the
release cycle, I'll do this cleanup in the next cycle.
Quoted 8 lines
Quoted 7 lines
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 138
Quoted 5 lines
+ return false;
mm/mprotect.c +2
-0+} +Thanks
Barry
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
In reply to Dev JainQuoted 56 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)mm/mprotect.c +40
-0@@ -83,6 +83,59 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, +pte_t oldpte,pte_t *pte,int target_node) +{ +struct folio *folio; +bool toptier; +int nid; + +/* Avoid TLB flush if possible */ +if (pte_protnone (oldpte)) +return true; + + folio =vm_normal_folio (vma, addr, oldpte); +if (!folio) +return true; + +if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) +return true; + +/* Also skip shared copy-on-write pages */ +if (is_cow_mapping (vma->vm_flags) && + (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) +return true; + +/* +* While migration can move some dirty pages, +* it cannot move them all from MIGRATE_ASYNC +* context. +*/ +if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) +return true; + +/* +* Don't mess with PTEs if page is already on the node +* a single-threaded process is running on. +*/ + nid =folio_nid (folio); +if (target_node == nid) +return true; + + toptier =node_is_toptier (nid);
mm/mprotect.c bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, · line 135
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) +return true; + +if (folio_use_access_time (folio)) +folio_xchg_access_time (folio,jiffies_to_msecs (jiffies));
Perhaps this bit should be kept in the original location? It's got nothing to do with determining if the pte should be skipped.
Thanks,
Ryan
Quoted 62 lines
mm/mprotect.c +5
-46+return false; +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -117,53 +170,9 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -struct folio *folio; -int nid; -bool toptier; - -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) -continue ; - - folio =vm_normal_folio (vma, addr, oldpte); -if (!folio ||folio_is_zone_device (folio) || -folio_test_ksm (folio)) -continue ; - -/* Also skip shared copy-on-write pages */ -if (is_cow_mapping (vma->vm_flags) && - (folio_maybe_dma_pinned (folio) || -folio_maybe_mapped_shared (folio))) -continue ; - -/* -* While migration can move some dirty pages, -* it cannot move them all from MIGRATE_ASYNC -* context. -*/ -if (folio_is_file_lru (folio) && -folio_test_dirty (folio)) -continue ; - -/* -* Don't mess with PTEs if page is already on the node -* a single-threaded process is running on. -*/ - nid =folio_nid (folio); -if (target_node == nid) -continue ; - toptier =node_is_toptier (nid); - -/* -* Skip scanning top tier node if normal numa -* balancing is disabled -*/ -if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && - toptier) +if (prot_numa_skip (vma, addr, oldpte, pte, + target_node))continue ; -if (folio_use_access_time (folio)) -folio_xchg_access_time (folio, -jiffies_to_msecs (jiffies)); } oldpte =ptep_modify_prot_start (vma, addr, pte);
Re: [PATCH v5 1/7] mm: Refactor MM_CP_PROT_NUMA skipping case into new function
In reply to Dev JainQuoted 8 lines · Dev Jain
Reduce indentation by refactoring the
prot_numacase into a new function. No functional change intended.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 101 +++++++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 46 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
PATCH v5 2/7 mm: Optimize mprotect() for MM_CP_PROT_NUMA by batch-skipping PTEs
In reply to Dev JainFor the MM_CP_PROT_NUMA skipping case, observe that, if we skip an iteration due to the underlying folio satisfying any of the skip conditions, then for all subsequent ptes which map the same folio, the iteration will be skipped for them too. Therefore, we can optimize by using folio_pte_batch() to batch skip the iterations.
Use prot_numa_skip() introduced in the previous patch to determine whether we need to skip the iteration. Change its signature to have a double pointer to a folio, which will be used by mprotect_folio_pte_batch() to determine the number of iterations we can safely skip.
--- mm/mprotect.c | 55 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 42 insertions(+), 13 deletions(-)
mm/mprotect.c +42 -13
@@ -83,28 +83,43 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, +pte_t pte,int max_nr_ptes) +{ +/* No underlying folio, so cannot batch */ +if (!folio) +return 1 ; + +if (!folio_test_large (folio)) +return 1 ; + +return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +} +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, -pte_t oldpte,pte_t *pte,int target_node) +pte_t oldpte,pte_t *pte,int target_node, +struct folio **foliop) { -struct folio *folio; +struct folio *folio =NULL ; +bool ret = true;bool toptier;int nid;/* Avoid TLB flush if possible */ if (pte_protnone (oldpte)) -return true; + goto skip; folio =vm_normal_folio (vma, addr, oldpte);if (!folio) -return true; + goto skip;if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) -return true; + goto skip;/* Also skip shared copy-on-write pages */ if (is_cow_mapping (vma->vm_flags) && (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) -return true; + goto skip;/* * While migration can move some dirty pages,
@@ -112,7 +127,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* context. */ if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) -return true; + goto skip;/* * Don't mess with PTEs if page is already on the node
@@ -120,7 +135,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,*/ nid =folio_nid (folio);if (target_node == nid) -return true; + goto skip; toptier =node_is_toptier (nid);
@@ -129,11 +144,15 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* balancing is disabled */ if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) -return true; + goto skip; + ret = false;if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -return false; + +skip: + *foliop = folio; +return ret; } static long change_pte_range(struct mmu_gather *tlb,
@@ -147,6 +166,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr_ptes;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);
@@ -161,8 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr_ptes = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr_ptes = (end - addr) >> PAGE_SHIFT; + struct folio *folio; pte_t ptent; /*
@@ -170,9 +193,15 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -if (prot_numa_skip (vma, addr, oldpte, pte, - target_node)) +int ret =prot_numa_skip (vma, addr, oldpte, pte, + target_node, &folio); +if (ret) { + +/* determine batch to skip */ + nr_ptes =mprotect_folio_pte_batch (folio, + pte, oldpte, max_nr_ptes);continue ; + } } oldpte =ptep_modify_prot_start (vma, addr, pte);
@@ -289,7 +318,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr_ptes, addr += nr_ptes *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl);
Signature · 1 line
2.30.2
Re: [PATCH v5 2/7] mm: Optimize mprotect() for MM_CP_PROT_NUMA by batch-skipping PTEs
Quoted 10 lines · Dev Jain
For the
MM_CP_PROT_NUMAskipping case, observe that, if we skip an iteration due to the underlying folio satisfying any of the skip conditions, then for all subsequent ptes which map the same folio, the iteration will be skipped for them too. Therefore, we can optimize by usingfolio_pte_batch()to batch skip the iterations.Use
prot_numa_skip()introduced in the previous patch to determine whether we need to skip the iteration. Change its signature to have a double pointer to a folio, which will be used bymprotect_folio_pte_batch()to determine the number of iterations we can safely skip.
On reflection I don't think this is too bad, it seems neat in this version at least :)
Quoted 2 lines
Signed-off-by Dev Jain <[email protected]>
Overall looks good to me, so:
Quoted 144 lines
--- mm/mprotect.c | 55 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 42 insertions(+), 13 deletions(-)mm/mprotect.c +42
-14@@ -83,28 +83,43 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, +pte_t pte,int max_nr_ptes) +{ +/* No underlying folio, so cannot batch */ +if (!folio) +return 1 ; + +if (!folio_test_large (folio)) +return 1 ; + +return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +} +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, -pte_t oldpte,pte_t *pte,int target_node) +pte_t oldpte,pte_t *pte,int target_node, +struct folio **foliop) { -struct folio *folio; +struct folio *folio =NULL ; +bool ret = true;bool toptier;int nid;/* Avoid TLB flush if possible */ if (pte_protnone (oldpte)) -return true; + goto skip; folio =vm_normal_folio (vma, addr, oldpte);if (!folio) -return true; + goto skip;if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) -return true; + goto skip;/* Also skip shared copy-on-write pages */ if (is_cow_mapping (vma->vm_flags) && (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) -return true; + goto skip;/* * While migration can move some dirty pages, @@ -112,7 +127,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* context. */ if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) -return true; + goto skip;/* * Don't mess with PTEs if page is already on the node @@ -120,7 +135,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,*/ nid =folio_nid (folio);if (target_node == nid) -return true; + goto skip; toptier =node_is_toptier (nid);@@ -129,11 +144,15 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* balancing is disabled */ if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) -return true; + goto skip; + ret = false;if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -return false; + +skip: + *foliop = folio; +return ret; } static long change_pte_range(struct mmu_gather *tlb,@@ -147,6 +166,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr_ptes;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);@@ -161,8 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr_ptes = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr_ptes = (end - addr) >> PAGE_SHIFT; + struct folio *folio; pte_t ptent; /*@@ -170,9 +193,15 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -if (prot_numa_skip (vma, addr, oldpte, pte, - target_node)) +int ret =prot_numa_skip (vma, addr, oldpte, pte, + target_node, &folio); +if (ret) { + +/* determine batch to skip */ + nr_ptes =mprotect_folio_pte_batch (folio, + pte, oldpte, max_nr_ptes);continue ; + } } oldpte =ptep_modify_prot_start (vma, addr, pte);@@ -289,7 +318,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr_ptes, addr += nr_ptes *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl); --2.30.2
Re: [PATCH v5 2/7] mm: Optimize mprotect() for MM_CP_PROT_NUMA by batch-skipping PTEs
In reply to Dev JainQuoted 6 lines · Dev Jain
For the
MM_CP_PROT_NUMAskipping case, observe that, if we skip an iteration due to the underlying folio satisfying any of the skip conditions, then for all subsequent ptes which map the same folio, the iteration will be skipped for them too. Therefore, we can optimize by usingfolio_pte_batch()to batch skip the iterations.
Quoted 6 lines
Use
prot_numa_skip()introduced in the previous patch to determine whether we need to skip the iteration. Change its signature to have a double pointer to a folio, which will be used bymprotect_folio_pte_batch()to determine the number of iterations we can safely skip.Signed-off-by Dev Jain <[email protected]>
LGTM:
Quoted 141 lines
--- mm/mprotect.c | 55 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 42 insertions(+), 13 deletions(-)mm/mprotect.c +42
-13@@ -83,28 +83,43 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, +pte_t pte,int max_nr_ptes) +{ +/* No underlying folio, so cannot batch */ +if (!folio) +return 1 ; + +if (!folio_test_large (folio)) +return 1 ; + +return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +} +static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr, -pte_t oldpte,pte_t *pte,int target_node) +pte_t oldpte,pte_t *pte,int target_node, +struct folio **foliop) { -struct folio *folio; +struct folio *folio =NULL ; +bool ret = true;bool toptier;int nid;/* Avoid TLB flush if possible */ if (pte_protnone (oldpte)) -return true; + goto skip; folio =vm_normal_folio (vma, addr, oldpte);if (!folio) -return true; + goto skip;if (folio_is_zone_device (folio) ||folio_test_ksm (folio)) -return true; + goto skip;/* Also skip shared copy-on-write pages */ if (is_cow_mapping (vma->vm_flags) && (folio_maybe_dma_pinned (folio) ||folio_maybe_mapped_shared (folio))) -return true; + goto skip;/* * While migration can move some dirty pages, @@ -112,7 +127,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* context. */ if (folio_is_file_lru (folio) &&folio_test_dirty (folio)) -return true; + goto skip;/* * Don't mess with PTEs if page is already on the node @@ -120,7 +135,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,*/ nid =folio_nid (folio);if (target_node == nid) -return true; + goto skip; toptier =node_is_toptier (nid);@@ -129,11 +144,15 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,* balancing is disabled */ if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && toptier) -return true; + goto skip; + ret = false;if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -return false; + +skip: + *foliop = folio; +return ret; } static long change_pte_range(struct mmu_gather *tlb,@@ -147,6 +166,7 @@ static long change_pte_range(struct mmu_gather *tlb,bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +int nr_ptes;tlb_change_page_size (tlb,PAGE_SIZE ); pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);@@ -161,8 +181,11 @@ static long change_pte_range(struct mmu_gather *tlb,flush_tlb_batched_pending (vma->vm_mm);arch_enter_lazy_mmu_mode (); do { + nr_ptes = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + int max_nr_ptes = (end - addr) >> PAGE_SHIFT; + struct folio *folio; pte_t ptent; /*@@ -170,9 +193,15 @@ static long change_pte_range(struct mmu_gather *tlb,* pages. See similar comment in change_huge_pmd. */ if (prot_numa) { -if (prot_numa_skip (vma, addr, oldpte, pte, - target_node)) +int ret =prot_numa_skip (vma, addr, oldpte, pte, + target_node, &folio); +if (ret) { + +/* determine batch to skip */ + nr_ptes =mprotect_folio_pte_batch (folio, + pte, oldpte, max_nr_ptes);continue ; + } } oldpte =ptep_modify_prot_start (vma, addr, pte);@@ -289,7 +318,7 @@ static long change_pte_range(struct mmu_gather *tlb, pages++; } } - }while (pte++, addr +=PAGE_SIZE , addr != end); + }while (pte += nr_ptes, addr += nr_ptes *PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (pte -1 , ptl);
Re: [PATCH v5 2/7] mm: Optimize mprotect() for MM_CP_PROT_NUMA by batch-skipping PTEs
In reply to Dev JainQuoted 11 lines · Dev Jain
For the
MM_CP_PROT_NUMAskipping case, observe that, if we skip an iteration due to the underlying folio satisfying any of the skip conditions, then for all subsequent ptes which map the same folio, the iteration will be skipped for them too. Therefore, we can optimize by usingfolio_pte_batch()to batch skip the iterations.Use
prot_numa_skip()introduced in the previous patch to determine whether we need to skip the iteration. Change its signature to have a double pointer to a folio, which will be used bymprotect_folio_pte_batch()to determine the number of iterations we can safely skip.
Quoted 5 lines
Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 55 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 42 insertions(+), 13 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
PATCH v5 3/7 mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainBatch ptep_modify_prot_start/commit in preparation for optimizing mprotect, implementing them as a simple loop over the corresponding single pte helpers. Architecture may override these helpers.
--- include/linux/pgtable.h | 84 ++++++++++++++++++++++++++++++++++++++++- mm/mprotect.c | 4 +- 2 files changed, 85 insertions(+), 3 deletions(-)
include/linux/pgtable.h +83 -1
@@ -1331,7 +1331,9 @@ static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,/* * Commit an update to a pte, leaving any hardware-controlled bits in -* the PTE unmodified. +* the PTE unmodified. The pte returned from ptep_modify_prot_start() may +* additionally have young and/or dirty bits set where previously they were not, +* so the updated pte may have these additional changes. */ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr,
@@ -1340,6 +1342,86 @@ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,__ptep_modify_prot_commit (vma, addr, ptep, pte); }#endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/** +* modify_prot_start_ptes - Start a pte protection read-modify-write transaction +* over a batch of ptes, which protects against asynchronous hardware +* modifications to the ptes. The intention is not to prevent the hardware from +* making pte updates, but to prevent any updates it may make from being lost. +* Please see the comment above ptep_modify_prot_start() for full description. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_start(), collecting the a/d bits from each pte +* in the batch. +* +* Note that PTE bits in the PTE batch besides the PFN can differ. +* +* Context: The caller holds the page table lock. The PTEs map consecutive +* pages that belong to the same folio. All other PTE bits must be identical for +* all PTEs in the batch except for young and dirty bits. The PTEs are all in +* the same PMD. +*/ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/** +* modify_prot_commit_ptes - Commit an update to a batch of ptes, leaving any +* hardware-controlled bits in the PTE unmodified. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @old_pte: Old page table entry (for the first entry) which is now cleared. +* @pte: New page table entry to be set. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_commit(). +* +* Context: The caller holds the page table lock. The PTEs are all in the same +* PMD. On exit, the set ptes in the batch map the same folio. The ptes set by +* ptep_modify_prot_start() may additionally have young and/or dirty bits set +* where previously they were not, so the updated ptes may have these +* additional changes. +*/ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +int i; + +for (i =0 ; i < nr; ++i, ++ptep, addr +=PAGE_SIZE ) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); + +/* Advance PFN only, set same prot */ + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +} +#endif +#endif /* CONFIG_MMU */ /*
mm/mprotect.c +2 -2
@@ -204,7 +204,7 @@ static long change_pte_range(struct mmu_gather *tlb, } } - oldpte =ptep_modify_prot_start (vma, addr, pte); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);if (uffd_wp)
@@ -230,7 +230,7 @@ static long change_pte_range(struct mmu_gather *tlb, can_change_pte_writable(vma, addr, ptent)) ptent =pte_mkwrite (ptent, vma); -ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); +modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes);if (pte_needs_flush (oldpte, ptent))tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++;
Signature · 1 line
2.30.2
Re: [PATCH v5 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
Quoted 5 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect, implementing them as a simple loop over the corresponding single pte helpers. Architecture may override these helpers.
Signed-off-by Dev Jain <[email protected]>
LGTM, so:
Quoted 132 lines
--- include/linux/pgtable.h | 84 ++++++++++++++++++++++++++++++++++++++++- mm/mprotect.c | 4 +- 2 files changed, 85 insertions(+), 3 deletions(-)include/linux/pgtable.h +83
-1@@ -1331,7 +1331,9 @@ static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,/* * Commit an update to a pte, leaving any hardware-controlled bits in -* the PTE unmodified. +* the PTE unmodified. The pte returned from ptep_modify_prot_start() may +* additionally have young and/or dirty bits set where previously they were not, +* so the updated pte may have these additional changes. */ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr,@@ -1340,6 +1342,86 @@ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,__ptep_modify_prot_commit (vma, addr, ptep, pte); }#endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/** +* modify_prot_start_ptes - Start a pte protection read-modify-write transaction +* over a batch of ptes, which protects against asynchronous hardware +* modifications to the ptes. The intention is not to prevent the hardware from +* making pte updates, but to prevent any updates it may make from being lost. +* Please see the comment above ptep_modify_prot_start() for full description. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_start(), collecting the a/d bits from each pte +* in the batch. +* +* Note that PTE bits in the PTE batch besides the PFN can differ. +* +* Context: The caller holds the page table lock. The PTEs map consecutive +* pages that belong to the same folio. All other PTE bits must be identical for +* all PTEs in the batch except for young and dirty bits. The PTEs are all in +* the same PMD. +*/ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/** +* modify_prot_commit_ptes - Commit an update to a batch of ptes, leaving any +* hardware-controlled bits in the PTE unmodified. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @old_pte: Old page table entry (for the first entry) which is now cleared. +* @pte: New page table entry to be set. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_commit(). +* +* Context: The caller holds the page table lock. The PTEs are all in the same +* PMD. On exit, the set ptes in the batch map the same folio. The ptes set by +* ptep_modify_prot_start() may additionally have young and/or dirty bits set +* where previously they were not, so the updated ptes may have these +* additional changes. +*/ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +int i; + +for (i =0 ; i < nr; ++i, ++ptep, addr +=PAGE_SIZE ) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); + +/* Advance PFN only, set same prot */ + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +} +#endif +#endif /* CONFIG_MMU */ /* mm/mprotect.c +2
-3@@ -204,7 +204,7 @@ static long change_pte_range(struct mmu_gather *tlb, } } - oldpte =ptep_modify_prot_start (vma, addr, pte); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);if (uffd_wp)@@ -230,7 +230,7 @@ static long change_pte_range(struct mmu_gather *tlb, can_change_pte_writable(vma, addr, ptent)) ptent =pte_mkwrite (ptent, vma); -ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); +modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes);if (pte_needs_flush (oldpte, ptent))tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++; --2.30.2
Re: [PATCH v5 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 6 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect, implementing them as a simple loop over the corresponding single pte helpers. Architecture may override these helpers.
Signed-off-by Dev Jain <[email protected]>
Quoted 5 lines
--- include/linux/pgtable.h | 84 ++++++++++++++++++++++++++++++++++++++++- mm/mprotect.c | 4 +- 2 files changed, 85 insertions(+), 3 deletions(-)
[...]
Quoted 5 lines
include/linux/pgtable.h +5
-0+#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte;
include/linux/pgtable.h static inline void ptep_modify_prot_commit(struct vm_area_struct *vma, · line 1383
Quoted 10 lines
include/linux/pgtable.h +10
-0+ + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte);
It might be interesting to explore whether a similar optimization could apply here as well:
https://lore.kernel.org/linux-mm/[email protected]/
I suspect it would, but it's probably not worth including in this patch.
include/linux/pgtable.h static inline void ptep_modify_prot_commit(struct vm_area_struct *vma, · line 1388
Quoted 5 lines
include/linux/pgtable.h +5
-0+ } +return pte; +} +#endif +
Thanks
Barry
Re: [PATCH v5 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 5 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect, implementing them as a simple loop over the corresponding single pte helpers. Architecture may override these helpers.
Signed-off-by Dev Jain <[email protected]>
Quoted 10 lines
--- include/linux/pgtable.h | 84 ++++++++++++++++++++++++++++++++++++++++- mm/mprotect.c | 4 +- 2 files changed, 85 insertions(+), 3 deletions(-)include/linux/pgtable.h +0
-0
include/linux/pgtable.h static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma, · line 1336
Quoted 7 lines
include/linux/pgtable.h +3
-1@@ -1331,7 +1331,9 @@ static inline pte_t ptep_modify_prot_start(struct vm_area_struct *vma,/* * Commit an update to a pte, leaving any hardware-controlled bits in -* the PTE unmodified. +* the PTE unmodified. The pte returned from ptep_modify_prot_start() may +* additionally have young and/or dirty bits set where previously they were not, +* so the updated pte may have these additional changes.
nit: I still find this difficult to parse (Although I expect you might tell me that this is the text I suggested last time around :) ). I think you mean that "it is permissable for young and/or dirty bits to be set in old_pte, despite beling clear when originally returned by ptep_modify_prot_start()".
Anyway, no big deal. I think we all know what it's getting at.
Thanks,
Ryan
Quoted 112 lines
include/linux/pgtable.h +80
-0*/ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr,@@ -1340,6 +1342,86 @@ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,__ptep_modify_prot_commit (vma, addr, ptep, pte); }#endif /* __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION */ + +/** +* modify_prot_start_ptes - Start a pte protection read-modify-write transaction +* over a batch of ptes, which protects against asynchronous hardware +* modifications to the ptes. The intention is not to prevent the hardware from +* making pte updates, but to prevent any updates it may make from being lost. +* Please see the comment above ptep_modify_prot_start() for full description. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_start(), collecting the a/d bits from each pte +* in the batch. +* +* Note that PTE bits in the PTE batch besides the PFN can differ. +* +* Context: The caller holds the page table lock. The PTEs map consecutive +* pages that belong to the same folio. All other PTE bits must be identical for +* all PTEs in the batch except for young and dirty bits. The PTEs are all in +* the same PMD. +*/ +#ifndef modify_prot_start_ptes +static inline pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,unsigned int nr) +{ +pte_t pte, tmp_pte; + + pte =ptep_modify_prot_start (vma, addr, ptep); +while (--nr) { + ptep++; + addr +=PAGE_SIZE ; + tmp_pte =ptep_modify_prot_start (vma, addr, ptep); +if (pte_dirty (tmp_pte)) + pte =pte_mkdirty (pte); +if (pte_young (tmp_pte)) + pte =pte_mkyoung (pte); + } +return pte; +} +#endif + +/** +* modify_prot_commit_ptes - Commit an update to a batch of ptes, leaving any +* hardware-controlled bits in the PTE unmodified. +* +* @vma: The virtual memory area the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @old_pte: Old page table entry (for the first entry) which is now cleared. +* @pte: New page table entry to be set. +* @nr: Number of entries. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_modify_prot_commit(). +* +* Context: The caller holds the page table lock. The PTEs are all in the same +* PMD. On exit, the set ptes in the batch map the same folio. The ptes set by +* ptep_modify_prot_start() may additionally have young and/or dirty bits set +* where previously they were not, so the updated ptes may have these +* additional changes. +*/ +#ifndef modify_prot_commit_ptes +static inline void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte,unsigned int nr) +{ +int i; + +for (i =0 ; i < nr; ++i, ++ptep, addr +=PAGE_SIZE ) { +ptep_modify_prot_commit (vma, addr, ptep, old_pte, pte); + +/* Advance PFN only, set same prot */ + old_pte =pte_next_pfn (old_pte); + pte =pte_next_pfn (pte); + } +} +#endif +#endif /* CONFIG_MMU */ /* mm/mprotect.c +2
-2@@ -204,7 +204,7 @@ static long change_pte_range(struct mmu_gather *tlb, } } - oldpte =ptep_modify_prot_start (vma, addr, pte); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);if (uffd_wp)@@ -230,7 +230,7 @@ static long change_pte_range(struct mmu_gather *tlb, can_change_pte_writable(vma, addr, ptent)) ptent =pte_mkwrite (ptent, vma); -ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); +modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes);if (pte_needs_flush (oldpte, ptent))tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++;
Re: [PATCH v5 3/7] mm: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 10 lines · Dev Jain
Batch ptep_modify_prot_start/commit in preparation for optimizing mprotect, implementing them as a simple loop over the corresponding single pte helpers. Architecture may override these helpers.
Signed-off-by Dev Jain <[email protected]>--- include/linux/pgtable.h | 84 ++++++++++++++++++++++++++++++++++++++++- mm/mprotect.c | 4 +- 2 files changed, 85 insertions(+), 3 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
PATCH v5 4/7 mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
In reply to Dev JainPatch 6 optimizes mprotect() by batch clearing the ptes, masking in the new protections, and batch setting the ptes. Suppose that the first pte of the batch is writable - with the current implementation of folio_pte_batch(), it is not guaranteed that the other ptes in the batch are already writable too, so we may incorrectly end up setting the writable bit on all ptes via modify_prot_commit_ptes().
Therefore, introduce FPB_RESPECT_WRITE so that all ptes in the batch are writable or not.
--- mm/internal.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
mm/internal.h +8 -3
@@ -208,17 +208,20 @@ typedef int __bitwise fpb_t;/* Compare PTEs respecting the soft-dirty bit. */ #define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1)) +/* Compare PTEs respecting the writable bit. */ +#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2)) +/* * Merge PTE write bits: if any PTE in the batch is writable, modify the * PTE at @ptentp to be writable. */ -#define FPB_MERGE_WRITE ((__force fpb_t)BIT(2)) +#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))/* * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty, * modify the PTE at @ptentp to be young or dirty, respectively. */ -#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(3)) +#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))static inline pte_t __pte_batch_clear_ignored (pte_t pte,fpb_t flags) {
@@ -226,7 +229,9 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) pte =pte_mkclean (pte);if (likely (!(flags &FPB_RESPECT_SOFT_DIRTY ))) pte =pte_clear_soft_dirty (pte); -return pte_wrprotect (pte_mkold (pte)); +if (likely (!(flags &FPB_RESPECT_WRITE ))) + pte =pte_wrprotect (pte); +return pte_mkold (pte); }/**
Signature · 1 line
2.30.2
Re: [PATCH v5 4/7] mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
Quoted 11 lines · Dev Jain
Patch 6 optimizes
mprotect()by batch clearing the ptes, masking in the new protections, and batch setting the ptes. Suppose that the first pte of the batch is writable - with the current implementation offolio_pte_batch(), it is not guaranteed that the other ptes in the batch are already writable too, so we may incorrectly end up setting the writable bit on all ptes viamodify_prot_commit_ptes().Therefore, introduce
FPB_RESPECT_WRITEso that all ptes in the batch are writable or not.Signed-off-by Dev Jain <[email protected]>
LGTM, so:
Quoted 45 lines
--- mm/internal.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)mm/internal.h +8
-4@@ -208,17 +208,20 @@ typedef int __bitwise fpb_t;/* Compare PTEs respecting the soft-dirty bit. */ #define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1)) +/* Compare PTEs respecting the writable bit. */ +#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2)) +/* * Merge PTE write bits: if any PTE in the batch is writable, modify the * PTE at @ptentp to be writable. */ -#define FPB_MERGE_WRITE ((__force fpb_t)BIT(2)) +#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))/* * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty, * modify the PTE at @ptentp to be young or dirty, respectively. */ -#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(3)) +#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))static inline pte_t __pte_batch_clear_ignored (pte_t pte,fpb_t flags) {@@ -226,7 +229,9 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) pte =pte_mkclean (pte);if (likely (!(flags &FPB_RESPECT_SOFT_DIRTY ))) pte =pte_clear_soft_dirty (pte); -return pte_wrprotect (pte_mkold (pte)); +if (likely (!(flags &FPB_RESPECT_WRITE ))) + pte =pte_wrprotect (pte); +return pte_mkold (pte); }/** -- 2.30.2
Re: [PATCH v5 4/7] mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
In reply to Dev JainQuoted 11 lines · Dev Jain
Patch 6 optimizes
mprotect()by batch clearing the ptes, masking in the new protections, and batch setting the ptes. Suppose that the first pte of the batch is writable - with the current implementation offolio_pte_batch(), it is not guaranteed that the other ptes in the batch are already writable too, so we may incorrectly end up setting the writable bit on all ptes viamodify_prot_commit_ptes().Therefore, introduce
FPB_RESPECT_WRITEso that all ptes in the batch are writable or not.Signed-off-by Dev Jain <[email protected]>
Quoted 42 lines
--- mm/internal.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)mm/internal.h +8
-3@@ -208,17 +208,20 @@ typedef int __bitwise fpb_t;/* Compare PTEs respecting the soft-dirty bit. */ #define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1)) +/* Compare PTEs respecting the writable bit. */ +#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2)) +/* * Merge PTE write bits: if any PTE in the batch is writable, modify the * PTE at @ptentp to be writable. */ -#define FPB_MERGE_WRITE ((__force fpb_t)BIT(2)) +#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))/* * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty, * modify the PTE at @ptentp to be young or dirty, respectively. */ -#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(3)) +#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))static inline pte_t __pte_batch_clear_ignored (pte_t pte,fpb_t flags) {@@ -226,7 +229,9 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) pte =pte_mkclean (pte);if (likely (!(flags &FPB_RESPECT_SOFT_DIRTY ))) pte =pte_clear_soft_dirty (pte); -return pte_wrprotect (pte_mkold (pte)); +if (likely (!(flags &FPB_RESPECT_WRITE ))) + pte =pte_wrprotect (pte); +return pte_mkold (pte); }/**
Re: [PATCH v5 4/7] mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
In reply to Dev JainQuoted 1 line · Dev Jain
Patch 6 optimizes
mprotect()by batch clearing the ptes, masking in the new
“Patch 6” might not make sense when reading it in the git log. Something like below might be better:
mprotect() will be optimized by batch clearing the ptes, masking in the new protections, and batch setting the ptes in an upcoming commit.
No need to repin for this one.
Quoted 6 lines
protections, and batch setting the ptes. Suppose that the first pte of the batch is writable - with the current implementation of
folio_pte_batch(), it is not guaranteed that the other ptes in the batch are already writable too, so we may incorrectly end up setting the writable bit on all ptes viamodify_prot_commit_ptes().
Quoted 8 lines
Therefore, introduce
FPB_RESPECT_WRITEso that all ptes in the batch are writable or not.Signed-off-by Dev Jain <[email protected]>--- mm/internal.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
Re: [PATCH v5 4/7] mm: Introduce FPB_RESPECT_WRITE for PTE batching infrastructure
Quoted 5 lines · Zi Yan
Quoted 1 line · Dev Jain
Patch 6 optimizes
mprotect()by batch clearing the ptes, masking in the new“Patch 6” might not make sense when reading it in the git log. Something like below might be better:
Andrew has fixed that for me :)
Quoted 15 lines
mprotect()will be optimized by batch clearing the ptes, masking in the new protections, and batch setting the ptes in an upcoming commit.No need to repin for this one.
Quoted 9 lines
protections, and batch setting the ptes. Suppose that the first pte of the batch is writable - with the current implementation of
folio_pte_batch(), it is not guaranteed that the other ptes in the batch are already writable too, so we may incorrectly end up setting the writable bit on all ptes viamodify_prot_commit_ptes().Therefore, introduce
FPB_RESPECT_WRITEso that all ptes in the batch are writable or not.
Quoted 6 lines
Quoted 5 lines
Signed-off-by Dev Jain <[email protected]>--- mm/internal.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)LGTM. Reviewed-by: Zi Yan <[email protected]>
Thanks.
Quoted 3 lines
Best Regards,
Yan, Zi
PATCH v5 5/7 mm: Split can_change_pte_writable() into private and shared parts
In reply to Dev JainIn preparation for patch 6 and modularizing the code in general, split can_change_pte_writable() into private and shared VMA parts. No functional change intended.
--- mm/mprotect.c | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-)
mm/mprotect.c +36 -14
@@ -40,11 +40,8 @@#include "internal.h" -bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +static bool maybe_change_pte_writable (struct vm_area_struct *vma,pte_t pte) { -struct page *page; -if (WARN_ON_ONCE (!(vma->vm_flags &VM_WRITE )))return false;
@@ -60,16 +57,32 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,if (userfaultfd_pte_wp (vma, pte))return false; -if (!(vma->vm_flags &VM_SHARED )) { -/* -* Writable MAP_PRIVATE mapping: We can only special-case on -* exclusive anonymous pages, because we know that our -* write-fault handler similarly would map them writable without -* any additional checks while holding the PT lock. -*/ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); - } +return true; +} + +static bool can_change_private_pte_writable (struct vm_area_struct *vma, +unsigned long addr,pte_t pte) +{ +struct page *page; + +if (!maybe_change_pte_writable (vma, pte)) +return false; + +/* +* Writable MAP_PRIVATE mapping: We can only special-case on +* exclusive anonymous pages, because we know that our +* write-fault handler similarly would map them writable without +* any additional checks while holding the PT lock. +*/ + page =vm_normal_page (vma, addr, pte); +return page &&PageAnon (page) &&PageAnonExclusive (page); +} + +static bool can_change_shared_pte_writable (struct vm_area_struct *vma, +pte_t pte) +{ +if (!maybe_change_pte_writable (vma, pte)) +return false;VM_WARN_ON_ONCE (is_zero_pfn (pte_pfn (pte)) &&pte_dirty (pte));
@@ -83,6 +96,15 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte) +{ +if (!(vma->vm_flags &VM_SHARED )) +return can_change_private_pte_writable (vma, addr, pte); + +return can_change_shared_pte_writable (vma, pte); +} +static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep,pte_t pte,int max_nr_ptes) {
Signature · 1 line
2.30.2
Re: [PATCH v5 5/7] mm: Split can_change_pte_writable() into private and shared parts
Quoted 6 lines · Dev Jain
In preparation for patch 6 and modularizing the code in general, split
can_change_pte_writable()into private and shared VMA parts. No functional change intended.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Dev Jain <[email protected]>
Great thanks! This is much clearer I think (of course being the Suggested-by here makes me somewhat biased :P) :>)
LGTM, so:
Quoted 83 lines
--- mm/mprotect.c | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-)mm/mprotect.c +36
-15@@ -40,11 +40,8 @@#include "internal.h" -bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, -pte_t pte) +static bool maybe_change_pte_writable (struct vm_area_struct *vma,pte_t pte) { -struct page *page; -if (WARN_ON_ONCE (!(vma->vm_flags &VM_WRITE )))return false;@@ -60,16 +57,32 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,if (userfaultfd_pte_wp (vma, pte))return false; -if (!(vma->vm_flags &VM_SHARED )) { -/* -* Writable MAP_PRIVATE mapping: We can only special-case on -* exclusive anonymous pages, because we know that our -* write-fault handler similarly would map them writable without -* any additional checks while holding the PT lock. -*/ - page =vm_normal_page (vma, addr, pte); -return page &&PageAnon (page) &&PageAnonExclusive (page); - } +return true; +} + +static bool can_change_private_pte_writable (struct vm_area_struct *vma, +unsigned long addr,pte_t pte) +{ +struct page *page; + +if (!maybe_change_pte_writable (vma, pte)) +return false; + +/* +* Writable MAP_PRIVATE mapping: We can only special-case on +* exclusive anonymous pages, because we know that our +* write-fault handler similarly would map them writable without +* any additional checks while holding the PT lock. +*/ + page =vm_normal_page (vma, addr, pte); +return page &&PageAnon (page) &&PageAnonExclusive (page); +} + +static bool can_change_shared_pte_writable (struct vm_area_struct *vma, +pte_t pte) +{ +if (!maybe_change_pte_writable (vma, pte)) +return false;VM_WARN_ON_ONCE (is_zero_pfn (pte_pfn (pte)) &&pte_dirty (pte));@@ -83,6 +96,15 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return pte_dirty (pte); } +bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr, +pte_t pte) +{ +if (!(vma->vm_flags &VM_SHARED )) +return can_change_private_pte_writable (vma, addr, pte); + +return can_change_shared_pte_writable (vma, pte); +} +static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep,pte_t pte,int max_nr_ptes) { --2.30.2
Re: [PATCH v5 5/7] mm: Split can_change_pte_writable() into private and shared parts
In reply to Dev JainQuoted 10 lines · Dev Jain
In preparation for patch 6 and modularizing the code in general, split
can_change_pte_writable()into private and shared VMA parts. No functional change intended.Suggested-by Lorenzo Stoakes <[email protected]>Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
PATCH v5 6/7 mm: Optimize mprotect() by PTE batching
In reply to Dev JainUse folio_pte_batch to batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.
The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via modify_prot_start_ptes()).
Observe that the flag combination we pass to mprotect_folio_pte_batch() guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.
Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.
So now we need to reason whether something can go wrong via can_change_pte_writable(). The pte_protnone, pte_needs_soft_dirty_wp, and userfaultfd_pte_wp cases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.
Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around pte_dirty() in can_change_shared_pte_writable() (in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).
To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For pte_needs_flush():
ppc does not care about the a/d bits.
For x86, PAGE_SAVED_DIRTY is ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.
--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)
mm/mprotect.c +113 -12
@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)
@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,
@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +} + +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx; + +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) { +if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ +static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) { + expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx); + len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, + first_page, expected_anon_exclusive); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } +} + +static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +bool set_write; + +if (vma->vm_flags &VM_SHARED ) { + set_write =can_change_shared_pte_writable (vma, ptent); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } + + set_write =maybe_change_pte_writable (vma, ptent) && + (folio &&folio_test_anon (folio)); +if (!set_write) { +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } +commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)
@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes =1 ; oldpte =ptep_get (pte);if (pte_present (oldpte)) { +const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ; -struct folio *folio; +struct folio *folio =NULL ;pte_t ptent;/*
@@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);
@@ -248,14 +350,13 @@ static long change_pte_range(struct mmu_gather *tlb,* COW or special handling is required. */ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && - !pte_write (ptent) && -can_change_pte_writable (vma, addr, ptent)) - ptent =pte_mkwrite (ptent, vma); - -modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes); -if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; + !pte_write (ptent)) +set_write_prot_commit_flush_ptes (vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb); +else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb); + pages += nr_ptes; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte;
Signature · 1 line
2.30.2
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 49 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
Quoted 9 lines
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.
Thanks for great commit message!
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>
This is looking MUCH better :) Thanks!
Some nits below, but I've gone through this carefully and can't find anything that seems obviously wrong here, so:
Quoted 51 lines
--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)mm/mprotect.c +23
-2@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +}
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 210
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx;
Nit but:
int end = start_idx + max_len;
for (idx = start_idx + 1; idx < end; idx++) {Would be a little neater here.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 212
Quoted 2 lines
mm/mprotect.c +2
-0+ +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) {
Nitty again but the below might be a little clearer?
struct page *page = &firstpage[idx]; if (expected_anon_exclusive != PageAnonExclusive(page))
Quoted 10 lines
mm/mprotect.c +10
-0+if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 232
Quoted 10 lines
mm/mprotect.c +10
-0+* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/
Nice comment thanks.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 242
Quoted 10 lines
mm/mprotect.c +10
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) {
I'd prefer this to be:
int i;
...
for (i = 0; i < nr_ptes; i += len, sub_batch_idx += len) {mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 243
Quoted 1 line
mm/mprotect.c +1
-0+ expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx);
Nit but would prefer:
struct page *page = &first_page[sub_batch_idx]; expected_anon_exclusive = PageAnonExclusive(page);
Quoted 89 lines
mm/mprotect.c +47
-11+ len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, + first_page, expected_anon_exclusive); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } +} + +static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +bool set_write; + +if (vma->vm_flags &VM_SHARED ) { + set_write =can_change_shared_pte_writable (vma, ptent); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } + + set_write =maybe_change_pte_writable (vma, ptent) && + (folio &&folio_test_anon (folio)); +if (!set_write) { +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } +commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes =1 ; oldpte =ptep_get (pte);if (pte_present (oldpte)) { +const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ; -struct folio *folio; +struct folio *folio =NULL ;pte_t ptent;/* @@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);@@ -248,14 +350,13 @@ static long change_pte_range(struct mmu_gather *tlb,* COW or special handling is required. */ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && - !pte_write (ptent) && -can_change_pte_writable (vma, addr, ptent)) - ptent =pte_mkwrite (ptent, vma); - -modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes); -if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; + !pte_write (ptent)) +set_write_prot_commit_flush_ptes (vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb); +else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb); + pages += nr_ptes; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte; --2.30.2
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 61 lines · Lorenzo Stoakes
Quoted 49 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
Quoted 9 lines
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.Thanks for great commit message!
Quoted 7 lines
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>This is looking MUCH better :) Thanks!
Some nits below, but I've gone through this carefully and can't find anything that seems obviously wrong here, so:
Reviewed-by Lorenzo Stoakes <[email protected]>
Thanks!
Quoted 64 lines
Quoted 52 lines
--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)mm/mprotect.c +23
-2@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +}mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 210
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx;Nit but:
Quoted 5 lines
int end = start_idx + max_len; for (idx = start_idx + 1; idx < end; idx++) {Would be a little neater here.
I politely disagree :) start_idx + max_len is *obviously* the end index, no need to add one more line of code asserting that.
Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 212
Quoted 3 lines
mm/mprotect.c +2
-0+ +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) {Nitty again but the below might be a little clearer?
struct page *page = &firstpage[idx]; if (expected_anon_exclusive != PageAnonExclusive(page))
I don't think so. first_page[idx] may confuse us into thinking that we have an array of pages. Also, the way you define it assigns a stack address to struct page *page; this is not a problem in theory and the code will still be correct, but I will prefer struct page *page containing the actual address of the linear map struct page, which is vmemmap + PFN. The way I write it is, I initialize first_page from folio_page() which will derive the address from folio->page, and folio was derived from vm_normal_folio() (which was derived from the PFN in the PTE), therefore first_page will contain the actual vmemmap address of corresponding struct page, hence it is guaranteed that first_page + x will give me the x'th page in the folio.
Quoted 36 lines
Quoted 12 lines
mm/mprotect.c +10
-0+if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 232
Quoted 10 lines
mm/mprotect.c +10
-0+* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ Nice comment thanks.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 242
Quoted 10 lines
mm/mprotect.c +10
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) {I'd prefer this to be:
Quoted 7 lines
int i; ... for (i = 0; i < nr_ptes; i += len, sub_batch_idx += len) {mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 243
Quoted 1 line
mm/mprotect.c +1
-0+ expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx);
We won't be able to do nr_ptes -= len with this. And personally a while loop is clearer to me here.
Quoted 95 lines
Nit but would prefer:
struct page *page = &first_page[sub_batch_idx]; expected_anon_exclusive = PageAnonExclusive(page);Quoted 89 lines
mm/mprotect.c +47
-11+ len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, + first_page, expected_anon_exclusive); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } +} + +static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +bool set_write; + +if (vma->vm_flags &VM_SHARED ) { + set_write =can_change_shared_pte_writable (vma, ptent); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } + + set_write =maybe_change_pte_writable (vma, ptent) && + (folio &&folio_test_anon (folio)); +if (!set_write) { +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } +commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes =1 ; oldpte =ptep_get (pte);if (pte_present (oldpte)) { +const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ; -struct folio *folio; +struct folio *folio =NULL ;pte_t ptent;/* @@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);@@ -248,14 +350,13 @@ static long change_pte_range(struct mmu_gather *tlb,* COW or special handling is required. */ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && - !pte_write (ptent) && -can_change_pte_writable (vma, addr, ptent)) - ptent =pte_mkwrite (ptent, vma); - -modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes); -if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; + !pte_write (ptent)) +set_write_prot_commit_flush_ptes (vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb); +else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb); + pages += nr_ptes; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte; --2.30.2
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 6 lines · Dev Jain
Quoted 2 lines · Lorenzo Stoakes
On Fri, Jul 18, 2025 at 02:32:43PM +0530, Dev Jain wrote:
Reviewed-by Lorenzo Stoakes <[email protected]>Thanks!
You're welcome :)
Quoted 65 lines
Quoted 65 lines
Quoted 53 lines
--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)mm/mprotect.c +23
-2@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +}mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 210
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx;Nit but:
Quoted 8 lines
Quoted 5 lines
int end = start_idx + max_len; for (idx = start_idx + 1; idx < end; idx++) {Would be a little neater here.
I politely disagree :)
start_idx+max_lenis *obviously* the end index, no need to add one more line of code asserting that.
Haha, well disagreement is permitted you know ;) as long as it's polite of course...
That's fine, this isn't a big deal.
Quoted 12 lines
Quoted 10 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 212
Quoted 5 lines
mm/mprotect.c +2
-0+ +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) {Nitty again but the below might be a little clearer?
struct page *page = &firstpage[idx]; if (expected_anon_exclusive != PageAnonExclusive(page))
Quoted 11 lines
I don't think so. first_page[idx] may confuse us into thinking that we have an array of pages. Also, the way you define it assigns a stack address to struct page *page; this is not a problem in theory and the code will still be correct, but I will prefer struct page *page containing the actual address of the linear map struct page, which is vmemmap + PFN. The way I write it is, I initialize
first_pagefromfolio_page()which will derive the address fromfolio->page, and folio was derived fromvm_normal_folio()(which was derived from the PFN in the PTE), thereforefirst_pagewill contain the actual vmemmap address of corresponding struct page, hence it is guaranteed thatfirst_page+ x will give me the x'th page in the folio.
OK, I don't think this is an issue, but I"m not going to press this, as it's a trivial thing, it's fine as-is :)
Quoted 40 lines
Quoted 40 lines
Quoted 14 lines
mm/mprotect.c +10
-0+if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 232
Quoted 10 lines
mm/mprotect.c +10
-0+* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ Nice comment thanks.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 242
Quoted 10 lines
mm/mprotect.c +10
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) {I'd prefer this to be:
int i;
Quoted 8 lines
Quoted 5 lines
... for (i = 0; i < nr_ptes; i += len, sub_batch_idx += len) {mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 243
Quoted 1 line
mm/mprotect.c +1
-0+ expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx);We won't be able to do
nr_ptes-= len with this. And personally a while loop is clearer to me here.
Well, you don't need to :) maybe rename i to pte_idx + pass nr_ptes - pte_idx.
Buuuut I'm not going to press this, it's not a big deal, and I see your point!
Overall the R-b tag still stands with the above unchanged.
Thanks for doing this series and being open to feedback, I feel we're iterated to something nice here!
Cheers, Lorenzo
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 149 lines · Lorenzo Stoakes
Quoted 4 lines · Dev Jain
Quoted 2 lines · Lorenzo Stoakes
On Fri, Jul 18, 2025 at 02:32:43PM +0530, Dev Jain wrote:
Reviewed-by Lorenzo Stoakes <[email protected]>Thanks!
You're welcome :)
Quoted 63 lines
Quoted 63 lines
Quoted 51 lines
--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)mm/mprotect.c +23
-2@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +}mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 210
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx;Nit but:
Quoted 7 lines
Quoted 5 lines
int end = start_idx + max_len; for (idx = start_idx + 1; idx < end; idx++) {Would be a little neater here.
I politely disagree :)
start_idx+max_lenis *obviously* the end index, no need to add one more line of code asserting that.Haha, well disagreement is permitted you know ;) as long as it's polite of course...
That's fine, this isn't a big deal.
Quoted 8 lines
Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 212
Quoted 3 lines
mm/mprotect.c +2
-0+ +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) {Nitty again but the below might be a little clearer?
struct page *page = &firstpage[idx]; if (expected_anon_exclusive != PageAnonExclusive(page))Quoted 11 lines
I don't think so. first_page[idx] may confuse us into thinking that we have an array of pages. Also, the way you define it assigns a stack address to struct page *page; this is not a problem in theory and the code will still be correct, but I will prefer struct page *page containing the actual address of the linear map struct page, which is vmemmap + PFN. The way I write it is, I initialize
first_pagefromfolio_page()which will derive the address fromfolio->page, and folio was derived fromvm_normal_folio()(which was derived from the PFN in the PTE), thereforefirst_pagewill contain the actual vmemmap address of corresponding struct page, hence it is guaranteed thatfirst_page+ x will give me the x'th page in the folio.OK, I don't think this is an issue, but I"m not going to press this, as it's a trivial thing, it's fine as-is :)
Quoted 38 lines
Quoted 38 lines
Quoted 12 lines
mm/mprotect.c +10
-0+if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 232
Quoted 10 lines
mm/mprotect.c +10
-0+* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ Nice comment thanks.
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 242
Quoted 10 lines
mm/mprotect.c +10
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) {I'd prefer this to be:
int i;Quoted 7 lines
Quoted 5 lines
... for (i = 0; i < nr_ptes; i += len, sub_batch_idx += len) {mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 243
Quoted 1 line
mm/mprotect.c +1
-0+ expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx);We won't be able to do
nr_ptes-= len with this. And personally a while loop is clearer to me here.
Quoted 8 lines
Well, you don't need to :) maybe rename i to
pte_idx+ passnr_ptes-pte_idx.Buuuut I'm not going to press this, it's not a big deal, and I see your point!
Overall the R-b tag still stands with the above unchanged.
Thanks for doing this series and being open to feedback, I feel we're iterated to something nice here!
We definitely iterated to something nice! I definitely found your review very useful; as contributors trying to solve our own problems, we simply keep adding new features and the code becomes a mess of if-else statements - this is what I felt when I began reading kernel code. I am reading code and suddenly some conditional handling a uffd-wp pte appears (I know you have some nice words about uffd :))
Thank you for actually forcing me to think into how I can implement my ideas into code better!
Quoted 2 lines
Cheers, Lorenzo
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to Dev JainQuoted 205 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.Signed-off-by Dev Jain <[email protected]>--- mm/mprotect.c | 125 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 113 insertions(+), 12 deletions(-)mm/mprotect.c +101
-4@@ -106,7 +106,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, }static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, -pte_t pte,int max_nr_ptes) +pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */ if (!folio)@@ -115,7 +115,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,if (!folio_test_large (folio))return 1 ; -return folio_pte_batch (folio, ptep, pte, max_nr_ptes); +return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr_ptes, flags); } static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,@@ -177,6 +177,102 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,return ret; } +/* Set nr_ptes number of ptes, starting from idx */ +static void prot_commit_flush_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes, +int idx,bool set_write,struct mmu_gather *tlb) +{ +/* +* Advance the position in the batch by idx; note that if idx > 0, +* then the nr_ptes passed here is <= batch size - idx. +*/ + addr += idx *PAGE_SIZE ; + ptep += idx; + oldpte =pte_advance_pfn (oldpte, idx); + ptent =pte_advance_pfn (ptent, idx); + +if (set_write) + ptent =pte_mkwrite (ptent, vma); + +modify_prot_commit_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr, nr_ptes *PAGE_SIZE ); +} + +/* +* Get max length of consecutive ptes pointing to PageAnonExclusive() pages or +* !PageAnonExclusive() pages, starting from start_idx. Caller must enforce +* that the ptes point to consecutive pages of the same anon large folio. +*/ +static int page_anon_exclusive_sub_batch (int start_idx,int max_len, +struct page *first_page,bool expected_anon_exclusive) +{ +int idx; + +for (idx = start_idx +1 ; idx < start_idx + max_len; ++idx) { +if (expected_anon_exclusive !=PageAnonExclusive (first_page + idx)) +break ; + } +return idx - start_idx; +} + +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ +static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 ); +bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) { + expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx); + len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, + first_page, expected_anon_exclusive); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } +} + +static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +bool set_write; + +if (vma->vm_flags &VM_SHARED ) { + set_write =can_change_shared_pte_writable (vma, ptent); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } + + set_write =maybe_change_pte_writable (vma, ptent) && + (folio &&folio_test_anon (folio)); +if (!set_write) { +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } +commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes =1 ; oldpte =ptep_get (pte);if (pte_present (oldpte)) { +const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ; -struct folio *folio; +struct folio *folio =NULL ;pte_t ptent;/* @@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 331
Quoted 10 lines
mm/mprotect.c +5
-0continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes);
Here both mprotect_folio_pte_batch() and modify_prot_start_ptes() can collect dirty and access bits from PTEs in the batch. But modify_prot_start_ptes() clears PTEs to prevent concurrent hardware modification. It took me a while to figure this out, since I was wondering why oldpte is not used after mprotect_folio_pte_batch().
I wish folio_pte_batch_flags() used by mprotect_folio_pte_batch() can take a function pointer to allow either read PTEs or clear PTEs, so that there could be one function. But that could complicate folio_pte_batch_flags(). Just a comment. :)
Best Regards,
Yan, Zi
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to Dev JainQuoted 51 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():
Quoted 8 lines
ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.Signed-off-by Dev Jain <[email protected]>
I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
[...]
Quoted 10 lines
mm/mprotect.c +10
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 237
Quoted 10 lines
mm/mprotect.c +10
-0+* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ +static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 );
Who says that we have the first page of the folio mapped into the first PTE of the batch?
Quoted 75 lines
mm/mprotect.c +46
-3+bool expected_anon_exclusive; +int sub_batch_idx =0 ; +int len; + +while (nr_ptes) { + expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx); + len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, + first_page, expected_anon_exclusive); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, + sub_batch_idx, expected_anon_exclusive, tlb); + sub_batch_idx += len; + nr_ptes -= len; + } +} + +static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +bool set_write; + +if (vma->vm_flags &VM_SHARED ) { + set_write =can_change_shared_pte_writable (vma, ptent); +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } + + set_write =maybe_change_pte_writable (vma, ptent) && + (folio &&folio_test_anon (folio)); +if (!set_write) { +prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, nr_ptes, +/* idx = */ 0 , set_write, tlb); +return ; + } +commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +} +static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes =1 ; oldpte =ptep_get (pte);if (pte_present (oldpte)) { +const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ; -struct folio *folio; +struct folio *folio =NULL ;pte_t ptent;/* @@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);@@ -248,14 +350,13 @@ static long change_pte_range(struct mmu_gather *tlb,* COW or special handling is required. */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && - !pte_write(ptent) &&
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 355
Quoted 10 lines
mm/mprotect.c +3
-7- can_change_pte_writable(vma, addr, ptent)) - ptent =pte_mkwrite (ptent, vma); - -modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes); -if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; + !pte_write(ptent)) +set_write_prot_commit_flush_ptes (vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb);
While staring at this:
Very broken indentation.
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 358
Quoted 3 lines
mm/mprotect.c +3
-0+else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb);
Semi-broken intendation.
Quoted 4 lines
mm/mprotect.c +1
-0+ pages += nr_ptes; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte;
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 82 lines · David Hildenbrand
Quoted 51 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():Quoted 8 lines
ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
[...]
Quoted 15 lines
mm/mprotect.c +15
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/
Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 237
Quoted 5 lines
mm/mprotect.c +5
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 );Who says that we have the first page of the folio mapped into the first PTE of the batch?
For the record, I *hate* that we moved from vm_normal_folio() to vm_normal_page(). Please undo that and forward the proper mapped page.
Signature · 3 lines
Cheers, David / dhildenb
Will Deacon
maintainerRe: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to David HildenbrandQuoted 59 lines · David Hildenbrand
Quoted 58 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.
Quoted 5 lines
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
Please excuse my laziness, but do you have a link to the report? I've been looking at some oddities on arm64 coming back from some of the CI systems and was heading in the direction of a recent mm regression judging by the first-known-bad-build in linux-next.
https://lore.kernel.org/r/CA+G9fYumD2MGjECCv0wx2V_96_FKNtFQpT63qVNrrCmomoPYVQ@mail.gmail.com
Will
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 60 lines · Will Deacon
Quoted 59 lines · David Hildenbrand
Quoted 58 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
Please excuse my laziness, but do you have a link to the report?
I was lazy :)
https://lkml.kernel.org/r/[email protected]
Quoted 6 lines
I've
been looking at some oddities on arm64 coming back from some of the CI systems and was heading in the direction of a recent mm regression judging by the first-known-bad-build in linux-next.https://lore.kernel.org/r/CA+G9fYumD2MGjECCv0wx2V_96_FKNtFQpT63qVNrrCmomoPYVQ@mail.gmail.com
Hm, mprotect seems to be involved. So it might or might not correlate.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to David HildenbrandQuoted 59 lines · David Hildenbrand
Quoted 58 lines · Dev Jain
Use
folio_pte_batchto batch process a large folio. Note that, PTE batching here will save a few function calls, and this strategy in certain cases (not this one) batches atomic operations in general, so we have a performance win for all arches. This patch paves the way for patch 7 which will help us elide the TLBI per contig block on arm64.The correctness of this patch lies on the correctness of setting the new ptes based upon information only from the first pte of the batch (which may also have accumulated a/d bits via
modify_prot_start_ptes()).Observe that the flag combination we pass to
mprotect_folio_pte_batch()guarantees that the batch is uniform w.r.t the soft-dirty bit and the writable bit. Therefore, the only bits which may differ are the a/d bits. So we only need to worry about code which is concerned about the a/d bits of the PTEs.Setting extra a/d bits on the new ptes where previously they were not set, is fine - setting access bit when it was not set is not an incorrectness problem but will only possibly delay the reclaim of the page mapped by the pte (which is in fact intended because the kernel just operated on this region via
mprotect()!). Setting dirty bit when it was not set is again not an incorrectness problem but will only possibly force an unnecessary writeback.So now we need to reason whether something can go wrong via
can_change_pte_writable(). Thepte_protnone,pte_needs_soft_dirty_wp, anduserfaultfd_pte_wpcases are solved due to uniformity in the corresponding bits guaranteed by the flag combination. The ptes all belong to the same VMA (since callers guarantee that [start, end) will lie within the VMA) therefore the conditional based on the VMA is also safe to batch around.Since the dirty bit on the PTE really is just an indication that the folio got written to - even if the PTE is not actually dirty but one of the PTEs in the batch is, the wp-fault optimization can be made. Therefore, it is safe to batch around
pte_dirty()incan_change_shared_pte_writable()(in fact this is better since without batching, it may happen that some ptes aren't changed to writable just because they are not dirty, even though the other ptes mapping the same large folio are dirty).To batch around the PageAnonExclusive case, we must check the corresponding condition for every single page. Therefore, from the large folio batch, we process sub batches of ptes mapping pages with the same PageAnonExclusive condition, and process that sub batch, then determine and process the next sub batch, and so on. Note that this does not cause any extra overhead; if suppose the size of the folio batch is 512, then the sub batch processing in total will take 512 iterations, which is the same as what we would have done before.
For
pte_needs_flush():ppc does not care about the a/d bits.
For x86,
PAGE_SAVED_DIRTYis ignored. We will flush only when a/d bits get cleared; since we can only have extra a/d bits due to batching, we will only have an extra flush, not a case where we elide a flush due to batching when we shouldn't have.
Quoted 4 lines
Quoted 1 line
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
Thank you for CCing, and quickly spotting the problem!
Quoted 27 lines
[...]
Quoted 24 lines
mm/mprotect.c +4
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In
can_change_pte_writable(),
+ * if the vma is a private vma, and we cannot determine whether to change
+ * the pte to writable just from the vma and the pte, we then need to look
+ * at the actual page pointed to by the pte. Unfortunately, if we have a
+ * batch of ptes pointing to consecutive pages of the same anon large folio,
+ * the anon-exclusivity (or the negation) of the first page does not guarantee
+ * the anon-exclusivity (or the negation) of the other pages corresponding to
+ * the pte batch; hence in this case it is incorrect to decide to change or
+ * not change the ptes to writable just by using information from the first
+ * pte of the batch. Therefore, we must individually check all pages and
+ * retrieve sub-batches.
Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 237
Quoted 6 lines
+ */
mm/mprotect.c +5
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, + struct folio *folio, unsigned long addr, pte_t *ptep, + pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb) +{ + struct page *first_page = folio_page(folio, 0);Who says that we have the first page of the folio mapped into the first PTE of the batch?
Oops, I thought I had taken care of this. We make the folio from vm_normal_folio(vma, addr, oldpte),
which makes the struct page from the actual pte, but then I missed that page_folio() will reset the folio->page
to the head page, I was under the impression it will retain the original page. And indeed, if it didn't do that
then the purpose of vm_normal_folio() is nullified, should have thought about that :( Lemme send a fix patch.
Quoted 92 lines
Quoted 81 lines
+ bool
expected_anon_exclusive;
+ intsub_batch_idx= 0;
+ int len;
+
+ while (nr_ptes) {
+expected_anon_exclusive=PageAnonExclusive(first_page + sub_batch_idx);
+ len = page_anon_exclusive_sub_batch(sub_batch_idx,nr_ptes, +first_page,expected_anon_exclusive); + prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, len, +sub_batch_idx,expected_anon_exclusive, tlb); +sub_batch_idx+= len;
+nr_ptes-= len;
+ }mm/mprotect.c +3
-0+} + +static void set_write_prot_commit_flush_ptes(struct vm_area_struct*vma,
+ struct folio *folio, unsigned long addr,pte_t*ptep, +pte_toldpte,pte_tptent, intnr_ptes, structmmu_gather*tlb)mm/mprotect.c +18
-0+{ + bool set_write; + + if (vma->vm_flags & VM_SHARED) { + set_write = can_change_shared_pte_writable(vma, ptent); + prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, nr_ptes, + /* idx = */ 0, set_write, tlb); + return; + } + + set_write = maybe_change_pte_writable(vma, ptent) && + (folio && folio_test_anon(folio)); + if (!set_write) { + prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, nr_ptes, + /* idx = */ 0, set_write, tlb); + return; + } + commit_anon_folio_batch(vma, folio, addr, ptep, oldpte, ptent,
nr_ptes, tlb);(quoted diff) +2
-0+} +static long change_pte_range(struct
mmu_gather*tlb, structvm_area_struct*vma,pmd_t*pmd, unsigned long addr, unsigned long end,pgprot_tnewprot, unsigned longcp_flags)mm/mprotect.c +7
-2@@ -206,8 +302,9 @@ static long change_pte_range(struct mmu_gather *tlb, nr_ptes = 1; oldpte = ptep_get(pte); if (pte_present(oldpte)) { + const fpb_t flags = FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ; int max_nr_ptes = (end - addr) >> PAGE_SHIFT; - struct folio *folio; + struct folio *folio = NULL; pte_t ptent; /*@@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb, /* determine batch to skip */ nr_ptes = mprotect_folio_pte_batch(folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes, /* flags = */ 0); continue; } } + if (!folio) + folio = vm_normal_folio(vma, addr, oldpte); + + nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags); + oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes); ptent = pte_modify(oldpte, newprot);@@ -248,14 +350,13 @@ static long change_pte_range(struct
mmu_gather*tlb,
* COW or special handling is required.
*/
if ((cp_flags&MM_CP_TRY_CHANGE_WRITABLE) &&
- !pte_write(ptent) &&
-can_change_pte_writable(vma, addr, ptent))
- ptent =pte_mkwrite(ptent, vma);
-Quoted 8 lines
-
modify_prot_commit_ptes(vma, addr, pte, oldpte, ptent, nr_ptes);
- if (pte_needs_flush(oldpte, ptent))
-tlb_flush_pte_range(tlb, addr, PAGE_SIZE); - pages++;
+ !pte_write(ptent))
+set_write_prot_commit_flush_ptes(vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb);While staring at this:
Quoted 1 line
Very broken indentation.
Indeed, but then in order to fix the indentation if we start positioning
every argument just below the opening bracket then it will take at least four lines :)
Quoted 7 lines
Quoted 5 lines
+ else
+ prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, +nr_ptes, /* idx = */ 0, /*set_write= */ false, tlb);Semi-broken intendation.
Yup I can position nr_ptes below vma and set_write below nr_ptes. But this
should probably not be a part of the fix patch.
Quoted 7 lines
Quoted 5 lines
+ pages +=
nr_ptes;
} else if (is_swap_pte(oldpte)) {swp_entry_tentry =pte_to_swp_entry(oldpte);pte_tnewpte;
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 9 lines
Quoted 4 lines
While staring at this:
Very broken indentation.
Indeed, but then in order to fix the indentation if we start positioning
every argument just below the opening bracket then it will take at least four lines :)
You are allowed to use more than 80 chars if there is good reason :)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to David HildenbrandQuoted 24 lines · David Hildenbrand
Quoted 1 line · Dev Jain
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
[...]
Quoted 15 lines
mm/mprotect.c +15
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/
Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 237
Quoted 5 lines
mm/mprotect.c +5
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 );Who says that we have the first page of the folio mapped into the first PTE of the batch?
Yikes, missed this sorry. Got too tied up in alogrithm here.
You mean in _this_ PTE of the batch right? As we're invoking these on each part of the PTE table.
I mean I guess we can simply do:
struct page *first_page = pte_page(ptent);
Right?
Presuming ptent is the PTE entry we are curently examining (and applying the PAE sub-batching to etc.)
[snip]
Quoted 33 lines
Quoted 20 lines
mm/mprotect.c +6
-2@@ -221,11 +318,16 @@ static long change_pte_range(struct mmu_gather *tlb,/* determine batch to skip */ nr_ptes =mprotect_folio_pte_batch (folio, - pte, oldpte, max_nr_ptes); + pte, oldpte, max_nr_ptes,/* flags = */ 0 );continue ; } } +if (!folio) + folio =vm_normal_folio (vma, addr, oldpte); + + nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); + oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes); ptent =pte_modify (oldpte, newprot);@@ -248,14 +350,13 @@ static long change_pte_range(struct mmu_gather *tlb,* COW or special handling is required. */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && - !pte_write(ptent) &&mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 355
Quoted 10 lines
mm/mprotect.c +3
-7- can_change_pte_writable(vma, addr, ptent)) - ptent =pte_mkwrite (ptent, vma); - -modify_prot_commit_ptes (vma, addr, pte, oldpte, ptent, nr_ptes); -if (pte_needs_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); - pages++; + !pte_write(ptent)) +set_write_prot_commit_flush_ptes (vma, folio, + addr, pte, oldpte, ptent, nr_ptes, tlb);While staring at this:
Quoted 1 line
Very broken indentation.
Hmm... I wonder if checkpatch.pl would have picked this up actually.
Yeah that 2nd line is just wrong...
Quoted 6 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 358
Quoted 4 lines
mm/mprotect.c +3
-0+else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb);Semi-broken intendation.
Because of else then 2 lines after?
Quoted 10 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 362
Quoted 5 lines
mm/mprotect.c +1
-0+ pages += nr_ptes; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);pte_t newpte;--
Cheers,
Quoted 2 lines
David / dhildenb
I think on this series I was so happy to see the implementation evolve to something that was so much nicer than it originally was, I did not scrutinise the impl. details of this one close enough, but as Jamie Lannister said, "there are always lessons in failures" :)
Cheers, Lorenzo
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 36 lines · Lorenzo Stoakes
Quoted 24 lines · David Hildenbrand
Quoted 1 line · Dev Jain
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
[...]
Quoted 15 lines
mm/mprotect.c +15
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In can_change_pte_writable(), +* if the vma is a private vma, and we cannot determine whether to change +* the pte to writable just from the vma and the pte, we then need to look +* at the actual page pointed to by the pte. Unfortunately, if we have a +* batch of ptes pointing to consecutive pages of the same anon large folio, +* the anon-exclusivity (or the negation) of the first page does not guarantee +* the anon-exclusivity (or the negation) of the other pages corresponding to +* the pte batch; hence in this case it is incorrect to decide to change or +* not change the ptes to writable just by using information from the first +* pte of the batch. Therefore, we must individually check all pages and +* retrieve sub-batches. +*/ Quoted 8 lines
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, · line 237
Quoted 5 lines
mm/mprotect.c +5
-0+static void commit_anon_folio_batch (struct vm_area_struct *vma, +struct folio *folio,unsigned long addr,pte_t *ptep, +pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) +{ +struct page *first_page =folio_page (folio,0 );Who says that we have the first page of the folio mapped into the first PTE of the batch?
Yikes, missed this sorry. Got too tied up in alogrithm here.
Quoted 8 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part of the PTE table.
I mean I guess we can simply do:
struct page *first_page = pte_page(ptent);Right?
Yes, but we should forward the result from vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.
...
Quoted 8 lines
Quoted 6 lines
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 358
Quoted 4 lines
mm/mprotect.c +3
-0+else +prot_commit_flush_ptes (vma, addr, pte, oldpte, ptent, + nr_ptes,/* idx = */ 0 ,/* set_write = */ false, tlb);Semi-broken intendation.
Because of else then 2 lines after?
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
Is what I would have expected.
I think a smart man once said, that if you need more than one line per statement in an if/else clause, a set of {} can aid readability. But I don't particularly care :)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 53 lines · David Hildenbrand
Quoted 52 lines · Lorenzo Stoakes
Quoted 35 lines · David Hildenbrand
Quoted 1 line · Dev Jain
Signed-off-by Dev Jain <[email protected]>I wanted to review this, but looks like it's already upstream and I suspect it's buggy (see the upstream report I cc'ed you on)
[...]
Quoted 26 lines
mm/mprotect.c +4
-0+ +/* +* This function is a result of trying our very best to retain the +* "avoid the write-fault handler" optimization. In
can_change_pte_writable(),
+ * if the vma is a private vma, and we cannot determine whether to change
+ * the pte to writable just from the vma and the pte, we then need to look
+ * at the actual page pointed to by the pte. Unfortunately, if we have a
+ * batch of ptes pointing to consecutive pages of the same anon large folio,
+ * the anon-exclusivity (or the negation) of the first page does not guarantee
+ * the anon-exclusivity (or the negation) of the other pages corresponding to
+ * the pte batch; hence in this case it is incorrect to decide to change or
+ * not change the ptes to writable just by using information from the first
+ * pte of the batch. Therefore, we must individually check all pages and
+ * retrieve sub-batches.
+ */
+static void commit_anon_folio_batch(structvm_area_struct*vma,Quoted 8 lines
Quoted 5 lines
+ struct folio *folio, unsigned long addr,
pte_t*ptep, +pte_toldpte,pte_tptent, intnr_ptes, structmmu_gather*tlb)
+{
+ struct page*first_page=folio_page(folio, 0);Who says that we have the first page of the folio mapped into the first PTE of the batch?
Yikes, missed this sorry. Got too tied up in alogrithm here.
You mean in _this_ PTE of the batch right? As we're invoking these on each part of the PTE table.
I mean I guess we can simply do:
Quoted 7 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.
Makes sense, so I guess I will have to change the signature of prot_numa_skip()
to pass a double ptr to a page instead of folio and derive the folio in the caller,
and pass down both the folio and the page to set_write_prot_commit_flush_ptes.
Quoted 22 lines
...
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
+ else
+ prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, +nr_ptes, /* idx = */ 0, /*set_write= */ false, tlb);Semi-broken intendation.
Because of else then 2 lines after?
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);Is what I would have expected.
I think a smart man once said, that if you need more than one line per statement in an if/else clause, a set of {} can aid readability. But I don't particularly care :)
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 15 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 7 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.
Quoted 8 lines
Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.
I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
I foresee some debate otherwise...
Quoted 18 lines
Quoted 18 lines
...
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
+ else
+ prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, +nr_ptes, /* idx = */ 0, /*set_write= */ false, tlb);Semi-broken intendation.
Because of else then 2 lines after?
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, nr_ptes, /* idx = */ 0, /* set_write = */ false, tlb);
Quoted 8 lines
Quoted 8 lines
Is what I would have expected.
I think a smart man once said, that if you need more than one line per statement in
an if/else clause, a set of {} can aid readability. But I don't particularly care :)
Can do this in follow up too.
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 21 lines · Lorenzo Stoakes
Quoted 12 lines · Dev Jain
Quoted 4 lines
Quoted 4 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.Quoted 8 lines
Quoted 5 lines
I mean I guess we can simply do:
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Quoted 8 lines
Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.
Quoted 7 lines
I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
I foresee some debate otherwise...
Yup I would personally prefer that. Although if you would like to see the churn, here it is (based on mm-hotfixes-unstable, untested):
---mm/mprotect.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
mm/mprotect.c +0 -0
@@ -120,9 +120,10 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,
static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, pte_t oldpte, pte_t *pte, int target_node,
- struct folio **foliop)
+ struct page **pagep)
{
struct folio *folio = NULL;
+ struct page *page = NULL;
bool ret = true;
bool toptier;
int nid;
@@ -131,7 +132,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, if (pte_protnone(oldpte))
goto skip;
- folio = vm_normal_folio(vma, addr, oldpte);
+ page = vm_normal_page(vma, addr, oldpte);
+ if (page)
+ folio = page_folio(page);
if (!folio)
goto skip;
@@ -173,7 +176,7 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
skip:
- *foliop = folio;
+ *pagep = page;
return ret;
}
@@ -231,18 +234,17 @@ static int page_anon_exclusive_sub_batch(int start_idx, int max_len,
* retrieve sub-batches.
*/
static void commit_anon_folio_batch(struct vm_area_struct *vma,
- struct folio *folio, unsigned long addr, pte_t *ptep,
+ struct folio *folio,struct page *page, unsigned long addr, pte_t *ptep,pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb)
{
- struct page *first_page = folio_page(folio, 0);
bool expected_anon_exclusive;
int sub_batch_idx = 0;
int len;
while (nr_ptes) {
- expected_anon_exclusive = PageAnonExclusive(first_page + sub_batch_idx);
+ expected_anon_exclusive = PageAnonExclusive(nth_page(page, sub_batch_idx)); len = page_anon_exclusive_sub_batch(sub_batch_idx, nr_ptes,
- first_page, expected_anon_exclusive);
+ page, expected_anon_exclusive);
prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, len,sub_batch_idx, expected_anon_exclusive, tlb);sub_batch_idx += len;
@@ -251,7 +253,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma, }
static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma,
- struct folio *folio, unsigned long addr, pte_t *ptep,
+ struct folio *folio, struct page *page, unsigned long addr, pte_t *ptep, pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb)
{
bool set_write;
@@ -270,7 +272,7 @@ static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, /* idx = */ 0, set_write, tlb);
return;
}
- commit_anon_folio_batch(vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb);
+ commit_anon_folio_batch(vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb); }
static long change_pte_range(struct mmu_gather *tlb,
@@ -305,6 +307,7 @@ static long change_pte_range(struct mmu_gather *tlb,
const fpb_t flags = FPB_RESPECT_SOFT_DIRTY | FPB_RESPECT_WRITE;
int max_nr_ptes = (end - addr) >> PAGE_SHIFT;
struct folio *folio = NULL;
+ struct page *page = NULL;pte_t ptent;
/*
@@ -313,7 +316,9 @@ static long change_pte_range(struct mmu_gather *tlb,
*/
if (prot_numa) {
int ret = prot_numa_skip(vma, addr, oldpte, pte,
- target_node, &folio);
+ target_node, &page);
+ if (page)
+ folio = page_folio(page);
if (ret) {
/* determine batch to skip */
@@ -351,7 +356,7 @@ static long change_pte_range(struct mmu_gather *tlb,
*/
if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
!pte_write(ptent))
- set_write_prot_commit_flush_ptes(vma, folio,
+ set_write_prot_commit_flush_ptes(vma, folio, page,
addr, pte, oldpte, ptent, nr_ptes, tlb);
else
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
Signature · 5 lines
2.30.2 I think the really correct way here is to pass double ptr to both folio and page, since the way I do it now, I do a page_folio() twice potentially, which kind of defeats the purpose of the trouble we took to save on a vm_normal_folio() :) So I would prefer the pte_page(ptent) thingy for the fix patch.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
...
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
+ else
+ prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, +nr_ptes, /* idx = */ 0, /*set_write= */ false, tlb);Semi-broken intendation.
Because of else then 2 lines after?
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
nr_ptes, /* idx = */ 0, /*set_write= */ false, tlb);Quoted 8 lines
Quoted 8 lines
Is what I would have expected.
I think a smart man once said, that if you need more than one line per statement in
an if/else clause, a set of {} can aid readability. But I don't particularly care :)Can do this in follow up too.
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
No idea why Thunderbird decides to send an HTML email whenever
I copy code into it. Will fix this for future once and for all.
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to Dev JainQuoted 54 lines · Dev Jain
Quoted 21 lines · Lorenzo Stoakes
Quoted 12 lines · Dev Jain
Quoted 4 lines
Quoted 4 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.Quoted 8 lines
Quoted 5 lines
I mean I guess we can simply do:
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Quoted 8 lines
Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.Quoted 7 lines
I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
I foresee some debate otherwise...
Yup I would personally prefer that. Although if you would like to see the churn, here it is (based on mm-hotfixes-unstable, untested):
---
mm/mprotect.c| 27 ++++++++++++++++-----------1 file changed, 16 insertions(+), 11 deletions(-)mm/mprotect.c +2
-1@@ -120,9 +120,10 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep, static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,pte_t oldpte,pte_t *pte,int target_node, -struct folio **foliop) +struct page **pagep) {struct folio *folio =NULL ; +struct page *page =NULL ;bool ret = true;bool toptier;int nid;
mm/mprotect.c static bool prot_numa_skip(struct vm_area_struct · line 137
Quoted 7 lines
mm/mprotect.c +3
-1@@ -131,7 +132,9 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, if (pte_protnone(oldpte)) goto skip; - folio = vm_normal_folio(vma, addr, oldpte); + page = vm_normal_page(vma, addr, oldpte); +if (page) + folio =page_folio (page);
See the draft I just send, where we move that vm_normal_page() into the caller.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to Lorenzo StoakesQuoted 22 lines · Lorenzo Stoakes
Quoted 21 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 7 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
Quoted 8 lines
Quoted 2 lines
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
This is not an issue in any released kernel, so we can do this properly.
We should just remove that nested vm_normal_folio().
Untested, but should give an idea what we can do.
mm/mprotect.c +12 -14
@@ -120,7 +120,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr,pte_t oldpte,pte_t *pte,int target_node, -struct folio **foliop) +struct folio *folio) {struct folio *folio =NULL ;bool ret = true;
@@ -131,7 +131,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,if (pte_protnone (oldpte)) goto skip; - folio =vm_normal_folio (vma, addr, oldpte);if (!folio) goto skip;
@@ -172,8 +171,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -skip: - *foliop = folio;return ret; }
@@ -231,10 +228,9 @@ static int page_anon_exclusive_sub_batch(int start_idx, int max_len,* retrieve sub-batches. */ static void commit_anon_folio_batch (struct vm_area_struct *vma, -struct folio *folio,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *first_page,unsigned long addr,pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) { -struct page *first_page =folio_page (folio,0 );bool expected_anon_exclusive;int sub_batch_idx =0 ;int len;
@@ -243,7 +239,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma, expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx); len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, first_page, expected_anon_exclusive); -prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, +prot_commit_flush_ptes (vma, addr, ptep, page, oldpte, ptent, len, sub_batch_idx, expected_anon_exclusive, tlb); sub_batch_idx += len; nr_ptes -= len;
@@ -251,7 +247,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma, }static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, -struct folio *folio,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *page,unsigned long addr,pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) {bool set_write;
@@ -270,7 +266,7 @@ static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, /* idx = */ 0, set_write, tlb);return ; } -commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +commit_anon_folio_batch (vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb); } static long change_pte_range(struct mmu_gather *tlb,
@@ -305,15 +301,20 @@ static long change_pte_range(struct mmu_gather *tlb,const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ;struct folio *folio =NULL ; +struct page *page;pte_t ptent; + page =vm_normal_folio (vma, addr, oldpte); +if (page) + folio =page_folio (page); +/* * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ if (prot_numa) {int ret =prot_numa_skip (vma, addr, oldpte, pte, - target_node, &folio); + target_node, folio);if (ret) {/* determine batch to skip */
@@ -323,9 +324,6 @@ static long change_pte_range(struct mmu_gather *tlb, } } -if (!folio) - folio =vm_normal_folio (vma, addr, oldpte); - nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes);
@@ -351,7 +349,7 @@ static long change_pte_range(struct mmu_gather *tlb,*/ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pte_write (ptent)) -set_write_prot_commit_flush_ptes (vma, folio, +set_write_prot_commit_flush_ptes (vma, folio, page, addr, pte, oldpte, ptent, nr_ptes, tlb);else prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
Signature · 7 lines
2.50.1 -- Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Lorenzo Stoakes
Quoted 15 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 7 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Quoted 8 lines
Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
Quoted 8 lines
Quoted 2 lines
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
This is not an issue in any released kernel, so we can do this properly.
We should just remove that nested
vm_normal_folio().Untested, but should give an idea what we can do.
This puts the overhead of vm_normal_folio() unconditionally into the pte_present path.
Although I am guessing that is already happening assuming prot_numa case is not the
hot path. This is fine by me. So I guess I shouldn't have done that "reuse the folio
from prot_numa case if possible" thingy at all :)
Quoted 125 lines
mm/mprotect.c +1
-1@@ -120,7 +120,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep, static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, pte_t oldpte, pte_t *pte, int target_node, - struct folio **foliop) + struct folio *folio) { struct folio *folio = NULL;bool ret = true;
mm/mprotect.c +1
-5@@ -131,7 +131,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, if (pte_protnone(oldpte)) goto skip; - folio = vm_normal_folio(vma, addr, oldpte); if (!folio) goto skip;@@ -172,8 +171,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr, if (folio_use_access_time(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); -skip: - *foliop = folio; return ret; }@@ -231,10 +228,9 @@ static int page_anon_exclusive_sub_batch(intstart_idx, int max_len, * retrieve sub-batches. */ static void commit_anon_folio_batch(struct vm_area_struct *vma, - struct folio *folio, unsigned long addr, pte_t *ptep, + struct folio *folio, struct page *first_page, unsigned long addr,pte_t *ptep, pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb) { - struct page *first_page = folio_page(folio, 0); bool expected_anon_exclusive;int
sub_batch_idx= 0;
int len;mm/mprotect.c +1
-1@@ -243,7 +239,7 @@ static void commit_anon_folio_batch(structvm_area_struct *vma, expected_anon_exclusive = PageAnonExclusive(first_page + sub_batch_idx); len = page_anon_exclusive_sub_batch(sub_batch_idx, nr_ptes, first_page, expected_anon_exclusive); - prot_commit_flush_ptes(vma, addr, ptep, oldpte, ptent, len, + prot_commit_flush_ptes(vma, addr, ptep, page, oldpte, ptent, len, sub_batch_idx, expected_anon_exclusive, tlb);
sub_batch_idx+= len;
nr_ptes-= len;mm/mprotect.c +1
-1@@ -251,7 +247,7 @@ static void commit_anon_folio_batch(structvm_area_struct *vma, } static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, - struct folio *folio, unsigned long addr, pte_t *ptep, + struct folio *folio, struct page *page, unsigned long addr,pte_t *ptep, pte_t oldpte, pte_t ptent, int nr_ptes, struct mmu_gather *tlb){
boolset_write;mm/mprotect.c +1
-1@@ -270,7 +266,7 @@ static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, /* idx = */ 0, set_write, tlb); return; } - commit_anon_folio_batch(vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); + commit_anon_folio_batch(vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb);}
static long change_pte_range(struct
mmu_gather*tlb,mm/mprotect.c +6
-1@@ -305,15 +301,20 @@ static long change_pte_range(struct mmu_gather *tlb, const fpb_t flags = FPB_RESPECT_SOFT_DIRTY | FPB_RESPECT_WRITE; int max_nr_ptes = (end - addr) >> PAGE_SHIFT; struct folio *folio = NULL; + struct page *page; pte_t ptent; + page = vm_normal_folio(vma, addr, oldpte); + if (page) + folio = page_folio(page); + /* * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ if (prot_numa) { int ret = prot_numa_skip(vma, addr, oldpte, pte, - target_node, &folio); + target_node, folio); if (ret) {/* determine batch to skip */
mm/mprotect.c +0
-3@@ -323,9 +324,6 @@ static long change_pte_range(struct mmu_gather *tlb, } } - if (!folio) - folio = vm_normal_folio(vma, addr, oldpte); - nr_ptes = mprotect_folio_pte_batch(folio, pte, oldpte, max_nr_ptes, flags); oldpte = modify_prot_start_ptes(vma, addr, pte, nr_ptes);@@ -351,7 +349,7 @@ static long change_pte_range(struct mmu_gather *tlb,*/ if ((
cp_flags&MM_CP_TRY_CHANGE_WRITABLE) && !pte_write(ptent))
- set_write_prot_commit_flush_ptes(vma, folio, +set_write_prot_commit_flush_ptes(vma, folio, page, addr, pte, oldpte, ptent, nr_ptes, tlb); else
prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent,
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
Quoted 47 lines · Dev Jain
Quoted 36 lines · David Hildenbrand
Quoted 27 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 8 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does
exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Makes sense, so I guess I will have to change the signature of
prot_numa_skip()Quoted 6 lines
to pass a double ptr to a page instead of folio and derive the folio in the
caller,and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.Quoted 7 lines
I already don't love how we psas the folio back from there for very dubious
benefit. I really hate the idea of having a struct **page parameter...I wonder if we should just have a quick fixup for hotfix, and refine this more
later?Quoted 5 lines
This is not an issue in any released kernel, so we can do this properly.
We should just remove that nested
vm_normal_folio().Untested, but should give an idea what we can do.
This puts the overhead of
vm_normal_folio()unconditionally into thepte_presentpath.
Quoted 7 lines
Although I am guessing that is already happening assuming
prot_numacase is not thehot path. This is fine by me. So I guess I shouldn't have done that "reuse the folio
from
prot_numacase if possible" thingy at all :)
I mean, it only applies when trying to NUMA-protect something that is already protected. Not sure how relevant that is in practice.
As we don't even batch these today, we could just do:
mm/mprotect.c +3 -4
@@ -127,10 +127,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,bool toptier;int nid; -/* Avoid TLB flush if possible */ -if (pte_protnone (oldpte)) - goto skip; -if (!folio) goto skip;
@@ -304,6 +300,9 @@ static long change_pte_range(struct mmu_gather *tlb,struct page *page;pte_t ptent; +if (prot_numa &&pte_protnone (oldpte)) +continue ; + page =vm_normal_folio (vma, addr, oldpte);if (page) folio =page_folio (page);
But with my change, we could actually batch-skip such large folios, because mprotect_folio_pte_batch() would see the folio.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to Dev JainQuoted 47 lines · Dev Jain
Quoted 36 lines · David Hildenbrand
Quoted 27 lines · Lorenzo Stoakes
Quoted 19 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 8 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does
exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Makes sense, so I guess I will have to change the signature of
prot_numa_skip()Quoted 6 lines
to pass a double ptr to a page instead of folio and derive the folio in the
caller,and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.Quoted 7 lines
I already don't love how we psas the folio back from there for very dubious
benefit. I really hate the idea of having a struct **page parameter...I wonder if we should just have a quick fixup for hotfix, and refine this more
later?Quoted 5 lines
This is not an issue in any released kernel, so we can do this properly.
We should just remove that nested
vm_normal_folio().Untested, but should give an idea what we can do.
This puts the overhead of
vm_normal_folio()unconditionally into thepte_presentpath.
Quoted 7 lines
Although I am guessing that is already happening assuming
prot_numacase is not thehot path. This is fine by me. So I guess I shouldn't have done that "reuse the folio
from
prot_numacase if possible" thingy at all :)
Don't worry it was a reasonable suggestion at the time, I think let's just make our lives easier here :)
Can you check David's patch and make sure it's ok? It looks reasonable to me (other than the eobvious typo I pointed out).
Re: [PATCH v5 6/7] mm: Optimize mprotect() by PTE batching
In reply to David HildenbrandQuoted 102 lines · David Hildenbrand
Quoted 22 lines · Lorenzo Stoakes
Quoted 21 lines · Dev Jain
Quoted 7 lines
Quoted 7 lines
You mean in _this_ PTE of the batch right? As we're invoking these on each part
of the PTE table.I mean I guess we can simply do:
Quoted 7 lines
Quoted 3 lines
struct page
*first_page=pte_page(ptent);Right?
Yes, but we should forward the result from
vm_normal_page(), which does exactly that for you, and increment the page accordingly as required, just like with the pte we are processing.Makes sense, so I guess I will have to change the signature of
prot_numa_skip()to pass a double ptr to a page instead of folio and derive the folio in the caller,
Quoted 8 lines
Quoted 2 lines
and pass down both the folio and the page to
set_write_prot_commit_flush_ptes.I already don't love how we psas the folio back from there for very dubious benefit. I really hate the idea of having a struct **page parameter...
I wonder if we should just have a quick fixup for hotfix, and refine this more later?
This is not an issue in any released kernel, so we can do this properly.
We should just remove that nested
vm_normal_folio().Untested, but should give an idea what we can do.
mm/mprotect.c +5
-9@@ -120,7 +120,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,static bool prot_numa_skip (struct vm_area_struct *vma,unsigned long addr,pte_t oldpte,pte_t *pte,int target_node, -struct folio **foliop) +struct folio *folio) {struct folio *folio =NULL ;bool ret = true;@@ -131,7 +131,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,if (pte_protnone (oldpte)) goto skip; - folio =vm_normal_folio (vma, addr, oldpte);if (!folio) goto skip;@@ -172,8 +171,6 @@ static bool prot_numa_skip(struct vm_area_struct *vma, unsigned long addr,if (folio_use_access_time (folio))folio_xchg_access_time (folio,jiffies_to_msecs (jiffies)); -skip: - *foliop = folio;return ret; }@@ -231,10 +228,9 @@ static int page_anon_exclusive_sub_batch(int start_idx, int max_len,* retrieve sub-batches. */ static void commit_anon_folio_batch (struct vm_area_struct *vma, -struct folio *folio,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *first_page,unsigned long addr,pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) { -struct page *first_page =folio_page (folio,0 );bool expected_anon_exclusive;int sub_batch_idx =0 ;int len;@@ -243,7 +239,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma, expected_anon_exclusive =PageAnonExclusive (first_page + sub_batch_idx); len =page_anon_exclusive_sub_batch (sub_batch_idx, nr_ptes, first_page, expected_anon_exclusive); -prot_commit_flush_ptes (vma, addr, ptep, oldpte, ptent, len, +prot_commit_flush_ptes (vma, addr, ptep, page, oldpte, ptent, len, sub_batch_idx, expected_anon_exclusive, tlb); sub_batch_idx += len; nr_ptes -= len;@@ -251,7 +247,7 @@ static void commit_anon_folio_batch(struct vm_area_struct *vma, }static void set_write_prot_commit_flush_ptes (struct vm_area_struct *vma, -struct folio *folio,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *page,unsigned long addr,pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) {bool set_write;@@ -270,7 +266,7 @@ static void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, /* idx = */ 0, set_write, tlb);return ; } -commit_anon_folio_batch (vma, folio, addr, ptep, oldpte, ptent, nr_ptes, tlb); +commit_anon_folio_batch (vma, folio, page, addr, ptep, oldpte, ptent, nr_ptes, tlb); } static long change_pte_range(struct mmu_gather *tlb,
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 306
Quoted 6 lines
mm/mprotect.c +2
-0@@ -305,15 +301,20 @@ static long change_pte_range(struct mmu_gather *tlb,const fpb_t flags =FPB_RESPECT_SOFT_DIRTY |FPB_RESPECT_WRITE ;int max_nr_ptes = (end - addr) >>PAGE_SHIFT ;struct folio *folio =NULL ; +struct page *page;pte_t ptent; + page =vm_normal_folio (vma, addr, oldpte);
Surely vm_normal_page()? :P
Quoted 14 lines
mm/mprotect.c +4
-1+if (page) + folio =page_folio (page); +/* * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd. */ if (prot_numa) {int ret =prot_numa_skip (vma, addr, oldpte, pte, - target_node, &folio); + target_node, folio);if (ret) {/* determine batch to skip */
mm/mprotect.c static long change_pte_range(struct mmu_gather *tlb, · line 327
Quoted 5 lines
mm/mprotect.c +0
-3@@ -323,9 +324,6 @@ static long change_pte_range(struct mmu_gather *tlb, } } -if (!folio) - folio =vm_normal_folio (vma, addr, oldpte); -
Yes :) thanks :>)
Quoted 20 lines
mm/mprotect.c +1
-2nr_ptes =mprotect_folio_pte_batch (folio, pte, oldpte, max_nr_ptes, flags); oldpte =modify_prot_start_ptes (vma, addr, pte, nr_ptes);@@ -351,7 +349,7 @@ static long change_pte_range(struct mmu_gather *tlb,*/ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pte_write (ptent)) -set_write_prot_commit_flush_ptes (vma, folio, +set_write_prot_commit_flush_ptes (vma, folio, page, addr, pte, oldpte, ptent, nr_ptes, tlb);else prot_commit_flush_ptes(vma, addr, pte, oldpte, ptent, --2.50.1
--
Cheers,David / dhildenb
PATCH v5 7/7 arm64: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainOverride the generic definition of modify_prot_start_ptes() to use get_and_clear_full_ptes(). This helper does a TLBI only for the starting and ending contpte block of the range, whereas the current implementation will call ptep_get_and_clear() for every contpte block, thus doing a TLBI on every contpte block. Therefore, we have a performance win.
The arm64 definition of pte_accessible() allows us to batch in the errata specific case:
#define pte_accessible(mm, pte) \
(mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid(pte))
All ptes are obviously present in the folio batch, and they are also valid.
Override the generic definition of modify_prot_commit_ptes() to simply use set_ptes() to map the new ptes into the pagetable.
--- arch/arm64/include/asm/pgtable.h | 10 ++++++++++ arch/arm64/mm/mmu.c | 28 +++++++++++++++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-)
arch/arm64/include/asm/pgtable.h +10 -0
@@ -1643,6 +1643,16 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep, +unsigned int nr); + +#define modify_prot_commit_ptes modify_prot_commit_ptes +extern void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte, +unsigned int nr); +#ifdef CONFIG_ARM64_CONTPTE /*
arch/arm64/mm/mmu.c +23 -5
@@ -26,6 +26,7 @@#include <linux/set_memory.h> #include <linux/kfence.h> #include <linux/pkeys.h> +#include <linux/mm_inline.h> #include <asm/barrier.h> #include <asm/cputype.h>
@@ -1524,24 +1525,41 @@ static int __init prevent_bootmem_remove_init(void)early_initcall (prevent_bootmem_remove_init);#endif -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +pte_t modify_prot_start_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,unsigned int nr) { +pte_t pte =get_and_clear_full_ptes (vma->vm_mm, addr, ptep, nr,/* full = */ 0 ); +if (alternative_has_cap_unlikely (ARM64_WORKAROUND_2645198 )) {/* * Break-before-make (BBM) is required for all user space mappings * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ -if (pte_user_exec (ptep_get (ptep))) -return ptep_clear_flush (vma, addr, ptep); +if (pte_accessible (vma->vm_mm, pte) &&pte_user_exec (pte)) +__flush_tlb_range (vma, addr, nr *PAGE_SIZE , +PAGE_SIZE , true,3 ); } -return ptep_get_and_clear (vma->vm_mm, addr, ptep); + +return pte; +} + +pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +{ +return modify_prot_start_ptes (vma, addr, ptep,1 ); +} + +void modify_prot_commit_ptes (struct vm_area_struct *vma,unsigned long addr, +pte_t *ptep,pte_t old_pte,pte_t pte, +unsigned int nr) +{ +set_ptes (vma->vm_mm, addr, ptep, pte, nr); }void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t pte) { -set_pte_at (vma->vm_mm, addr, ptep, pte); +modify_prot_commit_ptes (vma, addr, ptep, old_pte, pte,1 ); }/*
Signature · 1 line
2.30.2
Re: [PATCH v5 7/7] arm64: Add batched versions of ptep_modify_prot_start/commit
Note I'm not forgetting this patch, it's just that I see Ryan has reviewed it, and on the arm64 side I defer to his expertise.
Cheers, Lorenzo
Re: [PATCH v5 7/7] arm64: Add batched versions of ptep_modify_prot_start/commit
In reply to Dev JainQuoted 12 lines · Dev Jain
Override the generic definition of
modify_prot_start_ptes()to useget_and_clear_full_ptes(). This helper does a TLBI only for the starting and ending contpte block of the range, whereas the current implementation will callptep_get_and_clear()for every contpte block, thus doing a TLBI on every contpte block. Therefore, we have a performance win.The arm64 definition of
pte_accessible()allows us to batch in the errata specific case:#define
pte_accessible(mm, pte)\
(mm_tlb_flush_pending(mm)?pte_present(pte):pte_valid(pte))
Quoted 7 lines
All ptes are obviously present in the folio batch, and they are also valid.
Override the generic definition of
modify_prot_commit_ptes()to simply useset_ptes()to map the new ptes into the pagetable.Reviewed-by Ryan Roberts <[email protected]>Signed-off-by Dev Jain <[email protected]>
The arm64 changes look fine to me:
Quoted 27 lines · Dev Jain
Use
folio_pte_batch()to optimizechange_pte_range(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_modify_prot_start()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.For split folios, there will be no pte batching; the batch size returned by
folio_pte_batch()will be 1. For pagetable split folios, the ptes will still point to the same large folio; for arm64, this results in the optimization described above, and for other arches, a minor improvement is expected due to a reduction in the number of function calls.mm-selftests pass on arm64. I have some failing tests on my x86 VM already; no new tests fail as a result of this patchset.
We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pages
Quoted 7 lines
Average execution time on arm64, Apple M3:
Before the patchset:
T1: 2.1 seconds T2: 2 seconds T3: 1 secondAfter the patchset:
T1: 0.65 seconds T2: 0.7 seconds T3: 1.1 seconds
For the note: the numbers are different from the previous versions. I must have run the test for more number of iterations and then pasted the test program here for 40 iterations, that's why the mismatch.
Quoted 33 lines · Dev Jain
Quoted 31 lines · Dev Jain
Use
folio_pte_batch()to optimizechange_pte_range(). On arm64, if the ptes are painted with the contig bit, thenptep_get()will iterate through all 16 entries to collect a/d bits. Hence this optimization will result in a 16x reduction in the number ofptep_get()calls. Next,ptep_modify_prot_start()will eventually callcontpte_try_unfold()on every contig block, thus flushing the TLB for the complete large folio range. Instead, useget_and_clear_full_ptes()so as to elide TLBIs on each contig block, and only do them on the starting and ending contig block.For split folios, there will be no pte batching; the batch size returned by
folio_pte_batch()will be 1. For pagetable split folios, the ptes will still point to the same large folio; for arm64, this results in the optimization described above, and for other arches, a minor improvement is expected due to a reduction in the number of function calls.mm-selftests pass on arm64. I have some failing tests on my x86 VM already; no new tests fail as a result of this patchset.
We use the following test cases to measure performance, mprotect()'ing the mapped memory to read-only then read-write 40 times:
Test case 1: Mapping 1G of memory, touching it to get PMD-THPs, then pte-mapping those THPs
Test case 2: Mapping 1G of memory with 64K mTHPs
Test case 3: Mapping 1G of memory with 4K pagesAverage execution time on arm64, Apple M3:
Before the patchset:
T1: 2.1 seconds T2: 2 seconds T3: 1 second
Quoted 8 lines
Quoted 3 lines
After the patchset:
T1: 0.65 seconds T2: 0.7 seconds T3: 1.1 secondsFor the note: the numbers are different from the previous versions. I must have run the test for more number of iterations and then pasted the test program here for 40 iterations, that's why the mismatch.
Thanks for this clarification!