mm/mprotect: avoid unnecessary TLB flushes
Subsystems (8)
v6 cover mm/mprotect: avoid unnecessary TLB flushes
From: Nadav Amit <[email protected]>
This patch-set is intended to remove unnecessary TLB flushes during mprotect() syscalls. Once this patch-set make it through, similar and further optimizations for MADV_COLD and userfaultfd would be possible.
Basically, there are 3 optimizations in this patch-set:
1. Use TLB batching infrastructure to batch flushes across VMAs and do better/fewer flushes. This would also be handy for later userfaultfd enhancements.
2. Avoid unnecessary TLB flushes. This optimization is the one that provides most of the performance benefits. Unlike previous versions, we now only avoid flushes that would not result in spurious page-faults.
3. Avoiding TLB flushes on change_huge_pmd() that are only needed to prevent the A/D bits from changing.
Andrew asked for some benchmark numbers. I do not have an easy determinate macrobenchmark in which it is easy to show benefit. I therre ran a microbenchmark: a loop that does the following on anonymous memory, just as a sanity check to see that time is saved by avoiding TLB flushes. The loop goes:
mprotect(p, PAGE_SIZE, PROT_READ) mprotect(p, PAGE_SIZE, PROT_READ|PROT_WRITE) *p = 0; // make the page writable
The test was run in KVM guest with 1 or 2 threads (the second thread was busy-looping). I measured the time (cycles) of each operation:
1 thread 2 threads
mmots +patch mmots +patchPROT_READ 3494 2725 (-22%) 8630 7788 (-10%)
PROT_READ|WRITE 3952 2724 (-31%) 9075 2865 (-68%)
[ mmots = v5.17-rc6-mmots-2022-03-06-20-38 ]
The exact numbers are really meaningless, but the benefit is clear. There are 2 interesting results though.
(1) PROT_READ is cheaper, while one can expect it not to be affected. This is presumably due to TLB miss that is saved
(2) Without memory access (*p = 0), the speedup of the patch is even greater. In that scenario mprotect(PROT_READ) also avoids the TLB flush. As a result both operations on the patched kernel take roughly ~1500 cycles (with either 1 or 2 threads), whereas on mmotm their cost is as high as presented in the table.
--
v5 -> v6:
* Wrong patch 2 was sent on v5
v4 -> v5:
* Avoid only TLB flushes that would not result in spurious PF [Dave]
* Better comments, names in pte_flags_need_flush() [Dave]
v3 -> v4:
* Remove KNL-related stuff [Dave]
* Check error code sanity on every PF [Dave]
* Reduce nesting, simplify access_error() changes [Dave]
* Remove redundant present->non-present check
* Use break instead of goto in do_mprotect_pkey()
* Add missing change_prot_numa() chunk
v2 -> v3:
* Fix orders of patches (order could lead to breakage)
* Better comments
* Clearer KNL detection [Dave]
* Assertion on PF error-code [Dave]
* Comments, code, function names improvements [PeterZ]
* Flush on access-bit clearing on PMD changes to follow the way flushing on x86 is done today in the kernel.
v1 -> v2:
* Wrong detection of permission demotion [Andrea]
* Better comments [Andrea]
* Handle THP [Andrea]
* Batching across VMAs [Peter Xu]
* Avoid open-coding PTE analysis
* Fix wrong use of the mmu_gather()
Nadav Amit (3):
mm/mprotect: use mmu_gather
mm/mprotect: do not flush when not required architecturally
mm: avoid unnecessary flush on change_huge_pmd()
arch/x86/include/asm/pgtable.h | 5 ++ arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 121 +++++++++++++++++++++++++++ arch/x86/mm/pgtable.c | 10 +++ fs/exec.c | 6 +- include/asm-generic/tlb.h | 14 ++++ include/linux/huge_mm.h | 5 +- include/linux/mm.h | 5 +- include/linux/pgtable.h | 20 +++++ mm/huge_memory.c | 19 +++-- mm/mempolicy.c | 9 +- mm/mprotect.c | 93 ++++++++++---------- mm/pgtable-generic.c | 8 ++ mm/userfaultfd.c | 6 +- 14 files changed, 268 insertions(+), 55 deletions(-)
Signature · 1 line
2.25.1
PATCH v6 1/3 mm/mprotect: use mmu_gather
From: Nadav Amit <[email protected]>
change_pXX_range() currently does not use mmu_gather, but instead implements its own deferred TLB flushes scheme. This both complicates the code, as developers need to be aware of different invalidation schemes, and prevents opportunities to avoid TLB flushes or perform them in finer granularity.
The use of mmu_gather for modified PTEs has benefits in various scenarios even if pages are not released. For instance, if only a single page needs to be flushed out of a range of many pages, only that page would be flushed. If a THP page is flushed, on x86 a single TLB invlpg instruction can be used instead of 512 instructions (or a full TLB flush, which would Linux would actually use by default). mprotect() over multiple VMAs requires a single flush.
Use mmu_gather in change_pXX_range(). As the pages are not released, only record the flushed range using tlb_flush_pXX_range().
Handle THP similarly and get rid of flush_cache_range() which becomes redundant since tlb_start_vma() calls it when needed.
--- fs/exec.c | 6 ++- include/linux/huge_mm.h | 5 ++- include/linux/mm.h | 5 ++- mm/huge_memory.c | 10 ++++- mm/mempolicy.c | 9 +++- mm/mprotect.c | 92 ++++++++++++++++++++++------------------- mm/userfaultfd.c | 6 ++- 7 files changed, 82 insertions(+), 51 deletions(-)
fs/exec.c +5 -1
@@ -758,6 +758,7 @@ int setup_arg_pages(struct linux_binprm *bprm,unsigned long stack_size;unsigned long stack_expand;unsigned long rlim_stack; +struct mmu_gather tlb;#ifdef CONFIG_STACK_GROWSUP /* Limit stack size */
@@ -812,8 +813,11 @@ int setup_arg_pages(struct linux_binprm *bprm, vm_flags |= mm->def_flags; vm_flags |=VM_STACK_INCOMPLETE_SETUP ; - ret =mprotect_fixup (vma, &prev, vma->vm_start, vma->vm_end, +tlb_gather_mmu (&tlb, mm); + ret =mprotect_fixup (&tlb, vma, &prev, vma->vm_start, vma->vm_end, vm_flags); +tlb_finish_mmu (&tlb); +if (ret) goto out_unlock;BUG_ON (prev != vma);
include/linux/huge_mm.h +3 -2
@@ -36,8 +36,9 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr);bool move_huge_pmd (struct vm_area_struct *vma,unsigned long old_addr,unsigned long new_addr,pmd_t *old_pmd,pmd_t *new_pmd); -int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr, -pgprot_t newprot,unsigned long cp_flags); +int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags);vm_fault_t vmf_insert_pfn_pmd_prot (struct vm_fault *vmf,pfn_t pfn,pgprot_t pgprot,bool write);
include/linux/mm.h +3 -2
@@ -1970,10 +1970,11 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,#define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) -extern unsigned long change_protection (struct vm_area_struct *vma,unsigned long start, +extern unsigned long change_protection (struct mmu_gather *tlb, +struct vm_area_struct *vma,unsigned long start,unsigned long end,pgprot_t newprot,unsigned long cp_flags); -extern int mprotect_fixup (struct vm_area_struct *vma, +extern int mprotect_fixup (struct mmu_gather *tlb,struct vm_area_struct *vma,struct vm_area_struct **pprev,unsigned long start,unsigned long end,unsigned long newflags);
mm/huge_memory.c +8 -2
@@ -1709,8 +1709,9 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,* or if prot_numa but THP migration is not supported * - HPAGE_PMD_NR if protections changed and TLB flush necessary */ -int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long addr,pgprot_t newprot,unsigned long cp_flags) +int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma, +pmd_t *pmd,unsigned long addr,pgprot_t newprot, +unsigned long cp_flags) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;
@@ -1721,6 +1722,8 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,bool uffd_wp = cp_flags &MM_CP_UFFD_WP ;bool uffd_wp_resolve = cp_flags &MM_CP_UFFD_WP_RESOLVE ; +tlb_change_page_size (tlb,HPAGE_PMD_SIZE ); +if (prot_numa && !thp_migration_supported ())return 1 ;
@@ -1819,6 +1822,9 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, } ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); + +tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE ); +BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:spin_unlock (ptl);
mm/mempolicy.c +8 -1
@@ -104,6 +104,7 @@#include <linux/swapops.h> #include <asm/tlbflush.h> +#include <asm/tlb.h> #include <linux/uaccess.h> #include "internal.h"
@@ -636,12 +637,18 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,unsigned long change_prot_numa (struct vm_area_struct *vma,unsigned long addr,unsigned long end) { +struct mmu_gather tlb;int nr_updated; - nr_updated =change_protection (vma, addr, end,PAGE_NONE ,MM_CP_PROT_NUMA ); +tlb_gather_mmu (&tlb, vma->vm_mm); + + nr_updated =change_protection (&tlb, vma, addr, end,PAGE_NONE , +MM_CP_PROT_NUMA );if (nr_updated)count_vm_numa_events (NUMA_PTE_UPDATES , nr_updated); +tlb_finish_mmu (&tlb); +return nr_updated; }#else
mm/mprotect.c +50 -42
247 lines not on this page · raw message
mm/userfaultfd.c +5 -1
@@ -16,6 +16,7 @@#include <linux/hugetlb.h> #include <linux/shmem_fs.h> #include <asm/tlbflush.h> +#include <asm/tlb.h> #include "internal.h" static __always_inline
@@ -687,6 +688,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, atomic_t *mmap_changing) {struct vm_area_struct *dst_vma; +struct mmu_gather tlb;pgprot_t newprot;int err;
@@ -728,8 +730,10 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start,else newprot =vm_get_page_prot (dst_vma->vm_flags); -change_protection (dst_vma, start, start + len, newprot, +tlb_gather_mmu (&tlb, dst_mm); +change_protection (&tlb, dst_vma, start, start + len, newprot, enable_wp ?MM_CP_UFFD_WP :MM_CP_UFFD_WP_RESOLVE ); +tlb_finish_mmu (&tlb); err =0 ; out_unlock:
Signature · 1 line
2.25.1