mm/mprotect: do not flush when not required architecturally
Subsystems (8)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH RESEND v5 2/3 mm/mprotect: do not flush when not required architecturally
From: Nadav Amit <[email protected]>
Currently, using mprotect() to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. However, in such cases the PTE is often not modified (i.e., remain RO) and therefore not TLB flush is needed.
Add an arch-specific pte_needs_flush() which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86 pte_needs_flush().
Always flush the TLB when it is architecturally needed even when skipping a TLB flush might only result in a spurious page-faults by skipping the flush.
--- arch/x86/include/asm/pgtable_types.h | 2 + arch/x86/include/asm/tlbflush.h | 121 +++++++++++++++++++++++++++ include/asm-generic/tlb.h | 14 ++++ mm/huge_memory.c | 9 +- mm/mprotect.c | 3 +- 5 files changed, 144 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
@@ -110,9 +110,11 @@#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_NX (_AT(pteval_t, 1) << _PAGE_BIT_NX)#define _PAGE_DEVMAP (_AT(u64, 1) << _PAGE_BIT_DEVMAP) +#define _PAGE_SOFTW4 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW4)#else #define _PAGE_NX (_AT(pteval_t, 0))#define _PAGE_DEVMAP (_AT(pteval_t, 0)) +#define _PAGE_SOFTW4 (_AT(pteval_t, 0))#endif #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)
arch/x86/include/asm/tlbflush.h +121 -0
127 lines not on this page · raw message
include/asm-generic/tlb.h +14 -0
@@ -654,6 +654,20 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, } while (0)#endif +#ifndef pte_needs_flush +static inline bool pte_needs_flush (pte_t oldpte,pte_t newpte) +{ +return true; +} +#endif + +#ifndef huge_pmd_needs_flush +static inline bool huge_pmd_needs_flush (pmd_t oldpmd,pmd_t newpmd) +{ +return true; +} +#endif +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */
mm/huge_memory.c +5 -4
@@ -1698,7 +1698,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl; -pmd_t entry; +pmd_t oldpmd, entry;bool preserve_write;int ret;bool prot_numa = cp_flags &MM_CP_PROT_NUMA ;
@@ -1784,9 +1784,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,* pmdp_invalidate() is required to make sure we don't miss * dirty/young flags set by hardware. */ - entry =pmdp_invalidate (vma, addr, pmd); + oldpmd =pmdp_invalidate (vma, addr, pmd); - entry =pmd_modify (entry, newprot); + entry =pmd_modify (oldpmd, newprot);if (preserve_write) entry =pmd_mk_savedwrite (entry);if (uffd_wp) {
@@ -1803,7 +1803,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); -tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE ); +if (huge_pmd_needs_flush (oldpmd, entry)) +tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE );BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:
mm/mprotect.c +2 -1
@@ -152,7 +152,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_needs_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);
Signature · 1 line
2.25.1
Quoted 11 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, using
mprotect()to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. However, in such cases the PTE is often not modified (i.e., remain RO) and therefore not TLB flush is needed.
Quoted 7 lines
Add an arch-specific
pte_needs_flush()which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86pte_needs_flush().Always flush the TLB when it is architecturally needed even when skipping a TLB flush might only result in a spurious page-faults by skipping the flush.
Errr.. Sent the wrong version. Will resend with new version number.
PATCH RESEND v5 3/3 mm: avoid unnecessary flush on change_huge_pmd()
In reply to a message not in the archiveFrom: Nadav Amit <[email protected]>
Calls to change_protection_range() on THP can trigger, at least on x86, two TLB flushes for one page: one immediately, when pmdp_invalidate() is called by change_huge_pmd(), and then another one later (that can be batched) when change_protection_range() finishes.
The first TLB flush is only necessary to prevent the dirty bit (and with a lesser importance the access bit) from changing while the PTE is modified. However, this is not necessary as the x86 CPUs set the dirty-bit atomically with an additional check that the PTE is (still) present. One caveat is Intel's Knights Landing that has a bug and does not do so.
Leverage this behavior to eliminate the unnecessary TLB flush in change_huge_pmd(). Introduce a new arch specific pmdp_invalidate_ad() that only invalidates the access and dirty bit from further changes.
--- arch/x86/include/asm/pgtable.h | 5 +++++ arch/x86/mm/pgtable.c | 10 ++++++++++ include/linux/pgtable.h | 20 ++++++++++++++++++++ mm/huge_memory.c | 4 ++-- mm/pgtable-generic.c | 8 ++++++++ 5 files changed, 45 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable.h +5 -0
@@ -1173,6 +1173,11 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } }#endif + +#define __HAVE_ARCH_PMDP_INVALIDATE_AD +extern pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); +/* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel.
arch/x86/mm/pgtable.c +10 -0
@@ -608,6 +608,16 @@ int pmdp_clear_flush_young(struct vm_area_struct *vma, return young; } + +pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp) +{ +/* +* No flush is necessary. Once an invalid PTE is established, the PTE's +* access and dirty bits cannot be updated. +*/ +return pmdp_establish (vma, address, pmdp,pmd_mkinvalid (*pmdp)); +}#endif /**
include/linux/pgtable.h +20 -0
@@ -570,6 +570,26 @@ extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);#endif +#ifndef __HAVE_ARCH_PMDP_INVALIDATE_AD + +/* +* pmdp_invalidate_ad() invalidates the PMD while changing a transparent +* hugepage mapping in the page tables. This function is similar to +* pmdp_invalidate(), but should only be used if the access and dirty bits would +* not be cleared by the software in the new PMD value. The function ensures +* that hardware changes of the access and dirty bits updates would not be lost. +* +* Doing so can allow in certain architectures to avoid a TLB flush in most +* cases. Yet, another TLB flush might be necessary later if the PMD update +* itself requires such flush (e.g., if protection was set to be stricter). Yet, +* even when a TLB flush is needed because of the update, the caller may be able +* to batch these TLB flushing operations, so fewer TLB flush operations are +* needed. +*/ +extern pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma, +unsigned long address,pmd_t *pmdp); +#endif +#ifndef __HAVE_ARCH_PTE_SAMEstatic inline int pte_same (pte_t pte_a,pte_t pte_b) {
mm/huge_memory.c +2 -2
@@ -1781,10 +1781,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,* The race makes MADV_DONTNEED miss the huge pmd and don't clear it * which may break userspace. * -* pmdp_invalidate() is required to make sure we don't miss +* pmdp_invalidate_ad() is required to make sure we don't miss * dirty/young flags set by hardware. */ - oldpmd =pmdp_invalidate (vma, addr, pmd); + oldpmd =pmdp_invalidate_ad (vma, addr, pmd); entry =pmd_modify (oldpmd, newprot);if (preserve_write)
mm/pgtable-generic.c +8 -0
@@ -201,6 +201,14 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, }#endif +#ifndef __HAVE_ARCH_PMDP_INVALIDATE_AD +pmd_t pmdp_invalidate_ad (struct vm_area_struct *vma,unsigned long address, +pmd_t *pmdp) +{ +return pmdp_invalidate (vma, address, pmdp); +} +#endif +#ifndef pmdp_collapse_flushpmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
Signature · 1 line
2.25.1