TLB batching consolidation and enhancements
Subsystems (30)
cover TLB batching consolidation and enhancements
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using mmu_gather (e.g., zap_page_range()).
2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g., change_protection_range()).
3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (move_ptes()).
5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (try_to_unmap_one() on x86).
It seems that (1)-(4) can be consolidated. In addition, it seems that
(5) is racy. It also seems there can be many redundant TLB flushes, and potentially TLB-shootdown storms, for instance during batched reclamation (using try_to_unmap_one()) if at the same time mmu_gather defers TLB flushes.
More aggressive TLB batching may be possible, but this patch-set does not add such batching. The proposed changes would enable such batching in a later time.
Admittedly, I do not understand how things are not broken today, which frightens me to make further batching before getting things in order. For instance, why is ok for zap_pte_range() to batch dirty-PTE flushes for each page-table (but not in greater granularity). Can't ClearPageDirty() be called before the flush, causing writes after ClearPageDirty() and before the flush to be lost?
This patch-set therefore performs the following changes:
1. Change mprotect, task_mmu and mapping_dirty_helpers to use mmu_gather instead of {inc|dec}_tlb_flush_pending().
2. Avoid TLB flushes if PTE permission is not demoted.
3. Cleans up mmu_gather to be less arch-dependant.
4. Uses mm's generations to track in finer granularity, either per-VMA or per page-table, whether a pending mmu_gather operation is outstanding. This should allow to avoid some TLB flushes when KSM or memory reclamation takes place while another operation such as munmap() or mprotect() is running.
5. Changes try_to_unmap_one() flushing scheme, as the current seems broken to track in a bitmap which CPUs have outstanding TLB flushes instead of having a flag.
Further optimizations are possible, such as changing move_ptes() to use mmu_gather.
The patches were very very lightly tested. I am looking forward for your feedback regarding the overall approaches, and whether to split them into multiple patch-sets.
Nadav Amit (20):
mm/tlb: fix fullmm semantics
mm/mprotect: use mmu_gather
mm/mprotect: do not flush on permission promotion mm/mapping_dirty_helpers: use mmu_gather
mm/tlb: move BATCHED_UNMAP_TLB_FLUSH to tlb.h fs/task_mmu: use mmu_gather interface of clear-soft-dirty
mm: move x86 tlb_gen to generic code
mm: store completed TLB generation
mm: create pte/pmd_tlb_flush_pending()
mm: add pte_to_page()
mm/tlb: remove arch-specific tlb_start/end_vma() mm/tlb: save the VMA that is flushed during tlb_start_vma() mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
mm: move inc/dec_tlb_flush_pending() to mmu_gather.c
mm: detect deferred TLB flushes in vma granularity mm/tlb: per-page table generation tracking mm/tlb: updated completed deferred TLB flush conditionally
mm: make mm_cpumask() volatile
lib/cpumask: introduce cpumask_atomic_or() mm/rmap: avoid potential races
arch/arm/include/asm/bitops.h | 4 +- arch/arm/include/asm/pgtable.h | 4 +- arch/arm64/include/asm/pgtable.h | 4 +- arch/csky/Kconfig | 1 + arch/csky/include/asm/tlb.h | 12 -- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/tlb.h | 2 - arch/s390/Kconfig | 1 + arch/s390/include/asm/tlb.h | 3 - arch/sparc/Kconfig | 1 + arch/sparc/include/asm/pgtable_64.h | 9 +- arch/sparc/include/asm/tlb_64.h | 2 - arch/sparc/mm/init_64.c | 2 +- arch/x86/Kconfig | 3 + arch/x86/hyperv/mmu.c | 2 +- arch/x86/include/asm/mmu.h | 10 - arch/x86/include/asm/mmu_context.h | 1 - arch/x86/include/asm/paravirt_types.h | 2 +- arch/x86/include/asm/pgtable.h | 24 +-- arch/x86/include/asm/tlb.h | 21 +- arch/x86/include/asm/tlbbatch.h | 15 -- arch/x86/include/asm/tlbflush.h | 61 ++++-- arch/x86/mm/tlb.c | 52 +++-- arch/x86/xen/mmu_pv.c | 2 +- drivers/firmware/efi/efi.c | 1 + fs/proc/task_mmu.c | 29 ++- include/asm-generic/bitops/find.h | 8 +- include/asm-generic/tlb.h | 291 +++++++++++++++++++++----- include/linux/bitmap.h | 21 +- include/linux/cpumask.h | 40 ++-- include/linux/huge_mm.h | 3 +- include/linux/mm.h | 29 ++- include/linux/mm_types.h | 166 ++++++++++----- include/linux/mm_types_task.h | 13 -- include/linux/pgtable.h | 2 +- include/linux/smp.h | 6 +- init/Kconfig | 21 ++ kernel/fork.c | 2 + kernel/smp.c | 8 +- lib/bitmap.c | 33 ++- lib/cpumask.c | 8 +- lib/find_bit.c | 10 +- mm/huge_memory.c | 6 +- mm/init-mm.c | 1 + mm/internal.h | 16 -- mm/ksm.c | 2 +- mm/madvise.c | 6 +- mm/mapping_dirty_helpers.c | 52 +++-- mm/memory.c | 2 + mm/mmap.c | 1 + mm/mmu_gather.c | 59 +++++- mm/mprotect.c | 55 ++--- mm/mremap.c | 2 +- mm/pgtable-generic.c | 2 +- mm/rmap.c | 42 ++-- mm/vmscan.c | 1 + 56 files changed, 803 insertions(+), 374 deletions(-) delete mode 100644 arch/x86/include/asm/tlbbatch.h
Signature · 1 line
2.25.1
RFC 1/20 mm/tlb: fix fullmm semantics
From: Nadav Amit <[email protected]>
fullmm in mmu_gather is supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However, tlb_finish_mmu() sets fullmm, when in fact it want to say that the TLB should be fully flushed.
Change tlb_finish_mmu() to set need_flush_all and check this flag in tlb_flush_mmu_tlbonly() when deciding whether a flush is needed.
--- include/asm-generic/tlb.h | 2 +- mm/mmu_gather.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/asm-generic/tlb.h +1 -1
@@ -420,7 +420,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)* these bits. */ if (!(tlb->freed_tables || tlb->cleared_ptes || tlb->cleared_pmds || - tlb->cleared_puds || tlb->cleared_p4ds)) + tlb->cleared_puds || tlb->cleared_p4ds || tlb->need_flush_all))return ;tlb_flush (tlb);
mm/mmu_gather.c +1 -1
@@ -323,7 +323,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb)* On x86 non-fullmm doesn't yield significant difference * against fullmm. */ - tlb->fullmm =1 ; + tlb->need_flush_all =1 ;__tlb_reset_range (tlb); tlb->freed_tables =1 ; }
Signature · 1 line
2.25.1
Andy Lutomirski
maintainerRe: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 7 lines · Nadav Amit
From: Nadav Amit <[email protected]>
fullmm in
mmu_gatheris supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However,tlb_finish_mmu()sets fullmm, when in fact it want to say that the TLB should be fully flushed.
Maybe also rename fullmm?
Re: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 11 lines · Andy Lutomirski
Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
fullmm in
mmu_gatheris supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However,tlb_finish_mmu()sets fullmm, when in fact it want to say that the TLB should be fully flushed.Maybe also rename fullmm?
Possible. How about mm_torn_down?
I should have also changed the comment in tlb_finish_mmu().
Andy Lutomirski
maintainerRe: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 6 lines · Nadav Amit
Quoted 6 lines · Andy Lutomirski
Quoted 2 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
fullmm in
mmu_gatheris supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However,tlb_finish_mmu()sets fullmm, when in fact it want to say that the TLB should be fully flushed.Maybe also rename fullmm?
Possible. How about
mm_torn_down?
Sure. Or mm_exiting, perhaps?
Quoted 2 lines
I should have also changed the comment in
tlb_finish_mmu().
Re: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 13 lines · Andy Lutomirski
Quoted 10 lines · Nadav Amit
Quoted 10 lines · Andy Lutomirski
Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
fullmm in
mmu_gatheris supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However,tlb_finish_mmu()sets fullmm, when in fact it want to say that the TLB should be fully flushed.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Maybe also rename fullmm?
Possible. How about
mm_torn_down?Sure. Or
mm_exiting, perhaps?
mm_exiting indeed sounds better.
Peter Zijlstra
maintainerRe: [RFC 01/20] mm/tlb: fix fullmm semantics
In reply to Nadav AmitRe: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 4 lines · Peter Zijlstra
I have seen this series, and applied my patches on it.
Despite Will’s patches, there were still inconsistencies between fullmm and need_flush_all.
Am I missing something?
Peter Zijlstra
maintainerRe: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 11 lines · Nadav Amit
Quoted 4 lines · Peter Zijlstra
I have seen this series, and applied my patches on it.
Despite Will’s patches, there were still inconsistencies between fullmm and
need_flush_all.Am I missing something?
I wasn't aware you were on top. I'll look again.
Re: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Nadav Amit
Quoted 4 lines · Peter Zijlstra
Quoted 8 lines
Quoted 6 lines
I have seen this series, and applied my patches on it.
Despite Will’s patches, there were still inconsistencies between fullmm and
need_flush_all.Am I missing something?
I wasn't aware you were on top. I'll look again.
Looking on arm64’s tlb_flush() makes me think that there is currently a bug that this patch fixes. Arm64’s tlb_flush() does:
/*
* If we're tearing down the address space then we only care about
* invalidating the walk-cache, since the ASID allocator won't
* reallocate our ASID without invalidating the entire TLB.
*/
if (tlb->fullmm) {
if (!last_level)
flush_tlb_mm(tlb->mm);
return;
} But currently tlb_mmu_finish() can mistakenly set fullmm incorrectly (if mm_tlb_flush_nested() is true), which might skip the TLB flush.
Lucky for us, arm64 flushes each VMA separately (which as we discussed separately may not be necessary), so the only PTEs that might not be flushed are PTEs that are updated concurrently by another thread that also defer their flushes. It therefore seems that the implications are more on the correctness of certain syscalls (e.g., madvise(DONT_NEED)) without implications on security or memory corruptions.
Let me know if you want me to send this patch separately with an updated commit log for faster inclusion.
Will Deacon
maintainerRe: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 31 lines · Nadav Amit
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Nadav Amit
Quoted 4 lines · Peter Zijlstra
Quoted 8 lines
Quoted 6 lines
I have seen this series, and applied my patches on it.
Despite Will’s patches, there were still inconsistencies between fullmm and
need_flush_all.Am I missing something?
I wasn't aware you were on top. I'll look again.
Looking on arm64’s
tlb_flush()makes me think that there is currently a bug that this patch fixes. Arm64’stlb_flush()does:/* * If we're tearing down the address space then we only care about * invalidating the walk-cache, since the ASID allocator won't * reallocate our ASID without invalidating the entire TLB. */ if (tlb->fullmm) { if (!last_level) flush_tlb_mm(tlb->mm); return; }
Quoted 2 lines
But currently
tlb_mmu_finish()can mistakenly set fullmm incorrectly (ifmm_tlb_flush_nested()is true), which might skip the TLB flush.
But in that case isn't 'freed_tables' set to 1, so 'last_level' will be false and we'll do the flush in the code above?
Will
Re: [RFC 01/20] mm/tlb: fix fullmm semantics
Quoted 34 lines · Will Deacon
Quoted 31 lines · Nadav Amit
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Nadav Amit
Quoted 4 lines · Peter Zijlstra
Quoted 8 lines
Quoted 6 lines
I have seen this series, and applied my patches on it.
Despite Will’s patches, there were still inconsistencies between fullmm and
need_flush_all.Am I missing something?
I wasn't aware you were on top. I'll look again.
Looking on arm64’s
tlb_flush()makes me think that there is currently a bug that this patch fixes. Arm64’stlb_flush()does:/* * If we're tearing down the address space then we only care about * invalidating the walk-cache, since the ASID allocator won't * reallocate our ASID without invalidating the entire TLB. */ if (tlb->fullmm) { if (!last_level) flush_tlb_mm(tlb->mm); return; }
Quoted 5 lines
Quoted 2 lines
But currently
tlb_mmu_finish()can mistakenly set fullmm incorrectly (ifmm_tlb_flush_nested()is true), which might skip the TLB flush.But in that case isn't '
freed_tables' set to 1, so 'last_level' will be false and we'll do the flush in the code above?
Indeed. You are right. So no rush.
RFC 2/20 mm/mprotect: use mmu_gather
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
change_pXX_range() currently does not use mmu_gather, but instead implements its own deferred TLB flushes scheme. This both complicates the code, as developers need to be aware of different invalidation schemes, and prevents.
Use mmu_gather in change_pXX_range(). As the pages are not released, only record the flushed range using tlb_flush_pXX_range().
--- include/linux/huge_mm.h | 3 ++- mm/huge_memory.c | 4 +++- mm/mprotect.c | 51 ++++++++++++++++++++--------------------- 3 files changed, 30 insertions(+), 28 deletions(-)
include/linux/huge_mm.h +2 -1
@@ -37,7 +37,8 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud,bool move_huge_pmd (struct vm_area_struct *vma,unsigned long old_addr,unsigned long new_addr,pmd_t *old_pmd,pmd_t *new_pmd);int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr, -pgprot_t newprot,unsigned long cp_flags); +pgprot_t newprot,unsigned long cp_flags, +struct mmu_gather *tlb);vm_fault_t vmf_insert_pfn_pmd_prot (struct vm_fault *vmf,pfn_t pfn,pgprot_t pgprot,bool write);
mm/huge_memory.c +3 -1
@@ -1797,7 +1797,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,* - HPAGE_PMD_NR is protections changed and TLB flush necessary */ int change_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd, -unsigned long addr,pgprot_t newprot,unsigned long cp_flags) +unsigned long addr,pgprot_t newprot,unsigned long cp_flags, +struct mmu_gather *tlb) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;
@@ -1885,6 +1886,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, entry =pmd_clear_uffd_wp (entry); } ret =HPAGE_PMD_NR ; +tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE );set_pmd_at (mm, addr, pmd, entry);BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:
mm/mprotect.c +25 -26
124 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC 3/20 mm/mprotect: do not flush on permission promotion
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Currently, using mprotect() to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. At least on x86, as protection is promoted, no TLB flush is needed.
Add an arch-specific pte_may_need_flush() which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86 pte_may_need_flush().
For x86, besides the simple logic that PTE protection promotion or changes of software bits does require a flush, also add logic that considers the dirty-bit. If the dirty-bit is clear and write-protect is set, no TLB flush is needed, as x86 updates the dirty-bit atomically on write, and if the bit is clear, the PTE is reread.
--- arch/x86/include/asm/tlbflush.h | 44 +++++++++++++++++++++++++++++++++ include/asm-generic/tlb.h | 4 +++ mm/mprotect.c | 3 ++- 3 files changed, 50 insertions(+), 1 deletion(-)
arch/x86/include/asm/tlbflush.h +44 -0
@@ -255,6 +255,50 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) +{ +const pteval_t ignore_mask = _PAGE_SOFTW1 | _PAGE_SOFTW2 | + _PAGE_SOFTW3 | _PAGE_ACCESSED; +const pteval_t enable_mask = _PAGE_RW | _PAGE_DIRTY | _PAGE_GLOBAL; +pteval_t oldval =pte_val (oldpte); +pteval_t newval =pte_val (newpte); +pteval_t diff = oldval ^ newval; +pteval_t disable_mask =0 ; + +if (IS_ENABLED (CONFIG_X86_64 ) ||IS_ENABLED (CONFIG_X86_PAE )) + disable_mask = _PAGE_NX; + +/* new is non-present: need only if old is present */ +if (pte_none (newpte)) +return !pte_none (oldpte); + +/* +* If, excluding the ignored bits, only RW and dirty are cleared and the +* old PTE does not have the dirty-bit set, we can avoid a flush. This +* is possible since x86 architecture set the dirty bit atomically while +* it caches the PTE in the TLB. +* +* The condition considers any change to RW and dirty as not requiring +* flush if the old PTE is not dirty or not writable for simplification +* of the code and to consider (unlikely) cases of changing dirty-bit of +* write-protected PTE. +*/ +if (!(diff & ~(_PAGE_RW | _PAGE_DIRTY | ignore_mask)) && + (!(pte_dirty (oldpte) || !pte_write (oldpte)))) +return false; + +/* +* Any change of PFN and any flag other than those that we consider +* requires a flush (e.g., PAT, protection keys). To save flushes we do +* not consider the access bit as it is considered by the kernel as +* best-effort. +*/ +return diff & ((oldval & enable_mask) | + (newval & disable_mask) | + ~(enable_mask | disable_mask | ignore_mask)); +} +#define pte_may_need_flush pte_may_need_flush +#endif /* !MODULE */ #endif /* _ASM_X86_TLBFLUSH_H */
include/asm-generic/tlb.h +4 -0
@@ -654,6 +654,10 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, } while (0)#endif +#ifndef pte_may_need_flush +static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) {return true; } +#endif +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */
mm/mprotect.c +2 -1
@@ -139,7 +139,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_may_need_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);
Signature · 1 line
2.25.1
Andy Lutomirski
maintainerRe: [RFC 03/20] mm/mprotect: do not flush on permission promotion
Adding Andrew Cooper, who has a distressingly extensive understanding of the x86 PTE magic.
Quoted 39 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, using
mprotect()to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. At least on x86, as protection is promoted, no TLB flush is needed.Add an arch-specific
pte_may_need_flush()which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86pte_may_need_flush().For x86, besides the simple logic that PTE protection promotion or changes of software bits does require a flush, also add logic that considers the dirty-bit. If the dirty-bit is clear and write-protect is set, no TLB flush is needed, as x86 updates the dirty-bit atomically on write, and if the bit is clear, the PTE is reread.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>--- arch/x86/include/asm/tlbflush.h | 44 +++++++++++++++++++++++++++++++++ include/asm-generic/tlb.h | 4 +++ mm/mprotect.c | 3 ++- 3 files changed, 50 insertions(+), 1 deletion(-)arch/x86/include/asm/tlbflush.h +0
-0
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 261
Quoted 7 lines
arch/x86/include/asm/tlbflush.h +4
-0@@ -255,6 +255,50 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) +{ +const pteval_t ignore_mask = _PAGE_SOFTW1 | _PAGE_SOFTW2 | + _PAGE_SOFTW3 | _PAGE_ACCESSED;
Why is accessed ignored? Surely clearing the accessed bit needs a flush if the old PTE is present.
Quoted 7 lines
arch/x86/include/asm/tlbflush.h +7
-0+const pteval_t enable_mask = _PAGE_RW | _PAGE_DIRTY | _PAGE_GLOBAL; +pteval_t oldval =pte_val (oldpte); +pteval_t newval =pte_val (newpte); +pteval_t diff = oldval ^ newval; +pteval_t disable_mask =0 ; + +if (IS_ENABLED (CONFIG_X86_64 ) ||IS_ENABLED (CONFIG_X86_PAE ))
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 278
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+ disable_mask = _PAGE_NX; + +/* new is non-present: need only if old is present */ +if (pte_none (newpte)) +return !pte_none (oldpte); + +/* +* If, excluding the ignored bits, only RW and dirty are cleared and the +* old PTE does not have the dirty-bit set, we can avoid a flush. This +* is possible since x86 architecture set the dirty bit atomically while
s/set/sets/
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 288
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+* it caches the PTE in the TLB. +* +* The condition considers any change to RW and dirty as not requiring +* flush if the old PTE is not dirty or not writable for simplification +* of the code and to consider (unlikely) cases of changing dirty-bit of +* write-protected PTE. +*/ +if (!(diff & ~(_PAGE_RW | _PAGE_DIRTY | ignore_mask)) && + (!(pte_dirty (oldpte) || !pte_write (oldpte)))) +return false;
This logic seems confusing to me. Is your goal to say that, if the old PTE was clean and writable and the new PTE is write-protected, then no flush is needed? If so, I would believe you're right, but I'm not convinced you've actually implemented this. Also, there may be other things going on that need flushing, e.g. a change of the address or an accessed bit or NX change.
Also, CET makes this extra bizarre.
Quoted 48 lines
arch/x86/include/asm/tlbflush.h +13
-0+ +/* +* Any change of PFN and any flag other than those that we consider +* requires a flush (e.g., PAT, protection keys). To save flushes we do +* not consider the access bit as it is considered by the kernel as +* best-effort. +*/ +return diff & ((oldval & enable_mask) | + (newval & disable_mask) | + ~(enable_mask | disable_mask | ignore_mask)); +} +#define pte_may_need_flush pte_may_need_flush +#endif /* !MODULE */ #endif /* _ASM_X86_TLBFLUSH_H */ include/asm-generic/tlb.h +4
-0@@ -654,6 +654,10 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, } while (0)#endif +#ifndef pte_may_need_flush +static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) {return true; } +#endif +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */ mm/mprotect.c +2
-2@@ -139,7 +139,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_may_need_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte); --2.25.1
Re: [RFC 03/20] mm/mprotect: do not flush on permission promotion
Quoted 47 lines · Andy Lutomirski
Adding Andrew Cooper, who has a distressingly extensive understanding of the x86 PTE magic.
Quoted 41 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, using
mprotect()to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. At least on x86, as protection is promoted, no TLB flush is needed.Add an arch-specific
pte_may_need_flush()which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86pte_may_need_flush().For x86, besides the simple logic that PTE protection promotion or changes of software bits does require a flush, also add logic that considers the dirty-bit. If the dirty-bit is clear and write-protect is set, no TLB flush is needed, as x86 updates the dirty-bit atomically on write, and if the bit is clear, the PTE is reread.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>---
arch/x86/include/asm/tlbflush.h | 44 +++++++++++++++++++++++++++++++++ include/asm-generic/tlb.h | 4 +++ mm/mprotect.c | 3 ++- 3 files changed, 50 insertions(+), 1 deletion(-)arch/x86/include/asm/tlbflush.h +0
-0@@ -255,6 +255,50 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch);
Quoted 7 lines
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 261
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4
-0+static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) +{ +const pteval_t ignore_mask = _PAGE_SOFTW1 | _PAGE_SOFTW2 | + _PAGE_SOFTW3 | _PAGE_ACCESSED;Why is accessed ignored? Surely clearing the accessed bit needs a flush if the old PTE is present.
I am just following the current scheme in the kernel (x86):
int ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
/*
* On x86 CPUs, clearing the accessed bit without a TLB flush
* doesn't cause data corruption. [ It could cause incorrect
* page aging and the (mistaken) reclaim of hot pages, but the
* chance of that should be relatively low. ]
*
* So as a performance optimization don't flush the TLB when
* clearing the accessed bit, it will eventually be flushed by
* a context switch or a VM operation anyway. [ In the rare
* event of it not getting flushed for a long time the delay
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
return ptep_test_and_clear_young(vma, address, ptep);
}
Quoted 32 lines
Quoted 8 lines
arch/x86/include/asm/tlbflush.h +7
-0+const pteval_t enable_mask = _PAGE_RW | _PAGE_DIRTY | _PAGE_GLOBAL; +pteval_t oldval =pte_val (oldpte); +pteval_t newval =pte_val (newpte); +pteval_t diff = oldval ^ newval; +pteval_t disable_mask =0 ; + +if (IS_ENABLED (CONFIG_X86_64 ) ||IS_ENABLED (CONFIG_X86_PAE ))arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 278
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+ disable_mask = _PAGE_NX; + +/* new is non-present: need only if old is present */ +if (pte_none (newpte)) +return !pte_none (oldpte); + +/* +* If, excluding the ignored bits, only RW and dirty are cleared and the +* old PTE does not have the dirty-bit set, we can avoid a flush. This +* is possible since x86 architecture set the dirty bit atomically while s/set/sets/
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 288
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+* it caches the PTE in the TLB. +* +* The condition considers any change to RW and dirty as not requiring +* flush if the old PTE is not dirty or not writable for simplification +* of the code and to consider (unlikely) cases of changing dirty-bit of +* write-protected PTE. +*/ +if (!(diff & ~(_PAGE_RW | _PAGE_DIRTY | ignore_mask)) && + (!(pte_dirty (oldpte) || !pte_write (oldpte)))) +return false;
Quoted 3 lines
This logic seems confusing to me. Is your goal to say that, if the old PTE was clean and writable and the new PTE is write-protected, then no flush is needed?
Yes.
Quoted 4 lines
If so, I would believe you're right, but I'm
not convinced you've actually implemented this. Also, there may be other things going on that need flushing, e.g. a change of the address or an accessed bit or NX change.
The first part (diff & ~(_PAGE_RW | _PAGE_DIRTY | ignore_mask) is supposed to capture changes of address, NX-bit, etc.
The second part is indeed wrong. It should have been:
(!pte_dirty(oldpte) || !pte_write(oldpte))
Quoted 2 lines
Also, CET makes this extra bizarre.
I saw something about the not-writeable-and-dirty considered differently. I need to have a look, but I am not sure it affects anything.
Andy Lutomirski
maintainerRe: [RFC 03/20] mm/mprotect: do not flush on permission promotion
Quoted 58 lines · Nadav Amit
Quoted 48 lines · Andy Lutomirski
Adding Andrew Cooper, who has a distressingly extensive understanding of the x86 PTE magic.
Quoted 41 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, using
mprotect()to unprotect a memory region or uffd to unprotect a memory region causes a TLB flush. At least on x86, as protection is promoted, no TLB flush is needed.Add an arch-specific
pte_may_need_flush()which tells whether a TLB flush is needed based on the old PTE and the new one. Implement an x86pte_may_need_flush().For x86, besides the simple logic that PTE protection promotion or changes of software bits does require a flush, also add logic that considers the dirty-bit. If the dirty-bit is clear and write-protect is set, no TLB flush is needed, as x86 updates the dirty-bit atomically on write, and if the bit is clear, the PTE is reread.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>---
arch/x86/include/asm/tlbflush.h | 44 +++++++++++++++++++++++++++++++++ include/asm-generic/tlb.h | 4 +++ mm/mprotect.c | 3 ++- 3 files changed, 50 insertions(+), 1 deletion(-)arch/x86/include/asm/tlbflush.h +0
-0@@ -255,6 +255,50 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch);Quoted 7 lines
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 261
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4
-0+static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) +{ +const pteval_t ignore_mask = _PAGE_SOFTW1 | _PAGE_SOFTW2 | + _PAGE_SOFTW3 | _PAGE_ACCESSED;Why is accessed ignored? Surely clearing the accessed bit needs a flush if the old PTE is present.
I am just following the current scheme in the kernel (x86):
Quoted 9 lines
int ptep_clear_flush_young(struct
vm_area_struct*vma,
unsigned long address,pte_t*ptep)
{
/*
* On x86 CPUs, clearing the accessed bit without a TLB flush
* doesn't cause data corruption. [ It could cause incorrect
* page aging and the (mistaken) reclaim of hot pages, but the
* chance of that should be relatively low. ]
*
If anyone ever implements the optimization of skipping the flush when unmapping a !accessed page, then this will cause data corruption.
If nothing else, this deserves a nice comment in the new code.
Quoted 57 lines
* So as a performance optimization don't flush the TLB when
* clearing the accessed bit, it will eventually be flushed by
* a context switch or a VM operation anyway. [ In the rare
* event of it not getting flushed for a long time the delay
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
returnptep_test_and_clear_young(vma, address, ptep);
}Quoted 32 lines
Quoted 8 lines
arch/x86/include/asm/tlbflush.h +7
-0+const pteval_t enable_mask = _PAGE_RW | _PAGE_DIRTY | _PAGE_GLOBAL; +pteval_t oldval =pte_val (oldpte); +pteval_t newval =pte_val (newpte); +pteval_t diff = oldval ^ newval; +pteval_t disable_mask =0 ; + +if (IS_ENABLED (CONFIG_X86_64 ) ||IS_ENABLED (CONFIG_X86_PAE ))arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 278
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+ disable_mask = _PAGE_NX; + +/* new is non-present: need only if old is present */ +if (pte_none (newpte)) +return !pte_none (oldpte); + +/* +* If, excluding the ignored bits, only RW and dirty are cleared and the +* old PTE does not have the dirty-bit set, we can avoid a flush. This +* is possible since x86 architecture set the dirty bit atomically while s/set/sets/
arch/x86/include/asm/tlbflush.h static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, · line 288
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+* it caches the PTE in the TLB. +* +* The condition considers any change to RW and dirty as not requiring +* flush if the old PTE is not dirty or not writable for simplification +* of the code and to consider (unlikely) cases of changing dirty-bit of +* write-protected PTE. +*/ +if (!(diff & ~(_PAGE_RW | _PAGE_DIRTY | ignore_mask)) && + (!(pte_dirty (oldpte) || !pte_write (oldpte)))) +return false;Quoted 3 lines
This logic seems confusing to me. Is your goal to say that, if the old PTE was clean and writable and the new PTE is write-protected, then no flush is needed?
Yes.
Quoted 4 lines
If so, I would believe you're right, but I'm
not convinced you've actually implemented this. Also, there may be other things going on that need flushing, e.g. a change of the address or an accessed bit or NX change.The first part (diff & ~(_PAGE_RW |
_PAGE_DIRTY|ignore_mask) is supposed to capture changes of address, NX-bit, etc.
Quoted 9 lines
The second part is indeed wrong. It should have been:
(!pte_dirty(oldpte) || !pte_write(oldpte))Quoted 2 lines
Also, CET makes this extra bizarre.
I saw something about the not-writeable-and-dirty considered differently. I need to have a look, but I am not sure it affects anything.
It affects everyone's sanity. I don't yet have an opinion as to whether it affects correctness :)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC 03/20] mm/mprotect: do not flush on permission promotion
Quoted 28 lines · Andrew Cooper
Quoted 2 lines · Andy Lutomirski
Adding Andrew Cooper, who has a distressingly extensive understanding of the x86 PTE magic.
Pretty sure it is all learning things the hard way...
Quoted 12 lines · Nadav Amit
mm/mprotect.c static unsigned long change_pte_range(struct mmu_gather *tlb, · line 143
Quoted 11 lines
mm/mprotect.c +2
-1@@ -139,7 +139,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_may_need_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE );You're choosing to avoid the flush, based on A/D bits read ahead of the actual modification of the PTE.
In this example, another thread can write into the range (sets A and D), and get a suitable TLB entry which goes unflushed while the rest of the kernel thinks the memory is write-protected and clean.
Quoted 5 lines
The only safe way to do this is to use XCHG/etc to modify the PTE, and base flush calculations on the results. Atomic operations are ordered with A/D updates from pagewalks on other CPUs, even on AMD where A updates are explicitly not ordered with regular memory reads, for performance reasons.
Thanks Andrew for the feedback, but I think the patch does it exactly in this safe manner that you describe (at least on native x86, but I see a similar path elsewhere as well):
oldpte = ptep_modify_prot_start()
-> __ptep_modify_prot_start()
-> ptep_get_and_clear
-> native_ptep_get_and_clear()
-> xchg()
Note that the xchg() will clear the PTE (i.e., making it non-present), and no further updates of A/D are possible until ptep_modify_prot_commit() is called.
On non-SMP setups this is not atomic (no xchg), but since we hold the lock, we should be safe.
I guess you are right and a pte_may_need_flush() deserves a comment to clarify that oldpte must be obtained by an atomic operation to ensure no A/D bits are lost (as you say).
Yet, I do not see a correctness problem. Am I missing something?
Re: [RFC 03/20] mm/mprotect: do not flush on permission promotion
Quoted 45 lines · Nadav Amit
Quoted 26 lines · Andrew Cooper
Quoted 2 lines · Andy Lutomirski
Adding Andrew Cooper, who has a distressingly extensive understanding of the x86 PTE magic.
Pretty sure it is all learning things the hard way...
Quoted 12 lines · Nadav Amit
mm/mprotect.c static unsigned long change_pte_range(struct mmu_gather *tlb, · line 143
Quoted 11 lines
mm/mprotect.c +2
-1@@ -139,7 +139,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent =pte_mkwrite (ptent); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +if (pte_may_need_flush (oldpte, ptent)) +tlb_flush_pte_range (tlb, addr,PAGE_SIZE );You're choosing to avoid the flush, based on A/D bits read ahead of the actual modification of the PTE.
In this example, another thread can write into the range (sets A and D), and get a suitable TLB entry which goes unflushed while the rest of the kernel thinks the memory is write-protected and clean.
Quoted 5 lines
The only safe way to do this is to use XCHG/etc to modify the PTE, and base flush calculations on the results. Atomic operations are ordered with A/D updates from pagewalks on other CPUs, even on AMD where A updates are explicitly not ordered with regular memory reads, for performance reasons.
Thanks Andrew for the feedback, but I think the patch does it exactly in this safe manner that you describe (at least on native x86, but I see a similar path elsewhere as well):
oldpte =
ptep_modify_prot_start()
->__ptep_modify_prot_start()
->ptep_get_and_clear
->native_ptep_get_and_clear()
->xchg()Note that the
xchg()will clear the PTE (i.e., making it non-present), and no further updates of A/D are possible untilptep_modify_prot_commit()is called.
Quoted 8 lines
On non-SMP setups this is not atomic (no xchg), but since we hold the lock, we should be safe.
I guess you are right and a
pte_may_need_flush()deserves a comment to clarify that oldpte must be obtained by an atomic operation to ensure no A/D bits are lost (as you say).Yet, I do not see a correctness problem. Am I missing something?
No(ish) - I failed to spot that path.
But native_ptep_get_and_clear() is broken on !SMP builds. It needs to be an XCHG even in that case, to spot A/D updates from prefetch or shared-virtual-memory DMA.
~Andrew
RFC 4/20 mm/mapping_dirty_helpers: use mmu_gather
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Avoid open-coding mmu_gather for no reason. There is no apparent reason not to use the existing mmu_gather interfaces.
Use the newly introduced pte_may_need_flush() to check whether a flush is needed to avoid unnecassary flushes.
--- mm/mapping_dirty_helpers.c | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-)
mm/mapping_dirty_helpers.c +11 -26
@@ -4,7 +4,7 @@#include <linux/bitops.h> #include <linux/mmu_notifier.h> #include <asm/cacheflush.h> -#include <asm/tlbflush.h> +#include <asm/tlb.h> /** * struct wp_walk - Private struct for pagetable walk callbacks
@@ -15,8 +15,7 @@*/ struct wp_walk {struct mmu_notifier_range range; -unsigned long tlbflush_start; -unsigned long tlbflush_end; +struct mmu_gather tlb;unsigned long total; };
@@ -42,9 +41,9 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end, ptent =pte_wrprotect (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++; - wpwalk->tlbflush_start =min (wpwalk->tlbflush_start, addr); - wpwalk->tlbflush_end =max (wpwalk->tlbflush_end, - addr +PAGE_SIZE ); + +if (pte_may_need_flush (old_pte, ptent)) +tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); }return 0 ;
@@ -101,9 +100,7 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++; - wpwalk->tlbflush_start =min (wpwalk->tlbflush_start, addr); - wpwalk->tlbflush_end =max (wpwalk->tlbflush_end, - addr +PAGE_SIZE ); +tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE );__set_bit (pgoff, cwalk->bitmap); cwalk->start =min (cwalk->start, pgoff);
@@ -184,20 +181,13 @@ static int wp_clean_pre_vma(unsigned long start, unsigned long end, {struct wp_walk *wpwalk = walk->private; - wpwalk->tlbflush_start = end; - wpwalk->tlbflush_end = start; -mmu_notifier_range_init (&wpwalk->range,MMU_NOTIFY_PROTECTION_PAGE ,0 , walk->vma, walk->mm, start, end);mmu_notifier_invalidate_range_start (&wpwalk->range);flush_cache_range (walk->vma, start, end); -/* -* We're not using tlb_gather_mmu() since typically -* only a small subrange of PTEs are affected, whereas -* tlb_gather_mmu() records the full range. -*/ -inc_tlb_flush_pending (walk->mm); +tlb_gather_mmu (&wpwalk->tlb, walk->mm); +tlb_start_vma (&wpwalk->tlb, walk->vma);return 0 ; }
@@ -212,15 +202,10 @@ static void wp_clean_post_vma(struct mm_walk *walk) {struct wp_walk *wpwalk = walk->private; -if (mm_tlb_flush_nested (walk->mm)) -flush_tlb_range (walk->vma, wpwalk->range.start, - wpwalk->range.end); -else if (wpwalk->tlbflush_end > wpwalk->tlbflush_start) -flush_tlb_range (walk->vma, wpwalk->tlbflush_start, - wpwalk->tlbflush_end); -mmu_notifier_invalidate_range_end (&wpwalk->range); -dec_tlb_flush_pending (walk->mm); + +tlb_end_vma (&wpwalk->tlb, walk->vma); +tlb_finish_mmu (&wpwalk->tlb); }/*
Signature · 1 line
2.25.1
RFC 5/20 mm/tlb: move BATCHED_UNMAP_TLB_FLUSH to tlb.h
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Arguably, tlb.h is the natural place for TLB related code. In addition, task_mmu needs to be able to call to flush_tlb_batched_pending() and therefore cannot (or should not) use mm/internal.h.
Move all the functions that are controlled by CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH to tlb.h.
--- include/asm-generic/tlb.h | 17 +++++++++++++++++ mm/internal.h | 16 ---------------- mm/mremap.c | 2 +- mm/vmscan.c | 1 + 4 files changed, 19 insertions(+), 17 deletions(-)
include/asm-generic/tlb.h +17 -0
@@ -658,6 +658,23 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb,static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) {return true; }#endif +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +void try_to_unmap_flush (void ); +void try_to_unmap_flush_dirty (void ); +void flush_tlb_batched_pending (struct mm_struct *mm); +#else +static inline void try_to_unmap_flush (void ) +{ +} +static inline void try_to_unmap_flush_dirty (void ) +{ +} +static inline void flush_tlb_batched_pending (struct mm_struct *mm) +{ +} +static inline void tlb_batch_init (void ) { } +#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */
mm/internal.h +0 -16
@@ -585,22 +585,6 @@ struct tlbflush_unmap_batch;*/ extern struct workqueue_struct *mm_percpu_wq; -#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH -void try_to_unmap_flush (void ); -void try_to_unmap_flush_dirty (void ); -void flush_tlb_batched_pending (struct mm_struct *mm); -#else -static inline void try_to_unmap_flush (void ) -{ -} -static inline void try_to_unmap_flush_dirty (void ) -{ -} -static inline void flush_tlb_batched_pending (struct mm_struct *mm) -{ -} -#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ -extern const struct trace_print_flags pageflag_names[];extern const struct trace_print_flags vmaflag_names[];extern const struct trace_print_flags gfpflag_names[];
mm/mremap.c +1 -1
@@ -26,7 +26,7 @@#include <linux/userfaultfd_k.h> #include <asm/cacheflush.h> -#include <asm/tlbflush.h> +#include <asm/tlb.h> #include "internal.h"
mm/vmscan.c +1 -0
@@ -52,6 +52,7 @@#include <asm/tlbflush.h> #include <asm/div64.h> +#include <asm/tlb.h> #include <linux/swapops.h> #include <linux/balloon_compaction.h>
Signature · 1 line
2.25.1
RFC 6/20 fs/task_mmu: use mmu_gather interface of clear-soft-dirty
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Use mmu_gather interface in task_mmu instead of {inc|dec}_tlb_flush_pending(). This would allow to consolidate the code and to avoid potential bugs.
--- fs/proc/task_mmu.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)
fs/proc/task_mmu.c +24 -3
@@ -1032,8 +1032,25 @@ enum clear_refs_types {struct clear_refs_private {enum clear_refs_types type; +struct mmu_gather tlb; }; +static int tlb_pre_vma (unsigned long start,unsigned long end, +struct mm_walk *walk) +{ +struct clear_refs_private *cp = walk->private; + +tlb_start_vma (&cp->tlb, walk->vma); +return 0 ; +} + +static void tlb_post_vma (struct mm_walk *walk) +{ +struct clear_refs_private *cp = walk->private; + +tlb_end_vma (&cp->tlb, walk->vma); +} +#ifdef CONFIG_MEM_SOFT_DIRTY #define is_cow_mapping (flags) (((flags) & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE)
@@ -1140,6 +1157,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,/* Clear accessed and referenced bits. */ pmdp_test_and_clear_young (vma, addr, pmd);test_and_clear_page_young (page); +tlb_flush_pmd_range (&cp->tlb, addr,HPAGE_PMD_SIZE );ClearPageReferenced (page); out:spin_unlock (ptl);
@@ -1155,6 +1173,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,if (cp->type ==CLEAR_REFS_SOFT_DIRTY ) {clear_soft_dirty (vma, addr, pte); +tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );continue ; }
@@ -1168,6 +1187,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,/* Clear accessed and referenced bits. */ ptep_test_and_clear_young (vma, addr, pte);test_and_clear_page_young (page); +tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );ClearPageReferenced (page); }pte_unmap_unlock (pte -1 , ptl);
@@ -1198,6 +1218,8 @@ static int clear_refs_test_walk(unsigned long start, unsigned long end, }static const struct mm_walk_ops clear_refs_walk_ops = { + .pre_vma = tlb_pre_vma, + .post_vma = tlb_post_vma, .pmd_entry = clear_refs_pte_range, .test_walk = clear_refs_test_walk, };
@@ -1248,6 +1270,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, goto out_unlock; } +tlb_gather_mmu (&cp.tlb, mm);if (type ==CLEAR_REFS_SOFT_DIRTY ) {for (vma = mm->mmap; vma; vma = vma->vm_next) {if (!(vma->vm_flags &VM_SOFTDIRTY ))
@@ -1256,7 +1279,6 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,vma_set_page_prot (vma); } -inc_tlb_flush_pending (mm);mmu_notifier_range_init (&range,MMU_NOTIFY_SOFT_DIRTY ,0 ,NULL , mm,0 ,-1UL );mmu_notifier_invalidate_range_start (&range);
@@ -1265,10 +1287,9 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, &cp);if (type ==CLEAR_REFS_SOFT_DIRTY ) {mmu_notifier_invalidate_range_end (&range); -flush_tlb_mm (mm); -dec_tlb_flush_pending (mm); } out_unlock: +tlb_finish_mmu (&cp.tlb);mmap_write_unlock (mm); out_mm:mmput (mm);
Signature · 1 line
2.25.1
RFC 7/20 mm: move x86 tlb_gen to generic code
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
x86 currently has a TLB-generation tracking logic that can be used by additional architectures (as long as they implement some additional logic).
Extract the relevant pieces of code from x86 to general TLB code. This would be useful to allow to write the next "fine granularity deferred TLB flushes detection" patches without making them x86-specific.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/mmu.h | 10 -------- arch/x86/include/asm/mmu_context.h | 1 - arch/x86/include/asm/tlbflush.h | 18 -------------- arch/x86/mm/tlb.c | 8 +++--- drivers/firmware/efi/efi.c | 1 + include/linux/mm_types.h | 39 ++++++++++++++++++++++++++++++ init/Kconfig | 6 +++++ kernel/fork.c | 2 ++ mm/init-mm.c | 1 + mm/rmap.c | 9 ++++++- 11 files changed, 62 insertions(+), 34 deletions(-)
arch/x86/Kconfig +1 -0
@@ -86,6 +86,7 @@ config X86 select ARCH_HAS_SYSCALL_WRAPPER select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_HAS_DEBUG_WX + select ARCH_HAS_TLB_GENERATIONS select ARCH_HAVE_NMI_SAFE_CMPXCHG select ARCH_MIGHT_HAVE_ACPI_PDC if ACPI select ARCH_MIGHT_HAVE_PC_PARPORT
arch/x86/include/asm/mmu.h +0 -10
@@ -23,16 +23,6 @@ typedef struct {*/ u64 ctx_id; -/* -* Any code that needs to do any sort of TLB flushing for this -* mm will first make its changes to the page tables, then -* increment tlb_gen, then flush. This lets the low-level -* flushing code keep track of what needs flushing. -* -* This is not used on Xen PV. -*/ -atomic64_t tlb_gen; -#ifdef CONFIG_MODIFY_LDT_SYSCALL struct rw_semaphore ldt_usr_sem;struct ldt_struct *ldt;
arch/x86/include/asm/mmu_context.h +0 -1
@@ -105,7 +105,6 @@ static inline int init_new_context(struct task_struct *tsk,mutex_init (&mm->context.lock); mm->context.ctx_id =atomic64_inc_return (&last_mm_ctx_id); -atomic64_set (&mm->context.tlb_gen,0 );#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS if (cpu_feature_enabled (X86_FEATURE_OSPKE )) {
arch/x86/include/asm/tlbflush.h +0 -18
@@ -235,24 +235,6 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); } -static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) -{ -/* -* Bump the generation count. This also serves as a full barrier -* that synchronizes with switch_mm(): callers are required to order -* their read of mm_cpumask after their writes to the paging -* structures. -*/ -return atomic64_inc_return (&mm->context.tlb_gen); -} - -static inline void arch_tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch, -struct mm_struct *mm) -{ -inc_mm_tlb_gen (mm); -cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); -} -extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch);static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte)
arch/x86/mm/tlb.c +4 -4
@@ -511,7 +511,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,* the TLB shootdown code. */ smp_mb (); - next_tlb_gen =atomic64_read (&next->context.tlb_gen); + next_tlb_gen =atomic64_read (&next->tlb_gen);if (this_cpu_read (cpu_tlbstate.ctxs[prev_asid].tlb_gen) == next_tlb_gen)return ;
@@ -546,7 +546,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,*/ if (next != &init_mm)cpumask_set_cpu (cpu,mm_cpumask (next)); - next_tlb_gen =atomic64_read (&next->context.tlb_gen); + next_tlb_gen =atomic64_read (&next->tlb_gen);choose_new_asid (next, next_tlb_gen, &new_asid, &need_flush);
@@ -618,7 +618,7 @@ void initialize_tlbstate_and_flush(void) {int i;struct mm_struct *mm =this_cpu_read (cpu_tlbstate.loaded_mm); -u64 tlb_gen =atomic64_read (&init_mm.context.tlb_gen); +u64 tlb_gen =atomic64_read (&init_mm.tlb_gen);unsigned long cr3 =__read_cr3 ();/* Assert that CR3 already references the right mm. */
@@ -667,7 +667,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f,*/ struct mm_struct *loaded_mm =this_cpu_read (cpu_tlbstate.loaded_mm);u32 loaded_mm_asid =this_cpu_read (cpu_tlbstate.loaded_mm_asid); -u64 mm_tlb_gen =atomic64_read (&loaded_mm->context.tlb_gen); +u64 mm_tlb_gen =atomic64_read (&loaded_mm->tlb_gen);u64 local_tlb_gen =this_cpu_read (cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen);/* This code cannot presently handle being reentered. */
drivers/firmware/efi/efi.c +1 -0
@@ -62,6 +62,7 @@ struct mm_struct efi_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock), .mmlist = LIST_HEAD_INIT(efi_mm.mmlist), .cpu_bitmap = { [BITS_TO_LONGS(NR_CPUS)] = 0}, + INIT_TLB_GENERATIONS };struct workqueue_struct *efi_rts_wq;
include/linux/mm_types.h +39 -0
@@ -560,6 +560,17 @@ struct mm_struct {#ifdef CONFIG_IOMMU_SUPPORT u32 pasid; +#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +/* +* Any code that needs to do any sort of TLB flushing for this +* mm will first make its changes to the page tables, then +* increment tlb_gen, then flush. This lets the low-level +* flushing code keep track of what needs flushing. +* +* This is not used on Xen PV. +*/ +atomic64_t tlb_gen;#endif } __randomize_layout;
@@ -676,6 +687,34 @@ static inline bool mm_tlb_flush_nested(struct mm_struct *mm)return atomic_read (&mm->tlb_flush_pending) >1 ; } +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +static inline void init_mm_tlb_gen (struct mm_struct *mm) +{ +atomic64_set (&mm->tlb_gen,0 ); +} + +static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) +{ +/* +* Bump the generation count. This also serves as a full barrier +* that synchronizes with switch_mm(): callers are required to order +* their read of mm_cpumask after their writes to the paging +* structures. +*/ +return atomic64_inc_return (&mm->tlb_gen); +} + +#define INIT_TLB_GENERATIONS \ + .tlb_gen = ATOMIC64_INIT(0), + +#else + +static inline void init_mm_tlb_gen (struct mm_struct *mm) { } + +#define INIT_TLB_GENERATION + +#endif +struct vm_fault ;/**
init/Kconfig +6 -0
@@ -842,6 +842,12 @@ config ARCH_SUPPORTS_NUMA_BALANCING # and the refill costs are offset by the savings of sending fewer IPIs. config ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH bool + depends on ARCH_HAS_TLB_GENERATIONS + +# +# For architectures that track for each address space the TLB generation. +config ARCH_HAS_TLB_GENERATIONS + bool config CC_HAS_INT128 def_bool !$(cc-option,$(m64-flag) -D__SIZEOF_INT128__=0) && 64BIT
kernel/fork.c +2 -0
@@ -1027,6 +1027,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,RCU_INIT_POINTER (mm->exe_file,NULL );mmu_notifier_subscriptions_init (mm);init_tlb_flush_pending (mm); +init_mm_tlb_gen (mm); +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte =NULL ;#endif
mm/init-mm.c +1 -0
@@ -38,5 +38,6 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, + INIT_TLB_GENERATIONS INIT_MM_CONTEXT(init_mm) };
mm/rmap.c +8 -1
@@ -613,11 +613,18 @@ void try_to_unmap_flush_dirty(void)try_to_unmap_flush (); } +static inline void tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm) +{ +inc_mm_tlb_gen (mm); +cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); +} +static void set_tlb_ubc_flush_pending (struct mm_struct *mm,bool writable) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; -arch_tlbbatch_add_mm (&tlb_ubc->arch, mm); +tlbbatch_add_mm (&tlb_ubc->arch, mm); tlb_ubc->flush_required = true;/*
Signature · 1 line
2.25.1
Andy Lutomirski
maintainerRe: [RFC 07/20] mm: move x86 tlb_gen to generic code
Quoted 10 lines · Nadav Amit
From: Nadav Amit <[email protected]>
x86 currently has a TLB-generation tracking logic that can be used by additional architectures (as long as they implement some additional logic).
Extract the relevant pieces of code from x86 to general TLB code. This would be useful to allow to write the next "fine granularity deferred TLB flushes detection" patches without making them x86-specific.
Tentative ACK.
My biggest concern about this is that, once it's exposed to core code, people might come up with clever-but-incorrect ways to abuse it. Oh well.
Quoted 8 lines
struct
workqueue_struct*efi_rts_wq;include/linux/mm_types.h +0
-0@@ -560,6 +560,17 @@ struct mm_struct {#ifdef CONFIG_IOMMU_SUPPORT
include/linux/mm_types.h struct mm_struct { · line 571
Quoted 10 lines
include/linux/mm_types.h +9
-0u32 pasid; +#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +/* +* Any code that needs to do any sort of TLB flushing for this +* mm will first make its changes to the page tables, then +* increment tlb_gen, then flush. This lets the low-level +* flushing code keep track of what needs flushing. +* +* This is not used on Xen PV.
That last comment should probably go away.
RFC 8/20 mm: store completed TLB generation
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
To detect deferred TLB flushes in fine granularity, we need to keep track on the completed TLB flush generation for each mm.
Add logic to track for each mm the tlb_gen_completed, which tracks the completed TLB generation. It is the arch responsibility to call mark_mm_tlb_gen_done() whenever a TLB flush is completed.
Start the generation numbers from 1 instead of 0. This would allow later to detect whether flushes of a certain generation were completed.
--- arch/x86/mm/tlb.c | 10 ++++++++++ include/asm-generic/tlb.h | 33 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 15 ++++++++++++++- 3 files changed, 57 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +10 -0
@@ -14,6 +14,7 @@#include <asm/nospec-branch.h> #include <asm/cache.h> #include <asm/apic.h> +#include <asm/tlb.h> #include "mm_internal.h"
@@ -915,6 +916,9 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), info); +/* Update the completed generation */ +mark_mm_tlb_gen_done (mm, new_tlb_gen); +put_flush_tlb_info ();put_cpu (); }
@@ -1147,6 +1151,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)cpumask_clear (&batch->cpumask); +/* +* We cannot call mark_mm_tlb_gen_done() since we do not know which +* mm's should be flushed. This may lead to some unwarranted TLB +* flushes, but not to correction problems. +*/ +put_cpu (); }
include/asm-generic/tlb.h +33 -0
@@ -513,6 +513,39 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm }#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + +/* +* Helper function to update a generation to have a new value, as long as new +* value is greater or equal to gen. +*/ +static inline void tlb_update_generation (atomic64_t *gen,u64 new_gen) +{ +u64 cur_gen =atomic64_read (gen); + +while (cur_gen < new_gen) { +u64 old_gen =atomic64_cmpxchg (gen, cur_gen, new_gen); + +/* Check if we succeeded in the cmpxchg */ +if (likely (cur_gen == old_gen)) +break ; + + cur_gen = old_gen; + }; +} + + +static inline void mark_mm_tlb_gen_done (struct mm_struct *mm,u64 gen) +{ +/* +* Update the completed generation to the new generation if the new +* generation is greater than the previous one. +*/ +tlb_update_generation (&mm->tlb_gen_completed, gen); +} + +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*.
include/linux/mm_types.h +14 -1
@@ -571,6 +571,13 @@ struct mm_struct {* This is not used on Xen PV. */ atomic64_t tlb_gen; + +/* +* TLB generation which is guarnateed to be flushed, including +* all the PTE changes that were performed before tlb_gen was +* incremented. +*/ +atomic64_t tlb_gen_completed;#endif } __randomize_layout;
@@ -690,7 +697,13 @@ static inline bool mm_tlb_flush_nested(struct mm_struct *mm)#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS static inline void init_mm_tlb_gen (struct mm_struct *mm) { -atomic64_set (&mm->tlb_gen,0 ); +/* +* Start from generation of 1, so default generation 0 will be +* considered as flushed and would not be regarded as an outstanding +* deferred invalidation. +*/ +atomic64_set (&mm->tlb_gen,1 ); +atomic64_set (&mm->tlb_gen_completed,1 ); }static inline u64 inc_mm_tlb_gen (struct mm_struct *mm)
Signature · 1 line
2.25.1
Andy Lutomirski
maintainerRe: [RFC 08/20] mm: store completed TLB generation
Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
To detect deferred TLB flushes in fine granularity, we need to keep track on the completed TLB flush generation for each mm.
Quoted 6 lines
Add logic to track for each mm the
tlb_gen_completed, which tracks the completed TLB generation. It is the arch responsibility to callmark_mm_tlb_gen_done()whenever a TLB flush is completed.Start the generation numbers from 1 instead of 0. This would allow later to detect whether flushes of a certain generation were completed.
Can you elaborate on how this helps?
I think you should document that tlb_gen_completed only means that no outdated TLB entries will be observably used. In the x86 implementation it's possible for older TLB entries to still exist, unused, in TLBs of cpus running other mms.
How does this work with arch_tlbbatch_flush()?
Quoted 58 lines
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>--- arch/x86/mm/tlb.c | 10 ++++++++++ include/asm-generic/tlb.h | 33 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 15 ++++++++++++++- 3 files changed, 57 insertions(+), 1 deletion(-)arch/x86/mm/tlb.c +10
-0@@ -14,6 +14,7 @@#include <asm/nospec-branch.h> #include <asm/cache.h> #include <asm/apic.h> +#include <asm/tlb.h> #include "mm_internal.h" @@ -915,6 +916,9 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), info); +/* Update the completed generation */ +mark_mm_tlb_gen_done (mm, new_tlb_gen); +put_flush_tlb_info ();put_cpu (); }@@ -1147,6 +1151,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)cpumask_clear (&batch->cpumask); +/* +* We cannot call mark_mm_tlb_gen_done() since we do not know which +* mm's should be flushed. This may lead to some unwarranted TLB +* flushes, but not to correction problems. +*/ +put_cpu (); }include/asm-generic/tlb.h +0
-0
include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 521
Quoted 9 lines
include/asm-generic/tlb.h +6
-0@@ -513,6 +513,39 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm }#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + +/* +* Helper function to update a generation to have a new value, as long as new +* value is greater or equal to gen. +*/
I read this a couple of times, and I don't understand it. How about:
Helper function to atomically set *gen = max(*gen, new_gen)
Quoted 35 lines
include/asm-generic/tlb.h +27
-0+static inline void tlb_update_generation (atomic64_t *gen,u64 new_gen) +{ +u64 cur_gen =atomic64_read (gen); + +while (cur_gen < new_gen) { +u64 old_gen =atomic64_cmpxchg (gen, cur_gen, new_gen); + +/* Check if we succeeded in the cmpxchg */ +if (likely (cur_gen == old_gen)) +break ; + + cur_gen = old_gen; + }; +} + + +static inline void mark_mm_tlb_gen_done (struct mm_struct *mm,u64 gen) +{ +/* +* Update the completed generation to the new generation if the new +* generation is greater than the previous one. +*/ +tlb_update_generation (&mm->tlb_gen_completed, gen); +} + +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. include/linux/mm_types.h +0
-0
include/linux/mm_types.h struct mm_struct { · line 576
Quoted 6 lines
include/linux/mm_types.h +3
-0@@ -571,6 +571,13 @@ struct mm_struct {* This is not used on Xen PV. */ atomic64_t tlb_gen; + +/* +* TLB generation which is guarnateed to be flushed, including
guaranteed
include/linux/mm_types.h struct mm_struct { · line 579
Quoted 3 lines
include/linux/mm_types.h +3
-0+* all the PTE changes that were performed before tlb_gen was +* incremented. +*/
I will defer judgment to future patches before I believe that this isn't racy :)
Quoted 5 lines
include/linux/mm_types.h +1
-0+atomic64_t tlb_gen_completed;#endif } __randomize_layout;
include/linux/mm_types.h static inline bool mm_tlb_flush_nested(struct mm_struct *mm) · line 704
Quoted 9 lines
include/linux/mm_types.h +5
-1@@ -690,7 +697,13 @@ static inline bool mm_tlb_flush_nested(struct mm_struct *mm)#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS static inline void init_mm_tlb_gen (struct mm_struct *mm) { -atomic64_set (&mm->tlb_gen,0 ); +/* +* Start from generation of 1, so default generation 0 will be +* considered as flushed and would not be regarded as an outstanding +* deferred invalidation. +*/
Aha, this makes sense.
Quoted 8 lines
include/linux/mm_types.h +2
-1+atomic64_set (&mm->tlb_gen,1 ); +atomic64_set (&mm->tlb_gen_completed,1 ); }static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) --2.25.1
Re: [RFC 08/20] mm: store completed TLB generation
Quoted 8 lines · Andy Lutomirski
Quoted 5 lines · Nadav Amit
From: Nadav Amit <[email protected]>
To detect deferred TLB flushes in fine granularity, we need to keep track on the completed TLB flush generation for each mm.
Quoted 8 lines
Quoted 6 lines
Add logic to track for each mm the
tlb_gen_completed, which tracks the completed TLB generation. It is the arch responsibility to callmark_mm_tlb_gen_done()whenever a TLB flush is completed.Start the generation numbers from 1 instead of 0. This would allow later to detect whether flushes of a certain generation were completed.
Can you elaborate on how this helps?
I guess it should have gone to patch 15.
The relevant code it interacts with is in read_defer_tlb_flush_gen(). It allows to use a single check to see “outdated” deferred TLB gen. Initially tlb->defer_gen is zero. We are going to do inc_mm_tlb_gen() both on the first time we defer TLB entries and whenever we see mm_gen is newer than tlb->defer_gen:
+ mm_gen = atomic64_read(&mm->tlb_gen);
+
+ /*
+ * This condition checks for both first deferred TLB flush and for other + * TLB pending or executed TLB flushes after the last table that we + * updated. In the latter case, we are going to skip a generation, which + * would lead to a full TLB flush. This should therefore not cause + * correctness issues, and should not induce overheads, since anyhow in + * TLB storms it is better to perform full TLB flush. + */
+ if (mm_gen != tlb->defer_gen) {
+ VM_BUG_ON(mm_gen < tlb->defer_gen);
+
+ tlb->defer_gen = inc_mm_tlb_gen(mm);
+ }
Quoted 5 lines
I think you should document that
tlb_gen_completedonly means that no outdated TLB entries will be observably used. In the x86 implementation it's possible for older TLB entries to still exist, unused, in TLBs of cpus running other mms.
You mean entries that be later flushed during switch_mm_irqs_off(), right? I think that overall my comments need some work. Yes.
Quoted 1 line
How does this work with
arch_tlbbatch_flush()?
completed_gen is not updated by arch_tlbbatch_flush(), since I couldn’t find a way to combine them. completed_gen might not catch up with tlb_gen in this case until another TLB flush takes place. I do not see correctness issue, but it might result in redundant TLB flush.
Quoted 115 lines
Quoted 57 lines
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>---
arch/x86/mm/tlb.c | 10 ++++++++++ include/asm-generic/tlb.h | 33 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 15 ++++++++++++++- 3 files changed, 57 insertions(+), 1 deletion(-)arch/x86/mm/tlb.c +10
-0@@ -14,6 +14,7 @@#include <asm/nospec-branch.h> #include <asm/cache.h> #include <asm/apic.h> +#include <asm/tlb.h> #include "mm_internal.h" @@ -915,6 +916,9 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), info); +/* Update the completed generation */ +mark_mm_tlb_gen_done (mm, new_tlb_gen); +put_flush_tlb_info ();put_cpu (); }@@ -1147,6 +1151,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)cpumask_clear (&batch->cpumask); +/* +* We cannot call mark_mm_tlb_gen_done() since we do not know which +* mm's should be flushed. This may lead to some unwarranted TLB +* flushes, but not to correction problems. +*/ +put_cpu (); }include/asm-generic/tlb.h +0
-0include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 521
Quoted 9 lines
include/asm-generic/tlb.h +6
-0@@ -513,6 +513,39 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm }#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + +/* +* Helper function to update a generation to have a new value, as long as new +* value is greater or equal to gen. +*/ I read this a couple of times, and I don't understand it. How about:
Helper function to atomically set *gen =
max(*gen, new_gen)Quoted 35 lines
include/asm-generic/tlb.h +27
-0+static inline void tlb_update_generation (atomic64_t *gen,u64 new_gen) +{ +u64 cur_gen =atomic64_read (gen); + +while (cur_gen < new_gen) { +u64 old_gen =atomic64_cmpxchg (gen, cur_gen, new_gen); + +/* Check if we succeeded in the cmpxchg */ +if (likely (cur_gen == old_gen)) +break ; + + cur_gen = old_gen; + }; +} + + +static inline void mark_mm_tlb_gen_done (struct mm_struct *mm,u64 gen) +{ +/* +* Update the completed generation to the new generation if the new +* generation is greater than the previous one. +*/ +tlb_update_generation (&mm->tlb_gen_completed, gen); +} + +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. include/linux/mm_types.h +0
-0include/linux/mm_types.h struct mm_struct { · line 576
Quoted 6 lines
include/linux/mm_types.h +3
-0@@ -571,6 +571,13 @@ struct mm_struct {* This is not used on Xen PV. */ atomic64_t tlb_gen; + +/* +* TLB generation which is guarnateed to be flushed, including guaranteed
Quoted 5 lines
include/linux/mm_types.h struct mm_struct { · line 579
Quoted 3 lines
include/linux/mm_types.h +3
-0+* all the PTE changes that were performed before tlb_gen was +* incremented. +*/ I will defer judgment to future patches before I believe that this isn't racy :)
Fair enough. Thanks for the review.
Andy Lutomirski
maintainerRe: [RFC 08/20] mm: store completed TLB generation
Quoted 44 lines · Nadav Amit
Quoted 9 lines · Andy Lutomirski
Quoted 5 lines · Nadav Amit
From: Nadav Amit <[email protected]>
To detect deferred TLB flushes in fine granularity, we need to keep track on the completed TLB flush generation for each mm.
Quoted 8 lines
Quoted 6 lines
Add logic to track for each mm the
tlb_gen_completed, which tracks the completed TLB generation. It is the arch responsibility to callmark_mm_tlb_gen_done()whenever a TLB flush is completed.Start the generation numbers from 1 instead of 0. This would allow later to detect whether flushes of a certain generation were completed.
Can you elaborate on how this helps?
I guess it should have gone to patch 15.
The relevant code it interacts with is in
read_defer_tlb_flush_gen(). It allows to use a single check to see “outdated” deferred TLB gen. Initiallytlb->defer_genis zero. We are going to doinc_mm_tlb_gen()both on the first time we defer TLB entries and whenever we seemm_genis newer thantlb->defer_gen:(quoted diff) +15
-0+ mm_gen = atomic64_read(&mm->tlb_gen); + + /* + * This condition checks for both first deferred TLB flush and for other + * TLB pending or executed TLB flushes after the last table that we + * updated. In the latter case, we are going to skip a generation, which + * would lead to a full TLB flush. This should therefore not cause + * correctness issues, and should not induce overheads, since anyhow in + * TLB storms it is better to perform full TLB flush. + */ + if (mm_gen != tlb->defer_gen) { + VM_BUG_ON(mm_gen < tlb->defer_gen); + + tlb->defer_gen = inc_mm_tlb_gen(mm); + }Quoted 3 lines
Quoted 7 lines
Quoted 4 lines
I think you should document that
tlb_gen_completedonly means that no outdated TLB entries will be observably used. In the x86 implementation it's possible for older TLB entries to still exist, unused, in TLBs of cpus running other mms.You mean entries that be later flushed during
switch_mm_irqs_off(), right? I think that overall my comments need some work. Yes.
That's exactly what I mean.
Quoted 7 lines
Quoted 2 lines
How does this work with
arch_tlbbatch_flush()?
completed_genis not updated byarch_tlbbatch_flush(), since I couldn’t find a way to combine them.completed_genmight not catch up withtlb_genin this case until another TLB flush takes place. I do not see correctness issue, but it might result in redundant TLB flush.
Please at least document this.
FWIW, arch_tlbbatch_flush() is gross. I'm not convinced it's really supportable with proper broadcast invalidation. I suppose we could remove it or explicitly track the set of mms that need flushing.
Quoted 124 lines
Quoted 116 lines
Quoted 58 lines
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>---
arch/x86/mm/tlb.c | 10 ++++++++++ include/asm-generic/tlb.h | 33 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 15 ++++++++++++++- 3 files changed, 57 insertions(+), 1 deletion(-)arch/x86/mm/tlb.c +10
-0@@ -14,6 +14,7 @@#include <asm/nospec-branch.h> #include <asm/cache.h> #include <asm/apic.h> +#include <asm/tlb.h> #include "mm_internal.h" @@ -915,6 +916,9 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), info); +/* Update the completed generation */ +mark_mm_tlb_gen_done (mm, new_tlb_gen); +put_flush_tlb_info ();put_cpu (); }@@ -1147,6 +1151,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)cpumask_clear (&batch->cpumask); +/* +* We cannot call mark_mm_tlb_gen_done() since we do not know which +* mm's should be flushed. This may lead to some unwarranted TLB +* flushes, but not to correction problems. +*/ +put_cpu (); }include/asm-generic/tlb.h +0
-0include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 521
Quoted 9 lines
include/asm-generic/tlb.h +6
-0@@ -513,6 +513,39 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm }#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + +/* +* Helper function to update a generation to have a new value, as long as new +* value is greater or equal to gen. +*/ I read this a couple of times, and I don't understand it. How about:
Helper function to atomically set *gen =
max(*gen, new_gen)Quoted 35 lines
include/asm-generic/tlb.h +27
-0+static inline void tlb_update_generation (atomic64_t *gen,u64 new_gen) +{ +u64 cur_gen =atomic64_read (gen); + +while (cur_gen < new_gen) { +u64 old_gen =atomic64_cmpxchg (gen, cur_gen, new_gen); + +/* Check if we succeeded in the cmpxchg */ +if (likely (cur_gen == old_gen)) +break ; + + cur_gen = old_gen; + }; +} + + +static inline void mark_mm_tlb_gen_done (struct mm_struct *mm,u64 gen) +{ +/* +* Update the completed generation to the new generation if the new +* generation is greater than the previous one. +*/ +tlb_update_generation (&mm->tlb_gen_completed, gen); +} + +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. include/linux/mm_types.h +0
-0include/linux/mm_types.h struct mm_struct { · line 576
Quoted 6 lines
include/linux/mm_types.h +3
-0@@ -571,6 +571,13 @@ struct mm_struct {* This is not used on Xen PV. */ atomic64_t tlb_gen; + +/* +* TLB generation which is guarnateed to be flushed, including guaranteed
Quoted 5 lines
include/linux/mm_types.h struct mm_struct { · line 579
Quoted 3 lines
include/linux/mm_types.h +3
-0+* all the PTE changes that were performed before tlb_gen was +* incremented. +*/ I will defer judgment to future patches before I believe that this isn't racy :)
Fair enough. Thanks for the review.
Peter Zijlstra
maintainerRe: [RFC 08/20] mm: store completed TLB generation
In reply to Nadav AmitQuoted 4 lines · Nadav Amit
include/asm-generic/tlb.h +4
-0+static inline void tlb_update_generation (atomic64_t *gen,u64 new_gen) +{ +u64 cur_gen =atomic64_read (gen); +
include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 535
Quoted 10 lines
include/asm-generic/tlb.h +10
-0+while (cur_gen < new_gen) { +u64 old_gen =atomic64_cmpxchg (gen, cur_gen, new_gen); + +/* Check if we succeeded in the cmpxchg */ +if (likely (cur_gen == old_gen)) +break ; + + cur_gen = old_gen; + }; +}
u64 cur_gen = atomic64_read(gen); while (cur_gen < new_gen && !atomic64_try_cmpxchg(gen, &cur_gen, new_gen)) ;
RFC 9/20 mm: create pte/pmd_tlb_flush_pending()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
In preparation for fine(r) granularity, introduce pte_tlb_flush_pending() and pmd_tlb_flush_pending(). Right now the function directs to mm_tlb_flush_pending().
Change pte_accessible() to provide the vma as well.
No functional change. Next patches will use this information on architectures that use per-table deferred TLB tracking.
--- arch/arm/include/asm/pgtable.h | 4 +++- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/sparc/include/asm/pgtable_64.h | 9 ++++++--- arch/sparc/mm/init_64.c | 2 +- arch/x86/include/asm/pgtable.h | 7 +++---- include/linux/mm_types.h | 10 ++++++++++ include/linux/pgtable.h | 2 +- mm/huge_memory.c | 2 +- mm/ksm.c | 2 +- mm/pgtable-generic.c | 2 +- 10 files changed, 29 insertions(+), 15 deletions(-)
arch/arm/include/asm/pgtable.h +3 -1
@@ -190,7 +190,9 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)#define pte_none (pte) (!pte_val(pte))#define pte_present (pte) (pte_isset((pte), L_PTE_PRESENT))#define pte_valid (pte) (pte_isset((pte), L_PTE_VALID)) -#define pte_accessible (mm, pte) (mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid(pte)) +#define pte_accessible (vma, pte) \ + (pte_tlb_flush_pending(vma, pte) ? \ + pte_present(*pte) : pte_valid(*pte))#define pte_write (pte) (pte_isclear((pte), L_PTE_RDONLY))#define pte_dirty (pte) (pte_isset((pte), L_PTE_DIRTY))#define pte_young (pte) (pte_isset((pte), L_PTE_YOUNG))
arch/arm64/include/asm/pgtable.h +2 -2
@@ -126,8 +126,8 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];* flag, since ptep_clear_flush_young() elides a DSB when invalidating the * TLB. */ -#define pte_accessible (mm, pte) \ - (mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid(pte)) +#define pte_accessible (vma, pte) \ + (pte_tlb_flush_pending(vma, pte) ? pte_present(*pte) : pte_valid(*pte))/* * p??_access_permitted() is true for valid user mappings (subject to the
arch/sparc/include/asm/pgtable_64.h +6 -3
@@ -673,9 +673,9 @@ static inline unsigned long pte_present(pte_t pte) }#define pte_accessible pte_accessible -static inline unsigned long pte_accessible (struct mm_struct *mm,pte_t a) +static inline unsigned long pte_accessible (struct vm_area_struct *vma,pte_t *a) { -return pte_val (a) & _PAGE_VALID; +return pte_val (*a) & _PAGE_VALID; }static inline unsigned long pte_special (pte_t pte)
@@ -906,8 +906,11 @@ static void maybe_tlb_batch_add(struct mm_struct *mm, unsigned long vaddr,* * SUN4V NOTE: _PAGE_VALID is the same value in both the SUN4U * and SUN4V pte layout, so this inline test is fine. +* +* The vma is not propagated to this point, but it is not used by +* sparc's pte_accessible(). We therefore provide NULL. */ -if (likely (mm != &init_mm) &&pte_accessible (mm, orig)) +if (likely (mm != &init_mm) &&pte_accessible (NULL , ptep))tlb_batch_add (mm, vaddr, ptep, orig, fullmm, hugepage_shift); }
arch/sparc/mm/init_64.c +1 -1
@@ -404,7 +404,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t * mm = vma->vm_mm;/* Don't insert a non-valid PTE into the TSB, we'll deadlock. */ -if (!pte_accessible (mm, pte)) +if (!pte_accessible (vma, ptep))return ;spin_lock_irqsave (&mm->context.lock, flags);
arch/x86/include/asm/pgtable.h +3 -4
@@ -775,13 +775,12 @@ static inline int pte_devmap(pte_t a)#endif #define pte_accessible pte_accessible -static inline bool pte_accessible (struct mm_struct *mm,pte_t a) +static inline bool pte_accessible (struct vm_area_struct *vma,pte_t *a) { -if (pte_flags (a) & _PAGE_PRESENT) +if (pte_flags (*a) & _PAGE_PRESENT)return true; -if ((pte_flags (a) & _PAGE_PROTNONE) && -mm_tlb_flush_pending (mm)) +if ((pte_flags (*a) & _PAGE_PROTNONE) &&pte_tlb_flush_pending (vma, a))return true;return false;
include/linux/mm_types.h +10 -0
@@ -682,6 +682,16 @@ static inline bool mm_tlb_flush_pending(struct mm_struct *mm)return atomic_read (&mm->tlb_flush_pending); } +static inline bool pte_tlb_flush_pending (struct vm_area_struct *vma,pte_t *pte) +{ +return mm_tlb_flush_pending (vma->vm_mm); +} + +static inline bool pmd_tlb_flush_pending (struct vm_area_struct *vma,pmd_t *pmd) +{ +return mm_tlb_flush_pending (vma->vm_mm); +} +static inline bool mm_tlb_flush_nested (struct mm_struct *mm) {/*
include/linux/pgtable.h +1 -1
@@ -725,7 +725,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct page *page)#endif #ifndef pte_accessible -# define pte_accessible (mm, pte) ((void)(pte), 1) +# define pte_accessible (vma, pte) ((void)(pte), 1)#endif #ifndef flush_tlb_fix_spurious_fault
mm/huge_memory.c +1 -1
@@ -1514,7 +1514,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)* We are not sure a pending tlb flush here is for a huge page * mapping or not. Hence use the tlb range variant */ -if (mm_tlb_flush_pending (vma->vm_mm)) { +if (pmd_tlb_flush_pending (vma, vmf->pmd)) {flush_tlb_range (vma, haddr, haddr +HPAGE_PMD_SIZE );/* * change_huge_pmd() released the pmd lock before
mm/ksm.c +1 -1
@@ -1060,7 +1060,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page,if (pte_write (*pvmw.pte) ||pte_dirty (*pvmw.pte) || (pte_protnone (*pvmw.pte) &&pte_savedwrite (*pvmw.pte)) || -mm_tlb_flush_pending (mm)) { +pte_tlb_flush_pending (vma, pvmw.pte)) {pte_t entry; swapped =PageSwapCache (page);
mm/pgtable-generic.c +1 -1
@@ -93,7 +93,7 @@ pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,struct mm_struct *mm = (vma)->vm_mm;pte_t pte; pte =ptep_get_and_clear (mm, address, ptep); -if (pte_accessible (mm, pte)) +if (pte_accessible (vma, ptep))flush_tlb_page (vma, address);return pte; }
Signature · 1 line
2.25.1
RFC 10/20 mm: add pte_to_page()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Add a pte_to_page(), which is similar to pmd_to_page, which will be used later.
Inline pmd_to_page() as well.
--- include/linux/mm.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
include/linux/mm.h +8 -1
@@ -2161,6 +2161,13 @@ static inline spinlock_t *ptlock_ptr(struct page *page) }#endif /* ALLOC_SPLIT_PTLOCKS */ +static inline struct page *pte_to_page (pte_t *pte) +{ +unsigned long mask = ~(PTRS_PER_PTE *sizeof (pte_t) -1 ); + +return virt_to_page ((void *)((unsigned long ) pte & mask)); +} +static inline spinlock_t *pte_lockptr (struct mm_struct *mm,pmd_t *pmd) {return ptlock_ptr (pmd_page (*pmd));
@@ -2246,7 +2253,7 @@ static inline void pgtable_pte_page_dtor(struct page *page)#if USE_SPLIT_PMD_PTLOCKS -static struct page *pmd_to_page (pmd_t *pmd) +static inline struct page *pmd_to_page (pmd_t *pmd) {unsigned long mask = ~(PTRS_PER_PMD *sizeof (pmd_t) -1 );return virt_to_page ((void *)((unsigned long ) pmd & mask));
Signature · 1 line
2.25.1
RFC 11/20 mm/tlb: remove arch-specific tlb_start/end_vma()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Architecture-specific tlb_start_vma() and tlb_end_vma() seem unnecessary. They are currently used for:
1. Avoid per-VMA TLB flushes. This can be determined by introducing a new config option.
2. Avoid saving information on the vma that is being flushed. Saving this information, even for architectures that do not need it, is cheap and we will need it for per-VMA deferred TLB flushing.
3. Avoid calling flush_cache_range().
Remove the architecture specific tlb_start_vma() and tlb_end_vma() in the following manner, corresponding to the previous requirements:
1. Introduce a new config option - ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING - to allow architectures to define whether they want aggressive TLB flush batching (instead of flushing mappings of each VMA separately).
2. Save information on the vma regardless of architecture. Saving this information should have negligible overhead, and they will be needed for fine granularity TLB flushes.
3. flush_cache_range() is anyhow not defined for the architectures that implement tlb_start/end_vma().
No functional change intended.
--- arch/csky/Kconfig | 1 + arch/csky/include/asm/tlb.h | 12 ------------ arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/tlb.h | 2 -- arch/s390/Kconfig | 1 + arch/s390/include/asm/tlb.h | 3 --- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/tlb_64.h | 2 -- arch/x86/Kconfig | 1 + arch/x86/include/asm/tlb.h | 3 --- include/asm-generic/tlb.h | 15 +++++---------- init/Kconfig | 8 ++++++++ 12 files changed, 18 insertions(+), 32 deletions(-)
arch/csky/Kconfig +1 -0
@@ -8,6 +8,7 @@ config CSKY select ARCH_HAS_SYNC_DMA_FOR_DEVICE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS if NR_CPUS>2 + select ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING select ARCH_WANT_FRAME_POINTERS if !CPU_CK610 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select COMMON_CLK
arch/csky/include/asm/tlb.h +0 -12
@@ -6,18 +6,6 @@#include <asm/cacheflush.h> -#define tlb_start_vma (tlb, vma) \ - do { \ - if (!(tlb)->fullmm) \ - flush_cache_range(vma, (vma)->vm_start, (vma)->vm_end); \ - } while (0) - -#define tlb_end_vma (tlb, vma) \ - do { \ - if (!(tlb)->fullmm) \ - flush_tlb_range(vma, (vma)->vm_start, (vma)->vm_end); \ - } while (0) -#define tlb_flush (tlb) flush_tlb_mm((tlb)->mm)#include <asm-generic/tlb.h>
arch/powerpc/Kconfig +1 -0
@@ -151,6 +151,7 @@ config PPC select ARCH_USE_CMPXCHG_LOCKREF if PPC64 select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS + select ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN
arch/powerpc/include/asm/tlb.h +0 -2
@@ -19,8 +19,6 @@#include <linux/pagemap.h> -#define tlb_start_vma (tlb, vma) do { } while (0) -#define tlb_end_vma (tlb, vma) do { } while (0)#define __tlb_remove_tlb_entry __tlb_remove_tlb_entry#define tlb_flush tlb_flush
arch/s390/Kconfig +1 -0
@@ -113,6 +113,7 @@ config S390 select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF select ARCH_WANTS_DYNAMIC_TASK_STRUCT + select ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION select BUILDTIME_TABLE_SORT
arch/s390/include/asm/tlb.h +0 -3
@@ -27,9 +27,6 @@ static inline void tlb_flush(struct mmu_gather *tlb);static inline bool __tlb_remove_page_size (struct mmu_gather *tlb,struct page *page,int page_size); -#define tlb_start_vma (tlb, vma) do { } while (0) -#define tlb_end_vma (tlb, vma) do { } while (0) -#define tlb_flush tlb_flush#define pte_free_tlb pte_free_tlb#define pmd_free_tlb pmd_free_tlb
arch/sparc/Kconfig +1 -0
@@ -51,6 +51,7 @@ config SPARC select NEED_DMA_MAP_STATE select NEED_SG_DMA_LENGTH select SET_FS + select ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING config SPARC32 def_bool !64BIT
arch/sparc/include/asm/tlb_64.h +0 -2
@@ -22,8 +22,6 @@ void smp_flush_tlb_mm(struct mm_struct *mm);void __flush_tlb_pending (unsigned long ,unsigned long ,unsigned long *);void flush_tlb_pending (void ); -#define tlb_start_vma (tlb, vma) do { } while (0) -#define tlb_end_vma (tlb, vma) do { } while (0)#define tlb_flush (tlb) flush_tlb_pending()/*
arch/x86/Kconfig +1 -0
@@ -101,6 +101,7 @@ config X86 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_USE_SYM_ANNOTATIONS + select ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_DEFAULT_BPF_JIT if X86_64 select ARCH_WANTS_DYNAMIC_TASK_STRUCT
arch/x86/include/asm/tlb.h +0 -3
@@ -2,9 +2,6 @@#ifndef _ASM_X86_TLB_H#define _ASM_X86_TLB_H -#define tlb_start_vma (tlb, vma) do { } while (0) -#define tlb_end_vma (tlb, vma) do { } while (0) -#define tlb_flush tlb_flushstatic inline void tlb_flush (struct mmu_gather *tlb);
include/asm-generic/tlb.h +5 -10
@@ -334,8 +334,8 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb)#ifdef CONFIG_MMU_GATHER_NO_RANGE -#if defined(tlb_flush) || defined(tlb_start_vma) || defined(tlb_end_vma) -#error MMU_GATHER_NO_RANGE relies on default tlb_flush(), tlb_start_vma() and tlb_end_vma() +#if defined(tlb_flush) +#error MMU_GATHER_NO_RANGE relies on default tlb_flush()#endif /*
@@ -362,10 +362,6 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm#ifndef tlb_flush -#if defined(tlb_start_vma) || defined(tlb_end_vma) -#error Default tlb_flush() relies on default tlb_start_vma() and tlb_end_vma() -#endif -/* * When an architecture does not provide its own tlb_flush() implementation * but does have a reasonably efficient flush_vma_range() implementation
@@ -486,7 +482,6 @@ static inline unsigned long tlb_get_unmap_size(struct mmu_gather *tlb)* case where we're doing a full MM flush. When we're doing a munmap, * the vmas are adjusted to only cover the region to be torn down. */ -#ifndef tlb_start_vmastatic inline void tlb_start_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) {if (tlb->fullmm)
@@ -495,14 +490,15 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct *tlb_update_vma_flags (tlb, vma);flush_cache_range (vma, vma->vm_start, vma->vm_end); } -#endif -#ifndef tlb_end_vmastatic inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) {if (tlb->fullmm)return ; +if (IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )) +return ; +/* * Do a TLB flush and reset the range at VMA boundaries; this avoids * the ranges growing with the unused space between consecutive VMAs,
@@ -511,7 +507,6 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm*/ tlb_flush_mmu_tlbonly (tlb); } -#endif #ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS
init/Kconfig +8 -0
@@ -849,6 +849,14 @@ config ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH config ARCH_HAS_TLB_GENERATIONS bool +# +# For architectures that prefer to batch TLB flushes aggressively, i.e., +# not to flush after changing or removing each VMA. The architecture must +# provide its own tlb_flush() function. +config ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING + bool + depends on !CONFIG_MMU_GATHER_NO_GATHER + config CC_HAS_INT128 def_bool !$(cc-option,$(m64-flag) -D__SIZEOF_INT128__=0) && 64BIT
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Quoted 16 lines · Nadav Amit
include/asm-generic/tlb.h +2
-2@@ -334,8 +334,8 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb)#ifdef CONFIG_MMU_GATHER_NO_RANGE -#if defined(tlb_flush) || defined(tlb_start_vma) || defined(tlb_end_vma) -#error MMU_GATHER_NO_RANGE relies on default tlb_flush(), tlb_start_vma() and tlb_end_vma() +#if defined(tlb_flush) +#error MMU_GATHER_NO_RANGE relies on default tlb_flush()#endif /*
include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 367
Quoted 6 lines
include/asm-generic/tlb.h +0
-3@@ -362,10 +362,6 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm#ifndef tlb_flush -#if defined(tlb_start_vma) || defined(tlb_end_vma) -#error Default tlb_flush() relies on default tlb_start_vma() and tlb_end_vma() -#endif
#ifdef CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING #error ....
#endif
goes here...
include/asm-generic/tlb.h static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct * · line 500
Quoted 7 lines
include/asm-generic/tlb.h +2
-0static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) {if (tlb->fullmm)return ; +if (IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )) +return ;
Also, can you please stick to the CONFIG_MMU_GATHER_* namespace?
I also don't think AGRESSIVE_FLUSH_BATCHING quite captures what it does. How about:
CONFIG_MMU_GATHER_NO_PER_VMA_FLUSH
?
Nicholas Piggin
maintainerRe: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Excerpts from Peter Zijlstra's message of February 1, 2021 10:09 pm:
Quoted 42 lines · Nadav Amit
Quoted 17 lines
include/asm-generic/tlb.h +2
-2@@ -334,8 +334,8 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb)#ifdef CONFIG_MMU_GATHER_NO_RANGE -#if defined(tlb_flush) || defined(tlb_start_vma) || defined(tlb_end_vma) -#error MMU_GATHER_NO_RANGE relies on default tlb_flush(), tlb_start_vma() and tlb_end_vma() +#if defined(tlb_flush) +#error MMU_GATHER_NO_RANGE relies on default tlb_flush()#endif /* include/asm-generic/tlb.h static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm · line 367
Quoted 6 lines
include/asm-generic/tlb.h +0
-3@@ -362,10 +362,6 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm#ifndef tlb_flush -#if defined(tlb_start_vma) || defined(tlb_end_vma) -#error Default tlb_flush() relies on default tlb_start_vma() and tlb_end_vma() -#endif #ifdef
CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING#error ....
#endifgoes here...
include/asm-generic/tlb.h static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct * · line 500
Quoted 7 lines
include/asm-generic/tlb.h +2
-0static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) {if (tlb->fullmm)return ; +if (IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )) +return ;Also, can you please stick to the CONFIG_MMU_GATHER_* namespace?
Quoted 4 lines
I also don't think
AGRESSIVE_FLUSH_BATCHINGquite captures what it does. How about:
CONFIG_MMU_GATHER_NO_PER_VMA_FLUSH
Yes please, have to have descriptive names.
I didn't quite see why this was much of an improvement though. Maybe follow up patches take advantage of it? I didn't see how they all fit together.
Thanks,
Nick
Re: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Quoted 9 lines · Nicholas Piggin
Excerpts from Peter Zijlstra's message of February 1, 2021 10:09 pm:
Quoted 4 lines
I also don't think
AGRESSIVE_FLUSH_BATCHINGquite captures what it does. How about:
CONFIG_MMU_GATHER_NO_PER_VMA_FLUSHYes please, have to have descriptive names.
Point taken. I will fix it.
Quoted 4 lines
I didn't quite see why this was much of an improvement though. Maybe follow up patches take advantage of it? I didn't see how they all fit together.
They do, but I realized as I said in other emails that I have a serious bug in the deferred invalidation scheme.
Having said that, I think there is an advantage of having an explicit config option instead of relying on whether tlb_end_vma is defined. For instance, Arm does not define tlb_end_vma, and consequently it flushes the TLB after each VMA. I suspect it is not intentional.
Peter Zijlstra
maintainerRe: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Quoted 2 lines · Nadav Amit
Arm does not define
tlb_end_vma, and consequently it flushes the TLB after each VMA. I suspect it is not intentional.
ARM is one of those that look at the VM_EXEC bit to explicitly flush ITLB IIRC, so it has to.
Re: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Nadav Amit
Arm does not define
tlb_end_vma, and consequently it flushes the TLB after each VMA. I suspect it is not intentional.ARM is one of those that look at the
VM_EXECbit to explicitly flush ITLB IIRC, so it has to.
Hmm… I don’t think Arm is doing that. At least arm64 does not use the default tlb_flush(), and it does not seem to consider VM_EXEC (at least in this path):
static inline void tlb_flush(struct mmu_gather *tlb)
{
struct vm_area_struct vma = TLB_FLUSH_VMA(tlb->mm, 0);
bool last_level = !tlb->freed_tables;
unsigned long stride = tlb_get_unmap_size(tlb);
int tlb_level = tlb_get_level(tlb);
/*
* If we're tearing down the address space then we only care about
* invalidating the walk-cache, since the ASID allocator won't
* reallocate our ASID without invalidating the entire TLB.
*/
if (tlb->mm_exiting) {
if (!last_level)
flush_tlb_mm(tlb->mm);
return;
}
__flush_tlb_range(&vma, tlb->start, tlb->end, stride,
last_level, tlb_level);
}Peter Zijlstra
maintainerRe: [RFC 11/20] mm/tlb: remove arch-specific tlb_start/end_vma()
Quoted 6 lines · Nadav Amit
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Nadav Amit
Arm does not define
tlb_end_vma, and consequently it flushes the TLB after each VMA. I suspect it is not intentional.
Quoted 7 lines
Quoted 2 lines
ARM is one of those that look at the
VM_EXECbit to explicitly flush ITLB IIRC, so it has to.Hmm… I don’t think Arm is doing that. At least arm64 does not use the default
tlb_flush(), and it does not seem to considerVM_EXEC(at least in this path):
ARM != ARM64. ARM certainly does, but you're right, I don't think ARM64 does this.
RFC 12/20 mm/tlb: save the VMA that is flushed during tlb_start_vma()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Certain architectures need information about the vma that is about to be flushed. Currently, an artificial vma is constructed using the original vma infromation. Instead of saving the flags, record the vma during tlb_start_vma() and use this vma when calling flush_tlb_range().
Record the vma unconditionally as it would be needed for per-VMA deferred TLB flush tracking and the overhead of tracking it unconditionally should be negligible.
--- include/asm-generic/tlb.h | 56 +++++++++++++-------------------------- 1 file changed, 19 insertions(+), 37 deletions(-)
include/asm-generic/tlb.h +19 -37
@@ -252,6 +252,13 @@ extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,struct mmu_gather {struct mm_struct *mm; +/* +* The current vma. This information is changing upon tlb_start_vma() +* and is therefore only valid between tlb_start_vma() and tlb_end_vma() +* calls. +*/ +struct vm_area_struct *vma; +#ifdef CONFIG_MMU_GATHER_TABLE_FREE struct mmu_table_batch *batch;#endif
@@ -283,12 +290,6 @@ struct mmu_gather { unsigned int cleared_puds : 1; unsigned int cleared_p4ds : 1; -/* -* tracks VM_EXEC | VM_HUGETLB in tlb_start_vma -*/ - unsigned int vma_exec : 1; - unsigned int vma_huge : 1; -unsigned int batch_count;#ifndef CONFIG_MMU_GATHER_NO_GATHER
@@ -352,10 +353,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm(tlb->mm); } -static inline void -tlb_update_vma_flags (struct mmu_gather *tlb,struct vm_area_struct *vma) { } - -#define tlb_end_vma tlb_end_vmastatic inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { }#else /* CONFIG_MMU_GATHER_NO_RANGE */
@@ -364,7 +361,7 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm/* * When an architecture does not provide its own tlb_flush() implementation -* but does have a reasonably efficient flush_vma_range() implementation +* but does have a reasonably efficient flush_tlb_range() implementation * use that. */ static inline void tlb_flush (struct mmu_gather *tlb)
@@ -372,38 +369,20 @@ static inline void tlb_flush(struct mmu_gather *tlb)if (tlb->fullmm || tlb->need_flush_all) {flush_tlb_mm (tlb->mm); }else if (tlb->end) { -struct vm_area_struct vma = { - .vm_mm = tlb->mm, - .vm_flags = (tlb->vma_exec ?VM_EXEC :0 ) | - (tlb->vma_huge ?VM_HUGETLB :0 ), - }; - -flush_tlb_range (&vma, tlb->start, tlb->end); +VM_BUG_ON (!tlb->vma); +flush_tlb_range (tlb->vma, tlb->start, tlb->end); } }static inline void -tlb_update_vma_flags (struct mmu_gather *tlb,struct vm_area_struct *vma) +tlb_update_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { -/* -* flush_tlb_range() implementations that look at VM_HUGETLB (tile, -* mips-4k) flush only large pages. -* -* flush_tlb_range() implementations that flush I-TLB also flush D-TLB -* (tile, xtensa, arm), so it's ok to just add VM_EXEC to an existing -* range. -* -* We rely on tlb_end_vma() to issue a flush, such that when we reset -* these values the batch is empty. -*/ - tlb->vma_huge =is_vm_hugetlb_page (vma); - tlb->vma_exec = !!(vma->vm_flags &VM_EXEC ); + tlb->vma = vma; } -#else static inline void -tlb_update_vma_flags (struct mmu_gather *tlb,struct vm_area_struct *vma) { } +tlb_update_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) { }#endif
@@ -487,17 +466,17 @@ static inline void tlb_start_vma(struct mmu_gather *tlb, struct vm_area_struct *if (tlb->fullmm)return ; -tlb_update_vma_flags (tlb, vma); +tlb_update_vma (tlb, vma);flush_cache_range (vma, vma->vm_start, vma->vm_end); }static inline void tlb_end_vma (struct mmu_gather *tlb,struct vm_area_struct *vma) {if (tlb->fullmm) -return ; + goto out;if (IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )) -return ; + goto out;/* * Do a TLB flush and reset the range at VMA boundaries; this avoids
@@ -506,6 +485,9 @@ static inline void tlb_end_vma(struct mmu_gather *tlb, struct vm_area_struct *vm* this. */ tlb_flush_mmu_tlbonly (tlb); +out: +/* Reset the VMA as a precaution. */ +tlb_update_vma (tlb,NULL ); }#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [RFC 12/20] mm/tlb: save the VMA that is flushed during tlb_start_vma()
Quoted 19 lines · Nadav Amit
include/asm-generic/tlb.h +0
-7@@ -283,12 +290,6 @@ struct mmu_gather { unsigned int cleared_puds : 1; unsigned int cleared_p4ds : 1; -/* -* tracks VM_EXEC | VM_HUGETLB in tlb_start_vma -*/ - unsigned int vma_exec : 1; - unsigned int vma_huge : 1; -unsigned int batch_count;#ifndef CONFIG_MMU_GATHER_NO_GATHER @@ -372,38 +369,20 @@ static inline void tlb_flush(struct mmu_gather *tlb)if (tlb->fullmm || tlb->need_flush_all) {flush_tlb_mm (tlb->mm); }else if (tlb->end) { -struct vm_area_struct vma = {
include/asm-generic/tlb.h static inline void tlb_flush(struct mmu_gather *tlb) · line 375
Quoted 10 lines
include/asm-generic/tlb.h +2
-6- .vm_mm = tlb->mm, - .vm_flags = (tlb->vma_exec ? VM_EXEC : 0) | - (tlb->vma_huge ? VM_HUGETLB : 0), - }; - -flush_tlb_range (&vma, tlb->start, tlb->end); +VM_BUG_ON (!tlb->vma); +flush_tlb_range (tlb->vma, tlb->start, tlb->end); } }
I don't much like this, and I think this is a step in the wrong direction.
The idea is to extend the tlb_{remove,flush}_*() API to provide the needed information to do TLB flushing. In fact, I think tlb_remove_huge*() is already sufficient to set the VM_EXEC 'hint'. We just don't have anything that covers the EXEC thing.
(also, I suspect the page_size crud we have also covers that)
Constructing a fake vma very much ensures arch tlb routines don't go about and look at anything else either.
Quoted 7 lines
include/asm-generic/tlb.h +1
-5+tlb_update_vma(struct mmu_gather *tlb, struct vm_area_struct *vma) { -/* -* flush_tlb_range() implementations that look at VM_HUGETLB (tile, -* mips-4k) flush only large pages. -* -* flush_tlb_range() implementations that flush I-TLB also flush D-TLB
include/asm-generic/tlb.h static inline void tlb_flush(struct mmu_gather *tlb) · line 381
Quoted 10 lines
include/asm-generic/tlb.h +1
-8-* (tile, xtensa, arm), so it's ok to just add VM_EXEC to an existing -* range. -* -* We rely on tlb_end_vma() to issue a flush, such that when we reset -* these values the batch is empty. -*/ - tlb->vma_huge =is_vm_hugetlb_page (vma); - tlb->vma_exec = !!(vma->vm_flags &VM_EXEC ); + tlb->vma = vma; }
And you're also removing the useful information about arch tlb flush functions.
RFC 13/20 mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Introduce tlb_start_ptes() and tlb_end_ptes() which would be called before and after PTEs are updated and TLB flushes are deferred. This will be later be used for fine granualrity deferred TLB flushing detection.
In the meanwhile, move flush_tlb_batched_pending() into tlb_start_ptes(). It was not called from mapping_dirty_helpers by wp_pte() and clean_record_pte(), which might be a bug.
No additional functional change is intended.
--- fs/proc/task_mmu.c | 2 ++ include/asm-generic/tlb.h | 18 ++++++++++++++++++ mm/madvise.c | 6 ++++-- mm/mapping_dirty_helpers.c | 15 +++++++++++++-- mm/memory.c | 2 ++ mm/mprotect.c | 3 ++- 6 files changed, 41 insertions(+), 5 deletions(-)
fs/proc/task_mmu.c +2 -0
@@ -1168,6 +1168,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,return 0 ; pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); +tlb_start_ptes (&cp->tlb);for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;
@@ -1190,6 +1191,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );ClearPageReferenced (page); } +tlb_end_ptes (&cp->tlb);pte_unmap_unlock (pte -1 , ptl);cond_resched ();return 0 ;
include/asm-generic/tlb.h +18 -0
@@ -58,6 +58,11 @@* Defaults to flushing at tlb_end_vma() to reset the range; helps when * there's large holes between the VMAs. * +* - tlb_start_ptes() / tlb_end_ptes; makr the start / end of PTEs change. +* +* Does internal accounting to allow fine(r) granularity checks for +* pte_accessible() on certain configuration. +* * - tlb_remove_table() * * tlb_remove_table() is the basic primitive to free page-table directories
@@ -373,6 +378,10 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_range(tlb->vma, tlb->start, tlb->end); } } +#endif + +#if __is_defined (tlb_flush) || \ +IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )static inline void tlb_update_vma (struct mmu_gather *tlb,struct vm_area_struct *vma)
@@ -523,6 +532,15 @@ static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen)#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +#define tlb_start_ptes (tlb) \ + do { \ + struct mmu_gather *_tlb = (tlb); \ + \ + flush_tlb_batched_pending(_tlb->mm); \ + } while (0) + +static inline void tlb_end_ptes (struct mmu_gather *tlb) { } +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*.
mm/madvise.c +4 -2
@@ -392,7 +392,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,#endif tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;
@@ -468,6 +468,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);if (pageout)reclaim_pages (&page_list);
@@ -588,7 +589,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;
@@ -692,6 +693,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,add_mm_counter (mm,MM_SWAPENTS , nr_swap); }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);cond_resched (); next:
mm/mapping_dirty_helpers.c +13 -2
@@ -6,6 +6,8 @@#include <asm/cacheflush.h> #include <asm/tlb.h> +#include "internal.h" +/** * struct wp_walk - Private struct for pagetable walk callbacks * @range: Range for mmu notifiers
@@ -36,7 +38,10 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,pte_t ptent = *pte;if (pte_write (ptent)) { -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_wrprotect (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent);
@@ -44,6 +49,7 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,if (pte_may_need_flush (old_pte, ptent))tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb); }return 0 ;
@@ -94,13 +100,18 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,if (pte_dirty (ptent)) {pgoff_t pgoff = ((addr - walk->vma->vm_start) >>PAGE_SHIFT ) + walk->vma->vm_pgoff - cwalk->bitmap_pgoff; -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_mkclean (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++;tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb);__set_bit (pgoff, cwalk->bitmap); cwalk->start =min (cwalk->start, pgoff);
mm/memory.c +2 -0
@@ -1221,6 +1221,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,init_rss_vec (rss); start_pte =pte_offset_map_lock (mm, pmd, addr, &ptl); pte = start_pte; +tlb_start_ptes (tlb);flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); do {
@@ -1314,6 +1315,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,add_mm_rss_vec (mm, rss);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);/* Do the actual TLB flush before dropping ptl */ if (force_flush)tlb_flush_mmu_tlbonly (tlb);
mm/mprotect.c +2 -1
@@ -70,7 +70,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, atomic_read(&vma->vm_mm->mm_users) == 1) target_node =numa_node_id (); -flush_tlb_batched_pending (vma->vm_mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode (); do { oldpte = *pte;
@@ -182,6 +182,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (pte -1 , ptl);return pages;
Signature · 1 line
2.25.1
Re: [RFC 13/20] mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
Quoted 58 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Introduce
tlb_start_ptes()andtlb_end_ptes()which would be called before and after PTEs are updated and TLB flushes are deferred. This will be later be used for fine granualrity deferred TLB flushing detection.In the meanwhile, move
flush_tlb_batched_pending()intotlb_start_ptes(). It was not called frommapping_dirty_helpersbywp_pte()andclean_record_pte(), which might be a bug.No additional functional change is intended.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>--- fs/proc/task_mmu.c | 2 ++ include/asm-generic/tlb.h | 18 ++++++++++++++++++ mm/madvise.c | 6 ++++-- mm/mapping_dirty_helpers.c | 15 +++++++++++++-- mm/memory.c | 2 ++ mm/mprotect.c | 3 ++- 6 files changed, 41 insertions(+), 5 deletions(-)fs/proc/task_mmu.c +2
-0@@ -1168,6 +1168,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,return 0 ; pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); +tlb_start_ptes (&cp->tlb);for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -1190,6 +1191,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );ClearPageReferenced (page); } +tlb_end_ptes (&cp->tlb);pte_unmap_unlock (pte -1 , ptl);cond_resched ();return 0 ;include/asm-generic/tlb.h +0
-0
include/asm-generic/tlb.h line 61
Quoted 4 lines
include/asm-generic/tlb.h +1
-0@@ -58,6 +58,11 @@* Defaults to flushing at tlb_end_vma() to reset the range; helps when * there's large holes between the VMAs. * +* - tlb_start_ptes() / tlb_end_ptes; makr the start / end of PTEs change.
Hello Nadav,
short nid makr/mark
Damian
Quoted 170 lines
include/asm-generic/tlb.h +17
-0+* +* Does internal accounting to allow fine(r) granularity checks for +* pte_accessible() on certain configuration. +* * - tlb_remove_table() * * tlb_remove_table() is the basic primitive to free page-table directories @@ -373,6 +378,10 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_range(tlb->vma, tlb->start, tlb->end); } } +#endif + +#if __is_defined (tlb_flush) || \ +IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )static inline void tlb_update_vma (struct mmu_gather *tlb,struct vm_area_struct *vma)@@ -523,6 +532,15 @@ static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen)#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +#define tlb_start_ptes (tlb) \ + do { \ + struct mmu_gather *_tlb = (tlb); \ + \ + flush_tlb_batched_pending(_tlb->mm); \ + } while (0) + +static inline void tlb_end_ptes (struct mmu_gather *tlb) { } +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. mm/madvise.c +4
-2@@ -392,7 +392,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,#endif tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -468,6 +468,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);if (pageout)reclaim_pages (&page_list);@@ -588,7 +589,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -692,6 +693,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,add_mm_counter (mm,MM_SWAPENTS , nr_swap); }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);cond_resched (); next:mm/mapping_dirty_helpers.c +13
-2@@ -6,6 +6,8 @@#include <asm/cacheflush.h> #include <asm/tlb.h> +#include "internal.h" +/** * struct wp_walk - Private struct for pagetable walk callbacks * @range: Range for mmu notifiers @@ -36,7 +38,10 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,pte_t ptent = *pte;if (pte_write (ptent)) { -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_wrprotect (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent);@@ -44,6 +49,7 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,if (pte_may_need_flush (old_pte, ptent))tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb); }return 0 ;@@ -94,13 +100,18 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,if (pte_dirty (ptent)) {pgoff_t pgoff = ((addr - walk->vma->vm_start) >>PAGE_SHIFT ) + walk->vma->vm_pgoff - cwalk->bitmap_pgoff; -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_mkclean (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++;tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb);__set_bit (pgoff, cwalk->bitmap); cwalk->start =min (cwalk->start, pgoff);mm/memory.c +2
-0@@ -1221,6 +1221,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,init_rss_vec (rss); start_pte =pte_offset_map_lock (mm, pmd, addr, &ptl); pte = start_pte; +tlb_start_ptes (tlb);flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); do {@@ -1314,6 +1315,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,add_mm_rss_vec (mm, rss);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);/* Do the actual TLB flush before dropping ptl */ if (force_flush)tlb_flush_mmu_tlbonly (tlb);mm/mprotect.c +2
-1@@ -70,7 +70,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, atomic_read(&vma->vm_mm->mm_users) == 1) target_node =numa_node_id (); -flush_tlb_batched_pending (vma->vm_mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode (); do { oldpte = *pte;@@ -182,6 +182,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (pte -1 , ptl);return pages;Signature · 1 line
2.25.1
Re: [RFC 13/20] mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
In reply to Nadav AmitQuoted 58 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Introduce
tlb_start_ptes()andtlb_end_ptes()which would be called before and after PTEs are updated and TLB flushes are deferred. This will be later be used for fine granualrity deferred TLB flushing detection.In the meanwhile, move
flush_tlb_batched_pending()intotlb_start_ptes(). It was not called frommapping_dirty_helpersbywp_pte()andclean_record_pte(), which might be a bug.No additional functional change is intended.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>--- fs/proc/task_mmu.c | 2 ++ include/asm-generic/tlb.h | 18 ++++++++++++++++++ mm/madvise.c | 6 ++++-- mm/mapping_dirty_helpers.c | 15 +++++++++++++-- mm/memory.c | 2 ++ mm/mprotect.c | 3 ++- 6 files changed, 41 insertions(+), 5 deletions(-)fs/proc/task_mmu.c +2
-0@@ -1168,6 +1168,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,return 0 ; pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); +tlb_start_ptes (&cp->tlb);for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -1190,6 +1191,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );ClearPageReferenced (page); } +tlb_end_ptes (&cp->tlb);pte_unmap_unlock (pte -1 , ptl);cond_resched ();return 0 ;include/asm-generic/tlb.h +0
-0
include/asm-generic/tlb.h line 61
Quoted 4 lines
include/asm-generic/tlb.h +1
-0@@ -58,6 +58,11 @@* Defaults to flushing at tlb_end_vma() to reset the range; helps when * there's large holes between the VMAs. * +* - tlb_start_ptes() / tlb_end_ptes; makr the start / end of PTEs change.
Hello Nadav,
short nid makr/mark
Damian
Quoted 170 lines
include/asm-generic/tlb.h +17
-0+* +* Does internal accounting to allow fine(r) granularity checks for +* pte_accessible() on certain configuration. +* * - tlb_remove_table() * * tlb_remove_table() is the basic primitive to free page-table directories @@ -373,6 +378,10 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_range(tlb->vma, tlb->start, tlb->end); } } +#endif + +#if __is_defined (tlb_flush) || \ +IS_ENABLED (CONFIG_ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING )static inline void tlb_update_vma (struct mmu_gather *tlb,struct vm_area_struct *vma)@@ -523,6 +532,15 @@ static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen)#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ +#define tlb_start_ptes (tlb) \ + do { \ + struct mmu_gather *_tlb = (tlb); \ + \ + flush_tlb_batched_pending(_tlb->mm); \ + } while (0) + +static inline void tlb_end_ptes (struct mmu_gather *tlb) { } +/* * tlb_flush_{pte|pmd|pud|p4d}_range() adjust the tlb->start and tlb->end, * and set corresponding cleared_*. mm/madvise.c +4
-2@@ -392,7 +392,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,#endif tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -468,6 +468,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);if (pageout)reclaim_pages (&page_list);@@ -588,7 +589,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -692,6 +693,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,add_mm_counter (mm,MM_SWAPENTS , nr_swap); }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);cond_resched (); next:mm/mapping_dirty_helpers.c +13
-2@@ -6,6 +6,8 @@#include <asm/cacheflush.h> #include <asm/tlb.h> +#include "internal.h" +/** * struct wp_walk - Private struct for pagetable walk callbacks * @range: Range for mmu notifiers @@ -36,7 +38,10 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,pte_t ptent = *pte;if (pte_write (ptent)) { -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_wrprotect (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent);@@ -44,6 +49,7 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end,if (pte_may_need_flush (old_pte, ptent))tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb); }return 0 ;@@ -94,13 +100,18 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,if (pte_dirty (ptent)) {pgoff_t pgoff = ((addr - walk->vma->vm_start) >>PAGE_SHIFT ) + walk->vma->vm_pgoff - cwalk->bitmap_pgoff; -pte_t old_pte =ptep_modify_prot_start (walk->vma, addr, pte); +pte_t old_pte; + +tlb_start_ptes (&wpwalk->tlb); + + old_pte =ptep_modify_prot_start (walk->vma, addr, pte); ptent =pte_mkclean (old_pte);ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++;tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_end_ptes (&wpwalk->tlb);__set_bit (pgoff, cwalk->bitmap); cwalk->start =min (cwalk->start, pgoff);mm/memory.c +2
-0@@ -1221,6 +1221,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,init_rss_vec (rss); start_pte =pte_offset_map_lock (mm, pmd, addr, &ptl); pte = start_pte; +tlb_start_ptes (tlb);flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); do {@@ -1314,6 +1315,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,add_mm_rss_vec (mm, rss);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);/* Do the actual TLB flush before dropping ptl */ if (force_flush)tlb_flush_mmu_tlbonly (tlb);mm/mprotect.c +2
-1@@ -70,7 +70,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, atomic_read(&vma->vm_mm->mm_users) == 1) target_node =numa_node_id (); -flush_tlb_batched_pending (vma->vm_mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode (); do { oldpte = *pte;@@ -182,6 +182,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, } }while (pte++, addr +=PAGE_SIZE , addr != end);arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (pte -1 , ptl);return pages;Signature · 1 line
2.25.1
Re: [RFC 13/20] mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
Quoted 68 lines · Damian Tometzki
Quoted 58 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Introduce
tlb_start_ptes()andtlb_end_ptes()which would be called before and after PTEs are updated and TLB flushes are deferred. This will be later be used for fine granualrity deferred TLB flushing detection.In the meanwhile, move
flush_tlb_batched_pending()intotlb_start_ptes(). It was not called frommapping_dirty_helpersbywp_pte()andclean_record_pte(), which might be a bug.No additional functional change is intended.
Signed-off-by Nadav Amit <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Andrew Morton <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Dave Hansen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Will Deacon <[email protected]>Cc Yu Zhao <[email protected]>Cc Nick Piggin <[email protected]>---
fs/proc/task_mmu.c | 2 ++ include/asm-generic/tlb.h | 18 ++++++++++++++++++ mm/madvise.c | 6 ++++-- mm/mapping_dirty_helpers.c | 15 +++++++++++++-- mm/memory.c | 2 ++ mm/mprotect.c | 3 ++- 6 files changed, 41 insertions(+), 5 deletions(-)fs/proc/task_mmu.c +2
-0@@ -1168,6 +1168,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,return 0 ; pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); +tlb_start_ptes (&cp->tlb);for (; addr != end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;@@ -1190,6 +1191,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE );ClearPageReferenced (page); } +tlb_end_ptes (&cp->tlb);pte_unmap_unlock (pte -1 , ptl);cond_resched ();return 0 ;include/asm-generic/tlb.h +0
-0include/asm-generic/tlb.h line 61
Quoted 4 lines
include/asm-generic/tlb.h +1
-0@@ -58,6 +58,11 @@* Defaults to flushing at tlb_end_vma() to reset the range; helps when * there's large holes between the VMAs. * +* - tlb_start_ptes() / tlb_end_ptes; makr the start / end of PTEs change. Hello Nadav,
Quoted 1 line
short nid makr/mark
Thanks! I will fix it.
Peter Zijlstra
maintainerRe: [RFC 13/20] mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
In reply to Nadav AmitQuoted 17 lines · Nadav Amit
include/asm-generic/tlb.h +9
-1+#define tlb_start_ptes (tlb) \ + do { \ + struct mmu_gather *_tlb = (tlb); \ + \ + flush_tlb_batched_pending(_tlb->mm); \ + } while (0) + +static inline void tlb_end_ptes (struct mmu_gather *tlb) { }tlb_change_page_size (tlb,PAGE_SIZE ); orig_pte = pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl); -flush_tlb_batched_pending (mm); +tlb_start_ptes (tlb);arch_enter_lazy_mmu_mode ();for (; addr < end; pte++, addr +=PAGE_SIZE ) { ptent = *pte;
include/asm-generic/tlb.h static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 474
Quoted 7 lines
include/asm-generic/tlb.h +1
-0@@ -468,6 +468,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);if (pageout)reclaim_pages (&page_list);
I don't like how you're dubbling up on arch_*_lazy_mmu_mode(). It seems to me those should be folded into tlb_{start,end}_ptes().
Alternatively, the even more work approach would be to, add an optional @tlb argument to pte_offset_map_lock()/pte_unmap_unlock() and friends.
Re: [RFC 13/20] mm/tlb: introduce tlb_start_ptes() and tlb_end_ptes()
Quoted 31 lines · Peter Zijlstra
Quoted 17 lines · Nadav Amit
include/asm-generic/tlb.h +8
-0+#define tlb_start_ptes (tlb) \ + do { \ + struct mmu_gather *_tlb = (tlb); \ + \ + flush_tlb_batched_pending(_tlb->mm); \ + } while (0) + +static inline void tlb_end_ptes (struct mmu_gather *tlb) { }tlb_change_page_size(tlb, PAGE_SIZE); orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);(quoted diff) +1
-1- flush_tlb_batched_pending(mm); + tlb_start_ptes(tlb);arch_enter_lazy_mmu_mode(); for (; addr < end; pte++, addr += PAGE_SIZE) { ptent = *pte;mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 474
Quoted 7 lines
mm/madvise.c +1
-0@@ -468,6 +468,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, }arch_leave_lazy_mmu_mode (); +tlb_end_ptes (tlb);pte_unmap_unlock (orig_pte, ptl);if (pageout)reclaim_pages (&page_list);I don't like how you're dubbling up on arch_*_lazy_mmu_mode(). It seems to me those should be folded into tlb_{start,end}_ptes().
Quoted 2 lines
Alternatively, the even more work approach would be to, add an optional @tlb argument to pte_offset_map_lock()/pte_unmap_unlock() and friends.
Not too fund of the “even more approach”. I still have debts I need to pay to the kernel community on old patches that didn’t make it through.
I will fold arch_*_lazy_mmu_mode() as you suggested. Admittedly, I do not understand this arch_*_lazy_mmu_mode() very well - I would have assumed they would be needed only when PTEs are established, and in other cases the arch code will hook directly to the TLB flushing interface.
However, based on the code, it seems that powerpc does not even flush PTEs that are established (only removed/demoted). Probably I am missing something. I will just blindly fold it.
RFC 14/20 mm: move inc/dec_tlb_flush_pending() to mmu_gather.c
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Reduce the chances that inc/dec_tlb_flush_pending() will be abused by moving them into mmu_gather.c, which is more of their natural place. This also allows to reduce the clutter on mm_types.h.
--- include/linux/mm_types.h | 54 ---------------------------------------- mm/mmu_gather.c | 54 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 54 deletions(-)
include/linux/mm_types.h +0 -54
@@ -615,60 +615,6 @@ static inline void init_tlb_flush_pending(struct mm_struct *mm)atomic_set (&mm->tlb_flush_pending,0 ); } -static inline void inc_tlb_flush_pending (struct mm_struct *mm) -{ -atomic_inc (&mm->tlb_flush_pending); -/* -* The only time this value is relevant is when there are indeed pages -* to flush. And we'll only flush pages after changing them, which -* requires the PTL. -* -* So the ordering here is: -* -* atomic_inc(&mm->tlb_flush_pending); -* spin_lock(&ptl); -* ... -* set_pte_at(); -* spin_unlock(&ptl); -* -* spin_lock(&ptl) -* mm_tlb_flush_pending(); -* .... -* spin_unlock(&ptl); -* -* flush_tlb_range(); -* atomic_dec(&mm->tlb_flush_pending); -* -* Where the increment if constrained by the PTL unlock, it thus -* ensures that the increment is visible if the PTE modification is -* visible. After all, if there is no PTE modification, nobody cares -* about TLB flushes either. -* -* This very much relies on users (mm_tlb_flush_pending() and -* mm_tlb_flush_nested()) only caring about _specific_ PTEs (and -* therefore specific PTLs), because with SPLIT_PTE_PTLOCKS and RCpc -* locks (PPC) the unlock of one doesn't order against the lock of -* another PTL. -* -* The decrement is ordered by the flush_tlb_range(), such that -* mm_tlb_flush_pending() will not return false unless all flushes have -* completed. -*/ -} - -static inline void dec_tlb_flush_pending (struct mm_struct *mm) -{ -/* -* See inc_tlb_flush_pending(). -* -* This cannot be smp_mb__before_atomic() because smp_mb() simply does -* not order against TLB invalidate completion, which is what we need. -* -* Therefore we must rely on tlb_flush_*() to guarantee order. -*/ -atomic_dec (&mm->tlb_flush_pending); -} -static inline bool mm_tlb_flush_pending (struct mm_struct *mm) {/*
mm/mmu_gather.c +54 -0
60 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC 15/20 mm: detect deferred TLB flushes in vma granularity
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Currently, deferred TLB flushes are detected in the mm granularity: if there is any deferred TLB flush in the entire address space due to NUMA migration, pte_accessible() in x86 would return true, and ptep_clear_flush() would require a TLB flush. This would happen even if the PTE resides in a completely different vma.
Recent code changes and possible future enhancements might require to detect TLB flushes in finer granularity. Detection in finer granularity can also enable more aggressive TLB deferring in the future.
Record for each vma the last mm's TLB generation after the last deferred PTE/PMD change while the page-table lock is still held. Increase the mm generation before recording to indicate that a pending TLB flush is pending. Record in the mmu_gather struct the mm's TLB generation at the time in which the last TLB flushing was deferred.
Once the TLB flushing of deferred request takes place, use the deferred TLB generation that is recorded in mmu_gather. Detection of deferred TLB flushes is performed by checking whether the mm's completed TLB generation is the lower/equal than the mm's TLB generation. Architectures that use the TLB generation logic are required to perform a full TLB flush if they detect that a new TLB flush request "skips" a generation (as already done by x86 code).
To indicate that a deferred TLB flush takes place, increase the mm's TLB generation after updating the PTEs. However, try to avoid increasing the mm's generation after subsequent PTE updates, as increasing it again would lead to a full TLB flush once the deferred TLB flushes are performed (due to the "skipped" TLB generation). Therefore, if the mm generation did not change after subsequent PTE update, use the previous generation.
As multiple updates of the vma generation can be performed concurrently, use atomic operations to ensure that the TLB generation as recorded in the vma is the last (most recent) one.
Once a deferred TLB flush is eventually performed it might be redundant, if due to another TLB flush the deferred flush was performed (by doing a full TLB flush once detecting the "skipped" generation). This case can be detected if the deferred TLB generation, as recorded in mmu_gather was already completed. However, we do not record deferred PUD/P4D flushes, and freeing tables also requires a flush on cores in lazy TLB mode. In such cases a TLB flush is needed even if the mm's completed TLB generation indicates the flush was already "performed".
--- arch/x86/include/asm/tlb.h | 18 ++++-- arch/x86/include/asm/tlbflush.h | 5 ++ arch/x86/mm/tlb.c | 14 ++++- include/asm-generic/tlb.h | 104 ++++++++++++++++++++++++++++++-- include/linux/mm_types.h | 19 ++++++ mm/mmap.c | 1 + mm/mmu_gather.c | 3 + 7 files changed, 150 insertions(+), 14 deletions(-)
arch/x86/include/asm/tlb.h +13 -5
@@ -9,15 +9,23 @@ static inline void tlb_flush(struct mmu_gather *tlb);static inline void tlb_flush (struct mmu_gather *tlb) { -unsigned long start =0UL , end =TLB_FLUSH_ALL ;unsigned int stride_shift =tlb_get_unmap_shift (tlb); -if (!tlb->fullmm && !tlb->need_flush_all) { - start = tlb->start; - end = tlb->end; +/* Perform full flush when needed */ +if (tlb->fullmm || tlb->need_flush_all) { +flush_tlb_mm_range (tlb->mm,0 ,TLB_FLUSH_ALL , stride_shift, + tlb->freed_tables); +return ; } -flush_tlb_mm_range (tlb->mm, start, end, stride_shift, tlb->freed_tables); +/* Check if flush was already performed */ +if (!tlb->freed_tables && !tlb->cleared_puds && + !tlb->cleared_p4ds && +atomic64_read (&tlb->mm->tlb_gen_completed) > tlb->defer_gen) +return ; + +flush_tlb_mm_range_gen (tlb->mm, tlb->start, tlb->end, stride_shift, + tlb->freed_tables, tlb->defer_gen); }/*
arch/x86/include/asm/tlbflush.h +5 -0
@@ -225,6 +225,11 @@ void flush_tlb_others(const struct cpumask *cpumask, : PAGE_SHIFT, false)extern void flush_tlb_all (void ); + +extern void flush_tlb_mm_range_gen (struct mm_struct *mm,unsigned long start, +unsigned long end,unsigned int stride_shift, +bool freed_tables,u64 gen); +extern void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start,unsigned long end,unsigned int stride_shift,bool freed_tables);
arch/x86/mm/tlb.c +11 -3
@@ -883,12 +883,11 @@ static inline void put_flush_tlb_info(void)#endif } -void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, +void flush_tlb_mm_range_gen (struct mm_struct *mm,unsigned long start,unsigned long end,unsigned int stride_shift, -bool freed_tables) +bool freed_tables,u64 new_tlb_gen) {struct flush_tlb_info *info; -u64 new_tlb_gen;int cpu; cpu =get_cpu ();
@@ -923,6 +922,15 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,put_cpu (); } +void flush_tlb_mm_range (struct mm_struct *mm,unsigned long start, +unsigned long end,unsigned int stride_shift, +bool freed_tables) +{ +u64 new_tlb_gen =inc_mm_tlb_gen (mm); + +flush_tlb_mm_range_gen (mm, start, end, stride_shift, freed_tables, + new_tlb_gen); +}static void do_flush_tlb_all (void *info) {
include/asm-generic/tlb.h +98 -6
162 lines not on this page · raw message
include/linux/mm_types.h +19 -0
@@ -367,6 +367,9 @@ struct vm_area_struct {#endif #ifdef CONFIG_NUMA struct mempolicy *vm_policy;/* NUMA policy for the VMA */ +#endif +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +atomic64_t defer_tlb_gen;/* Deferred TLB flushes generation */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout;
@@ -628,6 +631,21 @@ static inline bool mm_tlb_flush_pending(struct mm_struct *mm)return atomic_read (&mm->tlb_flush_pending); } +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +static inline bool pte_tlb_flush_pending (struct vm_area_struct *vma,pte_t *pte) +{ +struct mm_struct *mm = vma->vm_mm; + +return atomic64_read (&vma->defer_tlb_gen) <atomic64_read (&mm->tlb_gen_completed); +} + +static inline bool pmd_tlb_flush_pending (struct vm_area_struct *vma,pmd_t *pmd) +{ +struct mm_struct *mm = vma->vm_mm; + +return atomic64_read (&vma->defer_tlb_gen) <atomic64_read (&mm->tlb_gen_completed); +} +#else /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ static inline bool pte_tlb_flush_pending (struct vm_area_struct *vma,pte_t *pte) {return mm_tlb_flush_pending (vma->vm_mm);
@@ -637,6 +655,7 @@ static inline bool pmd_tlb_flush_pending(struct vm_area_struct *vma, pmd_t *pmd) {return mm_tlb_flush_pending (vma->vm_mm); } +#endif /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ static inline bool mm_tlb_flush_nested (struct mm_struct *mm) {
mm/mmap.c +1 -0
@@ -3337,6 +3337,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,get_file (new_vma->vm_file);if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open (new_vma); +init_vma_tlb_generation (new_vma);vma_link (mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; }
mm/mmu_gather.c +3 -0
@@ -329,6 +329,9 @@ static void __tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,#endif tlb_table_init (tlb); +#ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS + tlb->defer_gen =0 ; +#endif #ifdef CONFIG_MMU_GATHER_PAGE_SIZE tlb->page_size =0 ;#endif
Signature · 1 line
2.25.1
Re: [RFC 15/20] mm: detect deferred TLB flushes in vma granularity
Quoted 9 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, deferred TLB flushes are detected in the mm granularity: if there is any deferred TLB flush in the entire address space due to NUMA migration,
pte_accessible()in x86 would return true, andptep_clear_flush()would require a TLB flush. This would happen even if the PTE resides in a completely different vma.
[ snip ]
Quoted 20 lines
include/asm-generic/tlb.h +20
-0+static inline void read_defer_tlb_flush_gen (struct mmu_gather *tlb) +{ +struct mm_struct *mm = tlb->mm; +u64 mm_gen; + +/* +* Any change of PTE before calling __track_deferred_tlb_flush() must be +* performed using RMW atomic operation that provides a memory barriers, +* such as ptep_modify_prot_start(). The barrier ensure the PTEs are +* written before the current generation is read, synchronizing +* (implicitly) with flush_tlb_mm_range(). +*/ +smp_mb__after_atomic (); + + mm_gen =atomic64_read (&mm->tlb_gen); + +/* +* This condition checks for both first deferred TLB flush and for other +* TLB pending or executed TLB flushes after the last table that we +* updated. In the latter case, we are going to skip a generation, which
include/asm-generic/tlb.h static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen) · line 573
Quoted 10 lines
include/asm-generic/tlb.h +10
-0+* would lead to a full TLB flush. This should therefore not cause +* correctness issues, and should not induce overheads, since anyhow in +* TLB storms it is better to perform full TLB flush. +*/ +if (mm_gen != tlb->defer_gen) { +VM_BUG_ON (mm_gen < tlb->defer_gen); + + tlb->defer_gen =inc_mm_tlb_gen (mm); + } +}
Andy’s comments managed to make me realize this code is wrong. We must call inc_mm_tlb_gen(mm) every time.
Otherwise, a CPU that saw the old tlb_gen and updated it in its local cpu_tlbstate on a context-switch. If the process was not running when the TLB flush was issued, no IPI will be sent to the CPU. Therefore, later switch_mm_irqs_off() back to the process will not flush the local TLB.
I need to think if there is a better solution. Multiple calls to inc_mm_tlb_gen() during deferred flushes would trigger a full TLB flush instead of one that is specific to the ranges, once the flush actually takes place. On x86 it’s practically a non-issue, since anyhow any update of more than 33-entries or so would cause a full TLB flush, but this is still ugly.
Re: [RFC 15/20] mm: detect deferred TLB flushes in vma granularity
Quoted 56 lines · Nadav Amit
Quoted 10 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Currently, deferred TLB flushes are detected in the mm granularity: if there is any deferred TLB flush in the entire address space due to NUMA migration,
pte_accessible()in x86 would return true, andptep_clear_flush()would require a TLB flush. This would happen even if the PTE resides in a completely different vma.[ snip ]
Quoted 20 lines
include/asm-generic/tlb.h +20
-0+static inline void read_defer_tlb_flush_gen (struct mmu_gather *tlb) +{ +struct mm_struct *mm = tlb->mm; +u64 mm_gen; + +/* +* Any change of PTE before calling __track_deferred_tlb_flush() must be +* performed using RMW atomic operation that provides a memory barriers, +* such as ptep_modify_prot_start(). The barrier ensure the PTEs are +* written before the current generation is read, synchronizing +* (implicitly) with flush_tlb_mm_range(). +*/ +smp_mb__after_atomic (); + + mm_gen =atomic64_read (&mm->tlb_gen); + +/* +* This condition checks for both first deferred TLB flush and for other +* TLB pending or executed TLB flushes after the last table that we +* updated. In the latter case, we are going to skip a generation, which include/asm-generic/tlb.h static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen) · line 573
Quoted 10 lines
include/asm-generic/tlb.h +10
-0+* would lead to a full TLB flush. This should therefore not cause +* correctness issues, and should not induce overheads, since anyhow in +* TLB storms it is better to perform full TLB flush. +*/ +if (mm_gen != tlb->defer_gen) { +VM_BUG_ON (mm_gen < tlb->defer_gen); + + tlb->defer_gen =inc_mm_tlb_gen (mm); + } +}Andy’s comments managed to make me realize this code is wrong. We must call
inc_mm_tlb_gen(mm)every time.Otherwise, a CPU that saw the old
tlb_genand updated it in its localcpu_tlbstateon a context-switch. If the process was not running when the TLB flush was issued, no IPI will be sent to the CPU. Therefore, laterswitch_mm_irqs_off()back to the process will not flush the local TLB.
Quoted 6 lines
I need to think if there is a better solution. Multiple calls to
inc_mm_tlb_gen()during deferred flushes would trigger a full TLB flush instead of one that is specific to the ranges, once the flush actually takes place. On x86 it’s practically a non-issue, since anyhow any update of more than 33-entries or so would cause a full TLB flush, but this is still ugly.
What if we had a per-mm ring buffer of flushes? When starting a flush, we would stick the range in the ring buffer and, when flushing, we would read the ring buffer to catch up. This would mostly replace the flush_tlb_info struct, and it would let us process multiple partial flushes together.
Re: [RFC 15/20] mm: detect deferred TLB flushes in vma granularity
Quoted 13 lines · Andy Lutomirski
Quoted 12 lines · Nadav Amit
Andy’s comments managed to make me realize this code is wrong. We must call
inc_mm_tlb_gen(mm)every time.Otherwise, a CPU that saw the old
tlb_genand updated it in its localcpu_tlbstateon a context-switch. If the process was not running when the TLB flush was issued, no IPI will be sent to the CPU. Therefore, laterswitch_mm_irqs_off()back to the process will not flush the local TLB.
Quoted 7 lines
Quoted 5 lines
I need to think if there is a better solution. Multiple calls to
inc_mm_tlb_gen()during deferred flushes would trigger a full TLB flush instead of one that is specific to the ranges, once the flush actually takes place. On x86 it’s practically a non-issue, since anyhow any update of more than 33-entries or so would cause a full TLB flush, but this is still ugly.What if we had a per-mm ring buffer of flushes? When starting a flush, we would stick the range in the ring buffer and, when flushing, we would read the ring buffer to catch up. This would mostly replace the
flush_tlb_infostruct, and it would let us process multiple partial flushes together.
I wanted to sleep on it, and went back and forth on whether it is the right direction, hence the late response.
I think that what you say make sense. I think that I even tried to do once something similar for some reason, but my memory plays tricks on me.
So tell me what you think on this ring-based solution. As you said, you keep per-mm ring of flush_tlb_info. When you queue an entry, you do something like:
#define RING_ENTRY_INVALID (0)
gen = inc_mm_tlb_gen(mm); struct flush_tlb_info *info = mm->ring[gen % RING_SIZE]; spin_lock(&mm->ring_lock); WRITE_ONCE(info->new_tlb_gen, RING_ENTRY_INVALID); smp_wmb(); info->start = start; info->end = end; info->stride_shift = stride_shift; info->freed_tables = freed_tables; smp_store_release(&info->new_tlb_gen, gen); spin_unlock(&mm->ring_lock);
When you flush you use the entry generation as a sequence lock. On overflow of the ring (i.e., sequence number mismatch) you perform a full flush:
for (gen = mm->tlb_gen_completed; gen < mm->tlb_gen; gen++) {
struct flush_tlb_info *info = &mm->ring[gen % RING_SIZE];
// detect overflow and invalid entries
if (smp_load_acquire(info->new_tlb_gen) != gen)
goto full_flush;
start = min(start, info->start);
end = max(end, info->end);
stride_shift = min(stride_shift, info->stride_shift);
freed_tables |= info.freed_tables;
smp_rmb();
// seqlock-like check that the information was not updated
if (READ_ONCE(info->new_tlb_gen) != gen)
goto full_flush;}
On x86 I suspect that performing a full TLB flush would anyhow be the best thing to do if there is more than a single entry. I am also not sure that it makes sense to check the ring from flush_tlb_func_common() (i.e., in each IPI handler) as it might cause cache thrashing.
Instead it may be better to do so from flush_tlb_mm_range(), when the flushes are initiated, and use an aggregated flush_tlb_info for the flush.
It may also be better to have the ring arch-independent, so it would resemble more of mmu_gather (the parts about the TLB flush information, without the freed pages stuff).
We can detect deferred TLB flushes either by storing “deferred_gen” in the page-tables/VMA (as I did) or by going over the ring, from tlb_gen_completed to tlb_gen, and checking for an overlap. I think page-tables would be most efficient/scalable, but perhaps going over the ring would be easier to understand logic.
Makes sense? Thoughts?
Re: [RFC 15/20] mm: detect deferred TLB flushes in vma granularity
Quoted 37 lines · Nadav Amit
Quoted 14 lines
Quoted 14 lines · Andy Lutomirski
Quoted 3 lines · Nadav Amit
Andy’s comments managed to make me realize this code is wrong. We must call
inc_mm_tlb_gen(mm)every time.Otherwise, a CPU that saw the old
tlb_genand updated it in its localcpu_tlbstateon a context-switch. If the process was not running when the TLB flush was issued, no IPI will be sent to the CPU. Therefore, laterswitch_mm_irqs_off()back to the process will not flush the local TLB.Quoted 7 lines
Quoted 5 lines
I need to think if there is a better solution. Multiple calls to
inc_mm_tlb_gen()during deferred flushes would trigger a full TLB flush instead of one that is specific to the ranges, once the flush actually takes place. On x86 it’s practically a non-issue, since anyhow any update of more than 33-entries or so would cause a full TLB flush, but this is still ugly.What if we had a per-mm ring buffer of flushes? When starting a flush, we would stick the range in the ring buffer and, when flushing, we would read the ring buffer to catch up. This would mostly replace the
flush_tlb_infostruct, and it would let us process multiple partial flushes together.I wanted to sleep on it, and went back and forth on whether it is the right direction, hence the late response.
I think that what you say make sense. I think that I even tried to do once something similar for some reason, but my memory plays tricks on me.
So tell me what you think on this ring-based solution. As you said, you keep per-mm ring of
flush_tlb_info. When you queue an entry, you do something like:
Quoted 5 lines
#define
RING_ENTRY_INVALID(0)gen =
inc_mm_tlb_gen(mm);
structflush_tlb_info*info = mm->ring[gen %RING_SIZE];spin_lock(&mm->ring_lock);
Once you are holding the lock, you should presumably check that the ring didn’t overflow while you were getting the lock — if new_tlb_gen > gen, abort.
Quoted 9 lines
WRITE_ONCE(info->new_tlb_gen, RING_ENTRY_INVALID);smp_wmb();
info->start= start;
info->end= end;
info->stride_shift=stride_shift;info->freed_tables=freed_tables;smp_store_release(&info->new_tlb_gen, gen);spin_unlock(&mm->ring_lock);
Seems reasonable. I’m curious how this ends up getting used.
Quoted 40 lines
When you flush you use the entry generation as a sequence lock. On overflow of the ring (i.e., sequence number mismatch) you perform a full flush:
for (gen = mm->tlb_gen_completed; gen < mm->tlb_gen; gen++) { struct flush_tlb_info *info = &mm->ring[gen % RING_SIZE]; // detect overflow and invalid entries if (smp_load_acquire(info->new_tlb_gen) != gen) goto full_flush; start = min(start, info->start); end = max(end, info->end); stride_shift = min(stride_shift, info->stride_shift); freed_tables |= info.freed_tables; smp_rmb(); // seqlock-like check that the information was not updated if (READ_ONCE(info->new_tlb_gen) != gen) goto full_flush;}
On x86 I suspect that performing a full TLB flush would anyhow be the best thing to do if there is more than a single entry. I am also not sure that it makes sense to check the ring from
flush_tlb_func_common()(i.e., in each IPI handler) as it might cause cache thrashing.Instead it may be better to do so from
flush_tlb_mm_range(), when the flushes are initiated, and use an aggregatedflush_tlb_infofor the flush.It may also be better to have the ring arch-independent, so it would resemble more of
mmu_gather(the parts about the TLB flush information, without the freed pages stuff).We can detect deferred TLB flushes either by storing “deferred_gen” in the page-tables/VMA (as I did) or by going over the ring, from
tlb_gen_completedtotlb_gen, and checking for an overlap. I think page-tables would be most efficient/scalable, but perhaps going over the ring would be easier to understand logic.Makes sense? Thoughts?
RFC 16/20 mm/tlb: per-page table generation tracking
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Detecting deferred TLB flushes per-VMA has two drawbacks:
1. It requires an atomic cmpxchg to record mm's TLB generation at the time of the last TLB flush, as two deferred TLB flushes on the same VMA can race.
2. It might be in coarse granularity for large VMAs.
On 64-bit architectures that have available space on page-struct, we can resolve these two drawbacks by recording the TLB generation at the time of the last deferred flush in page-struct of page-table whose TLB flushes were deferred.
Introduce a new CONFIG_PER_TABLE_DEFERRED_FLUSHES config option. Record when enabled the deferred TLB flush generation on page-struct, which is protected by the page-table lock.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 23 ++++++------ fs/proc/task_mmu.c | 6 ++-- include/asm-generic/tlb.h | 65 ++++++++++++++++++++++++++-------- include/linux/mm.h | 13 +++++++ include/linux/mm_types.h | 22 ++++++++++++ init/Kconfig | 7 ++++ mm/huge_memory.c | 2 +- mm/mapping_dirty_helpers.c | 4 +-- mm/mprotect.c | 2 +- 10 files changed, 113 insertions(+), 32 deletions(-)
arch/x86/Kconfig +1 -0
@@ -250,6 +250,7 @@ config X86 select X86_FEATURE_NAMES if PROC_FS select PROC_PID_ARCH_STATUS if PROC_FS select MAPPING_DIRTY_HELPERS + select PER_TABLE_DEFERRED_FLUSHES if X86_64 imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI config INSTRUCTION_DECODER
arch/x86/include/asm/pgtable.h +12 -11
@@ -774,17 +774,18 @@ static inline int pte_devmap(pte_t a) }#endif -#define pte_accessible pte_accessible -static inline bool pte_accessible (struct vm_area_struct *vma,pte_t *a) -{ -if (pte_flags (*a) & _PAGE_PRESENT) -return true; - -if ((pte_flags (*a) & _PAGE_PROTNONE) &&pte_tlb_flush_pending (vma, a)) -return true; - -return false; -} +#define pte_accessible (vma, a) \ + ({ \ + pte_t *_a = (a); \ + bool r = false; \ + \ + if (pte_flags(*_a) & _PAGE_PRESENT) \ + r = true; \ + else \ + r = ((pte_flags(*_a) & _PAGE_PROTNONE) && \ + pte_tlb_flush_pending((vma), _a)); \ + r; \ + })static inline int pmd_present (pmd_t pmd) {
fs/proc/task_mmu.c +3 -3
@@ -1157,7 +1157,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,/* Clear accessed and referenced bits. */ pmdp_test_and_clear_young (vma, addr, pmd);test_and_clear_page_young (page); -tlb_flush_pmd_range (&cp->tlb, addr,HPAGE_PMD_SIZE ); +tlb_flush_pmd_range (&cp->tlb, pmd, addr,HPAGE_PMD_SIZE );ClearPageReferenced (page); out:spin_unlock (ptl);
@@ -1174,7 +1174,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,if (cp->type ==CLEAR_REFS_SOFT_DIRTY ) {clear_soft_dirty (vma, addr, pte); -tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE ); +tlb_flush_pte_range (&cp->tlb, pte, addr,PAGE_SIZE );continue ; }
@@ -1188,7 +1188,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,/* Clear accessed and referenced bits. */ ptep_test_and_clear_young (vma, addr, pte);test_and_clear_page_young (page); -tlb_flush_pte_range (&cp->tlb, addr,PAGE_SIZE ); +tlb_flush_pte_range (&cp->tlb, pte, addr,PAGE_SIZE );ClearPageReferenced (page); }tlb_end_ptes (&cp->tlb);
include/asm-generic/tlb.h +51 -14
151 lines not on this page · raw message
include/linux/mm.h +13 -0
@@ -2208,11 +2208,21 @@ static inline void pgtable_init(void)pgtable_cache_init (); } +#ifdef CONFIG_PER_TABLE_DEFERRED_FLUSHES +static inline void page_table_tlb_gen_init (struct page *page) +{ + page->deferred_tlb_gen =0 ; +} +#else /* CONFIG_PER_TABLE_DEFERRED_FLUSHES */ +static inline void page_table_tlb_gen_init (struct page *page) { } +#endif /* CONFIG_PER_TABLE_DEFERRED_FLUSHES */ +static inline bool pgtable_pte_page_ctor (struct page *page) {if (!ptlock_init (page))return false;__SetPageTable (page); +page_table_tlb_gen_init (page);inc_lruvec_page_state (page,NR_PAGETABLE );return true; }
@@ -2221,6 +2231,7 @@ static inline void pgtable_pte_page_dtor(struct page *page) {ptlock_free (page);__ClearPageTable (page); +page_table_tlb_gen_init (page);dec_lruvec_page_state (page,NR_PAGETABLE ); }
@@ -2308,6 +2319,7 @@ static inline bool pgtable_pmd_page_ctor(struct page *page)if (!pmd_ptlock_init (page))return false;__SetPageTable (page); +page_table_tlb_gen_init (page);inc_lruvec_page_state (page,NR_PAGETABLE );return true; }
@@ -2316,6 +2328,7 @@ static inline void pgtable_pmd_page_dtor(struct page *page) {pmd_ptlock_free (page);__ClearPageTable (page); +page_table_tlb_gen_init (page);dec_lruvec_page_state (page,NR_PAGETABLE ); }
include/linux/mm_types.h +22 -0
@@ -148,6 +148,9 @@ struct page {pgtable_t pmd_huge_pte;/* protected by page->ptl */ unsigned long _pt_pad_2;/* mapping */ union { +#ifdef CONFIG_PER_TABLE_DEFERRED_FLUSHES +u64 deferred_tlb_gen;/* x86 non-pgd protected by page->ptl */ +#endif struct mm_struct *pt_mm;/* x86 pgds only */ atomic_t pt_frag_refcount;/* powerpc */ };
@@ -632,6 +635,7 @@ static inline bool mm_tlb_flush_pending(struct mm_struct *mm) } #ifdef CONFIG_ARCH_HAS_TLB_GENERATIONS +#ifndef CONFIG_PER_TABLE_DEFERRED_FLUSHES static inline bool pte_tlb_flush_pending(struct vm_area_struct *vma, pte_t *pte) { struct mm_struct *mm = vma->vm_mm;
@@ -645,6 +649,24 @@ static inline bool pmd_tlb_flush_pending(struct vm_area_struct *vma, pmd_t *pmd) return atomic64_read(&vma->defer_tlb_gen) < atomic64_read(&mm->tlb_gen_completed); } +#else /* CONFIG_PER_TABLE_DEFERRED_FLUSHES */ +#define pte_tlb_flush_pending (vma, pte) \ + ({ \ + struct mm_struct *mm = (vma)->vm_mm; \ + \ + (pte_to_page(pte))->deferred_tlb_gen < \ + atomic64_read(&mm->tlb_gen_completed); \ + }) + +#define pmd_tlb_flush_pending (vma, pmd) \ + ({ \ + struct mm_struct *mm = (vma)->vm_mm; \ + \ + (pmd_to_page(pmd))->deferred_tlb_gen < \ + atomic64_read(&mm->tlb_gen_completed); \ + }) + +#endif /* CONFIG_PER_TABLE_DEFERRED_FLUSHES */ #else /* CONFIG_ARCH_HAS_TLB_GENERATIONS */ static inline bool pte_tlb_flush_pending (struct vm_area_struct *vma,pte_t *pte) {
init/Kconfig +7 -0
@@ -857,6 +857,13 @@ config ARCH_WANT_AGGRESSIVE_TLB_FLUSH_BATCHING bool depends on !CONFIG_MMU_GATHER_NO_GATHER +# +# For architectures that prefer to save deferred TLB generations in the +# page-table instead of the VMA. +config PER_TABLE_DEFERRED_FLUSHES + bool + depends on ARCH_HAS_TLB_GENERATIONS && 64BIT + config CC_HAS_INT128 def_bool !$(cc-option,$(m64-flag) -D__SIZEOF_INT128__=0) && 64BIT
mm/huge_memory.c +1 -1
@@ -1886,7 +1886,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, entry =pmd_clear_uffd_wp (entry); } ret =HPAGE_PMD_NR ; -tlb_flush_pmd_range (tlb, addr,HPAGE_PMD_SIZE ); +tlb_flush_pmd_range (tlb, pmd, addr,HPAGE_PMD_SIZE );set_pmd_at (mm, addr, pmd, entry);BUG_ON (vma_is_anonymous (vma) && !preserve_write &&pmd_write (entry)); unlock:
mm/mapping_dirty_helpers.c +2 -2
@@ -48,7 +48,7 @@ static int wp_pte(pte_t *pte, unsigned long addr, unsigned long end, wpwalk->total++;if (pte_may_need_flush (old_pte, ptent)) -tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_flush_pte_range (&wpwalk->tlb, pte, addr,PAGE_SIZE );tlb_end_ptes (&wpwalk->tlb); }
@@ -110,7 +110,7 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,ptep_modify_prot_commit (walk->vma, addr, pte, old_pte, ptent); wpwalk->total++; -tlb_flush_pte_range (&wpwalk->tlb, addr,PAGE_SIZE ); +tlb_flush_pte_range (&wpwalk->tlb, pte, addr,PAGE_SIZE );tlb_end_ptes (&wpwalk->tlb);__set_bit (pgoff, cwalk->bitmap);
mm/mprotect.c +1 -1
@@ -140,7 +140,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent);if (pte_may_need_flush (oldpte, ptent)) -tlb_flush_pte_range (tlb, addr,PAGE_SIZE ); +tlb_flush_pte_range (tlb, pte, addr,PAGE_SIZE ); pages++; }else if (is_swap_pte (oldpte)) {swp_entry_t entry =pte_to_swp_entry (oldpte);
Signature · 1 line
2.25.1
RFC 17/20 mm/tlb: updated completed deferred TLB flush conditionally
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
If all the deferred TLB flushes were completed, there is no need to update the completed TLB flush. This update requires an atomic cmpxchg, so we would like to skip it.
To do so, save for each mm the last TLB generation in which TLB flushes were deferred. While saving this information requires another atomic cmpxchg, assume that deferred TLB flushes are less frequent than TLB flushes.
--- include/asm-generic/tlb.h | 23 ++++++++++++++++++----- include/linux/mm_types.h | 5 +++++ 2 files changed, 23 insertions(+), 5 deletions(-)
include/asm-generic/tlb.h +18 -5
@@ -536,6 +536,14 @@ static inline void tlb_update_generation(atomic64_t *gen, u64 new_gen)static inline void mark_mm_tlb_gen_done (struct mm_struct *mm,u64 gen) { +/* +* If we all the deferred TLB generations were completed, we can skip +* the update of tlb_gen_completed and save a few cycles on cmpxchg. +*/ +if (atomic64_read (&mm->tlb_gen_deferred) == +atomic64_read (&mm->tlb_gen_completed)) +return ; +/* * Update the completed generation to the new generation if the new * generation is greater than the previous one.
@@ -546,7 +554,7 @@ static inline void mark_mm_tlb_gen_done(struct mm_struct *mm, u64 gen)static inline void read_defer_tlb_flush_gen (struct mmu_gather *tlb) {struct mm_struct *mm = tlb->mm; -u64 mm_gen; +u64 mm_gen, new_gen;/* * Any change of PTE before calling __track_deferred_tlb_flush() must be
@@ -567,11 +575,16 @@ static inline void read_defer_tlb_flush_gen(struct mmu_gather *tlb)* correctness issues, and should not induce overheads, since anyhow in * TLB storms it is better to perform full TLB flush. */ -if (mm_gen != tlb->defer_gen) { -VM_BUG_ON (mm_gen < tlb->defer_gen); +if (mm_gen == tlb->defer_gen) +return ; - tlb->defer_gen =inc_mm_tlb_gen (mm); - } +VM_BUG_ON (mm_gen < tlb->defer_gen); + + new_gen =inc_mm_tlb_gen (mm); + tlb->defer_gen = new_gen; + +/* Update mm->tlb_gen_deferred */ +tlb_update_generation (&mm->tlb_gen_deferred, new_gen); }#ifndef CONFIG_PER_TABLE_DEFERRED_FLUSHES
include/linux/mm_types.h +5 -0
@@ -578,6 +578,11 @@ struct mm_struct {*/ atomic64_t tlb_gen; +/* +* The last TLB generation which was deferred. +*/ +atomic64_t tlb_gen_deferred; +/* * TLB generation which is guarnateed to be flushed, including * all the PTE changes that were performed before tlb_gen was
Signature · 1 line
2.25.1
RFC 18/20 mm: make mm_cpumask() volatile
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
mm_cpumask() is volatile: a bit might be turned on or off at any given moment, and it is not protected by any lock. While the kernel coding guidelines are very prohibitive against the use of volatile, not marking mm_cpumask() as volatile seems wrong.
Cpumask and bitmap manipulation functions may work fine, as they are allowed to use either the new or old value. Apparently they do, as no bugs were reported. However, the fact that mm_cpumask() is not volatile might lead to theoretical bugs due to compiler optimizations.
For example, cpumask_next() uses _find_next_bit(). A compiler might add to _find_next_bit() invented loads that would cause __ffs() to run on different value than the one read before. Consequently, if something like that happens, the result might be a CPU that was neither set on the old nor the new mask. I could not find what might go wrong in such a case, but it seems as an improper result.
Mark mm_cpumask() result as volatile and propagate the "volatile" qualifier according to the compiler shouts.
--- arch/arm/include/asm/bitops.h | 4 ++-- arch/x86/hyperv/mmu.c | 2 +- arch/x86/include/asm/paravirt_types.h | 2 +- arch/x86/include/asm/tlbflush.h | 2 +- arch/x86/mm/tlb.c | 4 ++-- arch/x86/xen/mmu_pv.c | 2 +- include/asm-generic/bitops/find.h | 8 ++++---- include/linux/bitmap.h | 16 +++++++-------- include/linux/cpumask.h | 28 +++++++++++++-------------- include/linux/mm_types.h | 4 ++-- include/linux/smp.h | 6 +++--- kernel/smp.c | 8 ++++---- lib/bitmap.c | 8 ++++---- lib/cpumask.c | 8 ++++---- lib/find_bit.c | 10 +++++----- 15 files changed, 56 insertions(+), 56 deletions(-)
arch/arm/include/asm/bitops.h +2 -2
@@ -162,8 +162,8 @@ extern int _test_and_change_bit(int nr, volatile unsigned long * p);*/ extern int _find_first_zero_bit_le (const unsigned long *p,unsigned size);extern int _find_next_zero_bit_le (const unsigned long *p,int size,int offset); -extern int _find_first_bit_le (const unsigned long *p,unsigned size); -extern int _find_next_bit_le (const unsigned long *p,int size,int offset); +extern int _find_first_bit_le (const volatile unsigned long *p,unsigned size); +extern int _find_next_bit_le (const volatile unsigned long *p,int size,int offset);/* * Big endian assembly bitops. nr = 0 -> byte 3 bit 0.
arch/x86/hyperv/mmu.c +1 -1
@@ -52,7 +52,7 @@ static inline int fill_gva_list(u64 gva_list[], int offset,return gva_n - offset; } -static void hyperv_flush_tlb_others (const struct cpumask *cpus, +static void hyperv_flush_tlb_others (const volatile struct cpumask *cpus,const struct flush_tlb_info *info) {int cpu, vcpu, gva_n, max_gvas;
arch/x86/include/asm/paravirt_types.h +1 -1
@@ -201,7 +201,7 @@ struct pv_mmu_ops {void (*flush_tlb_user)(void );void (*flush_tlb_kernel)(void );void (*flush_tlb_one_user)(unsigned long addr); -void (*flush_tlb_others)(const struct cpumask *cpus, +void (*flush_tlb_others)(const volatile struct cpumask *cpus,const struct flush_tlb_info *info);void (*tlb_remove_table)(struct mmu_gather *tlb,void *table);
arch/x86/include/asm/tlbflush.h +1 -1
@@ -208,7 +208,7 @@ struct flush_tlb_info {void flush_tlb_local (void );void flush_tlb_one_user (unsigned long addr);void flush_tlb_one_kernel (unsigned long addr); -void flush_tlb_others (const struct cpumask *cpumask, +void flush_tlb_others (const volatile struct cpumask *cpumask,const struct flush_tlb_info *info);#ifdef CONFIG_PARAVIRT
arch/x86/mm/tlb.c +2 -2
@@ -796,7 +796,7 @@ static bool tlb_is_not_lazy(int cpu, void *data)return !per_cpu (cpu_tlbstate.is_lazy, cpu); } -STATIC_NOPV void native_flush_tlb_others(const struct cpumask *cpumask, +STATIC_NOPV void native_flush_tlb_others(const volatile struct cpumask *cpumask,const struct flush_tlb_info *info) {count_vm_tlb_event (NR_TLB_REMOTE_FLUSH );
@@ -824,7 +824,7 @@ STATIC_NOPV void native_flush_tlb_others(const struct cpumask *cpumask, (void *)info, 1, cpumask); } -void flush_tlb_others (const struct cpumask *cpumask, +void flush_tlb_others (const volatile struct cpumask *cpumask,const struct flush_tlb_info *info) {__flush_tlb_others (cpumask, info);
arch/x86/xen/mmu_pv.c +1 -1
@@ -1247,7 +1247,7 @@ static void xen_flush_tlb_one_user(unsigned long addr)preempt_enable (); } -static void xen_flush_tlb_others (const struct cpumask *cpus, +static void xen_flush_tlb_others (const volatile struct cpumask *cpus,const struct flush_tlb_info *info) {struct {
include/asm-generic/bitops/find.h +4 -4
@@ -12,8 +12,8 @@* Returns the bit number for the next set bit * If no bits are set, returns @size. */ -extern unsigned long find_next_bit (const unsigned long *addr,unsigned long - size,unsigned long offset); +extern unsigned long find_next_bit (const volatile unsigned long *addr, +unsigned long size,unsigned long offset);#endif #ifndef find_next_and_bit
@@ -27,8 +27,8 @@ extern unsigned long find_next_bit(const unsigned long *addr, unsigned long* Returns the bit number for the next set bit * If no bits are set, returns @size. */ -extern unsigned long find_next_and_bit (const unsigned long *addr1, -const unsigned long *addr2,unsigned long size, +extern unsigned long find_next_and_bit (const volatile unsigned long *addr1, +const volatile unsigned long *addr2,unsigned long size,unsigned long offset);#endif
include/linux/bitmap.h +8 -8
@@ -141,8 +141,8 @@ extern void __bitmap_shift_left(unsigned long *dst, const unsigned long *src,extern void bitmap_cut (unsigned long *dst,const unsigned long *src,unsigned int first,unsigned int cut,unsigned int nbits); -extern int __bitmap_and (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,unsigned int nbits); +extern int __bitmap_and (unsigned long *dst,const volatile unsigned long *bitmap1, +const volatile unsigned long *bitmap2,unsigned int nbits);extern void __bitmap_or (unsigned long *dst,const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits); extern void __bitmap_xor(unsigned long *dst, const unsigned long *bitmap1,
@@ -152,8 +152,8 @@ extern int __bitmap_andnot(unsigned long *dst, const unsigned long *bitmap1,extern void __bitmap_replace (unsigned long *dst,const unsigned long *old,const unsigned long *new,const unsigned long *mask,unsigned int nbits); -extern int __bitmap_intersects (const unsigned long *bitmap1, -const unsigned long *bitmap2,unsigned int nbits); +extern int __bitmap_intersects (const volatile unsigned long *bitmap1, +const volatile unsigned long *bitmap2,unsigned int nbits);extern int __bitmap_subset (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits);extern int __bitmap_weight (const unsigned long *bitmap,unsigned int nbits);
@@ -278,8 +278,8 @@ extern void bitmap_to_arr32(u32 *buf, const unsigned long *bitmap, (const unsigned long *) (bitmap), (nbits))#endif -static inline int bitmap_and (unsigned long *dst,const unsigned long *src1, -const unsigned long *src2,unsigned int nbits) +static inline int bitmap_and (unsigned long *dst,const volatile unsigned long *src1, +const volatile unsigned long *src2,unsigned int nbits) {if (small_const_nbits (nbits))return (*dst = *src1 & *src2 &BITMAP_LAST_WORD_MASK (nbits)) !=0 ;
@@ -359,8 +359,8 @@ static inline bool bitmap_or_equal(const unsigned long *src1,return !(((*src1 | *src2) ^ *src3) &BITMAP_LAST_WORD_MASK (nbits)); } -static inline int bitmap_intersects (const unsigned long *src1, -const unsigned long *src2,unsigned int nbits) +static inline int bitmap_intersects (const volatile unsigned long *src1, +const volatile unsigned long *src2,unsigned int nbits) {if (small_const_nbits (nbits))return ((*src1 & *src2) &BITMAP_LAST_WORD_MASK (nbits)) !=0 ;
include/linux/cpumask.h +14 -14
@@ -158,7 +158,7 @@ static inline unsigned int cpumask_last(const struct cpumask *srcp) }/* Valid inputs for n are -1 and 0. */ -static inline unsigned int cpumask_next (int n,const struct cpumask *srcp) +static inline unsigned int cpumask_next (int n,const volatile struct cpumask *srcp) {return n+1 ; }
@@ -169,8 +169,8 @@ static inline unsigned int cpumask_next_zero(int n, const struct cpumask *srcp) }static inline unsigned int cpumask_next_and (int n, -const struct cpumask *srcp, -const struct cpumask *andp) +const volatile struct cpumask *srcp, +const volatile struct cpumask *andp) {return n+1 ; }
@@ -183,7 +183,7 @@ static inline unsigned int cpumask_next_wrap(int n, const struct cpumask *mask, }/* cpu must be a valid cpu, ie 0, so there's no other choice. */ -static inline unsigned int cpumask_any_but (const struct cpumask *mask, +static inline unsigned int cpumask_any_but (const volatile struct cpumask *mask,unsigned int cpu) {return 1 ;
@@ -235,7 +235,7 @@ static inline unsigned int cpumask_last(const struct cpumask *srcp)return find_last_bit (cpumask_bits (srcp), nr_cpumask_bits); } -unsigned int cpumask_next (int n,const struct cpumask *srcp); +unsigned int cpumask_next (int n,const volatile struct cpumask *srcp);/** * cpumask_next_zero - get the next unset cpu in a cpumask
@@ -252,8 +252,8 @@ static inline unsigned int cpumask_next_zero(int n, const struct cpumask *srcp)return find_next_zero_bit (cpumask_bits (srcp), nr_cpumask_bits, n+1 ); } -int cpumask_next_and (int n,const struct cpumask *,const struct cpumask *); -int cpumask_any_but (const struct cpumask *mask,unsigned int cpu); +int cpumask_next_and (int n,const volatile struct cpumask *,const volatile struct cpumask *); +int cpumask_any_but (const volatile struct cpumask *mask,unsigned int cpu);unsigned int cpumask_local_spread (unsigned int i,int node);int cpumask_any_and_distribute (const struct cpumask *src1p,const struct cpumask *src2p);
@@ -335,7 +335,7 @@ extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool* @cpu: cpu number (< nr_cpu_ids) * @dstp: the cpumask pointer */ -static inline void cpumask_set_cpu (unsigned int cpu,struct cpumask *dstp) +static inline void cpumask_set_cpu (unsigned int cpu,volatile struct cpumask *dstp) {set_bit (cpumask_check (cpu),cpumask_bits (dstp)); }
@@ -351,7 +351,7 @@ static inline void __cpumask_set_cpu(unsigned int cpu, struct cpumask *dstp)* @cpu: cpu number (< nr_cpu_ids) * @dstp: the cpumask pointer */ -static inline void cpumask_clear_cpu (int cpu,struct cpumask *dstp) +static inline void cpumask_clear_cpu (int cpu,volatile struct cpumask *dstp) {clear_bit (cpumask_check (cpu),cpumask_bits (dstp)); }
@@ -368,7 +368,7 @@ static inline void __cpumask_clear_cpu(int cpu, struct cpumask *dstp)* * Returns 1 if @cpu is set in @cpumask, else returns 0 */ -static inline int cpumask_test_cpu (int cpu,const struct cpumask *cpumask) +static inline int cpumask_test_cpu (int cpu,const volatile struct cpumask *cpumask) {return test_bit (cpumask_check (cpu),cpumask_bits ((cpumask))); }
@@ -428,8 +428,8 @@ static inline void cpumask_clear(struct cpumask *dstp)* If *@dstp is empty, returns 0, else returns 1 */ static inline int cpumask_and (struct cpumask *dstp, -const struct cpumask *src1p, -const struct cpumask *src2p) +const volatile struct cpumask *src1p, +const volatile struct cpumask *src2p) {return bitmap_and (cpumask_bits (dstp),cpumask_bits (src1p),cpumask_bits (src2p), nr_cpumask_bits);
@@ -521,8 +521,8 @@ static inline bool cpumask_or_equal(const struct cpumask *src1p,* @src1p: the first input * @src2p: the second input */ -static inline bool cpumask_intersects (const struct cpumask *src1p, -const struct cpumask *src2p) +static inline bool cpumask_intersects (const volatile struct cpumask *src1p, +const volatile struct cpumask *src2p) {return bitmap_intersects (cpumask_bits (src1p),cpumask_bits (src2p), nr_cpumask_bits);
include/linux/mm_types.h +2 -2
@@ -611,9 +611,9 @@ static inline void mm_init_cpumask(struct mm_struct *mm) }/* Future-safe accessor for struct mm_struct's cpu_vm_mask. */ -static inline cpumask_t *mm_cpumask (struct mm_struct *mm) +static inline volatile cpumask_t *mm_cpumask (struct mm_struct *mm) { -return (struct cpumask *)&mm->cpu_bitmap; +return (volatile struct cpumask *)&mm->cpu_bitmap; }struct mmu_gather ;
include/linux/smp.h +3 -3
@@ -59,7 +59,7 @@ void on_each_cpu(smp_call_func_t func, void *info, int wait);* Call a function on processors specified by mask, which might include * the local one. */ -void on_each_cpu_mask (const struct cpumask *mask,smp_call_func_t func, +void on_each_cpu_mask (const volatile struct cpumask *mask,smp_call_func_t func,void *info,bool wait);/*
@@ -71,7 +71,7 @@ void on_each_cpu_cond(smp_cond_func_t cond_func, smp_call_func_t func, void *info, bool wait);void on_each_cpu_cond_mask (smp_cond_func_t cond_func,smp_call_func_t func, -void *info,bool wait,const struct cpumask *mask); +void *info,bool wait,const volatile struct cpumask *mask);int smp_call_function_single_async (int cpu,call_single_data_t *csd);
@@ -118,7 +118,7 @@ extern void smp_cpus_done(unsigned int max_cpus);* Call a function on all other processors */ void smp_call_function (smp_call_func_t func,void *info,int wait); -void smp_call_function_many (const struct cpumask *mask, +void smp_call_function_many (const volatile struct cpumask *mask,smp_call_func_t func,void *info,bool wait); int smp_call_function_any(const struct cpumask *mask,
kernel/smp.c +4 -4
@@ -604,7 +604,7 @@ int smp_call_function_any(const struct cpumask *mask, }EXPORT_SYMBOL_GPL (smp_call_function_any); -static void smp_call_function_many_cond (const struct cpumask *mask, +static void smp_call_function_many_cond (const volatile struct cpumask *mask,smp_call_func_t func,void *info,bool wait,smp_cond_func_t cond_func) {
@@ -705,7 +705,7 @@ static void smp_call_function_many_cond(const struct cpumask *mask,* hardware interrupt handler or from a bottom half handler. Preemption * must be disabled when calling this function. */ -void smp_call_function_many (const struct cpumask *mask, +void smp_call_function_many (const volatile struct cpumask *mask,smp_call_func_t func,void *info,bool wait) {smp_call_function_many_cond (mask, func, info, wait,NULL );
@@ -853,7 +853,7 @@ EXPORT_SYMBOL(on_each_cpu);* exception is that it may be used during early boot while * early_boot_irqs_disabled is set. */ -void on_each_cpu_mask (const struct cpumask *mask,smp_call_func_t func, +void on_each_cpu_mask (const volatile struct cpumask *mask,smp_call_func_t func,void *info,bool wait) {int cpu =get_cpu ();
@@ -892,7 +892,7 @@ EXPORT_SYMBOL(on_each_cpu_mask);* from a hardware interrupt handler or from a bottom half handler. */ void on_each_cpu_cond_mask (smp_cond_func_t cond_func,smp_call_func_t func, -void *info,bool wait,const struct cpumask *mask) +void *info,bool wait,const volatile struct cpumask *mask) {int cpu =get_cpu ();
lib/bitmap.c +4 -4
@@ -235,8 +235,8 @@ void bitmap_cut(unsigned long *dst, const unsigned long *src, }EXPORT_SYMBOL (bitmap_cut); -int __bitmap_and (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,unsigned int bits) +int __bitmap_and (unsigned long *dst,const volatile unsigned long *bitmap1, +const volatile unsigned long *bitmap2,unsigned int bits) {unsigned int k;unsigned int lim = bits/BITS_PER_LONG ;
@@ -301,8 +301,8 @@ void __bitmap_replace(unsigned long *dst, }EXPORT_SYMBOL (__bitmap_replace); -int __bitmap_intersects (const unsigned long *bitmap1, -const unsigned long *bitmap2,unsigned int bits) +int __bitmap_intersects (const volatile unsigned long *bitmap1, +const volatile unsigned long *bitmap2,unsigned int bits) {unsigned int k, lim = bits/BITS_PER_LONG ;for (k =0 ; k < lim; ++k)
lib/cpumask.c +4 -4
@@ -15,7 +15,7 @@* * Returns >= nr_cpu_ids if no further cpus set. */ -unsigned int cpumask_next (int n,const struct cpumask *srcp) +unsigned int cpumask_next (int n,const volatile struct cpumask *srcp) {/* -1 is a legal arg here. */ if (n !=-1 )
@@ -32,8 +32,8 @@ EXPORT_SYMBOL(cpumask_next);* * Returns >= nr_cpu_ids if no further cpus set in both. */ -int cpumask_next_and (int n,const struct cpumask *src1p, -const struct cpumask *src2p) +int cpumask_next_and (int n,const volatile struct cpumask *src1p, +const volatile struct cpumask *src2p) {/* -1 is a legal arg here. */ if (n !=-1 )
@@ -51,7 +51,7 @@ EXPORT_SYMBOL(cpumask_next_and);* Often used to find any cpu but smp_processor_id() in a mask. * Returns >= nr_cpu_ids if no cpus set. */ -int cpumask_any_but (const struct cpumask *mask,unsigned int cpu) +int cpumask_any_but (const volatile struct cpumask *mask,unsigned int cpu) {unsigned int i;
lib/find_bit.c +5 -5
@@ -29,8 +29,8 @@* searching it for one bits. * - The optional "addr2", which is anded with "addr1" if present. */ -static unsigned long _find_next_bit (const unsigned long *addr1, -const unsigned long *addr2,unsigned long nbits, +static unsigned long _find_next_bit (const volatile unsigned long *addr1, +const volatile unsigned long *addr2,unsigned long nbits,unsigned long start,unsigned long invert,unsigned long le) {unsigned long tmp, mask;
@@ -74,7 +74,7 @@ static unsigned long _find_next_bit(const unsigned long *addr1,/* * Find the next set bit in a memory region. */ -unsigned long find_next_bit (const unsigned long *addr,unsigned long size, +unsigned long find_next_bit (const volatile unsigned long *addr,unsigned long size,unsigned long offset) {return _find_next_bit (addr,NULL , size, offset,0UL ,0 );
@@ -92,8 +92,8 @@ EXPORT_SYMBOL(find_next_zero_bit);#endif #if !defined(find_next_and_bit) -unsigned long find_next_and_bit (const unsigned long *addr1, -const unsigned long *addr2,unsigned long size, +unsigned long find_next_and_bit (const volatile unsigned long *addr1, +const volatile unsigned long *addr2,unsigned long size,unsigned long offset) {return _find_next_bit (addr1, addr2, size, offset,0UL ,0 );
Signature · 1 line
2.25.1
RFC 19/20 lib/cpumask: introduce cpumask_atomic_or()
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
Introduce cpumask_atomic_or() and bitmask_atomic_or() to allow to perform atomic or operations atomically on cpumasks. This will be used by the next patch.
To be more efficient, skip atomic operations when no changes are needed.
--- include/linux/bitmap.h | 5 +++++ include/linux/cpumask.h | 12 ++++++++++++ lib/bitmap.c | 25 +++++++++++++++++++++++++ 3 files changed, 42 insertions(+)
include/linux/bitmap.h +5 -0
@@ -76,6 +76,7 @@* bitmap_to_arr32(buf, src, nbits) Copy nbits from buf to u32[] dst * bitmap_get_value8(map, start) Get 8bit value from map at start * bitmap_set_value8(map, value, start) Set 8bit value to map at start +* bitmap_atomic_or(dst, src, nbits) *dst |= *src (atomically) * * Note, bitmap_zero() and bitmap_fill() operate over the region of * unsigned longs, that is, bits behind bitmap till the unsigned long
@@ -577,6 +578,10 @@ static inline void bitmap_set_value8(unsigned long *map, unsigned long value, map[index] |= value << offset; } +extern void bitmap_atomic_or (volatile unsigned long *dst, +const volatile unsigned long *bitmap,unsigned int bits); + +#endif /* __ASSEMBLY__ */ #endif /* __LINUX_BITMAP_H */
include/linux/cpumask.h +12 -0
@@ -699,6 +699,18 @@ static inline unsigned int cpumask_size(void)return BITS_TO_LONGS (nr_cpumask_bits) *sizeof (long ); } +/** +* cpumask_atomic_or - *dstp |= *srcp (*dstp is set atomically) +* @dstp: the cpumask result (and source which is or'd) +* @srcp: the source input +*/ +static inline void cpumask_atomic_or (volatile struct cpumask *dstp, +const volatile struct cpumask *srcp) +{ +bitmap_atomic_or (cpumask_bits (dstp),cpumask_bits (srcp), + nr_cpumask_bits); +} +/* * cpumask_var_t: struct cpumask for stack usage. *
lib/bitmap.c +25 -0
@@ -1310,3 +1310,28 @@ void bitmap_to_arr32(u32 *buf, const unsigned long *bitmap, unsigned int nbits)EXPORT_SYMBOL (bitmap_to_arr32);#endif + +void bitmap_atomic_or (volatile unsigned long *dst, +const volatile unsigned long *bitmap,unsigned int bits) +{ +unsigned int k; +unsigned int nr =BITS_TO_LONGS (bits); + +for (k =0 ; k < nr; k++) { +unsigned long src = bitmap[k]; + +/* +* Skip atomic operations when no bits are changed. Do not use +* bitmap[k] directly to avoid redundant loads as bitmap +* variable is volatile. +*/ +if (!(src & ~dst[k])) +continue ; + +if (BITS_PER_LONG ==64 ) +atomic64_or (src, (atomic64_t *)&dst[k]); +else +atomic_or (src, (atomic_t *)&dst[k]); + } +} +EXPORT_SYMBOL (bitmap_atomic_or);
Signature · 1 line
2.25.1
RFC 20/20 mm/rmap: avoid potential races
In reply to Nadav AmitFrom: Nadav Amit <[email protected]>
flush_tlb_batched_pending() appears to have a theoretical race:tlb_flush_batched is being cleared after the TLB flush, and if in between another core calls set_tlb_ubc_flush_pending() and sets the pending TLB flush indication, this indication might be lost. Holding the page-table lock when SPLIT_LOCK is set cannot eliminate this race.
The current batched TLB invalidation scheme therefore does not seem viable or easily repairable.
Introduce a new scheme, in which a cpumask is maintained for pending batched TLB flushes. When a full TLB flush is performed clear the corresponding bit on the CPU the performs the TLB flush.
This scheme is only suitable for architectures that use IPIs for TLB shootdowns. As x86 is the only architecture that currently uses batched TLB flushes, this is not an issue.
--- arch/x86/include/asm/tlbbatch.h | 15 ------------ arch/x86/include/asm/tlbflush.h | 2 +- arch/x86/mm/tlb.c | 18 ++++++++++----- include/linux/mm.h | 7 ++++++ include/linux/mm_types_task.h | 13 ----------- mm/rmap.c | 41 ++++++++++++++++----------------- 6 files changed, 40 insertions(+), 56 deletions(-) delete mode 100644 arch/x86/include/asm/tlbbatch.h
arch/x86/include/asm/tlbbatch.h deleted +0 -15
@@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ARCH_X86_TLBBATCH_H -#define _ARCH_X86_TLBBATCH_H - -#include <linux/cpumask.h> - -struct arch_tlbflush_unmap_batch { -/* -* Each bit set is a CPU that potentially has a TLB entry for one of -* the PFNs being flushed.. -*/ -struct cpumask cpumask; -}; - -#endif /* _ARCH_X86_TLBBATCH_H */
arch/x86/include/asm/tlbflush.h +1 -1
@@ -240,7 +240,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)flush_tlb_mm_range (vma->vm_mm, a, a +PAGE_SIZE ,PAGE_SHIFT , false); } -extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_flush (void );static inline bool pte_may_need_flush (pte_t oldpte,pte_t newpte) {
arch/x86/mm/tlb.c +12 -6
@@ -760,8 +760,15 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f,count_vm_tlb_events (NR_TLB_LOCAL_FLUSH_ONE , nr_invalidate);trace_tlb_flush (reason, nr_invalidate); }else { +int cpu =smp_processor_id (); +/* Full flush. */ flush_tlb_local (); + +/* If there are batched TLB flushes, mark they are done */ +if (cpumask_test_cpu (cpu, &tlb_flush_batched_cpumask)) +cpumask_clear_cpu (cpu, &tlb_flush_batched_cpumask); +if (local)count_vm_tlb_event (NR_TLB_LOCAL_FLUSH_ALL );trace_tlb_flush (reason,TLB_FLUSH_ALL );
@@ -1143,21 +1150,20 @@ static const struct flush_tlb_info full_flush_tlb_info = { .end = TLB_FLUSH_ALL, }; -void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch) +void arch_tlbbatch_flush (void ) {int cpu =get_cpu (); -if (cpumask_test_cpu (cpu, &batch->cpumask)) { +if (cpumask_test_cpu (cpu, &tlb_flush_batched_cpumask)) {lockdep_assert_irqs_enabled ();local_irq_disable ();flush_tlb_func_local (&full_flush_tlb_info,TLB_LOCAL_SHOOTDOWN );local_irq_enable (); } -if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids) -flush_tlb_others (&batch->cpumask, &full_flush_tlb_info); - -cpumask_clear (&batch->cpumask); +if (cpumask_any_but (&tlb_flush_batched_cpumask, cpu) < nr_cpu_ids) +flush_tlb_others (&tlb_flush_batched_cpumask, + &full_flush_tlb_info);/* * We cannot call mark_mm_tlb_gen_done() since we do not know which
include/linux/mm.h +7 -0
@@ -3197,5 +3197,12 @@ unsigned long wp_shared_mapping_range(struct address_space *mapping,extern int sysctl_nr_trim_pages; +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +extern volatile cpumask_t tlb_flush_batched_cpumask; +void tlb_batch_init (void ); +#else +static inline void tlb_batch_init (void ) { } +#endif +#endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */
include/linux/mm_types_task.h +0 -13
@@ -15,10 +15,6 @@#include <asm/page.h> -#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH -#include <asm/tlbbatch.h> -#endif -#define USE_SPLIT_PTE_PTLOCKS (NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS)#define USE_SPLIT_PMD_PTLOCKS (USE_SPLIT_PTE_PTLOCKS && \ IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK))
@@ -75,15 +71,6 @@ struct page_frag {/* Track pages that require TLB flushes */ struct tlbflush_unmap_batch {#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH -/* -* The arch code makes the following promise: generic code can modify a -* PTE, then call arch_tlbbatch_add_mm() (which internally provides all -* needed barriers), then call arch_tlbbatch_flush(), and the entries -* will be flushed on all CPUs by the time that arch_tlbbatch_flush() -* returns. -*/ -struct arch_tlbflush_unmap_batch arch; -/* True if a flush is needed. */ bool flush_required;
mm/rmap.c +20 -21
76 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [RFC 20/20] mm/rmap: avoid potential races
Hi, Nadav,
Quoted 7 lines · Nadav Amit
From: Nadav Amit <[email protected]>
flush_tlb_batched_pending()appears to have a theoretical race:tlb_flush_batchedis being cleared after the TLB flush, and if in between another core callsset_tlb_ubc_flush_pending()and sets the pending TLB flush indication, this indication might be lost. Holding the page-table lock whenSPLIT_LOCKis set cannot eliminate this race.
Recently, when I read the corresponding code, I find the exact same race too. Do you still think the race is possible at least in theory? If so, why hasn't your fix been merged?
Quoted 2 lines
The current batched TLB invalidation scheme therefore does not seem viable or easily repairable.
I have some idea to fix this without too much code. If necessary, I will send it out.
Best Regards,
Huang, Ying
Re: [RFC 20/20] mm/rmap: avoid potential races
Quoted 16 lines · Ying
Hi, Nadav,
Quoted 7 lines · Nadav Amit
From: Nadav Amit <[email protected]>
flush_tlb_batched_pending()appears to have a theoretical race:tlb_flush_batchedis being cleared after the TLB flush, and if in between another core callsset_tlb_ubc_flush_pending()and sets the pending TLB flush indication, this indication might be lost. Holding the page-table lock whenSPLIT_LOCKis set cannot eliminate this race.
Quoted 3 lines
Recently, when I read the corresponding code, I find the exact same race too. Do you still think the race is possible at least in theory? If so, why hasn't your fix been merged?
I think the race is possible. It didn’t get merged, IIRC, due to some addressable criticism and lack of enthusiasm from other people, and my laziness/busy-ness.
Quoted 6 lines
Quoted 3 lines
The current batched TLB invalidation scheme therefore does not seem viable or easily repairable.
I have some idea to fix this without too much code. If necessary, I will send it out.
Arguably, it would be preferable to have a small back-portable fix for this issue specifically. Just try to ensure that you do not introduce performance overheads. Any solution should be clear about its impact on additional TLB flushes on the worst-case scenario and the number of additional atomic operations that would be required.
Re: [RFC 20/20] mm/rmap: avoid potential races
Quoted 14 lines · Nadav Amit
Quoted 14 lines · Ying
Hi, Nadav,
Quoted 7 lines · Nadav Amit
From: Nadav Amit <[email protected]>
flush_tlb_batched_pending()appears to have a theoretical race:tlb_flush_batchedis being cleared after the TLB flush, and if in between another core callsset_tlb_ubc_flush_pending()and sets the pending TLB flush indication, this indication might be lost. Holding the page-table lock whenSPLIT_LOCKis set cannot eliminate this race.
Quoted 7 lines
Quoted 3 lines
Recently, when I read the corresponding code, I find the exact same race too. Do you still think the race is possible at least in theory? If so, why hasn't your fix been merged?
I think the race is possible. It didn’t get merged, IIRC, due to some addressable criticism and lack of enthusiasm from other people, and my laziness/busy-ness.
Got it! Thanks your information!
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
The current batched TLB invalidation scheme therefore does not seem viable or easily repairable.
I have some idea to fix this without too much code. If necessary, I will send it out.
Arguably, it would be preferable to have a small back-portable fix for this issue specifically. Just try to ensure that you do not introduce performance overheads. Any solution should be clear about its impact on additional TLB flushes on the worst-case scenario and the number of additional atomic operations that would be required.
Sure. Will do that.
Best Regards,
Huang, Ying
Andy Lutomirski
maintainerQuoted 12 lines · Nadav Amit
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using
mmu_gather(e.g.,zap_page_range()).2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g.,
change_protection_range()).
Quoted 6 lines
3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (
move_ptes()).5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (
try_to_unmap_one()on x86).
Are you referring to the arch_tlbbatch_add_mm/flush mechanism?
Quoted 14 lines · Andy Lutomirski
Quoted 11 lines · Nadav Amit
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using
mmu_gather(e.g.,zap_page_range()).2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g.,
change_protection_range()).
Quoted 8 lines
Quoted 6 lines
3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (
move_ptes()).5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (
try_to_unmap_one()on x86).Are you referring to the arch_tlbbatch_add_mm/flush mechanism?
Yes.
Nicholas Piggin
maintainerExcerpts from Nadav Amit's message of January 31, 2021 10:11 am:
Quoted 28 lines
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using
mmu_gather(e.g.,zap_page_range()).2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g.,
change_protection_range()).3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (
move_ptes()).5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (
try_to_unmap_one()on x86).It seems that (1)-(4) can be consolidated. In addition, it seems that
(5) is racy. It also seems there can be many redundant TLB flushes, and potentially TLB-shootdown storms, for instance during batched reclamation (usingtry_to_unmap_one()) if at the same timemmu_gatherdefers TLB flushes.More aggressive TLB batching may be possible, but this patch-set does not add such batching. The proposed changes would enable such batching in a later time.
Quoted 6 lines
Admittedly, I do not understand how things are not broken today, which frightens me to make further batching before getting things in order. For instance, why is ok for
zap_pte_range()to batch dirty-PTE flushes for each page-table (but not in greater granularity). Can'tClearPageDirty()be called before the flush, causing writes afterClearPageDirty()and before the flush to be lost?
Because it's holding the page table lock which stops page_mkclean from cleaning the page. Or am I misunderstanding the question?
I'll go through the patches a bit more closely when they all come through. Sparc and powerpc of course need the arch lazy mode to get per-page/pte information for operations that are not freeing pages, which is what mmu gather is designed for.
I wouldn't mind using a similar API so it's less of a black box when reading generic code, but it might not quite fit the mmu gather API exactly (most of these paths don't want a full mmu_gather on stack).
Quoted 16 lines
This patch-set therefore performs the following changes:
1. Change mprotect,
task_mmuandmapping_dirty_helpersto usemmu_gatherinstead of {inc|dec}_tlb_flush_pending().2. Avoid TLB flushes if PTE permission is not demoted.
3. Cleans up
mmu_gatherto be less arch-dependant.4. Uses mm's generations to track in finer granularity, either per-VMA or per page-table, whether a pending
mmu_gatheroperation is outstanding. This should allow to avoid some TLB flushes when KSM or memory reclamation takes place while another operation such asmunmap()ormprotect()is running.
Quoted 3 lines
5. Changes
try_to_unmap_one()flushing scheme, as the current seems broken to track in a bitmap which CPUs have outstanding TLB flushes instead of having a flag.
Putting fixes first, and cleanups and independent patches (like #2) next would help with getting stuff merged and backported.
Thanks,
Nick
Quoted 38 lines · Nicholas Piggin
Excerpts from Nadav Amit's message of January 31, 2021 10:11 am:
Quoted 28 lines
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using
mmu_gather(e.g.,zap_page_range()).2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g.,
change_protection_range()).3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (
move_ptes()).5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (
try_to_unmap_one()on x86).It seems that (1)-(4) can be consolidated. In addition, it seems that
(5) is racy. It also seems there can be many redundant TLB flushes, and potentially TLB-shootdown storms, for instance during batched reclamation (usingtry_to_unmap_one()) if at the same timemmu_gatherdefers TLB flushes.More aggressive TLB batching may be possible, but this patch-set does not add such batching. The proposed changes would enable such batching in a later time.
Quoted 6 lines
Admittedly, I do not understand how things are not broken today, which frightens me to make further batching before getting things in order. For instance, why is ok for
zap_pte_range()to batch dirty-PTE flushes for each page-table (but not in greater granularity). Can'tClearPageDirty()be called before the flush, causing writes afterClearPageDirty()and before the flush to be lost?
Quoted 2 lines
Because it's holding the page table lock which stops
page_mkcleanfrom cleaning the page. Or am I misunderstanding the question?
Thanks. I understood this part. Looking again at the code, I now understand my confusion: I forgot that the reverse mapping is removed after the PTE is zapped.
Makes me wonder whether it is ok to defer the TLB flush to tlb_finish_mmu(), by performing set_page_dirty() for the batched pages when needed in tlb_finish_mmu() [ i.e., by marking for each batched page whether set_page_dirty() should be issued for that page while collecting them ].
Quoted 4 lines
I'll go through the patches a bit more closely when they all come through. Sparc and powerpc of course need the arch lazy mode to get per-page/pte information for operations that are not freeing pages, which is what mmu gather is designed for.
IIUC you mean any PTE change requires a TLB flush. Even setting up a new PTE where no previous PTE was set, right?
Quoted 3 lines
I wouldn't mind using a similar API so it's less of a black box when reading generic code, but it might not quite fit the mmu gather API exactly (most of these paths don't want a full
mmu_gatheron stack).
I see your point. It may be possible to create two mmu_gather structs: a small one that only holds the flush information and another that also holds the pages.
Quoted 15 lines
Quoted 15 lines
This patch-set therefore performs the following changes:
1. Change mprotect,
task_mmuandmapping_dirty_helpersto usemmu_gatherinstead of {inc|dec}_tlb_flush_pending().2. Avoid TLB flushes if PTE permission is not demoted.
3. Cleans up
mmu_gatherto be less arch-dependant.4. Uses mm's generations to track in finer granularity, either per-VMA or per page-table, whether a pending
mmu_gatheroperation is outstanding. This should allow to avoid some TLB flushes when KSM or memory reclamation takes place while another operation such asmunmap()ormprotect()is running.
Quoted 6 lines
Quoted 3 lines
5. Changes
try_to_unmap_one()flushing scheme, as the current seems broken to track in a bitmap which CPUs have outstanding TLB flushes instead of having a flag.Putting fixes first, and cleanups and independent patches (like #2) next would help with getting stuff merged and backported.
I tried to do it mostly this way. There are some theoretical races which I did not manage (or try hard enough) to create, so I did not include in the “fixes” section. I will restructure the patch-set according to the feedback.
Thanks,
Nadav
Quoted 43 lines · Nadav Amit
Quoted 39 lines · Nicholas Piggin
Excerpts from Nadav Amit's message of January 31, 2021 10:11 am:
Quoted 28 lines
From: Nadav Amit <[email protected]>
There are currently (at least?) 5 different TLB batching schemes in the kernel:
1. Using
mmu_gather(e.g.,zap_page_range()).2. Using {inc|dec}_tlb_flush_pending() to inform other threads on the ongoing deferred TLB flush and flushing the entire range eventually (e.g.,
change_protection_range()).3. arch_{enter|leave}_lazy_mmu_mode() for sparc and powerpc (and Xen?).
4. Batching per-table flushes (
move_ptes()).5. By setting a flag on that a deferred TLB flush operation takes place, flushing when (
try_to_unmap_one()on x86).It seems that (1)-(4) can be consolidated. In addition, it seems that
(5) is racy. It also seems there can be many redundant TLB flushes, and potentially TLB-shootdown storms, for instance during batched reclamation (usingtry_to_unmap_one()) if at the same timemmu_gatherdefers TLB flushes.More aggressive TLB batching may be possible, but this patch-set does not add such batching. The proposed changes would enable such batching in a later time.
Quoted 6 lines
Admittedly, I do not understand how things are not broken today, which frightens me to make further batching before getting things in order. For instance, why is ok for
zap_pte_range()to batch dirty-PTE flushes for each page-table (but not in greater granularity). Can'tClearPageDirty()be called before the flush, causing writes afterClearPageDirty()and before the flush to be lost?Quoted 2 lines
Because it's holding the page table lock which stops
page_mkcleanfrom cleaning the page. Or am I misunderstanding the question?
Quoted 8 lines
Thanks. I understood this part. Looking again at the code, I now understand my confusion: I forgot that the reverse mapping is removed after the PTE is zapped.
Makes me wonder whether it is ok to defer the TLB flush to
tlb_finish_mmu(), by performingset_page_dirty()for the batched pages when needed intlb_finish_mmu()[ i.e., by marking for each batched page whetherset_page_dirty()should be issued for that page while collecting them ].
Correcting myself (I hope): no we cannot do so, since the buffers might be remove from the page at that point.
Peter Zijlstra
maintainerQuoted 9 lines · Nadav Amit
Quoted 6 lines · Nicholas Piggin
I'll go through the patches a bit more closely when they all come through. Sparc and powerpc of course need the arch lazy mode to get per-page/pte information for operations that are not freeing pages, which is what mmu gather is designed for.
IIUC you mean any PTE change requires a TLB flush. Even setting up a new PTE where no previous PTE was set, right?
These are the HASH architectures. Their hardware doesn't walk the page-tables, but it consults a hash-table to resolve page translations.
They _MUST_ flush the entries under the PTL to avoid ever seeing conflicting information, which will make them really unhappy. They can do this because they have TLBI broadcast.
There's a few more details I'm sure, but those seem to have slipped from my mind.
Nicholas Piggin
maintainerExcerpts from Peter Zijlstra's message of February 1, 2021 10:44 pm:
Quoted 10 lines · Nadav Amit
Quoted 9 lines
Quoted 6 lines · Nicholas Piggin
I'll go through the patches a bit more closely when they all come through. Sparc and powerpc of course need the arch lazy mode to get per-page/pte information for operations that are not freeing pages, which is what mmu gather is designed for.
IIUC you mean any PTE change requires a TLB flush. Even setting up a new PTE where no previous PTE was set, right?
In cases of increasing permissiveness of access, yes it may want to update the "TLB" (read hash table) to avoid taking hash table faults.
But whatever the reason for the flush, there may have to be more data carried than just the virtual address range and/or physical pages.
If you clear out the PTE then you have no guarantee of actually being able to go back and address the the in-memory or in-hardware translation structures to update them, depending on what exact scheme is used (powerpc probably could if all page sizes were the same, but THP or 64k/4k sub pages would throw a spanner in those works).
Quoted 2 lines
These are the HASH architectures. Their hardware doesn't walk the page-tables, but it consults a hash-table to resolve page translations.
Yeah, it's very cool in a masochistic way.
I actually don't know if it's worth doing a big rework of it, as much as I'd like to. Rather than just keep it in place and eventually dismantling some of the go-fast hooks from core code if we can one day deprecate it in favour of the much easier radix mode.
The whole thing is like a big steam train, years ago Paul and Ben and Anton and co got the boiler stoked up and set all the valves just right so it runs unbelievably well for what it's actually doing but look at it the wrong way and the whole thing could blow up. (at least that's what it feels like to me probably because I don't know the code that well).
Sparc could probably do the same, not sure about Xen. I don't suppose vmware is intending to add any kind of paravirt mode related to this stuff?
Thanks,
Nick